AI带你省钱旅游！精准预测民宿房源价格！( 二 ) _生活百科

机器学习实战 | 机器学习特征工程最全解读

字段清洗因为数据中的字段众多，有些字段比较乱，我们需要做一些数据清洗的工作，数据包含一些带有URL的列，对最后的预测作用不大，我们把它们清洗掉。

# 删除url字段def drop_function(df):df = df.drop(columns=['listing_url', 'description', 'host_thumbnail_url', 'host_picture_url', 'latitude', 'longitude', 'picture_url', 'host_url', 'host_location', 'neighbourhood', 'neighbourhood_cleansed', 'host_about', 'has_availability', 'availability_30', 'availability_60', 'availability_90', 'availability_365', 'calendar_last_scraped'])return dfgm_df = drop_function(gm_listings)

删除过后的数据如下，干净很多

文章插图
缺失值处理数据中也包含了一些缺失值，我们对它们进行分析处理：
# 查看缺失值百分比(gm_df.isnull().sum()/gm_df.shape[0])* 100得到如下结果

id0.000000scrape_id0.000000last_scraped0.000000name0.000000neighborhood_overview41.266741host_id0.000000host_name0.000000host_since0.000000host_response_time10.212054host_response_rate10.212054host_acceptance_rate5.636161host_is_superhost0.000000host_neighbourhood91.657366host_listings_count0.000000host_total_listings_count0.000000host_verifications0.000000host_has_profile_pic0.000000host_identity_verified0.000000neighbourhood_group_cleansed0.000000property_type0.000000room_type0.000000accommodates0.000000bathrooms100.000000bathrooms_text0.306920bedrooms4.687500beds2.120536amenities0.000000price0.000000minimum_nights0.000000maximum_nights0.000000minimum_minimum_nights0.000000maximum_minimum_nights0.000000minimum_maximum_nights0.000000maximum_maximum_nights0.000000minimum_nights_avg_ntm0.000000maximum_nights_avg_ntm0.000000calendar_updated100.000000number_of_reviews0.000000number_of_reviews_ltm0.000000number_of_reviews_l30d0.000000first_review19.810268last_review19.810268review_scores_rating19.810268review_scores_accuracy20.089286review_scores_cleanliness20.089286review_scores_checkin20.089286review_scores_communication20.089286review_scores_location20.089286review_scores_value20.089286license100.000000instant_bookable0.000000calculated_host_listings_count0.000000calculated_host_listings_count_entire_homes0.000000calculated_host_listings_count_private_rooms0.000000calculated_host_listings_count_shared_rooms0.000000reviews_per_month19.810268dtype: float64

我们分几种不同的比例情况对缺失值进行处理：

高缺失比例的字段，如license、calendar_updated、bathrooms、host_neighborhood等包含90%以上的NaN值，包括neighborhood overview是41%的NaN，并且包含文本数据。我们会直接剔除这些字段。
数值型字段，缺失不多的情况下，我们用字段平均值进行填充。这保证了这些值的分布被保留下来。这些列包括bedrooms、beds、review_scores_rating、review_scores_accuracy和其他打分字段。
类别型字段，像bathrooms_text和host_response_time，我们用众数进行填充。

# 剔除高缺失比例字段def drop_function_2(df):df = df.drop(columns=['license', 'calendar_updated', 'bathrooms', 'host_neighbourhood', 'neighborhood_overview'])return dfgm_df = drop_function_2(gm_df)# 均值填充def input_mean(df, column_list):for columns in column_list:df[columns].fillna(value = https://www.huyubaike.com/biancheng/df[columns].mean(), inplace=True)return dfcolumn_list = ['review_scores_rating', 'review_scores_accuracy', 'review_scores_cleanliness','review_scores_checkin', 'review_scores_communication', 'review_scores_location','review_scores_value', 'reviews_per_month','bedrooms', 'beds']gm_df = input_mean(gm_df, column_list)# 众数填充def input_mode(df, column_list):for columns in column_list:df[columns].fillna(value = https://www.huyubaike.com/biancheng/df[columns].mode()[0], inplace=True)return dfcolumn_list = ['first_review', 'last_review', 'bathrooms_text', 'host_acceptance_rate','host_response_rate', 'host_response_time']gm_df = input_mode(gm_df, column_list)

字段编码host_is_superhost 和 has_availability 等列对应的字符串含义为 true 或 false，我们对其编码替换为0或1 。

gm_df = gm_df.replace({'host_is_superhost': 't', 'host_has_profile_pic': 't', 'host_identity_verified': 't', 'has_availability': 't', 'instant_bookable': 't'}, 1)gm_df = gm_df.replace({'host_is_superhost': 'f', 'host_has_profile_pic': 'f', 'host_identity_verified': 'f', 'has_availability': 'f', 'instant_bookable': 'f'}, 0)

我们查看下替换后的数据分布
gm_df['host_is_superhost'].value_counts()