【独家发布】【2016新书】Foundations for Analytics with Python: From Non-Programmer to Hack

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

如果喜欢该文档，欢迎订阅【2016新书】文库，https://bbs.pinggu.org/forum.php?mod=collection&action=view&ctid=3187

图书名称：Foundations for Analytics with Python: From Non-Programmer to Hacker
作者：Clinton W. Brownley
出版社：O’Reilly Media
页数：356
出版时间：2016                           
语言：English
格式：pdf
内容简介：
If you’re like many of Excel’s 750 million users, you want to do more with your data—like repeating similar analyses over hundreds of files, or combining data in many files for analysis at one time. This practical guide shows ambitious non-programmers how to automate and scale the processing and analysis of data in different formats—by using Python.

After author Clinton Brownley takes you through Python basics, you’ll be able to write simple scripts for processing data in spreadsheets as well as databases. You’ll also learn how to use several Python modules for parsing files, grouping data, and producing statistics. No programming experience is necessary.

回复免费：

本帖隐藏的内容

Foundations for Analytics with Python_ From Non-Programmer to Hacke.pdf (6.39 MB)

2016-8-25 19:50:45 上传

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏7 回帖

关键词：Foundations foundation Programmer Analytics Programme action Maria

1. #!/usr/bin/env python3
   
2. import numpy as np
   
3. import pandas as pd
   
4. import seaborn as sns
   
5. import matplotlib.pyplot as plt
   
6. import statsmodels.api as sm
   
7. import statsmodels.formula.api as smf
   
8. 
   
9. # Read the data set into a pandas DataFrame
   
10. churn = pd.read_csv('churn.csv', sep=',', header=0)
    
11. 
    
12. churn.columns = [heading.lower() for heading in \
    
13. churn.columns.str.replace(' ', '_').str.replace("\'", "").str.strip('?')]
    
14. 
    
15. churn['churn01'] = np.where(churn['churn'] == 'True.', 1., 0.)
    
16. print(churn.head())
    
17. print(churn.describe())
    
18. 
    
19. 
    
20. # Calculate descriptive statistics for grouped data
    
21. print(churn.groupby(['churn'])[['day_charge', 'eve_charge', 'night_charge', 'intl_charge', 'account_length', 'custserv_calls']].agg(['count', 'mean', 'std']))
    
22. 
    
23. # Specify different statistics for different variables
    
24. print(churn.groupby(['churn']).agg({'day_charge' : ['mean', 'std'],
    
25.                                 'eve_charge' : ['mean', 'std'],
    
26.                                 'night_charge' : ['mean', 'std'],
    
27.                                 'intl_charge' : ['mean', 'std'],
    
28.                                 'account_length' : ['count', 'min', 'max'],
    
29.                                 'custserv_calls' : ['count', 'min', 'max']}))
    
30. 
    
31. # Create total_charges, split it into 5 groups, and
    
32. # calculate statistics for each of the groups
    
33. churn['total_charges'] = churn['day_charge'] + churn['eve_charge'] + \
    
34.                                                  churn['night_charge'] + churn['intl_charge']
    
35. factor_cut = pd.cut(churn.total_charges, 5, precision=2)
    
36. def get_stats(group):
    
37.         return {'min' : group.min(), 'max' : group.max(),
    
38.                         'count' : group.count(), 'mean' : group.mean(),
    
39.                         'std' : group.std()}
    
40. grouped = churn.custserv_calls.groupby(factor_cut)
    
41. print(grouped.apply(get_stats).unstack())
    
42. 
    
43. # Split account_length into quantiles and
    
44. # calculate statistics for each of the quantiles
    
45. factor_qcut = pd.qcut(churn.account_length, [0., 0.25, 0.5, 0.75, 1.])
    
46. grouped = churn.custserv_calls.groupby(factor_qcut)
    
47. print(grouped.apply(get_stats).unstack())
    
48. 
    
49. # Create binary/dummy indicator variables for intl_plan and vmail_plan
    
50. # and join them with the churn column in a new DataFrame
    
51. intl_dummies = pd.get_dummies(churn['intl_plan'], prefix='intl_plan')
    
52. vmail_dummies = pd.get_dummies(churn['vmail_plan'], prefix='vmail_plan')
    
53. churn_with_dummies = churn[['churn']].join([intl_dummies, vmail_dummies])
    
54. print(churn_with_dummies.head())
    
55. 
    
56. # Split total_charges into quartiles, create binary indicator variables
    
57. # for each of the quartiles, and add them to the churn DataFrame
    
58. qcut_names = ['1st_quartile', '2nd_quartile', '3rd_quartile', '4th_quartile']
    
59. total_charges_quartiles = pd.qcut(churn.total_charges, 4, labels=qcut_names)
    
60. dummies = pd.get_dummies(total_charges_quartiles, prefix='total_charges')
    
61. churn_with_dummies = churn.join(dummies)
    
62. print(churn_with_dummies.head())
    
63. 
    
64. # Create pivot tables
    
65. print(churn.pivot_table(['total_charges'], index=['churn', 'custserv_calls']))
    
66. print(churn.pivot_table(['total_charges'], index=['churn'], columns=['custserv_calls']))
    
67. print(churn.pivot_table(['total_charges'], index=['custserv_calls'], columns=['churn'], \
    
68.                                                 aggfunc='mean', fill_value='NaN', margins=True))
    
69. 
    
70. # Fit a logistic regression model
    
71. dependent_variable = churn['churn01']
    
72. independent_variables = churn[['account_length', 'custserv_calls', 'total_charges']]
    
73. independent_variables_with_constant = sm.add_constant(independent_variables, prepend=True)
    
74. logit_model = sm.Logit(dependent_variable, independent_variables_with_constant).fit()
    
75. #logit_model = smf.glm(output_variable, input_variables, family=sm.families.Binomial()).fit()
    
76. print(logit_model.summary())
    
77. print("\nQuantities you can extract from the result:\n%s" % dir(logit_model))
    
78. print("\nCoefficients:\n%s" % logit_model.params)
    
79. print("\nCoefficient Std Errors:\n%s" % logit_model.bse)
    
80. #logit_marginal_effects = logit_model.get_margeff(method='dydx', at='overall')
    
81. #print(logit_marginal_effects.summary())
    
82. 
    
83. print("\ninvlogit(-7.2205 + 0.0012*mean(account_length) + 0.4443*mean(custserv_calls) + 0.0729*mean(total_charges))")
    
84. 
    
85. def inverse_logit(model_formula):
    
86.         from math import exp
    
87.         return (1.0 / (1.0 + exp(-model_formula)))*100.0
    
88. 
    
89. at_means = float(logit_model.params[0]) + \
    
90.         float(logit_model.params[1])*float(churn['account_length'].mean()) + \
    
91.         float(logit_model.params[2])*float(churn['custserv_calls'].mean()) + \
    
92.         float(logit_model.params[3])*float(churn['total_charges'].mean())
    
93. 
    
94. print(churn['account_length'].mean())
    
95. print(churn['custserv_calls'].mean())
    
96. print(churn['total_charges'].mean())
    
97. print(at_means)
    
98. print("Probability of churn when independent variables are at their mean values: %.2f" % inverse_logit(at_means))
    
99. 
    
100. cust_serv_mean = float(logit_model.params[0]) + \
     
101.         float(logit_model.params[1])*float(churn['account_length'].mean()) + \
     
102.         float(logit_model.params[2])*float(churn['custserv_calls'].mean()) + \
     
103.         float(logit_model.params[3])*float(churn['total_charges'].mean())
     
104.                
     
105. cust_serv_mean_minus_one = float(logit_model.params[0]) + \
     
106.                 float(logit_model.params[1])*float(churn['account_length'].mean()) + \
     
107.                 float(logit_model.params[2])*float(churn['custserv_calls'].mean()-1.0) + \
     
108.                 float(logit_model.params[3])*float(churn['total_charges'].mean())
     
109. 
     
110. print(cust_serv_mean)
     
111. print(churn['custserv_calls'].mean()-1.0)
     
112. print(cust_serv_mean_minus_one)
     
113. print("Probability of churn when account length changes by 1: %.2f" % (inverse_logit(cust_serv_mean) - inverse_logit(cust_serv_mean_minus_one)))
     
114. 
     
115. # Predict churn for "new" observations
     
116. new_observations = churn.ix[churn.index.isin(xrange(10)), independent_variables.columns]
     
117. new_observations_with_constant = sm.add_constant(new_observations, prepend=True)
     
118. y_predicted = logit_model.predict(new_observations_with_constant)
     
119. y_predicted_rounded = [round(score, 2) for score in y_predicted]
     
120. print(y_predicted_rounded)
     
121. 
     
122. # Fit a logistic regression model
     
123. output_variable = churn['churn01']
     
124. vars_to_keep = churn[['account_length', 'custserv_calls', 'total_charges']]
     
125. inputs_standardized = (vars_to_keep - vars_to_keep.mean()) / vars_to_keep.std()
     
126. input_variables = sm.add_constant(inputs_standardized, prepend=False)
     
127. logit_model = sm.Logit(output_variable, input_variables).fit()
     
128. #logit_model = smf.glm(output_variable, input_variables, family=sm.families.Binomial()).fit()
     
129. print(logit_model.summary())
     
130. print(logit_model.params)
     
131. print(logit_model.bse)
     
132. #logit_marginal_effects = logit_model.get_margeff(method='dydx', at='overall')
     
133. #print(logit_marginal_effects.summary())
     
134. 
     
135. # Predict output value for a new observation based on its mean standardized input values
     
136. input_variables = [0., 0., 0., 1.]
     
137. predicted_value = logit_model.predict(input_variables)
     
138. print("Predicted value: %.5f") % predicted_value

复制代码

1. #!/usr/bin/env python3
   
2. import sys
   
3. from datetime import date
   
4. from xlrd import open_workbook, xldate_as_tuple
   
5. from xlwt import Workbook
   
6. 
   
7. input_file = sys.argv[1]
   
8. output_file = sys.argv[2]
   
9. 
   
10. output_workbook = Workbook()
    
11. output_worksheet = output_workbook.add_sheet('selected_columns_all_worksheets')
    
12. 
    
13. my_columns = ['Customer Name', 'Sale Amount']
    
14.        
    
15. first_worksheet = True
    
16. with open_workbook(input_file) as workbook:
    
17.         data = [my_columns]
    
18.         index_of_cols_to_keep = []
    
19.         for worksheet in workbook.sheets():
    
20.                 if first_worksheet:
    
21.                         header = worksheet.row_values(0)
    
22.                         for column_index in range(len(header)):
    
23.                                 if header[column_index] in my_columns:
    
24.                                         index_of_cols_to_keep.append(column_index)
    
25.                         first_worksheet = False
    
26.                 for row_index in range(1, worksheet.nrows):
    
27.                         row_list = []
    
28.                         for column_index in index_of_cols_to_keep:       
    
29.                                 cell_value = worksheet.cell_value(row_index, column_index)
    
30.                                 cell_type = worksheet.cell_type(row_index, column_index)
    
31.                                 if cell_type == 3:
    
32.                                         date_cell = xldate_as_tuple(cell_value,workbook.datemode)
    
33.                                         date_cell = date(*date_cell[0:3]).strftime('%m/%d/%Y')
    
34.                                         row_list.append(date_cell)
    
35.                                 else:
    
36.                                         row_list.append(cell_value)
    
37.                         data.append(row_list)
    
38. 
    
39.         for list_index, output_list in enumerate(data):
    
40.                 for element_index, element in enumerate(output_list):
    
41.                         output_worksheet.write(list_index, element_index, element)
    
42. 
    
43. output_workbook.save(output_file)

复制代码

1. #!/usr/bin/env python3
   
2. import sys
   
3. from datetime import date
   
4. from xlrd import open_workbook, xldate_as_tuple
   
5. from xlwt import Workbook
   
6. 
   
7. input_file = sys.argv[1]
   
8. output_file = sys.argv[2]
   
9. 
   
10. output_workbook = Workbook()
    
11. output_worksheet = output_workbook.add_sheet('set_of_worksheets')
    
12. 
    
13. my_sheets = [0,1]
    
14. threshold = 1900.0
    
15. sales_column_index = 3
    
16. 
    
17. first_worksheet = True
    
18. with open_workbook(input_file) as workbook:
    
19.         data = []
    
20.         for sheet_index in range(workbook.nsheets):
    
21.                 if sheet_index in my_sheets:
    
22.                         worksheet = workbook.sheet_by_index(sheet_index)
    
23.                         if first_worksheet:
    
24.                                 header_row = worksheet.row_values(0)
    
25.                                 data.append(header_row)
    
26.                                 first_worksheet = False
    
27.                         for row_index in range(1,worksheet.nrows):
    
28.                                 row_list = []
    
29.                                 sale_amount = worksheet.cell_value(row_index, sales_column_index)
    
30.                                 if sale_amount > threshold:
    
31.                                         for column_index in range(worksheet.ncols):
    
32.                                                 cell_value = worksheet.cell_value(row_index,column_index)
    
33.                                                 cell_type = worksheet.cell_type(row_index, column_index)
    
34.                                                 if cell_type == 3:
    
35.                                                         date_cell = xldate_as_tuple(cell_value,workbook.datemode)
    
36.                                                         date_cell = date(*date_cell[0:3]).strftime('%m/%d/%Y')
    
37.                                                         row_list.append(date_cell)
    
38.                                                 else:
    
39.                                                         row_list.append(cell_value)
    
40.                                 if row_list:
    
41.                                         data.append(row_list)
    
42. 
    
43.         for list_index, output_list in enumerate(data):
    
44.                 for element_index, element in enumerate(output_list):
    
45.                         output_worksheet.write(list_index, element_index, element)
    
46. 
    
47. output_workbook.save(output_file)

复制代码

1. #!/usr/bin/env python3
   
2. import glob
   
3. import os
   
4. import sys
   
5. from xlrd import open_workbook
   
6. 
   
7. input_directory = sys.argv[1]
   
8. 
   
9. workbook_counter = 0
   
10. for input_file in glob.glob(os.path.join(input_directory, '*.xls*')):
    
11.         workbook = open_workbook(input_file)
    
12.         print('Workbook: {}'.format(os.path.basename(input_file)))
    
13.         print('Number of worksheets: {}'.format(workbook.nsheets))
    
14.         for worksheet in workbook.sheets():
    
15.                 print('Worksheet name:', worksheet.name, '\tRows:',\
    
16.                                 worksheet.nrows, '\tColumns:', worksheet.ncols)
    
17.         workbook_counter += 1
    
18. print('Number of Excel workbooks: {}'.format(workbook_counter))

复制代码

1. #!/usr/bin/env python3
   
2. import glob
   
3. import os
   
4. import sys
   
5. from datetime import date
   
6. from xlrd import open_workbook, xldate_as_tuple
   
7. from xlwt import Workbook
   
8. 
   
9. input_folder = sys.argv[1]
   
10. output_file = sys.argv[2]
    
11. 
    
12. output_workbook = Workbook()
    
13. output_worksheet = output_workbook.add_sheet('sums_and_averages')
    
14. 
    
15. all_data = []
    
16. sales_column_index = 3
    
17. 
    
18. header = ['workbook', 'worksheet', 'worksheet_total', 'worksheet_average',\
    
19.                                         'workbook_total', 'workbook_average']
    
20. all_data.append(header)
    
21. 
    
22. for input_file in glob.glob(os.path.join(input_folder, '*.xls*')):
    
23.         with open_workbook(input_file) as workbook:
    
24.                 list_of_totals = []
    
25.                 list_of_numbers = []
    
26.                 workbook_output = []
    
27.                 for worksheet in workbook.sheets():
    
28.                         total_sales = 0
    
29.                         number_of_sales = 0
    
30.                         worksheet_list = []
    
31.                         worksheet_list.append(os.path.basename(input_file))
    
32.                         worksheet_list.append(worksheet.name)
    
33.                         for row_index in range(1,worksheet.nrows):
    
34.                                 try:
    
35.                                         total_sales += float(str(worksheet.cell_value(row_index,sales_column_index)).strip('
    
36. ).replace(',',''))
    
37.                                         number_of_sales += 1.
    
38.                                 except:
    
39.                                         total_sales += 0.
    
40.                                         number_of_sales += 0.
    
41.                         average_sales = '%.2f' % (total_sales / number_of_sales)
    
42.                         worksheet_list.append(total_sales)
    
43.                         worksheet_list.append(float(average_sales))
    
44.                         list_of_totals.append(total_sales)
    
45.                         list_of_numbers.append(float(number_of_sales))
    
46.                         workbook_output.append(worksheet_list)
    
47.                 workbook_total = sum(list_of_totals)
    
48.                 workbook_average = sum(list_of_totals)/sum(list_of_numbers)
    
49.                 for list_element in workbook_output:
    
50.                         list_element.append(workbook_total)
    
51.                         list_element.append(workbook_average)
    
52.                 all_data.extend(workbook_output)
    
53.                
    
54. for list_index, output_list in enumerate(all_data):
    
55.         for element_index, element in enumerate(output_list):
    
56.                 output_worksheet.write(list_index, element_index, element)
    
57. 
    
58. output_workbook.save(output_file)

复制代码

1. #!/usr/bin/env python3
   
2. import sys
   
3. from xlrd import open_workbook
   
4. from xlwt import Workbook
   
5. 
   
6. input_file = sys.argv[1]
   
7. output_file = sys.argv[2]
   
8. 
   
9. output_workbook = Workbook()
   
10. output_worksheet = output_workbook.add_sheet('jan_2013_output')
    
11. 
    
12. with open_workbook(input_file) as workbook:
    
13.         worksheet = workbook.sheet_by_name('january_2013')
    
14.         for row_index in range(worksheet.nrows):
    
15.                 for column_index in range(worksheet.ncols):
    
16.                         output_worksheet.write(row_index, column_index, worksheet.cell_value(row_index, column_index))
    
17. output_workbook.save(output_file)

复制代码

thanks for sharing