[BeijingTomorrow]Python Package for Beijing Pollution Forecast

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. BEIJING TOMORROW
   
2. ----------------
   
3. 
   
4. By Tavis Barr, tavisbarr@gmail.com, Copyright 2016
   
5. Licensed under the Gnu Public License V. 2.0
   
6. Contact me about other licensing arrangements
   
7. 
   
8. 
   
9. This program uses satellite data from the NASA MODIS program, and pollution
   
10. data from the US embassy in Beijing to develop a predictor of the next day's
    
11. change in pollution based on today's satellite images.
    
12. 
    
13. It is not designed as a first-rate forecaster -- it would need several
    
14. improvements for that, including addition of weather data and probably better
    
15. training of the models, or for that matter, just including the current
    
16. pollution level as a feature -- but rather as a demonstration of how to use
    
17. CaffeOnSpark to perform a complete modelling exercise from download to
    
18. training to prediction.  The CaffeOnSpark package is not very well documented
    
19. as of the time of this writing, so the code serves to illustrate its usage.
    
20. 
    
21. 
    
22. SETUP
    
23. -----
    
24. 
    
25. This program is designed to be run on Spark.  I execute this through the
    
26. Eclipse IDE using PyDev, but it can also be done via the command line.  To get
    
27. PyDev working with CaffeOnSpark, the following changes need to be made to the
    
28. Python interpreter:
    
29. 
    
30. (a) The following need to be added to PYTHONPATH, with ${SPARK_HOME} and
    
31. ${CAFFE_ON_SPARK} replaced with the actual absolute paths:
    
32. 
    
33. ${SPARK_HOME}
    
34. ${SPARK_HOME}/python
    
35. ${SPARK_HOME}/python/lib/py4j-[your_version]-src.zip
    
36. ${SPARK_HOME}/python/lib/pyspark.zip
    
37. ${CAFFE_ON_SPARK}/caffe-grid/target/caffeonsparkpythonapi.zip
    
38. 
    
39. (b) The following environmental variables need to be added:
    
40. 
    
41. SPARK_HOME needs to be set to the root of your Spark installation
    
42. PYSPARK_SUBMIT_ARGS needs to be set to the following (YMMV):
    
43. 
    
44. --master local[*]       # OR URL if you are running on YARN
    
45. --queue PyDevSpark1.6.1 # Can be called whatever  
    
46. --files ${CAFFE_ON_SPARK}/caffe-public/python/caffe/_caffe.so,
    
47.         ${CAFFE_ON_SPARK}/caffe-public/distribute/lib/libcaffe.so.1.0.0-rc3  
    
48. --jars "${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar,
    
49.                 ${CAFFE_ON_SPARK}/caffe-distri/target/caffe-distri-0.1-SNAPSHOT-jar-with-dependencies.jar"
    
50. --driver-library-path "${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar"
    
51. --driver-class-path "${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar"
    
52. --conf         spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}:
    
53.                 ${CAFFE_ON_SPARK}/caffe-public/distribute/lib:
    
54.                 ${CAFFE_ON_SPARK}/caffe-distri/distribute/lib"  
    
55. --conf         spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:
    
56.                 ${CAFFE_ON_SPARK}/caffe-public/distribute/lib:
    
57.                 ${CAFFE_ON_SPARK}/caffe-distri/distribute/lib"
    
58. pyspark-shell
    
59. 
    
60. These are roughly the same arguments that will need to be made when
    
61. invoking pyspark if this program is run from the command line.  Again,
    
62. ${CAFFE_ON_SPARK} should be replaced with its actual value.
    
63. 
    
64. 
    
65. 
    
66. Running the program takes place in four major steps, each of which uses its
    
67. own module.  These are: (1) Downloading the data, (2) Building the LMDB
    
68. database, (3) Training the module, and (4) Prediction on today's data.
    
69. Additionally, there is a module to train a standard (non-"deep") model
    
70. using the Random Forest library on Apache Spark for comparison.  Each of
    
71. these modules are described in order.
    
72. 
    
73. (1) Downloading the data
    
74. 
    
75. The module that is responsible for downloading the satellite data,
    
76. tilescraper, is available as a separate package, because it likely has uses
    
77. aside from training models etc.  The pollution data are much smaller, and
    
78. are pulled in real time using the beijingpollutioncompiler module.  The data
    
79. are pulled from the US Embassy web site in Beijing; the Chinese Ministry of
    
80. Environmental Protection also publishes data that are more accurate, but these
    
81. are more difficult to obtain and also do not go back quite as far.
    
82. 
    
83. 
    
84. (2) Building the database
    
85. 
    
86. CaffeOnSpark expects the data in one of a handful of datbase formats; I found
    
87. LMDB easiest to work with, so I transformed the data into this format to use
    
88. them.  Note that the images require to have the index order of the underlying
    
89. data changed from the standard image format for Caffe to use them.
    
90. 
    
91. Unfortunately, the Python module for Caffe also expects the outcome data to be
    
92. an integer; fixing this is a one-line change in the Caffe source code, but I
    
93. did not want to use a non-standard version of Caffe, so I just multiplied the
    
94. changes by 100.
    
95. 
    
96. When no satellite data is available, MODIS does not throw an exception but
    
97. merely returns a black image.  The pollution data may be blank for some days.
    
98. As the CaffeOnSpark trainer is expecting all observations to be valid, days
    
99. with such data are discarded at the time the database is built.
    
100. 
     
101. The outcome to be predicted is the logarithmic change between today's pollution
     
102. level and tomorrow's pollution level.
     
103. 
     
104. The features are expected to have a mean of zero, so the mean of each pixel is
     
105. subtratcted from every image of the training and test data before it is placed
     
106. in the database.
     
107. 
     
108. (3) Training the Model
     
109. 
     
110. The Python interface to CaffeOnSpark is somewhat limited as of this writing;
     
111. in particular, it is necessary to have the configuration of the solver and
     
112. network specified in .prototxt files rather than configured as Python objects
     
113. in the code (the Caffe interfaces to define them in the code will work, but
     
114. they cannot easily be attached to the CaffeOnSpark configuration).
     
115. 
     
116. The steps to train the model are pretty much the same as in the demonstration
     
117. examples widely available online; in fact, very little deviation from these
     
118. steps is supported by the Python interface to CaffeOnSpark.  The key is that
     
119. we obtain our model configuration from a folder in the resource directory,
     
120. and also deliver the model there.  When it is finished training, the routine
     
121. reports an R-squared of the regression on the test sample.  In any event,
     
122. most of the work in this section involves tweaking the .prototxt files to
     
123. improve the model.
     
124. 
     
125. (4) Prediction
     
126. 
     
127. Once the model is trained, it can be called from the regular Caffe
     
128. interface without requiring the overhead of Spark.  After checking that today's
     
129. image is not blank, we load it and transform it the same way as the training
     
130. images were transformed -- the axes are swapped and the mean is subtracted.
     
131. Finally, the test image is expected to be part of a batch, in this case a
     
132. batch of one, so we add an extra dimension to the image array.  
     
133. 
     
134. Additionally, we need a slight alteration to the .prototxt file to predict
     
135. using this model.  First, the top layers (loss and accuracy) need to be taken
     
136. out of the model, otherwise they will be returned and not the prediction.
     
137. Finally, the batch size for the test model needs to be set to one.
     
138. 
     
139. With that in hand, we are ready to make a prediction.  Obviously, this model
     
140. is quite rudimentary.
     
141. 
     
142. (5) Comparison
     
143. 
     
144. It may be interesting to see how well we can do without using "deep" learning.
     
145. The pollutionrflearner downscales the image into a 4x4 grid, and then builds
     
146. a random forest over the grid.  The clear challenge in building any model
     
147. of this phenomenon is that we have far more features than observations, so we
     
148. need to intelligently simplify the features before attempting to train.  In
     
149. any event, the random forest model does slightly worse than the "deep" model,
     
150. but not substantially so.  Frankly, neither model is highly predictive.

复制代码

本帖隐藏的内容

https://github.com/tavisbarr/BeijingTomorrow

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Pollution Forecast Tomorrow Beijing package

谢谢楼主分享！

day new month update

谢谢，学习你了。

多谢楼主