请选择 进入手机版 | 继续访问电脑版
楼主: oliyiyi
2335 12

Linear, Machine Learning and Probabilistic Approaches for Time Series Analysis [推广有奖]

版主

泰斗

0%

还不是VIP/贵宾

-

TA的文库  其他...

计量文库

威望
7
论坛币
272091 个
通用积分
31269.1729
学术水平
1435 点
热心指数
1554 点
信用等级
1345 点
经验
383778 点
帖子
9599
精华
66
在线时间
5466 小时
注册时间
2007-5-21
最后登录
2024-3-21

初级学术勋章 初级热心勋章 初级信用勋章 中级信用勋章 中级学术勋章 中级热心勋章 高级热心勋章 高级学术勋章 高级信用勋章 特级热心勋章 特级学术勋章 特级信用勋章

oliyiyi 发表于 2017-3-1 16:40:21 |显示全部楼层 |坛友微信交流群

+2 论坛币
k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群
赵安豆老师微信:zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

感谢您参与论坛问题回答

经管之家送您两个论坛币!

+2 论坛币

本帖隐藏的内容

In this post, we consider different approaches for time series modeling. The forecasting approaches using linear models, ARIMA alpgorithm, XGBoost machine learning algorithm are described. Results of different model combinations are shown. For probabilistic modeling the approaches using copulas and Bayesian inference are considered.

INTRODUCTION

Time series analysis, especially forecasting, is an important problem of modern predictive analytics. The goal of this study is to consider different aproaches for time series modeling.  For our analysis, we used stores sales historical data from a particular competition “Rossmann Store Sales”. These data represent the sales time series of Rossmann stores. For time series forecasting such approaches as linear models and ARIMA algorithm are widely used. Machine lerning algorithm make it possible to find patterns in the time series. Sometimes we need to forecast not only more probable values of sales but also their distribution. Especially we need it in the risk analysis for assessing different risks related to sales dynamics. In this case, we need to take into account sales distributions and dependencies between sales time series features (e.g. day of week, month, average sales, etc.) and external factors such as promo, distance to competitors, etc. One can consider sales as a stochastic variable with some marginal distributions. If we have sales distribution, we can calculate value at risk (VaR) which is one of risk assessment features. In probabilistic analysis of sales, we can use copulas which allows us to  analyze  the dependence between sales and different factors.  To find distributions of model parameters Bayesian inference approach can be used.


LINEAR MODELS AND MACHINE LEARNING APPROACHES

For our analysis, we used stores sales historical data. To compare different forecasting approaches we used two last  months of the historical data as validation data for accuracy scoring using root mean squared error (RMSE). For the comparison, we used the following methods: ARIMA using R package “forecast” [1], linear regression with LASSO regularization using  R package “lars” [2], conditional inference trees with linear regression on the leaf using mob() function from R package “party” [3], gradient boosting XGBoost model using R package “xgboost” [4]. Package  “xgboost”  (short term for eXtreme Gradient Boosting)  is an efficient and scalable implementation of gradient boosting framework [5,6]. The package includes efficient linear model solver and tree learning algorithm. We also used combined approaches such as linear blending ARIMA and gradient boosting model, stacking with the use of linear regression on the first step and gradient boosting on the second step.  We used two ways of classifications – the first way is based on the time series approach and the second one is based on the independent and identically distributed variables. We consider sales in the natural logarithmic scale. Figure 1 shows typical time series of store sales.



[color=rgb(255, 255, 255) !important]



Figure 1. Typical time series of store sales.


Figure 2 shows the example of time series forecasting by different methods with RMSE error metric. Let us consider the case of time series forecasting using linear blending of ARIMA and XGBoost models. For arbitrary chosen store (Store 285) we received   RMSE=0.11 for ARIMA model,  RMSE=0.107 for XGBoost model and RMSE=0.093 for   linear blending of ARIMA and XGBoost models. Let us consider the case of using stacking with linear regression on the first step and xgboost on the second step. For arbitrary chosen store (Store 95) we received RMSE=0.122 for XGBoost model  and RMSE=0.117 for stacking model. We also studied the case of time series forecasting using XGBoost model with time series approach  and xgboost model based on independent and identically distributed variables. For arbitrary chosen store (Store 95) we received   RMSE=0.138 for  XGBoost model with time series approach  and  RMSE=0.118  for XGBoost model with i.i.d approach.  The obtained results show that for different stores the best accuracy is released by different approaches.  For each type of time series, we may develop an optimized approach which can be based on the combination of different predictive models.



[color=rgb(255, 255, 255) !important]



Figure 2. Time series forecastings by different methods.



COPULAS APPROACH FOR MODELING  

A copula is a multivariate probability distribution for which the marginal probability distribution of each variable is uniform. Copulas are used to describe the dependence between random variables. Sklar's Theorem states that any multivariate joint distribution can be written in terms of univariate marginal distribution functions and a copula, which describes the dependence structure between the variables. The copula contains all information on the dependence structure between the variables, whereas the marginal cumulative distribution functions contain all information on the marginal distributions. For the case study, we use the same sales time series, which represent sales in the stores network. We used “copula” R package [7] for modeling. We consider sales in the natural logarithmic scale. For our analysis, we take such features as sales (variable logSales), previous day sales (variable prevLogSales), number of customers that visited a store (variable Customers). First of all we take one sales time series for one arbitrary store. Marginal distributions and dependencies with correlation coefficient are shown on the figure 3. On the figures 4,5 the pseudo observations of investigated features are shown. These figures represent stochastic dependencies of investigated variable. Our next objective is to find such copulas, which will approximate these dependencies. We chose t-copula for modeling. Using maximum likelihood method, one can find fitting parameters for copula. The probability density function for calculated fitted t-copula is shown on the figure 6.


[color=rgb(255, 255, 255) !important]


Figure 3. Marginal distributions and correlation coefficient.


[color=rgb(255, 255, 255) !important]


Figure 4. Pseudo observations for logSales and prevLogSales variables.


[color=rgb(255, 255, 255) !important]


Figure 5. Pseudo observations for logSales and Customers variables.



[color=rgb(255, 255, 255) !important]


Figure 6. The probability density function for fitted t-copula.


Having fitted copula and marginal distributions, we can generate pseudo-random samples of investigated variables by applying inverse marginal comulative distribution function (CDF) to each dimensional variable of fitted copula. To construct multivariate distribution of dependent variables, we chose gamma distribution for marginal distributions of logSales and Customers variables. Having fitted copula and finding parameters for these gamma distributions from historical data, we generate  pseudo-random samples with the probability density function (PDF), shown on the figure 7.


[color=rgb(255, 255, 255) !important]


Figure 7. The PDF of generated  pseudo-randomsamples using fitted t-copula and marginal distributions.


If we need to analyze multivariate dependences with more than two variables, it is effective to use vine copulas, which enable us to construct complex multivariate copula using bivariate ones. For studying vine copula, we used CDVine R package [8]. Let us consider such features of sales time series as sales (variable logSales), mean sales per day for store (variable meanLogSales) and promo action (variable Promo). In this case, we analyze sales for stores chain. To analyze the stochastic dependence we used canonical vine copula. First tree for fitted canonical vine copula with Kendall’s tau values is shown on the figure 8.


[color=rgb(255, 255, 255) !important]


Figure 8. The tree for fitted canonical vine copula with Kendall’s tau values.




二维码

扫码加我 拉你入群

请注明:姓名-公司-职位

以便审核进群资格,未注明则拒绝

关键词:Time Series APPROACHES Approach Analysis Learning Series

缺少币币的网友请访问有奖回帖集合
https://bbs.pinggu.org/thread-3990750-1-1.html
oliyiyi 发表于 2017-3-1 16:40:56 |显示全部楼层 |坛友微信交流群

We chose a t-copula for the logSales-Promo dependency and a normal for the logSales-meanLogSales dependency. The pseudo observations for constructed canonical vine copula are shown on the figure 9 in the dimension of logSales and meanLogSales .


[color=rgb(255, 255, 255) !important]


Figure 9. Pseudo-observations for constructed canonical vine copula.


As the case study shows, the use of copula make it possible to model stochastic dependencies between different factors of sales time series separately from their marginal distributions. This can be considered as an additional approach in the sales time series analysis.


BAYESIAN INFERENCE

For Bayesian inference case study, we take such features as promo, seasonality factors (week day, month day, month of year). As well as in the previous studies, we consider sales in the natural logarithmic scale. As the example we take one sales time series for one arbitrary store. For Bayesian inference, we used Markov Chain Monte Carlo (MCMC) algorithm from MCMCpack R package [9]. For time series modeling, we used the linear regression with Gaussian errors.  Trace plots of samples vs the simulation index can be very useful in assessing convergence. The trace plot for promo coefficient is shown on the figure 10.



[color=rgb(255, 255, 255) !important]


Figure 10. Trace plot for promo coefficient.


The trace plot demonstrates the stationary process, which means good convergence and   sufficient burn-in period in the MCMC algorithm. The similar trace plots were received for other coefficients in the linear regression. The density of distributions of some regression coefficients for chosen arbitrary store are shown on the figure 11.



[color=rgb(255, 255, 255) !important]


Figure 11. Density of distributions of regression coefficients.


The figure 12 shows the examples for box plots for some regression coefficients.


[color=rgb(255, 255, 255) !important]


Figure 12. Box plots for  regression coefficients.


Sales time series can have outliers and it is important to take into account this fact using heavy tails distributions instead of Gaussian distribution. For the linear regression with variables with different type of distributions we used Bayesian hierarchical model. We conducted the case study using JAGS sampler [10] software with “rjags” R package. For modeling, we take into account mean sales for the store, sales, and promo.  We consider sales and mean sales for the store in the natural logarithmic scale. For mean sales for stores, we used Gaussian distribution, for sales – Student distribution, and  for promo – Bernoulli distribution.  In this model, we consider sales as an independent and identically distributed random variable without separating sales for different stores. Info about the store is represented by mean sales for the store values. Mean sales for the store (variable meanLogSales) vs  sales (variable logSales) obtained for considered Bayesian model are shown on the  figure 13.


[color=rgb(255, 255, 255) !important]


Figure 13. Mean sales for the store (variable meanLogSales) vs  sales (variable logSales)

obtained for fitted Bayesian model.


As the case study shows, the use of Bayesian approach allows us to model stochastic dependencies between different factors of sales time series and receive the distributions for model parameters. Such an approach can be useful  for assessing different risks related to sales dynamics.


Out team “The Slippery Appraisals”  won this competition:


[color=rgb(255, 255, 255) !important]


For our soultion we used multilevel model. We built a lot of models on 1st level. The training method of most of 1st level models was XGBoost. For second level we used ExtraTrees classifier, linear model from Python scikit-learn and Neural Networks. The final result is weighted average of these models. The most important features are based on the lags of target variable grouped by factors and their combinations, aggregated features (min, max, mean, sum) of target variable grouped by factors and their combinations, frequency features of factors variables. For our work, we used R environment with Rstudio IDE, and Python with Jupyter. We combined our work on our personal computers with cloud computing on x1.32xlarge Amazon EC2 instance with web interface for access. The following diagram shows our  multilevel model:


[color=rgb(255, 255, 255) !important]





CONCLUSION

In our cases study we showed different approaches for time series modeling. Forecasting with using linear models, ARIMA algorithm, xgboost machine learning algorithm are described. The results of different models combinations are shown. For probabilistic modeling, the approach with using copulas is shown. The Bayesian inference was applied for time series linear regression case. For time series forecasting the different models combinations technics can give better RMSE accuracy comparing to single algorithms. The probabilistic approach for time series modeling is important in the risk assessment problems. The copula approach gives one the ability to model probabilistic dependence between target values and extreme factors which is useful when a target variable has non Gausian probability density function with heavy tails. Bayesian models can be used to find distributions of coefficients in the linear model of time series. Having model parameters distributions one can find the distribution of target values using Monte-Carlo approaches. For each type of time series, one can develop an optimized approach, which can be based on the combination of different predictive models. As the example of the combination of different models, we followed the winner solution for  competition.


References

  • Hyndman, R., & Khandakar, Y. (2008). “Automatic Time Series Forecasting: The forecast Package for R.” Journal of Statistical Software, 27(3), pp.1 – 22, 2008
  • Efron, Bradley, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. "Least angle regression." The Annals of statistics 32, no. 2,pp.407-499, 2004
  • Zeileis, Achim, Torsten Hothorn, and Kurt Hornik. "Model-based recursive partitioning." Journal of Computational and Graphical Statistics 17.2, pp.492-514, 2008
  • Chen, Tianqi, and Carlos Guestrin. "XGBoost: A Scalable Tree Boosting System." arXiv preprint arXiv:1603.02754 (2016).
  • J. Friedman. “Greedy function approximation: a gradient boosting machine.”, Annals of Statistics, 29(5):1189–1232, 2001.
  • J. Friedman. “Stochastic gradient boosting.”, Computational Statistics & Data Analysis, 38(4):367–378, 2002.
  • Yan, Jun. "Enjoy the joy of copulas: with a package copula." Journal of Statistical Software, 21.4, pp.1-21, 2007
  • Brechmann, Eike Christian, and Ulf Schepsmeier. "Modeling dependence with C-and D-vine copulas: The R-package CDVine." Journal of Statistical Software, 52.3, pp. 1-27, 2013
  • Quinn, Kevin M., Andrew D. Martin, and Jong Hee Park. "MCMCpack: Markov chain Monte Carlo in R." Journal of Statistical Software 10.II (2011).
  • Martyn Plummer. JAGS Version 3.4.0 user manual. URL:http://sourceforge.net/projects/mcmc-jags/files/Manuals/3.x/jags_user_manual.pdf

缺少币币的网友请访问有奖回帖集合
https://bbs.pinggu.org/thread-3990750-1-1.html

使用道具

sqy 发表于 2017-3-1 16:46:39 |显示全部楼层 |坛友微信交流群
ding!!!!!!!!!!!!

使用道具

ekscheng 发表于 2017-3-1 17:00:16 |显示全部楼层 |坛友微信交流群

使用道具

soccy 发表于 2017-3-1 23:58:36 |显示全部楼层 |坛友微信交流群

使用道具

leexiong73 发表于 2017-3-2 17:16:06 |显示全部楼层 |坛友微信交流群

使用道具

minixi 发表于 2017-3-2 19:46:11 |显示全部楼层 |坛友微信交流群
谢谢分享

使用道具

asnster 发表于 2017-3-3 10:40:30 |显示全部楼层 |坛友微信交流群
好麻烦啊。干脆点不好吗

使用道具

Thanks for Sharing!

使用道具

xiaoyouyu 发表于 2017-3-10 06:42:53 |显示全部楼层 |坛友微信交流群
xie xie fen xiang! lou zhu!

使用道具

您需要登录后才可以回帖 登录 | 我要注册

本版微信群
加好友,备注jltj
拉您入交流群

京ICP备16021002-2号 京B2-20170662号 京公网安备 11010802022788号 论坛法律顾问:王进律师 知识产权保护声明   免责及隐私声明

GMT+8, 2024-3-28 21:16