Research Details

Using Receiver Operating Characteristic Analysis to Evaluate Large-Change Forecast Accuracy


This paper applies receiver operating characteristics (ROC) analysis to M3 Competition, micro monthly time series for one-month-ahead forecasts. Using the partial area under the curve (PAUC) criterion as a forecast accuracy measure and paired-comparison testing via bootstrapping, we find that complex methods (AutomatANN, Flores-Pearce2, Forecast ProSmart FCS, and Theta) perform best for forecasting large declines in these time series, which tended as a group to decline over time. A regression model of PAUC on a judgmental index for forecast method complexity provides further confirming evidence. We also found that a combination forecast, consisting of the median value of the top three methods, to perform better than the component methods, although not statistically so. The classification of top methods matches that obtained using conventional forecast accuracy methods in the M3 Competition complex
methods forecast these series better than simple ones.

Publication Year: 2008
Type: Working Paper
Working Paper Number: 22

Adobe .pdf files require the Adobe Acrobat Reader.
Download and install the Adobe Acrobat Reader.