Research Details

Using Receiver Operating Characteristic Analysis to Evaluate Large-Change Forecast Accuracy



This paper applies receiver operating characteristics (ROC) analysis to M3 Competition, micro
monthly time series for one-month-ahead forecasts. Using the partial area under the curve
(PAUC) criterion as a forecast accuracy measure and paired-comparison testing via
bootstrapping, we find that complex methods (AutomatANN, Flores-Pearce2, Forecast ProSmart
FCS, and Theta) perform best for forecasting large declines in these time series, which tended as
a group to decline over time. A regression model of PAUC on a judgmental index for forecast
method complexity provides further confirming evidence. We also found that a combination
forecast, consisting of the median value of the top three methods, to perform better than the
component methods, although not statistically so. The classification of top methods matches that
obtained using conventional forecast accuracy methods in the M3 Competition?complex
methods forecast these series better than simple ones.

Publication Year: 2008
Type: Working Paper
Working Paper Number: 22

Adobe .pdf files require the Adobe Acrobat Reader.
Download and install the Adobe Acrobat Reader.