A Unified Statistical Framework for Evaluating Predictive Methods

Abstract

Predictive analytics is an important part of the business intelligence and decision support systems literature and likely to grow in importance with the emergence of big data as a discipline. Despite their importance, the accuracy of predictive methods is often not assessed using statistical hypothesis tests. Furthermore, there is no commonly agreed upon standard as to which questions should be examined when evaluating predictive methods. We fill this gap by defining three questions that involve the overall and comparative predictive accuracy of the new method. We then present a unified statistical framework for evaluating predictive methods that can be used to address all three of these questions. The framework is particularly versatile and can be applied to most problems and datasets. In addition to these practical advantages over hypotheses tests used in previous literature, the framework has the theoretical advantage that it is not necessary to assume a normal distribution