Abstract
Evaluating and documenting the robustness of forecasting models to different input conditions is important for their responsible deployment in real-world applications. Time series forecasting models often exhibit degraded performance in the form of unusually large errors, high uncertainty, or hubris (high errors coupled with low uncertainty). Traditional stress testing approaches rely on manually designed adverse scenarios that fail to systematically identify unknown stress factors, in which data characteristics indicate potential issues. To overcome this limitation, this paper introduces MAST (Meta-learning and data Augmentation for Stress Testing), a novel method for stress testing forecasting models. MAST leverages model outputs (error scores and prediction intervals) to automatically identify and characterize input conditions that induce stress. Specifically, MAST is a binary probabilistic classifier that predicts the likelihood of forecasting model stress based on time series features. An additional contribution is a novel time series data augmentation approach based on oversampling or synthetic time series generation, that improves the information about stress factors in the input space, resulting in increased stress classification performance. Experiments were conducted using 6 benchmark datasets containing a total of 97.829 time series. We demonstrate how MAST is able to identify and explain input conditions that lead to manifestations of stress, namely large errors, high uncertainty, or hubris.