Abstract: Typically, point forecasting methods are compared and assessed by means of an error measure or scoring function, such as the absolute error or the squared error. The individual scores are then averaged over forecast cases, to result in a summary measure of the predictive performance, such as the mean absolute error or the (root) mean squared error. I demonstrate that this common practice can lead to grossly misguided inferences, unless the scoring function and the forecasting task are carefully matched. Effective point forecasting requires that the scoring function be specified ex ante, or that the forecaster receives a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. If the scoring function is specified ex ante, the forecaster can issue the optimal point forecast, namely, the Bayes rule. If the forecaster receives a directive in the form of a functional, it is critical that the scoring function be consistent for it, in the sense that the expected score is minimized when following the directive. A functional is elicitable if there exists a scoring function that is strictly consistent for it. Expectations, ratios of expectations and quantiles are elicitable. For example, a scoring function is consistent for the mean functional if and only if it is a Bregman function. It is consistent for a quantile if and only if it is generalized piecewise linear. Similar characterizations apply to ratios of expectations and to expectiles. Weighted scoring functions are consistent for functionals that adapt to the weighting in peculiar ways. Not all functionals are elicitable; for instance, conditional value-at-risk is not, despite its popularity in quantitative finance.
Recommendations
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Order-sensitivity and equivariance of scoring functions
- Higher order elicitability and Osband's principle
- Point forecasting and forecast evaluation with generalized Huber loss
- Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
Cited in
(only showing first 100 items - show all)- Asymptotic stability of empirical processes and related functionals
- Forecasting Value at Risk and Expected Shortfall Using a Semiparametric Approach Based on the Asymmetric Laplace Distribution
- Backtesting extreme value theory models of expected shortfall
- Generalization error for Tweedie models: decomposition and error reduction with bagging
- Quantile-based risk sharing
- Isotonic recalibration under a low signal-to-noise ratio
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning
- Relative bound and asymptotic comparison of expectile with respect to expected shortfall
- Bias-corrected score decomposition for generalized quantiles
- Marked self-exciting point process modelling of information diffusion on twitter
- Expectile depth: theory and computation for bivariate datasets
- Uniform calibration tests for forecasting systems with small lead time
- Scoring predictions at extreme quantiles
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Dynamic semiparametric models for expected shortfall (and value-at-risk)
- A parsimonious parametric model for generating margin requirements for futures
- Comparative evaluation of point process forecasts
- On the \(L_p\)-quantiles for the Student \(t\) distribution
- Tail asymptotics of generalized deflated risks with insurance applications
- A coupled statistical and deterministic model for forecasting climate-driven dengue incidence in Selangor, Malaysia
- Range-based risk measures and their applications
- Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination
- Estimation and testing for spatially indexed curves with application to ionospheric and magnetic field trends
- Semi-parametric Bayesian tail risk forecasting incorporating realized measures of volatility
- Extreme M-quantiles as risk measures: from \(L^{1}\) to \(L^{p}\) optimization
- On elicitable risk measures
- Better the devil you know: improved forecasts from imperfect models
- Testing for auto-calibration with Lorenz and concentration curves
- Optimal estimation of the supremum and occupation times of a self-similar Lévy process
- Generic Conditions for Forecast Dominance
- Comparing Possibly Misspecified Forecasts
- Predicting the Global Minimum Variance Portfolio
- Proper Scoring Rules for Evaluating Density Forecasts with Asymmetric Loss Functions
- Retire: robust expectile regression in high dimensions
- Volatility prediction comparison via robust volatility proxies: an empirical deviation perspective
- Estimation of complier expected shortfall treatment effects with a binary instrumental variable
- Diversification quotients based on VaR and ES
- Generalized quantiles as risk measures
- Semi-parametric estimation of multivariate extreme expectiles
- Parametric quantile autoregressive moving average models with exogenous terms
- Bias-reduced and variance-corrected asymptotic Gaussian inference about extreme expectiles
- Calibrating Distribution Models from PELVE
- Optimal investment under VaR-regulation and minimum insurance
- Dynamic principal component regression for forecasting functional time series in a group structure
- scientific article; zbMATH DE number 7780127 (Why is no real title available?)
- Bayesian Forecasting of Many Count-Valued Time Series
- Risk measures with the CxLS property
- Time-varying sparsity in dynamic regression models
- Forecast dominance testing via sign randomization
- Elicitability of Instance and Object Ranking
- On the properties of the lambda value at risk: robustness, elicitability and consistency
- Dual representation of expectile-based expected shortfall and its properties
- Semiparametric empirical best prediction for small area estimation of unemployment indicators
- Dynamic quantile function models
- Scenario analysis for derivative portfolios via dynamic factor models
- Hierarchical probabilistic forecasting of electricity demand with smart meter data
- Optimal reinsurance with expectile
- Multivariate geometric expectiles
- Distributionally robust reinsurance with expectile
- Distortion riskmetrics on general spaces
- Grouped multivariate and functional time series forecasting: an application to annuity pricing
- Point forecasting and forecast evaluation with generalized Huber loss
- Joint inference on extreme expectiles for multivariate heavy-tailed distributions
- Risks in emerging markets equities: time-varying versus spatial risk analysis
- On the indirect elicitability of the mode and modal interval
- Risk bounds for factor models
- Bayesian spline method for assessing extreme loads on wind turbines
- What can we learn from telematics car driving data: a survey
- On a capital allocation by minimization of some risk indicators
- Scenario aggregation method for portfolio expectile optimization
- Expectile asymptotics
- Quantifying market risk with value-at-risk or expected shortfall? -- Consequences for capital requirements and model risk
- Forecasting overdispersed INAR(1) count time series with negative binomial marginal
- Mean-variance-VaR portfolios: MIQP formulation and performance analysis
- Estimating and backtesting risk under heavy tails
- Convex and Lorenz orders under balance correction in nonlife insurance pricing: review and new developments
- Probabilistic sensitivity measures as information value
- Coherence and elicitability
- Mixture EMOS model for calibrating ensemble forecasts of wind speed
- A dynamic nonstationary spatio-temporal model for short term prediction of precipitation
- Aggregation-robustness and model uncertainty of regulatory risk measures
- Comparison of value-at-risk models using the MCS approach
- The mode functional is not elicitable
- Robust estimation in regression and classification methods for large dimensional data
- Using proxies to improve forecast evaluation
- Reconstruction of late Holocene climate based on tree growth and mechanistic hierarchical models
- Local bias adjustment, duration-weighted probabilities, and automatic construction of tariff cells
- Evaluating Forecasts for High-Impact Events Using Transformed Kernel Scores
- The dependence structure between equity and foreign exchange markets and tail risk forecasts of foreign investments
- Order-sensitivity and equivariance of scoring functions
- Model selection with Gini indices under auto-calibration
- Joint generalized quantile and conditional tail expectation regression for insurance risk analysis
- The MLE of Aigner, Amemiya, and Poirier is \textit{not} the expectile MLE
- Comments on: Space-time wind speed forecasting for improved power system dispatch
- GARCH-UGH: a bias-reduced approach for dynamic extreme value-at-risk estimation in financial time series
- Lasso regularization within the LocalGLMnet architecture
- Backtesting VaR and expectiles with realized scores
- The role of the information set for forecasting -- with applications to risk management
- Multiple measures realized GARCH models
This page was built for publication: Making and evaluating point forecasts
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q91134)
