Abstract: Typically, point forecasting methods are compared and assessed by means of an error measure or scoring function, such as the absolute error or the squared error. The individual scores are then averaged over forecast cases, to result in a summary measure of the predictive performance, such as the mean absolute error or the (root) mean squared error. I demonstrate that this common practice can lead to grossly misguided inferences, unless the scoring function and the forecasting task are carefully matched. Effective point forecasting requires that the scoring function be specified ex ante, or that the forecaster receives a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. If the scoring function is specified ex ante, the forecaster can issue the optimal point forecast, namely, the Bayes rule. If the forecaster receives a directive in the form of a functional, it is critical that the scoring function be consistent for it, in the sense that the expected score is minimized when following the directive. A functional is elicitable if there exists a scoring function that is strictly consistent for it. Expectations, ratios of expectations and quantiles are elicitable. For example, a scoring function is consistent for the mean functional if and only if it is a Bregman function. It is consistent for a quantile if and only if it is generalized piecewise linear. Similar characterizations apply to ratios of expectations and to expectiles. Weighted scoring functions are consistent for functionals that adapt to the weighting in peculiar ways. Not all functionals are elicitable; for instance, conditional value-at-risk is not, despite its popularity in quantitative finance.
Recommendations
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Order-sensitivity and equivariance of scoring functions
- Higher order elicitability and Osband's principle
- Point forecasting and forecast evaluation with generalized Huber loss
- Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
Cited in
(only showing first 100 items - show all)- Asymptotic stability of empirical processes and related functionals
- Forecasting Value at Risk and Expected Shortfall Using a Semiparametric Approach Based on the Asymmetric Laplace Distribution
- Backtesting extreme value theory models of expected shortfall
- Generalization error for Tweedie models: decomposition and error reduction with bagging
- Quantile-based risk sharing
- Isotonic recalibration under a low signal-to-noise ratio
- A survey of Bayesian predictive methods for model assessment, selection and comparison
- Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning
- Relative bound and asymptotic comparison of expectile with respect to expected shortfall
- Conditional generalized quantiles as systemic risk measures: properties, estimation, and application
- Bias-corrected score decomposition for generalized quantiles
- Marked self-exciting point process modelling of information diffusion on twitter
- Expectile depth: theory and computation for bivariate datasets
- Uniform calibration tests for forecasting systems with small lead time
- Scoring predictions at extreme quantiles
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Dynamic semiparametric models for expected shortfall (and value-at-risk)
- A parsimonious parametric model for generating margin requirements for futures
- Comparative evaluation of point process forecasts
- On the \(L_p\)-quantiles for the Student \(t\) distribution
- Tail asymptotics of generalized deflated risks with insurance applications
- A coupled statistical and deterministic model for forecasting climate-driven dengue incidence in Selangor, Malaysia
- Range-based risk measures and their applications
- Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination
- On a general class of functionals: statistical inference and application to risk measures
- Estimation and testing for spatially indexed curves with application to ionospheric and magnetic field trends
- Semi-parametric Bayesian tail risk forecasting incorporating realized measures of volatility
- Extreme M-quantiles as risk measures: from \(L^{1}\) to \(L^{p}\) optimization
- On elicitable risk measures
- Better the devil you know: improved forecasts from imperfect models
- Testing for auto-calibration with Lorenz and concentration curves
- Isotonic conditional laws
- Optimal estimation of the supremum and occupation times of a self-similar Lévy process
- Mixed frequency data and portfolio selection: a novel approach integrating DEA with mixed frequency data sources
- Generic Conditions for Forecast Dominance
- Comparing Possibly Misspecified Forecasts
- Predicting the Global Minimum Variance Portfolio
- Proper Scoring Rules for Evaluating Density Forecasts with Asymmetric Loss Functions
- Retire: robust expectile regression in high dimensions
- Volatility prediction comparison via robust volatility proxies: an empirical deviation perspective
- Estimation of complier expected shortfall treatment effects with a binary instrumental variable
- Diversification quotients based on VaR and ES
- RGA: a unified measure of predictive accuracy
- Generalized quantiles as risk measures
- Optimal transport divergences induced by scoring functions
- Semi-parametric estimation of multivariate extreme expectiles
- Parametric quantile autoregressive moving average models with exogenous terms
- Bias-reduced and variance-corrected asymptotic Gaussian inference about extreme expectiles
- Calibrating Distribution Models from PELVE
- Optimal investment under VaR-regulation and minimum insurance
- Dynamic principal component regression for forecasting functional time series in a group structure
- scientific article; zbMATH DE number 7780127 (Why is no real title available?)
- Bayesian Forecasting of Many Count-Valued Time Series
- Risk measures with the CxLS property
- Time-varying sparsity in dynamic regression models
- Joint value-at-risk and expected shortfall regression for location-scale time series models
- New regression model and machine learning for fitting proportional data with application
- Forecast dominance testing via sign randomization
- Elicitability of Instance and Object Ranking
- On the properties of the lambda value at risk: robustness, elicitability and consistency
- Dual representation of expectile-based expected shortfall and its properties
- Semiparametric empirical best prediction for small area estimation of unemployment indicators
- The effect of exchange rates on statistical decisions
- Dynamic quantile function models
- Automatic locally stationary time series forecasting with application to predicting UK gross value added time series
- Expectile regression averaging method for probabilistic forecasting of electricity prices
- Scenario analysis for derivative portfolios via dynamic factor models
- Hierarchical probabilistic forecasting of electricity demand with smart meter data
- Optimal reinsurance with expectile
- Multivariate geometric expectiles
- Distributionally robust reinsurance with expectile
- Distortion riskmetrics on general spaces
- Grouped multivariate and functional time series forecasting: an application to annuity pricing
- Point forecasting and forecast evaluation with generalized Huber loss
- Joint inference on extreme expectiles for multivariate heavy-tailed distributions
- Risks in emerging markets equities: time-varying versus spatial risk analysis
- On the indirect elicitability of the mode and modal interval
- Computational analysis of expectile and deviation expectile portfolio optimization models
- Risk bounds for factor models
- Bayesian spline method for assessing extreme loads on wind turbines
- What can we learn from telematics car driving data: a survey
- Nonlinear network-based quantitative trait prediction from biological data
- On a capital allocation by minimization of some risk indicators
- Scenario aggregation method for portfolio expectile optimization
- Expectile asymptotics
- Quantifying market risk with value-at-risk or expected shortfall? -- Consequences for capital requirements and model risk
- Forecasting overdispersed INAR(1) count time series with negative binomial marginal
- Testing Quantile Forecast Optimality
- Mean-variance-VaR portfolios: MIQP formulation and performance analysis
- Estimating and backtesting risk under heavy tails
- Convex and Lorenz orders under balance correction in nonlife insurance pricing: review and new developments
- Probabilistic sensitivity measures as information value
- A risk measurement approach from risk-averse stochastic optimization of score functions
- Coherence and elicitability
- Risk-adaptive approaches to stochastic optimization: a survey
- Mixture EMOS model for calibrating ensemble forecasts of wind speed
- A dynamic nonstationary spatio-temporal model for short term prediction of precipitation
- Aggregation-robustness and model uncertainty of regulatory risk measures
- Comparison of value-at-risk models using the MCS approach
- The mode functional is not elicitable
This page was built for publication: Making and evaluating point forecasts
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q91134)
