Abstract: Typically, point forecasting methods are compared and assessed by means of an error measure or scoring function, such as the absolute error or the squared error. The individual scores are then averaged over forecast cases, to result in a summary measure of the predictive performance, such as the mean absolute error or the (root) mean squared error. I demonstrate that this common practice can lead to grossly misguided inferences, unless the scoring function and the forecasting task are carefully matched. Effective point forecasting requires that the scoring function be specified ex ante, or that the forecaster receives a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. If the scoring function is specified ex ante, the forecaster can issue the optimal point forecast, namely, the Bayes rule. If the forecaster receives a directive in the form of a functional, it is critical that the scoring function be consistent for it, in the sense that the expected score is minimized when following the directive. A functional is elicitable if there exists a scoring function that is strictly consistent for it. Expectations, ratios of expectations and quantiles are elicitable. For example, a scoring function is consistent for the mean functional if and only if it is a Bregman function. It is consistent for a quantile if and only if it is generalized piecewise linear. Similar characterizations apply to ratios of expectations and to expectiles. Weighted scoring functions are consistent for functionals that adapt to the weighting in peculiar ways. Not all functionals are elicitable; for instance, conditional value-at-risk is not, despite its popularity in quantitative finance.
Recommendations
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Order-sensitivity and equivariance of scoring functions
- Higher order elicitability and Osband's principle
- Point forecasting and forecast evaluation with generalized Huber loss
- Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
Cited in
(only showing first 100 items - show all)- Testing the reliability of forecasting systems
- Grouped multivariate and functional time series forecasting: an application to annuity pricing
- Robust VIF regression with application to variable selection in large data sets
- Bayes risk, elicitability, and the Expected Shortfall
- Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning
- On the elicitability of range value at risk
- Estimating value-at-risk and expected shortfall using the intraday low and range data
- Estimating and backtesting risk under heavy tails
- Tail asymptotics of generalized deflated risks with insurance applications
- Optimal insurance design in the presence of exclusion clauses
- Scoring interval forecasts: equal-tailed, shortest, and modal interval
- A relative error-based approach for variable selection
- Isotonic regression for elicitable functionals and their Bayes risk
- Using the Bayesian Shtarkov solution for predictions
- Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
- Spatio-temporal short-term wind forecast: a calibrated regime-switching method
- Joint generalized quantile and conditional tail expectation regression for insurance risk analysis
- Optimal operational service levels in vendor managed inventory contracts -- an exact approach
- Measurability of functionals and of ideal point forecasts
- The consistency and asymptotic normality of the kernel type expectile regression estimator for functional data
- Backtesting extreme value theory models of expected shortfall
- A parsimonious parametric model for generating margin requirements for futures
- Density forecast of financial returns using decomposition and maximum entropy
- Characterizing the optimal solutions to the isotonic regression problem for identifiable functionals
- Optimal estimation of the supremum and occupation times of a self-similar Lévy process
- Probabilistic sensitivity measures as information value
- Marked self-exciting point process modelling of information diffusion on twitter
- Distributional transforms, probability distortions, and their applications
- Inventory -- forecasting: mind the gap
- Feature extraction for functional time series: theory and application to NIR spectroscopy data
- Optimal trading policies for wind energy producer
- Point forecasting and forecast evaluation with generalized Huber loss
- Joint inference on extreme expectiles for multivariate heavy-tailed distributions
- Forecasting intra-individual changes of affective states taking into account inter-individual differences using intensive longitudinal data from a university student dropout study in math
- Risks in emerging markets equities: time-varying versus spatial risk analysis
- Focusing on regions of interest in forecast evaluation
- On the indirect elicitability of the mode and modal interval
- Functional prediction of intraday cumulative returns
- Bayesian spline method for assessing extreme loads on wind turbines
- Why scoring functions cannot assess tail properties
- Semi-parametric Bayesian tail risk forecasting incorporating realized measures of volatility
- Backtesting expected shortfall and beyond
- Econometric modeling of risk measures: a selective review of the recent literature
- Quantile evaluation, sensitivity to bracketing, and sharing business payoffs
- Uniform calibration tests for forecasting systems with small lead time
- Scoring predictions at extreme quantiles
- Measuring and adjusting for overconfidence
- Semiparametric empirical best prediction for small area estimation of unemployment indicators
- A theory for measures of tail risk
- On the properties of the lambda value at risk: robustness, elicitability and consistency
- Backtesting VaR and expectiles with realized scores
- Order-sensitivity and equivariance of scoring functions
- Comments on: Space-time wind speed forecasting for improved power system dispatch
- The role of the information set for forecasting -- with applications to risk management
- A dynamic nonstationary spatio-temporal model for short term prediction of precipitation
- Optimal robust insurance with a finite uncertainty set
- On the measurement of economic tail risk
- Using conditional kernel density estimation for wind power density forecasting
- Regulatory arbitrage of risk measures
- Relative bound and asymptotic comparison of expectile with respect to expected shortfall
- Comments on: Space-time wind speed forecasting for improved power system dispatch
- Bayesian structured additive distributional regression with an application to regional income inequality in Germany
- Optimal reinsurance with expectile
- Forecaster's dilemma: extreme events and forecast evaluation
- Generalized quantiles as risk measures
- Multivariate geometric expectiles
- Projecting the future burden of cancer: Bayesian age-period-cohort analysis with integrated nested Laplace approximations
- Extreme M-quantiles as risk measures: from \(L^{1}\) to \(L^{p}\) optimization
- Efficient regularized isotonic regression with application to gene-gene interaction search
- Performance measurement with expectiles
- Forecast dominance testing via sign randomization
- Risk measures with the CxLS property
- How superadditive can a risk measure be?
- ASYMPTOTIC EXPANSIONS OF GENERALIZED QUANTILES AND EXPECTILES FOR EXTREME RISKS
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Tail risk inference via expectiles in heavy-tailed time series
- Uncertainty quantification in complex simulation models using ensemble copula coupling
- Semi-parametric estimation of multivariate extreme expectiles
- Aggregation-robustness and model uncertainty of regulatory risk measures
- Probabilistic wind speed forecasting on a grid based on ensemble model output statistics
- Generalization error for Tweedie models: decomposition and error reduction with bagging
- Coherence and elicitability
- Estimation and testing for spatially indexed curves with application to ionospheric and magnetic field trends
- Understanding predictive information criteria for Bayesian models
- Comparison of value-at-risk models using the MCS approach
- Quantifying market risk with value-at-risk or expected shortfall? -- Consequences for capital requirements and model risk
- Expectile depth: theory and computation for bivariate datasets
- Dynamic quantile function models
- Optimal investment under VaR-regulation and minimum insurance
- Distortion riskmetrics on general spaces
- Verification of internal risk measure estimates
- A joint quantile and expected shortfall regression framework
- Elicitability and identifiability of set-valued measures of systemic risk
- The mode functional is not elicitable
- Higher order elicitability and Osband's principle
- A note on the use of empirical AUC for evaluating probabilistic forecasts
- On the \(L_p\)-quantiles for the Student \(t\) distribution
- Asymptotic stability of empirical processes and related functionals
- On a capital allocation by minimization of some risk indicators
- Dominating countably many forecasts
This page was built for publication: Making and evaluating point forecasts
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q91134)
