Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply

DOI10.1111/RSSB.12154zbMATH Open1414.62038arXiv1503.08195OpenAlexW2963870508MaRDI QIDQ5378146FDOQ5378146

Authors: Werner Ehm, Tilmann Gneiting, Alexander I. Jordan, Fabian Krüger 

Publication date: 12 June 2019

Published in: Journal of the Royal Statistical Society Series B: Statistical Methodology (Search for Journal in Brave)

Abstract: In the practice of point prediction, it is desirable that forecasters receive a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. When evaluating and comparing competing forecasts, it is then critical that the scoring function used for these purposes be consistent for the functional at hand, in the sense that the expected score is minimized when following the directive. We show that any scoring function that is consistent for a quantile or an expectile functional, respectively, can be represented as a mixture of extremal scoring functions that form a linearly parameterized family. Scoring functions for the mean value and probability forecasts of binary events constitute important examples. The quantile and expectile functionals along with the respective extremal scoring functions admit appealing economic interpretations in terms of thresholds in decision making. The Choquet type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function. In empirical settings it suffices to compare the average scores for only a finite number of extremal elements. Plots of the average scores with respect to the extremal scoring functions, which we call Murphy diagrams, permit detailed comparisons of the relative merits of competing forecasts.

Full work available at URL: https://arxiv.org/abs/1503.08195

Recommendations

• Making and evaluating point forecasts
• Order-sensitivity and equivariance of scoring functions
• Higher order elicitability and Osband's principle
• Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
• Consistent scoring functions for quantiles

zbMATH Keywords

quantiledecision theoryexpectileprobability forecastforecast rankingeconomic utilityChoquet representationpoint forecastconsistent scoring functionelicitableorder sensitivity

Mathematics Subject Classification ID

Point estimation (62F10) Nonparametric estimation (62G05) Statistical ranking and selection procedures (62F07) Inference from stochastic processes and prediction (62M20) Decision theory (91B06) General considerations in statistical decision theory (62C05) Research exposition (monographs, survey articles) pertaining to statistics (62-02)

Cited In (48)

• Calibrating sufficiently
• Making and evaluating point forecasts
• Bias-corrected score decomposition for generalized quantiles
• Comparative evaluation of point process forecasts
• Receiver operating characteristic (ROC) movies, universal ROC (UROC) curves, and coefficient of predictive ability (CPA)
• Range-based risk measures and their applications
• Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination
• Scalable spatio‐temporal Bayesian analysis of high‐dimensional electroencephalography data
• Testing for auto-calibration with Lorenz and concentration curves
• Generic Conditions for Forecast Dominance
• Comparing Possibly Misspecified Forecasts
• Forecast dominance testing via sign randomization
• A discrete density approach to Bayesian quantile and expectile regression with discrete responses
• Tail expectile process and risk assessment
• Point forecasting and forecast evaluation with generalized Huber loss
• Joint inference on extreme expectiles for multivariate heavy-tailed distributions
• Using proxies to improve forecast evaluation
• Order-sensitivity and equivariance of scoring functions
• ExpectHill estimation, extreme risk and heavy tails
• Backtesting VaR and expectiles with realized scores
• Comparing Predictive Accuracy in the Presence of a Loss Function Shape Parameter
• The local linear functional \(k\)NN estimator of the conditional expectile: uniform consistency in number of neighbors
• Scoring interval forecasts: equal-tailed, shortest, and modal interval
• Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
• An elastic-net penalized expectile regression with applications
• Isotonic regression for elicitable functionals and their Bayes risk
• Density Regression with Conditional Support Points
• The \(k\)th power expectile regression
• Forecaster's dilemma: extreme events and forecast evaluation
• A joint quantile and expected shortfall regression framework
• Why scoring functions cannot assess tail properties
• Quantile evaluation, sensitivity to bracketing, and sharing business payoffs
• Sensitivity measures based on scoring functions
• On automatic bias reduction for extreme expectile estimation
• On the elicitability of range value at risk
• Elicitability and identifiability of set-valued measures of systemic risk
• Higher order elicitability and Osband's principle
• A hierarchical Bayesian model for predicting ecological interactions using scaled evolutionary relationships
• PDE-regularised spatial quantile regression
• On the nonparametric estimation of the functional expectile regression
• Encoded value-at-risk: a machine learning approach for portfolio risk measurement
• Properization: constructing proper scoring rules via Bayes acts
• The functional \(k\mathrm{NN}\) estimator of the conditional expectile: uniform consistency in number of neighbors
• Characterizing the optimal solutions to the isotonic regression problem for identifiable functionals
• Simultaneous Semiparametric Estimation of Clustering and Regression
• Consistent scoring functions for quantiles
• Estimating value-at-risk and expected shortfall using the intraday low and range data
• The consistency and asymptotic normality of the kernel type expectile regression estimator for functional data

This page was built for publication: Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5378146)