Order-sensitivity and equivariance of scoring functions
From MaRDI portal
Abstract: The relative performance of competing point forecasts is usually measured in terms of loss or scoring functions. It is widely accepted that these scoring function should be strictly consistent in the sense that the expected score is minimized by the correctly specified forecast for a certain statistical functional such as the mean, median, or a certain risk measure. Thus, strict consistency opens the way to meaningful forecast comparison, but is also important in regression and M-estimation. Usually strictly consistent scoring functions for an elicitable functional are not unique. To give guidance on the choice of a scoring function, this paper introduces two additional quality criteria. Order-sensitivity opens the possibility to compare two deliberately misspecified forecasts given that the forecasts are ordered in a certain sense. On the other hand, equivariant scoring functions obey similar equivariance properties as the functional at hand - such as translation invariance or positive homogeneity. In our study, we consider scoring functions for popular functionals, putting special emphasis on vector-valued functionals, e.g. the pair (mean, variance) or (Value at Risk, Expected Shortfall).
Recommendations
- Making and evaluating point forecasts
- Higher order elicitability and Osband's principle
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- On scoring ordered classifications
- Should Scoring Rules be “Effective”?
Cited in
(22)- Making and evaluating point forecasts
- Bias-corrected score decomposition for generalized quantiles
- Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings. With discussion and authors' reply
- Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination
- Point forecasting and forecast evaluation with generalized Huber loss
- On the indirect elicitability of the mode and modal interval
- Osband's Principle for Identification Functions
- Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals
- Backtesting Systemic Risk Forecasts Using Multi-Objective Elicitability
- Isotonic regression for elicitable functionals and their Bayes risk
- Why scoring functions cannot assess tail properties
- Proper scoring rules, gradients, divergences, and entropies for paths and time series
- Sensitivity measures based on scoring functions
- On the elicitability of range value at risk
- Higher order elicitability and Osband's principle
- Elicitability and identifiability of set-valued measures of systemic risk
- Assessing point forecast accuracy by stochastic loss distance
- Elicitability and identifiability of tail risk measures
- Deep quantile and deep composite triplet regression
- Estimation and backtesting of risk measures with emphasis on distortion risk measures
- Measurability of functionals and of ideal point forecasts
- Consistent scoring functions for quantiles
This page was built for publication: Order-sensitivity and equivariance of scoring functions
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2414489)
