Easy Uncertainty Quantification (EasyUQ): Generating Predictive Distributions from Single-Valued Model Output

DOI10.1137/22M1541915arXiv2212.08376OpenAlexW4391651925WikidataQ128493737 ScholiaQ128493737MaRDI QIDQ6154539FDOQ6154539

Authors: Eva-Maria Walz, Alexander Henzi, Johanna F. Ziegel, Tilmann Gneiting 

Publication date: 15 February 2024

Published in: SIAM Review (Search for Journal in Brave)

Abstract: How can we quantify uncertainty if our favorite computational tool - be it a numerical, a statistical, or a machine learning approach, or just any computer model - provides single-valued output only? In this article, we introduce the Easy Uncertainty Quantification (EasyUQ) technique, which transforms real-valued model output into calibrated statistical distributions, based solely on training data of model output-outcome pairs, without any need to access model input. In its basic form, EasyUQ is a special case of the recently introduced Isotonic Distributional Regression (IDR) technique that leverages the pool-adjacent-violators algorithm for nonparametric isotonic regression. EasyUQ yields discrete predictive distributions that are calibrated and optimal in finite samples, subject to stochastic monotonicity. The workflow is fully automated, without any need for tuning. The Smooth EasyUQ approach supplements IDR with kernel smoothing, to yield continuous predictive distributions that preserve key properties of the basic form, including both, stochastic monotonicity with respect to the original model output, and asymptotic consistency. For the selection of kernel parameters, we introduce multiple one-fit grid search, a computationally much less demanding approximation to leave-one-out cross-validation. We use simulation examples and the WeatherBench challenge in data-driven weather prediction to illustrate the techniques. In a study of benchmark problems from machine learning, we show how EasyUQ and Smooth EasyUQ can be integrated into the workflow of modern neural network learning and hyperparameter tuning, and find EasyUQ to be competitive with more elaborate input-based approaches.

Full work available at URL: https://arxiv.org/abs/2212.08376

zbMATH Keywords

kernel smoothingproper scoring ruleisotonic regressionneural network trainingprobabilistic forecastpool-adjacent-violators (PAV) algorithmcomputational model output

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Learning and adaptive systems in artificial intelligence (68T05) Computational learning theory (68Q32)

Cites Work

• Calibrated Probabilistic Mesoscale Weather Field Forecasting
• Strictly Proper Scoring Rules, Prediction, and Estimation
• Probabilistic Forecasts, Calibration and Sharpness
• Title not available (Why is that?)
• Analyzing stochastic computer models: a review with opportunities
• Distributional regression forests for probabilistic precipitation forecasting in complex terrain
• Title not available (Why is that?)
• Bandwith selection for the smoothing of distribution functions
• A review and comparison of bandwidth selection methods for kernel regression
• Stochastic orders
• Multivariate Student-t regression models: Pitfalls and inference
• Uncertainty quantification. Theory, implementation, and applications
• Title not available (Why is that?)
• Inferences Under a Stochastic Ordering Constraint
• Uncertainty quantification in complex simulation models using ensemble copula coupling
• Scoring Rules for Continuous Probability Distributions
• A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing
• Ensemble forecasting
• Introduction to uncertainty quantification
• Computing electricity spot price prediction intervals using quantile regression and forecast averaging
• Handbook of uncertainty quantification. In 2 volumes
• Nonparametric shape-restricted regression
• Monotone least squares and isotonic quantiles
• Receiver operating characteristic (ROC) curves: equivalences, beta model, and minimum distance estimation
• Isotonic Distributional Regression
• Accelerating the pool-adjacent-violators algorithm for isotonic distributional regression
• ACCRUE: accurate and reliable uncertainty estimate in deterministic models
• Distributional (Single) Index Models

This page was built for publication: Easy Uncertainty Quantification (EasyUQ): Generating Predictive Distributions from Single-Valued Model Output

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6154539)