False confidence, non-additive beliefs, and valid statistical inference

DOI10.1016/J.IJAR.2019.06.005zbMATH Open1471.62236arXiv1607.05051OpenAlexW2943819575WikidataQ127581138 ScholiaQ127581138MaRDI QIDQ2302767FDOQ2302767

Authors: Ryan Martin 

Publication date: 26 February 2020

Published in: International Journal of Approximate Reasoning (Search for Journal in Brave)

Abstract: Statistics has made tremendous advances since the times of Fisher, Neyman, Jeffreys, and others, but the fundamental and practically relevant questions about probability and inference that puzzled our founding fathers remain unanswered. To bridge this gap, I propose to look beyond the two dominating schools of thought and ask the following three questions: what do scientists need out of statistics, do the existing frameworks meet these needs, and, if not, how to fill the void? To the first question, I contend that scientists seek to convert their data, posited statistical model, etc., into calibrated degrees of belief about quantities of interest. To the second question, I argue that any framework that returns additive beliefs, i.e., probabilities, necessarily suffers from {em false confidence}---certain false hypotheses tend to be assigned high probability---and, therefore, risks systematic bias. This reveals the fundamental importance of {em non-additive beliefs} in the context of statistical inference. But non-additivity alone is not enough so, to the third question, I offer a sufficient condition, called {em validity}, for avoiding false confidence, and present a framework, based on random sets and belief functions, that provably meets this condition. Finally, I discuss characterizations of p-values and confidence intervals in terms of valid non-additive beliefs, which imply that users of these classical procedures are already following the proposed framework without knowing it.

Full work available at URL: https://arxiv.org/abs/1607.05051

Recommendations

• Confidence is epistemic probability for empirical science
• Nonadditive probabilities in statistics
• Frequency-calibrated belief functions: review and new insights
• A frequentist framework of inductive reasoning
• Confidence, Likelihood, Probability

zbMATH Keywords

Bayesinferential modelrandom setfiducialplausibility function\( p \)-value

Mathematics Subject Classification ID

Foundations and philosophical topics in statistics (62A01) Nonparametric tolerance and confidence regions (62G15)

Cites Work

• Statistical Inference as Severe Testing
• Title not available (Why is that?)
• Principles of uncertainty
• Conditional Inferential Models: Combining Information for Prior-Free Probabilistic Inference
• The IM-based method for testing the non-inferiority of odds ratio in matched-pairs design
• Bayes' Theorem in the 21st Century
• On Some Principles of Statistical Inference
• Frequency-calibrated belief functions: review and new insights
• Safe probability
• On an inferential model construction using generalized associations
• Gibbs posterior inference on the minimum clinically important difference
• Satellite conjunction analysis and the false confidence theorem
• Generalized Fiducial Inference for Ultrahigh-Dimensional Regression
• Response to the ASA’s Statement on p-Values: Context, Process, and Purpose
• Calibrating general posterior credible regions
• A Statistical Inference Course Based on p-Values
• Fisher's “Inverse Probability” of 1930
• Discussion
• Assessing the Statistical Analyses Used in Basic and Applied Social Psychology After Their p-Value Ban
• Interval estimation for a binomial proportion. (With comments and a rejoinder).
• Title not available (Why is that?)
• Sampling and Bayes' Inference in Scientific Modelling and Robustness
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review
• Extreme value theory. An introduction.
• THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL
• Title not available (Why is that?)
• Robust Statistics
• Bayesianly justifiable and relevant frequency calculations for the applied statistician
• Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
• Title not available (Why is that?)
• Some Impossibility Theorems in Econometrics With Applications to Structural and Dynamic Models
• Theory of Random Sets
• Principles of Statistical Inference
• Title not available (Why is that?)
• R. A. Fisher and the fiducial argument
• Generalized Confidence Intervals
• Generalized fiducial inference for wavelet regression
• Confidence and Likelihood^*
• Title not available (Why is that?)
• Title not available (Why is that?)
• Fiducial Generalized Confidence Intervals
• The ASA Statement on p-Values: Context, Process, and Purpose
• Title not available (Why is that?)
• Title not available (Why is that?)
• Upper and Lower Probabilities Induced by a Multivalued Mapping
• Default reasoning and possibility theory
• Towards a unified theory of imprecise probability
• Bayes, reproducibility and the quest for truth
• The Dempster-Shafer calculus for statisticians
• The sources of Kolmogorov's \textit{Grundbegriffe}
• On the Birnbaum argument for the strong likelihood principle
• A note on p-values interpreted as plausibilities
• On the Foundations of Statistical Inference
• What does the proof of Birnbaum's theorem prove?
• Title not available (Why is that?)
• Is Bayes posterior just quick and dirty confidence?
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Pivotal Models and the Fiducial Argument
• Inferential Models: A Framework for Prior-Free Posterior Probabilistic Inference
• Generalized fiducial inference via discretization
• Title not available (Why is that?)
• Frequentist statistics as a theory of inductive inference
• Inferential Models
• Title not available (Why is that?)
• Integrated likelihood methods for eliminating nuisance parameters. (With comments and a rejoinder).
• The Behrens-Fisher problem revisited: a Bayes-frequentist synthesis
• Title not available (Why is that?)
• Default Priors for Bayesian and Frequentist Inference
• Three enigmatic examples and inference from likelihood
• Title not available (Why is that?)
• On random sets and belief functions
• Symmetric Measures on Cartesian Products
• An Example of Wide Discrepancy Between Fiducial and Confidence Intervals
• Title not available (Why is that?)
• Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector
• An Introduction to Random Sets
• Title not available (Why is that?)
• New Methods for Reasoning Towards Posterior Distributions Based on Sample Data
• Upper and Lower Probabilities Generated by a Random Closed Interval
• The nonexistence of \(100(1-\alpha)\%\) confidence sets of finite expected diameter in errors-in-variables and related models
• Theory of capacities
• Maximum Likelihood: An Introduction
• Exact prior-free probabilistic inference on the heritability coefficient in a linear mixed model
• Bayes' theorem for Choquet capacities
• What is a statistical model? (With comments and rejoinder).
• Minimax tests and the Neyman-Pearson lemma for capacities
• Title not available (Why is that?)
• On rereading R. A. Fisher
• R. A. Fisher's fiducial argument and Bayes' theorem
• A mathematical theory of hints. An approach to the Dempster-Shafer theory of evidence
• Classic works on the Dempster-Shafer theory of belief functions
• A mathematical theory of evidence turns 40
• On the Difficulties Inherent in Fisher's Fiducial Argument
• Upper and Lower Probability Inferences for Families of Hypotheses with Monotone Density Ratios
• Further Examples of Inconsistencies in the Fiducial Argument
• Allocations of probability
• A more powerful test for comparing two Poisson means
• Title not available (Why is that?)
• Mathematical foundations for a theory of confidence structures
• R. A. Fisher in the 21st century. Invited paper presented at the 1996 R. A. Fisher lecture. (With comments).
• R. A. Fisher on the history of inverse probability. With comments by Robin L. Plackett and G. A. Barnard and a rejoinder by the author
• Decision-making with belief functions: a review
• What did Fisher mean by inverse probability in 1912--1922?
• Inference about constrained parameters using the elastic belief method
• Marginal Inferential Models: Prior-Free Probabilistic Inference on Interest Parameters
• The Methodology of Scientific Research Programmes
• Understanding uncertainty
• Two classes of group divisible partial diallel crosses
• Practical Solutions of the Behrens-Fisher Problem
• R. A. Fisher on Bayes and Bayes' theorem
• Confidence, Likelihood, Probability
• Exact and efficient inference for partial Bayes problems
• Prior envelopes based on belief functions

Cited In (23)

• Generalized inferential models for censored data
• Direct and approximately valid probabilistic inference on a class of statistical functionals
• An inferential model-based method for testing homogeneity of several variances against tree-ordered alternatives
• Satellite conjunction analysis and the false confidence theorem
• Verified propagation of imprecise probabilities in non-linear ODEs
• Distribution-free inferential models: achieving finite-sample valid probabilistic inference, with emphasis on quantile regression
• Belief functions induced by random fuzzy sets: a general framework for representing uncertain and fuzzy evidence
• Games of incomplete information: a framework based on belief functions
• A practical strategy for valid partial prior-dependent possibilistic inference
• Confidence in confidence distributions!
• Valid Model-Free Prediction of Future Insurance Claims
• Towards a Theory of Valid Inferential Models with Partial Prior Information
• A comparison of learning rate selection methods in generalized Bayesian inference
• Gibbs posterior inference on multivariate quantiles
• A possibility-theoretic solution to Basu's Bayesian-frequentist via media
• Model-free posterior inference on the area under the receiver operating characteristic curve
• Validity, consonant plausibility measures, and Conformal prediction
• Game-theoretic statistical inference: optional sampling, universal inference, and multiple testing based on e-values. Abstracts from the workshop held May 5--10, 2024
• Special issue from the 5th international conference on belief functions (BELIEF 2018)
• Nonadditive probabilities in statistics
• Model misspecification, Bayesian versus credibility estimation, and Gibbs posteriors
• Valid inferential models for prediction in supervised learning problems
• New two-sided confidence intervals for binomial inference derived using Walley's imprecise posterior likelihood as a test statistic

Uses Software

• BayesDA

This page was built for publication: False confidence, non-additive beliefs, and valid statistical inference

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2302767)