Any Inspection is Manipulable
The calibration tests treated in the literature are pure: After any history it is known what specific event (for instance, `rain', `foggy') is being considered. Here, mixed calibration tests are introduced, where, after each history, a randomly chosen event is checked. A mixed test checks whether the average prediction of these randomly chosen events is asymptotically equal to the frequency of their realizations. Like a deterministic test, a mixed calibration test is also type \(I\)-error free. The main result of this paper shows that there is no mixed calibration test that rejects any inaccurate forecast. That is, for any test of this kind, there exists a forecasting scheme that is always consistent with the realized events. A constructive method to produce this scheme is provided as well. Another interpretation of the main result involves a forecaster and an inspector. The inspector randomly chooses a calibration test according to a known distribution. It is shown that a forecaster, without knowing anything regarding the way states are selected can pass almost every test chosen by the inspector, independently of the sequence of realizations. In other words, the forecaster has a forecasting method that is consistent with the observed facts in the meaning induced by almost all tests. Instead of randomly choosing one test, as discussed previously, the inspector can employ a set of tests simultaneously. The main result implies that if the inspector employs countably many calibration tests, for instance all those computable by a Turing machine, a forecaster can, likewise, pass them all simultaneously. It means that a consultant or a scientist (e.g., a weather forecaster) who knows the (countable) set of tests his model needs to pass, can manipulate. That is, no matter what the sequence of realizations is, his manipulative forecast will pass all the tests. The proof of the main result relies on an extension of Blackwell's approachability theorem to games with payoffs in infinite dimensional spaces.
- On calibration error of randomized forecasting algorithms
- A nonmanipulable test
- Special issue: Learning in games: a symposium in honor of David Blackwell.
- Calibration and Bayesian learning.
- Belief-based equilibrium
- Transferable deposits as a screening mechanism
- Mostly calibrated
- On comparison of experts
- Approachability, regret and calibration: implications and equivalences
- Non-asymptotic calibration and resolution
- Merging and testing opinions
- A strategy-proof test of portfolio returns
- A geometric proof of calibration
- Non-Bayesian Testing of a Stochastic Prediction
- On Sequences with Non-learnable Subsequences
- Testing Multiple Forecasters
- Expressible inspections
- Repeated games with incomplete information
- Learning in Games
- Efficient testing of forecasts
- Good Randomized Sequential Probability Forecasting is Always Possible
- Calibration with Many Checking Rules
- Testing theories with learnable and predictive representations
- Nonmanipulable Bayesian testing
- On universal algorithms for adaptive forecasting
This page was built for publication: Any Inspection is Manipulable
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4531035)