The potential and perils of preprocessing: building new foundations

DOI10.3150/13-BEJSP16zbMATH Open1440.62019arXiv1309.6790OpenAlexW2023338600MaRDI QIDQ373524FDOQ373524

Authors: Alexander W. Blocker, Xiao-Li Meng 

Publication date: 17 October 2013

Published in: Bernoulli (Search for Journal in Brave)

Abstract: Preprocessing forms an oft-neglected foundation for a wide range of statistical and scientific analyses. However, it is rife with subtleties and pitfalls. Decisions made in preprocessing constrain all later analyses and are typically irreversible. Hence, data analysis becomes a collaborative endeavor by all parties involved in data collection, preprocessing and curation, and downstream inference. Even if each party has done its best given the information and resources available to them, the final result may still fall short of the best possible in the traditional single-phase inference framework. This is particularly relevant as we enter the era of "big data". The technologies driving this data explosion are subject to complex new forms of measurement error. Simultaneously, we are accumulating increasingly massive databases of scientific analyses. As a result, preprocessing has become more vital (and potentially more dangerous) than ever before.

Full work available at URL: https://arxiv.org/abs/1309.6790

Recommendations

• Principled statistical inference in data science
• Statistical strategies for the analysis of massive data sets
• The role of statistics in the era of big data: a computational scientist' perspective
• Big data: the next challenge for statistics
• The future of statistics and data science

zbMATH Keywords

measurement errormultiple imputationdata compressiondata repositoriesmultiphase inferencestatistical principles

Mathematics Subject Classification ID

Computational methods for problems pertaining to statistics (62-08) Foundations and philosophical topics in statistics (62A01) Sampling theory, sample surveys (62D05) Missing data (62D10)

Cites Work

• A Predictive Approach to Model Selection
• Equivalent Comparisons of Experiments
• Partial likelihood
• Inference and missing data
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Consistent Estimates Based on Partially Consistent Observations
• Handbook of Markov Chain Monte Carlo
• Multiple Imputation After 18+ Years
• The Selection of Prior Distributions by Formal Rules
• Statistical decision theory and Bayesian analysis. 2nd ed
• Significance analysis of microarrays applied to the ionizing radiation response
• Exploration, normalization, and summaries of high density oligonucleotide array probe level data
• An invariant form for the prior probability in estimation problems
• Title not available (Why is that?)
• Invariant Prior Distributions
• Quantization
• Comparison of experiments and information measures
• On rereading R. A. Fisher
• On surrogate loss functions and \(f\)-divergences
• Several Bayesians: a review. (With discussion)
• On the Reconciliation of Probability Assessments
• Proper and Improper Multiple Imputation
• Sufficiency and Approximate Sufficiency
• Parameter Estimation for the Exponential-Normal Convolution Model for Background Correction of Affymetrix GeneChip Data
• On a Necessary and Sufficient Condition for Admissibility of Estimators When Strictly Convex Loss is Used
• Classifying Gene Expression Profiles from Pairwise mRNA Comparisons

Cited In (6)

• Multiple Improvements of Multiple Imputation Likelihood Ratio Tests
• Recent Developments in the Theory of Pre-processing
• Conducting highly principled data science: a statistician's job and joy
• Discussion: The Q‐q Dynamic for Deeper Learning and Research
• Nonstandard conditionally specified models for nonignorable missing data
• Multi-way blockmodels for analyzing coordinated high-dimensional responses

Uses Software

• Bioconductor
• gcrma
• limma

This page was built for publication: The potential and perils of preprocessing: building new foundations

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q373524)