Concise comparative summaries (CCS) of large text corpora with a human experiment

DOI10.1214/13-AOAS698MaRDI QIDQ2453687zbMATH OpenOpenAlexFDO

Authors Yanyan Li

Publication date 10 June 2014

Published in The Annals of Applied Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1404.7362, https://projecteuclid.org/euclid.aoas/1396966296

zbMATH Keywords

Lassosparse modelingtext summarizationco-occurrencehigh-dimensional analysisL1 regularized logistic regressionL2 normalizationtf-idf

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Ridge regression; shrinkage estimators (Lasso) (62J07) Applications of statistics (62P99)

Abstract: In this paper we propose a general framework for topic-specific summarization of large text corpora and illustrate how it can be used for the analysis of news databases. Our framework, concise comparative summarization (CCS), is built on sparse classification methods. CCS is a lightweight and flexible tool that offers a compromise between simple word frequency based methods currently in wide use and more heavyweight, model-intensive methods such as latent Dirichlet allocation (LDA). We argue that sparse methods have much to offer for text analysis and hope CCS opens the door for a new branch of research in this important field. For a particular topic of interest (e.g., China or energy), CSS automatically labels documents as being either on- or off-topic (usually via keyword search), and then uses sparse classification methods to predict these labels with the high-dimensional counts of all the other words and phrases in the documents. The resulting small set of phrases found as predictive are then harvested as the summary. To validate our tool, we, using news articles from the New York Times international section, designed and conducted a human survey to compare the different summarizers with human understanding. We demonstrate our approach with two case studies, a media analysis of the framing of "Egypt" in the New York Times throughout the Arab Spring and an informal comparison of the New York Times' and Wall Street Journal's coverage of "energy." Overall, we find that the Lasso with $L^2$ normalization can be effectively and usefully used to summarize large corpora, regardless of document size.

Recommendations

• Automatic text summarization using latent semantic analysis
• scientific article; zbMATH DE number 1843147
• Optimizing word set coverage for multi-event summarization
• Multi-document summarization via group sparse learning
• scientific article; zbMATH DE number 2086484

Cites work

• scientific article; zbMATH DE number 5957506 (Why is no real title available?)
• scientific article; zbMATH DE number 2014725 (Why is no real title available?)
• scientific article; zbMATH DE number 1869443 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• 10.1162/153244303322753670
• 10.1162/jmlr.2003.3.4-5.993
• Applied Bayesian and classical inference. The case of the Federalist papers. 2nd ed. of: Inference and disputed authorship: The Federalist
• Safe Feature Elimination in Sparse Supervised Learning

Cited in

(3)

• Sequential Text-Term Selection in Vector Space Models
• Conducting sparse feature selection on arbitrarily long phrases in text corpora with a focus on interpretability
• scientific article; zbMATH DE number 7370606 (Why is no real title available?)

This page was built for publication: Concise comparative summaries (CCS) of large text corpora with a human experiment

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2453687)