Mining events with declassified diplomatic documents

DOI10.1214/20-AOAS1344MaRDI QIDQ2078743zbMATH OpenOpenAlexFDO

Authors Yuanjun Gao, Jack Goetz, Matthew Connelly, Rahul Mazumder

Publication date 3 March 2022

Published in The Annals of Applied Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1712.07319

zbMATH Keywords

fused Lassooptimizationsignal estimationU.S. historyU.S. National Archives

Mathematics Subject Classification ID

Applications of statistics (62P99) History, political science (91F10)

Abstract: Since 1973 the State Department has been using electronic records systems to preserve classified communications. Recently, approximately 1.9 million of these records from 1973-77 have been made available by the U.S. National Archives. While some of these communication streams have periods witnessing an acceleration in the rate of transmission; others do not show any notable patterns in communication intensity. Given the sheer volume of these communications -- far greater than what had been available until now -- scholars need automated statistical techniques to identify the communications that warrant closer study. We develop a statistical framework that can semi-automatically identify from a large corpus of documents a handful that historians would consider more interesting electronic records. Our approach brings together related but distinct statistical concepts from nonparametric signal estimation and statistical hypothesis testing -- which when put together help us identify and analyze various geometrical aspects of the communication streams. Dominant periods of heightened and sustained activities aka bursts, as identified through these methods, correspond well with historical events recognized by standard reference works on the 1970s.

Recommendations

• Reading Encrypted Diplomatic Correspondence: An Undergraduate Research Project
• Extracting the information: espionage with double crossing

Cites work

• $\ell_1$ Trend Filtering
• A Cluster Analysis Method for Grouping Means in the Analysis of Variance
• A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
• Adaptive piecewise polynomial estimation via trend filtering
• Algorithms for the optimal identification of segment neighborhoods
• Best subset selection via a modern optimization lens
• Circular binary segmentation for the analysis of array-based DNA copy number data
• Consistencies and rates of convergence of jump-penalized least squares estimators
• CONTINUOUS INSPECTION SCHEMES
• Curve fitting by segmented straight lines
• Fast best subset selection: coordinate descent and local combinatorial optimization algorithms
• Gradient methods for minimizing composite functions
• Greedy Kernel Change-Point Detection
• High-dimensional variable selection
• scientific article; zbMATH DE number 425941 (Why is no real title available?)
• Introductory lectures on convex optimization. A basic course.
• Locally adaptive regression splines
• Optimal detection of changepoints with a linear computational cost
• Scan statistics
• SparseNet: coordinate descent with nonconvex penalties
• Sparsity and Smoothness Via the Fused Lasso

Describes a project that uses

Uses Software

• ElemStatLearn
• L0Learn
• sparsenet

This page was built for publication: Mining events with declassified diplomatic documents

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2078743)