PTNews Corpus
DOI10.5281/zenodo.3908507Zenodo3908507MaRDI QIDQ6701273FDOQ6701273
Dataset published at Zenodo repository.
Publication date: 25 June 2020
The PTNews Corpus is a collection of over 19 million tokens extracted from 10 years of political news articles (in Portuguese) from the Portuguese newspaper PÚBLICO. The corpus is available under the Creative Commons Attribution-NonCommercial-ShareAlike Licence. The material contained on the PTNews Corpus is 2010-2020 PÚBLICO Comunicao Social SA. The corpus sizes between the preprocessed version of Penn Treebank (PTB) and WikiText-103. Similarly to WikiText, PTNews has a larger vocabulary than PTB and retains the original case, punctuation and numbers. This corpus contains over 31000 publicly available full articles which makes it well suited for models that can take advantage of long-term dependencies. The corpus is available as a word-level collection of articles in two version: the first (ptnews_origin) contains a single file with all the articles in the form: title, URL, date, body; the second, contains only the title and body of the news articles and it is split into train, test, validation sets. In this processed version, the words with less than 3 occurrences are mapped to the unk token. Each sentence in an article body occupies a single line of the dataset and the end of paragraph is marked with the eop tag at the end of a sentence. Portuguese words resulting from contractions like desta, ou nesta are separated into d, esta, n, esta, respectively. Sample article: Carlos César : Cavaco " cansado e sem entusiasmo " quis afastar responsabilidades sobre a crise https://publico.pt/2010/06/10/politica/noticia/carlos-cesar-cavaco-cansado-e-sem-entusiasmo-quis-afastar-responsabilidades-sobre-a-crise-1441369 2010-06-10 15:38:00 O presidente do Governo Regional dos Açores , Carlos César , considerou hoje que Cavaco Silva esteve " cansado e sem entusiasmo " no discurso do Dia de Portugal , onde afastou responsabilidades sobre a actual crise . eop " O país ouviu um Presidente cansado e sem entusiasmo , que andou às voltas com os papéis para dizer que não tinha nada a ver com as razões da crise " , afirmou Carlos César , num comentário à Lusa sobre o discurso do Presidente da República na cerimónia oficial do 10 de Junho , realizada em Faro . eop Carlos César considerou , no entanto , " positivo " que Cavaco Silva tenha feito " um discurso alinhado com um tema recorrente na apreciação do momento que vivemos , o da coesão e da corresponsabilização " . eop No mesmo sentido , manifestou concordância com o apelo que Cavaco Silva fez " à responsabilidade dos empregadores e empregados " , mas deixou um alerta relativamente à referência do Presidente da República à necessidade de " limpar Portugal " . eop Para Carlos César , se essa referência " for despida de conteúdo institucional útil , tratou-se de mais um discurso que se perderá na babugem política d aquilo que Cavaco Silva entendeu recordar como o ' rectângulo ' " . eop Reporting Results If you wish to report results or other resources obtained on the PTNews contact Davide Nunes with the following information: Task: e.g. Language Modelling, Semantic Similarity, etc; Publication URL: url to published article or preprint; Type of Model: LSTM Neural Network, n-grams, GloVe vectors, etc; Evaluation Metrics: e.g. validation and testing perplexities in the case of language modelling. They will be displayed here Preprocessed Corpus Statistics articles: 31.919 articles by split: train: 25.537 test: 3.191 val: 3.191 unique tokens: 68.318 unique OoV Tokens: 76.157 total tokens: 19.021.661 total OoV tokens: 95.043 OoV rate: 0.5% tokens by split: train: 15.242.995 test: 1.895.184 val: 1.883.482 Contact Information If you have questions about the corpus or want to report benchmark results, contact Davide Nunes.
This page was built for dataset: PTNews Corpus
