Fact vs Prediction: Decoding the News
Paper + Venue + Topic
The paper Extracting Predictive Statements with Their Scope from News Articles by Navya Yarrabelly and Kamalakar Karlapalem was accepted at the Twelfth International AAAI Conference on Web and Social Media (ICWSM 2018). This research explores natural language processing (NLP) and temporality analysis in news articles. More specifically, it examines predictive statements and clauses, factual statements and clauses, and factual bases. To do this, the authors scraped online news data and used NLP techniques such as part-of-speech tagging, n-grams, triplets, parse trees, and dependency relations. They also made use of Stanford SUTime and DBPedia entity linking.
The Problem + Why it Matters
This work is motivated by the role of journalists writing news articles to inform the public. Furthermore, the authors narrow their focus to factual statements and references to the future, namely predictions. They also define a factual base for a prediction as “knowledge, facts, science, experiments etc, based on which the prediction is being made.” They estimate that news articles contain a large volume of predictions, where journalists seek to establish credibility and the public perceives which journalists to receive information from and, therefore, trust.
The main focus of this work is the linguistic interplay between facts, the truthfulness of facts, and future-oriented prediction components in news articles. This work does not establish factual truth or perform fact verification. I denote that distinguishing between facts and predictions is imperative because the public interprets each differently, which may lead to a swift reaction or deliberate preparation. Some scenarios where there is a difference between reaction and preparation include natural disasters, military conflicts, and pandemics. Thus, learning the differences can improve public decision-making and behavioral change.
In addition, previous methods cannot reliably capture both factual clauses and predictive clauses, nor how the clauses relate to one another. This can lead to classifying an entire news statement as factual while failing to extract the predictive clause within it. See Statement II in the image below as an example where there exists a factual clause (“not keep its promise”) and a predictive clause (“government promised to extend the maternity leave”).
Linguistic patterns and predictive keywords are intuitive or naive approaches. However, the authors argue that “there is an imperative need to address the solution with a new NLP perspective for processing of a sentence with its component clauses and analyze the dependencies between the clauses, which other methods lack in their approach.”
Following this, Yarrabelly and Karlapalem state, “An information retrieval and extraction system must handle the above-mentioned interplay in a systematic manner to address the problem of determining predictive statements and their scope in news articles.” Systems will be insufficient if they can't extract a predictive clause and its scope. Accurately extracting future-related information can save the public time (e.g., an approximate hurricane landfall time), improve safety (e.g., an expected military escalation or ceasefire), and support public health (e.g., a projected rise in pandemic cases or hospitalizations) by making explicit predictions easier to find.
Thus, the scope of this work is not only to classify a statement as factual or predictive, but also to classify the clauses within it as factual or predictive.
The Method
As previously stated, Yarrabelly and Karlapalem define a factual base for a prediction as “knowledge, facts, science, experiments etc, based on which the prediction is being made.” Additionally, they use the Merriam-Webster meaning of prediction-"any statement made in reference to future..." and further states, "classify sentences as predictive if it refers to an action or probability of a future course of an event...". This is the essence of how other works use this prediction term. Relying on these definitions guided their dataset collection and annotation process. They scraped English news articles from BBC and Times Now. More specifically, their domains of interests were Politics and Economy and Sports, where the distribution of these are:
| Domain | Dataset | Predictive Statements (+) | Factual Statements (-) | Predictive Clauses (+) | Factual Clauses (-) |
|---|---|---|---|---|---|
| Politics and Economy | Set3480 | 1,227 | 1,011 | 1,740 | 1,740 |
| Sports | Set200 | 70 | 58 | 100 | 100 |
Furthermore, we observe at the sentence level, Set3480 is a decently-balanced dataset across statements (54.8% predictive / 45.2% factual). Similarly, Set200 is decently-balanced across statements (54.7% predictive / 45.3% factual). At the clause level, both datasets are perfectly balanced (50.0% predictive / 50.0% factual). Since both datasets are balanced, this increases the likelihood that model performance is driven by an ability to distinguish between factual and predictive language rather than by class imbalance.
From defining problem, terms, to dataset construction and labelling, now we get into the three three hierarchial dependent tasks.
- Clausal dependency relations extraction: to split the sentence into its constituent clauses and extract the dependencies of how a clause modifies other clauses in the sentence.
- Classification of simple clauses (CSC): to label each free-standing clause as predictive or factual, by learning lexical and syntactic patterns of relation triplets in a clause.
- Clause labels disambiguation: a classifier (CCDR) to predict the label of a sentence as predictive or factual, from its clause labels and dependency relations.
Clausal dependency (CD) relations extraction:
This process involves a rigorous linguistic foundation—utilizing parse trees, part-of-speech tagging, and specific rules to define predictive versus factual clauses, even across multiple sentences. See screenshot for more:
This foundational step is critical because the subsequent classification tasks rely entirely on these identified clausal relationships.
Classification of simple clauses (CSC):
This task identifies and extracts component clauses by separating predictive elements from factual ones. For example, given the statement from above "Though the government promised to extend the maternity leave, it could not keep its promise," the system isolates the predictive clause ("the government promised to extend the maternity leave") from the factual clause ("Though it could not keep its promise").
This step is crucial because they use these features to train a classifier that can distinguish between predictive and factual clauses. This granularity is essential since some systems may miss a predictive clause buried within a factual sentence, causing them to skip critical downstream processing. Furthermore, these clause-level labels serve as the necessary input for the final sentence-level classification.
Clause labels disambiguation
Building on the previous two steps (CD and CSC), this task resolves linguistic ambiguities to determine if a sentence as a whole is predictive or factual. For example, the system would label the statement "Saulius Mikoliunas could also face action after three fans were arrested for throwing coins on the pitch" as predictive, while labeling "Though the government promised to extend the maternity leave, it could not keep its promise" as factual.
This final step is imperative because it synthesizes granular linguistic data into a single, usable label (fact or prediction). As mentioned in the problem section, from a systems perspective, automating this disambiguation process allows end-users to immediately distinguish forecasts from facts without manually parsing the text—validating my earlier hypothesis that such automation saves critical time for the user.
What is Genuinely Novel
The novelty of this work lies in the hierarchical execution of its three tasks: starting with in-sentence clausal relationships, identifying factual versus predictive clauses, and then performing sentence-level classification. This structure begins with the relationships among clauses and broadens to the sentence-level label.
Take, for example, “Statement II” mentioned earlier. At the sentence level, the authors labeled it as factual; if the analysis stopped there, a system could fail to extract the predictive clause for downstream tasks, such as informing the public or later verifying the prediction’s correctness. However, by decomposing the sentence into components, the system identifies the embedded predictive clause. This shows that a predictive element can exist even when the overall statement is factual—a nuance that would be missed without this hierarchical approach.
Limitations
A limitation is the lack of public access to the code and dataset, which prevents verification of the results. I have reached out to request the dataset and am currently waiting for a response.
The authors state that patterns appearing fewer than 15 times were removed from their datasets. Their approach is not simply a fixed-rule system: it uses rules and linguistic processing for clause extraction and feature construction, then trains a classifier to label clauses as predictive or factual. However, excluding rare patterns from the feature sets can reduce coverage for uncommon language forms and create a generalization risk when the model encounters expressions not sufficiently represented in the training data.
Ideas for Our Work
This paper reinforces the need for the broader temporal scope implemented in TOLSA-M. While this work explicitly defines predictive statements as referring to the future, TOLSA-M expands this by incorporating past and present tenses. This allows us to examine "past-tense predictive statements"—essentially tracking a prediction from its origin to its fruition—to see how a narrative evolves over time.
Additionally, the Classification of Simple Clauses (CSC) task—which extracts the "predictive clause"—aligns directly with our TOLSA-M Taxonomy. Specifically, Yarrabelly and Karlapalem focus on the predictive clause maps to our "Document" dimension and its "phrase" sub-dimension. By combining this granular clause extraction with an analysis of discourse genres (such as beliefs, claims, and facts).
Just as a predictive clause can hide within a statement, the attribute, metric, and slope can within the outcome property of TOLSA-M. Thus, instead of having our systems remain high level granuar, we'd need to dive deeper to extract the subcomponents.
While the paper relies on a fixed set of patterns and rules, the TOLSA-M Taxonomy allows us to either incorporate these rules as a baseline or transcend them through a more flexible, metadata-driven approach. This is critical because, as noted earlier, rule-based systems struggle to capture the dynamic variety of language when expressions fall outside their predefined lists.
For more information about our research, return to our homepage: ufdatastudio.com.
For more from Detravious on NLP research (including the TOLSA-M Taxonomy), research and industry collaborations, faith in Jesus, and broader writings, visit: Research Portfolio | LinkedIn | Medium, or contact via email at dj.brinkley@ufl.edu.
