Fact vs. Prediction: Decoding the News
Paper + Venue + Topic
The paper Extracting Predictive Statements with Their Scope from News Articles was presented at the Twelfth International AAAI Conference on Web and Social Media (ICWSM 2018). This research explores the fields of natural language processing and temporality analysis.
The Problem + Why it Matters
This work is motivated by the role of journalists writing news articles to inform the public. Furthermore, the authors narrow their focus on the information these journalists present into facts or references to the future, namely predictions. They estimate news articles to have a large volume of predictions.
The main argument of this work is the linguistic interplay between factual reporting, the truthfulness of a fact, and future-oriented predictions in news articles. Distinguishing between these is imperative because the public relies on the veracity of all three to make timely decisions. Furthermore, predictions regarding natural disasters, military conflicts, and pandemics provide the public with critical knowledge of what may occur. On this note, the journalist seeks to establish credibility; thus, accurate predictions allow the public to prepare in advance rather than simply react, and help them identify which journalists to adhere to for information and, therefore, trust.
The authors state, "An information retrieval and extraction system must handle the above-mentioned interplay in a systematic manner to address the problem of determining predictive statements and their scope in news articles." I hypothesize automating this process such that public users can see the distinctions will save time as the system will provide them with the predictions.
In addition, the authors accentuate the difficulty of differentiating factual phrases from predictive clauses. Linguistic patterns and predictive keywords are the intuitive or naive approaches. They fail due to not capturing the entirety of a predictive clause/sentence and when the fact/outcome is known. See Statement II in the image below as an example where there exists a predictive clause ("government promised to extend the maternity leave") and a fact ("not keep its promise").
Thus, the scope of this work is not only to classify a statement as fact or prediction, but also to classify the clause as factual or predictive.
The Method
The authors state, "we define factual base for a prediction as knowledge, facts, science, experiments etc, based on which the prediction is being made." Additionally, they use the Merriam-Webster meaning of prediction-"any statement made in reference to future..." and further states, "classify sentences as predictive if it refers to an action or probability of a future course of an event...". This is the essence of how other works use this prediction term.
The datasets breakdown are
| Dataset | Predictive Statements | Factual Statements | Predictive Clauses | Factual Clauses |
|---|---|---|---|---|
| Set3480 | 1,227 | 1,011 | 1,740 | 1,740 |
| Set200 | 70 | 58 | 100 | 100 |
Furthermore, Set3480 is a well-balanced dataset across statements (54.8% predictive / 45.2% factual). Similarly, Set200 is well-balanced across statements (54.7% predictive / 45.3% factual). At the clause level, both datasets are perfectly balanced (50.0% predictive / 50.0% factual). Since both datasets are well-balanced, this increases the likelihood that model performance is driven by an ability to distinguish between factual and predictive language rather than by class imbalance.
With the aforementioned, they performed three tasks that are hierarchy dependent.
Clausal dependency relations extraction: to split the sentence into its constituent clauses and extract the dependencies of how a clause modifies other clauses in the sentence.
This process involves a rigorous linguistic foundation—utilizing parse trees, part-of-speech tagging, and specific rules to define predictive versus factual clauses, even across multiple sentences. See screenshot for more:
This foundational step is critical because the subsequent classification tasks rely entirely on these identified clausal relationships.
Classification of simple clauses (CSC): to label each free-standing clause as predictive or factual, by learning lexical and syntactic patterns of relation triplets in a clause.
This task identifies and extracts component clauses by separating predictive elements from factual ones. For example, given the statement from above "Though the government promised to extend the maternity leave, it could not keep its promise," the system isolates the predictive clause ("the government promised to extend the maternity leave") from the factual clause ("Though it could not keep its promise").
This step is crucial because they use these features to train a classifier that can distinguish between predictive and factual clauses. This granularity is essential since some systems may miss a predictive clause buried within a factual sentence, causing them to skip critical downstream processing. Furthermore, these clause-level labels serve as the necessary input for the final sentence-level classification.
Clause labels disambiguation: a classifier (CCDR) to predict the label of a sentence as predictive or factual, from its clause labels and dependency relations.
Building on the previous two steps, this task resolves linguistic ambiguities to determine if a sentence as a whole is predictive or factual. For example, the system would label the statement "Saulius Mikoliunas could also face action after three fans were arrested for throwing coins on the pitch" as predictive, while labeling "Though the government promised to extend the maternity leave, it could not keep its promise" as factual.
This final step is imperative because it synthesizes granular linguistic data into a single, usable label. From a systems perspective, automating this disambiguation process allows end-users to immediately distinguish forecasts from facts without manually parsing the text—validating my earlier hypothesis that such automation saves critical time for the user.
What is Genuinely Novel
The novelty of this work lies in the hierarchical execution of its three tasks, utilizing traditional dependency relations and triplets to move from a broad to a granular analysis. This structure is critical because each task addresses the limitations of the previous one.
Take, for example, "Statement II" mentioned earlier. At the sentence level, the authors labeled it as a fact; if the analysis stopped there, the sentence would be ignored. However, by decomposing the sentence into components, the system discovers a hidden predictive clause. This proves that a predictive element can exist even when the overall statement is factual—a nuance that would be completely missed without this hierarchical approach.
Limitations
A significant limitation is the lack of public access to the code and the dataset, which prevents verification of the results. I have reached out to request the dataset and am currently waiting for a response.
The authors state that any patterns appearing fewer than 15 times were removed from their datasets. This implies that the model does not capture every possible linguistic pattern. This is a common downside of relying on fixed rules and patterns, as it ignores the dynamic variety of language that doesn't fit into a specific list.
Ideas for Our Work
This paper reinforces the need for the broader temporal scope implemented in TOLSA-M. While this work explicitly defines predictive statements as referring to the future, TOLSA-M expands this by incorporating past and present tenses. This allows us to examine "past-tense predictive statements"—essentially tracking a prediction from its origin to its fruition—to see how a narrative evolves over time.
Additionally, the authors' Classification of Simple Clauses (CSC) task—which extracts the "predictive clause"—aligns directly with our TOLSA-M Taxonomy. Specifically, their focus on the predictive clause maps to our "Document" dimension and its "span/phrase" sub-dimension. By combining this granular clause extraction with an analysis of discourse genres (such as beliefs, claims, and facts), we can move beyond simple keyword matching and rigid rules to achieve a more precise extraction of measurable outcomes and their associated temporal properties.
While the paper relies on a fixed set of patterns and rules, the TOLSA-M Taxonomy allows us to either incorporate these rules as a baseline or transcend them through a more flexible, metadata-driven approach. This is critical because, as noted earlier, rule-based systems struggle to capture the dynamic variety of language when expressions fall outside their predefined lists.
For more information about our research, return to our homepage: ufdatastudio.com.
For more from Detravious on NLP research (including the TOLSA-M Taxonomy), research and industry collaborations, faith in Jesus, and broader writings, visit: Research Portfolio | LinkedIn | Medium, or contact via email at dj.brinkley@ufl.edu.
