Veridicality
Paper + Venue + Topic
This paper “i have a feeling trump will win..................”: Forecasting Winners and Losers from User Predictions on Twitter Sandesh Swamy, Alan Ritter, Marie-Catherine de Marneffe was published at the Conference on Empirical Methods in Natural Language Processing in 2017. This research explores social media predictions, veridicality classification, and outcome forecasting using the wisdom of crowds.
The Problem + Why it Matters
The main research question this paper addresses is, "Can popular beliefs on social media predict who will win?" In this paper, the authors associate popular belief with explicit predictions. I posit that they leverage social media data because it contains a wealth of information from a wide variety of users across infinite topics. Additionally, social media data is generated at an incredibly high velocity.
While the volume, variety, and velocity exist in social media data, I hypothesize that a significant challenge to extracting value lies in its loose linguistic and grammatical structure (e.g., slang, abbreviations, sarcasm). Understanding this structural challenge matters because accurately detecting popular belief is highly complex; there is no single, rule-based approach that can reliably capture meaning across such messy, unstructured text.
Building on this linguistic challenge, the authors state that social media "statements vary in the degree of certainty the author expresses toward the outcome." This introduces the critical concept of veridicality—the degree to which an author firmly asserts that an event will happen. Accurately identifying a user's level of certainty matters because it fundamentally changes how a system should interpret their prediction. For example, consider the difference between someone stating a hurricane will hit versus a hurricane has a chance of hitting. Accurately identifying these varying levels of certainty allows a system to isolate genuine predictions from mere hopes or guesses, ultimately determining the reliability of the crowd's true belief.
To address these challenges—specifically extracting explicit predictions and their degree of certainty from unstructured text—this work presents TwiVer, a system that classifies the veridicality toward future contests. Furthermore, the predictions of interest are explicit statements regarding future events that do not yet have known outcomes. In this context, contests are defined as "events planned to occur on a specific date, where a number of contenders compete and a single winner is chosen."
The Method
The authors' methodology follows a clear three-step pipeline: data collection, human annotation, and modeling/forecasting.
Data Collection
To study explicit predictions about future events with still-unknown outcomes, the authors first built a corpus of tweets centered on contests. They treated contests as events scheduled for a specific date, with multiple contenders and one eventual winner. To gather the data, they queried Twitter using the keyword "win" together with the name of a contender and a date range set before the contest took place. I hypothesize that using this specific keyword reduces the challenge of loose linguistic structure by filtering for explicit statements. In this way, the tweets were collected only from a period when the outcome was still unknown.
These queries were generated automatically across several event types, including elections, awards, and sports competitions. After collecting the tweets, they peformed data cleaning--restricted the data to English-only messages, removed duplicates using Jaccard similarity with a threshold of 0.7, removed URLs, and kept only tweets that explicitly mentioned contenders in the text. After this cleaning process, they were left with 57,711 tweets for winners and 55,558 tweets for losers. See below.
| Dataset Name | Queries Generated | # Winners | # Losers |
|---|---|---|---|
| Presidential Primaries | 483 | 20,347 | 17,873 |
| Oscars | 176 | 1,498 | 872 |
| Ballon d'Or | 18 | 3,492 | 3,285 |
| Tennis Grand Slams | 52 | 10,785 | 19,745 |
| Eurovision | 162 | 261 | 1,421 |
| 2016 US presidential elections | 76 | 9,679 | 3,966 |
| 2014 Indian general elections | 68 | 920 | 736 |
| Rugby World Cup | 6 | 272 | 379 |
| Football World Cup | 12 | 8,129 | 5,489 |
| Cricket World Cup | 18 | 2,328 | 1,792 |
Human Annotation
Once the tweets were collected, the second step utilized Amazon Mechanical Turk crowdsourcing to label the data. Annotators were asked to classify the veridicality (degree of certainty) of the tweets using five options: "Definitely Yes," "Probably Yes," "Uncertain about the outcome," "Probably No," and "Definitely No." For the system's pipeline, these categories were later merged into three broader labels:
- "Definitely Yes" and "Probably Yes" became positive veridicality
- "Uncertain about the outcome" became neutral
- "Definitely No" and "Probably No" became negative veridicality
(Note: This crowdsourcing approach contrasts with the data collection and annotation methodology used for TOLSA-M, which relies on a stricter adjudication protocol involving shared guidelines, group discussions, and majority voting for difficult sentences.
Modeling and Forecasting
In the final step, this annotated corpus was used to train a log-linear classifier designed to detect positive veridicality based on features like context words, distance between entities, punctuation, and dependency paths. With the classifier in place, the system could automatically process new tweets and forecast uncertain outcomes by aggregating the predictions of various users—effectively leveraging the "wisdom of crowds."
They model the conditional distribution using the following five features:
They ended up with the data splits shown in the table below:
| Split | Size |
|---|---|
| Train | 2480 |
| Validate | 354 |
| Test | 709 |
What is Genuinely Novel
(Section pending)
Limitations
Querying user predictions from tweets: The authors state, "We restricted the time range of the queries to retrieve only messages written before the time of the contest to ensure that outcomes were unknown when the tweets were written." This is a limitation because they ignore examples where predictions are referenced retrospectively. For instance, a system might miss scenarios where an outcome is known, but a second message quotes the initial prediction in the past tense.
Jaccard Similarity Threshold: They use a 0.7 threshold for Jaccard similarity to remove duplicates, but provide no theoretical or empirical justification for why 0.7 was chosen. A 0.7 threshold could lead to more or fewer duplicates compared to an alternative threshold.
Loss of Granularity: They collapse their five classifications into three. By doing so, they lose fine-grained distinctions in veridicality.
Ideas for Our Work
Aligning Certainty with TOLSA-M: The authors' focus on the "degree of certainty" directly corresponds to our TOLSA-M Epistemic dimension, which includes the subdimensions: speculative, probability, assertive, certain, and deductive. See How confident are you?
Tracking User Reliability: A second idea is to verify the correctness of users' predictions. This aligns with our goal of maintaining a real-time system to verify outcomes. This matters because establishing a user's track record (reliability) informs others of how much they should trust that user's future predictions.
Multilingual Entity Resolution: For our TOLSA-M tests, we could integrate langid.py into our entity resolution system to handle two cases: (1) same entity, same language, and (2) same entity, different languages. This is crucial because we want to avoid misclassifying or missing entities simply due to a language barrier.
NER and Temporality: We could use the Ritter et al. (2011) Twitter NER system to preprocess tweets and identify named entities across various topics. Unlike this paper, we would also extend our collection dates to after the event to capture past-TOLSA-Ms. See Binary Temporality (Past or Future) Detection for News Content
Flexible Classification Granularity: Just as the authors collapsed their veridicality scale from five levels to three, we can collapse and expand our TOLSA-M outcome properties (attribute, metric, and slope) and dimensions for classification tasks. This flexible granularity matters because if a system fails to confidently identify a specific subdimension, we can fall back on capturing the broader dimension without losing the data entirely.
For more information about our research, return to our homepage: ufdatastudio.com.
For more from Detravious on NLP research (including the TOLSA-M Taxonomy), research and industry collaborations, faith in Jesus, and broader writings, visit: Research Portfolio | LinkedIn | Medium, or contact via email at dj.brinkley@ufl.edu.
