Learn + Act and Apply + Prepare
Paper + Venue + Topic
This paper Distinguishing Past, On-going, and Future Events: The EventStatus Corpus by Ruihong Huang, Ignacio Cases, Dan Jurafsky, Cleo Condoravdi, and Ellen Riloff was published in the Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). This research explores temporal analysis of multilingual civil unrests corpora (English and Spanish) for event extraction and classification of the temporal status. Additionally, this work explores the semantic compositionality, verb lexicons, support vector machines, and convolutional neural networks.
The Problem + Why it Matters
The problem with traditional event extraction is its reliance on past events in the news. This is not very useful in real-time systems, because on-going and future events are often left out. The importance of this work lies in its inclusion of on-going and future events while still keeping past events.
The authors state that a past event has already happened, an on-going event is currently happening, and a future event may happen later. I extrapolate that these three temporal categories matter for the following reasons:
- People can learn from the past. Take, for example, the murders of Emmett Till, George Floyd, Breyanna Taylor and countless other Black people. In light of those past events, people should be better prepared to understand and respond to the death of a more recent Nolan Wells, as well as the broader experiences of the Black community. Similarly, this applies to many other events, such as COVID-19 and military wars.
- People can act and apply in the present. Take, for example, knowing real-time traffic conditions on a major highway. This can help a driver decide whether to take an alternate route, wait, or leave at a different time. Similarly, this applies to tracking a live sports game's current score and knowing whether a delayed flight has finally departed.
- People can prepare for what may come about. Take, for example, Jesus' return one day. This should prompt people to follow Him. Similarly, this applies to gains/losses of financial investments to avoid the 2008–2009 market crash and forecasts of when a hurricane is expected to make landfall.
Moving from event extraction to event-status classification, the authors note that “the linguistic contexts of on-going and future events involve complex compositionality, and features like explicit time expressions are less useful.” This matters because a system that classifies these societal events should minimize both false positives and false negatives. False positives exist if the system predicts that an event is on-going or future when it is actually past, which can function as a false alarm and misinform people. On the other hand, false negatives exist if the system predicts that an event is past when it is actually on-going or future and it fails to surface important information that people may need in order to act or prepare. In other words, this is not just a technical issue; it has practical consequences for end users.
To illustrate this challenge, the authors provide examples of a state-of-the-art TempEval system misclassifying the temporal category of events:
To reiterate, this work matters because traditional event extraction has focused primarily on past events, which means current systems often miss or mishandle on-going and future ones. By addressing all three temporal categories together, this paper moves toward a more complete event analysis pipeline rather than a past-only solution.
The Method
The authors methods can be split into two distinct phases:
- Data collection and annotation
- Modeling
Data Collection and Annotation
In the data collection and annotation phase, they identify words, randomly select news stories, and then conduct a two-stage annotation process.
“The English keywords are ‘protest’, ‘strike’, ‘march’, ‘rally’, ‘riot’ and ‘occupy’. These correspond to the most frequent words in the relevant frame in the Media Frames corpus (Card et al., 2015). Because ‘march’ most commonly refers to the month, we removed the word itself and only kept its other morphological variations.”
“Spanish keywords: ‘marchar’, ‘protestar’, ‘amotinar(se)’, ‘manifestar(se)’, ‘huelga’, ‘manifestacion’, ‘distur-bio’, ‘motín’, ‘ocupar la calle’, ‘tomar la calle’, ‘salir * las calles’, ‘lanzarse a las calles’, ‘cacerolas vacías’, ‘cacerolazo’, ‘cacerolada’. Asterisks could be replaced by up to 4 words. The last three terms are common expressions for protest marches in many countries of Latin America and Spain.”
“46 (out of 3000) and 9 (out of 1500) stories were removed due to keyword errors.”
The news stories are selected at random from English Gigaword 5th edition and Spanish Gigaword 3rd edition corpora. The first stage in the annotation process is using three judges to annotate sentences with one key phrase and a specific civil unrest event.
In the second annotation process, the annotators assigned a label of:
The final data distributions:
Modeling
The authors utilized two classifiers—support vector machine (SVM) and convolutional neural network (CNN)—across three classes using the one-versus-rest method:
| Classifier # | One (Target Class) | Rest (Remaining Classes) |
|---|---|---|
| 1 | PAST | ON-GOING, FUTURE |
| 2 | ON-GOING | PAST, FUTURE |
| 3 | FUTURE | PAST, ON-GOING |
With the SVM, they abolate on features such as: bag of words, tense/aspect/time (employing TIPSem "to generate the tense and aspect of events and find time expressions in both languauges") governing words, the future-oriented lexicon, and all of them combined.
With the CNN, they used word2vec embeddings for English. For Spanish, they trained their own Spanish embeddings by running word2vec over Spanish Gigaword and tokenizing it using Stanford CoreNLP and the pre-tokenized Spanish Wikipedia dump for Spanish.
What is Genuinely Novel
The creation of this dataset is itself novel, as it distinguishes among PAST, ON-GOING, and FUTURE events, with future further divided into PLANNED, ALERT, and POSSIBLE. This differs from much prior work, which has focused primarily on past events. By analyzing all three temporal categories, the paper moves toward a more holistic system.
The feature ablation for the CNN is also valuable. It helps show the contribution of each feature and provides guidance for future research on what types of features may be worth ablating or refining.
Finally, the authors’ decision to create their own Spanish embeddings is personally interesting. Embeddings often serve as the foundation for many NLP tasks, so if that foundation is weak, the downstream tasks built on top of it may also suffer.
Limitations
In their paper, the authors state that they "applied several rules to create additional tense and aspect features based on the governing words of event mentions." By "additional," they mean supplementary feature values derived from syntactic context, not new training examples or additional tense categories. For instance, if an event like "strike" is governed by a word like "threatened to," the rule-based features would capture that relationship to help classify the event as future-oriented. However, a key question is whether these rules are generalizable beyond the training data. If the rules are too narrow or too tied to specific linguistic patterns in the corpus, they may fail to classify events correctly in new contexts. This matters in a real-time system deployed for news articles, where a user might select to be notified of future events—such as the proximity or expected landfall of an incoming hurricane. If the rule-based features are not generalizable, the system could miss classifying the hurricane warning as a future event, and the end-user would not receive the alert they need to prepare or evacuate.
The paper states that governing words (e.g., “threatened,” “entered,” or “called for”) “have been useful in prior work,” but this claim appears to be under-cited. I only see one citation associated with this discussion, which makes it difficult to determine the broader empirical basis for including governing words as a feature. Nevertheless, their results do suggest that governing words are useful, as the SVM shows improved recall, precision, and F-score when this feature is added relative to some of the other feature settings.
Their CNN description is mostly digestible, but a visual representation would help the reader better understand the flow. This is more of a nice-to-have limitation, since the paper remains understandable without it, and I am unsure why the authors chose not to include one.
The authors decompose future events into Future Planned, Future Alert, and Future Possible, but they do not report separate experiments for these subcategories. This is an important limitation, because a system that can classify future events at this finer level of granularity would be more useful in practice: each subtype conveys a different degree of certainty or urgency, and therefore informs people differently about how they may need to prepare or respond.
Ideas for Our Work
Temporal Alignment: We can employ TIPSem “to generate the tense and aspect of events and find time expressions” across our TOLSA-M datasets, including synthetic, financial phrasebank, chronicle2050, TimeBank, YouTube, News API, multimodal forecasting for climate, Clients, Rivals and Rogues, and Forecast Bench. This aligns naturally with our TOLSA-M Tense dimension, since we explicitly care about past, present, and future rather than only future-oriented language. It also aligns with our Date property, because EventStatus’s document creation time maps to our declaration date, while linked time expressions map to within-document dates. This matters for tracking narrative change over time and for deciding when to examine the veracity of a TOLSA-M.
One-vs-Rest for TOLSA-M Structure: Inspired by EventStatus’s one-vs-rest setup, we can experiment with one-vs-rest classification for core TOLSA-M properties such as source, target, date, and outcome. The challenge here is that these properties may not always be easy to distinguish linguistically—for example, entities (source and target) may appear as pronouns, and some documents may blur the distinction between target and outcome. Still, this setup could help us move from broad binary detection (TOLSA-M/Non-TOLSA-M) toward more fine-grained extraction of TOLSA-M structure.
One-vs-Rest for TOLSA-M Taxonomy: We can also extend the one-vs-rest idea to selected TOLSA-M taxonomy dimensions, especially Certainty Document, and Reaction. In this framing, EventStatus’s future subcategories—Planned, Alert, and Possible—map most naturally onto our Certainty dimension, since they reflect different levels of commitment or urgency rather than different tenses. This could help us build more fine-grained classifiers that go beyond simply identifying whether a TOLSA-M exists, and instead classify what kind of TOLSA-M it is within our taxonomy. Another more fine-grained classifier is with the Document where subdimensions are span/phrase, sentences, and paragraphs. This matters as a TOLSA-M can have a predictive phrase, while entire sentence reads as a fact. We'd still want to classify as TOLSA-M. ideas section for more details. The last fine-grained is dimension is Reaction (after outcome is known) where subdimensions are motivational, disappointed, behavioral, and skeptical. An example of this is when Stephen A. Smith listened to the weather forecasts and overdressed, hence behavior.
For more information about our research, return to our homepage: ufdatastudio.com.
For more from Detravious on NLP research (including the TOLSA-M Taxonomy), research and industry collaborations, faith in Jesus, and broader writings, visit: Research Portfolio | LinkedIn | Medium, or contact via email at dj.brinkley@ufl.edu.
