How confident are you?

Paper + Venue + Topic

This paper ''You should probably read this'': Hedge Detection in Text by Denys Katerenchuk and Rivka Levitan was published on arXiv in 2024. This research explores hedge detection, uncertainty detection, natural language processing, part-of-speech tagging, neural networks with attention mechanisms, and transformer-based language models using the CoNLL-2010 shared task dataset.

The Problem + Why it Matters

This paper, at large, addresses the confidence of humans when expressing ideas, beliefs, and statements across various domains such as medicine, finance, and engineering. More specifically, they examine the textual Wikipedia dataset from the CoNLL-2010 shared task, which is:

dedicated to the detection of uncertainty cues and their linguistic scope in natural language texts. The motivation on behind this task was that distinguishing factual and uncertain information in texts is of essential importance in information extraction.

To illustrate the stakes, the authors open with a striking example: a doctor telling a patient, "I think you need surgery immediately!" They further state that the statement contains a level of uncertainty, which is introduced by the phrase "I think." This need for certainty extends across critical domains and the lack leads to real-world consequences. In medicine, as the authors state in this paper/example, doctors need to be certain to prevent cases of malpractice and save lives, and this certainty is also essential for preserving the trust of their patients. Similarly, financial advisors cannot afford the loss of clients that results from uncertainty about market volatility, and engineers risk losing contracts if they cannot guarantee that their materials will remain sustainable through inclement weather. The degree of confidence matters because others use it to assess how to incorporate information into their decision-making.

The authors state that hedges indicate the degree of belief.

Hedge phrases can be expressed through modal verbs (”could”, ”might”, etc.), peacock expressions (”very likely”, ”everyone”, ”I think”, etc.), and weasel words (”some believe”, ”clearly”, etc.).

These expressions are context-dependent and the presence of these in a sentence doesn’t indicate uncertainty.

In this work, they detect sentences with hedges. Despite the initial method of using POS tags, more recent works have moved away from this approach. The initial reasoning was the rise of neural networks (NN). The popular and top-performing models for this task were convolutional neural networks (CNN) and CNNs with an attention layer. From this, other works gravitated toward the transformer architecture and transformer-based language models, such as fine-tuned BERT. This matters as this work is motivated by a combination of all three: POS tags, neural networks (including attention), and transformers.

The Method

Sentence alone The authors use CNN, GRU, and LSTM architectures alone, a transformer alone, and a CNN, GRU, and LSTM with attention for each. Since these models can't take in text directly, they must take in embedding vectors. They evaluate these models using the top two word embedding models with respect to F1 score: fastText 2M and GloVe 6B. (Side note: They perform a survey on embedding models. See image below). In addition, they also created a custom-trained wiki1G model. They state that this model is smaller than the larger models or smaller than the other models.

Sentence + pos The authors state that POS tags have not been utilized in recent works, but for the earlier works on hedge detection, the models utilized both words and POS tags. Here they state that they will use POS tags (alone) to improve the uncertainty detection of sentences. They decide to use an LSTM with attention and a GRU with attention. One result is that "this result shows that POS tags contain a predictive signal that can be leveraged to improve performance."

Words + pos With the success of POS tags containing a predictive signal, the authors then perform another task to leverage the POS information. They combined POS with the word embedding models for hedge prediction. They have two different methods for combining them in this task:

1. An early fusion-like technique based on concatenating the POS tag representations to the word embedding at the word level.

The benefit here is that the features learn together from the very beginning, maximizing their combined strengths to produce a single answer.

2. Another approach is more similar to a late fusion-like technique, where two separate networks—one for POS tags and the other for word embeddings—are used, and then joined in the last hidden layer.

The benefit here is that the models learn separately, maximizing their unique differences, before joining at the end to finalize an answer.

A helpful analogy for this is two individuals working on a presentation deck. Early fusion is like the two individuals discussing the project together from the very beginning and collaborating throughout the entire process, leveraging each other's strengths to build one final deck together. Late fusion is like each person independently creating their own separate deck based on their unique perspectives, and then coming together at the very end to merge the best parts of both into a final presentation.

What is Genuinely Novel

The novelty in this work is the combination of traditional linguistics of POS features with modeling choices to capture NNs (including attention) and transformer, including early and late fusion techniques. Comparing these across embedding models is a great approach as well to see which model is best representative of the data.

Limitations

While the authors stated that Wikipedia is "more general and not limited to any domain," a more thorough and rigorous analysis would have required evaluation on BioScope (the medical domain dataset from CoNLL-2010) alongside additional out-of-domain datasets to empirically validate their generalization claims. Alternatively, following the same systematic evaluation approach they applied to embedding models and classification architectures, they could have conducted a comprehensive cross-domain analysis consisting of three experimental configurations: (1) train in-domain on Wikipedia and evaluate out-of-domain on BioScope, (2) train in-domain on BioScope and evaluate out-of-domain on Wikipedia, and (3) train on the combined Wikipedia and BioScope datasets and evaluate in-domain on both. This survey study would have provided empirical evidence to substantiate their justification for using Wikipedia alone, rather than relying solely on the assumption of its domain-agnostic nature.

Ideas for Our Work

The first idea for our work confirms the TOLSA-M dimension called Epistemic, where sub-dimensions are speculative, probabilistic, assertive, and certain. The certain sub-dimension capures hedge phrases broadly and more specifically... . Furthermore, not only knowing which TOLSA-M exists, but also specifically identifying the hedge phrase would be beneficial.

They don't directly connect hedges to predictive statements or the future as I hypothesized before reading. Knowing the relationship between hedge words and TOLSA-Ms would provide a deeper understanding of whether all hedge words are TOLSA-Ms and whether a TOLSA-M must contain a hedge word or not. One can argue that hedge phrases broadly and more specificially modal verbs are future-tense indicators.

Our current experiments utlize ML (SVM, GB, etc), DL NN (RNN, LSTM, GRU), Transformer (BERT), and LLM Generation (llama, mistral, gemma, gpt) with taking into account linguistic markers like part-of-speech tags. Thus, we are inspired on this along with utilizing a combination of NN + attention, early and late fusion techniques because of the added benefit of "a new top score on the CoNLL-2010 Wikipedia dataset."

Insipred us to do a survey of embedding models: spaCy (small, medium, large, transformer), SentenceTransformer (all-mpnet-base-v2, all-distilroberta-v1, all-MiniLM-L12-v2, all-MiniLM-L6-v2) to see which model best represents our datasets.

Ideas for Our Work

The first idea for our work confirms the TOLSA-M dimension called Epistemic, where sub-dimensions are speculative, probabilistic, assertive, and certain. The speculative and probabilistic sub-dimensions capture hedge phrases broadly, and more specifically modal verbs like 'could,' 'might,' and 'should.' These linguistic markers signal a lack of certainty in the TOLSA-M. The certain sub-dimension should capture statements with no hedging—unqualified, confident assertions. Furthermore, not only knowing which TOLSA-M exists, but also specifically identifying the hedge phrase within it would be beneficial for downstream tasks.

They don't directly connect hedges to predictive statements or the future as I hypothesized before reading. Knowing the relationship between hedge words and TOLSA-Ms would provide a deeper understanding of whether all hedge words signal TOLSA-Ms and whether a TOLSA-M must contain a hedge word or not. One can argue that hedge phrases broadly—and more specifically modal verbs—are future-tense indicators, which ties back to the challenge of tense in identifying TOLSA-Ms. Additionally, understanding this relationship can help us distinguish TOLSA-Ms from related discourse genres like beliefs, claims, and events, reducing both false positives (inflating our dataset) and false negatives (missing valid TOLSA-Ms).

Our current experiments utilize ML (SVM, GB, etc.), NN (RNN, LSTM, GRU), Transformer encoder-only (BERT, Longformer, etc.) and Transformer decoder-only (Llama, Mistral, Gemma, GPT), but have not yet incorporated linguistic markers like part-of-speech tags. This work inspires us to explore a combination of NN + attention mechanisms along with early and late fusion techniques, given the added benefit of achieving "a new top score on the CoNLL-2010 Wikipedia dataset."

Finally, this work inspired us to conduct a survey of embedding models—spaCy (small, medium, large, transformer) and SentenceTransformer (all-mpnet-base-v2, all-distilroberta-v1, all-MiniLM-L12-v2, all-MiniLM-L6-v2)—to see which model best represents our datasets and improves TOLSA-M identification. Of course we'll couple these with the model architectures (ML, NN, Transformers).

For more information about our research, return to our homepage: ufdatastudio.com.

For more from Detravious on NLP research (including the TOLSA-M Taxonomy), research and industry collaborations, faith in Jesus, and broader writings, visit: Research Portfolio | LinkedIn | Medium, or contact via email at dj.brinkley@ufl.edu.