Academic Highlights From ACL26, San Diego

I recently travelled to The 64th Annual Meeting of the Association for Computational Linguistics (ACL26) in San Diego, CA over the week of July 4th. There, I gave a poster presentation on the rules-based recursive descent parsing research that Gabriel Ayoubi, Girish Salunke and I completed for the Mixtec Codices and participated extensively in the AmericasNLP and Comput-EL workshops which included extensive research about language documentation, machine translation and automatic speech recognition for endangered languages as well as presentations of submissions to the AmericasNLP Shared Task on Cultural Image Captioning for Indigenous Languages.

6th Workshop on NLP for Indigenous Languages of the Americas (AmericasNLP)

We begin with coverage of the 2026 AmericasNLP workshop. The UF Data Studio, the Data Science Research (DSR) Lab and University of Florida were all well represented at AmericasNLP. I was a co-author on the winning gators submission to the Shared Task on Cultural Image Captioning for Indigenous Languages, and I attended the final hour of the afternoon workshop to listen to the shared task presentations and to help present our winning submission through conversations with workshop attendees thereafter. The time for my poster presentation in the 4th Advances in Language and Vision Research Workshop (ALVR) nearly conflicted with the shared task presentations, and I joined the workshop from the poster room downstairs after presenting my work on Mixtec. With little time to spare, I joined Aashish for a quick snapshot with a certificate awarded to us for winning the competition ahead of his presentation. I would go on to present the work myself through extensive conversations with AmericasNLP, Comput-EL, general ACL attendees throughout the days of the conference.

Shared Task Papers

gators Submission: Retrieval-Augmented Long-Context Translation for Cultural Image Captioning: Gators submission for AmericasNLP 2026 shared task - Aashish Dhawan

The Data Science Research (DSR) Lab's own Aashish Dhawan presented Team gator's submission to the AmericasNLP shared task on cultural image captioning. The project was completed ahead of the workshop in collaboration with the UF Data Studio itself, and I was the second author on the paper.

The submission leverages Retrieval Augmented Generation (RAG) and a two-step pipeline to generate captions in the low-resource target language via Machine Translation (MT) of Spanish (Es) intermediate captions. This approach is espoused by the task organizers themselves and appears in the baseline model for the competition. To be thorough, we tried directly generating captions in the target language from image input and a prompt. Early tests show that the direct approach was not appropriate for the task, while the baseline's two-step pipeline, first captioning in Es and then translating Es captions into the low resource (LoRes) target language, is adopted as the baseline for the contest. Rather than altering the organizer's pipeline, we therefore innovate entirely on the MT method rather than the image captioning method.

The baseline method consists of a Qwen-2.5VL captioning model, accessed through HuggingFace API, which produces the Es captions, and Es-LoRes translation by the winning Sheffield submission from the AmericasNLP 2023 on machine translation. That method utilized an ensemble of No Language Left Behind (NLLB) models tuned for each language in the shared task, per its system description. Perhaps motivated by the zeitgeist, we quickly decided to attempt LoRes MT with Large Language Models (LLMs) rather than the specialized NLLB models. We understand that there is nowhere near enough data for training or even fine-tuning of large LLMs for the task, and the pretrained models did not show good enough performance to warrant direct captioning or MT, so we focused on few shot and many shot prompting methods as ways of delivering additional context to the models.

Naive few-shot and many-shot prompting with the development set still did not lift performance substantially higher than the baseline. It was at this time that we began to investigate RAG and utilization of data outside of the development sets that were furnished for the shared task. We retrieve data for most of the relevant languages from the previous shared tasks on MT and, for the Guaraní (Grn) language, we leverage synthetic data augmentation in the vein of our previous work in the proceedings of the LoResMT workshop at EACL26. In the EACL paper, we show that fine-tuning with a mixture of genuine and synthetic data for Es-Grn translation boosts performance of a specialized Neural MT (NMT) model.

In the present work, we apply synthetic exemplars to retrieval for the Grn language and collect thousand-scale retrieval banks $\mathcal{R}$ for each of the LoRes languages in the shared task except Yucatec Maya (Yua). Our approach consists of many-shot prompting with $r$ retrieved exemplars from the retrieval bank of the target language and $d$ dev exemplars from the development set provided for the shared task. We call the choice of $r$ and $d$ our retrieval hyperparameters for a specific captioning run and write them as a tuple $(r,d)$.

In any RAG pipeline, a systematic method is necessary for retrieved exemplar selection. For the shared task submission, we use BM25, a method similar to TF-IDF, to rank exemplars in the retrieval banks $\mathcal{R}$ based on keyword-similarity of the Es source language to the Spanish caption $q$ produced by Qwen, which we call the query in this context. The top-ranked $r$ exemplars from $\mathcal{R}$, along with $d$ exemplars in $\mathcal{D}$, are chosen as exemplars in the many-shot prompting for the MT model. To select runtime hyperparameters for our submission, we perform a grid search of a subspace of possible $(r,d)$ pairs covering $r \in \{0, 10, 20, 40, 80\}$ and $d \in \{0, 10, 20, 30, 40, 49\}$ for each language except Yua, where we fix $r = 0$ but still search for optimum $d$. We select $d = 49$ as the maximum $d$ value in our search, rather than using all $50$ dev exemplars, since at least one dev set image has to serve as the query for each inference in our experiments.

In the final submission, we only utilize pretrained Gemini 2.5 Flash for MT using the retrieval-augmented many-shot prompt, but in our paper, we report an ablation showing the effect of LLM choice and RAG configuration on the results for Grn captions. Further, we perform ablations on prompting and post-processing strategies for the Bribri (Bzd) language, taking into account its morphological complexity and text encoding differences between Gemini's output and the shared task labels. We find that normalizing the output and label encodings for the Bzd language improves performance dramatically while morphological prompting improves it further still. Finally, we perform ablations on the use of synthetic exemplars in the $\mathcal{R}$ set for Grn. We show that, controlling for the choice of runtime hyperparameters, the relative performance gain over the baseline is consistently 27-30 points greater for inference with synthetic data than without.

For more information about our submission, please feel free to read first author Aashish Dhawan's writeup on the DSR Lab's site and our system paper in the AmericasNLP Proceedings available on ACL Anthology.

Yaduha Submission: Schema-Constrained Image Captioning for Five Low-Resource Indigenous Languages - Jared Coleman

Jared Coleman is a member of the Big Pine Paiute Tribe of Owens Valley, California, USA. Their language is critically endangered and incredibly low resource. This inspires Coleman to develop LLM Assisted Rules Based MT (LLM-ARBMT), which combines favorable traits of rules-based models and LLMs, for his tribe's language but his team's submission applies LLM-ARBMT to the shared tasks languages.

The first step is to define a grammar for the target language. Coleman presents an interface for creating such rules in a python package, and coding agent is tasked with building target language sentences using the package. The agent, an LLM, never has to translate from the source language to the low resource language at inference time. This works well for the Paiute language Coleman focuses on, but his team does not speak the target languages in the shared task. For their submission, an agent created the indigenous language grammar packages for the relevant languages.

Aashish Dhawan asked Coleman about the disconnect between the ChrF++ metric and his team's final human evaluation results, referring to how the Yaduha submission won the human evaluation in two out of the five LoRes languages despite not performing nearly as well in the qualifying automatic evaluations. Although it won in human evaluation for the Bribri and Orizaba Nahuatl languages, the Yaduha submission is completely excluded from the human evaluation for Guaraní and Wixárika per the shared task procedure because it fell outside the top-5 submissions in automatic evaluations for those languages. Coleman laments the known issues with ChrF++ but acknowledges that there is not enough data for superior neural, semantic metrics to work. For more information on the Yaduha submission and this disconnect, refer to the AmericasNLP 2026 Github repo, the shared task findings, and the Yaduha system paper.

NAIST Submission: Nearest-Neighbor Retrieval for Indigenous Image Captioning

The NAIST team believed that generation in the target language would be the most difficult part of the task. So, they thought it made sense to edit captions in a high resource language instead. They prompted GPT-4 to make English captions of test images, and encoded the images and the dev set images into embedding spaces. The embedding method and corresponding space were altered as an ablation. The caption output by their submission is the one for the dev image closest to the input image in that embedding space, hence the name Nearest-Neighbor (NN) Retrieval.

Next, looking to improve on this baseline, NAIST preprocessed the dev set by hand to create an inter-lingual gloss. The complex method based on this gloss was actually worse than the naive NN Retrieval approach. NN with CLIP similarity was submitted and achieved a winning performances in automatic generation but fell relatively flat with human annotators, only winning the Wixárika language. The authors conjecture that the method's fall off with human readers stems from the fact that their model outputs a caption for another image in response to any input in the test set.

Oral Presentations

Bridging Digital Tools for Linguistic Documentation and Revitalization

Tools for Documentation and Revitalization of endangered languages have one of two sets of problems. Closed documentation tools require specialist training and raise concerns about participation and the sovereignty of the native speaker. Open tools are nowhere near as comprehensive and do not generalize often. A review of the available tools, falling into these camps, inspire four design choices.

The core idea of the final product is a unified grammar that contributes every tool. Under the hood, it provides optional pages, a SQLite database, and features that can be quickly extended by other contributors. Community control is built-in thanks to open-sourcing and modular design. Setup still needs specialist understanding, and the product does not yet have validation through user study, but mitigating measures are the team's next steps.

IndigiEval: Evaluating LLMs in North American Indigenous Languages - Julia Mainzinger

Mainzinger began with a greeting, presumably in Mvskoke, and quickly repeated it in English for the benefit of audience. She then introduced a qualitative evaluation framework for Mvskoke, Choctaw, and other Indigenous languages of North America wherein community insights were gathered to create a benchmarking corpus for each language and LLMs were evaluated against them. Then, she and her Choctaw collaborator Jacqueline Brixey, evaluated machine translation performance and vocabulary retrieval. LLMs are overwhelmingly better at Hawaiian than either language of interest.

In a generative test, the authors tasked LLMs with writing traditional stories in the languages. The models produced stories in recognizable Mvskoke and Choctaw, but it had errors from hallucinated words and phrases, and they characterize this as problematic because language learners may be fooled. Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) performance were particularly poor with $90\%$ or greater WER and TTS so garbled that the whole experiment was summarily dismissed after only one sentence was brought into speech as a pilot.

9th Workshop on the Use of Computational Methods in the Study of Endangered Languages (Comput-EL 9)

The 9th Workshop on the Use of Computational Methods in the Study of Endangered Languages (Comput-EL 9) began with remarks from University of Alberta's Antti Arppe. He summarized the history of the conference and its position in the space of NLP for low resource languages. It began in 2014 and was originally "the only game in town" for such languages, but the space has expanded to include a large number of other special interest workshops on the topics, including AmericasNLP. This has culminated in the creation of the SIGEL special interest group for Endangered Languages under the ACL. Researchers are free to join the SIGEL group through an ACL mailing list on the dedicated website.

This year, ComputEL-9 has been planned in conjunction with the AmericasNLP workshop, including a dinner on July 3 with authors and attendees from AmericasNLP. The workshop received 33 submissions with 19 accepted papers. It featured 13 oral presentations and 6 posters.

Arppe gave us quick view of the languages that are represented in submissions to the ComputEL-9 workshop at ACL26. He said that he gave in to the temptation to ask ChatGPT to summarize the languages and spent two hours working through its mistakes to build a map of the world, with continents highlighted in different colors, listing languages from each. He continued with a similar map of the world, this time with nations highlighted, showing the nationality of researchers based on their affiliations. The United States (18) and Canada (16) led the tally, and Arppe found this fitting since the workshop is taking place ACL26 in San Diego.

The opening remarks concluded with a summary of the schedule, including four oral sessions, a poster session after the lunch break, and a conclusion in time for attendees to watch the over-the-top fireworks displays planned for the evening, since the workshop took place on July 4, 2026.

Oral I

The Missing Middle: Language Documentation Needs Better Infrastructure, Not Better Models - Luke Gessler

Interestingly, the speaker introduced the talk as an introduction of Comput-EL's resume. In truth, the authors' work is a position paper including major recommendations from work in Comput-EL workshops. First, the authors would like to redesign the shared task. Model papers have, they argue, little impact.

High-Resource Language Technology (HLT) academics see a new SOTA method, cite it maybe and move on. Meanwhile, Language Documentation (LD) academics may not even be aware of such advancements and rarely have the resources necessary to utilize the new technology. The issue for shared tasks arises from the incentives of the HLT researchers, Gessler claims. HLT researchers are incentivized to publish numerous, short papers improving performance and then to move on to the next. HLT researchers compete to "solve" a task, guaranteed publication if they do, but the systems are typically abandoned thereafter and the only contact between HLT and LD researchers in these shared tasks occurs during the one or two days of the workshop.

The authors' first recommendation is to redesign shared tasks to require contact between HLT and LD researchers, particularly qualitative evaluation during the development cycle prior to submission. The second recommendation is to treat the design of LD infrastructure as research. While LD workflows are vital to the preservation of endangered languages, efforts to create these infrastructures are not treated as scholarly contributions and are therefore ignored. The authors acknowledge that this may be a radical change, but there are already movements by various venues in this direction. The third and final recommendation is to actually build the infrastructure necessary for HLT to connect with LD interfaces. In the short term, there is already compatibility between HLTs and the FLEX and ELAN file formats. In the long term, a new application is necessary, Gessler continues, because the FLEX and ELAN are not suited for indegration with ML models.

Questions following the presentation addressed logistics of these recommendations. Concerns included how and whether to train young LD researchers to be competent HLT researchers and the caveat that not every national government would recognize work on LD infrastrutures as research output.

Revitalising Endangered Languages and Cultural Heritage through Language Technology: A Pilot Study for Dzardzongke

In this work, the authors focus on Dzardzongke a Nepalese language, mostly oral, with a recently created latinized orthography that is spoken in a few villages by about 1200 people. The speaker said that studying language technologies for endangered languages is important since general purpose language technologies focus heavily on highly resourced languages. Compared to the English language, there is effectively $0.0\%$ representation of Dzardzongke on the internet, which makes learning the new orthography very difficult for members of the community outside the relevant villages. The authors built an application for enrichment in the Dzardzongke language that members of the community could use for exposure outside of the region itself. Dzardzongke content on the app is based on field work in the region, including recorded conversations and culturally relevant dialogue. Users are expected to know the language orally and to speak English rather than Nepali since this reflects conditions among Dzardzongke speakers in the field.

Likewise, there is a focus on teaching the latin orthography for Dzardzongke and improving literacy through exercises including Quizzes, Dialogues, and a fuzzy search Dictionary with audio examples to teach pronunciation. Examples focus heavily on the culture of Dzardzongke speakers in the region, which give members of the community in diaspora a window into lifestyles of the area to which Dzardzongke is endemic. Language resources are stored as JSON, audio, and images, materials are archived at ELAR, and the application is modular and offline-first with optional use of local storage because a stable internet connection cannot be assumed of users in the field.

In summary, the authors' contribution is the first mobile app for Dzardzongke acquisition and a keyboard designed for the language. It integrates cultural enrichment with language learning, and it provides a reusable framework for the design of such applications for learning the orthographies of other languages. In the future, the authors plan to evaluate the application with a user study in the field and to perform iterative redesign projects to improve the application.

Oral II

AvarLab: An Integrated Digital Ecosystem for Avar, a Morphologically Rich Low-Resource Language - Kebed Zagidov

Avar is a Dagestani language from the Avar region whose writing system uses an I-like character. Often, speakers improvise when typing in the Avar language because existing keyboards aimed at more well-represented languages do not support this character. Zagidov, a native speaker of Avar, tells the audience how Avar speakers write messages with I's or 1's in place of the unsupported character. From this unfortunate first impression, Zagidov continues to summarize the features of Avar's complex morphology, including agglutination, a subject-object-verb word order, and a class-based agreement system consisting of four different grammatical cases.

Recalling themes from Luke Gessler's position paper, Avar has some amount of data available, but there is not very much infrastructure for language documentation. The authors seek to remedy the issue with a rules-based generate-verify framework wherein documentation generated by the framework is attested by native Avar speakers. Zagidov showed the audience a demo site showcasing the architecture. Features include translation corpuses and conjugation guides for applicable cases. Users are able to attest the correctness of information.

The existing database from which the site draws contains 15,784 lemmas for a total of 1.22M total forms. The site tags data to generate NLP corpora for the Avar language, although only $7.4\%$ of the examples are attested. Though this is a small number, Zagidov assures the audience that it will grow when more Avar speakers are registered for attestation on the site.

Morphological Parsing for Media Lengua: When Accessibility Matters More Than State-of-the-Art - Jesse Stewart

Jesse Stewart presented his work virtually because he is presently doing field work, and he began with a short greeting in the subject language: Media Lengua. Media Lengua, he explained, is an endangered language spoken by people in just a few communities in Ecuador. It features a Spanish vocabulary with morphology from the Quichua language of the region. Atop endangerment, Media Lengua speakers face linguicism, discrimination based on the language they speak, and Media Lengua has not been treated as a distinct language by researchers in the past.

The authors introduce a parser for the Media Lengua language, but they faced tough challenges in the language's compounds and morphology. These are met with look-ahead and a left-to-right parsing strategy. Such a strict rule-based parsing is only possible, Stewart says, thanks to unusually strong documentation for the language.

Oral III

Choosing an ASR model for Dënë Sųłıné: Navigating polysynthesis and unstandardized orthography - Olga Kriukova

Size is not the obstacle to ASR for LoRes languages. Others include data quality topological differences across orthographies for the same language and code switching in common parlance. A key decision for researchers is what architecture to use, and the authors' work on the Dënë language Dënë Sųłıné serves as an example

Training data consist of recordings of 23 Dënë speakers and two other sources. The test data consists of 100 gender and age balanced utterances with or without English code switching. Challenges specific to Dënë Sųłıné include code switching and spelling differences arising from speaker familiarity with the orthagraphy devised for the language.

Two architectures are pitted against each other: Whisper and Wav2Vec. Medium and Large Whisper models, as well as BERT and XLS Wav2Vec, are trained. Whisper performs better on CER, but BERT achieves lowest WER. Whisper-Medium achieved a lower substitution rate, making it more appropriate for ASR transcription. In particular, Wav2Vec models had catastrophic forgetting for code-switching samples, reading out specious high resource language transcriptions from the point of code switching onward.

There are open questions, including what the effect on CER and WER would be if the models were tested on larger data sets, but Whisper is the clear winner in the present investigation thanks to lower substitution and Wav2Vec's catastrophic forgetting

Bottlenecks of In-Context Learning for Fieldwork ASR: A Case-study of Panãra

For each hour of recorded field audio, expert linguists can expect to spend 30-50 hours on the transcription, says the presenter. ML transcription exists to improve the situation. Traditional approach is fine-tuning, but ASR through In-Context Learning (ICL) also exists, and it improves performance in resource-constrained settings. The proposed system is built on the latter approach for the Panãra language, a critically endangered language from the Brazillian Amazon with a particularly rich vowel inventory and system of diacritics. The authors explain that about $17\%$ of characters carry a diacritic, and that there is ongoing field work to gather recordings in the Panãra language.

The field work furnishes data for ASR ICL and held-out test data. The first N utterances are used as context to transcribe the remaining utterances less the held out test data. This is referred to as N-shot ICL, and the authors evaluate ASR learned with 10-shot ICL. CER in the resultant transcriptions appears to correlate with genre and recording length. Also, CER improves for all recordings if diacritics are not included in the scoring, indicating a correlation between diacritics and CER, but not all diacritics contribute equally to the effect. Stripping diacritics from the context improves CER more than substitution of assistive consonants, meaning that how the tonal information is conveyed impacts what the ASR model "hears" during ICL.

Addressing Domain Mismatch in ASR for Akuzipik Language Documentation - Summer Chambers

The Akuzipik language is from the Upik family. It is spoken on St. Lawerence island in the Bering Sea and the easternmost peninsula of mainland Russia. The team is interested in ASR and found that the MMS model from Pratap et al. claimed support for the Akuzipik language. Investigation shows that the model is trained using Youtube scrapings and Bible recordings from the New Testament. Chambers explains that this poses ethical issues and creates a domain mismatch between the MMS support for Akuzipik and the use cases the authors wish to serve.

Six speakers feature in speech recordings from three different genres, and baseline CER was determined for each speaker-genre pair. The first experiment is fine-tuning for CER reduction as compared to this baseline. For read Phrases and Stories, the error rate went down, but for the Fable genre, including both the New Testament recordings and stories translated from other languages.

In a second experiment, the Spontaneous Story data from Speaker 1 was held out for validation and testing while the speaker's Phrases and Fables were utilized for training. There was improvement in this setting, too, but there are questions surrounding the effect of Speaker 1's inclusion in the train set. Atop the CER metric, the authors provide a detailed summary of character-level errors.

As for applications, the authors cite conflicting evidence for whether high WER disqualifies ASR methods from use in transcription by language documentation researchers. This is certainly a goal, but the overall rate of error remains doubtful. Anecdotal evidence, on the other hand, provides hope.

The authors close with acknowledging the ethical concerns surrounding the MMS toolkit, which they mitigated by consulting tribal councils and not publishing their data to the public out of respect for indigenous data sovereignty.

Speech Recognition and Synthesis Technologies Applied to Preservation and Revitalization of the Ainu Language - Tatsuya Kawahara

Endangered languages of the Japanese archipelago include the Okinawan languages and Ainu, Kawahara began, which is the traditional language of a people who were the primary residents of Hokkaido and whose descendants still live on the island today. However, the language is endangered in modern times. Large corpora of audio recordings of folklore in Ainu exist from government efforts to preserve the language in the 1970s.

Kawahara leads a three-day workshop with the National Ainu Museum and Foundation For Ainu Culture who have developed a web interface for viewing recordings of the Ainu language. This interface featured prominently throughout Kawahara's presentation. At the start of the authors' involvement, roughly $30$ hours out of the total $700$ hours of recordings collected had been placed on the archive because of the sheer workload of transcribing the data by hand.

The authors' team started with a review of work constructing ASR for LoRes languages. Major approaches include XLS-R and Whisper, but these do not support Ainu, and transfer learning from a related language was not possible because Ainu is an isolated language, per the project's paper. The authors found it necessary to engineer a new model, JP-90k, that exhibits comparative performance with Whisper-Large but converges for Ainu ASR with much less training data.

At the time of the presentations, the authors are applying JP-90k to ASR-assisted transcription of the remaining $670$ hours of folklore recordings provided by the Japanese government. ASR accuracy is $60-75\%$ and therefore provides a jumping-off point for transcription by non-native speakers. Phone recognition accuracy is above $90\%$ in particular.

Anecdotally, the Director of the National Ainu Museum used assistance from AI to make an opening address, and toward revitalization, stories that were never recorded have been reproduced with generative models that synthesize Ainu speech, although the latter raises concerns about personality rights and deepfakes of Ainu speakers. In the paper, however, the authors propose synthesis of Ainu speech for anime dialogue as a safer alternative application.

Oral IV

An Interactive System for Generating Revisable Grammar Lessons for Extremely Low-Resource Languages Without Expert Annotation - Sebastien Christian

Christian began with some background about the status of languages to French Polynesia, a massive French region of the South Pacific, in which there are about 1300 endemic languages, most of them endangered. His work focuses on efforts to revitalize many of these languages, which are often neglected even in the linguistics literature. As a result, although policies are place to teach many of the endemic languages in schools, teachers do not have the resources necessary to do so. Christian arrives at the following principles for a solution: Existence, "something that is not perfect is better than nothing," Christian says; Grounding, resources should not be generated in the target language nor translated from others automatically; Self-sufficiency, linguists should not have to be the ones building the grammar; Pragmatism, as outcomes matter more than rewards; and Accessibility and Governance, meaning that software should be aligned with CARE principles.

The result is a hybrid method that combines typological databases, automated observations, LLM generation (in English rather than the target language), augmentation through bayesian inference, and user query to build grammatical descriptions of the endangered Polynesian language with an interface featuring query and generation output pages. As an evaluation, Christian discusses how the interface adheres to each of the guiding principles he identifies in his introduction. Future works including aligning authoritativeness with strength of inference and field experiments in French Polynesia and Vanuatu and deployment planned for 2027.

Annotation Tools for Language Documentation: A Survey of Capabilities, Gaps, and Morphological Support - Changbing Yang

The final presentation was about a survey of current tools for morphological annotation and gaps in the literature on the topic. The challenge of documentation, Yang began, involves several steps including annotation, gloss, and transcription. ELAN and FLEX are presently the best known tools for the task, but these are of a previous era before the rise of machine learning or generative AI. Therefore, they lack native support for the new technology, and the creation of new annotation technologies is difficult and expensive.

Yang summarizes what sets documentation apart from mainstream NLP including limited annotators, non-standard orthography, complex morphology, spotty internet connection and obligations to respect data sovereignty. The landscape is growing despite these challenges but is unevaluated, and the authors ask whether existing NLP tools can be adapted to service the challenges of language documentation.

A survey of tools revealed 98 candidates but found that 32 could not be installed, and a further 22 of the remaining candidates were paid or speech oriented, leaving a total of 44 tools included in the final review. The 44 tools were evaluated according to a rubric on dimensions of Cost, Sustainability (Maintenance), Sensitivities, Linguistic Capabilities, Portability, User Friendliness, and Active Learning (AI-assisted annotation). In total, the tools are evaluated according to 32 features recorded manually from the documentation, manuals, website, and papers of the tools with manual inspection as a final resort when needed.

The survey identifies significant gaps in the literature. Few tools out of the 44 satisfy many of the dimensions of tool quality identified. While $>90\%$ of the free, accessible tools have unicode support, only 15 out of 44 support morpheme annotation. Only $7\%$ of free tools support the LD-standard IGT output, creating a design mismatch between the majority ($93\%$) of NLP tools and LD researchers. Only 6 of 15 tools with morpheme annotation support also support real-time collaboration. These gaps give rise to a small table of recommended tools the authors suggest for collaborative annotation and suggestions for future tool development with the LD researchers' needs in mind.

Closing Remarks - Sarah Moeller

Moeller thanked then gathered representatives of the linguistics community and particularly the reviewers. She said that things went smoother than previous iterations of the conference, emphasizing the role of reviewers in driving conferences forward. Then, she thanked the funders, including Google and the Social Sciences and Humanities Foundation of Canada. Finally, she thanked the Comput-EL and AmericasNLP organizers for setting up the dinner and collaborating so closely with the Comput-EL conference.

Moving forward, Moeller announced an exit survey and the fact that Comput-EL 10 will be in Manoa, Hawaii in conjunction with ICLDC-10. The tentative submission deadline is October 1 with tentative event date March 2-3. They recommended submitting both to Comput-EL 10 and ICLDC 10 to guarantee a registration spot. Moeller also reminded the audience about SIGEL, recalling Arppe's opening address, and provided a QR code to sign up.

Before revealing the final slide, Moeller wished her fellow Americans happy Independence Day and then finally showed a closing image of researchers at Comput-EL 2, which was also in Hawaii, to coincide with ACL that year.

Keynotes

June 5, 9:30AM: A New Balancing Act: Reflections on the Relationship Between Computational Linguistics and AI - Phillip Resnik, Univ. of Maryland

The history of NLP consists of the war between of statistical and rules based methods and their final synthesis. Resnik has been a party to this war and synthesis for decades as a member of both industry and academics, helping to lead one company to a nine-figure exit. Resnik is very concerned about recent developments in NLP and sees them as a potentially existential threat to its survival. Old men have complained about change, he says, since times immemorial. Socrates complained about the invention of writing, as an example, but Resnik says he is not upset about the changes to the field but acknowledging the changes in the field and warning us about potentially existential problems stemming from the success of NLP in recent years.

Resnik cites Herbert Simon's book Sciences of the Artificial. It develops the idea of science about something that is made by people, termed artificial science, as distinct from natural sciences. Engineering and studying computer models are examples of the artificial science since they pursue increases of knowledge about an artificial subject Research of the artificial is therefore set apart from product development by the search for new information about the project domain. Reviewers should therefore look at how an R&D project solves problems that matter.

Resnik moves forward, citing Donald Stoke's Pasteur's Quadrant, which introduces a two-dimensional scale of research in academic "basic" and applied directions. It is named for Louis Pasteur, inventor of Pasteurization, who did groundbreaking scientific research on sanitation and the germ theory of disease to improve public health, whose work therefore sits on the far end of both axes. Tinkering exists at the low end of both axes where neither new knowledge nor practical application are well serviced.

Resnik once was asked whether he incorporated philosophy into his work and answered that he would rather avoid doing philosophy at all, but the one who asked him said you can do philosophy with eyes open or eyes shut, meaning that it is unavoidable. For instance, Resnik continued, the bigram model on which language models are based today is actually described in terms of a state machine. The point is that the theory (philosophy) is often interchangeable but switching the theoretical framework can significantly improve research performance. More alarmingly, neglecting theoretical frameworks and moving forward with the genesis of products can be dangerous, with tragic consequences and a significant human toll.

NLP has a severe recency bias in its citations, Resnik cites, meaning that no theory persists over time, unlike other research disciplines. The ACL is the "home community" for NLP and therefore positioned to fix the problem. He says everyone is taking the same path forward, but Scientific progress requires the exploration of potential alternatives and failures along some of these paths. Then he cynically asked the crowd which ones of us were developing an alternative to transformer-based language models. One or two were, by show of hands, but the rest weren't. Resnik says that is because we do not have billions to spend and want jobs and funding.

ACL and EMNLP were once academically distinct conferences, but Resnik asserts this is no longer the case. Likewise, ACL is in danger of becoming another ML conference and a second-rate one at that. Language is treated as input and output of ML models, particularly LLMs, rather than the object of NLP research. There is some hope in the workshops, which Resnik displayed on the screen, including AmericasNLP and Comput-EL at the fore, but the CL in ACL otherwise takes a back seat to LLMs.

Bleak as this seems, Resnik apparently views the job market and industry even more bleakly. In the past, R&D labs produced scientific research like academics. Now, the labs are focused on improving performance on problems that have been defined already, with concerning parallels to the quarterly reports required of publicly traded companies. It is publish or perish; and as a result, the volume of submissions to top-tier NLP conferences balloons. Resnik blames this volume and vanishing thinness of reviewing talent for the erosion in the prestige of publishing at a conference like ACL in the first place.

There is a threat, Resnik concludes, to scientific pluralism because the value of theory has been neglected and language is not balanced as I/O and something worth studying for its own sake. The pursuit of jobs and industry agendas has completely outweighed the pursuit of knowledge.

June 6, 4:45PM: Explanation and Understanding: A Perspective from Cognitive Science - Tania Lombrozo, Princeton

Lombrozo is interested in explainability and tackles the question from a cognitive and psychological perspective as well as from conceptual principles. She proposes thinking about what an explanation does for us rather than what explanations are a priori.

She continues, saying explanation is a process and but one step of the greater process of understanding, real understanding distinct from the feeling of understanding. Colloquially, people understand explanations as an end product of the cognitive process of explanation, but Lombrozo's research has sought to isolate the process from the product of explanation to study the former. She touts her publication history on the subject but says a more succinct trade book Why We Ask Why is in production, announced for release in October.

Lombrozo outlines one of the experiments conducted to isolate the explanation process. In it, two groups are given the same information, a task and data, but only one is told to "explain" explicitly while the other is given another instruction as a control. This control instruction is varied across repetitions of the experiment. When both adults and children are asked to "explain" the data explicitly, they are more likely to find simple patterns, complex patterns, and patterns that confirm their previously held beliefs. For example, Lombrozo introduces experiments headed by Williams on the classification of drawn figures into Glorps and Drents, which are completely made-up categories. Half of participants are asked to explain what differentiates Glorps and Drents, but the control group in each experiment was asked something different each time. For this experiment and others probing other stimuli, the explainer group is far more likely to find a trait that differentiates all drawings into the two categories with $100\%$ accuracy. These experiments are repeated for other tasks with both adults and children, and the results show that asking for an explanation is not always better and may create tradeoffs. From the biases, performance boosts and tradeoffs, Lombrozo concludes that explanations function to promote learning and generalization because they provide structure and consistency.

Lombrozo says that we may and have considered explanation in isolation thus far, but it is part of the broader process of inquiry that she details with an elaborate figure projected onto the screen behind her. Explanation sits between curiosity and belief revision. Therefore, she posits, belief revision and reliance are affected by explanation as a cognitive process. As an example, experiments at Princeton show that people are more likely accept an answer to a question if there is an explanation along side it, even when the explanation is false. On the other side, explanation is downwind of curiosity, and this is confirmed by experiments showing that initial curiosity and satisfaction with explanations are driven by the perception that there is something to be learned from the explanation to begin with. This is good insight, but Lombrozo warns that there is danger in metacognition around perceptions of learning since people are not generally good at tracking what they have learned. Hence, senses of learning satisfaction are definitely not a reliable signal of understanding.

Likewise, senses of understanding are also not understanding or signals thereof, and Lombrozo reviews experiments that demonstrate that people find nearly circular scientific explanations better with jargon than without. The role of recall from the jargon is controlled by the fact the researchers use neologisms instead of real jargon in the jargon groups' explanations. Superficial analogies in explanations and simple knowledge that an explanation exists have a similar affect on perceptions of understanding, although true understanding cannot be achieved from cicular explanations or the mere suggestion that an explanation exists.

Lombrozo's conclusion is that human beings rely on external cues of their understanding rather than the internal contents of our own skulls. Without resources, people are therefore at a loss and dont know it, but we get away with this because we are not typically isolated from external sources of explanation or information.

While Lombrozo's talk had, up until that point, covered perceptions and formations of understanding, Lombrozo then introduced what understanding actually is. She finds this question particularly salient in the LLM-era because there are tons of claims being thrown about about what models do and do not "understand." Lombrozo introduces three views of machine understanding including Etiology Model, Model-based understanding, and Ability-based understanding. Lombrozo resonates with the Model-based understanding.

Characterization of mechanisms is particularly important to understanding, Lombrozo says as a number of titles of papers from ACL26 with the word "mechanism" in the title appear on screen. Experiments with data on art gallery patrons and mechanistic reliance of their donations on portrait room visits show that mechanisms support broad generalizations. Participants predict unseen questions based on the mechanisms established by the portrait gallery. Structurally, Lombrozo concluded, people want consistent, simple explanations. Phenomenologically, they want ones that satisfy them and answer curiosities.

Lombrozo turns to an example from her youngest child. She was interested in a toy atlas that colored nations and the US states according to absurd or unimportant categories such as what countries use what type of wall outlets and which US states prefer dogs more to cats and vice versa. This, and maps the reader will be more familiar with, raise the questions about we want to know about maps: Are they accurate, useful, and do they support generalization? These questions need not all be answered in complete affirmative for the maps, a stand in for explanations in general in Lombrozo's extended analogy, to serve their purpose.

Oral Presentations

OlymMATH: Olympiad level benchmark for Math Reasoning: An Olympiad-Level Math Benchmark for Large Language Models - Haoxiang Sun

As background, the speaker introduces the basic concept of math benchmarks. They ask models math questions and evaluate the answers. Issues of interest include data leakage and rigor in evaluation as well as saturation. In particular, Sun tackles the latter problem of saturation, wherein all SOTA models achieve nearly perfect marks on the benchmark. It effectively ceases to be a challenge for them.

To address the community's MathQA benchmark concerns, the authors curate questions into EASY and HARD datasets of 100 samples each. Additionally, the authors propose 100 Lean code proof questions, which are verifiable automatically thanks to the machinery of the Lean programming language. Nearly all frontier or specialized models achieve high performance on existing benchmarks. The new benchmarks pose a challenge as compared to existing benchmarks but results show the expected correlation between performances in older benches and the new ones.

To mitigate reliability issues with other benchmarks, the authors propose multilingual evaluation and new methods of posing questions. The authors expected and observe that performance in the non-English, Chinese version of the corpus lags behind the English version. To tackle rigor, the Lean code provides automatic theorem verification.

Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision - Bingsen Chen

Chen introduces Deep Research Agents (DRAs) as LLMs armed with tools for web browsing and search, but these are often evaluated in a method that does not reflect the cycles of revision that a person goes through. This inspires a new benchmarking experiment that determines how well the agents can revise their research.

The benchmark that stems from this research is called MR DRE Reports are evaluated in a unified manner and put through a feedback simulation pipeline that simulates both detailed feedback and lazy "fix this" feedback. SOTA DRAs are evaluated using popular benchmarks and metrics are varied.

MR DRE's short answer to the motivating question is that models cannot self-evaluate. DRAs struggle in this setting to improve or even maintain report comprehensiveness, failing to preserve content outside of the user's feedback. Citation faithfulness degrades in particular. Over many turns of revision, the DRAs lag far behind optimal performance with no sign of catching up.

The authors conclude that DRAs have a long way to improve in their revision ability and that such improvement is important since their MR DRE benchmark reflects a use case common among users. They also release MR DRE to the community as a contribution to the literature.

Query-Efficient Private RAG - Shuhua Yang

GraphRAG systems are distinct from Standard RAGs in the way that they are targeted by attacks. Sensitive text may be leaked by a normal RAG directly but the attack target for attacks on a graph-based RAG is the structure of the graph itself which, as Yang explains, contains sensitive associations and is more threatening. The authors propose AGEA, a complex network of components, featuring language models and databases connected by classifiers, that explores the graph until novelty drops below a threshold, $0.2$ in the presented experiments, and then enters an exploitation mode until another threshold is reached.

The authors test AGEA against various victim GraphRAGs and datasets and evaluate what percentage of the graph is extracted. AGEA outperforms all previous approaches and improves on edge recovery in particular. Ablations show that the combination of exploration and exploitation outperforms either mode separately and that a filtering step included in the network is a necessary addition because precision collapses without it. Yang lists traversal-aware monitoring, response sanitization, and Graph-aware output control as future works. Overall, defense mechanisms are the next stage of her research.

When asked about defending against graph leakage after the presentation, which Yang and her team show happens under normal use, she referred to access control and sanitization as possible mitigating strategies. In response to questions about whether releasing her code at all was a risk to the safety of data bound in existing GraphRAGs, Yang assured her audience that enterprise GraphRAG systems can defend against the attack and that her work is about probing security issues inherent to GraphRAG as a RAG architecture.

Special Thanks

Finally and most importantly, special thanks to the Association for Computing Linguistics for organizing the conference, to the city of San Diego for hosting the conference-goers through July 4th weekend, and finally to the USS Midway Museum and the San Diego Grand Hyatt resort for hosting the events of the conference.

---

For more information about our research, return to our homepage: ufdatastudio.com.