The OpenMantra Dataset for Multimodal Manga Translation

Introduction

In 2020 and 2021, The University of Tokyo's Aizawa-Yamasaki-Matsui Lab and Mantra Inc. collaborated to publish a dataset for manga translation called OpenMantra at the The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI21).

OpenMantra's first two authors, Ryota Hinami and Shonosuke Ishiwata, are co-founders of the eponymous Mantra Inc., that focuses on comic localization and automatic manga translation technology. Their website showcases their work and collaboration with big names in the Japanese manga industry including Shogakukan, Shueisha, and Kadokawa. The senior author, Yusuke Matsui, is involved extensively with Manga109, another popular manga understanding dataset that I have covered extensively.

This partnership between Mantra Inc. and the Aizawa-Yamasaki-Matsui Lab presents a dataset consisting of 214 pages of Japanese manga with bounding box annotations for text inside of speech balloons and parallel text in the Japanese (Ja), English (En) and Chinese (Zh) languages curated with the help of professional manga translators and a data curation pipeline evaluated against official translations of the manga in question.

In this post, we will discuss why a parallel text dataset for manga translation from 2021 is still relevant today and OpenMantra's and manga translation's importance to the greater landscape of comic understanding. Finally, we will touch upon what the OpenMantra dataset enables and inspires us do to at the UF Data Studio.

Why Investigate Manga Translation?

Manga translation is an important but understudied machine learning task that presents tremendous applications for the world-wide manga and comic book industry and difficult technical challenges for machine learning and comic understanding researchers. Mantra Inc. is well-positioned to serve the needs of the industry through their work and appear to already provide machine learning pipelines for manga, including multimodal manga translation, as a service to important publishers in Japan. However, before pipelines can be of use comic publishing firms, Mantra Inc. and other researchers must overcome the rigors of multimodal translation on the complex and cryptically structured medium that is manga.

While multimodal machine learning, Machine Translation (MT), and Multimodal Machine Translation (MMT) in particular are both investigated by a large body of prior literature, manga translation and multimodal manga translation receive little attention outside of laboratories interested in comic understanding. We are confident this was even more the case when OpenMantra was published at AAAI21. In their paper, Hinami et al. describe the task of multimodal manga translation and formulate a context-aware manga translation approach to meet its challenges. The authors distinguish multimodal manga translation from traditional MMT by the fact that MMT is typically formulated as the translation of a sentence with additional context supplied by a single image. This formulation is not suitable for manga translation, and they continue, conventional image encoders are not properly suited for manga data anyway because they are conditioned on "natural images." Finally, Hinami et al. identify no previous corpora for multimodal manga translation.

The authors address each of the challenges in the previous paragraph through their methodology, and in so doing, they build a considerable contribution to the literature that remains relevant years later. Because no data existed at the time for the task they wished to complete, the authors curate OpenMantra and provide it as a contribution to the literature. Showcasing the dataset and its effectiveness as evaluation data for MMT pipelines in the manga domain, the authors formulate the context-aware manga translation pipeline which performs MMT on manga data and does so image-to-image (I2I) rather than producing a transcript of translated text.

The OpenMantra dataset addresses a gap in the data mining literature for manga data whereas the translation pipeline Hinami et al. design innovates over traditional MMT approaches to circumvent the problem of context from multiple images and deliver translations in context, situated within the original manga images in place of the original Japanese text. By choosing to study multimodal manga translation, we observe that Hinami et al. fill a niche in machine translation that previously went unexplored and provide well-developed contributions to the literature.

Contributions to the Literature

In this section, we explore the contributions that Hinami et al. make to the literature through the paper introducing OpenMantra. We begin with the dataset itself, then discuss how the authors reconceptualize MMT for the manga medium, and finally provide an overview of their experiments.

OpenMantra

From Ja manga pages and En/Zh translations they commission for dataset construction, the authors curate OpenMantra's annotation through a pipeline of object detectors and OCR models. CNN + ResNet101 object detection detects bounding rectangles of text boxes within speech bubbles, and rule-based object detection picks out the text lines. OCR is performed on the text lines themselves rather than the whole page because the authors found that OCR systems did not perform well on manga data. Additionally, the OCR models for detecting text inside of text lines are trained on synthetic data that Hinami et al. generate using a text rendering engine.

The end result is a dataset consisting of 214 pages of Ja manga with 848 panels and 1593 sentences and parallel text annotations available in Ja, En, and Zh. The authors provide it as a contribution to the literature and curate an additional, withheld dataset named PubManga to evaluate OpenMantra's data curation pipeline. The latter consists of published translations of 9 Ja manga series, totaling 258 pages with Ja-En parallel text annotations.

Context-Aware Manga Translation

In addition to the OpenMantra dataset, Hinami et al. formulate context-aware manga translation for the comic understanding literature. The authors introduce it as a "manga translation approach that incorporates multimodal context."

Given image $I$, an unordered set of texts $\mathcal{T}$, and mapping $b(t)$ that produces bounding boxes for all $t \in \mathcal{T}$, the authors group texts into scenes according to maximum IoU of bounding boxes $b(t)$ and scene bounding boxes (boundaries) $s \in \mathcal{S}$, where $\mathcal{S}$ is the total set of scene boundaries. Hinami et al. call the mapping of text line $t$ to its scene $a(t)$. The authors order the texts in $\mathcal{T}$ recursively, according to the convention that manga are read panel-by-panel in rows of panels right-to-left, with rows containing columns of panels that are read top-to-bottom before they are read speech-bubble-to-speech-bubble. The method splits the pages into rows and columns to isolate panels. Ordering of texts within panels is determined by distance from the upper-right corner as a base case. Hinami utilize Illustration2Vec (Saito and Matsui, 2015) to extract semantic information from scenes $s \in \mathcal{S}$. For each scene $s$, Illustration2Vec identifies a subset $f(s) \subseteq \mathcal{L}$ of 512 predefined semantic tags $\mathcal{L}$.

From this formulation, Hinami et al. broadly define three conceptual models for multimodal manga translation that utilize or neglect different pieces of information it provides. Each translate the Ja texts of manga pages $t \in \mathcal{T}$ into texts $t'$ in the target language (En or Zh), and they are implemented and evaluated in the experiment we discuss below.

First, the 2+2 Model (Model 1), based on Tiedemann and Scherrer (2017), translates each $t_i$ into $t_i'$ with only the previous Ja text $t_{i-1}$ as context. The model is given $t_{i-1}\texttt{<SEP>}t_i$, where $\texttt{<SEP>}$ is a separator token and outputs translations $t_{i-1}'$ and $t_i'$ in the target language. This approach provides a baseline that does not account for scene-based context. Second, Scene-Based Translation (Model 2) generalizes the approach from Model 1 by providing all texts from the same scene $a(t_{i})$ as context for translating $t_i$ into $t_i'$ instead of just the previous text $t_{i-1}$. Lastly, Scene-Based Translation with Visual Features (Model 3) incorporates the visual semantic tags $f(a(t_i))$ into translation for each text $t_i$. As we progress from Model 1 to 3, we observe that models receive more context and that Model 3 is the point where visual features from manga pages are incorporated through scene-level tagging.

Experiment

Specifically for their experiment, Hinami et al. curate a third, unnamed training corpus of 842,097 Ja-En manga page pairs, amassing some 3,979,205 Ja-En parallel sentences. The authors draw on Google Translate and train two Neural MT (NMT) models on this corpus and on a baseline corpus OpenSubtitles2018 (OS18) (Lison, Tiedemann, and Kouylekov, 2018) to represent traditional sentence-level manga translation in the experiment. These traditional sentence-level approaches are baselines against which Hinami et al. compare the different context-aware manga translation models previously defined.

The authors evaluate each model automatically using Ja-En parallel annotations in the OpenMantra dataset and the BLEU metric. They also perform a human evaluation study in which five bilingual Ja/En participants are asked to rate fidelity of translations on a scale of one to five (five being best). Surprisingly, BLEU scores suggest that context-aware approaches lag behind the sentence-level NMT model trained on the authors large corpus of Ja-En parallel manga pages. These do not agree with the human evaluation, wherein Hinami et al. report that Model 2 and Model 3 score significantly better than the sentence-level approach ($p < 0.05$). The authors conclude that the BLEU metric may therefore not be well-suited to evaluating manga translation.

Novelty

Comic Understanding is a niche topic in the modern machine learning literature. To our knowledge, the dataset is the first and only dataset to offer parallel text data for translation of manga dialogue and narration plus the bounding box annotations necessary for situating the text into the original image for a complete I2I multimodal machine translation pipeline.

Whereas Manga109 and its sequel Manga109v2026 provide Japanese transcriptions and bounding box data necessary for a variety of image recognition tasks on manga data, they lack the parallel text transcriptions that OpenMantra provides us. Manga109 may have grown more famous and well-cited over the years, becoming the go-to dataset in comic understanding literature, we identify that OpenMantra is uniquely suited for manga translation and multimodal manga translation tasks in a way that Manga109 and its updates cannot presently match.

Alternative Datasets for Manga Translation

Thus, after reviewing the literature at length, we arrive at the conclusion that there are no datasets which rival OpenMantra for Manga or Comic Translation tasks, let alone for the specialized task of in-context I2I Multimodal Translation. Other datasets, including the famed Manga109 and its recent update Manga109v2026, track the position of Japanese text and transcribe it; but the others all lack parallel texts in potential target languages, which is the crucial component of training data for any supervised machine translation pipeline.

The dataset is not particularly large, a fact we will touch on in the next section, so a team could possibly construct a similar parallel text dataset. However, intellectual property concerns are a major obstacle that the OpenMantra and Manga109 projects have overcome. As a result, the authors provide the 214 pages in OpenMantra and 109 volumes in Manga109 freely and openly for academic use while a version of Manga109 exists with 87 volumes for commercial research. Additionally, the authors of OpenMantra utilize a sophisticated data curation pipeline to extract text and bounding boxes from the pages and enlist professional manga translators to create the En and Zh parallel texts in OpenMantra. While the mechanics behind creating a parallel text dataset for manga translation would not be prohibitively difficult for bilingual or polyglot researchers, the researchers in question would have difficulty curating the same quality of data and authoritative parallel texts for that data and further difficulty distributing the dataset to a broad audience of fellow researchers.

Limitations

Unlike other scientific disciplines, it is typical to consider machine learning results more than a few years old to be badly outdated. Thus, it is the fashion to not look at or cite publications more than a few years old. The OpenMantra dataset falls into this category, but it is still state-of-the-art because there is very little work on comic understanding datasets being published at any one time. The authors present a dataset of manga pages with annotations describing bounding boxes that frame dialogue and parallel text data for machine translation. We identify in the literature no datasets offering the same amenities as OpenMantra despite an extensive search. The OpenMantra dataset therefore stands alone as the one specialized dataset available for multimodal manga or comic translation.

While freely available to the comic understanding community, the OpenMantra dataset is still somewhat limited in scope and size. OpenMantra's authors advertise its inclusion of 214 pages, including 1,593 sentences and 848 panels, from a diverse range of genres including action, battle, slice of life, and more. This may seem large to start with, but as compared to other datasets in the comic understanding space, particularly Manga109, the OpenMantra dataset is lightweight. Manga109, which we have already discussed, contains 109 volumes and more than 21,000 pages of manga whereas the 214 pages in OpenMantra would typically account for just a single volume.

The translation unit for MT pipelines is typically a single sentence, so perhaps the number of pages is not the effective measure of a manga translation dataset's size. Still, the 1,593 sentences provided by the annotations as parallel text for MMT are a much smaller corpus than popular datasets for general MMT outside of the comic understanding space. For instance, Multi30k offers ~30,000 parallel sentences and images for four European languages. Moreover, context-aware manga translation models constructed for experiments in OpenMantra's paper cannot atomize translation sentence-by-sentence as in a traditional MT framework.

Whatever we might say about the dataset and its size, we now look at the experiments and find another limitation. The authors perform I2I manga translation and present several examples in which English and Chinese inference text has been deftly laid over the original speech balloons and text boxes, but they do not provide any objective automatic metrics of the I2I performance directly, only indirect metrics measuring constituent tasks including text recognition and text translation. This limitation in the existing automatic metrics is what likely motivates Hinami et al. to perform a human evaluation study.

We notice similar limitations in a variety of other generative comic understanding research. Existing metrics are able to measure dimensions of performance or performance of individual steps in the generation pipeline but fail to summarily capture a measure of the overall performance such as metrics in regular machine translation would. While creating these metrics falls well outside the purview of OpenMantra and other data mining or methods papers, it is nonetheless important to note this gap as we transition to the UF Data Studio's plans for OpenMantra.

Future Work for the UF Data Studio

I perform comic understanding research as my dissertation work, so the OpenMantra and Manga109 greatly interest me. Whereas the latter has featured prominently in my past research projects and in the comic understanding literature more broadly, I perceive that the OpenMantra has been relatively neglected by authors in the comic understanding space.

In the five years since its publication, we find that no significant work has been conducted with the OpenMantra dataset by the broader community. Instead, Mantra Inc. alone boasts about their work on manga translation and localization on their website and alone touts a publication history centering on these topics.

Because OpenMantra is available to the public as an artifact of the 2021 AAAI paper, the UF Data Studio is able to immediately take advantage of the data and begin working on automatic manga translation. However, the dataset's own authors take manga translation in the Ja-En and Ja-Zh directions sufficiently far that we do not wish to innovate directly on the method. We are instead interested in the pervasive limitations of automatic metrics in generative comic understanding research which we have already discussed in the previous section. The OpenMantra dataset may not be large enough to furnish training data for new translation pipelines but it gives plenty of samples for evaluation studies on novel metrics and evaluation pipelines and how they agree with human judgments of overall manga translation quality. Having made that observation, we plan to begin work in this direction as soon as possible.

For more information about our research, return to our homepage: ufdatastudio.com.