DreamingComics Comic Generation Method from CVPR26
DreamingComics (Patrick Kwon & Chen Chen, University of Central Florida)
DreamingComics was presented on Sunday June 7 as a poster at the 2026 Conference on Computer Vision and Pattern Recognition (CVPR26). While I attended CVPR26 in-person and even had a chance to connect with Dr. Chen Chen over lunch the first day of the conference, my return flight from Denver left too early on the 7th to see the poster or speak with Kwon at length about his work. Instead, I first read the work during the flight and return to it this week to give a full write-up on my thoughts. The paper is available online via the CVF Open Access Database, and the authors have developed a paper site.
Because this article relates to my trip to Denver for CVPR, we could simply edit the original post to include it. However, we instead place this article in a new blogpost to draw better attention to Kwon and Chen's groundbreaking research on comic generation.
With permission from DreamingComics authors Patrick Kwon and Chen Chen, we include a copy of the poster presented at CVPR26 for the benefit of our readers. Throughout our discussion of the authors' DreamingComics method, it will illuminate the details of their work and illustrate improvements over related methods.
Introduction
The problem of comic generation is central to the generative wing of comic understanding, and many of the publications that study generative comic understanding focus on this one task in particular. Since comic understanding is my dissertation topic, I immediately latch onto any new research that advances comic generation.
The goal of comic generation techniques is to generate new comic book data through a machine learning method conditioned on existing input data. While comic books are consistently represented as images, the modality of input data and the feature engineering applied to it is not fixed and varies from method to method. Approaches for generating comics fall broadly into Image-to-Image (I2I) and Text-to-Image (T2I) categories or Composed Image Generation (CIG) in which the input is itself multimodal, typically consisting of text prompts and images that provide the method with additional context.
DreamingComics: A Composed Image Generation Method
DreamingComics is one such CIG comic generation method. Kwon and Chen make the composition of character reference images and textual prompts clear through their diagrams of the method architecture. Input consists of a text prompt, called the "Input Text," that contains a rough transcript of the desired narrative and "Input Source Images" that provide references for character design. The method is a complex pipeline of modules, comprising a video diffusion-transformer (DiT) illustrator; a novel RegionalRoPE region-aware positional encoding scheme; and layout generation via a fine-tuned language model. The method is evaluated with ViStoryBench and DreamBench++, two benchmarks for narrative scene generation.
As we include in the introduction to this post, comic generation is an important and deeply studied task in comic understanding literature. While this situation provides us with many methods and analysis for this task, it implies that there are already numerous approaches to comic generation Kwon and Chen must outperform and from which Kwon and Chen must set themselves apart in order to produce relevant, genuinely novel work on the subject. The concept of CIG should itself show that the composition of image and text features in comic generation is not a novel idea, but Kwon and Chen innovate in their method's adherence to the character models provided in reference images. Their paper and the corresponding website provide numerous examples.
While other comic generation methods are capable of generating comics that stick roughly to the story transcripts provided by Kwon and Chen, the characters are often so off-model that they are not recognizable as the same characters in Kwon and Chen's reference images. DreamingComics deviates from the character models in the reference image much less severely, although the comparison the authors provide is subjective. Existing objective metrics from ViStoryBench capture it in pieces, and it is difficult to characterize succinctly in text, but the difference in performance and alignment with the provided context is clear at a glance. Readers can see this difference for themselves in the poster above. The figures inset on the bottom-center and right side of the photo provide a direct comparison of DreamingComics' generations as compared to other methods controlling for the same prompt, generated layout, and reference images where applicable.
Kwon and Chen attribute improvements in on-model generation to the authors' new method of integrating the reference images into the generated comic via the video DiT illustrator. The authors explain that the decision to use a video DiT was motivated by their spatiotemporal priors, which condition them to maintain visual consistency in generated images. Setting themselves apart from previous works that generate images from video models, Kwon and Chen boast of relatively low latency thanks to DreamingComics' generation of single frames from the reference images whereas previous methods must generate multiple frames per output image.
Addressing layout fidelity, RegionalRoPE, a newly modified 3D Rotary Position Embedding (RoPE), and the novel Masked Condition Loss condition DreamingComics to produce character illustrations bound to the appropriate regions determined by the method's layout generator model. RegionalRoPE maps the RoPE indices of generated character illustrations to bounding boxes in the generated comic layout, positioning and scaling them to the correct locations in the generated layout.
While it improves layout fidelity without additional training, Kwon and Chen do not find RegionalRoPE adequate on its own and introduce Masked Condition Loss. The novel loss regularizes the diffusion objective with a mask loss term based on the average of ReLU-saturated differences between time-averaged Cross-Attention Maps (CAMs) collected at each timestep of diffusion and a binary mask MASK. The authors explain that the masked loss term penalizes attention to regions outside of the bounding boxes thanks to its utilization of ReLU-constrained differences. The mask loss is parameterized by hyperparameter , and the final Masked Condition Loss is the sum .
Limitations
DreamingComics does not generate speech bubbles or include dialogue in the input text. This is a significant and disappointing limitation because speech bubbles and thought bubbles are central fixtures of comics that convey characters' dialog or inner monologue to the reader. While it is certainly possible to draw and roughly understand comic book narratives without them, users of comic generation methods would expect speech bubbles to appear in the finished product. This limitation is common to many comic generation techniques and methods of narrative scene generation more broadly. Thus, it does not set DreamingComics back against the field, especially considering the novelty of the method and its improvement over existing baselines exists in its ability to remain on-model when illustrating characters based on reference imagery.
Future Work for the UF Data Studio
Although the subjective performance of DreamingComics is very impressive, comic generation methods that include speech bubbles from the input text are a necessary step forward for the comic generation literature. We are inspired to research in this direction and recommend that anyone working in comic generation consider supporting speech bubbles and other types of text natively.
Secondarily, the objective metrics of comic generation performance leave something to be desired. Most capture aspects of the generation quality without a clear summative measure of overall generation quality. Two metrics utilized in the work come from the ViStoryBench benchmark, including Inception Score and Aesthetic Score, but these are not necessarily specialized for comic generation. Further, DreamingComics clearly outperforms the baseline methods on a subjective basis, but Kwon and Chen write in the paper that the proposed method's Inception and Aesthetic Scores "are comparable to the copy-paste baseline," saying it stems from their "deliberate preservation of stylized, non-photorealistic appearances." It is problematic if summary metrics penalize methods for retaining the style of reference images and departing from the photorealism common in narrative scene generation, as these statements appear to confirm. In this case, the metrics fail to capture a clear positive change from static photorealism and inconsistent character design to persistent generation of comics with art styles and characters successfully conditioned by input. We are interested in creating new, specialized summary metrics for comic generation that respect stylistic and character model consistency as dimensions of fidelity in the comic understanding task.
For more information about our research, return to our homepage: ufdatastudio.com.
