Beyond Co-Generation: Rethinking Jointness in Joint Audio–Video Generation

Abstract

Our aim is to advance the understanding of genuine audio-visual coordination in joint text-to-audio-video (T2AV) generation, with particular emphasis on semantic consistency. Evaluating audio-visual consistency in T2AV generation is nontrivial, as detailed text prompts obscure whether such consistency stems from genuine cross-modal coordination or merely from independent text following. To isolate the former, we introduce intrinsic jointness: the ability of audio and video to produce the same realization for details left underspecified by the prompt. Using semantically underspecified prompts (e.g., “musical instrument playing” rather than “electric guitar playing”), we investigate four recent models and reveal their limited intrinsic jointness. To address this limitation, we propose JointREPA, a simple representation alignment-based post-training method to facilitate cross-modal coordination. Since existing benchmarks use semantically detailed prompts, we introduce UNSAIDBench, a benchmark designed to evaluate intrinsic jointness under semantic underspecification. Experiments across two distinct models and three benchmarks, including UNSAIDBench, demonstrate that JointREPA consistently improves semantic consistency.

Intrinsic jointness

The ability of audio and video to converge on the same realization for details that the prompt leaves open to multiple possibilities.

For example, an underspecified prompt such as “A man is playing a musical instrument” (Figure 1(b)), which admits multiple valid realizations (e.g., guitar or violin), requires the two modalities to actively coordinate their choice to reliably produce consistent outputs.

Figure 1
Figure 1. Intrinsic jointness. Under detailed prompts, output consistency can arise from each modality independently following the text prompt alone, without audio–visual interaction. In contrast, underspecified prompts enable disentangling whether this consistency stems from mere prompt following or genuine audio–visual coordination. We define intrinsic jointness as the ability to maintain consistency under such underspecified prompts.

Diagnosing Semantic Jointness

Q1. Do existing T2AV models exhibit semantic jointness?

Figure 2
Figure 2. Q1. Semantic jointness under varying prompt specificity. We evaluate semantic consistency using three metrics (Agreement, AV-IB, and AVHScore) across different levels of prompt specificity. All metrics generally decrease as prompt specificity decreases (L3 → L2 → L1). At L1, Agreement scores for most models are close to the shuffled-pair baseline (marked by ×), obtained by randomly pairing audio and video generated with different seeds under the same prompt. This suggests that current T2AV models exhibit limited semantic jointness.

Q2. Does limited semantic jointness persist in internal representations?

Figure 3
Figure 3. Q2. Semantic jointness in internal representations. We evaluate probe agreement across Transformer blocks (horizontal axis) at different generation steps and levels of prompt specificity. Probe agreement generally decreases as prompt specificity decreases (L3 → L2 → L1). At L1, probe agreement is close to the shuffled-pair baseline (dashed line). This suggests that limited semantic jointness is also reflected in the models' internal representations.

Q3. How effective is AV-attention for semantic coordination?

Figure 4
Figure 4. Q3. Analysis of the role of AV-attention in semantic coordination. (a) Semantic consistency with and without AV-attention. Blocking AV-attention causes substantial drops across all metrics in the LTX family, while inducing only minor changes in the JavisDiT family. (b) Semantic identity following under asymmetric prompting (L3 for one modality, L1 for the other). For both model families, the L1-conditioned modality mL1 follows the target identity specified zL3 substantially less reliably than under symmetric L3 prompting (both-L3). Together, these findings suggest that AV-attention is not effectively utilized for semantic coordination.

Method

Motivated by the findings in our analysis, we propose JointREPA, which extends REPA to T2AV generation. JointREPA aligns each modality's representation not only with target features from its own pretrained encoder, but also with those from the other modality. The former builds semantically rich representations that provide clearer signals for AV-attention to leverage, while the latter actively promotes cross-modal semantic exchange.

Figure 5
Figure 5. Overview of JointREPA. JointREPA aligns each modality's features with pretrained encoder representations from both its own modality (unimodal alignment) and the other modality (cross-modal alignment).

Main Results

Table 2. Main results on UNSAIDBench, JavisBench-mini, and VerseBench. We apply JointREPA to JavisDiT++ and LTX-2.3 and evaluate AV-Consistency, AV-Synchrony, Text-Consistency, and AV-Quality on three benchmarks. JointREPA consistently improves AV-Consistency across all benchmarks and backbone models.
AV-ConsistencyAV-SynchronyText-ConsistencyAV-Quality
ModelSizeAV-IB ↑AVHScore ↑JavisScore ↑DeSync ↓TV-IB ↑TA-IB ↑CLIP ↑CLAP ↑VQ ↑AQ ↑
UNSAIDBench
JavisDiT++2.1B0.1760.1630.1330.8150.1800.1380.3130.3431.4554.595
 + JointREPA2.1B0.2070.1960.1640.8040.1740.1630.3100.3691.6554.628
LTX-2.322B0.2440.2360.2090.4680.1730.1010.2980.2872.3545.050
 + JointREPA22B0.3020.2880.2610.4400.1690.1240.2980.3272.2564.852
JavisBench-mini
JavisDiT++2.1B0.2000.1820.1570.7840.2830.1700.3160.4291.5384.987
 + JointREPA2.1B0.2210.2040.1790.7820.2780.1870.3160.4471.7154.917
LTX-2.322B0.2490.2380.2140.5550.2800.1710.3060.4122.4585.348
 + JointREPA22B0.2850.2710.2470.5750.2770.1860.3040.4332.3395.157
VerseBench
JavisDiT++2.1B0.2130.2010.1720.9050.1820.1890.3250.4791.5804.280
 + JointREPA2.1B0.2370.2230.1950.9580.1750.2160.3230.4891.8504.260
LTX-2.322B0.3320.3170.2870.2840.1670.1910.3170.4472.7605.004
 + JointREPA22B0.3690.3510.3220.2410.1650.2110.3170.4722.6774.809

Qualitative Results

UNSAIDBench (UNderspecified Semantics in Audio-vIDeo generation) adapts the complex, real-world prompts of JavisBench to inject one underspecified shared factor per prompt. Following the AudioSet ontology, fine-grained event identities are abstracted into their parent categories (for example, “harp” becomes “musical instrument” and “cat” becomes “pet”) while the surrounding context is kept intact. Each stream can resolve the underspecified factor on its own; intrinsic jointness requires both to resolve it the same way. The underspecified shared factor is highlighted in each prompt.

Each slide shows, for the model named on top, a clip from the base model (left) and from the same model post-trained with JointREPA (right), generated from the same prompt and seed. Below each clip, Video and Audio name what the video shows and what the audio sounds like (= the two streams agree, ≠ they do not). All clips have sound. Turn on audio; starting one clip pauses the others.

LTX-2.3

Prompt “An elderly man in a striped shirt and glasses plays a musical instrument in what appears to be a shopping mall or a similar public space filled with arcade machines and electronic devices. The tiled floor and colorful arcade machines create a lively atmosphere. As he focuses on the notes, a slow-paced classical musical instrument piece in G major with soft, intricate melodies and rich chord progressions plays in the background. The lighting is bright and natural, typical of an indoor commercial area. In the distance, kids can be heard talking by the door in English.”

Base

Video Flute≠Audio Piano

+ JointREPA (Ours)

Video Flute=Audio Flute

LTX-2.3

Prompt “Two pets sit on a concrete surface, with one of them vocalizing at various intervals, ranging from loud and fierce to smaller, more subdued calls, interspersed with calls.”

Base

Video Cat≠Audio Dog

+ JointREPA (Ours)

Video Cat=Audio Cat

LTX-2.3

Prompt “A person plays a mellow musical instrument melody in a room with a checkered wall, wearing a gray t-shirt and striped pants. Multiple musical instruments are visible in the background, including a blue one. The performance is passionate and emotional, despite the noisy, low-quality mono recording. The musician's hands move up and down the notes of the musical instrument, creating a bright and intimate atmosphere, possibly in a music studio or practice room.”

Base

Video Zither≠Audio Flute

+ JointREPA (Ours)

Video Guitar=Audio Guitar

JavisDiT++

Prompt “A person sits on a couch in a living room, playing an acoustic guitar. They are wearing a white t-shirt and have blonde hair. The room feels cozy, with a plaid-patterned blanket on the couch and a poster on the wall. As they play, they speak in English about four chord songs, their tone neutral and their pace slow.”

Base

Video Female≠Audio Male

+ JointREPA (Ours)

Video Female=Audio Female

JavisDiT++

Prompt “A pet with brown markings, wearing a strap, runs quickly through a dense forest filled with tall trees and green foliage. The air is filled with the sounds of pets vocalizing and snarling. Fallen logs covered in moss lie scattered on the ground. It is daytime, and the natural light filters through the trees.”

Base

Video Dog≠Audio Cat

+ JointREPA (Ours)

Video Dog=Audio Dog

JavisDiT++

Prompt “A group of farm animals and farm animal graze on a green hillside, their woolly coats in shades of white and brown visible against the backdrop of a tree with bare branches and a fence in the distance. The farm animal call softly, adding to the serene atmosphere of the scene under a clear sky.”

Base

Video Cattle≠Audio Chicken

+ JointREPA (Ours)

Video Cattle=Audio Cattle

JavisDiT++

Prompt “A man is playing the musical instrument outdoors in front of a modern building with large glass windows reflecting the surroundings. He is wearing a white shirt and dark pants, standing on a paved area with greenery in the background. The music he is playing is a melodic instrumental piece in C major with a tempo of 125 BPM, featuring a musical instrument over chords like F major, G major, and A minor. The piece has a 4/4 time signature and a complex chord progression.”

Base

Video Guitar≠Audio Organ

+ JointREPA (Ours)

Video Guitar=Audio Guitar
1 / 7

BibTeX

@article{jung2026beyond,
  title={Beyond Co-Generation: Rethinking Jointness in Joint Audio-Video Generation},
  author={Jung, Jihoo and Kim, Ji-Hoon and Lee, Suyeon and Pham, Gia Thinh and Chung, Joon Son},
  journal={arXiv preprint arXiv:2611.22857},
  year={2026}
}