CHAT — Conversational Human Audio-visual Talking Dialogue Generation

European Conference on Computer Vision (ECCV) 2026

A whole conversation.
From one line of text.

Conversational Human Audio-visual Talking Dialogue Generation

The repository opens after the conference.

CHAT compared with condition-driven face video generation, face video editing, facial reaction generation and interactive talking-face pair generation.

Other methods animate one face from a condition, edit a clip you already have, or react to a partner you supply. CHAT writes the conversation, speaks it, and performs both sides.

Two people. Talking, listening, answering each other. No script. No reference video. Nothing but a prompt.

Generated dialogues

Watch them take turns.

Five prompts, five conversations, ten identities that were never paired by hand. Both faces in a clip are generated as a responsive pair, so the one listening reacts to the one speaking, inside the same clip.

Who is speaking, on the one shared timeline

No unit list was kept for this clip, so there is no timeline to draw.

Five turns in ten seconds. Each speaker talks twice and drops a back-channel into the other's turn, all on one shared timeline.

Both sides interrupt politely: an "uh-huh" at two seconds, a "yeah" at eight, each landing inside the other speaker's sentence.

A disagreement about which database to use. The reply cuts in as soon as the first speaker stops, with a back-channel halfway through.

Two male voices this time. Voices are sampled independently of faces, so a same-gender pair is as likely a draw as a mixed one.

Identities need not be photorealistic. The same method drives illustrated faces with no change. This clip comes from the conference video, so it is lower resolution than the others.

Each clip is the full CHAT output for one prompt, at 512 px per face, then passed through the CodeFormer face upscaler that Hallo2 ships, so it holds up on a large screen. Every number on this page is measured before that step. The two speakers' 24 kHz tracks are brought to the same loudness before they are mixed, so neither voice sits under the other, and every clip is normalised to one level. Each timeline is drawn from that clip's own generated unit list, except the illustrated one, whose list was not kept and whose turns are measured back from the clip by pitch. Identity descriptors carry no gender, so a face and a voice are drawn independently and need not agree; these five were picked from runs where they do.

How it works

Three stages.
One conversation.

01
It writes the dialogue

A language model turns one scenario prompt into multi-turn dyadic scripts, each with a described identity for both speakers.
02
It speaks both parts

Back-channels, per-turn emotion, timing and a shared acoustic environment are added, then synthesised as two voices on one timeline.
03
It performs the faces

Talking segments are rendered, listening segments generated, each face made responsive to its partner, and every seam blended away.

The CHAT pipeline: textual dialogue generation, dyadic audio dialogue generation, and interactive facial behaviour generation.

The full pipeline. TDG writes the dialogue pairs and identity descriptors, DADG turns each dialogue into interactive emotion-aware audio, and IFBG synthesises the responsive face-video pair.

The responsive facial behaviour generation sub-block: identity and partner behaviour are downsampled to several scales, fused by cross-attention, and injected into the diffusion transformer across denoising steps.

Inside the responsive block. Identity and the partner's behaviour are fused at several spatial scales, then fed into the diffusion transformer coarse first and fine last, so early steps settle the shape of a movement and later steps bring in the detail.

Inside

Three mechanisms, drawn.

Each diagram below is the actual rule the paper specifies, animated as you scroll rather than described in prose.

One timeline, two speakers

Sentences and back-channels are laid on a single shared timeline. While one speaker holds the turn the other is silent, apart from the short interactive words that keep the exchange alive.

W_inter = { uh-huh, hmm, okay, yeah }

Coarse first, fine last

The partner's behaviour is fused with the identity at several spatial scales. Early denoising steps receive only the coarsest scale, which fixes the shape of a movement. Finer scales join as the steps go on, bringing in detail.

l(t) = max(1, ⌈L·t / T⌉) · Φ(t) = ⋃_l=l(t)^L C(l)

No visible seam

Segments are refined separately, so their join could show. A half-Gaussian cross-fade over a narrow window blends the two sides, weighted equally at the boundary itself and decaying outwards.

α(τ) = ½ exp( −(τ−1)² / 2σ² ), σ = W/3

Overview

Why this matters.

Large-scale dyadic interactive audio-visual dialogue datasets are the raw material for interactive virtual agents and digital humans. Collecting them is slow, expensive, and ethically sensitive.

CHAT generates them instead. It unifies large language models and talking-face models with interactive audio and facial-behaviour refinement, producing aligned dyadic clips with diverse content and diverse identities from a single prompt. It outperforms existing methods built for related tasks under both objective and subjective evaluation.

The synthesised CHAT-AVD-50k dataset then works as pre-training data for downstream interactive head generation, improving both PerFRDiff and ReactDiff on REACT 2024.

Benchmarks

Measured against the closest work.

No existing method addresses this task directly, so CHAT is compared with recent methods for the nearest related tasks, all given the same prompts and audio and standardised on the first ten seconds. Means over 1000 generated dialogues.

Quantitative comparison
Method	FID↓	FVD↓	LSE-C↑	LSE-D↓	FRCorr↑	FRDiv↑	CSIM↑	LPIPS↓
Hallo3	20.56	362.23	4.61	10.41	N/A	N/A	0.79	0.55
SadTalker	22.53	385.51	6.85	8.17	N/A	N/A	0.81	0.48
DIM	36.52	460.35	6.82	8.89	31.02	12.55	0.83	0.52
EDTalk	18.74	619.92	5.62	9.61	N/A	N/A	0.78	0.36
ReactDiff	21.36	386.22	5.23	9.02	48.05	15.31	0.77	0.58
CHAT	17.33	365.03	6.89	8.07	50.02	15.34	0.85	0.38

Best in bold, second underlined. FRCorr and FRDiv are ×10⁻². N/A marks a metric a method cannot produce.

User study · 60 participants · 1–5
Method	Visual	Audio	Sync	Expr.	Interact.
Hallo3	4.5	3.5	3.9	3.8	2.5
SadTalker	2.5	3.7	1.2	2.0	1.3
DIM	3.7	3.1	3.4	2.8	3.6
EDTalk	2.2	3.6	2.3	2.8	2.7
ReactDiff	4.2	3.5	3.8	4.1	3.2
CHAT	4.6	4.8	4.6	4.6	4.8

Ablation
Variant	MCD↓	Emo-Acc↑	ViSQOL↑	FID↓
w/o audio refinement	6.23	41.2%	2.97	21.34
w/o emotion	9.12	38.3%	2.67	23.45
w/o facial refinement	—	—	—	38.67
w/o boundary continuity	—	—	—	18.56
Full CHAT	4.23	78.4%	3.98	17.33

A dash marks an audio metric a visual-only ablation leaves unchanged.

Qualitative comparison against talking-face and facial-reaction baselines.

Talking-face methods run twice give two unrelated monologues. Reaction methods need a real speaker video as input. CHAT produces the pair as one exchange.

CHAT-AVD-50k

A dataset that writes itself.

50,000dyadic dialogues

100,000paired clips

100,000identities

1,388.9hours

Every clip carries turn-level emotion, scenario and speaker metadata, spanning casual, professional, emotional, academic and social scenarios. As pre-training data it lifts two different reaction models on the REACT 2024 benchmark.

Downstream interactive head generation
Pre-training	PerFRDiff FRCorr↑	FRDist↓	ReactDiff FRCorr↑	FRDist↓
REACT 2024 only	37.21	94.72	24.19	86.70
+ CHAT-AVD-50k	40.11	89.45	26.12	83.87

Poster

ECCV 2026.

Thursday 10 September 2026, 7:30–9:30 PDT, ExHall #163. Conference page ›

Open the poster to read it, or take the printed PDF, 1400 by 1000 mm.

The team

Cite this work.

Junhao Song¹, Lluis Guasch¹, Xilin He², Zhongyu Yang³, Yingfang Yuan⁴, Weicheng Xie⁵, Linlin Shen⁵, Haijun Lin⁶, Shizhe Liu⁷, Wei Pang³, Siyang Song^8,†

¹Imperial College London ²Mohamed bin Zayed University of Artificial Intelligence ³Heriot-Watt University ⁴Northumbria University ⁵Shenzhen University ⁶Hunan Normal University ⁷University of Oxford ⁸University of Exeter ^†Corresponding author

@inproceedings{song2026chat,
  title     = {Conversational Human Audio-visual Talking Dialogue Generation},
  author    = {Song, Junhao and Guasch, Lluis and He, Xilin and Yang, Zhongyu and
               Yuan, Yingfang and Xie, Weicheng and Shen, Linlin and Lin, Haijun and
               Liu, Shizhe and Pang, Wei and Song, Siyang},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

The publisher record is pending the ECCV 2026 proceedings.

Paper: arXiv:2607.02799·Video·ECCV 2026 page
Code: github.com/Rqcker/chat — opens after the conference
Dataset: CHAT-AVD-50k — released separately, with provenance metadata and bias auditing

CHAT CHAT

Two people. Talking, listening, answering each other. No script. No reference video. Nothing but a prompt.

Three stages.One conversation.

It writes the dialogue

It speaks both parts

It performs the faces

One timeline, two speakers

Coarse first, fine last

No visible seam

Three stages.
One conversation.