Conversational Human Audio-visual Talking Dialogue Generation

Song, Junhao; Guasch, Lluis; He, Xilin; Yang, Zhongyu; Yuan, Yingfang; Xie, Weicheng; Shen, Linlin; Lin, Haijun; Liu, Shizhe; Pang, Wei; Song, Siyang

doi:10.1007/978-3-032-37252-9_33

European Conference on Computer Vision (ECCV) 2026

A whole conversation.
From one line of text.

Conversational Human Audio-visual Talking Dialogue Generation

See it in motion

Code, weights and the evaluation protocol are on GitHub.

CHAT compared with condition-driven face video generation, face video editing, facial reaction generation and interactive talking-face pair generation.

Other methods animate one face from a condition, edit a clip you already have, or react to a partner you supply. CHAT writes the conversation, speaks it, and performs both sides.

Two people. Talking, listening, answering each other. No script. No reference video. Nothing but a prompt.

Generated dialogues

Watch them take turns.

Nine prompts, nine conversations, eighteen identities that were never paired by hand. Both faces in a clip are generated as a responsive pair, so the one listening reacts to the one speaking, inside the same clip.

Who is speaking, on the one shared timeline

No unit list was kept for this clip, so there is no timeline to draw.

Five turns in ten seconds. Each speaker talks twice and drops a back-channel into the other’s turn, all on one shared timeline.

Both sides interrupt politely: an "uh-huh" at two seconds, a "yeah" at eight, each landing inside the other speaker’s sentence.

A disagreement about which database to use. The reply cuts in as soon as the first speaker stops, with a back-channel halfway through.

An interview about a startup. The listener’s back-channel lands at 3.7 seconds, and the follow-up question waits for the answer to finish.

A grandparent and a grandchild. Four spans here, and the last back-channel starts in the final half second of the clip.

Talking someone down before they go on. Reassurance and doubt each get a sentence, and each takes a back-channel inside it.

A neighbour raises the mess left after a party. The apology begins two tenths of a second after the complaint ends.

Two organisers before the gates open. The reply runs five and a half seconds and takes a "yeah" at 8.2.

Identities need not be photorealistic. The same method drives illustrated faces with no change. This clip comes from the conference video, so it is lower resolution than the others.

Each clip is the full CHAT output for one prompt, generated at 512 px per face and then passed through the CodeFormer face upscaler that Hallo2 ships, which doubles it to 1024 px a side. Every number on this page is measured before that step. The eight real-identity clips play without sound on this page; the illustrated clip keeps the soundtrack from the conference video. Each timeline is drawn from that clip's own generated unit list, except the illustrated one, whose list was not kept and whose turns are measured back from the clip by pitch. Identity descriptors carry no gender, so a face and a voice are drawn independently and need not agree.

How it works

Three stages.
One conversation.

01
It writes the dialogue

A language model turns one scenario prompt into multi-turn dyadic scripts, each with a described identity for both speakers.
02
It speaks both parts

Back-channels, per-turn emotion, timing and a shared acoustic environment are added, then synthesised as two voices on one timeline.
03
It performs the faces

Talking segments are rendered, listening segments generated, each face made responsive to its partner, and every seam blended away.

The CHAT pipeline: textual dialogue generation, dyadic audio dialogue generation, and interactive facial behaviour generation.

The full pipeline. TDG writes the dialogue pairs and identity descriptors, DADG turns each dialogue into interactive emotion-aware audio, and IFBG synthesises the responsive face-video pair.

The responsive facial behaviour generation sub-block: identity and partner behaviour are downsampled to several scales, fused by cross-attention, and injected into the diffusion transformer across denoising steps.

Inside the responsive block. Identity and the partner's behaviour are fused at several spatial scales, then fed into the diffusion transformer coarse first and fine last, so early steps settle the shape of a movement and later steps bring in the detail.

Inside

Three mechanisms, drawn.

Each diagram below is the actual rule the paper specifies, animated as you scroll rather than described in prose.

One timeline, two speakers

Sentences and back-channels are laid on a single shared timeline. While one speaker holds the turn the other is silent, apart from the short interactive words that keep the exchange alive.

W_inter = { uh-huh, hmm, okay, yeah }

Coarse first, fine last

The partner's behaviour is fused with the identity at several spatial scales. Early denoising steps receive only the coarsest scale, which fixes the shape of a movement. Finer scales join as the steps go on, bringing in detail.

l(t) = max(1, ⌈L·t / T⌉) · Φ(t) = ⋃_l=l(t)^L C(l)

No visible seam

Segments are refined separately, so their join could show. A half-Gaussian cross-fade over a narrow window blends the two sides, weighted equally at the boundary itself and decaying outwards.

α(τ) = ½ exp( −(τ−1)² / 2σ² ), σ = W/3

Overview

Why this matters.

Large-scale dyadic interactive audio-visual dialogue datasets are the raw material for interactive virtual agents and digital humans. Collecting them is slow, expensive, and ethically sensitive.

CHAT generates them instead. It unifies large language models and talking-face models with interactive audio and facial-behaviour refinement, producing aligned dyadic clips with diverse content and diverse identities from a single prompt. It outperforms existing methods built for related tasks under both objective and subjective evaluation.

The synthesised CHAT-AVD-50k dataset then works as pre-training data for downstream interactive head generation, improving both PerFRDiff and ReactDiff on REACT 2024.

Benchmarks

Measured against the closest work.

No existing method addresses this task directly, so CHAT is compared with recent methods for the nearest related tasks, all given the same prompts and audio and standardised on the first ten seconds. Means over 1000 generated dialogues.

Quantitative comparison
Method	FID↓	FVD↓	LSE-C↑	LSE-D↓	FRCorr↑	FRDiv↑	CSIM↑	LPIPS↓
Hallo3	20.56	362.23	4.61	10.41	N/A	N/A	0.79	0.55
SadTalker	22.53	385.51	6.85	8.17	N/A	N/A	0.81	0.48
DIM	36.52	460.35	6.82	8.89	31.02	12.55	0.83	0.52
EDTalk	18.74	619.92	5.62	9.61	N/A	N/A	0.78	0.36
ReactDiff	21.36	386.22	5.23	9.02	48.05	15.31	0.77	0.58
CHAT	17.33	365.03	6.89	8.07	50.02	15.34	0.85	0.38

Best in bold, second underlined. FRCorr and FRDiv are ×10⁻². N/A marks a metric a method cannot produce.

User study · 60 participants · 1–5
Method	Visual	Audio	Sync	Expr.	Interact.
Hallo3	4.5	3.5	3.9	3.8	2.5
SadTalker	2.5	3.7	1.2	2.0	1.3
DIM	3.7	3.1	3.4	2.8	3.6
EDTalk	2.2	3.6	2.3	2.8	2.7
ReactDiff	4.2	3.5	3.8	4.1	3.2
CHAT	4.6	4.8	4.6	4.6	4.8

Ablation
Variant	MCD↓	Emo-Acc↑	ViSQOL↑	FID↓
w/o audio refinement	6.23	41.2%	2.97	21.34
w/o emotion	9.12	38.3%	2.67	23.45
w/o facial refinement	—	—	—	38.67
w/o boundary continuity	—	—	—	18.56
Full CHAT	4.23	78.4%	3.98	17.33

A dash marks an audio metric a visual-only ablation leaves unchanged.

Qualitative comparison against talking-face and facial-reaction baselines.

Talking-face methods run twice give two unrelated monologues. Reaction methods need a real speaker video as input. CHAT produces the pair as one exchange.

CHAT-AVD-50k

A dataset that writes itself.

50,000dyadic dialogues

100,000paired clips

100,000identities

1,388.9hours

Every clip carries turn-level emotion, scenario and speaker metadata, spanning casual, professional, emotional, academic and social scenarios. As pre-training data it lifts two different reaction models on the REACT 2024 benchmark.

Downstream interactive head generation
Pre-training	PerFRDiff FRCorr↑	FRDist↓	ReactDiff FRCorr↑	FRDist↓
REACT 2024 only	37.21	94.72	24.19	86.70
+ CHAT-AVD-50k	40.11	89.45	26.12	83.87

Poster

ECCV 2026.

Thursday 10 September 2026, 7:30–9:30 PDT, ExHall #163. Conference page ›

Open the poster to read it, or take the printed PDF, 1400 by 1000 mm.

The team

Cite this work.

Junhao Song¹, Lluis Guasch¹, Xilin He², Zhongyu Yang³, Yingfang Yuan⁴, Weicheng Xie⁵, Linlin Shen⁵, Haijun Lin⁶, Shizhe Liu⁷, Wei Pang³, Siyang Song^8,†

¹Imperial College London ²Mohamed bin Zayed University of Artificial Intelligence ³Heriot-Watt University ⁴Northumbria University ⁵Shenzhen University ⁶Hunan Normal University ⁷University of Oxford ⁸University of Exeter ^†Corresponding author

@inproceedings{song2026chat,
  title     = {Conversational Human Audio-visual Talking Dialogue Generation},
  author    = {Song, Junhao and Guasch, Lluis and He, Xilin and Yang, Zhongyu and
               Yuan, Yingfang and Xie, Weicheng and Shen, Linlin and Lin, Haijun and
               Liu, Shizhe and Pang, Wei and Song, Siyang},
  booktitle = {Computer Vision -- ECCV 2026},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {597--616},
  doi       = {10.1007/978-3-032-37252-9_33}
}

Paper

Computer Vision – ECCV 2026

Springer Nature Switzerland, pages 597–616.

Chapter arXiv Talk ECCV

Published

Code

github.com/Rqcker/chat

Reference implementation, with the two IFBR weights.

GitHub

Available

Dataset

CHAT-AVD-50k

Provenance metadata and bias auditing ship with the clips.

What is in it

Released separately

CHAT CHAT

Two people. Talking, listening, answering each other. No script. No reference video. Nothing but a prompt.

Three stages.One conversation.

It writes the dialogue

It speaks both parts

It performs the faces