TurnBench
A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
Leaderboard
Models evaluated against TurnBench's held-out test set, ranked by recall. The ideal model is capable of high recall while maintaining a low false-positive rate and low latency.
- Recall:
- share of true events detected (higher is better).
- FPR:
- false-positive rate on test (lower is better); above 0.15 does not qualify.
- Latency:
- median delay after the event, p50 in ms (lower is better).
Dataset
TurnBench is built on a studio-recorded corpus of dual-channel dyadic speech, balanced across 6 conversation types1 and split evenly between clean and noisy recording conditions. Every dialogue is labeled by three independent annotators (Fleiss's κ = 0.76) where the ground truth label is derived from 2/3 consensus.
- hours of speech
- 30
- dialogues
- 154
- voice actors
- 106
- conversation types
- 6
Additionally, we include a 104-hour hand-labeled training set, annotated under the same protocol, released alongside the benchmark.
Evaluation
Per conversation, TurnBench requires models to output a discrete list of times2 for the following events:
- End-of-turn
- When a speaker has finished speaking and the floor is open for the other speaker to take.FP: declaring end-of-turn during a mid-turn pause.
- Interruption
- When a speaker takes the floor while the other is still talking.FP: declaring an interruption during a backchannel or because of noise, channel bleed, or echo.