Research / Robust speech recognition and spoken search / [3]
Who spoke when in rural interviews: second place at ALBAYZIN 2026
Robust Speaker Diarization under Spontaneous Rural Interviews via Cross-Architecture Fusion: The IberSPEECH 2026 COSER Challenge. IberSPEECH 2026, ALBAYZIN Speech-COSER Challenge, 2026.
Fine-tuning pyannote's segmentation cuts missed speech from 18 % to 2 %, and fusing it with a WavLM-based diarizer that confuses speakers 2.5 times less brings the blind-test diarization error rate to 15.72 %.
The problem
Speaker diarization answers “who spoke when”. COSER makes it hard: interviews of 50 minutes to 2.5 hours, three to five speakers each, short turns, back-channels and up to 85 % overlapped speech in one file. The challenge scores every frame, with no tolerance at segment boundaries and overlap included. Two errors dominate: speech the system misses, and speakers it confuses.
What we did
Two systems with opposite strengths, fused.
- Branch A is the pyannote pipeline with its segmentation model fine-tuned on the eight labelled COSER recordings. That one change cuts missed speech from about 18 % to 2 %. Its clustering threshold was then tuned with Optuna, on features computed so that no recording was ever scored by a model that had seen it.
- Branch B is DiariZen, a diarizer built on the self-supervised WavLM-Large model, used without any adaptation. It misses a quarter of the speech, but confuses speakers 2.5 times less than Branch A (6.3 % against 15.6 %).
DOVER-Lap combines the two by overlap-aware voting: the result keeps Branch A’s coverage and pulls speaker confusion down to 11.5 %.
What we found
The fusion reaches 15.72 % DER on the blind test. The more useful finding is about what survives a change of data. Tuning the clustering threshold gained 2.61 points on development but only 0.57 on the test set. Fusing with an architecturally different model gained 0.70 points on development and 1.59 on test: the benefit grew on the harder data. Diversity of the wrong kind did not help: fusing three copies of the same pipeline trained with different seeds, or swapping DiariZen for a stronger embedding of the same family, gave little or nothing, because their errors were correlated.
Why it matters
When test data differ from development data, a model built differently is a more reliable source of gain than a tuned threshold. On the challenge’s final leaderboard the system finished second of six. The story of both tracks is in Two second places at ALBAYZIN 2026.
Cite
Ramírez-Sánchez J.M., Piñeiro-Martín A., Docío-Fernández L., García-Mateo C. Robust Speaker Diarization under Spontaneous Rural Interviews via Cross-Architecture Fusion: The IberSPEECH 2026 COSER Challenge. IberSPEECH 2026, ALBAYZIN Speech-COSER Challenge, 2026.