Working papers · speech, health & trustworthy AIVigo, 2026

Research / Robust speech recognition and spoken search / [3]

Who spoke when in rural interviews: second place at ALBAYZIN 2026

Robust Speaker Diarization under Spontaneous Rural Interviews via Cross-Architecture Fusion: The IberSPEECH 2026 COSER Challenge. IberSPEECH 2026, ALBAYZIN Speech-COSER Challenge, 2026.

In one sentence

Fine-tuning pyannote's segmentation cuts missed speech from 18 % to 2 %, and fusing it with a WavLM-based diarizer that confuses speakers 2.5 times less brings the blind-test diarization error rate to 15.72 %.

2nddiarization track, final leaderboard
18 → 2%missed speech after fine-tuning
2.5×less speaker confusion in the second branch

The problem

Speaker diarization answers “who spoke when”. COSER makes it hard: interviews of 50 minutes to 2.5 hours, three to five speakers each, short turns, back-channels and up to 85 % overlapped speech in one file. The challenge scores every frame, with no tolerance at segment boundaries and overlap included. Two errors dominate: speech the system misses, and speakers it confuses.

What we did

Two systems with opposite strengths, fused.

  • Branch A is the pyannote pipeline with its segmentation model fine-tuned on the eight labelled COSER recordings. That one change cuts missed speech from about 18 % to 2 %. Its clustering threshold was then tuned with Optuna, on features computed so that no recording was ever scored by a model that had seen it.
  • Branch B is DiariZen, a diarizer built on the self-supervised WavLM-Large model, used without any adaptation. It misses a quarter of the speech, but confuses speakers 2.5 times less than Branch A (6.3 % against 15.6 %).

DOVER-Lap combines the two by overlap-aware voting: the result keeps Branch A’s coverage and pulls speaker confusion down to 11.5 %.

What we found

The fusion reaches 15.72 % DER on the blind test. The more useful finding is about what survives a change of data. Tuning the clustering threshold gained 2.61 points on development but only 0.57 on the test set. Fusing with an architecturally different model gained 0.70 points on development and 1.59 on test: the benefit grew on the harder data. Diversity of the wrong kind did not help: fusing three copies of the same pipeline trained with different seeds, or swapping DiariZen for a stronger embedding of the same family, gave little or nothing, because their errors were correlated.

Why it matters

When test data differ from development data, a model built differently is a more reliable source of gain than a tuned threshold. On the challenge’s final leaderboard the system finished second of six. The story of both tracks is in Two second places at ALBAYZIN 2026.

0255075100 Fine-tuned pyannote 18.2 DiariZen 34.9 Fusion 14.9
Figure 1. Error components on development data (leave-one-out, %). Branch A rarely misses speech; Branch B rarely confuses speakers; the fusion keeps the best of each.

Cite

Ramírez-Sánchez J.M., Piñeiro-Martín A., Docío-Fernández L., García-Mateo C. Robust Speaker Diarization under Spontaneous Rural Interviews via Cross-Architecture Fusion: The IberSPEECH 2026 COSER Challenge. IberSPEECH 2026, ALBAYZIN Speech-COSER Challenge, 2026.