Working papers · speech, health & trustworthy AIVigo, 2026

Writing /

Two second places at ALBAYZIN 2026: recognising and diarizing rural Spanish

In one sentence

My systems finished second in both the speech recognition and the speaker diarization tracks of the ALBAYZIN 2026 Speech-COSER challenge, on dialectal rural Spanish, trained on nothing but the challenge's own COSER data.

The ALBAYZIN evaluations are the long-running benchmark for speech technology in the languages of the Iberian Peninsula, held with the IberSPEECH conference. Every team gets the same data and is scored by the organisers on a blind test set that no one has seen. In the 2026 edition, Speech-COSER, my systems finished second in automatic speech recognition and second in speaker diarization, out of six teams in each track, among them research groups and companies such as Telefónica and pyannoteAI.

2nd / 6speech recognition
2nd / 6speaker diarization
0.09WER points from first place
COSER onlyno training data outside the challenge

Three things give these results their weight. In speech recognition, first place was 0.09 points of word error rate away, and Telefónica’s team finished behind. In diarization, the only team ahead was pyannoteAI, the company behind pyannote, the reference toolkit for diarization and the one my own system builds on. And both systems were the only ones in the challenge trained on no data other than the COSER recordings the organisers provided, on top of publicly available pretrained models.

Why COSER is hard

COSER, the Corpus Oral y Sonoro del Español Rural, collects ethnographic interviews recorded across rural Spain: conversations of fifty minutes to two and a half hours between an interviewer and three to five local speakers, in strongly marked dialectal varieties. The speech overlaps (up to 85 % of one file), the signal-to-noise ratio ranges from about 8 to 30 dB, and some provinces mix Spanish with Galician, Catalan or Basque. The transcripts are verbatim: pos, pa and torcío are written as spoken, so any system that “corrects” them towards standard Spanish is penalised.

The two systems answer that difficulty with two families of methods. Speech recognition relies on parameter-efficient fine-tuning: adapt one strong model to the domain while touching under 1 % of its weights. Speaker diarization relies on output voting: combine two systems that make different mistakes.

Speech recognition: parameter-efficient fine-tuning with R-LoRA

0102030 1st place 20.5 This work 20.6 Telefónica 22.2 4th 22.6 5th 23.0 6th 28.9
Figure 1. Final leaderboard of the speech recognition track, normalised WER (%), by position. Lower is better.

Why parameter-efficient

Whisper-large-v3 has 1.5 billion parameters and the challenge gives about 23 hours of in-domain audio. Fine-tuning every weight on so little data is expensive in memory and tends to erase what the model already knows. LoRA keeps the pretrained weights frozen and learns a small low-rank update next to each adapted layer: two thin matrices, A and B, whose product has the shape of the original weight.

What R-LoRA changes

Plain LoRA’s update is linear in its input, so it reacts to the size of a hidden activation as much as to its direction. R-LoRA (Radial LoRA) starts from the hypothesis that the useful adaptation signal is mostly in the direction. Before the down-projection it passes the input through a radial operator that divides it by its norm raised to a power λ. Each adapted layer learns its own λ between 0 and 1: at 0 it is plain LoRA, at 1 the update sees only the direction. The cost over LoRA is one parameter per layer. I introduced R-LoRA for clinical speech; Radial LoRA: when the direction matters more than the size explains it in full.

LoRA linear in x x W₀ · frozen, 1.5 B params A B × α / r + h r × kd × r R-LoRA learns λ x W₀ · frozen, 1.5 B params T_λ A B × α / r + h r × kd × r radial tempering λ = 0 plain LoRA λ = 0.5 partly normalised λ = 1 direction only h = W₀x + (α/r) · B A T_λ(x) T_λ(x) = x / (‖x‖ + ε)^λ, λ ∈ [0, 1] B starts at zero, so training starts from the pretrained model. T_λ is non-linear, so the update cannot be merged into W₀.
Figure 2. LoRA and R-LoRA, after the paper. Both keep the pretrained weight frozen and add a low-rank branch; R-LoRA inserts one operator, Tλ, that tempers the size of the input before it. Right: the same input after tempering with λ = 0, 0.5 and 1.

The system

  • Adapters: rank 16 on the attention projections (query, key, value, output) of encoder and decoder, about 15 million trainable parameters, 0.98 % of the model. λ starts at 0.5.
  • Training: about 23 hours from 35 recordings, with online SpecAugment, learning rate 2 × 10⁻⁵, batch 32, early stopping. It fits in about 10 GB of GPU memory.
  • Model selection: on whole held-out recordings, not on short segments. The best checkpoint on full audio came earlier than the best on segments, so the convenient proxy would have chosen the wrong model.
  • Inference: loudness normalisation (the test audio was about 7 dB hotter than the training audio), Silero voice activity detection, beam search, and a language prior from the province of each recording.
  • Packaging: because the radial operator is non-linear, the adapters cannot be merged into the weights as LoRA’s can. The model ships as one self-contained Hugging Face model with its own code, so deployment is a single load call.

What the numbers say

Blind test, 22 recordingsRaw WERNormalised WER
Whisper-large-v3, zero-shot45.2531.32
+ LoRA45.1029.24
+ R-LoRA41.5826.31

On development data LoRA and R-LoRA were only 0.33 points apart; on the blind test the gap grew to 2.93. LoRA was also more fragile: in the ablation, one checkpoint after its best, its WER jumped from 39.8 % to 55.6 %, while R-LoRA degraded gracefully. Both observations suggest the same thing: tempering the input helps the adaptation generalise and train more stably.

Voting and correcting the transcript: ROVER and language models

Once a recogniser has produced its hypotheses, two kinds of post-processing are common. ROVER aligns several transcriptions word by word into one network and keeps, at each position, the word with the most confidence-weighted votes. Language-model correction rewrites the transcript towards what a language model finds more likely.

On COSER only ROVER helped: voting over the model’s own n-best hypotheses removed 0.7 points of WER. Every language-model corrector made things worse, by 0.2 to 2.9 points: an in-domain n-gram rescorer, a fine-tuned mT5 and Gemma 3 12B instructed to preserve dialect. All of them pull the transcript towards standard Spanish (coge becomes cogen), and a verbatim reference counts every such change as an error. Those tests ran on an earlier decoding path, about 4 points weaker than the submitted one, so neither ended up in the final system. The ASR paper has the full ablation.

Speaker diarization: voting between systems that fail differently

05101520 pyannoteAI 10.4 This work 12.1 3rd 12.4 4th 13.0 5th 13.3 6th 18.6
Figure 3. Final leaderboard of the speaker diarization track, diarization error rate (%), no collar, overlap scored, by position. Lower is better.

Diarization answers “who spoke when”. Its error rate adds three parts: false alarms (speech where there is none), missed speech, and speaker confusion (speech given to the wrong speaker). On COSER the last two dominate.

Two branches with opposite strengths

  • Branch A, fine-tuned pyannote. The pyannote pipeline segments the audio into local speaker activity, embeds each segment and clusters the embeddings. Fine-tuning only its segmentation model on the eight labelled interviews (5 epochs) cut missed speech from about 18 % to 2 %. The clustering threshold was then tuned with Optuna on features cached per fold, so no recording was ever scored by a model that had seen it. The speaker-embedding model was left untouched: with about 40 different speakers in total, adapting it overfitted.
  • Branch B, DiariZen. An end-to-end diarizer built on the self-supervised WavLM-Large model, used without adaptation. On its own it is worse, because it misses a quarter of the speech, but it confuses speakers 2.5 times less than Branch A.
Development, leave-one-out (%)False alarmMissedConfusionDER
A · fine-tuned pyannote≈ 02.015.618.2
B · DiariZen, zero-shot5.423.36.334.9
DOVER-Lap fusion≈ 0.52.011.514.9

DOVER-Lap: ROVER for diarization

DOVER (diarization output voting error reduction) carries ROVER’s idea from words to speakers. Two diarizations cannot be compared directly, because each names its speakers arbitrarily: “speaker 1” of one system may be “speaker 3” of the other. DOVER-Lap first finds one global mapping between the labels of the systems, pairing those that overlap most in time. It then votes region by region, with each system’s vote weighted by its rank, and it allows several speakers to win the same region, which is what overlapping speech needs.

A · fine-tuned pyannote almost all speech B · DiariZen (WavLM) gaps, few confusions Fused (DOVER-Lap) A’s coverage 1 · map labels across systems pair the labels that overlap most in time 2 · vote region by region weights by rank; two speakers can share a region speaker 1 speaker 2 extra label (a confusion) time →
Figure 4. DOVER-Lap, as a schematic: segments are illustrative. The fused output keeps Branch A's coverage while the vote corrects labels. In the real system the vote lowered speaker confusion from 15.6 % to 11.5 %, not to zero.

The fusion inherits Branch A’s near-complete coverage and lowers its speaker confusion from 15.6 % to 11.5 %. On the blind test it reached 15.72 % DER in the paper’s runs.

What transfers to unseen data

ChangeGain on developmentGain on blind test
Tuning the clustering threshold−2.61 points−0.57 points
Adding DiariZen through DOVER-Lap−0.70 points−1.59 points

The tuned threshold lost most of its value on the test set; the second architecture gained more on the test set than on development. Not every kind of diversity works: fusing three copies of the same pipeline trained with different seeds, or replacing DiariZen with a stronger embedding of the same family, gave little or made things worse, because their errors were correlated. Voting only helps when the voters fail differently. The diarization paper has the analysis.

What this shows

  • Results under evaluation conditions, against industry and academic teams, on data none of us had seen.
  • Methods chosen for the data, not for the leaderboard: a parameter-efficient adapter for a small in-domain set, and a fusion of two architectures for a domain shift.
  • Data discipline: no external training data, leave-one-out validation on whole recordings, and decisions taken on held-out audio rather than on convenient proxies.
  • Engineering that ships: the ASR system is one self-contained model with its inference stack; the diarization system is two existing components, one adapted, fused with a standard method.

Both systems were built with Andrés Piñeiro-Martín, Laura Docío-Fernández and Carmen García-Mateo, and the speech recognition system also with Alejandro Esperon-Couceiro, at the Universidade de Vigo and Balidea. Both are presented at IberSPEECH 2026.

Positions and scores in Figures 1 and 3 are from the challenge’s final leaderboard, closed on 30 September 2026; the full table, with every team, is public on Codabench. Method details and the other tables come from the two papers, which report their own blind-test runs, so their numbers differ from the leaderboard’s.