Working papers · speech, health & trustworthy AIVigo, 2026

Writing /

Radial LoRA: when the direction matters more than the size

In one sentence

R-LoRA is a variant of LoRA that tempers the size of the adapter's input with one learnable exponent per layer. On four clinical speech corpora it beats LoRA by 12.4 points of Macro-F1, and it is a drop-in change for any Transformer that LoRA can adapt.

Large pretrained speech models know a lot about the voice, but clinical datasets are small: a few hundred speakers at most. Adapting a model of 95 million parameters to 150 patients is the everyday problem of health AI from speech, and the subject of my thesis. Radial LoRA (R-LoRA) is the method I propose for it, presented at IberSPEECH 2026: a one-parameter change to LoRA that, on four clinical corpora, raises Macro-F1 by 12.4 points at the same parameter budget.

+12.4Macro-F1 points over LoRA
p = 4.4 × 10⁻¹⁰paired test, 160 comparisons
+24parameters over LoRA
< 1 %added inference time over unmerged LoRA

The idea in one paragraph

LoRA adapts a frozen weight matrix by adding a small low-rank update, and that update is linear in its input: it reacts to how large a hidden activation is just as much as to which direction it points. Inside a Transformer, layer normalisation keeps representations on a sphere of fixed radius, and in many dimensions the angle between two vectors says more than their distance. R-LoRA asks the adapter to listen to the direction first: before the low-rank projection, it divides the input by its norm raised to a learnable power λ.

LoRA linear in x x W₀ · frozen A B × α / r + h r × kd × r R-LoRA learns λ x W₀ · frozen T_λ A B × α / r + h r × kd × r radial tempering λ = 0 plain LoRA λ = 0.5 partly normalised λ = 1 direction only h = W₀x + (α/r) · B A T_λ(x) T_λ(x) = x / (‖x‖ + ε)^λ, λ ∈ [0, 1] B starts at zero, so training starts from the pretrained model. T_λ is non-linear, so the update cannot be merged into W₀.
Figure 1. LoRA and R-LoRA. Both keep the pretrained weight frozen and add a low-rank branch; R-LoRA inserts one operator, Tλ, that tempers the size of the input. Right: the same input after tempering with λ = 0, 0.5 and 1.

How it works

For a frozen weight W₀, LoRA computes h = W₀x + (α/r)·BAx, where A and B are thin matrices of rank r. R-LoRA changes one thing: the branch receives Tλ(x) = x / (‖x‖ + ε)λ instead of x.

  • λ = 0 gives back plain LoRA.
  • λ = 1 projects the input onto the unit sphere: the update depends only on its direction.
  • Values in between keep part of the magnitude.

λ is learned, one value per adapted projection, through a sigmoid that keeps it between 0 and 1. As in LoRA, B starts at zero, so training starts exactly from the pretrained model and finds by itself how much normalisation each layer needs. On Wav2Vec2 base, with adapters on the query and value projections of its 12 layers, that adds 24 parameters to LoRA’s 290,000 at rank 8.

Why it should help on clinical speech

The paper first measures the geometry of the frozen model on each corpus. The representations are crowded: their vectors point in nearly the same direction, they use a small fraction of the space, and healthy and pathological speakers are separated by small angles.

CorpusTaskMean cosineDimensions holding 90 % of varianceAngle between classes
NeuroVozParkinson’s disease0.81308.5°
COPERIA /a/Long COVID, sustained vowel0.842421.1°
COPERIA coughLong COVID, cough0.94225.4°
VisIAsuicidal ideation, free speech0.901612.2°

When everything points almost the same way, the useful signal is in small differences of direction, and a linear adapter spends part of its capacity following the size of the input instead.

What we found

The comparison is strict: Wav2Vec2 base, eight ranks from 2 to 256, R-LoRA and LoRA at the same rank, 5 × 5 speaker-stratified nested cross-validation, and every metric computed per speaker, the unit that matters clinically.

0102030 NeuroVoz 1.1 COPERIA /a/ 14.1 COPERIA cough 10.2 VisIA 24.1 All corpora 12.4
Figure 2. Mean Macro-F1 gain of R-LoRA over LoRA (points), over eight ranks and five outer folds. Significant on COPERIA and VisIA after Holm correction; not significant on NeuroVoz.
  • Overall: +12.4 points of Macro-F1 (95 % CI 8.9 to 15.8), winning 113 of 160 paired comparisons. Significant on three corpora, never significantly worse.
  • At every budget: the margin is widest at the smallest rank (+17.4 points at rank 2) and still +10.7 at rank 256.
  • Against full fine-tuning: updating all 95 million weights trails at every rank, by 14.2 points on average. R-LoRA matches or beats it on every corpus while training about 322 times fewer parameters.

The case that matters most is VisIA:

VisIA, best rank per methodMacro-F1Recall, active ideationPrecision, active ideation
Full fine-tuning36.30.00.0
LoRA46.40.00.0
R-LoRA67.050.080.0

Both baselines collapse to the majority class and never flag a person with active suicidal ideation. R-LoRA finds half of them. In screening, a missed case is the costly error, and across all four corpora R-LoRA raises recall of the pathological class from 51.9 % to 59.0 %.

Where LoRA already does well, there is less to gain: on NeuroVoz it comes within two points of the best score any method reaches, and R-LoRA adds only 1.1 points, not significant.

What the model chooses

01 plain LoRA direction only median 0.52 0.080.94 screened band: 0.1 to 0.9, where 88 % of the learned values fall
Figure 3. Learned exponents λ across the 32 corpus × rank cells on Wav2Vec2. They range from 0.08 to 0.94 with median 0.52: the model picks partial normalisation, never plain LoRA and never direction only.

Left free, every layer settles somewhere in between. That is the mechanism behind the gain: neither LoRA’s raw input nor a purely directional one, but a learned compromise that differs by layer and by corpus.

The cost

R-LoRA gives up one of LoRA’s conveniences. A LoRA update is linear, so after training it can be folded into the weights and costs nothing at inference. The radial operator is non-linear, so R-LoRA’s branch has to run at every forward pass. Measured at rank 8, inference takes 3.30 ms per sample against 3.28 for an unmerged LoRA branch, within run-to-run noise; training runs at 0.96 times LoRA’s speed, with the same memory. For offline screening on scarce data, that is a small price.

Not only for clinical speech

Nothing in R-LoRA is specific to speech or to health. It changes the input of the adapter, not the model, so it applies wherever LoRA applies: any linear projection of any Transformer, for classification or for generation.

The evidence so far covers two settings. In the paper, Wav2Vec2 classifying four clinical conditions. And in my submission to the ALBAYZIN 2026 speech recognition challenge, Whisper-large-v3, an encoder-decoder trained for transcription, adapted on rural dialectal Spanish: on the blind test, R-LoRA reached 26.31 % normalised word error rate against 29.24 % for LoRA, with the same rank and the same data, and trained more stably.

What does not transfer automatically is the size of the gain. It is largest where the representations are most collapsed and the data scarcest, and the paper expects stronger encoders, whose spaces are better conditioned, to show a smaller effect. Text and vision Transformers are untested. The next steps are WavLM and HuBERT, and a head-to-head with DoRA, which normalises the weights rather than the input; the two act on different things and could be combined.

The paper, with the full protocol and the statistics, is on its page in Research. To implement it yourself, the tutorial Radial LoRA on Whisper, implemented from scratch builds it in PyTorch and trains it next to LoRA on a laptop CPU.

With Laura Docío-Fernández, Alejandro García-Caballero and Carmen García-Mateo (Universidade de Vigo and Complexo Hospitalario Universitario de Ourense). Funded by the Xunta de Galicia (GPC ED431B 2024/36) and the European Union. Presented at IberSPEECH 2026.