Working papers · speech, health & trustworthy AIVigo, 2026

Tutorials / T4

Radial LoRA on Whisper, implemented from scratch

In one sentence

Write R-LoRA in a few lines of PyTorch, check it numerically, train a Whisper classifier next to a LoRA baseline, and learn the two things that change in practice: the learning rate and merging.

Why this tutorial

R-LoRA is a one-parameter change to LoRA that I introduced for clinical speech, and the peft library does not include it. This notebook writes it from scratch and runs it through the same steps as the LoRA tutorial, on the same data, so the two can be read side by side.

LoRA linear in x x W₀ · frozen A B × α / r + h r × kd × r R-LoRA learns λ x W₀ · frozen T_λ A B × α / r + h r × kd × r radial tempering λ = 0 plain LoRA λ = 0.5 partly normalised λ = 1 direction only h = W₀x + (α/r) · B A T_λ(x) T_λ(x) = x / (‖x‖ + ε)^λ, λ ∈ [0, 1] B starts at zero, so training starts from the pretrained model. T_λ is non-linear, so the update cannot be merged into W₀.
Figure 1. What the notebook implements: the LoRA branch with one extra operator, Tλ, that tempers the size of its input.

What you do in the notebook

  1. Write RLoRALinear, a module that wraps any nn.Linear, and check its three properties: it equals the frozen layer at initialisation, it equals LoRA when λ = 0, and it is not linear in its input.
  2. Inject it into Whisper with a small function that plays the role of target_modules, and train a classifier next to a peft LoRA baseline with the same rank and seed.
  3. Match the learning rate. At LoRA’s rate, R-LoRA learns more slowly, because its branch sees an input divided by ‖x‖λ, about 8 times smaller on Whisper’s attention inputs. Scaling the adapters’ rate by that measured factor makes it converge as fast as LoRA.
  4. Read the learned λ, one per adapted layer.
  5. Try to merge it. LoRA folds into the weights with no change in output; a naive merge of R-LoRA changes the logits, which is why it has to be served whole.
  6. Save and reload only the adapter, and build an embedding extractor with R-LoRA on the attention and feed-forward layers.

What to take away

  • R-LoRA is about ten lines on top of LoRA, and nothing in them is specific to Whisper or to speech.
  • Do not reuse LoRA’s learning rate blindly: measure the scale the branch sees, or tune the rate per method.
  • The synthetic task is easy, so both methods solve it. The gains of the paper come from scarce clinical data whose representations are crowded into few directions; the notebook teaches the mechanism, not the result.