Tutorials / T4
Radial LoRA on Whisper, implemented from scratch
In one sentence
Write R-LoRA in a few lines of PyTorch, check it numerically, train a Whisper classifier next to a LoRA baseline, and learn the two things that change in practice: the learning rate and merging.
Why this tutorial
R-LoRA is a one-parameter change to LoRA that I introduced for clinical speech, and the peft library does not include it. This notebook writes it from scratch and runs it through the same steps as the LoRA tutorial, on the same data, so the two can be read side by side.
What you do in the notebook
- Write
RLoRALinear, a module that wraps anynn.Linear, and check its three properties: it equals the frozen layer at initialisation, it equals LoRA when λ = 0, and it is not linear in its input. - Inject it into Whisper with a small function that plays the role of
target_modules, and train a classifier next to apeftLoRA baseline with the same rank and seed. - Match the learning rate. At LoRA’s rate, R-LoRA learns more slowly, because its branch sees an input divided by ‖x‖λ, about 8 times smaller on Whisper’s attention inputs. Scaling the adapters’ rate by that measured factor makes it converge as fast as LoRA.
- Read the learned λ, one per adapted layer.
- Try to merge it. LoRA folds into the weights with no change in output; a naive merge of R-LoRA changes the logits, which is why it has to be served whole.
- Save and reload only the adapter, and build an embedding extractor with R-LoRA on the attention and feed-forward layers.
What to take away
- R-LoRA is about ten lines on top of LoRA, and nothing in them is specific to Whisper or to speech.
- Do not reuse LoRA’s learning rate blindly: measure the scale the branch sees, or tune the rate per method.
- The synthetic task is easy, so both methods solve it. The gains of the paper come from scarce clinical data whose representations are crowded into few directions; the notebook teaches the mechanism, not the result.