adapters
eigenkit.orthogonal.torch.adapters
Orthogonal fine-tuning: adapt a pretrained layer by rotating its weight.
An OrthogonalAdapter keeps the pretrained layer frozen and trains only the angles
of an orthogonal matrix \(R\), applied as \(W' = RW\) (or \(W' = WR^\top\) on the
inputs). \(R\) starts at the identity, so the adapted model starts at the
pretrained function, and \(RW\) has exactly the singular values of \(W\) at every
step: the Lipschitz constant of the layer never changes. merge() folds \(R\) into
the weight, so the adapter costs nothing at inference.
Example
import torch.nn as nn
from eigenkit.orthogonal.torch import (
apply_orthogonal_adapters,
merge_adapters,
trainable_parameters,
)
model = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 10))
apply_orthogonal_adapters(model, ["0", "2"], topology="pyramid")
params = list(trainable_parameters(model)) # the angles only
merge_adapters(model) # plain nn.Linear layers again
OrthogonalAdapter
Bases: Module
A frozen nn.Linear or nn.Conv2d whose weight is rotated by a trainable
orthogonal matrix.
side="out" gives \(W' = RW\), with \(R\) on the output channels;
side="in" gives \(W' = WR^\top\), with \(R\) on the input channels. The bias of the
base layer is kept as it is. The weight of a convolution is read as the matrix
(out, in·k·k) (see rotate_weight).
With the butterfly layout and a width that is not a power of two, the rotation
is a PartitionedOrthogonal when partition is None (the default) or True.
Any other layout takes any width it accepts, and partition=True uses the
partitioned form for it too. The angles take the dtype and the device of the
base weight.
An adapter cannot be re-initialised (see reset_parameters), so a model with
adapters is trained with a training loop of your own: the estimators
GenericClassifier and GenericRegressor re-initialise every layer in
fit() and refuse it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base
|
Linear | Conv2d
|
The pretrained layer. Its parameters are frozen in place. |
required |
topology
|
TopologyName | Topology
|
The layout of the rotation.
Default: |
'butterfly'
|
side
|
Side
|
|
'out'
|
partition
|
bool | None
|
Whether to use |
None
|
stages
|
int
|
The stages of a partitioned rotation. Default: |
2
|
perm_seed
|
int
|
The permutation seed of a partitioned rotation.
Default: |
0
|
bias
property
The bias of the base layer, base.bias, which is not rotated.
Returns:
| Type | Description |
|---|---|
Tensor | None
|
torch.Tensor | None: The bias, or |
weight
property
The adapted weight rotate_weight(R, base.weight, side).
A property, like the weight of the layer it replaces, so code that reads
layer.weight directly (nn.MultiheadAttention reads out_proj.weight)
gets the adapted weight and trains the angles. It is rebuilt from the
angles at every read.
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: A tensor of the shape of |
forward(x)
Run the base layer with the adapted weight.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
The input of the base layer. |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The output of the base layer with weight |
merge()
A copy of the base layer with the rotation folded into its weight.
Its parameters are frozen, like those of the base layer.
Returns:
| Type | Description |
|---|---|
Linear | Conv2d
|
nn.Linear | nn.Conv2d: A plain layer that computes what the adapter does. |
reset_parameters()
Refuse to re-initialise: an adapter starts from its pretrained weight.
model.apply(lambda m: m.reset_parameters()), which the estimators
GenericClassifier and GenericRegressor run at the start of fit(),
resets the children of a module before the module itself, so by the time
this method runs base already holds a new random weight. Training would
then rotate that random weight instead of the pretrained one, without any
error. Raising here stops it. Train an adapted model with a training loop
of your own, as in the fine-tuning example.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
Always. |
apply_orthogonal_adapters(model, target_modules, topology='butterfly', side='out', freeze_rest=True, **adapter_kwargs)
Replace layers of a model by OrthogonalAdapters, in place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
The model to adapt. |
required |
target_modules
|
str | Iterable[str] | Callable[[str, Module], bool]
|
The qualified name of the layer to adapt, or the names of several
layers (as in |
required |
topology
|
TopologyName | Topology
|
The layout of every rotation.
Default: |
'butterfly'
|
side
|
Side
|
|
'out'
|
freeze_rest
|
bool
|
Whether to freeze every parameter that is not an angle
of an adapter. Default: |
True
|
**adapter_kwargs
|
Any
|
Passed to |
{}
|
Returns:
| Type | Description |
|---|---|
Module
|
nn.Module: The same model. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If a name is not a module of the model. |
TypeError
|
If a named module is not |
ValueError
|
If nothing matches, or a named module is inside an adapter. |
fold_bn(conv, bn)
Fold an eval-mode BatchNorm into the convolution before it.
Numerical contract, all in float64 on the CPU:
scale = bn.weight / sqrt(bn.running_var + bn.eps),
w = conv.weight * scale.view(-1, 1, 1, 1) (one 1 per axis of the weight
after the first, so view(-1, 1, 1) for a Conv1d) and
b = bn.bias - scale * bn.running_mean. A convolution with a bias of its own
gives b = bn.bias - scale * (bn.running_mean - conv.bias) instead. A
BatchNorm with affine=False counts as weight = 1 and bias = 0. Cast the
results to the dtype you need.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conv
|
Conv1d | Conv2d | Conv3d
|
The convolution. |
required |
bn
|
BatchNorm1d | BatchNorm2d | BatchNorm3d
|
The BatchNorm
that follows it, in eval mode, with one feature per output channel
of |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
tuple[torch.Tensor, torch.Tensor]: The folded weight, of the shape of |
Tensor
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If the BatchNorm keeps no running statistics, or its number
of features is not the number of output channels of |
merge_adapters(model)
Replace every OrthogonalAdapter of a model by its merged layer, in place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
A model with adapters. If the model is itself an adapter, its merged layer is returned. |
required |
Returns:
| Type | Description |
|---|---|
Module
|
nn.Module: The model, with plain layers where the adapters were. |
trainable_parameters(model)
The parameters that require a gradient, for the optimiser.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
The model. |
required |
Yields:
| Type | Description |
|---|---|
Parameter
|
nn.Parameter: Each parameter with |