Skip to content

adapters

eigenkit.orthogonal.torch.adapters

Orthogonal fine-tuning: adapt a pretrained layer by rotating its weight.

An OrthogonalAdapter keeps the pretrained layer frozen and trains only the angles of an orthogonal matrix \(R\), applied as \(W' = RW\) (or \(W' = WR^\top\) on the inputs). \(R\) starts at the identity, so the adapted model starts at the pretrained function, and \(RW\) has exactly the singular values of \(W\) at every step: the Lipschitz constant of the layer never changes. merge() folds \(R\) into the weight, so the adapter costs nothing at inference.

Example
import torch.nn as nn

from eigenkit.orthogonal.torch import (
    apply_orthogonal_adapters,
    merge_adapters,
    trainable_parameters,
)

model = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 10))
apply_orthogonal_adapters(model, ["0", "2"], topology="pyramid")
params = list(trainable_parameters(model))  # the angles only
merge_adapters(model)  # plain nn.Linear layers again

OrthogonalAdapter

Bases: Module

A frozen nn.Linear or nn.Conv2d whose weight is rotated by a trainable orthogonal matrix.

side="out" gives \(W' = RW\), with \(R\) on the output channels; side="in" gives \(W' = WR^\top\), with \(R\) on the input channels. The bias of the base layer is kept as it is. The weight of a convolution is read as the matrix (out, in·k·k) (see rotate_weight).

With the butterfly layout and a width that is not a power of two, the rotation is a PartitionedOrthogonal when partition is None (the default) or True. Any other layout takes any width it accepts, and partition=True uses the partitioned form for it too. The angles take the dtype and the device of the base weight.

An adapter cannot be re-initialised (see reset_parameters), so a model with adapters is trained with a training loop of your own: the estimators GenericClassifier and GenericRegressor re-initialise every layer in fit() and refuse it.

Parameters:

Name Type Description Default
base Linear | Conv2d

The pretrained layer. Its parameters are frozen in place.

required
topology TopologyName | Topology

The layout of the rotation. Default: "butterfly".

'butterfly'
side Side

"out" or "in". Default: "out".

'out'
partition bool | None

Whether to use PartitionedOrthogonal; None chooses it for a butterfly whose width is not a power of two. Default: None.

None
stages int

The stages of a partitioned rotation. Default: 2.

2
perm_seed int

The permutation seed of a partitioned rotation. Default: 0.

0

bias property

The bias of the base layer, base.bias, which is not rotated.

Returns:

Type Description
Tensor | None

torch.Tensor | None: The bias, or None if the base layer has none.

weight property

The adapted weight rotate_weight(R, base.weight, side).

A property, like the weight of the layer it replaces, so code that reads layer.weight directly (nn.MultiheadAttention reads out_proj.weight) gets the adapted weight and trains the angles. It is rebuilt from the angles at every read.

Returns:

Type Description
Tensor

torch.Tensor: A tensor of the shape of base.weight.

forward(x)

Run the base layer with the adapted weight.

Parameters:

Name Type Description Default
x Tensor

The input of the base layer.

required

Returns:

Type Description
Tensor

torch.Tensor: The output of the base layer with weight self.weight.

merge()

A copy of the base layer with the rotation folded into its weight.

Its parameters are frozen, like those of the base layer.

Returns:

Type Description
Linear | Conv2d

nn.Linear | nn.Conv2d: A plain layer that computes what the adapter does.

reset_parameters()

Refuse to re-initialise: an adapter starts from its pretrained weight.

model.apply(lambda m: m.reset_parameters()), which the estimators GenericClassifier and GenericRegressor run at the start of fit(), resets the children of a module before the module itself, so by the time this method runs base already holds a new random weight. Training would then rotate that random weight instead of the pretrained one, without any error. Raising here stops it. Train an adapted model with a training loop of your own, as in the fine-tuning example.

Raises:

Type Description
RuntimeError

Always.

apply_orthogonal_adapters(model, target_modules, topology='butterfly', side='out', freeze_rest=True, **adapter_kwargs)

Replace layers of a model by OrthogonalAdapters, in place.

Parameters:

Name Type Description Default
model Module

The model to adapt.

required
target_modules str | Iterable[str] | Callable[[str, Module], bool]

The qualified name of the layer to adapt, or the names of several layers (as in named_modules()), or a predicate on the name and the module. Only nn.Linear and nn.Conv2d layers can be adapted.

required
topology TopologyName | Topology

The layout of every rotation. Default: "butterfly".

'butterfly'
side Side

"out" or "in". Default: "out".

'out'
freeze_rest bool

Whether to freeze every parameter that is not an angle of an adapter. Default: True.

True
**adapter_kwargs Any

Passed to OrthogonalAdapter, such as partition.

{}

Returns:

Type Description
Module

nn.Module: The same model.

Raises:

Type Description
KeyError

If a name is not a module of the model.

TypeError

If a named module is not nn.Linear or nn.Conv2d.

ValueError

If nothing matches, or a named module is inside an adapter.

fold_bn(conv, bn)

Fold an eval-mode BatchNorm into the convolution before it.

Numerical contract, all in float64 on the CPU: scale = bn.weight / sqrt(bn.running_var + bn.eps), w = conv.weight * scale.view(-1, 1, 1, 1) (one 1 per axis of the weight after the first, so view(-1, 1, 1) for a Conv1d) and b = bn.bias - scale * bn.running_mean. A convolution with a bias of its own gives b = bn.bias - scale * (bn.running_mean - conv.bias) instead. A BatchNorm with affine=False counts as weight = 1 and bias = 0. Cast the results to the dtype you need.

Parameters:

Name Type Description Default
conv Conv1d | Conv2d | Conv3d

The convolution.

required
bn BatchNorm1d | BatchNorm2d | BatchNorm3d

The BatchNorm that follows it, in eval mode, with one feature per output channel of conv.

required

Returns:

Type Description
Tensor

tuple[torch.Tensor, torch.Tensor]: The folded weight, of the shape of

Tensor

conv.weight, and bias, float64 on the CPU.

Raises:

Type Description
TypeError

If conv is not nn.Conv1d, nn.Conv2d or nn.Conv3d (a transposed convolution keeps its output channels on another axis).

ValueError

If the BatchNorm keeps no running statistics, or its number of features is not the number of output channels of conv.

merge_adapters(model)

Replace every OrthogonalAdapter of a model by its merged layer, in place.

Parameters:

Name Type Description Default
model Module

A model with adapters. If the model is itself an adapter, its merged layer is returned.

required

Returns:

Type Description
Module

nn.Module: The model, with plain layers where the adapters were.

trainable_parameters(model)

The parameters that require a gradient, for the optimiser.

Parameters:

Name Type Description Default
model Module

The model.

required

Yields:

Type Description
Parameter

nn.Parameter: Each parameter with requires_grad=True.