torch
eigenkit.orthogonal.torch
The PyTorch ring of eigenkit.orthogonal: trainable orthogonal modules and
orthogonal fine-tuning adapters.
It needs the torch extra (pip install 'eigenkit[torch]'). The NumPy part of
eigenkit.orthogonal loads this package only when one of its names is accessed.
MAX_MATERIALISE = 2 ** 31
module-attribute
The largest \(\mathrm{depth} \cdot n^2\) for which mode="materialise" builds
the dense matrix.
Certificate
Bases: NamedTuple
The prediction, the top-1 margin and the certified radius of every row.
Attributes:
| Name | Type | Description |
|---|---|---|
pred |
Tensor
|
The predicted class of each row, |
margin |
Tensor
|
The gap between the two largest logits, float64. |
radius |
Tensor
|
The certified L2 radius, float64; |
LLNHead
Bases: Module
A linear head with unit-norm rows (last-layer normalisation).
The rows \(w_k\) of the weight are normalised at every call, so the certificate can use the distances \(\lVert w_p - w_j \rVert\) directly. The bias cancels in every logit difference, so the certificate does not see it.
Numerical contract:
F.linear(h, F.normalize(self.linear.weight, dim=1), self.linear.bias). The
parameters are linear.weight and linear.bias.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
in_features
|
int
|
The number of input features. |
required |
out_features
|
int
|
The number of classes. |
required |
forward(h)
Apply the head.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
h
|
Tensor
|
Features |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: Logits |
get_weight()
The weight with unit-norm rows, F.normalize(self.linear.weight, dim=1).
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: A |
MaxMin
Bases: Module
The MaxMin activation: sort the channels within disjoint pairs.
It is 1-Lipschitz and keeps the norm of its input, because it only permutes
values. It is not distance-preserving: it maps (1, 0) and (0, 1) to the
same point (Anil, Lucas & Grosse, ICML 2019). It acts on dim 1.
Two pairings are in use and give different functions:
"adjacent"(GroupSort) pairs channels \(2k\) and \(2k + 1\) and writes the minimum first. Numerical contract:g = x.reshape(b, c // 2, 2, *rest),lo = torch.minimum(g[:, :, 0], g[:, :, 1]),hi = torch.maximum(...), thentorch.stack((lo, hi), dim=2).reshape(x.shape)."halves"(BRONet's MinMax) pairs channel \(k\) with \(k + C/2\) and writes the maximum of each pair in the first half. Numerical contract:a, b = torch.split(x, c // 2, dim=1),index = a > b, thentorch.cat([torch.where(index, a, b), torch.where(index, b, a)], dim=1).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pairing
|
Pairing
|
|
'adjacent'
|
forward(x)
Apply the activation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
A tensor |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The output, of the shape of |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Orthogonal
Bases: _Rotation
An \(n \times n\) orthogonal matrix parameterised by the angles of a network of RBS (Givens) rotations.
A Givens rotation is orthogonal for every angle, and a product of orthogonal matrices is orthogonal, so the matrix is orthogonal at every training step with no projection and no penalty. On quantum hardware each rotation is one RBS gate, so this is the classical simulation of a quantum orthogonal layer.
The state_dict holds only theta. The pair indices are buffers that are not
saved, so .to(device) moves them and a checkpoint stays small. They are
rebuilt from the topology whenever the module moves, so a module built on the
meta device, then to_empty() and load_state_dict(), works.
Numerical contracts:
- Initialisation:
init="identity"setstheta = torch.zeros(n_angles)and draws no random numbers;init="random"setstheta = torch.randn(n_angles) * std, one call on the global generator, made before any other random draw of the module. - The matrix: starting from
W = torch.eye(n)in the dtype and on the device oftheta, each layer takes the nextkangles,c = cos(theta[at:at+k]).unsqueeze(1),s = sin(...).unsqueeze(1), and setsW = W.index_copy(0, i, c*Wi + s*Wj).index_copy(0, j, -s*Wi + c*Wj)withWi = W.index_select(0, i),Wj = W.index_select(0, j). With a reflected topology the last row is negated at the end. - Caching: the matrix is cached only when
not self.training and not torch.is_grad_enabled(), under the key(theta._version, theta.device, theta.dtype), and it is reused only whilethetastill holds the values it was built from (torch.equal), so a write throughtheta.datacannot leave it stale. The cache decides only when the matrix is rebuilt, never its values. On themetadevice nothing is cached. A frozen matrix (freeze_matrix) takes precedence.
mode="materialise" builds the dense matrix, at a cost of
\(O(\mathrm{depth} \cdot n^2)\), and applies it as one matrix product. It is
refused when \(\mathrm{depth} \cdot n^2\) exceeds MAX_MATERIALISE
(\(2^{31}\)), which rules out a pyramid wider than 1024. mode="apply"
applies the rotations to the input layer by layer, at a cost of
\(O(\mathrm{depth} \cdot n \cdot B)\) for a batch of \(B\), and never builds the
matrix in forward; it has no contract with the materialised path, whose last
bits it does not reproduce.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n
|
int
|
The width. |
required |
topology
|
TopologyName | Topology
|
A layout name, or a |
'butterfly'
|
init
|
Init
|
|
'identity'
|
std
|
float
|
The standard deviation of the random angles. Default: |
0.1
|
mode
|
Mode
|
|
'materialise'
|
n_angles
property
The number of angles, one per RBS gate.
forward(x, dim=-1)
Rotate x along one axis: \(y = Rx\) on that axis.
In mode="materialise", or with a frozen matrix: R =
self.matrix().to(x.dtype), then x @ R.t() for a 2-D x along its last
axis and torch.einsum("oc,bc...->bo...", R, x) for an x of 3 or more
dimensions along dim=1. In mode="apply" the rotations act on x
directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
A 2-D batch |
required |
dim
|
int
|
|
-1
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The rotated tensor, of the shape of |
Raises:
| Type | Description |
|---|---|
ValueError
|
For any other combination of shape and |
matrix()
The materialised \(n \times n\) orthogonal matrix (see the class contract).
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The matrix, in the dtype and on the device of |
reset_parameters()
Initialise the angles again, by the rule of init.
init="identity" sets them to zero and draws nothing; init="random"
draws torch.randn(n_angles) * std from the global generator, as the
constructor does, and copies it into theta in place. The estimators call
this after seeding, so random_state controls the random angles, and an
identity start stays the identity.
OrthogonalAdapter
Bases: Module
A frozen nn.Linear or nn.Conv2d whose weight is rotated by a trainable
orthogonal matrix.
side="out" gives \(W' = RW\), with \(R\) on the output channels;
side="in" gives \(W' = WR^\top\), with \(R\) on the input channels. The bias of the
base layer is kept as it is. The weight of a convolution is read as the matrix
(out, in·k·k) (see rotate_weight).
With the butterfly layout and a width that is not a power of two, the rotation
is a PartitionedOrthogonal when partition is None (the default) or True.
Any other layout takes any width it accepts, and partition=True uses the
partitioned form for it too. The angles take the dtype and the device of the
base weight.
An adapter cannot be re-initialised (see reset_parameters), so a model with
adapters is trained with a training loop of your own: the estimators
GenericClassifier and GenericRegressor re-initialise every layer in
fit() and refuse it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base
|
Linear | Conv2d
|
The pretrained layer. Its parameters are frozen in place. |
required |
topology
|
TopologyName | Topology
|
The layout of the rotation.
Default: |
'butterfly'
|
side
|
Side
|
|
'out'
|
partition
|
bool | None
|
Whether to use |
None
|
stages
|
int
|
The stages of a partitioned rotation. Default: |
2
|
perm_seed
|
int
|
The permutation seed of a partitioned rotation.
Default: |
0
|
bias
property
The bias of the base layer, base.bias, which is not rotated.
Returns:
| Type | Description |
|---|---|
Tensor | None
|
torch.Tensor | None: The bias, or |
weight
property
The adapted weight rotate_weight(R, base.weight, side).
A property, like the weight of the layer it replaces, so code that reads
layer.weight directly (nn.MultiheadAttention reads out_proj.weight)
gets the adapted weight and trains the angles. It is rebuilt from the
angles at every read.
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: A tensor of the shape of |
forward(x)
Run the base layer with the adapted weight.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
The input of the base layer. |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The output of the base layer with weight |
merge()
A copy of the base layer with the rotation folded into its weight.
Its parameters are frozen, like those of the base layer.
Returns:
| Type | Description |
|---|---|
Linear | Conv2d
|
nn.Linear | nn.Conv2d: A plain layer that computes what the adapter does. |
reset_parameters()
Refuse to re-initialise: an adapter starts from its pretrained weight.
model.apply(lambda m: m.reset_parameters()), which the estimators
GenericClassifier and GenericRegressor run at the start of fit(),
resets the children of a module before the module itself, so by the time
this method runs base already holds a new random weight. Training would
then rotate that random weight instead of the pretrained one, without any
error. Raising here stops it. Train an adapted model with a training loop
of your own, as in the fine-tuning example.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
Always. |
OrthogonalConv1x1
Bases: Orthogonal
An orthogonal 1x1 convolution: Orthogonal applied to the channels of a
feature map, plus an optional bias.
Numerical contract: forward(x) = F.conv2d(x, W.reshape(n, n, 1, 1), bias)
with W = self.matrix().to(x.dtype) and bias = self.bias.to(x.dtype), as
Orthogonal.forward casts its matrix to the dtype of the input. The bias
parameter is created after theta, so the state_dict holds theta and
bias.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n
|
int
|
The number of channels. |
required |
topology
|
TopologyName | Topology
|
A layout name, or a |
'butterfly'
|
init
|
Init
|
|
'identity'
|
std
|
float
|
The standard deviation of the random angles. Default: |
0.1
|
bias
|
bool
|
Whether to add a trainable bias, initialised to zero.
Default: |
True
|
mode
|
Mode
|
|
'materialise'
|
forward(x)
Apply the orthogonal 1x1 convolution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
A feature map of shape |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The output, of the shape of |
reset_parameters()
Initialise the angles again by the rule of init (see
Orthogonal.reset_parameters), and the bias to zero, as the constructor
does.
PartitionedOrthogonal
Bases: _Rotation
An orthogonal \(\mathrm{width} \times \mathrm{width}\) matrix for any width, from orthogonal blocks on the binary parts of the width, mixed by fixed permutations.
The butterfly needs a power-of-two width. This module splits the width into its
binary parts (\(588 = 512 + 64 + 8 + 4\)), rotates every part of size 2 or more
with its own Orthogonal, and repeats this over stages stages, \(S\) of them,
with a fixed channel permutation \(P_s\) in between:
Every factor is orthogonal, so \(R\) is too. All angles start at zero, so \(R = I\) at initialisation. A part of size 1 (an odd width) is left as it is.
Numerical contracts:
- Construction: stages outer, parts inner, one
Orthogonal(part, topology)withinit="identity"per part of size 2 or more, so no random numbers are drawn; then, fors = 0 .. stages - 2, the buffersperm_s = mixing_permutation(width, perm_seed * 1000 + s)and its inverseinv_s = torch.argsort(perm_s), not saved in thestate_dictand rebuilt the same way whenever the module moves (soto_empty()after ameta-device build works). - Stage matrix:
torch.block_diag(*part_matrices), withtorch.ones(1, 1)in the dtype and on the device of the other parts appended for a part of size 1. - The matrix:
R = stage_0; then fors >= 1,R = B_s[inv][:, inv] @ Rwithinv = inv_{s-1}, the storedtorch.argsort(perm_{s-1}). forward(x, dim)follows the contract ofOrthogonal.forwardinmode="materialise".
The state_dict keys are blocks.<stage>.<part>.theta.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
The width, at least 2. |
required |
stages
|
int
|
The number of block-diagonal stages, at least 1. Default: |
2
|
perm_seed
|
int
|
The seed of the mixing permutations. Default: |
0
|
topology
|
TopologyName
|
The layout of every part. Default: |
'butterfly'
|
n_angles
property
The number of angles over every stage and part.
matrix()
The materialised \(\mathrm{width} \times \mathrm{width}\) orthogonal matrix (see the class contract).
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The matrix. |
apply_orthogonal_adapters(model, target_modules, topology='butterfly', side='out', freeze_rest=True, **adapter_kwargs)
Replace layers of a model by OrthogonalAdapters, in place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
The model to adapt. |
required |
target_modules
|
str | Iterable[str] | Callable[[str, Module], bool]
|
The qualified name of the layer to adapt, or the names of several
layers (as in |
required |
topology
|
TopologyName | Topology
|
The layout of every rotation.
Default: |
'butterfly'
|
side
|
Side
|
|
'out'
|
freeze_rest
|
bool
|
Whether to freeze every parameter that is not an angle
of an adapter. Default: |
True
|
**adapter_kwargs
|
Any
|
Passed to |
{}
|
Returns:
| Type | Description |
|---|---|
Module
|
nn.Module: The same model. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If a name is not a module of the model. |
TypeError
|
If a named module is not |
ValueError
|
If nothing matches, or a named module is inside an adapter. |
binary_parts(width)
The powers of two that sum to width, largest first: 588 gives
[512, 64, 8, 4].
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
A positive integer. |
required |
Returns:
| Type | Description |
|---|---|
list[int]
|
list[int]: The binary digits of |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
cert_ce(logits, y, head_weight, L, eps, tau=4.0)
A certification-aware cross-entropy.
Every wrong logit is raised by the most a perturbation of norm eps could
raise it against the true class,
\(L \cdot \mathrm{eps} \cdot \lVert w_y - w_j \rVert\), and the result is scored
with a cross-entropy at temperature 1/tau. Minimising it pushes the margins
beyond eps.
Numerical contract: d = (head_weight[y].unsqueeze(1) -
head_weight.unsqueeze(0)).norm(dim=-1), z = logits + L * eps * d, then
F.cross_entropy(tau * z, y). The row of the true class has d = 0.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
logits
|
Tensor
|
Logits |
required |
y
|
Tensor
|
The labels |
required |
head_weight
|
Tensor
|
The head weight |
required |
L
|
float
|
The Lipschitz constant of the feature map. |
required |
eps
|
float
|
The radius to train for. |
required |
tau
|
float
|
The scale of the logits. Default: |
4.0
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The loss, a scalar. |
certified_accuracy(correct, radius, eps)
The fraction of rows that are correct and certified beyond eps.
Numerical contract: float(correct.float().mean()) for eps == 0, which is
the plain accuracy, and float((correct & (radius > eps)).float().mean())
otherwise.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
correct
|
Tensor
|
A boolean tensor, |
required |
radius
|
Tensor
|
The certified radii, from |
required |
eps
|
float
|
The L2 radius to certify. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
float |
float
|
The certified accuracy at |
certified_radius(logits, head_weight, L)
The certified L2 radius of every row of logits, for a feature map with
Lipschitz constant L and a linear head with weight head_weight.
Numerical contract, in float64 on the CPU: z = logits.double(),
d = head_distances(head_weight), pred = z.argmax(1),
gap = z.gather(1, pred[:, None]) - z, dist = d[pred],
r = gap / (L * dist.clamp_min(1e-300)); then r is set to inf on the
predicted class and wherever dist == 0 (a class with the same row can never
overtake the prediction), and the radius is r.min(1).values. The margin is
the difference of the two largest logits, from z.topk(2, dim=1).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
logits
|
Tensor
|
Logits |
required |
head_weight
|
Tensor
|
The head weight |
required |
L
|
float
|
The Lipschitz constant of the feature map in front of the head. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Certificate |
Certificate
|
The prediction, the margin and the radius of every row. |
fold_bn(conv, bn)
Fold an eval-mode BatchNorm into the convolution before it.
Numerical contract, all in float64 on the CPU:
scale = bn.weight / sqrt(bn.running_var + bn.eps),
w = conv.weight * scale.view(-1, 1, 1, 1) (one 1 per axis of the weight
after the first, so view(-1, 1, 1) for a Conv1d) and
b = bn.bias - scale * bn.running_mean. A convolution with a bias of its own
gives b = bn.bias - scale * (bn.running_mean - conv.bias) instead. A
BatchNorm with affine=False counts as weight = 1 and bias = 0. Cast the
results to the dtype you need.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conv
|
Conv1d | Conv2d | Conv3d
|
The convolution. |
required |
bn
|
BatchNorm1d | BatchNorm2d | BatchNorm3d
|
The BatchNorm
that follows it, in eval mode, with one feature per output channel
of |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
tuple[torch.Tensor, torch.Tensor]: The folded weight, of the shape of |
Tensor
|
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If the BatchNorm keeps no running statistics, or its number
of features is not the number of output channels of |
head_distances(head_weight)
The distances \(\lVert w_i - w_j \rVert\) between the rows of a head weight, in float64.
Numerical contract: w = head_weight.detach().double(), then
torch.cdist(w, w, compute_mode="donot_use_mm_for_euclid_dist"). The direct
differences keep small distances exact; the default mode switches to a matrix
product above 25 rows, which can shrink a small distance and so overstate a
certified radius.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
head_weight
|
Tensor
|
The head weight |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The |
lipschitz_bound(model)
An upper bound on the L2 Lipschitz constant of a chain of layers.
The bound is the product of a bound for every layer of an nn.Sequential
(nested ones are flattened): the largest singular value of the weight of an
nn.Linear, an LLNHead (its normalised weight) or an OrthogonalAdapter
(its rotated weight); for an nn.Conv2d, that of the weight read as a matrix
(out, in·kh·kw), times \(\sqrt{k_h k_w}\), which bounds the convolution for
any stride and dilation with "zeros" padding, and with "circular" padding
no wider than dilation * (kernel_size - 1) / 2 on each axis (wider circular
padding, "reflect" and "replicate" feed a pixel to one kernel offset more
than once and raise a ValueError); 1 for the
orthogonal modules and for MaxMin, ReLU, LeakyReLU (slope at most 1),
Tanh, Sigmoid, Flatten and Identity. The singular values are computed
with sigma_max in float64, on the weight matrix or its transpose, whichever
has fewer rows. A 1x1 convolution and a linear layer are bounded
exactly.
An orthogonal adapter keeps every singular value of the weight it rotates, so adapting a chain leaves its bound unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
An |
required |
Returns:
| Name | Type | Description |
|---|---|---|
float |
float
|
The bound. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a layer is not one of the above, or a convolution pads with
|
merge_adapters(model)
Replace every OrthogonalAdapter of a model by its merged layer, in place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
A model with adapters. If the model is itself an adapter, its merged layer is returned. |
required |
Returns:
| Type | Description |
|---|---|
Module
|
nn.Module: The model, with plain layers where the adapters were. |
mixing_permutation(width, random_state)
A fixed random permutation of width channels, from its own CPU generator.
Numerical contract:
torch.randperm(width, generator=torch.Generator().manual_seed(random_state)).
The global generator is not touched.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
The number of channels. |
required |
random_state
|
int
|
The seed of the generator. |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: A permutation of |
rotate_weight(R, W, side='out')
Rotate the output or the input channels of a weight tensor by an orthogonal matrix.
The weight is read as the matrix W.reshape(out, in·k·k). Rotating its rows
(side="out", \(RW\)) or its input channels (side="in",
\(W (R^\top \otimes I)\)) keeps
every singular value of that matrix, so the operator norm, and the Lipschitz
constant, of the layer are unchanged. For a convolution R acts pixel by
pixel, so the same holds for the convolution operator.
Numerical contract for side="out":
(R @ W.reshape(W.shape[0], -1)).reshape(W.shape), which is R @ W for a 2-D
weight. side="in" computes
torch.einsum("oi...,ji->oj...", W, R).contiguous(), which is W @ R.T for a
2-D weight.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
R
|
Tensor
|
An orthogonal matrix, |
required |
W
|
Tensor
|
A weight |
required |
side
|
Side
|
|
'out'
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The rotated weight, of the shape of |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
sigma_max(w)
The largest singular value of a matrix, through its Gram matrix.
Exact and differentiable. Numerical contract:
torch.linalg.eigvalsh(w @ w.t()).amax().clamp_min(0).sqrt().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
w
|
Tensor
|
A matrix |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: A scalar tensor. |
trainable_parameters(model)
The parameters that require a gradient, for the optimiser.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Module
|
The model. |
required |
Yields:
| Type | Description |
|---|---|
Parameter
|
nn.Parameter: Each parameter with |
unary_loader_angles(x, layout='diagonal')
The input angles of UnaryLoader that load \(x / \lVert x \rVert\),
differentiable in x.
The PyTorch form of eigenkit.orthogonal.unary_loader_angles: the same gate
plan, with torch.atan2 on the signed component or the norm of the wires each
gate keeps and passes on. Each row is first scaled by a power of two, which
is exact, so the angles do not depend on the scale of x, at any finite
scale.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
One vector |
required |
layout
|
LoaderLayout
|
The loader layout. Default: |
'diagonal'
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The angles, |
Tensor
|
(float64 for an integer |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |