Skip to content

torch

eigenkit.orthogonal.torch

The PyTorch ring of eigenkit.orthogonal: trainable orthogonal modules and orthogonal fine-tuning adapters.

It needs the torch extra (pip install 'eigenkit[torch]'). The NumPy part of eigenkit.orthogonal loads this package only when one of its names is accessed.

MAX_MATERIALISE = 2 ** 31 module-attribute

The largest \(\mathrm{depth} \cdot n^2\) for which mode="materialise" builds the dense matrix.

Certificate

Bases: NamedTuple

The prediction, the top-1 margin and the certified radius of every row.

Attributes:

Name Type Description
pred Tensor

The predicted class of each row, long.

margin Tensor

The gap between the two largest logits, float64.

radius Tensor

The certified L2 radius, float64; inf when no other class can overtake the prediction.

LLNHead

Bases: Module

A linear head with unit-norm rows (last-layer normalisation).

The rows \(w_k\) of the weight are normalised at every call, so the certificate can use the distances \(\lVert w_p - w_j \rVert\) directly. The bias cancels in every logit difference, so the certificate does not see it.

Numerical contract: F.linear(h, F.normalize(self.linear.weight, dim=1), self.linear.bias). The parameters are linear.weight and linear.bias.

Parameters:

Name Type Description Default
in_features int

The number of input features.

required
out_features int

The number of classes.

required

forward(h)

Apply the head.

Parameters:

Name Type Description Default
h Tensor

Features (B, in_features).

required

Returns:

Type Description
Tensor

torch.Tensor: Logits (B, out_features).

get_weight()

The weight with unit-norm rows, F.normalize(self.linear.weight, dim=1).

Returns:

Type Description
Tensor

torch.Tensor: A (out_features, in_features) tensor.

MaxMin

Bases: Module

The MaxMin activation: sort the channels within disjoint pairs.

It is 1-Lipschitz and keeps the norm of its input, because it only permutes values. It is not distance-preserving: it maps (1, 0) and (0, 1) to the same point (Anil, Lucas & Grosse, ICML 2019). It acts on dim 1.

Two pairings are in use and give different functions:

  • "adjacent" (GroupSort) pairs channels \(2k\) and \(2k + 1\) and writes the minimum first. Numerical contract: g = x.reshape(b, c // 2, 2, *rest), lo = torch.minimum(g[:, :, 0], g[:, :, 1]), hi = torch.maximum(...), then torch.stack((lo, hi), dim=2).reshape(x.shape).
  • "halves" (BRONet's MinMax) pairs channel \(k\) with \(k + C/2\) and writes the maximum of each pair in the first half. Numerical contract: a, b = torch.split(x, c // 2, dim=1), index = a > b, then torch.cat([torch.where(index, a, b), torch.where(index, b, a)], dim=1).

Parameters:

Name Type Description Default
pairing Pairing

"adjacent" or "halves". Default: "adjacent".

'adjacent'

forward(x)

Apply the activation.

Parameters:

Name Type Description Default
x Tensor

A tensor (B, C, ...) with an even C.

required

Returns:

Type Description
Tensor

torch.Tensor: The output, of the shape of x.

Raises:

Type Description
ValueError

If C is odd.

Orthogonal

Bases: _Rotation

An \(n \times n\) orthogonal matrix parameterised by the angles of a network of RBS (Givens) rotations.

A Givens rotation is orthogonal for every angle, and a product of orthogonal matrices is orthogonal, so the matrix is orthogonal at every training step with no projection and no penalty. On quantum hardware each rotation is one RBS gate, so this is the classical simulation of a quantum orthogonal layer.

The state_dict holds only theta. The pair indices are buffers that are not saved, so .to(device) moves them and a checkpoint stays small. They are rebuilt from the topology whenever the module moves, so a module built on the meta device, then to_empty() and load_state_dict(), works.

Numerical contracts:

  • Initialisation: init="identity" sets theta = torch.zeros(n_angles) and draws no random numbers; init="random" sets theta = torch.randn(n_angles) * std, one call on the global generator, made before any other random draw of the module.
  • The matrix: starting from W = torch.eye(n) in the dtype and on the device of theta, each layer takes the next k angles, c = cos(theta[at:at+k]).unsqueeze(1), s = sin(...).unsqueeze(1), and sets W = W.index_copy(0, i, c*Wi + s*Wj).index_copy(0, j, -s*Wi + c*Wj) with Wi = W.index_select(0, i), Wj = W.index_select(0, j). With a reflected topology the last row is negated at the end.
  • Caching: the matrix is cached only when not self.training and not torch.is_grad_enabled(), under the key (theta._version, theta.device, theta.dtype), and it is reused only while theta still holds the values it was built from (torch.equal), so a write through theta.data cannot leave it stale. The cache decides only when the matrix is rebuilt, never its values. On the meta device nothing is cached. A frozen matrix (freeze_matrix) takes precedence.

mode="materialise" builds the dense matrix, at a cost of \(O(\mathrm{depth} \cdot n^2)\), and applies it as one matrix product. It is refused when \(\mathrm{depth} \cdot n^2\) exceeds MAX_MATERIALISE (\(2^{31}\)), which rules out a pyramid wider than 1024. mode="apply" applies the rotations to the input layer by layer, at a cost of \(O(\mathrm{depth} \cdot n \cdot B)\) for a batch of \(B\), and never builds the matrix in forward; it has no contract with the materialised path, whose last bits it does not reproduce.

Parameters:

Name Type Description Default
n int

The width.

required
topology TopologyName | Topology

A layout name, or a Topology on n wires. Default: "butterfly".

'butterfly'
init Init

"identity" for zero angles, "random" for normal angles of standard deviation std. Default: "identity".

'identity'
std float

The standard deviation of the random angles. Default: 0.1.

0.1
mode Mode

"materialise" or "apply". Default: "materialise".

'materialise'

n_angles property

The number of angles, one per RBS gate.

forward(x, dim=-1)

Rotate x along one axis: \(y = Rx\) on that axis.

In mode="materialise", or with a frozen matrix: R = self.matrix().to(x.dtype), then x @ R.t() for a 2-D x along its last axis and torch.einsum("oc,bc...->bo...", R, x) for an x of 3 or more dimensions along dim=1. In mode="apply" the rotations act on x directly.

Parameters:

Name Type Description Default
x Tensor

A 2-D batch (B, n), or a tensor (B, n, ...).

required
dim int

-1 (or 1) for a 2-D input, 1 for a larger one. Default: -1.

-1

Returns:

Type Description
Tensor

torch.Tensor: The rotated tensor, of the shape of x.

Raises:

Type Description
ValueError

For any other combination of shape and dim.

matrix()

The materialised \(n \times n\) orthogonal matrix (see the class contract).

Returns:

Type Description
Tensor

torch.Tensor: The matrix, in the dtype and on the device of theta.

reset_parameters()

Initialise the angles again, by the rule of init.

init="identity" sets them to zero and draws nothing; init="random" draws torch.randn(n_angles) * std from the global generator, as the constructor does, and copies it into theta in place. The estimators call this after seeding, so random_state controls the random angles, and an identity start stays the identity.

OrthogonalAdapter

Bases: Module

A frozen nn.Linear or nn.Conv2d whose weight is rotated by a trainable orthogonal matrix.

side="out" gives \(W' = RW\), with \(R\) on the output channels; side="in" gives \(W' = WR^\top\), with \(R\) on the input channels. The bias of the base layer is kept as it is. The weight of a convolution is read as the matrix (out, in·k·k) (see rotate_weight).

With the butterfly layout and a width that is not a power of two, the rotation is a PartitionedOrthogonal when partition is None (the default) or True. Any other layout takes any width it accepts, and partition=True uses the partitioned form for it too. The angles take the dtype and the device of the base weight.

An adapter cannot be re-initialised (see reset_parameters), so a model with adapters is trained with a training loop of your own: the estimators GenericClassifier and GenericRegressor re-initialise every layer in fit() and refuse it.

Parameters:

Name Type Description Default
base Linear | Conv2d

The pretrained layer. Its parameters are frozen in place.

required
topology TopologyName | Topology

The layout of the rotation. Default: "butterfly".

'butterfly'
side Side

"out" or "in". Default: "out".

'out'
partition bool | None

Whether to use PartitionedOrthogonal; None chooses it for a butterfly whose width is not a power of two. Default: None.

None
stages int

The stages of a partitioned rotation. Default: 2.

2
perm_seed int

The permutation seed of a partitioned rotation. Default: 0.

0

bias property

The bias of the base layer, base.bias, which is not rotated.

Returns:

Type Description
Tensor | None

torch.Tensor | None: The bias, or None if the base layer has none.

weight property

The adapted weight rotate_weight(R, base.weight, side).

A property, like the weight of the layer it replaces, so code that reads layer.weight directly (nn.MultiheadAttention reads out_proj.weight) gets the adapted weight and trains the angles. It is rebuilt from the angles at every read.

Returns:

Type Description
Tensor

torch.Tensor: A tensor of the shape of base.weight.

forward(x)

Run the base layer with the adapted weight.

Parameters:

Name Type Description Default
x Tensor

The input of the base layer.

required

Returns:

Type Description
Tensor

torch.Tensor: The output of the base layer with weight self.weight.

merge()

A copy of the base layer with the rotation folded into its weight.

Its parameters are frozen, like those of the base layer.

Returns:

Type Description
Linear | Conv2d

nn.Linear | nn.Conv2d: A plain layer that computes what the adapter does.

reset_parameters()

Refuse to re-initialise: an adapter starts from its pretrained weight.

model.apply(lambda m: m.reset_parameters()), which the estimators GenericClassifier and GenericRegressor run at the start of fit(), resets the children of a module before the module itself, so by the time this method runs base already holds a new random weight. Training would then rotate that random weight instead of the pretrained one, without any error. Raising here stops it. Train an adapted model with a training loop of your own, as in the fine-tuning example.

Raises:

Type Description
RuntimeError

Always.

OrthogonalConv1x1

Bases: Orthogonal

An orthogonal 1x1 convolution: Orthogonal applied to the channels of a feature map, plus an optional bias.

Numerical contract: forward(x) = F.conv2d(x, W.reshape(n, n, 1, 1), bias) with W = self.matrix().to(x.dtype) and bias = self.bias.to(x.dtype), as Orthogonal.forward casts its matrix to the dtype of the input. The bias parameter is created after theta, so the state_dict holds theta and bias.

Parameters:

Name Type Description Default
n int

The number of channels.

required
topology TopologyName | Topology

A layout name, or a Topology on n wires. Default: "butterfly".

'butterfly'
init Init

"identity" or "random". Default: "identity".

'identity'
std float

The standard deviation of the random angles. Default: 0.1.

0.1
bias bool

Whether to add a trainable bias, initialised to zero. Default: True.

True
mode Mode

"materialise" or "apply". Default: "materialise".

'materialise'

forward(x)

Apply the orthogonal 1x1 convolution.

Parameters:

Name Type Description Default
x Tensor

A feature map of shape (B, n, H, W), or (n, H, W) without a batch axis, as F.conv2d takes.

required

Returns:

Type Description
Tensor

torch.Tensor: The output, of the shape of x.

reset_parameters()

Initialise the angles again by the rule of init (see Orthogonal.reset_parameters), and the bias to zero, as the constructor does.

PartitionedOrthogonal

Bases: _Rotation

An orthogonal \(\mathrm{width} \times \mathrm{width}\) matrix for any width, from orthogonal blocks on the binary parts of the width, mixed by fixed permutations.

The butterfly needs a power-of-two width. This module splits the width into its binary parts (\(588 = 512 + 64 + 8 + 4\)), rotates every part of size 2 or more with its own Orthogonal, and repeats this over stages stages, \(S\) of them, with a fixed channel permutation \(P_s\) in between:

\[ R = (P_S^\top B_S P_S) \cdots (P_2^\top B_2 P_2) \, B_1, \qquad B_s = \operatorname{blockdiag}(\text{parts of stage } s) \]

Every factor is orthogonal, so \(R\) is too. All angles start at zero, so \(R = I\) at initialisation. A part of size 1 (an odd width) is left as it is.

Numerical contracts:

  • Construction: stages outer, parts inner, one Orthogonal(part, topology) with init="identity" per part of size 2 or more, so no random numbers are drawn; then, for s = 0 .. stages - 2, the buffers perm_s = mixing_permutation(width, perm_seed * 1000 + s) and its inverse inv_s = torch.argsort(perm_s), not saved in the state_dict and rebuilt the same way whenever the module moves (so to_empty() after a meta-device build works).
  • Stage matrix: torch.block_diag(*part_matrices), with torch.ones(1, 1) in the dtype and on the device of the other parts appended for a part of size 1.
  • The matrix: R = stage_0; then for s >= 1, R = B_s[inv][:, inv] @ R with inv = inv_{s-1}, the stored torch.argsort(perm_{s-1}).
  • forward(x, dim) follows the contract of Orthogonal.forward in mode="materialise".

The state_dict keys are blocks.<stage>.<part>.theta.

Parameters:

Name Type Description Default
width int

The width, at least 2.

required
stages int

The number of block-diagonal stages, at least 1. Default: 2.

2
perm_seed int

The seed of the mixing permutations. Default: 0.

0
topology TopologyName

The layout of every part. Default: "butterfly".

'butterfly'

n_angles property

The number of angles over every stage and part.

matrix()

The materialised \(\mathrm{width} \times \mathrm{width}\) orthogonal matrix (see the class contract).

Returns:

Type Description
Tensor

torch.Tensor: The matrix.

apply_orthogonal_adapters(model, target_modules, topology='butterfly', side='out', freeze_rest=True, **adapter_kwargs)

Replace layers of a model by OrthogonalAdapters, in place.

Parameters:

Name Type Description Default
model Module

The model to adapt.

required
target_modules str | Iterable[str] | Callable[[str, Module], bool]

The qualified name of the layer to adapt, or the names of several layers (as in named_modules()), or a predicate on the name and the module. Only nn.Linear and nn.Conv2d layers can be adapted.

required
topology TopologyName | Topology

The layout of every rotation. Default: "butterfly".

'butterfly'
side Side

"out" or "in". Default: "out".

'out'
freeze_rest bool

Whether to freeze every parameter that is not an angle of an adapter. Default: True.

True
**adapter_kwargs Any

Passed to OrthogonalAdapter, such as partition.

{}

Returns:

Type Description
Module

nn.Module: The same model.

Raises:

Type Description
KeyError

If a name is not a module of the model.

TypeError

If a named module is not nn.Linear or nn.Conv2d.

ValueError

If nothing matches, or a named module is inside an adapter.

binary_parts(width)

The powers of two that sum to width, largest first: 588 gives [512, 64, 8, 4].

Parameters:

Name Type Description Default
width int

A positive integer.

required

Returns:

Type Description
list[int]

list[int]: The binary digits of width, as powers of two.

Raises:

Type Description
ValueError

If width is not a positive integer.

cert_ce(logits, y, head_weight, L, eps, tau=4.0)

A certification-aware cross-entropy.

Every wrong logit is raised by the most a perturbation of norm eps could raise it against the true class, \(L \cdot \mathrm{eps} \cdot \lVert w_y - w_j \rVert\), and the result is scored with a cross-entropy at temperature 1/tau. Minimising it pushes the margins beyond eps.

Numerical contract: d = (head_weight[y].unsqueeze(1) - head_weight.unsqueeze(0)).norm(dim=-1), z = logits + L * eps * d, then F.cross_entropy(tau * z, y). The row of the true class has d = 0.

Parameters:

Name Type Description Default
logits Tensor

Logits (B, K).

required
y Tensor

The labels (B,).

required
head_weight Tensor

The head weight (K, D), in the graph if the head trains.

required
L float

The Lipschitz constant of the feature map.

required
eps float

The radius to train for.

required
tau float

The scale of the logits. Default: 4.0.

4.0

Returns:

Type Description
Tensor

torch.Tensor: The loss, a scalar.

certified_accuracy(correct, radius, eps)

The fraction of rows that are correct and certified beyond eps.

Numerical contract: float(correct.float().mean()) for eps == 0, which is the plain accuracy, and float((correct & (radius > eps)).float().mean()) otherwise.

Parameters:

Name Type Description Default
correct Tensor

A boolean tensor, pred == labels.

required
radius Tensor

The certified radii, from certified_radius.

required
eps float

The L2 radius to certify.

required

Returns:

Name Type Description
float float

The certified accuracy at eps.

certified_radius(logits, head_weight, L)

The certified L2 radius of every row of logits, for a feature map with Lipschitz constant L and a linear head with weight head_weight.

Numerical contract, in float64 on the CPU: z = logits.double(), d = head_distances(head_weight), pred = z.argmax(1), gap = z.gather(1, pred[:, None]) - z, dist = d[pred], r = gap / (L * dist.clamp_min(1e-300)); then r is set to inf on the predicted class and wherever dist == 0 (a class with the same row can never overtake the prediction), and the radius is r.min(1).values. The margin is the difference of the two largest logits, from z.topk(2, dim=1).

Parameters:

Name Type Description Default
logits Tensor

Logits (B, K), with K >= 2.

required
head_weight Tensor

The head weight (K, D), such as LLNHead.get_weight().

required
L float

The Lipschitz constant of the feature map in front of the head.

required

Returns:

Name Type Description
Certificate Certificate

The prediction, the margin and the radius of every row.

fold_bn(conv, bn)

Fold an eval-mode BatchNorm into the convolution before it.

Numerical contract, all in float64 on the CPU: scale = bn.weight / sqrt(bn.running_var + bn.eps), w = conv.weight * scale.view(-1, 1, 1, 1) (one 1 per axis of the weight after the first, so view(-1, 1, 1) for a Conv1d) and b = bn.bias - scale * bn.running_mean. A convolution with a bias of its own gives b = bn.bias - scale * (bn.running_mean - conv.bias) instead. A BatchNorm with affine=False counts as weight = 1 and bias = 0. Cast the results to the dtype you need.

Parameters:

Name Type Description Default
conv Conv1d | Conv2d | Conv3d

The convolution.

required
bn BatchNorm1d | BatchNorm2d | BatchNorm3d

The BatchNorm that follows it, in eval mode, with one feature per output channel of conv.

required

Returns:

Type Description
Tensor

tuple[torch.Tensor, torch.Tensor]: The folded weight, of the shape of

Tensor

conv.weight, and bias, float64 on the CPU.

Raises:

Type Description
TypeError

If conv is not nn.Conv1d, nn.Conv2d or nn.Conv3d (a transposed convolution keeps its output channels on another axis).

ValueError

If the BatchNorm keeps no running statistics, or its number of features is not the number of output channels of conv.

head_distances(head_weight)

The distances \(\lVert w_i - w_j \rVert\) between the rows of a head weight, in float64.

Numerical contract: w = head_weight.detach().double(), then torch.cdist(w, w, compute_mode="donot_use_mm_for_euclid_dist"). The direct differences keep small distances exact; the default mode switches to a matrix product above 25 rows, which can shrink a small distance and so overstate a certified radius.

Parameters:

Name Type Description Default
head_weight Tensor

The head weight (K, D).

required

Returns:

Type Description
Tensor

torch.Tensor: The (K, K) matrix of distances.

lipschitz_bound(model)

An upper bound on the L2 Lipschitz constant of a chain of layers.

The bound is the product of a bound for every layer of an nn.Sequential (nested ones are flattened): the largest singular value of the weight of an nn.Linear, an LLNHead (its normalised weight) or an OrthogonalAdapter (its rotated weight); for an nn.Conv2d, that of the weight read as a matrix (out, in·kh·kw), times \(\sqrt{k_h k_w}\), which bounds the convolution for any stride and dilation with "zeros" padding, and with "circular" padding no wider than dilation * (kernel_size - 1) / 2 on each axis (wider circular padding, "reflect" and "replicate" feed a pixel to one kernel offset more than once and raise a ValueError); 1 for the orthogonal modules and for MaxMin, ReLU, LeakyReLU (slope at most 1), Tanh, Sigmoid, Flatten and Identity. The singular values are computed with sigma_max in float64, on the weight matrix or its transpose, whichever has fewer rows. A 1x1 convolution and a linear layer are bounded exactly.

An orthogonal adapter keeps every singular value of the weight it rotates, so adapting a chain leaves its bound unchanged.

Parameters:

Name Type Description Default
model Module

An nn.Sequential, or a single layer.

required

Returns:

Name Type Description
float float

The bound.

Raises:

Type Description
ValueError

If a layer is not one of the above, or a convolution pads with "reflect", "replicate", or "circular" wider than dilation * (kernel_size - 1) / 2.

merge_adapters(model)

Replace every OrthogonalAdapter of a model by its merged layer, in place.

Parameters:

Name Type Description Default
model Module

A model with adapters. If the model is itself an adapter, its merged layer is returned.

required

Returns:

Type Description
Module

nn.Module: The model, with plain layers where the adapters were.

mixing_permutation(width, random_state)

A fixed random permutation of width channels, from its own CPU generator.

Numerical contract: torch.randperm(width, generator=torch.Generator().manual_seed(random_state)). The global generator is not touched.

Parameters:

Name Type Description Default
width int

The number of channels.

required
random_state int

The seed of the generator.

required

Returns:

Type Description
Tensor

torch.Tensor: A permutation of 0 .. width - 1, as a long tensor.

rotate_weight(R, W, side='out')

Rotate the output or the input channels of a weight tensor by an orthogonal matrix.

The weight is read as the matrix W.reshape(out, in·k·k). Rotating its rows (side="out", \(RW\)) or its input channels (side="in", \(W (R^\top \otimes I)\)) keeps every singular value of that matrix, so the operator norm, and the Lipschitz constant, of the layer are unchanged. For a convolution R acts pixel by pixel, so the same holds for the convolution operator.

Numerical contract for side="out": (R @ W.reshape(W.shape[0], -1)).reshape(W.shape), which is R @ W for a 2-D weight. side="in" computes torch.einsum("oi...,ji->oj...", W, R).contiguous(), which is W @ R.T for a 2-D weight.

Parameters:

Name Type Description Default
R Tensor

An orthogonal matrix, (out, out) for side="out" or (in, in) for side="in".

required
W Tensor

A weight (out, in) or (out, in, *kernel).

required
side Side

"out" or "in". Default: "out".

'out'

Returns:

Type Description
Tensor

torch.Tensor: The rotated weight, of the shape of W.

Raises:

Type Description
ValueError

If side is unknown or R does not match the axis.

sigma_max(w)

The largest singular value of a matrix, through its Gram matrix.

Exact and differentiable. Numerical contract: torch.linalg.eigvalsh(w @ w.t()).amax().clamp_min(0).sqrt().

Parameters:

Name Type Description Default
w Tensor

A matrix (m, n).

required

Returns:

Type Description
Tensor

torch.Tensor: A scalar tensor.

trainable_parameters(model)

The parameters that require a gradient, for the optimiser.

Parameters:

Name Type Description Default
model Module

The model.

required

Yields:

Type Description
Parameter

nn.Parameter: Each parameter with requires_grad=True.

unary_loader_angles(x, layout='diagonal')

The input angles of UnaryLoader that load \(x / \lVert x \rVert\), differentiable in x.

The PyTorch form of eigenkit.orthogonal.unary_loader_angles: the same gate plan, with torch.atan2 on the signed component or the norm of the wires each gate keeps and passes on. Each row is first scaled by a power of two, which is exact, so the angles do not depend on the scale of x, at any finite scale.

Parameters:

Name Type Description Default
x Tensor

One vector (n,) or a batch (B, n).

required
layout LoaderLayout

The loader layout. Default: "diagonal".

'diagonal'

Returns:

Type Description
Tensor

torch.Tensor: The angles, (n - 1,) or (B, n - 1), in the dtype of x

Tensor

(float64 for an integer x).

Raises:

Type Description
ValueError

If x is not 1-D or 2-D, a row is zero or not finite, or the layout does not accept the width.