Skip to content

modules

eigenkit.orthogonal.torch.modules

PyTorch modules for orthogonal networks: the angles are an nn.Parameter, and the orthogonal matrix is rebuilt from them, so it stays orthogonal at every training step.

Numerical contracts. Each module documents the exact sequence of torch operations it runs, and the tests pin it bit for bit. Changing one of them is a breaking change, because it changes the last bits of every result and so the trajectory of any training run.

Example
import torch

from eigenkit.orthogonal.torch import Orthogonal

rotation = Orthogonal(8, "butterfly", init="random", std=0.5)
x = torch.randn(4, 8)
y = rotation(x)  # x @ R.T
torch.allclose(y.norm(dim=1), x.norm(dim=1))  # True

MAX_MATERIALISE = 2 ** 31 module-attribute

The largest \(\mathrm{depth} \cdot n^2\) for which mode="materialise" builds the dense matrix.

Orthogonal

Bases: _Rotation

An \(n \times n\) orthogonal matrix parameterised by the angles of a network of RBS (Givens) rotations.

A Givens rotation is orthogonal for every angle, and a product of orthogonal matrices is orthogonal, so the matrix is orthogonal at every training step with no projection and no penalty. On quantum hardware each rotation is one RBS gate, so this is the classical simulation of a quantum orthogonal layer.

The state_dict holds only theta. The pair indices are buffers that are not saved, so .to(device) moves them and a checkpoint stays small. They are rebuilt from the topology whenever the module moves, so a module built on the meta device, then to_empty() and load_state_dict(), works.

Numerical contracts:

  • Initialisation: init="identity" sets theta = torch.zeros(n_angles) and draws no random numbers; init="random" sets theta = torch.randn(n_angles) * std, one call on the global generator, made before any other random draw of the module.
  • The matrix: starting from W = torch.eye(n) in the dtype and on the device of theta, each layer takes the next k angles, c = cos(theta[at:at+k]).unsqueeze(1), s = sin(...).unsqueeze(1), and sets W = W.index_copy(0, i, c*Wi + s*Wj).index_copy(0, j, -s*Wi + c*Wj) with Wi = W.index_select(0, i), Wj = W.index_select(0, j). With a reflected topology the last row is negated at the end.
  • Caching: the matrix is cached only when not self.training and not torch.is_grad_enabled(), under the key (theta._version, theta.device, theta.dtype), and it is reused only while theta still holds the values it was built from (torch.equal), so a write through theta.data cannot leave it stale. The cache decides only when the matrix is rebuilt, never its values. On the meta device nothing is cached. A frozen matrix (freeze_matrix) takes precedence.

mode="materialise" builds the dense matrix, at a cost of \(O(\mathrm{depth} \cdot n^2)\), and applies it as one matrix product. It is refused when \(\mathrm{depth} \cdot n^2\) exceeds MAX_MATERIALISE (\(2^{31}\)), which rules out a pyramid wider than 1024. mode="apply" applies the rotations to the input layer by layer, at a cost of \(O(\mathrm{depth} \cdot n \cdot B)\) for a batch of \(B\), and never builds the matrix in forward; it has no contract with the materialised path, whose last bits it does not reproduce.

Parameters:

Name Type Description Default
n int

The width.

required
topology TopologyName | Topology

A layout name, or a Topology on n wires. Default: "butterfly".

'butterfly'
init Init

"identity" for zero angles, "random" for normal angles of standard deviation std. Default: "identity".

'identity'
std float

The standard deviation of the random angles. Default: 0.1.

0.1
mode Mode

"materialise" or "apply". Default: "materialise".

'materialise'

n_angles property

The number of angles, one per RBS gate.

forward(x, dim=-1)

Rotate x along one axis: \(y = Rx\) on that axis.

In mode="materialise", or with a frozen matrix: R = self.matrix().to(x.dtype), then x @ R.t() for a 2-D x along its last axis and torch.einsum("oc,bc...->bo...", R, x) for an x of 3 or more dimensions along dim=1. In mode="apply" the rotations act on x directly.

Parameters:

Name Type Description Default
x Tensor

A 2-D batch (B, n), or a tensor (B, n, ...).

required
dim int

-1 (or 1) for a 2-D input, 1 for a larger one. Default: -1.

-1

Returns:

Type Description
Tensor

torch.Tensor: The rotated tensor, of the shape of x.

Raises:

Type Description
ValueError

For any other combination of shape and dim.

matrix()

The materialised \(n \times n\) orthogonal matrix (see the class contract).

Returns:

Type Description
Tensor

torch.Tensor: The matrix, in the dtype and on the device of theta.

reset_parameters()

Initialise the angles again, by the rule of init.

init="identity" sets them to zero and draws nothing; init="random" draws torch.randn(n_angles) * std from the global generator, as the constructor does, and copies it into theta in place. The estimators call this after seeding, so random_state controls the random angles, and an identity start stays the identity.

OrthogonalConv1x1

Bases: Orthogonal

An orthogonal 1x1 convolution: Orthogonal applied to the channels of a feature map, plus an optional bias.

Numerical contract: forward(x) = F.conv2d(x, W.reshape(n, n, 1, 1), bias) with W = self.matrix().to(x.dtype) and bias = self.bias.to(x.dtype), as Orthogonal.forward casts its matrix to the dtype of the input. The bias parameter is created after theta, so the state_dict holds theta and bias.

Parameters:

Name Type Description Default
n int

The number of channels.

required
topology TopologyName | Topology

A layout name, or a Topology on n wires. Default: "butterfly".

'butterfly'
init Init

"identity" or "random". Default: "identity".

'identity'
std float

The standard deviation of the random angles. Default: 0.1.

0.1
bias bool

Whether to add a trainable bias, initialised to zero. Default: True.

True
mode Mode

"materialise" or "apply". Default: "materialise".

'materialise'

forward(x)

Apply the orthogonal 1x1 convolution.

Parameters:

Name Type Description Default
x Tensor

A feature map of shape (B, n, H, W), or (n, H, W) without a batch axis, as F.conv2d takes.

required

Returns:

Type Description
Tensor

torch.Tensor: The output, of the shape of x.

reset_parameters()

Initialise the angles again by the rule of init (see Orthogonal.reset_parameters), and the bias to zero, as the constructor does.

PartitionedOrthogonal

Bases: _Rotation

An orthogonal \(\mathrm{width} \times \mathrm{width}\) matrix for any width, from orthogonal blocks on the binary parts of the width, mixed by fixed permutations.

The butterfly needs a power-of-two width. This module splits the width into its binary parts (\(588 = 512 + 64 + 8 + 4\)), rotates every part of size 2 or more with its own Orthogonal, and repeats this over stages stages, \(S\) of them, with a fixed channel permutation \(P_s\) in between:

\[ R = (P_S^\top B_S P_S) \cdots (P_2^\top B_2 P_2) \, B_1, \qquad B_s = \operatorname{blockdiag}(\text{parts of stage } s) \]

Every factor is orthogonal, so \(R\) is too. All angles start at zero, so \(R = I\) at initialisation. A part of size 1 (an odd width) is left as it is.

Numerical contracts:

  • Construction: stages outer, parts inner, one Orthogonal(part, topology) with init="identity" per part of size 2 or more, so no random numbers are drawn; then, for s = 0 .. stages - 2, the buffers perm_s = mixing_permutation(width, perm_seed * 1000 + s) and its inverse inv_s = torch.argsort(perm_s), not saved in the state_dict and rebuilt the same way whenever the module moves (so to_empty() after a meta-device build works).
  • Stage matrix: torch.block_diag(*part_matrices), with torch.ones(1, 1) in the dtype and on the device of the other parts appended for a part of size 1.
  • The matrix: R = stage_0; then for s >= 1, R = B_s[inv][:, inv] @ R with inv = inv_{s-1}, the stored torch.argsort(perm_{s-1}).
  • forward(x, dim) follows the contract of Orthogonal.forward in mode="materialise".

The state_dict keys are blocks.<stage>.<part>.theta.

Parameters:

Name Type Description Default
width int

The width, at least 2.

required
stages int

The number of block-diagonal stages, at least 1. Default: 2.

2
perm_seed int

The seed of the mixing permutations. Default: 0.

0
topology TopologyName

The layout of every part. Default: "butterfly".

'butterfly'

n_angles property

The number of angles over every stage and part.

matrix()

The materialised \(\mathrm{width} \times \mathrm{width}\) orthogonal matrix (see the class contract).

Returns:

Type Description
Tensor

torch.Tensor: The matrix.

binary_parts(width)

The powers of two that sum to width, largest first: 588 gives [512, 64, 8, 4].

Parameters:

Name Type Description Default
width int

A positive integer.

required

Returns:

Type Description
list[int]

list[int]: The binary digits of width, as powers of two.

Raises:

Type Description
ValueError

If width is not a positive integer.

mixing_permutation(width, random_state)

A fixed random permutation of width channels, from its own CPU generator.

Numerical contract: torch.randperm(width, generator=torch.Generator().manual_seed(random_state)). The global generator is not touched.

Parameters:

Name Type Description Default
width int

The number of channels.

required
random_state int

The seed of the generator.

required

Returns:

Type Description
Tensor

torch.Tensor: A permutation of 0 .. width - 1, as a long tensor.