modules
eigenkit.orthogonal.torch.modules
PyTorch modules for orthogonal networks: the angles are an nn.Parameter, and the
orthogonal matrix is rebuilt from them, so it stays orthogonal at every training step.
Numerical contracts. Each module documents the exact sequence of torch operations it runs, and the tests pin it bit for bit. Changing one of them is a breaking change, because it changes the last bits of every result and so the trajectory of any training run.
Example
MAX_MATERIALISE = 2 ** 31
module-attribute
The largest \(\mathrm{depth} \cdot n^2\) for which mode="materialise" builds
the dense matrix.
Orthogonal
Bases: _Rotation
An \(n \times n\) orthogonal matrix parameterised by the angles of a network of RBS (Givens) rotations.
A Givens rotation is orthogonal for every angle, and a product of orthogonal matrices is orthogonal, so the matrix is orthogonal at every training step with no projection and no penalty. On quantum hardware each rotation is one RBS gate, so this is the classical simulation of a quantum orthogonal layer.
The state_dict holds only theta. The pair indices are buffers that are not
saved, so .to(device) moves them and a checkpoint stays small. They are
rebuilt from the topology whenever the module moves, so a module built on the
meta device, then to_empty() and load_state_dict(), works.
Numerical contracts:
- Initialisation:
init="identity"setstheta = torch.zeros(n_angles)and draws no random numbers;init="random"setstheta = torch.randn(n_angles) * std, one call on the global generator, made before any other random draw of the module. - The matrix: starting from
W = torch.eye(n)in the dtype and on the device oftheta, each layer takes the nextkangles,c = cos(theta[at:at+k]).unsqueeze(1),s = sin(...).unsqueeze(1), and setsW = W.index_copy(0, i, c*Wi + s*Wj).index_copy(0, j, -s*Wi + c*Wj)withWi = W.index_select(0, i),Wj = W.index_select(0, j). With a reflected topology the last row is negated at the end. - Caching: the matrix is cached only when
not self.training and not torch.is_grad_enabled(), under the key(theta._version, theta.device, theta.dtype), and it is reused only whilethetastill holds the values it was built from (torch.equal), so a write throughtheta.datacannot leave it stale. The cache decides only when the matrix is rebuilt, never its values. On themetadevice nothing is cached. A frozen matrix (freeze_matrix) takes precedence.
mode="materialise" builds the dense matrix, at a cost of
\(O(\mathrm{depth} \cdot n^2)\), and applies it as one matrix product. It is
refused when \(\mathrm{depth} \cdot n^2\) exceeds MAX_MATERIALISE
(\(2^{31}\)), which rules out a pyramid wider than 1024. mode="apply"
applies the rotations to the input layer by layer, at a cost of
\(O(\mathrm{depth} \cdot n \cdot B)\) for a batch of \(B\), and never builds the
matrix in forward; it has no contract with the materialised path, whose last
bits it does not reproduce.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n
|
int
|
The width. |
required |
topology
|
TopologyName | Topology
|
A layout name, or a |
'butterfly'
|
init
|
Init
|
|
'identity'
|
std
|
float
|
The standard deviation of the random angles. Default: |
0.1
|
mode
|
Mode
|
|
'materialise'
|
n_angles
property
The number of angles, one per RBS gate.
forward(x, dim=-1)
Rotate x along one axis: \(y = Rx\) on that axis.
In mode="materialise", or with a frozen matrix: R =
self.matrix().to(x.dtype), then x @ R.t() for a 2-D x along its last
axis and torch.einsum("oc,bc...->bo...", R, x) for an x of 3 or more
dimensions along dim=1. In mode="apply" the rotations act on x
directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
A 2-D batch |
required |
dim
|
int
|
|
-1
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The rotated tensor, of the shape of |
Raises:
| Type | Description |
|---|---|
ValueError
|
For any other combination of shape and |
matrix()
The materialised \(n \times n\) orthogonal matrix (see the class contract).
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The matrix, in the dtype and on the device of |
reset_parameters()
Initialise the angles again, by the rule of init.
init="identity" sets them to zero and draws nothing; init="random"
draws torch.randn(n_angles) * std from the global generator, as the
constructor does, and copies it into theta in place. The estimators call
this after seeding, so random_state controls the random angles, and an
identity start stays the identity.
OrthogonalConv1x1
Bases: Orthogonal
An orthogonal 1x1 convolution: Orthogonal applied to the channels of a
feature map, plus an optional bias.
Numerical contract: forward(x) = F.conv2d(x, W.reshape(n, n, 1, 1), bias)
with W = self.matrix().to(x.dtype) and bias = self.bias.to(x.dtype), as
Orthogonal.forward casts its matrix to the dtype of the input. The bias
parameter is created after theta, so the state_dict holds theta and
bias.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n
|
int
|
The number of channels. |
required |
topology
|
TopologyName | Topology
|
A layout name, or a |
'butterfly'
|
init
|
Init
|
|
'identity'
|
std
|
float
|
The standard deviation of the random angles. Default: |
0.1
|
bias
|
bool
|
Whether to add a trainable bias, initialised to zero.
Default: |
True
|
mode
|
Mode
|
|
'materialise'
|
forward(x)
Apply the orthogonal 1x1 convolution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
A feature map of shape |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The output, of the shape of |
reset_parameters()
Initialise the angles again by the rule of init (see
Orthogonal.reset_parameters), and the bias to zero, as the constructor
does.
PartitionedOrthogonal
Bases: _Rotation
An orthogonal \(\mathrm{width} \times \mathrm{width}\) matrix for any width, from orthogonal blocks on the binary parts of the width, mixed by fixed permutations.
The butterfly needs a power-of-two width. This module splits the width into its
binary parts (\(588 = 512 + 64 + 8 + 4\)), rotates every part of size 2 or more
with its own Orthogonal, and repeats this over stages stages, \(S\) of them,
with a fixed channel permutation \(P_s\) in between:
Every factor is orthogonal, so \(R\) is too. All angles start at zero, so \(R = I\) at initialisation. A part of size 1 (an odd width) is left as it is.
Numerical contracts:
- Construction: stages outer, parts inner, one
Orthogonal(part, topology)withinit="identity"per part of size 2 or more, so no random numbers are drawn; then, fors = 0 .. stages - 2, the buffersperm_s = mixing_permutation(width, perm_seed * 1000 + s)and its inverseinv_s = torch.argsort(perm_s), not saved in thestate_dictand rebuilt the same way whenever the module moves (soto_empty()after ameta-device build works). - Stage matrix:
torch.block_diag(*part_matrices), withtorch.ones(1, 1)in the dtype and on the device of the other parts appended for a part of size 1. - The matrix:
R = stage_0; then fors >= 1,R = B_s[inv][:, inv] @ Rwithinv = inv_{s-1}, the storedtorch.argsort(perm_{s-1}). forward(x, dim)follows the contract ofOrthogonal.forwardinmode="materialise".
The state_dict keys are blocks.<stage>.<part>.theta.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
The width, at least 2. |
required |
stages
|
int
|
The number of block-diagonal stages, at least 1. Default: |
2
|
perm_seed
|
int
|
The seed of the mixing permutations. Default: |
0
|
topology
|
TopologyName
|
The layout of every part. Default: |
'butterfly'
|
n_angles
property
The number of angles over every stage and part.
matrix()
The materialised \(\mathrm{width} \times \mathrm{width}\) orthogonal matrix (see the class contract).
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The matrix. |
binary_parts(width)
The powers of two that sum to width, largest first: 588 gives
[512, 64, 8, 4].
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
A positive integer. |
required |
Returns:
| Type | Description |
|---|---|
list[int]
|
list[int]: The binary digits of |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
mixing_permutation(width, random_state)
A fixed random permutation of width channels, from its own CPU generator.
Numerical contract:
torch.randperm(width, generator=torch.Generator().manual_seed(random_state)).
The global generator is not touched.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
The number of channels. |
required |
random_state
|
int
|
The seed of the generator. |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: A permutation of |