## `eigenkit.orthogonal.torch`

The PyTorch ring of `eigenkit.orthogonal`: trainable orthogonal modules and orthogonal fine-tuning adapters.

It needs the `torch` extra (`pip install 'eigenkit[torch]'`). The NumPy part of `eigenkit.orthogonal` loads this package only when one of its names is accessed.

### `MAX_MATERIALISE = 2 ** 31`

The largest `$\mathrm{depth} \cdot n^2$` for which `mode="materialise"` builds the dense matrix.

### `Certificate`

Bases: `NamedTuple`

The prediction, the top-1 margin and the certified radius of every row.

Attributes:

| Name     | Type     | Description                                                                            |
| -------- | -------- | -------------------------------------------------------------------------------------- |
| `pred`   | `Tensor` | The predicted class of each row, long.                                                 |
| `margin` | `Tensor` | The gap between the two largest logits, float64.                                       |
| `radius` | `Tensor` | The certified L2 radius, float64; inf when no other class can overtake the prediction. |

### `LLNHead`

Bases: `Module`

A linear head with unit-norm rows (last-layer normalisation).

The rows `$w_k$` of the weight are normalised at every call, so the certificate can use the distances `$\lVert w_p - w_j \rVert$` directly. The bias cancels in every logit difference, so the certificate does not see it.

Numerical contract: `F.linear(h, F.normalize(self.linear.weight, dim=1), self.linear.bias)`. The parameters are `linear.weight` and `linear.bias`.

Parameters:

| Name           | Type  | Description                   | Default    |
| -------------- | ----- | ----------------------------- | ---------- |
| `in_features`  | `int` | The number of input features. | *required* |
| `out_features` | `int` | The number of classes.        | *required* |

#### `forward(h)`

Apply the head.

Parameters:

| Name | Type     | Description                | Default    |
| ---- | -------- | -------------------------- | ---------- |
| `h`  | `Tensor` | Features (B, in_features). | *required* |

Returns:

| Type     | Description                             |
| -------- | --------------------------------------- |
| `Tensor` | torch.Tensor: Logits (B, out_features). |

#### `get_weight()`

The weight with unit-norm rows, `F.normalize(self.linear.weight, dim=1)`.

Returns:

| Type     | Description                                         |
| -------- | --------------------------------------------------- |
| `Tensor` | torch.Tensor: A (out_features, in_features) tensor. |

### `MaxMin`

Bases: `Module`

The MaxMin activation: sort the channels within disjoint pairs.

It is 1-Lipschitz and keeps the norm of its input, because it only permutes values. It is not distance-preserving: it maps `(1, 0)` and `(0, 1)` to the same point (Anil, Lucas & Grosse, ICML 2019). It acts on dim 1.

Two pairings are in use and give different functions:

- `"adjacent"` (GroupSort) pairs channels `$2k$` and `$2k + 1$` and writes the minimum first. Numerical contract: `g = x.reshape(b, c // 2, 2, *rest)`, `lo = torch.minimum(g[:, :, 0], g[:, :, 1])`, `hi = torch.maximum(...)`, then `torch.stack((lo, hi), dim=2).reshape(x.shape)`.
- `"halves"` (BRONet's MinMax) pairs channel `$k$` with `$k + C/2$` and writes the maximum of each pair in the first half. Numerical contract: `a, b = torch.split(x, c // 2, dim=1)`, `index = a > b`, then `torch.cat([torch.where(index, a, b), torch.where(index, b, a)], dim=1)`.

Parameters:

| Name      | Type      | Description                                  | Default      |
| --------- | --------- | -------------------------------------------- | ------------ |
| `pairing` | `Pairing` | "adjacent" or "halves". Default: "adjacent". | `'adjacent'` |

#### `forward(x)`

Apply the activation.

Parameters:

| Name | Type     | Description                          | Default    |
| ---- | -------- | ------------------------------------ | ---------- |
| `x`  | `Tensor` | A tensor (B, C, ...) with an even C. | *required* |

Returns:

| Type     | Description                                  |
| -------- | -------------------------------------------- |
| `Tensor` | torch.Tensor: The output, of the shape of x. |

Raises:

| Type         | Description  |
| ------------ | ------------ |
| `ValueError` | If C is odd. |

### `Orthogonal`

Bases: `_Rotation`

An `$n \times n$` orthogonal matrix parameterised by the angles of a network of RBS (Givens) rotations.

A Givens rotation is orthogonal for every angle, and a product of orthogonal matrices is orthogonal, so the matrix is orthogonal at every training step with no projection and no penalty. On quantum hardware each rotation is one RBS gate, so this is the classical simulation of a quantum orthogonal layer.

The `state_dict` holds only `theta`. The pair indices are buffers that are not saved, so `.to(device)` moves them and a checkpoint stays small. They are rebuilt from the topology whenever the module moves, so a module built on the `meta` device, then `to_empty()` and `load_state_dict()`, works.

Numerical contracts:

- Initialisation: `init="identity"` sets `theta = torch.zeros(n_angles)` and draws no random numbers; `init="random"` sets `theta = torch.randn(n_angles) * std`, one call on the global generator, made before any other random draw of the module.
- The matrix: starting from `W = torch.eye(n)` in the dtype and on the device of `theta`, each layer takes the next `k` angles, `c = cos(theta[at:at+k]).unsqueeze(1)`, `s = sin(...).unsqueeze(1)`, and sets `W = W.index_copy(0, i, c*Wi + s*Wj).index_copy(0, j, -s*Wi + c*Wj)` with `Wi = W.index_select(0, i)`, `Wj = W.index_select(0, j)`. With a reflected topology the last row is negated at the end.
- Caching: the matrix is cached only when `not self.training and not torch.is_grad_enabled()`, under the key `(theta._version, theta.device, theta.dtype)`, and it is reused only while `theta` still holds the values it was built from (`torch.equal`), so a write through `theta.data` cannot leave it stale. The cache decides only when the matrix is rebuilt, never its values. On the `meta` device nothing is cached. A frozen matrix (`freeze_matrix`) takes precedence.

`mode="materialise"` builds the dense matrix, at a cost of `$O(\mathrm{depth} \cdot n^2)$`, and applies it as one matrix product. It is refused when `$\mathrm{depth} \cdot n^2$` exceeds `MAX_MATERIALISE` (`$2^{31}$`), which rules out a pyramid wider than 1024. `mode="apply"` applies the rotations to the input layer by layer, at a cost of `$O(\mathrm{depth} \cdot n \cdot B)$` for a batch of `$B$`, and never builds the matrix in `forward`; it has no contract with the materialised path, whose last bits it does not reproduce.

Parameters:

| Name       | Type           | Description                                                                                            | Default                                                        |
| ---------- | -------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- |
| `n`        | `int`          | The width.                                                                                             | *required*                                                     |
| `topology` | \`TopologyName | Topology\`                                                                                             | A layout name, or a Topology on n wires. Default: "butterfly". |
| `init`     | `Init`         | "identity" for zero angles, "random" for normal angles of standard deviation std. Default: "identity". | `'identity'`                                                   |
| `std`      | `float`        | The standard deviation of the random angles. Default: 0.1.                                             | `0.1`                                                          |
| `mode`     | `Mode`         | "materialise" or "apply". Default: "materialise".                                                      | `'materialise'`                                                |

#### `n_angles`

The number of angles, one per RBS gate.

#### `forward(x, dim=-1)`

Rotate `x` along one axis: `$y = Rx$` on that axis.

In `mode="materialise"`, or with a frozen matrix: `R = self.matrix().to(x.dtype)`, then `x @ R.t()` for a 2-D `x` along its last axis and `torch.einsum("oc,bc...->bo...", R, x)` for an `x` of 3 or more dimensions along `dim=1`. In `mode="apply"` the rotations act on `x` directly.

Parameters:

| Name  | Type     | Description                                                 | Default    |
| ----- | -------- | ----------------------------------------------------------- | ---------- |
| `x`   | `Tensor` | A 2-D batch (B, n), or a tensor (B, n, ...).                | *required* |
| `dim` | `int`    | -1 (or 1) for a 2-D input, 1 for a larger one. Default: -1. | `-1`       |

Returns:

| Type     | Description                                          |
| -------- | ---------------------------------------------------- |
| `Tensor` | torch.Tensor: The rotated tensor, of the shape of x. |

Raises:

| Type         | Description                                 |
| ------------ | ------------------------------------------- |
| `ValueError` | For any other combination of shape and dim. |

#### `matrix()`

The materialised `$n \times n$` orthogonal matrix (see the class contract).

Returns:

| Type     | Description                                                        |
| -------- | ------------------------------------------------------------------ |
| `Tensor` | torch.Tensor: The matrix, in the dtype and on the device of theta. |

#### `reset_parameters()`

Initialise the angles again, by the rule of `init`.

`init="identity"` sets them to zero and draws nothing; `init="random"` draws `torch.randn(n_angles) * std` from the global generator, as the constructor does, and copies it into `theta` in place. The estimators call this after seeding, so `random_state` controls the random angles, and an identity start stays the identity.

### `OrthogonalAdapter`

Bases: `Module`

A frozen `nn.Linear` or `nn.Conv2d` whose weight is rotated by a trainable orthogonal matrix.

`side="out"` gives `$W' = RW$`, with `$R$` on the output channels; `side="in"` gives `$W' = WR^\top$`, with `$R$` on the input channels. The bias of the base layer is kept as it is. The weight of a convolution is read as the matrix `(out, in·k·k)` (see `rotate_weight`).

With the butterfly layout and a width that is not a power of two, the rotation is a `PartitionedOrthogonal` when `partition` is `None` (the default) or `True`. Any other layout takes any width it accepts, and `partition=True` uses the partitioned form for it too. The angles take the dtype and the device of the base weight.

An adapter cannot be re-initialised (see `reset_parameters`), so a model with adapters is trained with a training loop of your own: the estimators `GenericClassifier` and `GenericRegressor` re-initialise every layer in `fit()` and refuse it.

Parameters:

| Name        | Type           | Description                                                 | Default                                                                                                                 |
| ----------- | -------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `base`      | \`Linear       | Conv2d\`                                                    | The pretrained layer. Its parameters are frozen in place.                                                               |
| `topology`  | \`TopologyName | Topology\`                                                  | The layout of the rotation. Default: "butterfly".                                                                       |
| `side`      | `Side`         | "out" or "in". Default: "out".                              | `'out'`                                                                                                                 |
| `partition` | \`bool         | None\`                                                      | Whether to use PartitionedOrthogonal; None chooses it for a butterfly whose width is not a power of two. Default: None. |
| `stages`    | `int`          | The stages of a partitioned rotation. Default: 2.           | `2`                                                                                                                     |
| `perm_seed` | `int`          | The permutation seed of a partitioned rotation. Default: 0. | `0`                                                                                                                     |

#### `bias`

The bias of the base layer, `base.bias`, which is not rotated.

Returns:

| Type     | Description |
| -------- | ----------- |
| \`Tensor | None\`      |

#### `weight`

The adapted weight `rotate_weight(R, base.weight, side)`.

A property, like the `weight` of the layer it replaces, so code that reads `layer.weight` directly (`nn.MultiheadAttention` reads `out_proj.weight`) gets the adapted weight and trains the angles. It is rebuilt from the angles at every read.

Returns:

| Type     | Description                                         |
| -------- | --------------------------------------------------- |
| `Tensor` | torch.Tensor: A tensor of the shape of base.weight. |

#### `forward(x)`

Run the base layer with the adapted weight.

Parameters:

| Name | Type     | Description                  | Default    |
| ---- | -------- | ---------------------------- | ---------- |
| `x`  | `Tensor` | The input of the base layer. | *required* |

Returns:

| Type     | Description                                                         |
| -------- | ------------------------------------------------------------------- |
| `Tensor` | torch.Tensor: The output of the base layer with weight self.weight. |

#### `merge()`

A copy of the base layer with the rotation folded into its weight.

Its parameters are frozen, like those of the base layer.

Returns:

| Type     | Description |
| -------- | ----------- |
| \`Linear | Conv2d\`    |

#### `reset_parameters()`

Refuse to re-initialise: an adapter starts from its pretrained weight.

`model.apply(lambda m: m.reset_parameters())`, which the estimators `GenericClassifier` and `GenericRegressor` run at the start of `fit()`, resets the children of a module before the module itself, so by the time this method runs `base` already holds a new random weight. Training would then rotate that random weight instead of the pretrained one, without any error. Raising here stops it. Train an adapted model with a training loop of your own, as in the fine-tuning example.

Raises:

| Type           | Description |
| -------------- | ----------- |
| `RuntimeError` | Always.     |

### `OrthogonalConv1x1`

Bases: `Orthogonal`

An orthogonal 1x1 convolution: `Orthogonal` applied to the channels of a feature map, plus an optional bias.

Numerical contract: `forward(x) = F.conv2d(x, W.reshape(n, n, 1, 1), bias)` with `W = self.matrix().to(x.dtype)` and `bias = self.bias.to(x.dtype)`, as `Orthogonal.forward` casts its matrix to the dtype of the input. The bias parameter is created after `theta`, so the `state_dict` holds `theta` and `bias`.

Parameters:

| Name       | Type           | Description                                                          | Default                                                        |
| ---------- | -------------- | -------------------------------------------------------------------- | -------------------------------------------------------------- |
| `n`        | `int`          | The number of channels.                                              | *required*                                                     |
| `topology` | \`TopologyName | Topology\`                                                           | A layout name, or a Topology on n wires. Default: "butterfly". |
| `init`     | `Init`         | "identity" or "random". Default: "identity".                         | `'identity'`                                                   |
| `std`      | `float`        | The standard deviation of the random angles. Default: 0.1.           | `0.1`                                                          |
| `bias`     | `bool`         | Whether to add a trainable bias, initialised to zero. Default: True. | `True`                                                         |
| `mode`     | `Mode`         | "materialise" or "apply". Default: "materialise".                    | `'materialise'`                                                |

#### `forward(x)`

Apply the orthogonal 1x1 convolution.

Parameters:

| Name | Type     | Description                                                                                | Default    |
| ---- | -------- | ------------------------------------------------------------------------------------------ | ---------- |
| `x`  | `Tensor` | A feature map of shape (B, n, H, W), or (n, H, W) without a batch axis, as F.conv2d takes. | *required* |

Returns:

| Type     | Description                                  |
| -------- | -------------------------------------------- |
| `Tensor` | torch.Tensor: The output, of the shape of x. |

#### `reset_parameters()`

Initialise the angles again by the rule of `init` (see `Orthogonal.reset_parameters`), and the bias to zero, as the constructor does.

### `PartitionedOrthogonal`

Bases: `_Rotation`

An orthogonal `$\mathrm{width} \times \mathrm{width}$` matrix for any width, from orthogonal blocks on the binary parts of the width, mixed by fixed permutations.

The butterfly needs a power-of-two width. This module splits the width into its binary parts (`$588 = 512 + 64 + 8 + 4$`), rotates every part of size 2 or more with its own `Orthogonal`, and repeats this over `stages` stages, `$S$` of them, with a fixed channel permutation `$P_s$` in between:

```math
R = (P_S^\top B_S P_S) \cdots (P_2^\top B_2 P_2) \, B_1,
\qquad B_s = \operatorname{blockdiag}(\text{parts of stage } s)
```

Every factor is orthogonal, so `$R$` is too. All angles start at zero, so `$R = I$` at initialisation. A part of size 1 (an odd width) is left as it is.

Numerical contracts:

- Construction: stages outer, parts inner, one `Orthogonal(part, topology)` with `init="identity"` per part of size 2 or more, so no random numbers are drawn; then, for `s = 0 .. stages - 2`, the buffers `perm_s = mixing_permutation(width, perm_seed * 1000 + s)` and its inverse `inv_s = torch.argsort(perm_s)`, not saved in the `state_dict` and rebuilt the same way whenever the module moves (so `to_empty()` after a `meta`-device build works).
- Stage matrix: `torch.block_diag(*part_matrices)`, with `torch.ones(1, 1)` in the dtype and on the device of the other parts appended for a part of size 1.
- The matrix: `R = stage_0`; then for `s >= 1`, `R = B_s[inv][:, inv] @ R` with `inv = inv_{s-1}`, the stored `torch.argsort(perm_{s-1})`.
- `forward(x, dim)` follows the contract of `Orthogonal.forward` in `mode="materialise"`.

The `state_dict` keys are `blocks.<stage>.<part>.theta`.

Parameters:

| Name        | Type           | Description                                                  | Default       |
| ----------- | -------------- | ------------------------------------------------------------ | ------------- |
| `width`     | `int`          | The width, at least 2.                                       | *required*    |
| `stages`    | `int`          | The number of block-diagonal stages, at least 1. Default: 2. | `2`           |
| `perm_seed` | `int`          | The seed of the mixing permutations. Default: 0.             | `0`           |
| `topology`  | `TopologyName` | The layout of every part. Default: "butterfly".              | `'butterfly'` |

#### `n_angles`

The number of angles over every stage and part.

#### `matrix()`

The materialised `$\mathrm{width} \times \mathrm{width}$` orthogonal matrix (see the class contract).

Returns:

| Type     | Description               |
| -------- | ------------------------- |
| `Tensor` | torch.Tensor: The matrix. |

### `apply_orthogonal_adapters(model, target_modules, topology='butterfly', side='out', freeze_rest=True, **adapter_kwargs)`

Replace layers of a model by `OrthogonalAdapter`s, in place.

Parameters:

| Name               | Type           | Description                                                                          | Default                                             |
| ------------------ | -------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------- |
| `model`            | `Module`       | The model to adapt.                                                                  | *required*                                          |
| `target_modules`   | \`str          | Iterable[str]                                                                        | Callable\[[str, Module], bool\]\`                   |
| `topology`         | \`TopologyName | Topology\`                                                                           | The layout of every rotation. Default: "butterfly". |
| `side`             | `Side`         | "out" or "in". Default: "out".                                                       | `'out'`                                             |
| `freeze_rest`      | `bool`         | Whether to freeze every parameter that is not an angle of an adapter. Default: True. | `True`                                              |
| `**adapter_kwargs` | `Any`          | Passed to OrthogonalAdapter, such as partition.                                      | `{}`                                                |

Returns:

| Type     | Description                |
| -------- | -------------------------- |
| `Module` | nn.Module: The same model. |

Raises:

| Type         | Description                                                 |
| ------------ | ----------------------------------------------------------- |
| `KeyError`   | If a name is not a module of the model.                     |
| `TypeError`  | If a named module is not nn.Linear or nn.Conv2d.            |
| `ValueError` | If nothing matches, or a named module is inside an adapter. |

### `binary_parts(width)`

The powers of two that sum to `width`, largest first: 588 gives `[512, 64, 8, 4]`.

Parameters:

| Name    | Type  | Description         | Default    |
| ------- | ----- | ------------------- | ---------- |
| `width` | `int` | A positive integer. | *required* |

Returns:

| Type        | Description                                                |
| ----------- | ---------------------------------------------------------- |
| `list[int]` | list\[int\]: The binary digits of width, as powers of two. |

Raises:

| Type         | Description                         |
| ------------ | ----------------------------------- |
| `ValueError` | If width is not a positive integer. |

### `cert_ce(logits, y, head_weight, L, eps, tau=4.0)`

A certification-aware cross-entropy.

Every wrong logit is raised by the most a perturbation of norm `eps` could raise it against the true class, `$L \cdot \mathrm{eps} \cdot \lVert w_y - w_j \rVert$`, and the result is scored with a cross-entropy at temperature `1/tau`. Minimising it pushes the margins beyond `eps`.

Numerical contract: `d = (head_weight[y].unsqueeze(1) - head_weight.unsqueeze(0)).norm(dim=-1)`, `z = logits + L * eps * d`, then `F.cross_entropy(tau * z, y)`. The row of the true class has `d = 0`.

Parameters:

| Name          | Type     | Description                                              | Default    |
| ------------- | -------- | -------------------------------------------------------- | ---------- |
| `logits`      | `Tensor` | Logits (B, K).                                           | *required* |
| `y`           | `Tensor` | The labels (B,).                                         | *required* |
| `head_weight` | `Tensor` | The head weight (K, D), in the graph if the head trains. | *required* |
| `L`           | `float`  | The Lipschitz constant of the feature map.               | *required* |
| `eps`         | `float`  | The radius to train for.                                 | *required* |
| `tau`         | `float`  | The scale of the logits. Default: 4.0.                   | `4.0`      |

Returns:

| Type     | Description                       |
| -------- | --------------------------------- |
| `Tensor` | torch.Tensor: The loss, a scalar. |

### `certified_accuracy(correct, radius, eps)`

The fraction of rows that are correct and certified beyond `eps`.

Numerical contract: `float(correct.float().mean())` for `eps == 0`, which is the plain accuracy, and `float((correct & (radius > eps)).float().mean())` otherwise.

Parameters:

| Name      | Type     | Description                                 | Default    |
| --------- | -------- | ------------------------------------------- | ---------- |
| `correct` | `Tensor` | A boolean tensor, pred == labels.           | *required* |
| `radius`  | `Tensor` | The certified radii, from certified_radius. | *required* |
| `eps`     | `float`  | The L2 radius to certify.                   | *required* |

Returns:

| Name    | Type    | Description                    |
| ------- | ------- | ------------------------------ |
| `float` | `float` | The certified accuracy at eps. |

### `certified_radius(logits, head_weight, L)`

The certified L2 radius of every row of `logits`, for a feature map with Lipschitz constant `L` and a linear head with weight `head_weight`.

Numerical contract, in float64 on the CPU: `z = logits.double()`, `d = head_distances(head_weight)`, `pred = z.argmax(1)`, `gap = z.gather(1, pred[:, None]) - z`, `dist = d[pred]`, `r = gap / (L * dist.clamp_min(1e-300))`; then `r` is set to `inf` on the predicted class and wherever `dist == 0` (a class with the same row can never overtake the prediction), and the radius is `r.min(1).values`. The margin is the difference of the two largest logits, from `z.topk(2, dim=1)`.

Parameters:

| Name          | Type     | Description                                                     | Default    |
| ------------- | -------- | --------------------------------------------------------------- | ---------- |
| `logits`      | `Tensor` | Logits (B, K), with K >= 2.                                     | *required* |
| `head_weight` | `Tensor` | The head weight (K, D), such as LLNHead.get_weight().           | *required* |
| `L`           | `float`  | The Lipschitz constant of the feature map in front of the head. | *required* |

Returns:

| Name          | Type          | Description                                             |
| ------------- | ------------- | ------------------------------------------------------- |
| `Certificate` | `Certificate` | The prediction, the margin and the radius of every row. |

### `fold_bn(conv, bn)`

Fold an eval-mode BatchNorm into the convolution before it.

Numerical contract, all in float64 on the CPU: `scale = bn.weight / sqrt(bn.running_var + bn.eps)`, `w = conv.weight * scale.view(-1, 1, 1, 1)` (one `1` per axis of the weight after the first, so `view(-1, 1, 1)` for a `Conv1d`) and `b = bn.bias - scale * bn.running_mean`. A convolution with a bias of its own gives `b = bn.bias - scale * (bn.running_mean - conv.bias)` instead. A BatchNorm with `affine=False` counts as `weight = 1` and `bias = 0`. Cast the results to the dtype you need.

Parameters:

| Name   | Type          | Description | Default       |
| ------ | ------------- | ----------- | ------------- |
| `conv` | \`Conv1d      | Conv2d      | Conv3d\`      |
| `bn`   | \`BatchNorm1d | BatchNorm2d | BatchNorm3d\` |

Returns:

| Type     | Description                                                             |
| -------- | ----------------------------------------------------------------------- |
| `Tensor` | tuple\[torch.Tensor, torch.Tensor\]: The folded weight, of the shape of |
| `Tensor` | conv.weight, and bias, float64 on the CPU.                              |

Raises:

| Type         | Description                                                                                                            |
| ------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `TypeError`  | If conv is not nn.Conv1d, nn.Conv2d or nn.Conv3d (a transposed convolution keeps its output channels on another axis). |
| `ValueError` | If the BatchNorm keeps no running statistics, or its number of features is not the number of output channels of conv.  |

### `head_distances(head_weight)`

The distances `$\lVert w_i - w_j \rVert$` between the rows of a head weight, in float64.

Numerical contract: `w = head_weight.detach().double()`, then `torch.cdist(w, w, compute_mode="donot_use_mm_for_euclid_dist")`. The direct differences keep small distances exact; the default mode switches to a matrix product above 25 rows, which can shrink a small distance and so overstate a certified radius.

Parameters:

| Name          | Type     | Description             | Default    |
| ------------- | -------- | ----------------------- | ---------- |
| `head_weight` | `Tensor` | The head weight (K, D). | *required* |

Returns:

| Type     | Description                                   |
| -------- | --------------------------------------------- |
| `Tensor` | torch.Tensor: The (K, K) matrix of distances. |

### `lipschitz_bound(model)`

An upper bound on the L2 Lipschitz constant of a chain of layers.

The bound is the product of a bound for every layer of an `nn.Sequential` (nested ones are flattened): the largest singular value of the weight of an `nn.Linear`, an `LLNHead` (its normalised weight) or an `OrthogonalAdapter` (its rotated weight); for an `nn.Conv2d`, that of the weight read as a matrix `(out, in·kh·kw)`, times `$\sqrt{k_h k_w}$`, which bounds the convolution for any stride and dilation with `"zeros"` padding, and with `"circular"` padding no wider than `dilation * (kernel_size - 1) / 2` on each axis (wider circular padding, `"reflect"` and `"replicate"` feed a pixel to one kernel offset more than once and raise a `ValueError`); 1 for the orthogonal modules and for `MaxMin`, `ReLU`, `LeakyReLU` (slope at most 1), `Tanh`, `Sigmoid`, `Flatten` and `Identity`. The singular values are computed with `sigma_max` in float64, on the weight matrix or its transpose, whichever has fewer rows. A 1x1 convolution and a linear layer are bounded exactly.

An orthogonal adapter keeps every singular value of the weight it rotates, so adapting a chain leaves its bound unchanged.

Parameters:

| Name    | Type     | Description                          | Default    |
| ------- | -------- | ------------------------------------ | ---------- |
| `model` | `Module` | An nn.Sequential, or a single layer. | *required* |

Returns:

| Name    | Type    | Description |
| ------- | ------- | ----------- |
| `float` | `float` | The bound.  |

Raises:

| Type         | Description                                                                                                                                       |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ValueError` | If a layer is not one of the above, or a convolution pads with "reflect", "replicate", or "circular" wider than dilation * (kernel_size - 1) / 2. |

### `merge_adapters(model)`

Replace every `OrthogonalAdapter` of a model by its merged layer, in place.

Parameters:

| Name    | Type     | Description                                                                             | Default    |
| ------- | -------- | --------------------------------------------------------------------------------------- | ---------- |
| `model` | `Module` | A model with adapters. If the model is itself an adapter, its merged layer is returned. | *required* |

Returns:

| Type     | Description                                                      |
| -------- | ---------------------------------------------------------------- |
| `Module` | nn.Module: The model, with plain layers where the adapters were. |

### `mixing_permutation(width, random_state)`

A fixed random permutation of `width` channels, from its own CPU generator.

Numerical contract: `torch.randperm(width, generator=torch.Generator().manual_seed(random_state))`. The global generator is not touched.

Parameters:

| Name           | Type  | Description                | Default    |
| -------------- | ----- | -------------------------- | ---------- |
| `width`        | `int` | The number of channels.    | *required* |
| `random_state` | `int` | The seed of the generator. | *required* |

Returns:

| Type     | Description                                                      |
| -------- | ---------------------------------------------------------------- |
| `Tensor` | torch.Tensor: A permutation of 0 .. width - 1, as a long tensor. |

### `rotate_weight(R, W, side='out')`

Rotate the output or the input channels of a weight tensor by an orthogonal matrix.

The weight is read as the matrix `W.reshape(out, in·k·k)`. Rotating its rows (`side="out"`, `$RW$`) or its input channels (`side="in"`, `$W (R^\top \otimes I)$`) keeps every singular value of that matrix, so the operator norm, and the Lipschitz constant, of the layer are unchanged. For a convolution `R` acts pixel by pixel, so the same holds for the convolution operator.

Numerical contract for `side="out"`: `(R @ W.reshape(W.shape[0], -1)).reshape(W.shape)`, which is `R @ W` for a 2-D weight. `side="in"` computes `torch.einsum("oi...,ji->oj...", W, R).contiguous()`, which is `W @ R.T` for a 2-D weight.

Parameters:

| Name   | Type     | Description                                                                | Default    |
| ------ | -------- | -------------------------------------------------------------------------- | ---------- |
| `R`    | `Tensor` | An orthogonal matrix, (out, out) for side="out" or (in, in) for side="in". | *required* |
| `W`    | `Tensor` | A weight (out, in) or (out, in, \*kernel).                                 | *required* |
| `side` | `Side`   | "out" or "in". Default: "out".                                             | `'out'`    |

Returns:

| Type     | Description                                          |
| -------- | ---------------------------------------------------- |
| `Tensor` | torch.Tensor: The rotated weight, of the shape of W. |

Raises:

| Type         | Description                                      |
| ------------ | ------------------------------------------------ |
| `ValueError` | If side is unknown or R does not match the axis. |

### `sigma_max(w)`

The largest singular value of a matrix, through its Gram matrix.

Exact and differentiable. Numerical contract: `torch.linalg.eigvalsh(w @ w.t()).amax().clamp_min(0).sqrt()`.

Parameters:

| Name | Type     | Description      | Default    |
| ---- | -------- | ---------------- | ---------- |
| `w`  | `Tensor` | A matrix (m, n). | *required* |

Returns:

| Type     | Description                    |
| -------- | ------------------------------ |
| `Tensor` | torch.Tensor: A scalar tensor. |

### `trainable_parameters(model)`

The parameters that require a gradient, for the optimiser.

Parameters:

| Name    | Type     | Description | Default    |
| ------- | -------- | ----------- | ---------- |
| `model` | `Module` | The model.  | *required* |

Yields:

| Type        | Description                                           |
| ----------- | ----------------------------------------------------- |
| `Parameter` | nn.Parameter: Each parameter with requires_grad=True. |

### `unary_loader_angles(x, layout='diagonal')`

The input angles of `UnaryLoader` that load `$x / \lVert x \rVert$`, differentiable in `x`.

The PyTorch form of `eigenkit.orthogonal.unary_loader_angles`: the same gate plan, with `torch.atan2` on the signed component or the norm of the wires each gate keeps and passes on. Each row is first scaled by a power of two, which is exact, so the angles do not depend on the scale of `x`, at any finite scale.

Parameters:

| Name     | Type           | Description                             | Default      |
| -------- | -------------- | --------------------------------------- | ------------ |
| `x`      | `Tensor`       | One vector (n,) or a batch (B, n).      | *required*   |
| `layout` | `LoaderLayout` | The loader layout. Default: "diagonal". | `'diagonal'` |

Returns:

| Type     | Description                                                         |
| -------- | ------------------------------------------------------------------- |
| `Tensor` | torch.Tensor: The angles, (n - 1,) or (B, n - 1), in the dtype of x |
| `Tensor` | (float64 for an integer x).                                         |

Raises:

| Type         | Description                                                                                   |
| ------------ | --------------------------------------------------------------------------------------------- |
| `ValueError` | If x is not 1-D or 2-D, a row is zero or not finite, or the layout does not accept the width. |
