## `eigenkit.orthogonal.torch.adapters`

Orthogonal fine-tuning: adapt a pretrained layer by rotating its weight.

An `OrthogonalAdapter` keeps the pretrained layer frozen and trains only the angles of an orthogonal matrix `$R$`, applied as `$W' = RW$` (or `$W' = WR^\top$` on the inputs). `$R$` starts at the identity, so the adapted model starts at the pretrained function, and `$RW$` has exactly the singular values of `$W$` at every step: the Lipschitz constant of the layer never changes. `merge()` folds `$R$` into the weight, so the adapter costs nothing at inference.

Example

```python
import torch.nn as nn

from eigenkit.orthogonal.torch import (
    apply_orthogonal_adapters,
    merge_adapters,
    trainable_parameters,
)

model = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 10))
apply_orthogonal_adapters(model, ["0", "2"], topology="pyramid")
params = list(trainable_parameters(model))  # the angles only
merge_adapters(model)  # plain nn.Linear layers again
```

### `OrthogonalAdapter`

Bases: `Module`

A frozen `nn.Linear` or `nn.Conv2d` whose weight is rotated by a trainable orthogonal matrix.

`side="out"` gives `$W' = RW$`, with `$R$` on the output channels; `side="in"` gives `$W' = WR^\top$`, with `$R$` on the input channels. The bias of the base layer is kept as it is. The weight of a convolution is read as the matrix `(out, in·k·k)` (see `rotate_weight`).

With the butterfly layout and a width that is not a power of two, the rotation is a `PartitionedOrthogonal` when `partition` is `None` (the default) or `True`. Any other layout takes any width it accepts, and `partition=True` uses the partitioned form for it too. The angles take the dtype and the device of the base weight.

An adapter cannot be re-initialised (see `reset_parameters`), so a model with adapters is trained with a training loop of your own: the estimators `GenericClassifier` and `GenericRegressor` re-initialise every layer in `fit()` and refuse it.

Parameters:

| Name        | Type           | Description                                                 | Default                                                                                                                 |
| ----------- | -------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `base`      | \`Linear       | Conv2d\`                                                    | The pretrained layer. Its parameters are frozen in place.                                                               |
| `topology`  | \`TopologyName | Topology\`                                                  | The layout of the rotation. Default: "butterfly".                                                                       |
| `side`      | `Side`         | "out" or "in". Default: "out".                              | `'out'`                                                                                                                 |
| `partition` | \`bool         | None\`                                                      | Whether to use PartitionedOrthogonal; None chooses it for a butterfly whose width is not a power of two. Default: None. |
| `stages`    | `int`          | The stages of a partitioned rotation. Default: 2.           | `2`                                                                                                                     |
| `perm_seed` | `int`          | The permutation seed of a partitioned rotation. Default: 0. | `0`                                                                                                                     |

#### `bias`

The bias of the base layer, `base.bias`, which is not rotated.

Returns:

| Type     | Description |
| -------- | ----------- |
| \`Tensor | None\`      |

#### `weight`

The adapted weight `rotate_weight(R, base.weight, side)`.

A property, like the `weight` of the layer it replaces, so code that reads `layer.weight` directly (`nn.MultiheadAttention` reads `out_proj.weight`) gets the adapted weight and trains the angles. It is rebuilt from the angles at every read.

Returns:

| Type     | Description                                         |
| -------- | --------------------------------------------------- |
| `Tensor` | torch.Tensor: A tensor of the shape of base.weight. |

#### `forward(x)`

Run the base layer with the adapted weight.

Parameters:

| Name | Type     | Description                  | Default    |
| ---- | -------- | ---------------------------- | ---------- |
| `x`  | `Tensor` | The input of the base layer. | *required* |

Returns:

| Type     | Description                                                         |
| -------- | ------------------------------------------------------------------- |
| `Tensor` | torch.Tensor: The output of the base layer with weight self.weight. |

#### `merge()`

A copy of the base layer with the rotation folded into its weight.

Its parameters are frozen, like those of the base layer.

Returns:

| Type     | Description |
| -------- | ----------- |
| \`Linear | Conv2d\`    |

#### `reset_parameters()`

Refuse to re-initialise: an adapter starts from its pretrained weight.

`model.apply(lambda m: m.reset_parameters())`, which the estimators `GenericClassifier` and `GenericRegressor` run at the start of `fit()`, resets the children of a module before the module itself, so by the time this method runs `base` already holds a new random weight. Training would then rotate that random weight instead of the pretrained one, without any error. Raising here stops it. Train an adapted model with a training loop of your own, as in the fine-tuning example.

Raises:

| Type           | Description |
| -------------- | ----------- |
| `RuntimeError` | Always.     |

### `apply_orthogonal_adapters(model, target_modules, topology='butterfly', side='out', freeze_rest=True, **adapter_kwargs)`

Replace layers of a model by `OrthogonalAdapter`s, in place.

Parameters:

| Name               | Type           | Description                                                                          | Default                                             |
| ------------------ | -------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------- |
| `model`            | `Module`       | The model to adapt.                                                                  | *required*                                          |
| `target_modules`   | \`str          | Iterable[str]                                                                        | Callable\[[str, Module], bool\]\`                   |
| `topology`         | \`TopologyName | Topology\`                                                                           | The layout of every rotation. Default: "butterfly". |
| `side`             | `Side`         | "out" or "in". Default: "out".                                                       | `'out'`                                             |
| `freeze_rest`      | `bool`         | Whether to freeze every parameter that is not an angle of an adapter. Default: True. | `True`                                              |
| `**adapter_kwargs` | `Any`          | Passed to OrthogonalAdapter, such as partition.                                      | `{}`                                                |

Returns:

| Type     | Description                |
| -------- | -------------------------- |
| `Module` | nn.Module: The same model. |

Raises:

| Type         | Description                                                 |
| ------------ | ----------------------------------------------------------- |
| `KeyError`   | If a name is not a module of the model.                     |
| `TypeError`  | If a named module is not nn.Linear or nn.Conv2d.            |
| `ValueError` | If nothing matches, or a named module is inside an adapter. |

### `fold_bn(conv, bn)`

Fold an eval-mode BatchNorm into the convolution before it.

Numerical contract, all in float64 on the CPU: `scale = bn.weight / sqrt(bn.running_var + bn.eps)`, `w = conv.weight * scale.view(-1, 1, 1, 1)` (one `1` per axis of the weight after the first, so `view(-1, 1, 1)` for a `Conv1d`) and `b = bn.bias - scale * bn.running_mean`. A convolution with a bias of its own gives `b = bn.bias - scale * (bn.running_mean - conv.bias)` instead. A BatchNorm with `affine=False` counts as `weight = 1` and `bias = 0`. Cast the results to the dtype you need.

Parameters:

| Name   | Type          | Description | Default       |
| ------ | ------------- | ----------- | ------------- |
| `conv` | \`Conv1d      | Conv2d      | Conv3d\`      |
| `bn`   | \`BatchNorm1d | BatchNorm2d | BatchNorm3d\` |

Returns:

| Type     | Description                                                             |
| -------- | ----------------------------------------------------------------------- |
| `Tensor` | tuple\[torch.Tensor, torch.Tensor\]: The folded weight, of the shape of |
| `Tensor` | conv.weight, and bias, float64 on the CPU.                              |

Raises:

| Type         | Description                                                                                                            |
| ------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `TypeError`  | If conv is not nn.Conv1d, nn.Conv2d or nn.Conv3d (a transposed convolution keeps its output channels on another axis). |
| `ValueError` | If the BatchNorm keeps no running statistics, or its number of features is not the number of output channels of conv.  |

### `merge_adapters(model)`

Replace every `OrthogonalAdapter` of a model by its merged layer, in place.

Parameters:

| Name    | Type     | Description                                                                             | Default    |
| ------- | -------- | --------------------------------------------------------------------------------------- | ---------- |
| `model` | `Module` | A model with adapters. If the model is itself an adapter, its merged layer is returned. | *required* |

Returns:

| Type     | Description                                                      |
| -------- | ---------------------------------------------------------------- |
| `Module` | nn.Module: The model, with plain layers where the adapters were. |

### `trainable_parameters(model)`

The parameters that require a gradient, for the optimiser.

Parameters:

| Name    | Type     | Description | Default    |
| ------- | -------- | ----------- | ---------- |
| `model` | `Module` | The model.  | *required* |

Yields:

| Type        | Description                                           |
| ----------- | ----------------------------------------------------- |
| `Parameter` | nn.Parameter: Each parameter with requires_grad=True. |
