Skip to content

functional

eigenkit.orthogonal.torch.functional

Functional forms of the orthogonal operations.

Example
import torch

from eigenkit.orthogonal.torch import Orthogonal, rotate_weight

conv = torch.nn.Conv2d(16, 32, 3)
R = Orthogonal(32, init="random", std=1.0).matrix()
W = rotate_weight(R, conv.weight)  # same shape, same singular values

rotate_weight(R, W, side='out')

Rotate the output or the input channels of a weight tensor by an orthogonal matrix.

The weight is read as the matrix W.reshape(out, in·k·k). Rotating its rows (side="out", \(RW\)) or its input channels (side="in", \(W (R^\top \otimes I)\)) keeps every singular value of that matrix, so the operator norm, and the Lipschitz constant, of the layer are unchanged. For a convolution R acts pixel by pixel, so the same holds for the convolution operator.

Numerical contract for side="out": (R @ W.reshape(W.shape[0], -1)).reshape(W.shape), which is R @ W for a 2-D weight. side="in" computes torch.einsum("oi...,ji->oj...", W, R).contiguous(), which is W @ R.T for a 2-D weight.

Parameters:

Name Type Description Default
R Tensor

An orthogonal matrix, (out, out) for side="out" or (in, in) for side="in".

required
W Tensor

A weight (out, in) or (out, in, *kernel).

required
side Side

"out" or "in". Default: "out".

'out'

Returns:

Type Description
Tensor

torch.Tensor: The rotated weight, of the shape of W.

Raises:

Type Description
ValueError

If side is unknown or R does not match the axis.

unary_loader_angles(x, layout='diagonal')

The input angles of UnaryLoader that load \(x / \lVert x \rVert\), differentiable in x.

The PyTorch form of eigenkit.orthogonal.unary_loader_angles: the same gate plan, with torch.atan2 on the signed component or the norm of the wires each gate keeps and passes on. Each row is first scaled by a power of two, which is exact, so the angles do not depend on the scale of x, at any finite scale.

Parameters:

Name Type Description Default
x Tensor

One vector (n,) or a batch (B, n).

required
layout LoaderLayout

The loader layout. Default: "diagonal".

'diagonal'

Returns:

Type Description
Tensor

torch.Tensor: The angles, (n - 1,) or (B, n - 1), in the dtype of x

Tensor

(float64 for an integer x).

Raises:

Type Description
ValueError

If x is not 1-D or 2-D, a row is zero or not finite, or the layout does not accept the width.