functional
eigenkit.orthogonal.torch.functional
Functional forms of the orthogonal operations.
Example
rotate_weight(R, W, side='out')
Rotate the output or the input channels of a weight tensor by an orthogonal matrix.
The weight is read as the matrix W.reshape(out, in·k·k). Rotating its rows
(side="out", \(RW\)) or its input channels (side="in",
\(W (R^\top \otimes I)\)) keeps
every singular value of that matrix, so the operator norm, and the Lipschitz
constant, of the layer are unchanged. For a convolution R acts pixel by
pixel, so the same holds for the convolution operator.
Numerical contract for side="out":
(R @ W.reshape(W.shape[0], -1)).reshape(W.shape), which is R @ W for a 2-D
weight. side="in" computes
torch.einsum("oi...,ji->oj...", W, R).contiguous(), which is W @ R.T for a
2-D weight.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
R
|
Tensor
|
An orthogonal matrix, |
required |
W
|
Tensor
|
A weight |
required |
side
|
Side
|
|
'out'
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The rotated weight, of the shape of |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
unary_loader_angles(x, layout='diagonal')
The input angles of UnaryLoader that load \(x / \lVert x \rVert\),
differentiable in x.
The PyTorch form of eigenkit.orthogonal.unary_loader_angles: the same gate
plan, with torch.atan2 on the signed component or the norm of the wires each
gate keeps and passes on. Each row is first scaled by a power of two, which
is exact, so the angles do not depend on the scale of x, at any finite
scale.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
One vector |
required |
layout
|
LoaderLayout
|
The loader layout. Default: |
'diagonal'
|
Returns:
| Type | Description |
|---|---|
Tensor
|
torch.Tensor: The angles, |
Tensor
|
(float64 for an integer |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |