benchmark

`eigenkit.benchmark`

Comparison of quantum and classical estimators.

The compare() function trains models that follow the scikit-learn interface on a dataset with several random seeds. It returns a BenchmarkResult with the aggregated metrics.

Example

from sklearn.svm import SVC
from eigenkit.qml import QSVC
from eigenkit.benchmark import compare

results = compare(
    models={
        "Quantum SVC": QSVC(
            feature_map="Angle", reps=1, probability=True
        ),
        "Regularized SVC": SVC(C=0.3),
        "Flexible SVC": SVC(C=3.0),
    },
    dataset="iris",
    metrics=["accuracy"],
    n_seeds=1,
)
print(list(results.fitted_models))
# ['Quantum SVC', 'Regularized SVC', 'Flexible SVC']

`BenchmarkResult` `dataclass`

Results of a benchmark comparison.

It stores the raw metric values of every seed, and aggregates them when you display them with table(), plot them with plot(), or analyse them with to_dataframe().

Each entry in raw is a dict with the keys "model", "seed", "metric" and "value".

fitted_models holds every fitted copy produced by compare(), keyed by the name of the model in the benchmark. The names usually match the class names, and repeated names are numbered (for example, SVC_0 and SVC_1). If compare(models={...}) receives a dict, its keys are used instead.

Example

from sklearn.svm import SVC
from eigenkit.qml import QSVC
from eigenkit.benchmark import compare

results = compare(
    models={
        "Quantum SVC": QSVC(
            feature_map="Angle", reps=1, probability=True
        ),
        "Regularized SVC": SVC(C=0.3),
        "Flexible SVC": SVC(C=3.0),
    },
    dataset="iris",
    metrics=["accuracy"],
    n_seeds=1,
)
print(list(results.fitted_models))
# ['Quantum SVC', 'Regularized SVC', 'Flexible SVC']
results.plot(save_path="svc_comparison.png")
df = results.to_dataframe()
X_new = [[5.1, 3.5, 1.4, 0.2]]

# Access the fitted estimators
winner = results.best_model(metric="accuracy")
preds = results.predict(X_new)
quantum_fits = results.fitted_models["Quantum SVC"]

# Run inference on new data, with all models or a subset
out = results.inference(X_new)
out = results.inference(X_new, models=["Quantum SVC"], seed=0)

`best_model(metric=None, seed='last')`

Return the fitted estimator with the best mean score on metric.

When metric is None, the default, the first metric recorded in raw that is not a timing metric is used. With the default metrics, this is usually "accuracy" for classification and "r2" for regression. If custom metrics were passed in a different order, the first of them is used.

Parameters:

Name	Type	Description	Default
`metric`	`str \| None`	The metric used for ranking, for example `"accuracy"` or `"r2"`. `None` selects the primary metric of the task. Built-in metrics use their registered direction, and custom metrics use the direction given to `custom_metric(...)`.	`None`
`seed`	`str \| int`	The seed whose estimator is returned for the winning model. `"last"`, the default, returns the estimator of the last seed. An `int` is a 0-based index into the list of seeds, in the order they ran.	`'last'`

Returns:

Type	Description
`Any`	The fitted estimator.

Raises:

Type	Description
`ValueError`	If no fitted models are stored, or if `metric` is not in the results.
`IndexError`	If `seed` is out of range.

`confusion_matrix(model_name=None, **kwargs)`

Return a scikit-learn confusion matrix computed from the stored predictions.

Parameters:

Name	Type	Description	Default
`model_name`	`str \| None`	The model whose predictions are used. `None`, the default, uses the first model with stored predictions.	`None`
`**kwargs`	`Any`	Passed on to `sklearn.metrics.confusion_matrix`.	`{}`

Returns:

Type	Description
`ndarray`	The confusion matrix, as a NumPy array.

Raises:

Type	Description
`ValueError`	If no predictions are stored (run `compare(..., store_predictions=True)`), or if there are none for `model_name`.
`ImportError`	If scikit-learn is not installed.

`inference(X, models=None, seed='last')`

Run every fitted model, or a subset of them, on new data.

Parameters:

Name	Type	Description	Default
`X`	`Any`	The input features to predict on.	required
`models`	`list[str] \| None`	The names of the models to run. `None`, the default, runs all fitted models. Use the exact keys of `fitted_models`: the names based on the class for a list input (for example, `"SVC"` or `"SVC_0"`), or your own keys for a dict input.	`None`
`seed`	`str \| int`	The seed whose estimator is used for each model. `"last"`, the default, uses the estimator of the last seed. An `int` is a 0-based index into the list of seeds, in the order they ran.	`'last'`

Returns:

Type	Description
`dict[str, dict[str, Any]]`	A dict keyed by model name. Each value contains two entries: `"predictions"`: the array returned by `estimator.predict(X)`. `"probabilities"`: the array returned by `estimator.predict_proba(X)` for classifiers that support it, and `None` for regressors and for classifiers without `predict_proba`.

Raises:

Type	Description
`ValueError`	If no fitted models are stored, or if an unknown model name is requested.
`IndexError`	If `seed` is out of range for any model.

Example

out = results.inference(X_new)
out = results.inference(X_new, models=["Flexible SVC"], seed=0)
preds = out["Flexible SVC"]["predictions"]
probs = out["Flexible SVC"]["probabilities"]  # None for regressors

`load(path)` `classmethod`

Load numeric benchmark results saved by BenchmarkResult.save().

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	The JSON file to read.	required

Returns:

Type	Description
`'BenchmarkResult'`	A `BenchmarkResult` with the stored values. It has no fitted models, because `save()` does not store them.

`plot(save_path=None, theme='light')`

Plot a grouped bar chart with error bars.

Parameters:

Name	Type	Description	Default
`save_path`	`str \| None`	If given, the figure is saved to this path instead of being shown.	`None`
`theme`	`str`	The plot theme, `"light"` (the default) or `"dark"`.	`'light'`

Raises:

Type	Description
`ImportError`	If matplotlib is not installed.

`plot_history(save_path=None, theme='light', seed='last')`

Plot the stored training and validation loss curves.

Models without a loss history are left out, and a warning names them.

Parameters:

Name	Type	Description	Default
`save_path`	`str \| None`	If given, the figure is saved to this path instead of being shown.	`None`
`theme`	`str`	The plot theme, `"light"` (the default) or `"dark"`.	`'light'`
`seed`	`str \| int`	The run to draw: `"last"` (the default), or an integer index into the list of seeds. With `"last"`, a model whose last seed failed shows its most recent recorded curve.	`'last'`

Raises:

Type	Description
`TypeError`	If `seed` is neither `"last"` nor an `int`.
`ValueError`	If no model has a training or validation loss curve.
`ImportError`	If matplotlib is not installed.

`predict(X, metric=None)`

Predict with the best model ranked by metric.

This is a shortcut for best_model(). It calls predict() on the estimator returned by best_model(metric=metric).

Parameters:

Name	Type	Description	Default
`X`	`Any`	The input features.	required
`metric`	`str \| None`	The metric used to select the best model. `None`, the default, selects the primary metric of the task: `"accuracy"` for classification and `"r2"` for regression.	`None`

Returns:

Type	Description
`Any`	The predictions of the best estimator's `predict()` method.

`save(path)`

Save the numeric results of the benchmark to a JSON file.

The fitted estimators are not saved, by design.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	The file to write. If it is an existing directory, a file named `bench_<timestamp>.json` is created inside it. If it has no extension, `.json` is added.	required

Returns:

Type	Description
`Path`	The path of the file that was written.

`significance_test(metric, method='wilcoxon')`

Return a matrix of p-values for each pair of models.

The p-values compare the values of metric on the seeds that both models share.

Parameters:

Name	Type	Description	Default
`metric`	`str`	The metric to compare.	required
`method`	`str`	`"wilcoxon"` (the default) for a Wilcoxon signed-rank test, or `"ttest"` for a paired t-test.	`'wilcoxon'`

Returns:

Type	Description
`'pd.DataFrame'`	A pandas DataFrame with one row and one column per model. The diagonal is `1`, and a pair with fewer than two shared seeds has the value `NaN`.

Raises:

Type	Description
`ImportError`	If pandas or scipy is not installed.
`ValueError`	If `method` is not supported, or if fewer than two models have values for `metric`.

`table(show_significance=False)`

Return the results as a text table.

The table is drawn with Unicode box-drawing characters and needs no optional dependency.

Parameters:

Name	Type	Description	Default
`show_significance`	`bool`	If `True`, mark the best model with `` or `*` when a Wilcoxon signed-rank test against another model gives `p < 0.05` or `p < 0.01`. The test needs at least `6` seeds shared by both models, so with fewer seeds no model is marked, and a warning explains why. The test needs pandas, from the `benchmark` extra.	`False`

Returns:

Type	Description
`str`	A string with one row per model and one column per metric. Each value is shown as `mean ± std`.

Raises:

Type	Description
`ImportError`	If `show_significance` is `True` and pandas is not installed.

`to_dataframe()`

Return the raw results of every seed as a pandas DataFrame.

Returns:

Type	Description
`'pd.DataFrame'`	A DataFrame with the columns `model`, `seed`, `metric` and `value`, and one row for each entry of `raw`.

Raises:

Type	Description
`ImportError`	If pandas is not installed.

`compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)`

Compare eigenkit models and built-in scikit-learn models on a dataset.

Each model must be a built-in scikit-learn estimator (from the sklearn package), or a subclass of BaseQuantumEstimator, BaseQuantumClassifier or BaseQuantumRegressor. For every seed, each model is copied with sklearn.base.clone().

Parameters:

Name	Type	Description	Default
`models`	`list \| dict[str, Any]`	The estimator instances to compare. If a list is passed, the display names come from `type(model).__name__`, and repeated names are numbered (for example, `SVC_0` and `SVC_1`). If a dict is passed, its keys are used as display names. Dict keys must be non-empty strings, with no whitespace at the start or the end and no control characters. Use a different key for each model. If the same key appears twice in a Python dict literal, Python keeps only the last model before `compare()` receives the dict. When `save_dir` is set, the keys must also be valid prefixes for file names.	required
`dataset`	`str \| dict`	A dataset name, or a dict that is already split, with `X_train`, `y_train`, `X_test` and `y_test`. Such a dict can also include `X_val` and `y_val`.	`'breast_cancer_wisconsin'`
`metrics`	`list \| None`	The metrics to compute. For classification, the options are `accuracy`, `f1`, `precision`, `recall`, `roc_auc`, `fit_time` and `predict_time`. For regression, they are `r2`, `mae`, `rmse`, `mse`, `fit_time` and `predict_time`. A custom function must be wrapped with `custom_metric(...)`, so that its direction (higher or lower is better) is explicit. The default depends on the type of task of the first model.	`None`
`n_seeds`	`int`	The number of seeds to average over. The default is `5`.	`5`
`test_size`	`float`	The fraction of the data kept for the final test set.	`0.15`
`show_progress`	`bool`	Whether to print the progress of the benchmark, with the current model and an estimate of the remaining time. The default is `False`, which runs silently.	`False`
`save_dir`	`str \| Path \| None`	If set, each fitted model is saved in this directory, as `{ModelName}_seed{N}.eigenkit` with `save()`, or as `{ModelName}_seed{N}.joblib` with joblib for scikit-learn estimators.	`None`
`store_predictions`	`bool`	If `True`, the result keeps `y_true`, `y_pred` and `y_prob` for the last seed of each model.	`False`
`validation_size`	`float`	The fraction of the data kept for validation.	`0.15`

Returns:

Type	Description
`BenchmarkResult`	An object with the results, and methods such as `table()`, `plot()` and `to_dataframe()`.

Raises:

Type	Description
`ValueError`	If `models` is empty, if a model name is not valid, or if `dataset` is not well formed.
`TypeError`	If a model name is not a string, or if a model is not a supported type of estimator.

`custom_metric(fn, *, direction, name=None)`

Define a custom metric and the direction in which it improves.

Parameters:

Name	Type	Description	Default
`fn`	`Callable[[ndarray, ndarray], float]`	The metric function. It receives `(y_true, y_pred)` and returns a number.	required
`direction`	`MetricDirection`	`"higher"` if higher values are better, or `"lower"` if lower values are better.	required
`name`	`str \| None`	The column label of the metric. The default is the name of `fn`.	`None`

Returns:

Type	Description
`CustomMetric`	A metric specification to pass to `compare(metrics=[...])`.

`load_checkpoint(path, *, model=None)`

Load a model checkpoint written by compare(..., save_dir=...).

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	The path to the checkpoint file (`.eigenkit` or `.joblib`).	required
`model`	`Any`	The `nn.Module` architecture for `GenericRegressor` or `GenericClassifier` checkpoints. It is ignored for other types of estimator.	`None`

Returns:

Type	Description
`Any`	The loaded estimator.

Raises:

Type	Description
`TypeError`	If a `.joblib` file does not contain a scikit-learn estimator.
`ValueError`	If the file extension is not supported.

`quick_baselines()`

Return a list of common scikit-learn classifiers with sensible defaults.

The list contains a SVC with a radial basis function (RBF) kernel and probabilities, so that roc_auc can be computed, a RandomForestClassifier, a LogisticRegression and a KNeighborsClassifier. XGBClassifier and LGBMClassifier are added when their packages can be imported.

scikit-learn 1.9 deprecates SVC(probability=True). From that version, the support vector classifier is wrapped as CalibratedClassifierCV(SVC(), ensemble=False), the replacement that scikit-learn documents, so the results table calls it CalibratedClassifierCV.

Returns:

Type	Description
`list[Any]`	A list of classifiers that follow the scikit-learn interface, created but not fitted.

Example

from eigenkit.benchmark import compare, quick_baselines
results = compare(models=quick_baselines(), dataset="iris", n_seeds=2)

`timer()`

Measure the wall-clock time of a block of code.

Yields:

Type	Description
`dict`	A dict. When the `with` block ends, its `"elapsed"` key holds the elapsed time in seconds.

Example

with timer() as t:
    do_work()
print(t["elapsed"])

benchmark

eigenkit.benchmark

BenchmarkResult dataclass

best_model(metric=None, seed='last')

confusion_matrix(model_name=None, **kwargs)

inference(X, models=None, seed='last')

load(path) classmethod

plot(save_path=None, theme='light')

plot_history(save_path=None, theme='light', seed='last')

predict(X, metric=None)

save(path)

significance_test(metric, method='wilcoxon')

table(show_significance=False)

to_dataframe()

compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)

custom_metric(fn, *, direction, name=None)

load_checkpoint(path, *, model=None)

quick_baselines()

timer()

`eigenkit.benchmark`

`BenchmarkResult` `dataclass`

`best_model(metric=None, seed='last')`

`confusion_matrix(model_name=None, **kwargs)`

`inference(X, models=None, seed='last')`

`load(path)` `classmethod`

`plot(save_path=None, theme='light')`

`plot_history(save_path=None, theme='light', seed='last')`

`predict(X, metric=None)`

`save(path)`

`significance_test(metric, method='wilcoxon')`

`table(show_significance=False)`

`to_dataframe()`

`compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)`

`custom_metric(fn, *, direction, name=None)`

`load_checkpoint(path, *, model=None)`

`quick_baselines()`

`timer()`