Benchmark

eigenkit.benchmark compares quantum and classical models. A single call to compare() trains several estimators on one dataset with several random seeds, and returns a BenchmarkResult. The result can show the scores as a formatted table, plot them as grouped bar charts, and export them as a pandas DataFrame.

The benchmark needs the benchmark extra, which installs matplotlib, pandas and scikit-learn: pip install eigenkit[benchmark]. The quantum models that you compare need their own extras: qml for QSVC and QSVR, and torch for HQNC, HQNR and the other differentiable estimators.

By default, named datasets are split into 70 % for training, 15 % for validation and 15 % for testing. The metrics are computed on the test split. The validation split is passed only to estimators whose fit() method accepts X_val and y_val.

compare() repeats the split into training, validation and test sets for each seed. It does not use k-fold cross-validation. For k-fold cross-validation, call scikit-learn directly, for example with cross_val_score or GridSearchCV(cv=...).

from eigenkit import QSVC, HQNC
from eigenkit.benchmark import compare
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier

result = compare(
    models=[
        QSVC(feature_map="Angle", reps=2),
        HQNC(
            in_features=4,
            depth=1,
            epochs=3,
            show_progress=False,
        ),
        SVC(C=0.1),
        RandomForestClassifier(n_estimators=20),
    ],
    dataset="iris",
    metrics=["accuracy", "f1"],
    n_seeds=3,
    validation_size=0.15,
    test_size=0.15,
)
print(result.table())
result.plot(save_path="comparison.png")
result.plot_history(save_path="losses.png")

The run used for this guide produced the following table:

┌────────────────────────┬───────────────┬───────────────┐
│ Model                  │ Accuracy      │ F1            │
├────────────────────────┼───────────────┼───────────────┤
│ QSVC                   │ 0.989 ± 0.019 │ 0.989 ± 0.019 │
│ HQNC                   │ 0.389 ± 0.096 │ 0.242 ± 0.131 │
│ SVC                    │ 0.911 ± 0.051 │ 0.909 ± 0.053 │
│ RandomForestClassifier │ 0.967 ± 0.033 │ 0.966 ± 0.034 │
└────────────────────────┴───────────────┴───────────────┘

Each cell shows the mean and the standard deviation across seeds, as mean ± std.

Supported models

Each entry in models must be one of the following:

An eigenkit estimator, that is, a subclass of BaseQuantumEstimator, BaseQuantumClassifier or BaseQuantumRegressor, such as HQNC, QSVC or CustomQuantumClassifier.
A built-in scikit-learn estimator, that is, a class from the sklearn package, such as SVC, RandomForestClassifier or LinearRegression.

Models from other libraries are not accepted yet. This includes XGBoost, LightGBM and custom BaseEstimator subclasses defined outside sklearn.

Every model must implement fit(X, y) and predict(X). Internally, compare() uses sklearn.base.clone to make a fresh copy of each model for every seed.

`compare()`

models is the only required argument. Every other argument has a default value.

Argument	Default	Description
`models`	required	A list of estimator instances, or a dict that maps display names to estimators.
`dataset`	`"breast_cancer_wisconsin"`	A dataset name from the registry, or a dict with `X_train`, `y_train`, `X_test` and `y_test`.
`metrics`	`None`	The names of the metrics to compute, or `custom_metric(...)` specifications. `None` selects the default metrics for the task, listed below.
`n_seeds`	`5`	The number of seeds to average over.
`test_size`	`0.15`	The fraction of the data kept for the final test metrics.
`show_progress`	`False`	Whether to print progress: the current seed, the current model and the estimated time remaining.
`save_dir`	`None`	If set, the directory where each fitted model is saved after every seed (see below).
`store_predictions`	`False`	If `True`, the result keeps `y_true`, `y_pred` and, when available, `y_prob` for the last seed.
`validation_size`	`0.15`	The fraction of the data kept for validation.

If you call compare(models) with no other argument, the benchmark uses the breast_cancer_wisconsin dataset. The default test_size here is 0.15, which is different from the 0.2 that load_dataset uses.

Set show_progress=True to see the progress of the run: which model is running and how many runs are left. By default nothing is printed, which keeps notebooks and continuous integration (CI) logs quiet.

Pass a dict to choose the names that appear in tables, plots, progress messages and saved file names. This is useful when you compare several instances of the same estimator class:

from sklearn.svm import SVC
from eigenkit import QSVC
from eigenkit.benchmark import compare

result = compare(
    models={
        "Quantum SVC": QSVC(feature_map="Angle", reps=1, probability=True),
        "Regularized SVC": SVC(C=0.3),
        "Flexible SVC": SVC(C=3.0),
    },
    dataset="iris",
    metrics=["accuracy"],
    n_seeds=1,
)
print(list(result.fitted_models))

['Quantum SVC', 'Regularized SVC', 'Flexible SVC']

Dict keys must not have whitespace at the start or the end, and must not contain control characters. When save_dir is set, the keys are also used as prefixes of file names. In that case, they must not end with a full stop and must not contain any of these characters: <>:"/\|?*. Use a different key for each model. If the same key appears twice in a Python dict literal, Python keeps only the last model before compare() receives the dict.

If you pass a dataset dict that is already split and you have a validation set, include X_val and y_val. If they are left out, compare() uses only the training and test sets that you provided.

Saving fitted models

When save_dir is set, each model is written to a file after fit():

Model type	Output file
eigenkit	`{ModelName}_seed{N}.eigenkit`, written with `save()`
built-in scikit-learn	`{ModelName}_seed{N}.joblib`, written with `joblib.dump`

compare(
    models=[HQNC(in_features=4, depth=2), SVC()],
    dataset="iris",
    n_seeds=2,
    save_dir="runs/iris_benchmark/",
)
# runs/iris_benchmark/HQNC_seed0.eigenkit, HQNC_seed1.eigenkit, SVC_seed0.joblib, ...

ModelName is the name of the model in the benchmark. When the same class appears more than once in a list, the names are numbered (SVC_0, SVC_1). When models is a dict, ModelName is the dict key.

To load an eigenkit checkpoint, use eigenkit.benchmark.load_checkpoint() or the estimator's load() method, for example HQNC.load(...). To load a scikit-learn checkpoint, use eigenkit.benchmark.load_checkpoint() or joblib.load(...). .joblib files use pickle internally, so only load them from sources that you trust.

See Model persistence for more details.

Classification metrics

The available metrics are accuracy, f1 (the F1 score), precision, recall, roc_auc (the area under the receiver operating characteristic curve), fit_time and predict_time.

When metrics is None, the benchmark computes accuracy, f1, fit_time and predict_time.

roc_auc needs predict_proba. QSVC with probability=True and HQNC both provide it. For a model without predict_proba, requesting roc_auc issues a UserWarning, and the metric is skipped.

Regression metrics

The available metrics are r2 (the coefficient of determination), mae (the mean absolute error), rmse (the root mean squared error), mse (the mean squared error), fit_time and predict_time. The type of task is taken from the first model: a model that inherits from RegressorMixin makes it a regression task.

When metrics is None, the benchmark computes r2, mae, mse, fit_time and predict_time. rmse is not computed by default, so request it by name if you need it.

Custom metrics

A custom metric must be wrapped with custom_metric(fn, *, direction, name=None). The function receives (y_true, y_pred). direction states whether higher or lower values are better. name sets the column label, which otherwise comes from the name of the function:

from eigenkit.benchmark import custom_metric


def error_rate(y_true, y_pred):
    return (y_true != y_pred).mean()


def match_rate(y_true, y_pred):
    return (y_true == y_pred).mean()


metrics = [
    custom_metric(error_rate, direction="lower"),
    custom_metric(match_rate, direction="higher"),
]

`BenchmarkResult`

compare() returns a BenchmarkResult. It holds the raw values for every seed, the fitted copies of the models, and methods to display, analyse and use the results.

Attributes

Attribute	Type	Description
`raw`	`list[dict]`	One row per value, with the keys `"model"`, `"seed"`, `"metric"` and `"value"`.
`fitted_models`	`dict[str, list]`	The fitted estimators for each model name, one per seed.
`predictions`	`dict[str, dict]`	The `y_true`, `y_pred` and `y_prob` of the last seed for each model, when `store_predictions=True`.
`model_artifacts`	`dict[str, list]`	The paths of the checkpoint files written to `save_dir`, grouped by model name.
`loss_curves`	`dict[str, list]`	The training and validation loss curves for each seed, when the fitted estimators record them.
`metric_directions`	`dict[str, str]`	The direction, `"higher"` or `"lower"`, of every metric in the run.

Methods

Method	Description
`table(show_significance=False)`	A text table with `mean ± std` for each model and metric. It needs no optional dependency. With `show_significance=True`, the best model is marked with `` or `*` when a test against another model gives `p < 0.05` or `p < 0.01`.
`plot(save_path=None, theme="light")`	A grouped bar chart with error bars. `theme` is `"light"` or `"dark"`. It needs `matplotlib`.
`plot_history(save_path=None, theme="light", seed="last")`	The training and validation loss curves of the models that record them. `seed` chooses the run to draw: `"last"` or an integer index. It needs `matplotlib`.
`to_dataframe()`	A DataFrame with one row per seed and metric. It needs `pandas`.
`best_model(metric=None, seed="last")`	The fitted estimator with the best mean on `metric`, taking the direction of the metric into account.
`predict(X, metric=None)`	A shortcut for `best_model(metric=metric).predict(X)`.
`inference(X, models=None, seed="last")`	The predictions and probabilities of all fitted models, or of the selected ones.
`significance_test(metric, method="wilcoxon")`	A matrix of p-values for each pair of models, computed from the scores of each seed. `method="ttest"` is also supported. It needs `pandas` and `scipy`.
`confusion_matrix(model_name=None, **kwargs)`	A scikit-learn confusion matrix computed from the stored predictions. `**kwargs` are passed to `sklearn.metrics.confusion_matrix`. It needs `store_predictions=True`.
`save(path)`	Writes the numeric results to a JSON file. The fitted estimators are not saved.
`BenchmarkResult.load(path)`	Restores a result from a JSON file written by `save()`.

The significance markers come from a Wilcoxon signed-rank test, which needs at least 6 seeds shared by the two models. With the default n_seeds=5, table(show_significance=True) cannot mark any model, and it issues a warning instead.

To use plotting, DataFrame export and significance tests, install the optional extra:

pip install eigenkit[benchmark]

plot_history() draws the models that record a loss history, and warns about the ones that do not. For example, HQNC produces training and validation curves, while SVC appears only in the metric table and the bar chart.

Accessing fitted models

Every fitted copy produced by compare() is stored in result.fitted_models, a dict[str, list[estimator]] keyed by the name of the model in the benchmark. With a list of models, the names come from the class (SVC, or SVC_0 and SVC_1 when the same class appears more than once). With a dict, the keys are exactly the names that you passed. Each value is a list of fitted copies, one per seed.

print(list(result.fitted_models))
# ['Quantum SVC', 'Regularized SVC', 'Flexible SVC']

quantum_fits = result.fitted_models["Quantum SVC"]
model_seed0 = quantum_fits[0]

`best_model(metric, seed)`

Returns the fitted estimator with the best mean on metric. When metric is not given, the primary metric is chosen automatically. It is the first metric in the results that is not a timing metric, usually "accuracy" for classification and "r2" for regression with the default metrics.

winner = result.best_model()  # auto-selects metric
winner = result.best_model(metric="f1", seed=0)  # explicit metric, seed 0
winner = result.best_model(metric="r2")  # explicit for regression

Parameter	Default	Description
`metric`	`None`	The metric used for ranking. `None` selects the primary metric of the task. Built-in metrics and custom metrics use their registered direction.
`seed`	`"last"`	The seed whose copy is returned: `"last"`, or a 0-based `int`.

`predict(X, metric)`

This is a shortcut for best_model(metric=metric).predict(X). It returns only the predictions of the selected model. Use inference() when you also need probabilities, or the outputs of several models.

preds = result.predict(X_new)  # auto-selects metric
preds = result.predict(X_new, metric="r2")  # explicit for regression

X is passed directly to the selected estimator. The return value is what that estimator's predict(X) returns, usually an array of shape (n_samples,).

Inference on new data

result.inference(X) runs all fitted models on new data and returns their predictions together with the class probabilities, in a single call.

out = result.inference(X_new)
# {
#   "HQNC": {"predictions": array([...]), "probabilities": array([[...]])},
#   "SVC":  {"predictions": array([...]), "probabilities": None},  # no predict_proba
# }

preds = out["HQNC"]["predictions"]  # shape (n_samples,)
probs = out["HQNC"]["probabilities"]  # shape (n_samples, n_classes), or None

Output key	Type	Description
`predictions`	`ndarray`	The output of `estimator.predict(X)`.
`probabilities`	`ndarray` or `None`	The output of `predict_proba(X)` for classifiers that support it, and `None` for regressors and for classifiers without `predict_proba`.

Selecting models and seed

By default, inference() runs all models, each with the copy fitted on the last seed. You can restrict both:

# Only a subset of models
out = result.inference(X_new, models=["SVC"])

# Specific seed (0-based index into the per-seed list)
out = result.inference(X_new, seed=0)
out = result.inference(X_new, models=["HQNC", "SVC"], seed=1)

Passing an unknown model name raises a ValueError that lists the available names.

Probabilities

probabilities is an array of shape (n_samples, n_classes) when the estimator provides predict_proba, and None otherwise:

It is filled for HQNC, QSVC(probability=True), CustomQuantumClassifier and SVC(probability=True).
It is None for all regressors, for classifiers without predict_proba (for example, a plain SVC() without probability=True), and for a QSVC without probability=True, which also issues a warning.

`quick_baselines()`

from eigenkit import QSVC
from eigenkit.benchmark import compare, quick_baselines

results = compare(
    models=[QSVC(feature_map="Angle"), *quick_baselines()],
    dataset="iris",
    n_seeds=2,
)

quick_baselines() returns built-in scikit-learn classifiers with sensible defaults: a SVC with a radial basis function (RBF) kernel and probabilities, a RandomForestClassifier, a LogisticRegression and a KNeighborsClassifier. scikit-learn 1.9 deprecates SVC(probability=True), so from that version the support vector classifier is wrapped as CalibratedClassifierCV(SVC(), ensemble=False).

Timing a block of code

timer() measures the wall-clock time of the code inside a with block. For example, it can measure the time needed to compute one kernel matrix on two providers:

import numpy as np

from eigenkit import QuantumKernel
from eigenkit.benchmark import timer

X = np.random.default_rng(0).uniform(0, np.pi, size=(150, 4))
for provider in ("numpy", "pennylane"):  # "pennylane" needs the pennylane extra
    kernel = QuantumKernel(n_features=4, feature_map="AngleCNOT", provider=provider)
    with timer() as t:
        kernel.compute_kernel_matrix(X)
    print(provider, f"{t['elapsed']:.2f} s")

API reference

`eigenkit.benchmark`

Comparison of quantum and classical estimators.

The compare() function trains models that follow the scikit-learn interface on a dataset with several random seeds. It returns a BenchmarkResult with the aggregated metrics.

Example

from sklearn.svm import SVC
from eigenkit.qml import QSVC
from eigenkit.benchmark import compare

results = compare(
    models={
        "Quantum SVC": QSVC(
            feature_map="Angle", reps=1, probability=True
        ),
        "Regularized SVC": SVC(C=0.3),
        "Flexible SVC": SVC(C=3.0),
    },
    dataset="iris",
    metrics=["accuracy"],
    n_seeds=1,
)
print(list(results.fitted_models))
# ['Quantum SVC', 'Regularized SVC', 'Flexible SVC']

`BenchmarkResult` `dataclass`

Results of a benchmark comparison.

It stores the raw metric values of every seed, and aggregates them when you display them with table(), plot them with plot(), or analyse them with to_dataframe().

Each entry in raw is a dict with the keys "model", "seed", "metric" and "value".

fitted_models holds every fitted copy produced by compare(), keyed by the name of the model in the benchmark. The names usually match the class names, and repeated names are numbered (for example, SVC_0 and SVC_1). If compare(models={...}) receives a dict, its keys are used instead.

Example

from sklearn.svm import SVC
from eigenkit.qml import QSVC
from eigenkit.benchmark import compare

results = compare(
    models={
        "Quantum SVC": QSVC(
            feature_map="Angle", reps=1, probability=True
        ),
        "Regularized SVC": SVC(C=0.3),
        "Flexible SVC": SVC(C=3.0),
    },
    dataset="iris",
    metrics=["accuracy"],
    n_seeds=1,
)
print(list(results.fitted_models))
# ['Quantum SVC', 'Regularized SVC', 'Flexible SVC']
results.plot(save_path="svc_comparison.png")
df = results.to_dataframe()
X_new = [[5.1, 3.5, 1.4, 0.2]]

# Access the fitted estimators
winner = results.best_model(metric="accuracy")
preds = results.predict(X_new)
quantum_fits = results.fitted_models["Quantum SVC"]

# Run inference on new data, with all models or a subset
out = results.inference(X_new)
out = results.inference(X_new, models=["Quantum SVC"], seed=0)

`best_model(metric=None, seed='last')`

Return the fitted estimator with the best mean score on metric.

When metric is None, the default, the first metric recorded in raw that is not a timing metric is used. With the default metrics, this is usually "accuracy" for classification and "r2" for regression. If custom metrics were passed in a different order, the first of them is used.

Parameters:

Name	Type	Description	Default
`metric`	`str \| None`	The metric used for ranking, for example `"accuracy"` or `"r2"`. `None` selects the primary metric of the task. Built-in metrics use their registered direction, and custom metrics use the direction given to `custom_metric(...)`.	`None`
`seed`	`str \| int`	The seed whose estimator is returned for the winning model. `"last"`, the default, returns the estimator of the last seed. An `int` is a 0-based index into the list of seeds, in the order they ran.	`'last'`

Returns:

Type	Description
`Any`	The fitted estimator.

Raises:

Type	Description
`ValueError`	If no fitted models are stored, or if `metric` is not in the results.
`IndexError`	If `seed` is out of range.

`confusion_matrix(model_name=None, **kwargs)`

Return a scikit-learn confusion matrix computed from the stored predictions.

Parameters:

Name	Type	Description	Default
`model_name`	`str \| None`	The model whose predictions are used. `None`, the default, uses the first model with stored predictions.	`None`
`**kwargs`	`Any`	Passed on to `sklearn.metrics.confusion_matrix`.	`{}`

Returns:

Type	Description
`ndarray`	The confusion matrix, as a NumPy array.

Raises:

Type	Description
`ValueError`	If no predictions are stored (run `compare(..., store_predictions=True)`), or if there are none for `model_name`.
`ImportError`	If scikit-learn is not installed.

`inference(X, models=None, seed='last')`

Run every fitted model, or a subset of them, on new data.

Parameters:

Name	Type	Description	Default
`X`	`Any`	The input features to predict on.	required
`models`	`list[str] \| None`	The names of the models to run. `None`, the default, runs all fitted models. Use the exact keys of `fitted_models`: the names based on the class for a list input (for example, `"SVC"` or `"SVC_0"`), or your own keys for a dict input.	`None`
`seed`	`str \| int`	The seed whose estimator is used for each model. `"last"`, the default, uses the estimator of the last seed. An `int` is a 0-based index into the list of seeds, in the order they ran.	`'last'`

Returns:

Type	Description
`dict[str, dict[str, Any]]`	A dict keyed by model name. Each value contains two entries: `"predictions"`: the array returned by `estimator.predict(X)`. `"probabilities"`: the array returned by `estimator.predict_proba(X)` for classifiers that support it, and `None` for regressors and for classifiers without `predict_proba`.

Raises:

Type	Description
`ValueError`	If no fitted models are stored, or if an unknown model name is requested.
`IndexError`	If `seed` is out of range for any model.

Example

out = results.inference(X_new)
out = results.inference(X_new, models=["Flexible SVC"], seed=0)
preds = out["Flexible SVC"]["predictions"]
probs = out["Flexible SVC"]["probabilities"]  # None for regressors

`load(path)` `classmethod`

Load numeric benchmark results saved by BenchmarkResult.save().

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	The JSON file to read.	required

Returns:

Type	Description
`'BenchmarkResult'`	A `BenchmarkResult` with the stored values. It has no fitted models, because `save()` does not store them.

`plot(save_path=None, theme='light')`

Plot a grouped bar chart with error bars.

Parameters:

Name	Type	Description	Default
`save_path`	`str \| None`	If given, the figure is saved to this path instead of being shown.	`None`
`theme`	`str`	The plot theme, `"light"` (the default) or `"dark"`.	`'light'`

Raises:

Type	Description
`ImportError`	If matplotlib is not installed.

`plot_history(save_path=None, theme='light', seed='last')`

Plot the stored training and validation loss curves.

Models without a loss history are left out, and a warning names them.

Parameters:

Name	Type	Description	Default
`save_path`	`str \| None`	If given, the figure is saved to this path instead of being shown.	`None`
`theme`	`str`	The plot theme, `"light"` (the default) or `"dark"`.	`'light'`
`seed`	`str \| int`	The run to draw: `"last"` (the default), or an integer index into the list of seeds. With `"last"`, a model whose last seed failed shows its most recent recorded curve.	`'last'`

Raises:

Type	Description
`TypeError`	If `seed` is neither `"last"` nor an `int`.
`ValueError`	If no model has a training or validation loss curve.
`ImportError`	If matplotlib is not installed.

`predict(X, metric=None)`

Predict with the best model ranked by metric.

This is a shortcut for best_model(). It calls predict() on the estimator returned by best_model(metric=metric).

Parameters:

Name	Type	Description	Default
`X`	`Any`	The input features.	required
`metric`	`str \| None`	The metric used to select the best model. `None`, the default, selects the primary metric of the task: `"accuracy"` for classification and `"r2"` for regression.	`None`

Returns:

Type	Description
`Any`	The predictions of the best estimator's `predict()` method.

`save(path)`

Save the numeric results of the benchmark to a JSON file.

The fitted estimators are not saved, by design.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	The file to write. If it is an existing directory, a file named `bench_<timestamp>.json` is created inside it. If it has no extension, `.json` is added.	required

Returns:

Type	Description
`Path`	The path of the file that was written.

`significance_test(metric, method='wilcoxon')`

Return a matrix of p-values for each pair of models.

The p-values compare the values of metric on the seeds that both models share.

Parameters:

Name	Type	Description	Default
`metric`	`str`	The metric to compare.	required
`method`	`str`	`"wilcoxon"` (the default) for a Wilcoxon signed-rank test, or `"ttest"` for a paired t-test.	`'wilcoxon'`

Returns:

Type	Description
`'pd.DataFrame'`	A pandas DataFrame with one row and one column per model. The diagonal is `1`, and a pair with fewer than two shared seeds has the value `NaN`.

Raises:

Type	Description
`ImportError`	If pandas or scipy is not installed.
`ValueError`	If `method` is not supported, or if fewer than two models have values for `metric`.

`table(show_significance=False)`

Return the results as a text table.

The table is drawn with Unicode box-drawing characters and needs no optional dependency.

Parameters:

Name	Type	Description	Default
`show_significance`	`bool`	If `True`, mark the best model with `` or `*` when a Wilcoxon signed-rank test against another model gives `p < 0.05` or `p < 0.01`. The test needs at least `6` seeds shared by both models, so with fewer seeds no model is marked, and a warning explains why. The test needs pandas, from the `benchmark` extra.	`False`

Returns:

Type	Description
`str`	A string with one row per model and one column per metric. Each value is shown as `mean ± std`.

Raises:

Type	Description
`ImportError`	If `show_significance` is `True` and pandas is not installed.

`to_dataframe()`

Return the raw results of every seed as a pandas DataFrame.

Returns:

Type	Description
`'pd.DataFrame'`	A DataFrame with the columns `model`, `seed`, `metric` and `value`, and one row for each entry of `raw`.

Raises:

Type	Description
`ImportError`	If pandas is not installed.

`compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)`

Compare eigenkit models and built-in scikit-learn models on a dataset.

Each model must be a built-in scikit-learn estimator (from the sklearn package), or a subclass of BaseQuantumEstimator, BaseQuantumClassifier or BaseQuantumRegressor. For every seed, each model is copied with sklearn.base.clone().

Parameters:

Name	Type	Description	Default
`models`	`list \| dict[str, Any]`	The estimator instances to compare. If a list is passed, the display names come from `type(model).__name__`, and repeated names are numbered (for example, `SVC_0` and `SVC_1`). If a dict is passed, its keys are used as display names. Dict keys must be non-empty strings, with no whitespace at the start or the end and no control characters. Use a different key for each model. If the same key appears twice in a Python dict literal, Python keeps only the last model before `compare()` receives the dict. When `save_dir` is set, the keys must also be valid prefixes for file names.	required
`dataset`	`str \| dict`	A dataset name, or a dict that is already split, with `X_train`, `y_train`, `X_test` and `y_test`. Such a dict can also include `X_val` and `y_val`.	`'breast_cancer_wisconsin'`
`metrics`	`list \| None`	The metrics to compute. For classification, the options are `accuracy`, `f1`, `precision`, `recall`, `roc_auc`, `fit_time` and `predict_time`. For regression, they are `r2`, `mae`, `rmse`, `mse`, `fit_time` and `predict_time`. A custom function must be wrapped with `custom_metric(...)`, so that its direction (higher or lower is better) is explicit. The default depends on the type of task of the first model.	`None`
`n_seeds`	`int`	The number of seeds to average over. The default is `5`.	`5`
`test_size`	`float`	The fraction of the data kept for the final test set.	`0.15`
`show_progress`	`bool`	Whether to print the progress of the benchmark, with the current model and an estimate of the remaining time. The default is `False`, which runs silently.	`False`
`save_dir`	`str \| Path \| None`	If set, each fitted model is saved in this directory, as `{ModelName}_seed{N}.eigenkit` with `save()`, or as `{ModelName}_seed{N}.joblib` with joblib for scikit-learn estimators.	`None`
`store_predictions`	`bool`	If `True`, the result keeps `y_true`, `y_pred` and `y_prob` for the last seed of each model.	`False`
`validation_size`	`float`	The fraction of the data kept for validation.	`0.15`

Returns:

Type	Description
`BenchmarkResult`	An object with the results, and methods such as `table()`, `plot()` and `to_dataframe()`.

Raises:

Type	Description
`ValueError`	If `models` is empty, if a model name is not valid, or if `dataset` is not well formed.
`TypeError`	If a model name is not a string, or if a model is not a supported type of estimator.

`custom_metric(fn, *, direction, name=None)`

Define a custom metric and the direction in which it improves.

Parameters:

Name	Type	Description	Default
`fn`	`Callable[[ndarray, ndarray], float]`	The metric function. It receives `(y_true, y_pred)` and returns a number.	required
`direction`	`MetricDirection`	`"higher"` if higher values are better, or `"lower"` if lower values are better.	required
`name`	`str \| None`	The column label of the metric. The default is the name of `fn`.	`None`

Returns:

Type	Description
`CustomMetric`	A metric specification to pass to `compare(metrics=[...])`.

`load_checkpoint(path, *, model=None)`

Load a model checkpoint written by compare(..., save_dir=...).

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	The path to the checkpoint file (`.eigenkit` or `.joblib`).	required
`model`	`Any`	The `nn.Module` architecture for `GenericRegressor` or `GenericClassifier` checkpoints. It is ignored for other types of estimator.	`None`

Returns:

Type	Description
`Any`	The loaded estimator.

Raises:

Type	Description
`TypeError`	If a `.joblib` file does not contain a scikit-learn estimator.
`ValueError`	If the file extension is not supported.

`quick_baselines()`

Return a list of common scikit-learn classifiers with sensible defaults.

The list contains a SVC with a radial basis function (RBF) kernel and probabilities, so that roc_auc can be computed, a RandomForestClassifier, a LogisticRegression and a KNeighborsClassifier. XGBClassifier and LGBMClassifier are added when their packages can be imported.

scikit-learn 1.9 deprecates SVC(probability=True). From that version, the support vector classifier is wrapped as CalibratedClassifierCV(SVC(), ensemble=False), the replacement that scikit-learn documents, so the results table calls it CalibratedClassifierCV.

Returns:

Type	Description
`list[Any]`	A list of classifiers that follow the scikit-learn interface, created but not fitted.

Example

from eigenkit.benchmark import compare, quick_baselines
results = compare(models=quick_baselines(), dataset="iris", n_seeds=2)

`timer()`

Measure the wall-clock time of a block of code.

Yields:

Type	Description
`dict`	A dict. When the `with` block ends, its `"elapsed"` key holds the elapsed time in seconds.

Example

with timer() as t:
    do_work()
print(t["elapsed"])

Benchmark

Supported models

compare()

Saving fitted models

Classification metrics

Regression metrics

Custom metrics

BenchmarkResult

Attributes

Methods

Accessing fitted models

best_model(metric, seed)

predict(X, metric)

Inference on new data

Selecting models and seed

Probabilities

quick_baselines()

Timing a block of code

API reference

eigenkit.benchmark

BenchmarkResult dataclass

best_model(metric=None, seed='last')

confusion_matrix(model_name=None, **kwargs)

inference(X, models=None, seed='last')

load(path) classmethod

plot(save_path=None, theme='light')

plot_history(save_path=None, theme='light', seed='last')

predict(X, metric=None)

save(path)

significance_test(metric, method='wilcoxon')

table(show_significance=False)

to_dataframe()

compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)

custom_metric(fn, *, direction, name=None)

load_checkpoint(path, *, model=None)

quick_baselines()

timer()

`compare()`

`BenchmarkResult`

`best_model(metric, seed)`

`predict(X, metric)`

`quick_baselines()`

`eigenkit.benchmark`

`BenchmarkResult` `dataclass`

`best_model(metric=None, seed='last')`

`confusion_matrix(model_name=None, **kwargs)`

`inference(X, models=None, seed='last')`

`load(path)` `classmethod`

`plot(save_path=None, theme='light')`

`plot_history(save_path=None, theme='light', seed='last')`

`predict(X, metric=None)`

`save(path)`

`significance_test(metric, method='wilcoxon')`

`table(show_significance=False)`

`to_dataframe()`

`compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)`

`custom_metric(fn, *, direction, name=None)`

`load_checkpoint(path, *, model=None)`

`quick_baselines()`

`timer()`