# Benchmark `eigenkit.benchmark` compares quantum and classical models. A single call to `compare()` trains several estimators on one dataset with several random seeds, and returns a `BenchmarkResult`. The result can show the scores as a formatted table, plot them as grouped bar charts, and export them as a pandas DataFrame. The benchmark needs the `benchmark` extra, which installs matplotlib, pandas and scikit-learn: `pip install eigenkit[benchmark]`. The quantum models that you compare need their own extras: `qml` for `QSVC` and `QSVR`, and `torch` for `HQNC`, `HQNR` and the other differentiable estimators. By default, named datasets are split into `70 %` for training, `15 %` for validation and `15 %` for testing. The metrics are computed on the test split. The validation split is passed only to estimators whose `fit()` method accepts `X_val` and `y_val`. `compare()` repeats the split into training, validation and test sets for each seed. It does not use k-fold cross-validation. For k-fold cross-validation, call scikit-learn directly, for example with `cross_val_score` or `GridSearchCV(cv=...)`. ```python from eigenkit import QSVC, HQNC from eigenkit.benchmark import compare from sklearn.svm import SVC from sklearn.ensemble import RandomForestClassifier result = compare( models=[ QSVC(feature_map="Angle", reps=2), HQNC( in_features=4, depth=1, epochs=3, show_progress=False, ), SVC(C=0.1), RandomForestClassifier(n_estimators=20), ], dataset="iris", metrics=["accuracy", "f1"], n_seeds=3, validation_size=0.15, test_size=0.15, ) print(result.table()) result.plot(save_path="comparison.png") result.plot_history(save_path="losses.png") ``` The run used for this guide produced the following table: ```text ┌────────────────────────┬───────────────┬───────────────┐ │ Model │ Accuracy │ F1 │ ├────────────────────────┼───────────────┼───────────────┤ │ QSVC │ 0.989 ± 0.019 │ 0.989 ± 0.019 │ │ HQNC │ 0.389 ± 0.096 │ 0.242 ± 0.131 │ │ SVC │ 0.911 ± 0.051 │ 0.909 ± 0.053 │ │ RandomForestClassifier │ 0.967 ± 0.033 │ 0.966 ± 0.034 │ └────────────────────────┴───────────────┴───────────────┘ ``` Each cell shows the mean and the standard deviation across seeds, as `mean ± std`. ## Supported models Each entry in `models` must be one of the following: 1. An **eigenkit estimator**, that is, a subclass of `BaseQuantumEstimator`, `BaseQuantumClassifier` or `BaseQuantumRegressor`, such as `HQNC`, `QSVC` or `CustomQuantumClassifier`. 1. A **built-in scikit-learn estimator**, that is, a class from the `sklearn` package, such as `SVC`, `RandomForestClassifier` or `LinearRegression`. Models from other libraries are not accepted yet. This includes XGBoost, LightGBM and custom `BaseEstimator` subclasses defined outside `sklearn`. Every model must implement `fit(X, y)` and `predict(X)`. Internally, `compare()` uses `sklearn.base.clone` to make a fresh copy of each model for every seed. ## `compare()` `models` is the only required argument. Every other argument has a default value. | Argument | Default | Description | | ------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `models` | required | A list of estimator instances, or a dict that maps display names to estimators. | | `dataset` | `"breast_cancer_wisconsin"` | A dataset name from the registry, or a dict with `X_train`, `y_train`, `X_test` and `y_test`. | | `metrics` | `None` | The names of the metrics to compute, or `custom_metric(...)` specifications. `None` selects the default metrics for the task, listed below. | | `n_seeds` | `5` | The number of seeds to average over. | | `test_size` | `0.15` | The fraction of the data kept for the final test metrics. | | `show_progress` | `False` | Whether to print progress: the current seed, the current model and the estimated time remaining. | | `save_dir` | `None` | If set, the directory where each fitted model is saved after every seed (see below). | | `store_predictions` | `False` | If `True`, the result keeps `y_true`, `y_pred` and, when available, `y_prob` for the last seed. | | `validation_size` | `0.15` | The fraction of the data kept for validation. | If you call `compare(models)` with no other argument, the benchmark uses the `breast_cancer_wisconsin` dataset. The default `test_size` here is `0.15`, which is different from the `0.2` that [`load_dataset`](https://docs.kit.eigenorbit.com/user-guide/datasets/index.md) uses. Set `show_progress=True` to see the progress of the run: which model is running and how many runs are left. By default nothing is printed, which keeps notebooks and continuous integration (CI) logs quiet. Pass a dict to choose the names that appear in tables, plots, progress messages and saved file names. This is useful when you compare several instances of the same estimator class: ```python from sklearn.svm import SVC from eigenkit import QSVC from eigenkit.benchmark import compare result = compare( models={ "Quantum SVC": QSVC(feature_map="Angle", reps=1, probability=True), "Regularized SVC": SVC(C=0.3), "Flexible SVC": SVC(C=3.0), }, dataset="iris", metrics=["accuracy"], n_seeds=1, ) print(list(result.fitted_models)) ``` ```text ['Quantum SVC', 'Regularized SVC', 'Flexible SVC'] ``` Dict keys must not have whitespace at the start or the end, and must not contain control characters. When `save_dir` is set, the keys are also used as prefixes of file names. In that case, they must not end with a full stop and must not contain any of these characters: `<>:"/\|?*`. Use a different key for each model. If the same key appears twice in a Python dict literal, Python keeps only the last model before `compare()` receives the dict. If you pass a dataset dict that is already split and you have a validation set, include `X_val` and `y_val`. If they are left out, `compare()` uses only the training and test sets that you provided. ### Saving fitted models When `save_dir` is set, each model is written to a file after `fit()`: | Model type | Output file | | --------------------- | -------------------------------------------------------- | | eigenkit | `{ModelName}_seed{N}.eigenkit`, written with `save()` | | built-in scikit-learn | `{ModelName}_seed{N}.joblib`, written with `joblib.dump` | ```python compare( models=[HQNC(in_features=4, depth=2), SVC()], dataset="iris", n_seeds=2, save_dir="runs/iris_benchmark/", ) # runs/iris_benchmark/HQNC_seed0.eigenkit, HQNC_seed1.eigenkit, SVC_seed0.joblib, ... ``` `ModelName` is the name of the model in the benchmark. When the same class appears more than once in a list, the names are numbered (`SVC_0`, `SVC_1`). When `models` is a dict, `ModelName` is the dict key. To load an eigenkit checkpoint, use `eigenkit.benchmark.load_checkpoint()` or the estimator's `load()` method, for example `HQNC.load(...)`. To load a scikit-learn checkpoint, use `eigenkit.benchmark.load_checkpoint()` or `joblib.load(...)`. `.joblib` files use pickle internally, so only load them from sources that you trust. See [Model persistence](https://docs.kit.eigenorbit.com/user-guide/model-persistence/index.md) for more details. ### Classification metrics The available metrics are `accuracy`, `f1` (the F1 score), `precision`, `recall`, `roc_auc` (the area under the receiver operating characteristic curve), `fit_time` and `predict_time`. When `metrics` is `None`, the benchmark computes `accuracy`, `f1`, `fit_time` and `predict_time`. `roc_auc` needs `predict_proba`. `QSVC` with `probability=True` and `HQNC` both provide it. For a model without `predict_proba`, requesting `roc_auc` issues a `UserWarning`, and the metric is skipped. ### Regression metrics The available metrics are `r2` (the coefficient of determination), `mae` (the mean absolute error), `rmse` (the root mean squared error), `mse` (the mean squared error), `fit_time` and `predict_time`. The type of task is taken from the first model: a model that inherits from `RegressorMixin` makes it a regression task. When `metrics` is `None`, the benchmark computes `r2`, `mae`, `mse`, `fit_time` and `predict_time`. `rmse` is not computed by default, so request it by name if you need it. ### Custom metrics A custom metric must be wrapped with `custom_metric(fn, *, direction, name=None)`. The function receives `(y_true, y_pred)`. `direction` states whether higher or lower values are better. `name` sets the column label, which otherwise comes from the name of the function: ```python from eigenkit.benchmark import custom_metric def error_rate(y_true, y_pred): return (y_true != y_pred).mean() def match_rate(y_true, y_pred): return (y_true == y_pred).mean() metrics = [ custom_metric(error_rate, direction="lower"), custom_metric(match_rate, direction="higher"), ] ``` ## `BenchmarkResult` `compare()` returns a `BenchmarkResult`. It holds the raw values for every seed, the fitted copies of the models, and methods to display, analyse and use the results. ### Attributes | Attribute | Type | Description | | ------------------- | ----------------- | --------------------------------------------------------------------------------------------------- | | `raw` | `list[dict]` | One row per value, with the keys `"model"`, `"seed"`, `"metric"` and `"value"`. | | `fitted_models` | `dict[str, list]` | The fitted estimators for each model name, one per seed. | | `predictions` | `dict[str, dict]` | The `y_true`, `y_pred` and `y_prob` of the last seed for each model, when `store_predictions=True`. | | `model_artifacts` | `dict[str, list]` | The paths of the checkpoint files written to `save_dir`, grouped by model name. | | `loss_curves` | `dict[str, list]` | The training and validation loss curves for each seed, when the fitted estimators record them. | | `metric_directions` | `dict[str, str]` | The direction, `"higher"` or `"lower"`, of every metric in the run. | ### Methods | Method | Description | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `table(show_significance=False)` | A text table with `mean ± std` for each model and metric. It needs no optional dependency. With `show_significance=True`, the best model is marked with `*` or `**` when a test against another model gives `p < 0.05` or `p < 0.01`. | | `plot(save_path=None, theme="light")` | A grouped bar chart with error bars. `theme` is `"light"` or `"dark"`. It needs `matplotlib`. | | `plot_history(save_path=None, theme="light", seed="last")` | The training and validation loss curves of the models that record them. `seed` chooses the run to draw: `"last"` or an integer index. It needs `matplotlib`. | | `to_dataframe()` | A DataFrame with one row per seed and metric. It needs `pandas`. | | `best_model(metric=None, seed="last")` | The fitted estimator with the best mean on `metric`, taking the direction of the metric into account. | | `predict(X, metric=None)` | A shortcut for `best_model(metric=metric).predict(X)`. | | `inference(X, models=None, seed="last")` | The predictions and probabilities of all fitted models, or of the selected ones. | | `significance_test(metric, method="wilcoxon")` | A matrix of p-values for each pair of models, computed from the scores of each seed. `method="ttest"` is also supported. It needs `pandas` and `scipy`. | | `confusion_matrix(model_name=None, **kwargs)` | A scikit-learn confusion matrix computed from the stored predictions. `**kwargs` are passed to `sklearn.metrics.confusion_matrix`. It needs `store_predictions=True`. | | `save(path)` | Writes the numeric results to a JSON file. The fitted estimators are not saved. | | `BenchmarkResult.load(path)` | Restores a result from a JSON file written by `save()`. | The significance markers come from a Wilcoxon signed-rank test, which needs at least `6` seeds shared by the two models. With the default `n_seeds=5`, `table(show_significance=True)` cannot mark any model, and it issues a warning instead. To use plotting, DataFrame export and significance tests, install the optional extra: ```bash pip install eigenkit[benchmark] ``` `plot_history()` draws the models that record a loss history, and warns about the ones that do not. For example, `HQNC` produces training and validation curves, while `SVC` appears only in the metric table and the bar chart. ### Accessing fitted models Every fitted copy produced by `compare()` is stored in `result.fitted_models`, a `dict[str, list[estimator]]` keyed by the name of the model in the benchmark. With a list of models, the names come from the class (`SVC`, or `SVC_0` and `SVC_1` when the same class appears more than once). With a dict, the keys are exactly the names that you passed. Each value is a list of fitted copies, one per seed. ```python print(list(result.fitted_models)) # ['Quantum SVC', 'Regularized SVC', 'Flexible SVC'] quantum_fits = result.fitted_models["Quantum SVC"] model_seed0 = quantum_fits[0] ``` #### `best_model(metric, seed)` Returns the fitted estimator with the best mean on `metric`. When `metric` is not given, the primary metric is chosen automatically. It is the first metric in the results that is not a timing metric, usually `"accuracy"` for classification and `"r2"` for regression with the default metrics. ```python winner = result.best_model() # auto-selects metric winner = result.best_model(metric="f1", seed=0) # explicit metric, seed 0 winner = result.best_model(metric="r2") # explicit for regression ``` | Parameter | Default | Description | | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `metric` | `None` | The metric used for ranking. `None` selects the primary metric of the task. Built-in metrics and custom metrics use their registered direction. | | `seed` | `"last"` | The seed whose copy is returned: `"last"`, or a 0-based `int`. | #### `predict(X, metric)` This is a shortcut for `best_model(metric=metric).predict(X)`. It returns only the predictions of the selected model. Use `inference()` when you also need probabilities, or the outputs of several models. ```python preds = result.predict(X_new) # auto-selects metric preds = result.predict(X_new, metric="r2") # explicit for regression ``` `X` is passed directly to the selected estimator. The return value is what that estimator's `predict(X)` returns, usually an array of shape `(n_samples,)`. ### Inference on new data `result.inference(X)` runs all fitted models on new data and returns their predictions together with the class probabilities, in a single call. ```python out = result.inference(X_new) # { # "HQNC": {"predictions": array([...]), "probabilities": array([[...]])}, # "SVC": {"predictions": array([...]), "probabilities": None}, # no predict_proba # } preds = out["HQNC"]["predictions"] # shape (n_samples,) probs = out["HQNC"]["probabilities"] # shape (n_samples, n_classes), or None ``` | Output key | Type | Description | | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `predictions` | `ndarray` | The output of `estimator.predict(X)`. | | `probabilities` | `ndarray` or `None` | The output of `predict_proba(X)` for classifiers that support it, and `None` for regressors and for classifiers without `predict_proba`. | #### Selecting models and seed By default, `inference()` runs all models, each with the copy fitted on the last seed. You can restrict both: ```python # Only a subset of models out = result.inference(X_new, models=["SVC"]) # Specific seed (0-based index into the per-seed list) out = result.inference(X_new, seed=0) out = result.inference(X_new, models=["HQNC", "SVC"], seed=1) ``` Passing an unknown model name raises a `ValueError` that lists the available names. #### Probabilities `probabilities` is an array of shape `(n_samples, n_classes)` when the estimator provides `predict_proba`, and `None` otherwise: - It is filled for `HQNC`, `QSVC(probability=True)`, `CustomQuantumClassifier` and `SVC(probability=True)`. - It is `None` for all regressors, for classifiers without `predict_proba` (for example, a plain `SVC()` without `probability=True`), and for a `QSVC` without `probability=True`, which also issues a warning. ## `quick_baselines()` ```python from eigenkit import QSVC from eigenkit.benchmark import compare, quick_baselines results = compare( models=[QSVC(feature_map="Angle"), *quick_baselines()], dataset="iris", n_seeds=2, ) ``` `quick_baselines()` returns built-in scikit-learn classifiers with sensible defaults: a `SVC` with a radial basis function (RBF) kernel and probabilities, a `RandomForestClassifier`, a `LogisticRegression` and a `KNeighborsClassifier`. scikit-learn 1.9 deprecates `SVC(probability=True)`, so from that version the support vector classifier is wrapped as `CalibratedClassifierCV(SVC(), ensemble=False)`. ## Timing a block of code `timer()` measures the wall-clock time of the code inside a `with` block. For example, it can measure the time needed to compute one kernel matrix on two providers: ```python import numpy as np from eigenkit import QuantumKernel from eigenkit.benchmark import timer X = np.random.default_rng(0).uniform(0, np.pi, size=(150, 4)) for provider in ("numpy", "pennylane"): # "pennylane" needs the pennylane extra kernel = QuantumKernel(n_features=4, feature_map="AngleCNOT", provider=provider) with timer() as t: kernel.compute_kernel_matrix(X) print(provider, f"{t['elapsed']:.2f} s") ``` ## API reference ### `eigenkit.benchmark` Comparison of quantum and classical estimators. The compare() function trains models that follow the scikit-learn interface on a dataset with several random seeds. It returns a BenchmarkResult with the aggregated metrics. Example ```python from sklearn.svm import SVC from eigenkit.qml import QSVC from eigenkit.benchmark import compare results = compare( models={ "Quantum SVC": QSVC( feature_map="Angle", reps=1, probability=True ), "Regularized SVC": SVC(C=0.3), "Flexible SVC": SVC(C=3.0), }, dataset="iris", metrics=["accuracy"], n_seeds=1, ) print(list(results.fitted_models)) # ['Quantum SVC', 'Regularized SVC', 'Flexible SVC'] ``` #### `BenchmarkResult` Results of a benchmark comparison. It stores the raw metric values of every seed, and aggregates them when you display them with `table()`, plot them with `plot()`, or analyse them with `to_dataframe()`. Each entry in `raw` is a dict with the keys `"model"`, `"seed"`, `"metric"` and `"value"`. `fitted_models` holds every fitted copy produced by compare(), keyed by the name of the model in the benchmark. The names usually match the class names, and repeated names are numbered (for example, `SVC_0` and `SVC_1`). If `compare(models={...})` receives a dict, its keys are used instead. Example ```python from sklearn.svm import SVC from eigenkit.qml import QSVC from eigenkit.benchmark import compare results = compare( models={ "Quantum SVC": QSVC( feature_map="Angle", reps=1, probability=True ), "Regularized SVC": SVC(C=0.3), "Flexible SVC": SVC(C=3.0), }, dataset="iris", metrics=["accuracy"], n_seeds=1, ) print(list(results.fitted_models)) # ['Quantum SVC', 'Regularized SVC', 'Flexible SVC'] results.plot(save_path="svc_comparison.png") df = results.to_dataframe() X_new = [[5.1, 3.5, 1.4, 0.2]] # Access the fitted estimators winner = results.best_model(metric="accuracy") preds = results.predict(X_new) quantum_fits = results.fitted_models["Quantum SVC"] # Run inference on new data, with all models or a subset out = results.inference(X_new) out = results.inference(X_new, models=["Quantum SVC"], seed=0) ``` ##### `best_model(metric=None, seed='last')` Return the fitted estimator with the best mean score on `metric`. When `metric` is `None`, the default, the first metric recorded in `raw` that is not a timing metric is used. With the default metrics, this is usually `"accuracy"` for classification and `"r2"` for regression. If custom metrics were passed in a different order, the first of them is used. Parameters: | Name | Type | Description | Default | | -------- | ----- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metric` | \`str | None\` | The metric used for ranking, for example "accuracy" or "r2". None selects the primary metric of the task. Built-in metrics use their registered direction, and custom metrics use the direction given to custom_metric(...). | | `seed` | \`str | int\` | The seed whose estimator is returned for the winning model. "last", the default, returns the estimator of the last seed. An int is a 0-based index into the list of seeds, in the order they ran. | Returns: | Type | Description | | ----- | --------------------- | | `Any` | The fitted estimator. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------- | | `ValueError` | If no fitted models are stored, or if metric is not in the results. | | `IndexError` | If seed is out of range. | ##### `confusion_matrix(model_name=None, **kwargs)` Return a scikit-learn confusion matrix computed from the stored predictions. Parameters: | Name | Type | Description | Default | | ------------ | ----- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `model_name` | \`str | None\` | The model whose predictions are used. None, the default, uses the first model with stored predictions. | | `**kwargs` | `Any` | Passed on to sklearn.metrics.confusion_matrix. | `{}` | Returns: | Type | Description | | --------- | --------------------------------------- | | `ndarray` | The confusion matrix, as a NumPy array. | Raises: | Type | Description | | ------------- | ------------------------------------------------------------------------------------------------------------- | | `ValueError` | If no predictions are stored (run compare(..., store_predictions=True)), or if there are none for model_name. | | `ImportError` | If scikit-learn is not installed. | ##### `inference(X, models=None, seed='last')` Run every fitted model, or a subset of them, on new data. Parameters: | Name | Type | Description | Default | | -------- | ----------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X` | `Any` | The input features to predict on. | *required* | | `models` | \`list[str] | None\` | The names of the models to run. None, the default, runs all fitted models. Use the exact keys of fitted_models: the names based on the class for a list input (for example, "SVC" or "SVC_0"), or your own keys for a dict input. | | `seed` | \`str | int\` | The seed whose estimator is used for each model. "last", the default, uses the estimator of the last seed. An int is a 0-based index into the list of seeds, in the order they ran. | Returns: | Type | Description | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dict[str, dict[str, Any]]` | A dict keyed by model name. Each value contains two entries: "predictions": the array returned by estimator.predict(X). "probabilities": the array returned by estimator.predict_proba(X) for classifiers that support it, and None for regressors and for classifiers without predict_proba. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------------- | | `ValueError` | If no fitted models are stored, or if an unknown model name is requested. | | `IndexError` | If seed is out of range for any model. | Example ```python out = results.inference(X_new) out = results.inference(X_new, models=["Flexible SVC"], seed=0) preds = out["Flexible SVC"]["predictions"] probs = out["Flexible SVC"]["probabilities"] # None for regressors ``` ##### `load(path)` Load numeric benchmark results saved by `BenchmarkResult.save()`. Parameters: | Name | Type | Description | Default | | ------ | ----- | ----------- | ---------------------- | | `path` | \`str | Path\` | The JSON file to read. | Returns: | Type | Description | | ------------------- | ------------------------------------------------------------------------------------------------------ | | `'BenchmarkResult'` | A BenchmarkResult with the stored values. It has no fitted models, because save() does not store them. | ##### `plot(save_path=None, theme='light')` Plot a grouped bar chart with error bars. Parameters: | Name | Type | Description | Default | | ----------- | ----- | ------------------------------------------------ | ------------------------------------------------------------------ | | `save_path` | \`str | None\` | If given, the figure is saved to this path instead of being shown. | | `theme` | `str` | The plot theme, "light" (the default) or "dark". | `'light'` | Raises: | Type | Description | | ------------- | ------------------------------- | | `ImportError` | If matplotlib is not installed. | ##### `plot_history(save_path=None, theme='light', seed='last')` Plot the stored training and validation loss curves. Models without a loss history are left out, and a warning names them. Parameters: | Name | Type | Description | Default | | ----------- | ----- | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `save_path` | \`str | None\` | If given, the figure is saved to this path instead of being shown. | | `theme` | `str` | The plot theme, "light" (the default) or "dark". | `'light'` | | `seed` | \`str | int\` | The run to draw: "last" (the default), or an integer index into the list of seeds. With "last", a model whose last seed failed shows its most recent recorded curve. | Raises: | Type | Description | | ------------- | ---------------------------------------------------- | | `TypeError` | If seed is neither "last" nor an int. | | `ValueError` | If no model has a training or validation loss curve. | | `ImportError` | If matplotlib is not installed. | ##### `predict(X, metric=None)` Predict with the best model ranked by `metric`. This is a shortcut for `best_model()`. It calls `predict()` on the estimator returned by `best_model(metric=metric)`. Parameters: | Name | Type | Description | Default | | -------- | ----- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X` | `Any` | The input features. | *required* | | `metric` | \`str | None\` | The metric used to select the best model. None, the default, selects the primary metric of the task: "accuracy" for classification and "r2" for regression. | Returns: | Type | Description | | ----- | --------------------------------------------------------- | | `Any` | The predictions of the best estimator's predict() method. | ##### `save(path)` Save the numeric results of the benchmark to a JSON file. The fitted estimators are not saved, by design. Parameters: | Name | Type | Description | Default | | ------ | ----- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `path` | \`str | Path\` | The file to write. If it is an existing directory, a file named bench\_.json is created inside it. If it has no extension, .json is added. | Returns: | Type | Description | | ------ | -------------------------------------- | | `Path` | The path of the file that was written. | ##### `significance_test(metric, method='wilcoxon')` Return a matrix of p-values for each pair of models. The p-values compare the values of `metric` on the seeds that both models share. Parameters: | Name | Type | Description | Default | | -------- | ----- | ----------------------------------------------------------------------------------------- | ------------ | | `metric` | `str` | The metric to compare. | *required* | | `method` | `str` | "wilcoxon" (the default) for a Wilcoxon signed-rank test, or "ttest" for a paired t-test. | `'wilcoxon'` | Returns: | Type | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `'pd.DataFrame'` | A pandas DataFrame with one row and one column per model. The diagonal is 1, and a pair with fewer than two shared seeds has the value NaN. | Raises: | Type | Description | | ------------- | ------------------------------------------------------------------------------- | | `ImportError` | If pandas or scipy is not installed. | | `ValueError` | If method is not supported, or if fewer than two models have values for metric. | ##### `table(show_significance=False)` Return the results as a text table. The table is drawn with Unicode box-drawing characters and needs no optional dependency. Parameters: | Name | Type | Description | Default | | ------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `show_significance` | `bool` | If True, mark the best model with * or \*\* when a Wilcoxon signed-rank test against another model gives p < 0.05 or p < 0.01. The test needs at least 6 seeds shared by both models, so with fewer seeds no model is marked, and a warning explains why. The test needs pandas, from the benchmark extra. | `False` | Returns: | Type | Description | | ----- | --------------------------------------------------------------------------------------------- | | `str` | A string with one row per model and one column per metric. Each value is shown as mean ± std. | Raises: | Type | Description | | ------------- | --------------------------------------------------------- | | `ImportError` | If show_significance is True and pandas is not installed. | ##### `to_dataframe()` Return the raw results of every seed as a pandas DataFrame. Returns: | Type | Description | | ---------------- | ---------------------------------------------------------------------------------------------- | | `'pd.DataFrame'` | A DataFrame with the columns model, seed, metric and value, and one row for each entry of raw. | Raises: | Type | Description | | ------------- | --------------------------- | | `ImportError` | If pandas is not installed. | #### `compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)` Compare eigenkit models and built-in scikit-learn models on a dataset. Each model must be a built-in scikit-learn estimator (from the `sklearn` package), or a subclass of BaseQuantumEstimator, BaseQuantumClassifier or BaseQuantumRegressor. For every seed, each model is copied with `sklearn.base.clone()`. Parameters: | Name | Type | Description | Default | | ------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `models` | \`list | dict[str, Any]\` | The estimator instances to compare. If a list is passed, the display names come from type(model).__name__, and repeated names are numbered (for example, SVC_0 and SVC_1). If a dict is passed, its keys are used as display names. Dict keys must be non-empty strings, with no whitespace at the start or the end and no control characters. Use a different key for each model. If the same key appears twice in a Python dict literal, Python keeps only the last model before compare() receives the dict. When save_dir is set, the keys must also be valid prefixes for file names. | | `dataset` | \`str | dict\` | A dataset name, or a dict that is already split, with X_train, y_train, X_test and y_test. Such a dict can also include X_val and y_val. | | `metrics` | \`list | None\` | The metrics to compute. For classification, the options are accuracy, f1, precision, recall, roc_auc, fit_time and predict_time. For regression, they are r2, mae, rmse, mse, fit_time and predict_time. A custom function must be wrapped with custom_metric(...), so that its direction (higher or lower is better) is explicit. The default depends on the type of task of the first model. | | `n_seeds` | `int` | The number of seeds to average over. The default is 5. | `5` | | `test_size` | `float` | The fraction of the data kept for the final test set. | `0.15` | | `show_progress` | `bool` | Whether to print the progress of the benchmark, with the current model and an estimate of the remaining time. The default is False, which runs silently. | `False` | | `save_dir` | \`str | Path | None\` | | `store_predictions` | `bool` | If True, the result keeps y_true, y_pred and y_prob for the last seed of each model. | `False` | | `validation_size` | `float` | The fraction of the data kept for validation. | `0.15` | Returns: | Type | Description | | ----------------- | ----------------------------------------------------------------------------------- | | `BenchmarkResult` | An object with the results, and methods such as table(), plot() and to_dataframe(). | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------------------------ | | `ValueError` | If models is empty, if a model name is not valid, or if dataset is not well formed. | | `TypeError` | If a model name is not a string, or if a model is not a supported type of estimator. | #### `custom_metric(fn, *, direction, name=None)` Define a custom metric and the direction in which it improves. Parameters: | Name | Type | Description | Default | | ----------- | ------------------------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | | `fn` | `Callable[[ndarray, ndarray], float]` | The metric function. It receives (y_true, y_pred) and returns a number. | *required* | | `direction` | `MetricDirection` | "higher" if higher values are better, or "lower" if lower values are better. | *required* | | `name` | \`str | None\` | The column label of the metric. The default is the name of fn. | Returns: | Type | Description | | -------------- | --------------------------------------------------------- | | `CustomMetric` | A metric specification to pass to compare(metrics=[...]). | #### `load_checkpoint(path, *, model=None)` Load a model checkpoint written by `compare(..., save_dir=...)`. Parameters: | Name | Type | Description | Default | | ------- | ----- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | `path` | \`str | Path\` | The path to the checkpoint file (\*.eigenkit or \*.joblib). | | `model` | `Any` | The nn.Module architecture for GenericRegressor or GenericClassifier checkpoints. It is ignored for other types of estimator. | `None` | Returns: | Type | Description | | ----- | --------------------- | | `Any` | The loaded estimator. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------ | | `TypeError` | If a .joblib file does not contain a scikit-learn estimator. | | `ValueError` | If the file extension is not supported. | #### `quick_baselines()` Return a list of common scikit-learn classifiers with sensible defaults. The list contains a `SVC` with a radial basis function (RBF) kernel and probabilities, so that `roc_auc` can be computed, a `RandomForestClassifier`, a `LogisticRegression` and a `KNeighborsClassifier`. `XGBClassifier` and `LGBMClassifier` are added when their packages can be imported. scikit-learn 1.9 deprecates `SVC(probability=True)`. From that version, the support vector classifier is wrapped as `CalibratedClassifierCV(SVC(), ensemble=False)`, the replacement that scikit-learn documents, so the results table calls it `CalibratedClassifierCV`. Returns: | Type | Description | | ----------- | ------------------------------------------------------------------------------------- | | `list[Any]` | A list of classifiers that follow the scikit-learn interface, created but not fitted. | Example ```python from eigenkit.benchmark import compare, quick_baselines results = compare(models=quick_baselines(), dataset="iris", n_seeds=2) ``` #### `timer()` Measure the wall-clock time of a block of code. Yields: | Type | Description | | ------ | -------------------------------------------------------------------------------------- | | `dict` | A dict. When the with block ends, its "elapsed" key holds the elapsed time in seconds. | Example ```python with timer() as t: do_work() print(t["elapsed"]) ```