## `eigenkit.benchmark` Comparison of quantum and classical estimators. The compare() function trains models that follow the scikit-learn interface on a dataset with several random seeds. It returns a BenchmarkResult with the aggregated metrics. Example ```python from sklearn.svm import SVC from eigenkit.qml import QSVC from eigenkit.benchmark import compare results = compare( models={ "Quantum SVC": QSVC( feature_map="Angle", reps=1, probability=True ), "Regularized SVC": SVC(C=0.3), "Flexible SVC": SVC(C=3.0), }, dataset="iris", metrics=["accuracy"], n_seeds=1, ) print(list(results.fitted_models)) # ['Quantum SVC', 'Regularized SVC', 'Flexible SVC'] ``` ### `BenchmarkResult` Results of a benchmark comparison. It stores the raw metric values of every seed, and aggregates them when you display them with `table()`, plot them with `plot()`, or analyse them with `to_dataframe()`. Each entry in `raw` is a dict with the keys `"model"`, `"seed"`, `"metric"` and `"value"`. `fitted_models` holds every fitted copy produced by compare(), keyed by the name of the model in the benchmark. The names usually match the class names, and repeated names are numbered (for example, `SVC_0` and `SVC_1`). If `compare(models={...})` receives a dict, its keys are used instead. Example ```python from sklearn.svm import SVC from eigenkit.qml import QSVC from eigenkit.benchmark import compare results = compare( models={ "Quantum SVC": QSVC( feature_map="Angle", reps=1, probability=True ), "Regularized SVC": SVC(C=0.3), "Flexible SVC": SVC(C=3.0), }, dataset="iris", metrics=["accuracy"], n_seeds=1, ) print(list(results.fitted_models)) # ['Quantum SVC', 'Regularized SVC', 'Flexible SVC'] results.plot(save_path="svc_comparison.png") df = results.to_dataframe() X_new = [[5.1, 3.5, 1.4, 0.2]] # Access the fitted estimators winner = results.best_model(metric="accuracy") preds = results.predict(X_new) quantum_fits = results.fitted_models["Quantum SVC"] # Run inference on new data, with all models or a subset out = results.inference(X_new) out = results.inference(X_new, models=["Quantum SVC"], seed=0) ``` #### `best_model(metric=None, seed='last')` Return the fitted estimator with the best mean score on `metric`. When `metric` is `None`, the default, the first metric recorded in `raw` that is not a timing metric is used. With the default metrics, this is usually `"accuracy"` for classification and `"r2"` for regression. If custom metrics were passed in a different order, the first of them is used. Parameters: | Name | Type | Description | Default | | -------- | ----- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metric` | \`str | None\` | The metric used for ranking, for example "accuracy" or "r2". None selects the primary metric of the task. Built-in metrics use their registered direction, and custom metrics use the direction given to custom_metric(...). | | `seed` | \`str | int\` | The seed whose estimator is returned for the winning model. "last", the default, returns the estimator of the last seed. An int is a 0-based index into the list of seeds, in the order they ran. | Returns: | Type | Description | | ----- | --------------------- | | `Any` | The fitted estimator. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------- | | `ValueError` | If no fitted models are stored, or if metric is not in the results. | | `IndexError` | If seed is out of range. | #### `confusion_matrix(model_name=None, **kwargs)` Return a scikit-learn confusion matrix computed from the stored predictions. Parameters: | Name | Type | Description | Default | | ------------ | ----- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `model_name` | \`str | None\` | The model whose predictions are used. None, the default, uses the first model with stored predictions. | | `**kwargs` | `Any` | Passed on to sklearn.metrics.confusion_matrix. | `{}` | Returns: | Type | Description | | --------- | --------------------------------------- | | `ndarray` | The confusion matrix, as a NumPy array. | Raises: | Type | Description | | ------------- | ------------------------------------------------------------------------------------------------------------- | | `ValueError` | If no predictions are stored (run compare(..., store_predictions=True)), or if there are none for model_name. | | `ImportError` | If scikit-learn is not installed. | #### `inference(X, models=None, seed='last')` Run every fitted model, or a subset of them, on new data. Parameters: | Name | Type | Description | Default | | -------- | ----------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X` | `Any` | The input features to predict on. | *required* | | `models` | \`list[str] | None\` | The names of the models to run. None, the default, runs all fitted models. Use the exact keys of fitted_models: the names based on the class for a list input (for example, "SVC" or "SVC_0"), or your own keys for a dict input. | | `seed` | \`str | int\` | The seed whose estimator is used for each model. "last", the default, uses the estimator of the last seed. An int is a 0-based index into the list of seeds, in the order they ran. | Returns: | Type | Description | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dict[str, dict[str, Any]]` | A dict keyed by model name. Each value contains two entries: "predictions": the array returned by estimator.predict(X). "probabilities": the array returned by estimator.predict_proba(X) for classifiers that support it, and None for regressors and for classifiers without predict_proba. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------------- | | `ValueError` | If no fitted models are stored, or if an unknown model name is requested. | | `IndexError` | If seed is out of range for any model. | Example ```python out = results.inference(X_new) out = results.inference(X_new, models=["Flexible SVC"], seed=0) preds = out["Flexible SVC"]["predictions"] probs = out["Flexible SVC"]["probabilities"] # None for regressors ``` #### `load(path)` Load numeric benchmark results saved by `BenchmarkResult.save()`. Parameters: | Name | Type | Description | Default | | ------ | ----- | ----------- | ---------------------- | | `path` | \`str | Path\` | The JSON file to read. | Returns: | Type | Description | | ------------------- | ------------------------------------------------------------------------------------------------------ | | `'BenchmarkResult'` | A BenchmarkResult with the stored values. It has no fitted models, because save() does not store them. | #### `plot(save_path=None, theme='light')` Plot a grouped bar chart with error bars. Parameters: | Name | Type | Description | Default | | ----------- | ----- | ------------------------------------------------ | ------------------------------------------------------------------ | | `save_path` | \`str | None\` | If given, the figure is saved to this path instead of being shown. | | `theme` | `str` | The plot theme, "light" (the default) or "dark". | `'light'` | Raises: | Type | Description | | ------------- | ------------------------------- | | `ImportError` | If matplotlib is not installed. | #### `plot_history(save_path=None, theme='light', seed='last')` Plot the stored training and validation loss curves. Models without a loss history are left out, and a warning names them. Parameters: | Name | Type | Description | Default | | ----------- | ----- | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `save_path` | \`str | None\` | If given, the figure is saved to this path instead of being shown. | | `theme` | `str` | The plot theme, "light" (the default) or "dark". | `'light'` | | `seed` | \`str | int\` | The run to draw: "last" (the default), or an integer index into the list of seeds. With "last", a model whose last seed failed shows its most recent recorded curve. | Raises: | Type | Description | | ------------- | ---------------------------------------------------- | | `TypeError` | If seed is neither "last" nor an int. | | `ValueError` | If no model has a training or validation loss curve. | | `ImportError` | If matplotlib is not installed. | #### `predict(X, metric=None)` Predict with the best model ranked by `metric`. This is a shortcut for `best_model()`. It calls `predict()` on the estimator returned by `best_model(metric=metric)`. Parameters: | Name | Type | Description | Default | | -------- | ----- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X` | `Any` | The input features. | *required* | | `metric` | \`str | None\` | The metric used to select the best model. None, the default, selects the primary metric of the task: "accuracy" for classification and "r2" for regression. | Returns: | Type | Description | | ----- | --------------------------------------------------------- | | `Any` | The predictions of the best estimator's predict() method. | #### `save(path)` Save the numeric results of the benchmark to a JSON file. The fitted estimators are not saved, by design. Parameters: | Name | Type | Description | Default | | ------ | ----- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `path` | \`str | Path\` | The file to write. If it is an existing directory, a file named bench\_.json is created inside it. If it has no extension, .json is added. | Returns: | Type | Description | | ------ | -------------------------------------- | | `Path` | The path of the file that was written. | #### `significance_test(metric, method='wilcoxon')` Return a matrix of p-values for each pair of models. The p-values compare the values of `metric` on the seeds that both models share. Parameters: | Name | Type | Description | Default | | -------- | ----- | ----------------------------------------------------------------------------------------- | ------------ | | `metric` | `str` | The metric to compare. | *required* | | `method` | `str` | "wilcoxon" (the default) for a Wilcoxon signed-rank test, or "ttest" for a paired t-test. | `'wilcoxon'` | Returns: | Type | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `'pd.DataFrame'` | A pandas DataFrame with one row and one column per model. The diagonal is 1, and a pair with fewer than two shared seeds has the value NaN. | Raises: | Type | Description | | ------------- | ------------------------------------------------------------------------------- | | `ImportError` | If pandas or scipy is not installed. | | `ValueError` | If method is not supported, or if fewer than two models have values for metric. | #### `table(show_significance=False)` Return the results as a text table. The table is drawn with Unicode box-drawing characters and needs no optional dependency. Parameters: | Name | Type | Description | Default | | ------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `show_significance` | `bool` | If True, mark the best model with * or \*\* when a Wilcoxon signed-rank test against another model gives p < 0.05 or p < 0.01. The test needs at least 6 seeds shared by both models, so with fewer seeds no model is marked, and a warning explains why. The test needs pandas, from the benchmark extra. | `False` | Returns: | Type | Description | | ----- | --------------------------------------------------------------------------------------------- | | `str` | A string with one row per model and one column per metric. Each value is shown as mean ± std. | Raises: | Type | Description | | ------------- | --------------------------------------------------------- | | `ImportError` | If show_significance is True and pandas is not installed. | #### `to_dataframe()` Return the raw results of every seed as a pandas DataFrame. Returns: | Type | Description | | ---------------- | ---------------------------------------------------------------------------------------------- | | `'pd.DataFrame'` | A DataFrame with the columns model, seed, metric and value, and one row for each entry of raw. | Raises: | Type | Description | | ------------- | --------------------------- | | `ImportError` | If pandas is not installed. | ### `compare(models, dataset='breast_cancer_wisconsin', metrics=None, n_seeds=5, test_size=0.15, show_progress=False, save_dir=None, store_predictions=False, validation_size=0.15)` Compare eigenkit models and built-in scikit-learn models on a dataset. Each model must be a built-in scikit-learn estimator (from the `sklearn` package), or a subclass of BaseQuantumEstimator, BaseQuantumClassifier or BaseQuantumRegressor. For every seed, each model is copied with `sklearn.base.clone()`. Parameters: | Name | Type | Description | Default | | ------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `models` | \`list | dict[str, Any]\` | The estimator instances to compare. If a list is passed, the display names come from type(model).__name__, and repeated names are numbered (for example, SVC_0 and SVC_1). If a dict is passed, its keys are used as display names. Dict keys must be non-empty strings, with no whitespace at the start or the end and no control characters. Use a different key for each model. If the same key appears twice in a Python dict literal, Python keeps only the last model before compare() receives the dict. When save_dir is set, the keys must also be valid prefixes for file names. | | `dataset` | \`str | dict\` | A dataset name, or a dict that is already split, with X_train, y_train, X_test and y_test. Such a dict can also include X_val and y_val. | | `metrics` | \`list | None\` | The metrics to compute. For classification, the options are accuracy, f1, precision, recall, roc_auc, fit_time and predict_time. For regression, they are r2, mae, rmse, mse, fit_time and predict_time. A custom function must be wrapped with custom_metric(...), so that its direction (higher or lower is better) is explicit. The default depends on the type of task of the first model. | | `n_seeds` | `int` | The number of seeds to average over. The default is 5. | `5` | | `test_size` | `float` | The fraction of the data kept for the final test set. | `0.15` | | `show_progress` | `bool` | Whether to print the progress of the benchmark, with the current model and an estimate of the remaining time. The default is False, which runs silently. | `False` | | `save_dir` | \`str | Path | None\` | | `store_predictions` | `bool` | If True, the result keeps y_true, y_pred and y_prob for the last seed of each model. | `False` | | `validation_size` | `float` | The fraction of the data kept for validation. | `0.15` | Returns: | Type | Description | | ----------------- | ----------------------------------------------------------------------------------- | | `BenchmarkResult` | An object with the results, and methods such as table(), plot() and to_dataframe(). | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------------------------ | | `ValueError` | If models is empty, if a model name is not valid, or if dataset is not well formed. | | `TypeError` | If a model name is not a string, or if a model is not a supported type of estimator. | ### `custom_metric(fn, *, direction, name=None)` Define a custom metric and the direction in which it improves. Parameters: | Name | Type | Description | Default | | ----------- | ------------------------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | | `fn` | `Callable[[ndarray, ndarray], float]` | The metric function. It receives (y_true, y_pred) and returns a number. | *required* | | `direction` | `MetricDirection` | "higher" if higher values are better, or "lower" if lower values are better. | *required* | | `name` | \`str | None\` | The column label of the metric. The default is the name of fn. | Returns: | Type | Description | | -------------- | --------------------------------------------------------- | | `CustomMetric` | A metric specification to pass to compare(metrics=[...]). | ### `load_checkpoint(path, *, model=None)` Load a model checkpoint written by `compare(..., save_dir=...)`. Parameters: | Name | Type | Description | Default | | ------- | ----- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | `path` | \`str | Path\` | The path to the checkpoint file (\*.eigenkit or \*.joblib). | | `model` | `Any` | The nn.Module architecture for GenericRegressor or GenericClassifier checkpoints. It is ignored for other types of estimator. | `None` | Returns: | Type | Description | | ----- | --------------------- | | `Any` | The loaded estimator. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------ | | `TypeError` | If a .joblib file does not contain a scikit-learn estimator. | | `ValueError` | If the file extension is not supported. | ### `quick_baselines()` Return a list of common scikit-learn classifiers with sensible defaults. The list contains a `SVC` with a radial basis function (RBF) kernel and probabilities, so that `roc_auc` can be computed, a `RandomForestClassifier`, a `LogisticRegression` and a `KNeighborsClassifier`. `XGBClassifier` and `LGBMClassifier` are added when their packages can be imported. scikit-learn 1.9 deprecates `SVC(probability=True)`. From that version, the support vector classifier is wrapped as `CalibratedClassifierCV(SVC(), ensemble=False)`, the replacement that scikit-learn documents, so the results table calls it `CalibratedClassifierCV`. Returns: | Type | Description | | ----------- | ------------------------------------------------------------------------------------- | | `list[Any]` | A list of classifiers that follow the scikit-learn interface, created but not fitted. | Example ```python from eigenkit.benchmark import compare, quick_baselines results = compare(models=quick_baselines(), dataset="iris", n_seeds=2) ``` ### `timer()` Measure the wall-clock time of a block of code. Yields: | Type | Description | | ------ | -------------------------------------------------------------------------------------- | | `dict` | A dict. When the with block ends, its "elapsed" key holds the elapsed time in seconds. | Example ```python with timer() as t: do_work() print(t["elapsed"]) ```