The Test Suite#

Overview#

The test suite works under pytest. The suite tests three different elements for every inference method:

  • a correctness/statistical-power test on simulated data

  • an exceptions/error-handling test class, and, where the object wraps a scikit-learn estimator

  • a scikit-learn estimator guidelines compliance test.

We present here the structure of the test suite organization, the logic behind tests, and quickly present important fixtures, and other important utility functions.

Tests are automatically being run after each push on a pull-request (PR) whose target branch is main. They are run on all combinations of python versions in [3.10, 3.14], and different OS structures [Windows, Mac, Linux]. We target a maximal code coverage with tests, meaning that tests should cover every line of existing code. To ensure this, we have the CodeCov utility bot that automatically computes the code coverage of tests after a successful commit on a PR, meaning that the entire test suite run is successful, and that the documentation generation process by Sphinx is executed without any error. The code coverage metric should then appear after the execution of tests and documentation generation process directly on the PR discussion on GitHub.

How the Suite Is Organized#

File Inventory#

File

What it tests

conftest.py

Shared pytest fixtures and helpers (no tests of its own)

__init__.py

Empty — makes the test folder a package so relative imports (from .conftest import ...) work

test_version.py

Sanity check that hidimstat.__version__ is a valid string

test_base_perturbation.py

BasePerturbation / BasePerturbationCV base classes shared by perturbation-style importance methods

test_base_variable_importance.py

BaseVariableImportance — the shared selection/plotting logic (top-k, percentile, threshold, FDR, FWER selection; plotting)

test_permutation_feature_importance.py

PFI / PFICV (Permutation Feature Importance)

test_conditional_feature_importance.py

CFI / CFICV (Conditional Feature Importance)

test_leave_one_covariate_in.py

LOCI / LOCICV (Leave-One-Covariate-In)

test_leave_one_covariate_out.py

LOCO / LOCOCV (Leave-One-Covariate-Out)

test_desparsified_lasso.py

DesparsifiedLasso, the reid/group-Reid noise estimators

test_knockoff.py

ModelXKnockoff, GaussianKnockoffs (Model-X knockoffs)

test_distilled_conditional_randomization_test.py

D0CRT (distilled conditional randomization test / dCRT)

test_ensemble_clustered_inference.py

CluDL / EnCluDL (clustered and ensembled-clustered desparsified Lasso inference on spatial/temporal data)

_utils/

The data generation methods, and various method specific and non-specific utility functions.

samplers/

Conditional sampling functions, and samplers for Gaussian knockoffs.

statistical_tools/

Statistical tests functions for test aggregations, multiple testing, Nadeau-Bengio T-tests, and computations of p-values.

The test suite follows the convention that there is a single test file for each single source python file. It ensures that each estimator or family of estimators has its own test module named after the corresponding source module. This keeps tests discoverable and lets a contributor working on one method run just pytest test_<method>.py.

Test organization logic#

For feature importance methods, tests fall into three categories:

  • Statistical correctness on simulated data:

    Tests generate synthetic data with a known support (important vs. non-important features) via data simulation functions found in hidimstat._utils.scenario, run the method, and check that:

    • importance scores are higher on truly important features than on null features,

    • a selection procedure keeps the empirical false discovery proportion (FDP) below the target level and achieves reasonable statistical power.

    Because these are stochastic tests, many are repeated over multiple seeds and the assertions compare an average FDP/power against a target plus a small tolerance, rather than requiring every single draw to succeed.

  • API and exception behavior:

    Dedicated classes or functions check that invalid inputs raise the right error type with the expected message, that calling .importance() before .fit() fails clearly, and that warnings are raised as expected.

  • Scikit-learn estimator compliance:

    As feature importance methods are written in compliance to scikit-learn standards, each one of them is checked against scikit-learn’s own estimator conformance checks, so they behave like any other scikit-learn estimator.

Reproducibility testing#

Modules that can be seeded run the same four-way check on random_state handling:

  • an integer seed gives identical results across repeated calls and across separate instances;

  • random_state=None gives different results on every call and every new instance;

  • a NumPy Generator object is consumed statefully. Repeated calls on the same instance differ, but re-creating the generator with the same seed reproduces the original result.

This is important for any Monte Carlo–style method where reproducibility guarantees matter.

All methods that involve randomness must include a dedicated test suite. The following test patterns help to ensure consistent behavior:

  • test_<method>_repeatability

  • test_<method>_randomness_with_none

  • test_<method>_reproducibility_with_integer

  • test_<method>_reproducibility_with_rng

Important fixtures and utilities#

Defined in the testing configuration file#

pytest_configure(config)

Forces matplotlib’s non-interactive Agg backend so plotting tests never pop up a window during a test run.

rng (function-scoped)

Returns a freshly seeded numpy.random.default_rng(42). Used throughout the suite whenever a test needs its own controlled randomness (shuffling arrays, drawing p-values, generating toy datasets) independent of the main data-generation fixture below.

data_generator (function-scoped, parametrized)

The workhorse fixture of the whole suite. It depends on eight parameters — n_samples, n_features, support_size, rho, seed, value, signal_noise_ratio, rho_serial — normally supplied via @pytest.mark.parametrize on the test (or test class), and calls multivariate_simulation to build a linear-model dataset with a known sparse support. It returns (X, y, important_features, not_important_features) — the design matrix, the response, and the index arrays a test needs to check that a method ranks/selects the right features. Because the parametrize decorator sits on the test, different tests can reuse this one fixture with completely different problem sizes and noise regimes (e.g. high-dimensional n_features=200 cases, correlated features via rho, or noisy targets via signal_noise_ratio).

fitted_linear_regression() / _fitted_linear_regression()

A plain helper (not a fixture) that fits a LinearRegression on a tiny random 2-column dataset, used to seed estimator-check lists with an already-fitted model — several checks specifically probe fitted-vs-unfitted behavior.

check_estimator(estimators, return_expected_failed_checks, valid=True)

The sklearn < 1.6 compatibility helper described above; not a fixture in the pytest sense, but a generator consumed by @pytest.mark.parametrize to build (estimator, check, name) tuples for valid or intentionally-invalid checks.

Local file-specific fixtures#

Several test files define their own narrower fixtures on top of the shared ones, generally to avoid refitting an expensive model in every test. Please make sure that you have checked module-scoped fixtures before implementing a utility function. Fixtures can generally be found at the beginning of the file.

Non-fixtures testing utilities#

If you ever need a non-fixture utility function, please make sure that you check existing functions in the separate _utils, samplers, and statistical_tools modules before reimplementing a method.

Summary#

The suite is thought to ensure having reusable testing recipes, applied consistently to every inference method in the library. The testing configuration file conftest.py supplies the two fixtures rng and data_generator, and one helper check_estimator that make this consistency possible, while each test module adds a thin, method-specific layer of fixtures and parametrizations on top.