Performance¶

pycatdap ships a benchmark suite (benchmarks/) that tracks the runtime of the hot paths so performance regressions can be caught. This page documents how to run it and records a baseline.

Scope

Phase 1 (issue #29) added the locally runnable harness and the baseline below. Phase 2 + 3 (issue #161) add a non-blocking benchmark workflow (see below): a nightly run plus a delta comment on code-changing PRs. Benchmarks are intentionally never part of make ci — they do not run in the standard test gate, and the workflow never blocks a PR or commit.

Running the benchmarks¶

make bench

This runs every benchmark under benchmarks/ via pytest-benchmark (a dev-only dependency in the bench dependency group; it is never installed for make ci and never shipped in the wheel). The suite lives outside testpaths, so make test and make ci do not collect it.

Pass extra flags with BENCH_ARGS:

# Skip the slow slice-discovery benchmark
make bench BENCH_ARGS="-m 'not slow'"

# Save a named baseline and compare a later run against it
make bench BENCH_ARGS="--benchmark-save=baseline"
make bench BENCH_ARGS="--benchmark-compare=0001 --benchmark-compare-fail=mean:20%"

# Emit machine-readable JSON
make bench BENCH_ARGS="--benchmark-json=results.json"

What is measured¶

Benchmark	Function	Workload
`bench_catdap1`	`catdap1`	100 / 1k / 10k / 100k rows × 10 all-categorical columns (default all-response call)
`bench_catdap2`	`catdap2`	100 / 1k / 10k rows × 5 / 10 mixed columns, `nvar=5`
`bench_pooling`	`optimal_binning`	1k / 10k / 100k rows, bottom-up (unequal) pooling
`bench_discovery`	`discover_error_slices`	Adult-Income-like synthetic frame (5k rows × 14 mixed columns), `max_vars=2` — marked `slow`

All inputs are generated by deterministic, fixed-seed synthetic generators (benchmarks/_data.py), so the workload is identical across runs and machines; only timing varies.

Baseline¶

Recorded with make bench on the following machine. Absolute numbers are hardware-dependent — treat them as a relative reference, not a contract.

CPU: Intel Core i9-14900K
OS / arch: Linux x86_64 (WSL2)
Python: 3.11.14
pycatdap: 0.13.x (develop)
Date: 2026-06-07

Times are the minimum over the measured rounds (least noisy estimator), in seconds.

catdap1 — pairwise categorical AIC (10 columns)¶

Rows	Min (s)
100	0.36
1,000	0.39
10,000	0.62
100,000	2.79

catdap2 — AIC subset search (`nvar=5`, mixed types)¶

Rows	5 cols (s)	10 cols (s)
100	0.08	0.20
1,000	0.26	0.64
10,000	1.81	4.30

optimal_binning — AIC-optimal continuous binning (bottom-up)¶

Rows	Min (s)
1,000	0.010
10,000	0.017
100,000	0.055

discover_error_slices — bounded slice discovery (14 columns)¶

Workload	Min (s)
Adult-Income-like, 5k rows, `max_vars=2`	6.98

This is the #20 acceptance scenario (CATDAP-02 on a 14-column dataset). At these bounded parameters it completes well under the 30 s target. See the caveats below for why it is bounded and synthetic.

Methodology and caveats¶

Synthetic, not real datasets. bench_discovery mimics the Adult Income shape (6 continuous + 8 categorical columns) rather than calling fetch_adult_income. Fetching from OpenML would inject network-timing variance into the measurement and, with scikit-learn installed, can hang the full test run for hours.
Bounded by design. Unbounded slice discovery on the full Adult Income has caused out-of-memory crashes. The discovery benchmark is therefore subsampled to 5k rows with max_vars=2 and a capped max_candidates, and is marked slow so it is opt-out via BENCH_ARGS="-m 'not slow'".
nvar is pinned for catdap2. With nvar=None, catdap2 searches every subset of all columns (2^cols), which is intractable past ~10 columns. The benchmark pins nvar=5 to measure a realistic, bounded search.
Continuous data has finite precision. optimal_binning starts from fine-grained bins of width accuracy (auto-detected as the smallest gap between sorted unique values), then greedily merges. All-unique random floats would explode the initial bin count, making the merge ~O(n_bins³). The auto-accuracy path therefore caps the initial grid at 256 bins and warns when it does so (issue #164 / H-0024), keeping the cost bounded for any input; pass an explicit accuracy for finer control. The generators still use bounded-precision (rounded) continuous values, so the benchmark measures the genuine O(rows) cost rather than the capped fallback.
Hosted-runner noise. GitHub-hosted runners share CPU and show high timing variance, which is why the CI regression check is a non-blocking nightly alert rather than a per-PR gate.

Continuous benchmarking (nightly)¶

The Benchmarks workflow (.github/workflows/benchmarks.yml) runs the suite on a GitHub-hosted runner and records the results so trends and regressions are visible over time. It is non-blocking by design — it never fails a PR or commit.

Schedule: nightly (03:00 UTC) plus manual workflow_dispatch. Scheduled runs fire from the default branch (main), so the nightly cadence activates once this workflow reaches main (a release); until then, trigger it manually on develop.
History + dashboard: stored by github-action-benchmark in the gh-pages branch under dev/bench/. The public docs site is served from the GitHub Actions artifact (not a branch), so the chart is not on the public site — view it by checking out the gh-pages branch. The raw output.json is also uploaded as a workflow artifact each run.
Regression alert: if a benchmark is ≥ 1.5× slower than the previous recorded run, the action posts a commit comment (alert-threshold: 150%, intentionally above +20% because hosted-runner timing is noisy). It does not fail the run (fail-on-alert: false).
On PRs touching src/pycatdap/** (or the benchmark plumbing), the workflow runs the non-slow subset, compares against the gh-pages baseline without writing history, and leaves a delta commit comment on the PR head commit (same-repo PRs; fork PRs validate read-only with no comment). It never blocks the PR.