Slice discovery, cohort comparison & drift¶

This tutorial closes the ML error-analysis arc. After the one-call error_analysis() wrapper and calibration tools told you how well a model is calibrated and which single variable drives errors, these APIs answer "which multivariable subgroup is the model failing on, and has the data shifted?"

Function	Purpose
discover_error_slices	Auto-discover multivariable cohorts where errors concentrate (DivExplorer / SliceLine analogue)
compare_cohorts	Quantify how two cohorts differ (ΔAIC + distribution diffs, Sweetviz-style)
detect_drift	Rank features by train→prod drift
regression_calibration_table / multiclass_calibration_table	Calibration beyond the binary case

The discovery search prunes on support (Apriori, anti-monotone) — not ΔAIC, which has no sound upper bound.

In [1]:

import numpy as np
import pandas as pd

import pycatdap
from pycatdap.error import (
    compare_cohorts,
    detect_drift,
    discover_error_slices,
    multiclass_calibration_table,
    multiclass_expected_calibration_error,
    regression_calibration_error,
    regression_calibration_table,
)

print("pycatdap", pycatdap.__version__)

import numpy as np import pandas as pd import pycatdap from pycatdap.error import ( compare_cohorts, detect_drift, discover_error_slices, multiclass_calibration_table, multiclass_expected_calibration_error, regression_calibration_error, regression_calibration_table, ) print("pycatdap", pycatdap.__version__)

pycatdap 0.7.1.dev7

1. Synthetic classification data¶

We craft a dataset where a model fails disproportionately on a specific cohort: older customers on the basic plan. Everywhere else the model is mostly right.

In [2]:

rng = np.random.default_rng(0)
n = 2000

age = rng.integers(18, 80, size=n).astype(float)
plan = rng.choice(["basic", "pro", "enterprise"], size=n)
region = rng.choice(["north", "south", "east", "west"], size=n)
df = pd.DataFrame({"age": age, "plan": plan, "region": region})

y_true = rng.integers(0, 2, size=n)
# errors concentrate where age >= 60 AND plan == "basic"
bad = (age >= 60) & (plan == "basic")
flip = np.where(bad, rng.random(n) < 0.75, rng.random(n) < 0.06)
y_pred = np.where(flip, 1 - y_true, y_true)

print(f"overall error rate: {(y_true != y_pred).mean():.3f}")
print(f"bad-cohort error rate: {(y_true != y_pred)[bad].mean():.3f}")

rng = np.random.default_rng(0) n = 2000 age = rng.integers(18, 80, size=n).astype(float) plan = rng.choice(["basic", "pro", "enterprise"], size=n) region = rng.choice(["north", "south", "east", "west"], size=n) df = pd.DataFrame({"age": age, "plan": plan, "region": region}) y_true = rng.integers(0, 2, size=n) # errors concentrate where age >= 60 AND plan == "basic" bad = (age >= 60) & (plan == "basic") flip = np.where(bad, rng.random(n) < 0.75, rng.random(n) < 0.06) y_pred = np.where(flip, 1 - y_true, y_true) print(f"overall error rate: {(y_true != y_pred).mean():.3f}") print(f"bad-cohort error rate: {(y_true != y_pred)[bad].mean():.3f}")

overall error rate: 0.143
bad-cohort error rate: 0.765

2. discover_error_slices¶

One call surfaces the worst multivariable cohorts, ranked by interestingness. Continuous columns (age) are AIC-binned against the error label, so the bins land where the error rate actually shifts.

In [3]:

result = discover_error_slices(
    df,
    y_true,
    y_pred,
    max_vars=3,
    measure="aic",
    top_k=8,
    min_support=40,
)
print(
    f"measure={result.measure}  evaluated={result.n_evaluated}  "
    f"pruned={result.n_pruned}"
)
print(
    f"search-space reduction: "
    f"{result.n_pruned / (result.n_evaluated + result.n_pruned):.1%}"
)
for s in result.slices[:6]:
    print(f"  err={s.error_metric:.2f}  n={s.size:4d}  | {s.description}")

result = discover_error_slices( df, y_true, y_pred, max_vars=3, measure="aic", top_k=8, min_support=40, ) print( f"measure={result.measure} evaluated={result.n_evaluated} " f"pruned={result.n_pruned}" ) print( f"search-space reduction: " f"{result.n_pruned / (result.n_evaluated + result.n_pruned):.1%}" ) for s in result.slices[:6]: print(f" err={s.error_metric:.2f} n={s.size:4d} | {s.description}")

measure=aic  evaluated=159  pruned=180
search-space reduction: 53.1%
  err=0.78  n= 151  | age ∈ [65.5895, 77.792] × plan = basic
  err=0.73  n=  52  | age ∈ [59.4883, 64.3693] × plan = basic
  err=0.79  n=  42  | age ∈ [65.5895, 77.792] × plan = basic × region = west
  err=0.73  n=  51  | age ∈ [65.5895, 77.792] × plan = basic × region = north
  err=0.31  n= 685  | plan = basic
  err=0.33  n= 390  | age ∈ [65.5895, 77.792]

The high-error cohort (age ∈ [...] × plan = basic) surfaces at the top. The search-space reduction quantifies how much support-pruning saved.

A flat, DivExplorer-compatible table is one call away:

In [4]:

result.to_divexplorer_format().head(6)

result.to_divexplorer_format().head(6)

Out[4]:

	description	size	error_rate	delta_aic	measure_value	n_error_in_slice
0	age ∈ [65.5895, 77.792] × plan = basic	151	0.781457	-583.006369	583.006369	118
1	age ∈ [59.4883, 64.3693] × plan = basic	52	0.730769	-583.006369	583.006369	38
2	age ∈ [65.5895, 77.792] × plan = basic × regio...	42	0.785714	-405.813627	405.813627	33
3	age ∈ [65.5895, 77.792] × plan = basic × regio...	51	0.725490	-405.813627	405.813627	37
4	plan = basic	685	0.310949	-223.188619	223.188619	213
5	age ∈ [65.5895, 77.792]	390	0.328205	-219.627548	219.627548	128

Pluggable interestingness measure¶

Swap measure= for any registered measure — "cramers_v", "mutual_info", or your own callable taking a contingency table.

In [5]:

cv = discover_error_slices(
    df, y_true, y_pred, measure="cramers_v", max_vars=2, top_k=3, min_support=40
)
[(round(s.measure_value, 3), s.description) for s in cv.slices]

cv = discover_error_slices( df, y_true, y_pred, measure="cramers_v", max_vars=2, top_k=3, min_support=40 ) [(round(s.measure_value, 3), s.description) for s in cv.slices]

Out[5]:

[(0.675, 'age ∈ [65.5895, 77.792] × plan = basic'),
 (0.675, 'age ∈ [59.4883, 64.3693] × plan = basic'),
 (0.387, 'age ∈ [65.5895, 77.792] × region = north')]

3. compare_cohorts¶

Treat cohort membership as a synthetic response: each feature's ΔAIC against it measures how strongly that feature distinguishes the two cohorts.

In [6]:

# cohort B over-represents the south region
a = df.copy()
b = df.copy()
b["region"] = rng.choice(
    ["north", "south", "east", "west"], size=n, p=[0.1, 0.7, 0.1, 0.1]
)

cmp = compare_cohorts(a, b)
cmp.summary

# cohort B over-represents the south region a = df.copy() b = df.copy() b["region"] = rng.choice( ["north", "south", "east", "west"], size=n, p=[0.1, 0.7, 0.1, 0.1] ) cmp = compare_cohorts(a, b) cmp.summary

Out[6]:

	variable	delta_aic	max_abs_diff
0	region	-813.331474	0.444
1	age	0.000000	0.000
2	plan	4.000000	0.000

In [7]:

cmp.distributions["region"]

cmp.distributions["region"]

Out[7]:

	value	prop_a	prop_b	diff
0	east	0.2460	0.1025	0.1435
1	north	0.2575	0.1015	0.1560
2	south	0.2535	0.6975	-0.4440
3	west	0.2430	0.0985	0.1445

4. detect_drift¶

A thin specialisation for the train→prod setting: features ranked by drift magnitude, plus the production error rate when predictions are supplied.

In [8]:

report = detect_drift(a, b, y_true=y_true, y_pred=y_pred)
print(f"prod error rate: {report.error_rate_prod:.3f}")
report.drift_ranking

report = detect_drift(a, b, y_true=y_true, y_pred=y_pred) print(f"prod error rate: {report.error_rate_prod:.3f}") report.drift_ranking

prod error rate: 0.143

Out[8]:

	variable	delta_aic
0	region	-813.331474
1	plan	4.000000
2	age	0.000000

5. Calibration beyond binary¶

The calibration tutorial covered binary calibration; here we add the regression and multi-class one-vs-rest variants.

In [9]:

# Regression: predicted-vs-actual quantile calibration
y_pred_reg = rng.uniform(0, 100, size=1000)
y_true_reg = y_pred_reg + rng.normal(0, 8, size=1000) + 5.0  # slight bias
reg_table = regression_calibration_table(y_true_reg, y_pred_reg, n_quantiles=5)
print(
    f"regression calibration error: "
    f"{regression_calibration_error(y_true_reg, y_pred_reg):.3f}"
)
reg_table

# Regression: predicted-vs-actual quantile calibration y_pred_reg = rng.uniform(0, 100, size=1000) y_true_reg = y_pred_reg + rng.normal(0, 8, size=1000) + 5.0 # slight bias reg_table = regression_calibration_table(y_true_reg, y_pred_reg, n_quantiles=5) print( f"regression calibration error: " f"{regression_calibration_error(y_true_reg, y_pred_reg):.3f}" ) reg_table

regression calibration error: 4.934

Out[9]:

	bin_low	bin_high	n	pred_mean	actual_mean	ci_low	ci_high
0	0.038105	21.292481	200.0	10.504333	15.793703	14.468704	17.118702
1	21.336990	40.782052	200.0	30.970713	36.744616	35.386071	38.103160
2	40.853202	57.984191	200.0	48.724230	52.342721	50.985176	53.700266
3	58.159458	79.727106	200.0	69.522185	74.368831	72.939651	75.798011
4	79.809989	99.974107	200.0	89.908993	95.051633	93.630640	96.472626

In [10]:

# Multi-class one-vs-rest
proba = rng.dirichlet(np.ones(3), size=1500)
y_mc = np.array([rng.choice(3, p=row) for row in proba])
tables = multiclass_calibration_table(y_mc, proba, strategy="equal_width", n_bins=5)
macro_ece = multiclass_expected_calibration_error(
    y_mc, proba, strategy="equal_width", n_bins=5
)
print(f"macro-averaged OvR ECE: {macro_ece:.3f}")
print("per-class tables:", list(tables.keys()))
tables[0].head()

# Multi-class one-vs-rest proba = rng.dirichlet(np.ones(3), size=1500) y_mc = np.array([rng.choice(3, p=row) for row in proba]) tables = multiclass_calibration_table(y_mc, proba, strategy="equal_width", n_bins=5) macro_ece = multiclass_expected_calibration_error( y_mc, proba, strategy="equal_width", n_bins=5 ) print(f"macro-averaged OvR ECE: {macro_ece:.3f}") print("per-class tables:", list(tables.keys())) tables[0].head()

macro-averaged OvR ECE: 0.018
per-class tables: [0, 1, 2]

Out[10]:

	bin_low	bin_high	n	prob_pred	prob_true	ci_low	ci_high
0	0.000151	0.199073	543	0.095738	0.090239	0.068931	0.117305
1	0.200604	0.399671	440	0.291537	0.277273	0.237510	0.320891
2	0.400722	0.599730	287	0.490762	0.512195	0.454588	0.569480
3	0.600176	0.798254	175	0.686831	0.680000	0.607658	0.744609
4	0.801854	0.992850	55	0.874059	0.800000	0.676351	0.884478

Summary¶

These APIs turn error analysis from "how good is the model?" into "where and why does it fail, and has the world changed?" — all on the same AIC foundation, scipy-free.

• discover_error_slices — support-pruned multivariable cohort search
• compare_cohorts / detect_drift — distribution & relationship shift
• regression + multi-class calibration — calibration for every task type

This is the final tutorial in the series.