One-call ML error analysis with error_analysis()¶

error_analysis() is the flagship ML error analysis entry point. It composes the error labellers (error_label / confusion_label / residual_label, Notebook 10) with target_analysis() (Notebook 09) into a single result object:

result = pycatdap.error_analysis(df, y_true, y_pred)
result.show()
result.to_html("report.html")
result.to_plotly_json()
result.to_divexplorer_format()

This notebook walks through:

1. Binary classification on German Credit with a synthetic classifier — confusion_label path with TP/FP/FN/TN, slice extraction
2. Multiclass on Palmer Penguins — automatic fallback to error_label (the multiclass guard)
3. Regression on synthetic data with engineered bias — residual_label path with under/over-prediction slices
4. The 5-method serialisation contract — show / to_dict / to_html / to_plotly_json / to_divexplorer_format
5. Implementation safeguards — the F-1 / F-2 / F-3 guards extracted by cross-check

All three datasets ship bundled with pycatdap. Three additional fetch loaders (fetch_california_housing / fetch_adult_income / fetch_compas) are available for larger benchmarks — they live under pycatdap[data] extras (pip install 'pycatdap[data]') and are demoed in the final section.

1. Binary classification — German Credit¶

In [ ]:

import numpy as np

import pycatdap
from pycatdap.datasets import load_german_credit

df = load_german_credit()
rng = np.random.default_rng(42)

# Synthetic classifier: harder on under-30 applicants and on those
# with low checking_status, so error_analysis should surface
# concentrated slices on those variables.
y_true = df["class"]
noise = np.where(df["age"] < 30, 0.4, 0.10)
noise = np.where(df["checking_status"] == "<0", noise + 0.10, noise)
flip = rng.random(len(df)) < noise
y_pred = y_true.copy()
y_pred.loc[flip] = y_true.loc[flip].map({"good": "bad", "bad": "good"})

accuracy = (y_true == y_pred).mean()
print(f"synthetic accuracy: {accuracy:.3f}")

import numpy as np import pycatdap from pycatdap.datasets import load_german_credit df = load_german_credit() rng = np.random.default_rng(42) # Synthetic classifier: harder on under-30 applicants and on those # with low checking_status, so error_analysis should surface # concentrated slices on those variables. y_true = df["class"] noise = np.where(df["age"] < 30, 0.4, 0.10) noise = np.where(df["checking_status"] == "<0", noise + 0.10, noise) flip = rng.random(len(df)) < noise y_pred = y_true.copy() y_pred.loc[flip] = y_true.loc[flip].map({"good": "bad", "bad": "good"}) accuracy = (y_true == y_pred).mean() print(f"synthetic accuracy: {accuracy:.3f}")

In [ ]:

result = pycatdap.error_analysis(
    df.drop(columns=["class"]),
    y_true.to_numpy(),
    y_pred.to_numpy(),
    top_k=4,
    positive="bad",
)

print(f"task         = {result.task}")
print(f"label_kind   = {result.label_kind}")
print(f"n_correct    = {result.n_correct}")
print(f"n_incorrect  = {result.n_incorrect}")
print()
print("Confusion (canonical TP/FP/FN/TN order):")
print(result.confusion)

result = pycatdap.error_analysis( df.drop(columns=["class"]), y_true.to_numpy(), y_pred.to_numpy(), top_k=4, positive="bad", ) print(f"task = {result.task}") print(f"label_kind = {result.label_kind}") print(f"n_correct = {result.n_correct}") print(f"n_incorrect = {result.n_incorrect}") print() print("Confusion (canonical TP/FP/FN/TN order):") print(result.confusion)

The feature ranking is the standard target_analysis output computed against the synthetic confusion label column. Variables with the most negative ΔAIC are the strongest discriminators of correct-vs-incorrect predictions.

In [ ]:

result.feature_ranking.head(8)

result.feature_ranking.head(8)

top_slices surfaces specific (variable, category) cells where one of FP / FN concentrates with |pearson_residual| ≥ 2.0. By construction we biased the classifier on age < 30 and checking_status='<0', so we expect slices on those.

In [ ]:

for s in result.top_slices:
    print(
        f"{s.variable:24s} {s.category!s:18s} "
        f"err={s.error_category:3s} "
        f"rate={s.error_rate:.3f} "
        f"residual={s.pearson_residual:+.2f} "
        f"ΔAIC(var)={s.delta_aic:+.2f}"
    )

for s in result.top_slices: print( f"{s.variable:24s} {s.category!s:18s} " f"err={s.error_category:3s} " f"rate={s.error_rate:.3f} " f"residual={s.pearson_residual:+.2f} " f"ΔAIC(var)={s.delta_aic:+.2f}" )

2. Multiclass — Palmer Penguins¶

confusion_label raises NotImplementedError on 3+ unique values (multiclass confusion labelling is not yet supported). The error_analysis() wrapper detects multiclass at the front door and routes through error_label automatically — no exception, the user gets a correct/incorrect labelled ErrorAnalysisResult without needing to know about the guard.

In [ ]:

from pycatdap.datasets import load_penguins

penguins = load_penguins().dropna(subset=["species", "body_mass_g"]).copy()
rng = np.random.default_rng(11)
y_true_mc = penguins["species"]
# Biased classifier: confuses Adelie ⇄ Chinstrap roughly 30% of the time,
# Gentoo stays mostly correct.
y_pred_mc = y_true_mc.copy()
swap_mask = (y_true_mc.isin(["Adelie", "Chinstrap"])) & (
    rng.random(len(penguins)) < 0.30
)
y_pred_mc.loc[swap_mask] = y_true_mc.loc[swap_mask].map(
    {"Adelie": "Chinstrap", "Chinstrap": "Adelie"}
)

mc_result = pycatdap.error_analysis(
    penguins.drop(columns=["species"]),
    y_true_mc.to_numpy(),
    y_pred_mc.to_numpy(),
    top_k=3,
)

print(f"task       = {mc_result.task}")
print(f"label_kind = {mc_result.label_kind}")  # error_label, not confusion_label
print(f"confusion  = {mc_result.confusion}")  # None for multiclass
print()
print(mc_result.feature_ranking)

from pycatdap.datasets import load_penguins penguins = load_penguins().dropna(subset=["species", "body_mass_g"]).copy() rng = np.random.default_rng(11) y_true_mc = penguins["species"] # Biased classifier: confuses Adelie ⇄ Chinstrap roughly 30% of the time, # Gentoo stays mostly correct. y_pred_mc = y_true_mc.copy() swap_mask = (y_true_mc.isin(["Adelie", "Chinstrap"])) & ( rng.random(len(penguins)) < 0.30 ) y_pred_mc.loc[swap_mask] = y_true_mc.loc[swap_mask].map( {"Adelie": "Chinstrap", "Chinstrap": "Adelie"} ) mc_result = pycatdap.error_analysis( penguins.drop(columns=["species"]), y_true_mc.to_numpy(), y_pred_mc.to_numpy(), top_k=3, ) print(f"task = {mc_result.task}") print(f"label_kind = {mc_result.label_kind}") # error_label, not confusion_label print(f"confusion = {mc_result.confusion}") # None for multiclass print() print(mc_result.feature_ranking)

3. Regression — engineered bias¶

We build a synthetic regression task where the predictor systematically under-predicts in one group. error_analysis should:

1. Auto-detect task="regression" from the continuous outputs
2. Bin residuals via residual_label(method="aic_pool") — F-3 guarantees bin_0 = under-prediction, bin_{n-1} = over-prediction
3. Surface slices on group=hi for the under-prediction bin

In [ ]:

import pandas as pd

rng = np.random.default_rng(23)
n = 400
group = rng.choice(["lo", "hi"], size=n)
noisy = rng.choice(["x", "y", "z"], size=n)
y_true_reg = np.where(group == "hi", 10.0, 0.0) + rng.normal(0, 1.0, size=n)
bias = np.where(group == "hi", -3.0, 0.0)
y_pred_reg = y_true_reg + bias + rng.normal(0, 0.5, size=n)

reg_df = pd.DataFrame({"group": group, "noisy": noisy})

reg_result = pycatdap.error_analysis(reg_df, y_true_reg, y_pred_reg, top_k=2)

print(f"task       = {reg_result.task}")
print(f"label_kind = {reg_result.label_kind}")
print(f"MAE        = {reg_result.mae:.3f}")
print(f"RMSE       = {reg_result.rmse:.3f}")
print()
print("residual_pooling:")
for k, v in reg_result.residual_pooling.items():
    print(f"  {k:8s}: {v}")
print()
print("top_slices:")
for s in reg_result.top_slices:
    print(
        f"  {s.variable}={s.category} [{s.error_category}]: "
        f"residual={s.pearson_residual:+.2f}"
    )

import pandas as pd rng = np.random.default_rng(23) n = 400 group = rng.choice(["lo", "hi"], size=n) noisy = rng.choice(["x", "y", "z"], size=n) y_true_reg = np.where(group == "hi", 10.0, 0.0) + rng.normal(0, 1.0, size=n) bias = np.where(group == "hi", -3.0, 0.0) y_pred_reg = y_true_reg + bias + rng.normal(0, 0.5, size=n) reg_df = pd.DataFrame({"group": group, "noisy": noisy}) reg_result = pycatdap.error_analysis(reg_df, y_true_reg, y_pred_reg, top_k=2) print(f"task = {reg_result.task}") print(f"label_kind = {reg_result.label_kind}") print(f"MAE = {reg_result.mae:.3f}") print(f"RMSE = {reg_result.rmse:.3f}") print() print("residual_pooling:") for k, v in reg_result.residual_pooling.items(): print(f" {k:8s}: {v}") print() print("top_slices:") for s in reg_result.top_slices: print( f" {s.variable}={s.category} [{s.error_category}]: " f"residual={s.pearson_residual:+.2f}" )

4. The 5-method serialisation contract¶

Every ErrorAnalysisResult exposes the same five output paths:

Method	Purpose
.show()	Inline notebook / stdout summary
.to_dict()	JSON-safe Python dict
.to_html(path=None)	Self-contained Plotly-embedded HTML report
.to_plotly_json()	Per-section Plotly figure specs (for web frontends)
.to_divexplorer_format()	Flat DataFrame for the DivExplorer subgroup API

In [ ]:

d = result.to_dict()
print("keys:", sorted(d.keys()))
print(f"top_slices count: {len(d['top_slices'])}")
print(f"confusion: {d['confusion']}")

d = result.to_dict() print("keys:", sorted(d.keys())) print(f"top_slices count: {len(d['top_slices'])}") print(f"confusion: {d['confusion']}")

In [ ]:

result.to_divexplorer_format().head(10)

result.to_divexplorer_format().head(10)

In [ ]:

spec = result.to_plotly_json()
print("sections:", sorted(spec.keys()))
# spec["feature_ranking"] / spec["confusion"] / spec["top_summaries"] are
# all Plotly figure dicts, ready to render via plotly.io.from_json or
# directly with go.Figure(spec[...]). LizyStudio consumes this format
# directly (DP-4).

spec = result.to_plotly_json() print("sections:", sorted(spec.keys())) # spec["feature_ranking"] / spec["confusion"] / spec["top_summaries"] are # all Plotly figure dicts, ready to render via plotly.io.from_json or # directly with go.Figure(spec[...]). LizyStudio consumes this format # directly (DP-4).

5. Implementation safeguards¶

Three subtle traps were caught by the cross-check review during design (2026-05-28). Each one is exercised end-to-end here.

F-1: reserved column-name collision¶

In [ ]:

bad = df.assign(__pycatdap_confusion_label__=0)
try:
    pycatdap.error_analysis(bad, "class", "class")
except ValueError as exc:
    print("F-1 guard fired as expected:")
    print(" ", exc)

bad = df.assign(__pycatdap_confusion_label__=0) try: pycatdap.error_analysis(bad, "class", "class") except ValueError as exc: print("F-1 guard fired as expected:") print(" ", exc)

F-2: perfect classifier (FP/FN rows dropped by pd.crosstab)¶

In [ ]:

perfect = pycatdap.error_analysis(
    df.drop(columns=["class"]), y_true.to_numpy(), y_true.to_numpy()
)
# All 4 canonical rows present even though FP/FN are 0:
print(perfect.confusion)

perfect = pycatdap.error_analysis( df.drop(columns=["class"]), y_true.to_numpy(), y_true.to_numpy() ) # All 4 canonical rows present even though FP/FN are 0: print(perfect.confusion)

F-3: residual bin order is monotonic¶

equal_pooling boundaries are sorted ascending, so the categorical labels of residual_label are monotonic in residual value. bin_0 is always the most-under-predicted bucket, bin_{n-1} the most-over-predicted. The slice extraction in PR-G2 relies on this.

In [ ]:

labels_resid = pycatdap.error.residual_label(
    y_true_reg, y_pred_reg, method="aic_pool", n_bins=4
)
bins = sorted(labels_resid.cat.categories.tolist())
residuals = y_true_reg - y_pred_reg
means = {b: residuals[labels_resid == b].mean() for b in bins}
for b, m in means.items():
    print(f"  {b}: mean residual = {m:+.3f}")

labels_resid = pycatdap.error.residual_label( y_true_reg, y_pred_reg, method="aic_pool", n_bins=4 ) bins = sorted(labels_resid.cat.categories.tolist()) residuals = y_true_reg - y_pred_reg means = {b: residuals[labels_resid == b].mean() for b in bins} for b, m in means.items(): print(f" {b}: mean residual = {m:+.3f}")

6. Large benchmark datasets (pycatdap[data])¶

Three additional fetch loaders ship with v0.8.0 for larger benchmarks. They live under the new pycatdap[data] extras (adds scikit-learn>=1.3) and delegate caching to sklearn (~/scikit_learn_data/, override with SCIKIT_LEARN_DATA):

# pip install 'pycatdap[data]'
df_ca   = pycatdap.datasets.fetch_california_housing()   # 20,640 × 9, regression
df_adult = pycatdap.datasets.fetch_adult_income()        # ~48,842 × 15, binary classification
df_comp  = pycatdap.datasets.fetch_compas()              # ~5,278 × 14, fairness demo

All three feed directly into error_analysis(df, y_true_column, y_pred_column). Fetchers raise ImportError with an explicit pip install 'pycatdap[data]' hint when scikit-learn is missing.

We don't execute these cells here so the notebook can run offline in CI. Try them locally after installing the extras.

What's next¶

error_analysis() is the ML error analysis one-call entry point. Related capabilities:

• Error visualisation — plot_confusion, residual_plot, residual_by_category for richer visual diagnostics
• Calibration — calibration curves plus Brier / ECE scores
• Slice discovery & drift — multivariable subgroup discovery (discover_error_slices), cohort comparison, drift detection — the natural extension of the single-variable slices surfaced here

Cross-references¶

• Error labelling building blocks: Notebook 10
• target_analysis() (the inner composition): Notebook 09
• One-call EDA profile(): Notebook 08
• Error visualisation (confusion & residual): Notebook 12