ML error labelling — error_label, confusion_label, residual_label, abs_residual_pool¶

This notebook opens the ML error analysis part of the library. It introduces 4 small label-generating functions that convert prediction outputs into categorical labels suitable for CATDAP analysis. The one-call error-analysis wrapper, error visualisation, calibration, and slice discovery all build on top of these labels.

API	Use it on	Returns
error_label(y_true, y_pred)	Any task	Categorical Series: "correct" / "incorrect"
confusion_label(y_true, y_pred, *, positive=...)	Binary classification	Categorical Series: "TP" / "FP" / "FN" / "TN"
residual_label(y_true, y_pred, *, method=...)	Regression	Categorical Series of binned residuals
abs_residual_pool(y_true, y_pred)	Regression	Categorical Series of AIC-pooled |residual| bins

All four return pd.Series — no new dataclass is introduced — so labels compose directly with the rest of the toolkit. The one-call error_analysis() wrapper layers an immutable ErrorAnalysisResult object on top of these labels.

This notebook uses the German Credit dataset to demo binary classification labels, and synthesized regression data for residual labels.

In [ ]:

import numpy as np

import pycatdap
from pycatdap.datasets import load_german_credit

df = load_german_credit()
print(df.shape, df["class"].value_counts().to_dict())

import numpy as np import pycatdap from pycatdap.datasets import load_german_credit df = load_german_credit() print(df.shape, df["class"].value_counts().to_dict())

1. error_label — "correct" / "incorrect"¶

error_label is the simplest label: it compares y_true and y_pred element-wise and returns a categorical Series with categories {"correct", "incorrect"}. Works for any task (binary, multiclass, regression discretized).

In [ ]:

# Train a quick model on German Credit to get realistic predictions.
rng = np.random.default_rng(42)
y_true = df["class"]
# Stand-in for a real classifier: 75% baseline accuracy with class-conditioned noise.
y_pred = y_true.copy()
wrong = rng.random(len(df)) < 0.25
y_pred.loc[wrong] = y_true.loc[wrong].map({"good": "bad", "bad": "good"})

labels = pycatdap.error.error_label(y_true, y_pred)
print("shape:", labels.shape)
print("counts:", labels.value_counts().to_dict())
print("dtype:", labels.dtype)

# Train a quick model on German Credit to get realistic predictions. rng = np.random.default_rng(42) y_true = df["class"] # Stand-in for a real classifier: 75% baseline accuracy with class-conditioned noise. y_pred = y_true.copy() wrong = rng.random(len(df)) < 0.25 y_pred.loc[wrong] = y_true.loc[wrong].map({"good": "bad", "bad": "good"}) labels = pycatdap.error.error_label(y_true, y_pred) print("shape:", labels.shape) print("counts:", labels.value_counts().to_dict()) print("dtype:", labels.dtype)

Combining with target_analysis¶

Once you have an error label, you can feed it back into the target-analysis and quality-report APIs to discover which explanatory variables explain the errors. This is the foundational pattern behind the one-call error_analysis() wrapper.

In [ ]:

df_with_errors = df.copy()
df_with_errors["was_correct"] = labels
# Drop columns that the (synthetic) classifier saw directly
explanatory_only = df_with_errors.drop(columns=["class"])
ta = pycatdap.target_analysis(explanatory_only, response="was_correct", top_k=3)
ta.ranking.head(5)

df_with_errors = df.copy() df_with_errors["was_correct"] = labels # Drop columns that the (synthetic) classifier saw directly explanatory_only = df_with_errors.drop(columns=["class"]) ta = pycatdap.target_analysis(explanatory_only, response="was_correct", top_k=3) ta.ranking.head(5)

2. confusion_label — TP / FP / FN / TN¶

confusion_label is binary-only; multiclass (one-vs-rest) is deferred to a later release. You pass positive= to designate the positive class, or omit it to let pycatdap pick the larger of the two unique values.

In [ ]:

conf = pycatdap.error.confusion_label(y_true, y_pred, positive="bad")
print(conf.value_counts().to_dict())
print("categories:", list(conf.cat.categories))

conf = pycatdap.error.confusion_label(y_true, y_pred, positive="bad") print(conf.value_counts().to_dict()) print("categories:", list(conf.cat.categories))

Multiclass: explicit NotImplementedError¶

Passing 3+ unique values raises NotImplementedError with a message pointing at the planned follow-up. The library ships a focused binary implementation now and designs multiclass deliberately later.

In [ ]:

from pycatdap.datasets import load_penguins

penguins = load_penguins().dropna(subset=["species"])
try:
    pycatdap.error.confusion_label(penguins["species"], penguins["species"])
except NotImplementedError as exc:
    print("Raised as expected:")
    print(exc)

from pycatdap.datasets import load_penguins penguins = load_penguins().dropna(subset=["species"]) try: pycatdap.error.confusion_label(penguins["species"], penguins["species"]) except NotImplementedError as exc: print("Raised as expected:") print(exc)

3. residual_label — bin regression residuals¶

For regression, residuals are continuous. residual_label bins them into categorical labels via one of three methods:

method	Backend	When to use
"aic_pool" (default)	CATDAP-01 AIC pooling (pycatdap._pooling)	Best balance — bin count chosen by AIC
"quantile"	pd.qcut	Equal-frequency bins
"equal_width"	pd.cut	Equal-width bins (sensitive to outliers)

In [ ]:

rng = np.random.default_rng(0)
y_true_reg = rng.normal(loc=10, scale=2, size=300)
y_pred_reg = y_true_reg + rng.normal(loc=0, scale=1, size=300)

for method in ("aic_pool", "quantile", "equal_width"):
    labels_r = pycatdap.error.residual_label(
        y_true_reg, y_pred_reg, method=method, n_bins=4
    )
    n_cats = len(labels_r.cat.categories)
    counts = labels_r.value_counts().to_dict()
    print(f"{method:13s} n_categories={n_cats:2d}")
    print(f"              counts={counts}")

rng = np.random.default_rng(0) y_true_reg = rng.normal(loc=10, scale=2, size=300) y_pred_reg = y_true_reg + rng.normal(loc=0, scale=1, size=300) for method in ("aic_pool", "quantile", "equal_width"): labels_r = pycatdap.error.residual_label( y_true_reg, y_pred_reg, method=method, n_bins=4 ) n_cats = len(labels_r.cat.categories) counts = labels_r.value_counts().to_dict() print(f"{method:13s} n_categories={n_cats:2d}") print(f" counts={counts}")

4. abs_residual_pool — AIC-pooled |residual| bins¶

abs_residual_pool is a convenience wrapper that takes the absolute residual |y_true - y_pred| and bins it via AIC pooling. Useful when you only care about magnitude of error, not sign.

In [ ]:

abs_bins = pycatdap.error.abs_residual_pool(y_true_reg, y_pred_reg, n_bins=4)
print("categories:", list(abs_bins.cat.categories))
print("counts:", abs_bins.value_counts().to_dict())

abs_bins = pycatdap.error.abs_residual_pool(y_true_reg, y_pred_reg, n_bins=4) print("categories:", list(abs_bins.cat.categories)) print("counts:", abs_bins.value_counts().to_dict())

5. _detect_task — heuristic task detection¶

The one-call error_analysis() wrapper accepts task="auto" and uses _detect_task under the hood. The heuristic:

• Strings / objects → classification
• Both integer dtype AND ≤ 20 unique values → classification
• y_pred in [0, 1] with binary y_true → classification (probabilities)
• Otherwise → regression

In [ ]:

from pycatdap.error import _detect_task

examples = [
    ("binary int", np.array([0, 1, 0, 1]), np.array([0, 1, 1, 1])),
    ("binary string", np.array(["cat", "dog"]), np.array(["cat", "cat"])),
    ("probability", np.array([0, 1, 0, 1]), np.array([0.1, 0.9, 0.3, 0.8])),
    ("continuous", y_true_reg[:5], y_pred_reg[:5]),
]
for name, yt, yp in examples:
    print(f"{name:13s}: {_detect_task(yt, yp)}")

from pycatdap.error import _detect_task examples = [ ("binary int", np.array([0, 1, 0, 1]), np.array([0, 1, 1, 1])), ("binary string", np.array(["cat", "dog"]), np.array(["cat", "cat"])), ("probability", np.array([0, 1, 0, 1]), np.array([0.1, 0.9, 0.3, 0.8])), ("continuous", y_true_reg[:5], y_pred_reg[:5]), ] for name, yt, yp in examples: print(f"{name:13s}: {_detect_task(yt, yp)}")

What's next¶

These labels are the building blocks. Coming up:

• One-call error analysis — error_analysis() wrapper. Combines _detect_task + error_label / confusion_label / residual_label + target_analysis into a single immutable result object.
• Error visualisation — plot_confusion, residual_plot, residual_by_category.
• Calibration — AIC binning calibration.
• Slice discovery — slice discovery, cohort comparison, drift detection.

Cross-references¶

• Target analysis and quality suite (profile, target_analysis, suite): Notebook 09
• One-call EDA: Notebook 08
• Bivariate association APIs: Notebook 07

Next: Notebook 11 — ML error analysis (one-call)