ML error visualisation — confusion & residual diagnostics¶

This notebook covers the dedicated visualisation layer for the error_analysis() result. Six functions, plus two convenience methods on ErrorAnalysisResult:

Function	Purpose
pycatdap.error.plot_confusion	Confusion matrix heatmap (binary or multi-class)
pycatdap.error.plot_confusion_by_slice	Per-category small-multiples confusion grid
pycatdap.error.confusion_aic	ΔAIC of predictions vs truth (negative when informative)
pycatdap.error.residual_plot	Residual vs prediction, prediction vs truth, residual histogram
pycatdap.error.residual_by_category	Box plot of residuals stratified by a variable
pycatdap.error.residual_pool_plot	AIC-pooled |residual| histogram with boundary overlay
ErrorAnalysisResult.plot_confusion	Delegation method using stored y_true/y_pred
ErrorAnalysisResult.residual_plot	Delegation method using stored y_true/y_pred

All plot functions support backend="matplotlib" (default, returns Axes) and backend="plotly" (returns Figure). plot_confusion_by_slice is the intentional exception — both backends return a Figure because it composes multiple axes.

This notebook walks the canonical flow:

1. Binary classification — confusion heatmap, slice grid, confusion_aic
2. Multi-class — N×N confusion heatmap (the one-call error_analysis() wrapper falls back to error_label on multi-class, but the heatmap layer is multi-class capable)
3. Regression — residual diagnostics, box-by-category, AIC pooling
4. The ErrorAnalysisResult.plot_confusion / .residual_plot delegation shortcuts

1. Binary classification — German Credit¶

In [ ]:

import numpy as np

import pycatdap
from pycatdap.datasets import load_german_credit

df = load_german_credit()
rng = np.random.default_rng(42)

y_true = df["class"]
noise = np.where(df["age"] < 30, 0.4, 0.10)
noise = np.where(df["checking_status"] == "<0", noise + 0.10, noise)
flip = rng.random(len(df)) < noise
y_pred = y_true.copy()
y_pred.loc[flip] = y_true.loc[flip].map({"good": "bad", "bad": "good"})

print(f"accuracy: {(y_true == y_pred).mean():.3f}")

import numpy as np import pycatdap from pycatdap.datasets import load_german_credit df = load_german_credit() rng = np.random.default_rng(42) y_true = df["class"] noise = np.where(df["age"] < 30, 0.4, 0.10) noise = np.where(df["checking_status"] == "<0", noise + 0.10, noise) flip = rng.random(len(df)) < noise y_pred = y_true.copy() y_pred.loc[flip] = y_true.loc[flip].map({"good": "bad", "bad": "good"}) print(f"accuracy: {(y_true == y_pred).mean():.3f}")

plot_confusion — the standard heatmap¶

In [ ]:

ax = pycatdap.error.plot_confusion(
    y_true.to_numpy(),
    y_pred.to_numpy(),
    labels=["good", "bad"],
    normalize="true",
    backend="matplotlib",
)
ax.figure

ax = pycatdap.error.plot_confusion( y_true.to_numpy(), y_pred.to_numpy(), labels=["good", "bad"], normalize="true", backend="matplotlib", ) ax.figure

confusion_aic — quantify how informative the predictions are¶

Sign convention follows pycatdap's existing catdap1 / target_analysis.ranking — negative = informative. (Issue #18 phrased this as "positive when informative"; pycatdap keeps the project-wide convention instead.)

In [ ]:

delta = pycatdap.error.confusion_aic(y_true.to_numpy(), y_pred.to_numpy())
verdict = "informative" if delta < 0 else "uninformative"
print(f"confusion_aic ΔAIC = {delta:+.2f} ({verdict})")

delta = pycatdap.error.confusion_aic(y_true.to_numpy(), y_pred.to_numpy()) verdict = "informative" if delta < 0 else "uninformative" print(f"confusion_aic ΔAIC = {delta:+.2f} ({verdict})")

plot_confusion_by_slice — small-multiples per slicing variable¶

Returns a Figure for both backends because multi-panel grids do not fit on a single Axes (an explicit, intentional exception to the usual "matplotlib returns Axes" rule).

In [ ]:

fig = pycatdap.error.plot_confusion_by_slice(
    df,
    y_true.to_numpy(),
    y_pred.to_numpy(),
    var="checking_status",
    labels=["good", "bad"],
    n_cols=2,
    backend="matplotlib",
)
fig

fig = pycatdap.error.plot_confusion_by_slice( df, y_true.to_numpy(), y_pred.to_numpy(), var="checking_status", labels=["good", "bad"], n_cols=2, backend="matplotlib", ) fig

2. Multi-class — Palmer Penguins¶

Even though the one-call error_analysis() wrapper falls back to error_label on multi-class (because confusion_label is binary-only), the visualisation layer happily renders the N×N confusion heatmap. The result delegation method (r.plot_confusion()) likewise works for any classification task because the raw y_true / y_pred are stored on the result.

In [ ]:

from pycatdap.datasets import load_penguins

penguins = load_penguins().dropna(subset=["species", "body_mass_g"]).copy()
rng = np.random.default_rng(11)
y_true_mc = penguins["species"]
y_pred_mc = y_true_mc.copy()
swap_mask = (y_true_mc.isin(["Adelie", "Chinstrap"])) & (
    rng.random(len(penguins)) < 0.30
)
y_pred_mc.loc[swap_mask] = y_true_mc.loc[swap_mask].map(
    {"Adelie": "Chinstrap", "Chinstrap": "Adelie"}
)

ax = pycatdap.error.plot_confusion(
    y_true_mc.to_numpy(),
    y_pred_mc.to_numpy(),
    normalize="true",
    backend="matplotlib",
)
ax.figure

from pycatdap.datasets import load_penguins penguins = load_penguins().dropna(subset=["species", "body_mass_g"]).copy() rng = np.random.default_rng(11) y_true_mc = penguins["species"] y_pred_mc = y_true_mc.copy() swap_mask = (y_true_mc.isin(["Adelie", "Chinstrap"])) & ( rng.random(len(penguins)) < 0.30 ) y_pred_mc.loc[swap_mask] = y_true_mc.loc[swap_mask].map( {"Adelie": "Chinstrap", "Chinstrap": "Adelie"} ) ax = pycatdap.error.plot_confusion( y_true_mc.to_numpy(), y_pred_mc.to_numpy(), normalize="true", backend="matplotlib", ) ax.figure

3. Regression — engineered bias¶

In [ ]:

import pandas as pd

rng = np.random.default_rng(23)
n = 400
group = rng.choice(["lo", "hi"], size=n)
y_true_reg = np.where(group == "hi", 10.0, 0.0) + rng.normal(0, 1.0, size=n)
bias = np.where(group == "hi", -3.0, 0.0)
y_pred_reg = y_true_reg + bias + rng.normal(0, 0.5, size=n)
reg_df = pd.DataFrame({"group": group})

import pandas as pd rng = np.random.default_rng(23) n = 400 group = rng.choice(["lo", "hi"], size=n) y_true_reg = np.where(group == "hi", 10.0, 0.0) + rng.normal(0, 1.0, size=n) bias = np.where(group == "hi", -3.0, 0.0) y_pred_reg = y_true_reg + bias + rng.normal(0, 0.5, size=n) reg_df = pd.DataFrame({"group": group})

residual_plot — three diagnostic styles¶

In [ ]:

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
pycatdap.error.residual_plot(y_true_reg, y_pred_reg, ax=axes[0])
pycatdap.error.residual_plot(
    y_true_reg, y_pred_reg, kind="scatter_true_pred", ax=axes[1]
)
pycatdap.error.residual_plot(y_true_reg, y_pred_reg, kind="histogram", ax=axes[2])
fig.tight_layout()
fig

import matplotlib.pyplot as plt fig, axes = plt.subplots(1, 3, figsize=(15, 4)) pycatdap.error.residual_plot(y_true_reg, y_pred_reg, ax=axes[0]) pycatdap.error.residual_plot( y_true_reg, y_pred_reg, kind="scatter_true_pred", ax=axes[1] ) pycatdap.error.residual_plot(y_true_reg, y_pred_reg, kind="histogram", ax=axes[2]) fig.tight_layout() fig

residual_by_category — stratify by an explanatory variable¶

By construction we biased predictions on group == "hi", so the high group should sit below the zero line.

In [ ]:

ax = pycatdap.error.residual_by_category(
    reg_df, y_true_reg, y_pred_reg, "group", backend="matplotlib"
)
ax.figure

ax = pycatdap.error.residual_by_category( reg_df, y_true_reg, y_pred_reg, "group", backend="matplotlib" ) ax.figure

residual_pool_plot — AIC pooling visualisation¶

Vertical dashed lines mark the AIC-pooled bin boundaries derived from residual_label(method="aic_pool"). The final bin count after AIC merging may be smaller than the n_bins= seed.

In [ ]:

ax = pycatdap.error.residual_pool_plot(
    y_true_reg, y_pred_reg, n_bins=4, backend="matplotlib"
)
ax.figure

ax = pycatdap.error.residual_pool_plot( y_true_reg, y_pred_reg, n_bins=4, backend="matplotlib" ) ax.figure

4. The ErrorAnalysisResult delegation shortcuts¶

After error_analysis() the result carries the raw y_true / y_pred so visualisations are one method call away.

In [ ]:

r = pycatdap.error_analysis(
    df.drop(columns=["class"]), y_true.to_numpy(), y_pred.to_numpy(), top_k=4
)
ax = r.plot_confusion(backend="matplotlib", normalize="true")
ax.figure

r = pycatdap.error_analysis( df.drop(columns=["class"]), y_true.to_numpy(), y_pred.to_numpy(), top_k=4 ) ax = r.plot_confusion(backend="matplotlib", normalize="true") ax.figure

In [ ]:

r_reg = pycatdap.error_analysis(reg_df, y_true_reg, y_pred_reg, top_k=1)
ax = r_reg.residual_plot(backend="matplotlib", kind="histogram")
ax.figure

r_reg = pycatdap.error_analysis(reg_df, y_true_reg, y_pred_reg, top_k=1) ax = r_reg.residual_plot(backend="matplotlib", kind="histogram") ax.figure

Multi-class delegation¶

Even though error_analysis() reports label_kind == "error_label" (multi-class fallback), r.plot_confusion() still produces the full 3×3 heatmap because the wrapper kept y_true / y_pred on the result.

In [ ]:

r_mc = pycatdap.error_analysis(
    penguins.drop(columns=["species"]),
    y_true_mc.to_numpy(),
    y_pred_mc.to_numpy(),
    top_k=2,
)
print(f"label_kind = {r_mc.label_kind}   (multiclass fallback)")
ax = r_mc.plot_confusion(backend="matplotlib", normalize="true")
ax.figure

r_mc = pycatdap.error_analysis( penguins.drop(columns=["species"]), y_true_mc.to_numpy(), y_pred_mc.to_numpy(), top_k=2, ) print(f"label_kind = {r_mc.label_kind} (multiclass fallback)") ax = r_mc.plot_confusion(backend="matplotlib", normalize="true") ax.figure

What's next¶

This completes the error visualisation layer. Upcoming:

• Calibration — calibration curves, Brier score, expected calibration error
• Slice discovery — multi-variable subgroup discovery (discover_error_slices), cohort comparison, drift detection (the natural extension of single-variable error slices)

Cross-references¶

• error_analysis() one-call wrapper: Notebook 11
• Error labels: Notebook 10
• target_analysis(): Notebook 09