ML calibration — reliability diagrams with AIC-optimal binning¶

Calibration answers the question that confusion and residual plots cannot: when the model says 70%, does it actually happen 70% of the time? This is probability calibration, and pycatdap brings its signature AIC-optimal binning to the reliability diagram.

Function	Purpose
pycatdap.error.calibration_curve	Reliability diagram (Wilson CIs); Axes / Figure
pycatdap.error.calibration_table	Per-bin numbers behind the diagram
pycatdap.error.brier_score	Mean squared error of probabilities
pycatdap.error.expected_calibration_error	Bin-weighted |observed − predicted| (ECE)
pycatdap.error.maximum_calibration_error	Worst-bin gap (MCE)
ErrorAnalysisResult.calibration_curve	Delegation method using stored y_true / y_proba

Three binning strategies via strategy=: aic (default — bins land where the observed positive-rate shifts), equal_width, quantile. Scope is binary classification; regression and multi-class calibration are planned follow-ups.

1. Calibrated vs over-confident probabilities¶

We synthesise two models over the same binary outcomes: one whose stated probability equals the empirical positive rate (well-calibrated), and one that sharpens the log-odds toward 0/1 (over-confident).

In [ ]:

import numpy as np

import pycatdap

rng = np.random.default_rng(0)
n = 3000

# Well-calibrated: stated probability == empirical positive rate.
proba_good = rng.uniform(0.02, 0.98, size=n)
y_true = (rng.random(n) < proba_good).astype(int)

# Over-confident on the SAME outcomes: sharpen the log-odds toward 0/1.
logit = np.log(proba_good / (1.0 - proba_good))
proba_over = 1.0 / (1.0 + np.exp(-1.8 * logit))

print(f"positive rate: {y_true.mean():.3f}")

import numpy as np import pycatdap rng = np.random.default_rng(0) n = 3000 # Well-calibrated: stated probability == empirical positive rate. proba_good = rng.uniform(0.02, 0.98, size=n) y_true = (rng.random(n) < proba_good).astype(int) # Over-confident on the SAME outcomes: sharpen the log-odds toward 0/1. logit = np.log(proba_good / (1.0 - proba_good)) proba_over = 1.0 / (1.0 + np.exp(-1.8 * logit)) print(f"positive rate: {y_true.mean():.3f}")

calibration_curve — the reliability diagram¶

Points on the y = x diagonal are perfectly calibrated. Error bars are Wilson 95% binomial intervals on the observed frequency.

In [ ]:

ax = pycatdap.error.calibration_curve(
    y_true, proba_good, strategy="aic", backend="matplotlib"
)
ax.set_title("Well-calibrated model (AIC bins)")
ax.figure

ax = pycatdap.error.calibration_curve( y_true, proba_good, strategy="aic", backend="matplotlib" ) ax.set_title("Well-calibrated model (AIC bins)") ax.figure

In [ ]:

ax = pycatdap.error.calibration_curve(
    y_true, proba_over, strategy="aic", backend="matplotlib"
)
ax.set_title("Over-confident model (AIC bins)")
ax.figure

ax = pycatdap.error.calibration_curve( y_true, proba_over, strategy="aic", backend="matplotlib" ) ax.set_title("Over-confident model (AIC bins)") ax.figure

Metrics — Brier, ECE, MCE¶

All lower-is-better. The over-confident model should score worse on every metric.

In [ ]:

for name, proba in [("well-calibrated", proba_good), ("over-confident", proba_over)]:
    brier = pycatdap.error.brier_score(y_true, proba)
    ece = pycatdap.error.expected_calibration_error(
        y_true, proba, strategy="equal_width", n_bins=10
    )
    mce = pycatdap.error.maximum_calibration_error(
        y_true, proba, strategy="equal_width", n_bins=10
    )
    print(f"{name:16s}  Brier={brier:.4f}  ECE={ece:.4f}  MCE={mce:.4f}")

for name, proba in [("well-calibrated", proba_good), ("over-confident", proba_over)]: brier = pycatdap.error.brier_score(y_true, proba) ece = pycatdap.error.expected_calibration_error( y_true, proba, strategy="equal_width", n_bins=10 ) mce = pycatdap.error.maximum_calibration_error( y_true, proba, strategy="equal_width", n_bins=10 ) print(f"{name:16s} Brier={brier:.4f} ECE={ece:.4f} MCE={mce:.4f}")

calibration_table — the numbers behind the diagram¶

The single source of truth for the curve and the metrics. One row per occupied bin, with the Wilson CI bounds.

In [ ]:

pycatdap.error.calibration_table(y_true, proba_over, strategy="aic")

pycatdap.error.calibration_table(y_true, proba_over, strategy="aic")

2. Binning strategies — aic vs equal_width vs quantile¶

ECE depends on the binning. AIC binning adapts the number and placement of bins to the data instead of imposing a fixed grid.

In [ ]:

import pandas as pd

rows = []
for strat in ["aic", "equal_width", "quantile"]:
    table = pycatdap.error.calibration_table(
        y_true, proba_over, strategy=strat, n_bins=10
    )
    ece = pycatdap.error.expected_calibration_error(
        y_true, proba_over, strategy=strat, n_bins=10
    )
    rows.append({"strategy": strat, "n_bins_used": len(table), "ECE": round(ece, 4)})

pd.DataFrame(rows)

import pandas as pd rows = [] for strat in ["aic", "equal_width", "quantile"]: table = pycatdap.error.calibration_table( y_true, proba_over, strategy=strat, n_bins=10 ) ece = pycatdap.error.expected_calibration_error( y_true, proba_over, strategy=strat, n_bins=10 ) rows.append({"strategy": strat, "n_bins_used": len(table), "ECE": round(ece, 4)}) pd.DataFrame(rows)

AIC binning is especially valuable for skewed probability predictions: where a model concentrates most of its outputs near 0 or 1, equal-width bins leave the interior nearly empty and waste resolution, while AIC bins merge uninformative regions and split where the positive-rate actually changes. The initial grid is bounded (a fixed probability-resolution) so continuous probabilities never blow up the bin count.

3. Plotly backend¶

In [ ]:

fig = pycatdap.error.calibration_curve(
    y_true, proba_over, strategy="aic", backend="plotly"
)
fig

fig = pycatdap.error.calibration_curve( y_true, proba_over, strategy="aic", backend="plotly" ) fig

4. The ErrorAnalysisResult.calibration_curve shortcut¶

Pass y_proba= to error_analysis() and the reliability diagram is one method call away — alongside the plot_confusion / residual_plot error-visualization shortcuts. Calibration requires 0/1-encoded y_true.

In [ ]:

df = pd.DataFrame(
    {
        "segment": rng.choice(["A", "B", "C"], size=n),
        "tenure_bucket": rng.choice(["new", "mid", "long"], size=n),
    }
)
y_pred = (proba_over >= 0.5).astype(int)

r = pycatdap.error_analysis(
    df, y_true, y_pred, y_proba=proba_over, task="classification"
)
ax = r.calibration_curve(backend="matplotlib", strategy="aic")
ax.set_title("Calibration via ErrorAnalysisResult delegation")
ax.figure

df = pd.DataFrame( { "segment": rng.choice(["A", "B", "C"], size=n), "tenure_bucket": rng.choice(["new", "mid", "long"], size=n), } ) y_pred = (proba_over >= 0.5).astype(int) r = pycatdap.error_analysis( df, y_true, y_pred, y_proba=proba_over, task="classification" ) ax = r.calibration_curve(backend="matplotlib", strategy="aic") ax.set_title("Calibration via ErrorAnalysisResult delegation") ax.figure

What's next¶

This completes the binary-classification calibration toolkit. Upcoming:

• Slice discovery & drift — multi-variable subgroup discovery (discover_error_slices), cohort comparison, drift detection.
• Calibration follow-ups — regression calibration (predicted-vs-actual quantiles) and multi-class (one-vs-rest).

Cross-references¶

• Error visualization (confusion & residual diagrams): Notebook 12
• error_analysis() one-call wrapper: Notebook 11
• Slice discovery & drift: Notebook 14