AIC-optimal binning on iris with pycatdap¶

This tutorial shows why the CATDAP-02 pooling search finds a more informative discretization of continuous variables than equal-width or quantile binning. We use Fisher's classic iris dataset (150 rows, 4 continuous features, 1 categorical response).

By the end of this notebook you'll know how to:

1. Run a baseline equal-width binning sweep and capture its ΔAIC scores
2. Call pycatdap.catdap2(..., pool=[0,...]) to let CATDAP-02 choose the bin boundaries
3. Compare the AIC-optimal cuts against equal-width cuts
4. Read intervals, aic, and subsets from a Catdap2Result

Background¶

CATDAP-02 treats binning as part of the model selection problem: every choice of bin count and boundary changes the contingency table and therefore the AIC. A pool=0 entry tells CATDAP-02 to search for the boundaries that minimize AIC, subject to a minimum bin width given by accuracy.

In [ ]:

import pandas as pd

import pycatdap

print(f"pycatdap version: {pycatdap.__version__}")
df = pycatdap.datasets.load_iris()
df.shape, list(df.columns)

import pandas as pd import pycatdap print(f"pycatdap version: {pycatdap.__version__}") df = pycatdap.datasets.load_iris() df.shape, list(df.columns)

1. Summarize the dataset¶

describe() confirms kind detection: 4 continuous + 1 categorical. The response variable (Species) is balanced (50 of each class).

In [ ]:

summary = pycatdap.describe(df)
summary.summary

summary = pycatdap.describe(df) summary.summary

In [ ]:

df["Species"].value_counts()

df["Species"].value_counts()

2. Look at one feature¶

plot_variable() infers the right plot kind from dtype. For continuous columns it produces a histogram.

In [ ]:

import matplotlib.pyplot as plt

cols = ["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"]
fig, axes = plt.subplots(1, 4, figsize=(16, 3))
for ax, col in zip(axes, cols, strict=True):
    pycatdap.plot_variable(df, col, ax=ax)
fig.tight_layout()
plt.show()

import matplotlib.pyplot as plt cols = ["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"] fig, axes = plt.subplots(1, 4, figsize=(16, 3)) for ax, col in zip(axes, cols, strict=True): pycatdap.plot_variable(df, col, ax=ax) fig.tight_layout() plt.show()

The two Petal features show a clear bimodal pattern that lines up with the species grouping. Sepal features overlap more.

3. Baseline: equal-width binning of Sepal.Length¶

Suppose we just bin Sepal.Length into equal-width buckets and run CATDAP-02 with both columns marked categorical (pool=2). The result is the ΔAIC of that fixed binning.

In [ ]:

def equal_width_aic(df: pd.DataFrame, col: str, response: str, n_bins: int) -> float:
    binned = pd.cut(df[col], bins=n_bins, include_lowest=True)
    work = pd.DataFrame({col: binned.astype(str), response: df[response]})
    result = pycatdap.catdap2(
        work, pool=[2, 2], response_name=response, accuracy=[0.0, 0.0]
    )
    return float(result.aic.loc[result.aic["variable"] == col, "aic"].iloc[0])


baseline = pd.DataFrame(
    {
        "n_bins": [2, 3, 4, 5, 6, 8, 10],
        "delta_aic": [
            equal_width_aic(df, "Sepal.Length", "Species", k)
            for k in [2, 3, 4, 5, 6, 8, 10]
        ],
    }
)
baseline

def equal_width_aic(df: pd.DataFrame, col: str, response: str, n_bins: int) -> float: binned = pd.cut(df[col], bins=n_bins, include_lowest=True) work = pd.DataFrame({col: binned.astype(str), response: df[response]}) result = pycatdap.catdap2( work, pool=[2, 2], response_name=response, accuracy=[0.0, 0.0] ) return float(result.aic.loc[result.aic["variable"] == col, "aic"].iloc[0]) baseline = pd.DataFrame( { "n_bins": [2, 3, 4, 5, 6, 8, 10], "delta_aic": [ equal_width_aic(df, "Sepal.Length", "Species", k) for k in [2, 3, 4, 5, 6, 8, 10] ], } ) baseline

More bins reduce ΔAIC up to a point, then start to penalize. Equal-width cuts can't react to where the boundaries between species actually fall.

4. AIC-optimal binning with catdap2¶

pool=0 is equal pooling — CATDAP-02 searches for the bin count and boundaries that minimize AIC. accuracy=0.1 keeps bins at least 0.1 cm wide, which is the measurement precision of the dataset.

In [ ]:

result = pycatdap.catdap2(
    df,
    pool=[0, 0, 0, 0, 2],
    response_name="Species",
    accuracy=[0.1, 0.1, 0.1, 0.1, 0.0],
)
result

result = pycatdap.catdap2( df, pool=[0, 0, 0, 0, 2], response_name="Species", accuracy=[0.1, 0.1, 0.1, 0.1, 0.0], ) result

In [ ]:

result.aic

result.aic

The AIC-optimal binning of Sepal.Length scores much lower than any equal-width baseline — the search picked boundaries that actually separate species.

5. Compare cut points¶

result.intervals gives the chosen interior boundaries for each pooled variable.

In [ ]:

{name: [round(x, 3) for x in cuts] for name, cuts in result.intervals.items()}

{name: [round(x, 3) for x in cuts] for name, cuts in result.intervals.items()}

In [ ]:

def overlay_cuts(col: str) -> None:
    fig, ax = plt.subplots(figsize=(8, 3))
    for species, sub in df.groupby("Species"):
        ax.hist(sub[col], bins=30, alpha=0.5, label=species)
    for x in result.intervals[col]:
        ax.axvline(x, color="black", linestyle="--", linewidth=1)
    ax.set_title(f"{col} — AIC-optimal cuts overlaid on species histograms")
    ax.set_xlabel(col)
    ax.legend()
    fig.tight_layout()
    plt.show()


overlay_cuts("Sepal.Length")
overlay_cuts("Petal.Length")

def overlay_cuts(col: str) -> None: fig, ax = plt.subplots(figsize=(8, 3)) for species, sub in df.groupby("Species"): ax.hist(sub[col], bins=30, alpha=0.5, label=species) for x in result.intervals[col]: ax.axvline(x, color="black", linestyle="--", linewidth=1) ax.set_title(f"{col} — AIC-optimal cuts overlaid on species histograms") ax.set_xlabel(col) ax.legend() fig.tight_layout() plt.show() overlay_cuts("Sepal.Length") overlay_cuts("Petal.Length")

The dashed lines mark the boundaries CATDAP-02 chose. For Petal.Length the cuts land squarely between species clusters — exactly where a thoughtful analyst would split.

6. Rank features by ΔAIC¶

aic_comparison_plot() is the canonical view of single-variable informativeness. Petal features dominate.

In [ ]:

ax = pycatdap.plot.aic_comparison_plot(result)
plt.show()

ax = pycatdap.plot.aic_comparison_plot(result) plt.show()

Plotly backend¶

The same call with backend="plotly" returns an interactive plotly.graph_objs.Figure — useful when embedding in LizyStudio or any web dashboard.

In [ ]:

fig = pycatdap.plot.aic_comparison_plot(result, backend="plotly")
fig.show()

fig = pycatdap.plot.aic_comparison_plot(result, backend="plotly") fig.show()

7. Best subsets¶

result.subsets is sorted by AIC. The top entries reveal which combinations of variables explain Species best.

In [ ]:

for s in result.subsets[:5]:
    print(f"AIC={s.aic:>9.2f}  n_vars={s.n_vars}  vars={s.variables}")

for s in result.subsets[:5]: print(f"AIC={s.aic:>9.2f} n_vars={s.n_vars} vars={s.variables}")

The single best predictor is Petal.Length. Adding Petal.Width marginally improves AIC, but adding sepal features barely moves the needle — their information is already captured by the petal axes.

Summary¶

Step	Function	What it showed
Summary	pycatdap.describe(df)	4 continuous + 1 categorical, no missing
Baseline	pd.cut + catdap2 with pool=[2,2]	Equal-width ΔAIC for n=2..10 bins
Optimal	catdap2(pool=[0,0,0,0,2])	AIC-optimal cuts via search
Cuts	result.intervals	Where the search placed boundaries
Ranking	aic_comparison_plot(result)	Petal features dominate
Subsets	result.subsets[:5]	Best variable combinations

Next steps¶

• See 02. EDA on Titanic for the all-categorical CATDAP-01 pipeline.
• See 05. Real-world EDA for an end-to-end workflow on messy data with missing values.