Target-pair analysis with target_summary and plot_target¶

This tutorial introduces the target-driven EDA API added in v0.3+ for asking one specific question: given a target variable, which explanatory variables explain it best, and how?

By the end of this notebook you'll know how to:

1. Cross-tabulate a (target, explanatory) pair with pycatdap.target_summary()
2. Read the four perspectives it returns — counts, row / column proportions, Pearson standardized residuals, and ΔAIC
3. Visualize the same pair with pycatdap.plot_target() and let the dispatch table pick the right plot kind
4. Use AIC-optimal binning for continuous explanatory variables — and see why equal-width binning often misses the signal
5. Rank all candidate explanatories by ΔAIC to find the strongest predictors
6. Export the result to standalone HTML or Plotly JSON for a web frontend

The dataset is the messy 891-passenger Titanic from seaborn (the same one used in notebook 05). It has both categorical and continuous columns plus real missing values — enough to exercise the full dispatch table.

!!! note "Companion notebooks" - 02 — Clean Titanic EDA introduces describe / plot_variable / plot_missing on the bundled contingency-table Titanic - 05 — Real-world EDA on seaborn Titanic walks through messy-data cleanup before applying CATDAP-01 / CATDAP-02

In [ ]:

import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

import pycatdap

print(f"pycatdap version: {pycatdap.__version__}")
df = sns.load_dataset("titanic")
# Drop columns derived from `survived` to avoid leakage in the ranking section
df = df.drop(columns=["alive", "adult_male", "who"])
print(f"shape: {df.shape}")
df.head()

import matplotlib.pyplot as plt import pandas as pd import seaborn as sns import pycatdap print(f"pycatdap version: {pycatdap.__version__}") df = sns.load_dataset("titanic") # Drop columns derived from `survived` to avoid leakage in the ranking section df = df.drop(columns=["alive", "adult_male", "who"]) print(f"shape: {df.shape}") df.head()

Why target-pair analysis?¶

The univariate EDA API (describe, plot_variable, plot_missing from notebook 02) tells you what each column looks like in isolation. CATDAP-01 / CATDAP-02 tell you which variables matter as a ranked set.

Target-pair analysis sits between those two views: you pick one explanatory and inspect its relationship with the target from multiple angles in a single call. It's the natural workflow for:

• Sanity-checking a feature before adding it to a model
• Investigating a single suspicious feature flagged by a CATDAP-01 ranking
• Explaining a model decision in narrative form ("survival depended on sex because — see this row-proportion table")

The shape of the API is intentionally similar to seaborn's hue=target pattern and sweetviz's target_feat=, but with AIC-based statistics instead of correlation or chi-squared.

1. Your first target pair: Survived × Sex¶

The minimal call returns a TargetSummary object containing four tables and a ΔAIC value.

In [ ]:

pair = pycatdap.target_summary(df, target="survived", explanatory="sex")
pair

pair = pycatdap.target_summary(df, target="survived", explanatory="sex") pair

In [ ]:

pair.show()

pair.show()

Reading the four tables:

• counts — raw cross-frequencies. Rows are target categories, columns are explanatory categories. Useful for spotting small cells that may need pooling.
• row_prop — each row sums to 1. Reads as: "within each target category, how were the passengers distributed across the explanatory?" (i.e., of survivors, what fraction were female vs male).
• col_prop — each column sums to 1. Reads as: "within each explanatory category, what's the target rate?" (i.e., of females, what fraction survived). This is usually the most actionable view.
• pearson_residuals — (observed − expected) / sqrt(expected). Cells with |residual| > 2 indicate a strong departure from independence. Positive residuals mean over-represented, negative means under-represented.

ΔAIC = AIC(survived modeled by sex) − AIC(survived independent of sex). A more negative ΔAIC means sex is more informative for survived. Here ΔAIC ≈ −266, which is large in absolute terms (anything beyond −10 is typically interpretable signal).

2. Visualize the pair¶

plot_target auto-selects the right plot kind by inspecting the dtypes of target and explanatory:

target dtype	explanatory dtype	auto kind
categorical / boolean	categorical (≤8 levels)	stacked bar
categorical / boolean	categorical (>8 levels)	mosaic
categorical	continuous	violin (matplotlib) / box (plotly)
boolean	continuous	overlaid histograms

For survived × sex (both categorical, 2 levels each), it picks stacked bar.

In [ ]:

fig, ax = plt.subplots(figsize=(5, 4))
pycatdap.plot_target(df, target="survived", explanatory="sex", ax=ax)
plt.show()

fig, ax = plt.subplots(figsize=(5, 4)) pycatdap.plot_target(df, target="survived", explanatory="sex", ax=ax) plt.show()

3. Multi-level categorical: Survived × Class¶

The dispatch table handles >2 levels transparently. With 3 classes, the stacked bar still works; for 8+ levels it would fall back to mosaic for space efficiency.

In [ ]:

pair_class = pycatdap.target_summary(df, target="survived", explanatory="class")
print(f"delta_aic = {pair_class.delta_aic:.3f}")
pair_class.col_prop.round(3)

pair_class = pycatdap.target_summary(df, target="survived", explanatory="class") print(f"delta_aic = {pair_class.delta_aic:.3f}") pair_class.col_prop.round(3)

In [ ]:

fig, ax = plt.subplots(figsize=(6, 4))
pycatdap.plot_target(df, target="survived", explanatory="class", ax=ax)
plt.show()

fig, ax = plt.subplots(figsize=(6, 4)) pycatdap.plot_target(df, target="survived", explanatory="class", ax=ax) plt.show()

The column-proportion table shows 1st-class survival ≈ 62%, 2nd ≈ 47%, 3rd ≈ 24% — a clear monotone effect. ΔAIC ≈ −91 — substantial signal, though weaker than sex (−266).

4. Continuous explanatory: Survived × Age (AIC-optimal binning)¶

For continuous explanatories, target_summary discretizes the column with AIC-optimal pooling — the same algorithm used by CATDAP-02 (see notebook 03 for a deep dive). The cut points minimize AIC of the resulting contingency table.

age has missing values, so drop them first.

In [ ]:

clean = df.dropna(subset=["age"]).copy()
pair_age = pycatdap.target_summary(clean, target="survived", explanatory="age")
print(f"delta_aic       = {pair_age.delta_aic:.3f}")
print(f"n_bins          = {len(pair_age.intervals) + 1}")
print(f"first 5 cuts    = {[round(b, 2) for b in pair_age.intervals[:5]]}")
print(f"last 5 cuts     = {[round(b, 2) for b in pair_age.intervals[-5:]]}")

clean = df.dropna(subset=["age"]).copy() pair_age = pycatdap.target_summary(clean, target="survived", explanatory="age") print(f"delta_aic = {pair_age.delta_aic:.3f}") print(f"n_bins = {len(pair_age.intervals) + 1}") print(f"first 5 cuts = {[round(b, 2) for b in pair_age.intervals[:5]]}") print(f"last 5 cuts = {[round(b, 2) for b in pair_age.intervals[-5:]]}")

The AIC-optimal binning of age produces many cuts because at N≈714 and binary target, the penalty per added bin (= 2) is small relative to the log-likelihood gains from capturing fine-grained survival patterns (children, young adults, elderly).

For the visualization, plot_target falls back to a violin plot comparing the raw age distribution per survival outcome — the binning result is used for delta_aic computation but not for the plot itself.

In [ ]:

fig, ax = plt.subplots(figsize=(6, 4))
pycatdap.plot_target(clean, target="survived", explanatory="age", ax=ax)
plt.show()

fig, ax = plt.subplots(figsize=(6, 4)) pycatdap.plot_target(clean, target="survived", explanatory="age", ax=ax) plt.show()

5. Why AIC binning matters — vs equal-width¶

A common reflex is to bin age into 4 equal-width buckets ("0–20, 20–40, 40–60, 60–80"). That choice is uninformative under AIC: it smooths over the very signal you're trying to detect.

In [ ]:

r_auto = pycatdap.target_summary(clean, target="survived", explanatory="age")
r_eq4 = pycatdap.target_summary(clean, target="survived", explanatory="age", bins=4)
r_dom = pycatdap.target_summary(
    clean, target="survived", explanatory="age", bins=[12, 18, 35, 60]
)

comparison = pd.DataFrame(
    {
        "n_bins": [
            len(r_auto.intervals) + 1,
            len(r_eq4.intervals) + 1,
            len(r_dom.intervals) + 1,
        ],
        "delta_aic": [r_auto.delta_aic, r_eq4.delta_aic, r_dom.delta_aic],
    },
    index=[
        "AIC-optimal (bins=None)",
        "Equal-width (bins=4)",
        "Domain bins [12,18,35,60]",
    ],
)
comparison.round(3)

r_auto = pycatdap.target_summary(clean, target="survived", explanatory="age") r_eq4 = pycatdap.target_summary(clean, target="survived", explanatory="age", bins=4) r_dom = pycatdap.target_summary( clean, target="survived", explanatory="age", bins=[12, 18, 35, 60] ) comparison = pd.DataFrame( { "n_bins": [ len(r_auto.intervals) + 1, len(r_eq4.intervals) + 1, len(r_dom.intervals) + 1, ], "delta_aic": [r_auto.delta_aic, r_eq4.delta_aic, r_dom.delta_aic], }, index=[ "AIC-optimal (bins=None)", "Equal-width (bins=4)", "Domain bins [12,18,35,60]", ], ) comparison.round(3)

Reading the comparison:

• AIC-optimal: ΔAIC ≈ −42. Strong signal — age is informative.
• Equal-width 4 bins: ΔAIC > 0. Equal-width says "age looks uninformative" — but this is just an artefact of the coarse partition averaging children's high survival rate against young-adult low survival rate.
• Domain bins [12, 18, 35, 60]: ΔAIC ≈ −5. The intuitive "infant / child / adult / elderly" split captures some signal but still misses the fine structure around early childhood.

The lesson: a "feature doesn't matter" result from an EDA tool is only as good as its binning choice. pycatdap's AIC-optimal binning is what lets age resurface as a useful predictor.

6. Rank all explanatories by ΔAIC¶

For target-driven EDA, the natural next step is to score every other column against the target and read off the ranking.

In [ ]:

def rank_explanatories(data: pd.DataFrame, target: str) -> pd.DataFrame:
    """Compute target_summary for each non-target column, return sorted by ΔAIC."""
    rows = []
    for col in data.columns:
        if col == target:
            continue
        sub = data.dropna(subset=[col])
        try:
            r = pycatdap.target_summary(sub, target=target, explanatory=col)
        except (ValueError, KeyError):
            continue
        rows.append({"explanatory": col, "n_obs": len(sub), "delta_aic": r.delta_aic})
    return pd.DataFrame(rows).sort_values("delta_aic").reset_index(drop=True)


ranking = rank_explanatories(df, "survived")
ranking

def rank_explanatories(data: pd.DataFrame, target: str) -> pd.DataFrame: """Compute target_summary for each non-target column, return sorted by ΔAIC.""" rows = [] for col in data.columns: if col == target: continue sub = data.dropna(subset=[col]) try: r = pycatdap.target_summary(sub, target=target, explanatory=col) except (ValueError, KeyError): continue rows.append({"explanatory": col, "n_obs": len(sub), "delta_aic": r.delta_aic}) return pd.DataFrame(rows).sort_values("delta_aic").reset_index(drop=True) ranking = rank_explanatories(df, "survived") ranking

In [ ]:

fig, ax = plt.subplots(figsize=(8, 4))
ax.barh(ranking["explanatory"], ranking["delta_aic"])
ax.set_xlabel("ΔAIC (more negative = more informative)")
ax.set_title("Explanatories ranked by ΔAIC against `survived`")
ax.invert_yaxis()
ax.axvline(0, color="black", linewidth=0.5)
plt.show()

fig, ax = plt.subplots(figsize=(8, 4)) ax.barh(ranking["explanatory"], ranking["delta_aic"]) ax.set_xlabel("ΔAIC (more negative = more informative)") ax.set_title("Explanatories ranked by ΔAIC against `survived`") ax.invert_yaxis() ax.axvline(0, color="black", linewidth=0.5) plt.show()

fare (continuous, auto-binned) and sex dominate. embark_town and alone are weakest. Note that this ranking is computed independently per pair — it does not account for redundancy (class ≈ pclass ≈ fare); for that, see CATDAP-02 in notebook 05 section 6.

7. Export for sharing¶

Every TargetSummary exposes .to_html() for a standalone report and .to_plotly_json() for a web frontend.

In [ ]:

html = pair.to_html()
print(f"HTML length: {len(html):,} bytes")
# To save:
# pair.to_html("survived_sex.html")

html = pair.to_html() print(f"HTML length: {len(html):,} bytes") # To save: # pair.to_html("survived_sex.html")

In [ ]:

spec = pair.to_plotly_json()
print("keys:", list(spec.keys()))
print("trace type:", spec["data"][0]["type"])

spec = pair.to_plotly_json() print("keys:", list(spec.keys())) print("trace type:", spec["data"][0]["type"])

8. Plotly backend for interactive plots¶

The same plot_target call accepts backend="plotly" to produce an interactive figure suitable for Jupyter, Streamlit, or a web frontend.

In [ ]:

fig = pycatdap.plot_target(df, target="survived", explanatory="class", backend="plotly")
fig.show()

fig = pycatdap.plot_target(df, target="survived", explanatory="class", backend="plotly") fig.show()

9. Continuous targets — Gaussian regression AIC¶

target_summary also natively supports continuous targets by switching from a multinomial contingency-table AIC to the Gaussian regression AIC of a piecewise-constant model:

AIC      = n · log(RSS / n) + penalty(k_bins + 1, n, criterion)
AIC_null = n · log(TSS / n) + penalty(2, n, criterion)
ΔAIC     = AIC − AIC_null = n · log(1 − R²) + Δpenalty

The target stays continuous; only the explanatory is binned. The return type is RegressionTargetSummary (sibling of TargetSummary).

To see it in action, treat fare (continuous) as the target and ask which features predict it.

In [ ]:

pair_fare_class = pycatdap.target_summary(clean, target="fare", explanatory="class")
print(repr(pair_fare_class))
pair_fare_class.bin_stats.round(2)

pair_fare_class = pycatdap.target_summary(clean, target="fare", explanatory="class") print(repr(pair_fare_class)) pair_fare_class.bin_stats.round(2)

Reading the regression summary:

• delta_aic — analogous to the categorical case (more negative = more informative).
• r_squared — the implicit 1 − RSS / TSS. Cross-readable as "X explains 36% of the variance in fare."
• bin_stats — count, target_mean, target_std per X bin. Compare to the contingency-table counts / col_prop of the categorical path: the regression view is more compact because the target isn't binned.
• n_effective — rows used. Rows missing in Y are dropped; rows missing in X get a _missing_ pseudo-bin so all (Y, X_i) pairs share the same null model.
• criterion — the AIC penalty family. Default is "bic" per Yao 1988's recommendation for piecewise-constant changepoint structures.

In [ ]:

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Categorical X: box plot of target by X bin (auto kind="box")
pycatdap.plot_target(clean, target="fare", explanatory="class", ax=axes[0])

# Continuous X: scatter + bin-mean overlay (auto kind="scatter")
pycatdap.plot_target(clean, target="fare", explanatory="age", ax=axes[1])

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(12, 4)) # Categorical X: box plot of target by X bin (auto kind="box") pycatdap.plot_target(clean, target="fare", explanatory="class", ax=axes[0]) # Continuous X: scatter + bin-mean overlay (auto kind="scatter") pycatdap.plot_target(clean, target="fare", explanatory="age", ax=axes[1]) plt.tight_layout() plt.show()

9.1 Comparing criterion penalties¶

criterion controls the model-complexity penalty. The three choices have different trade-offs:

criterion	penalty(k, n)	when to use
"bic" (default)	log(n) · k	conservative; recommended by Yao 1988 for changepoint structures
"aic"	2 · k	classical AIC; tends to over-select bins in piecewise-constant settings
"aicc"	2k + 2k(k+1)/(n-k-1)	small-sample correction (Hurvich & Tsai 1989 style)

R² is the same across criteria; only delta_aic shifts.

In [ ]:

criteria = ["aic", "aicc", "bic"]
deltas = [
    pycatdap.target_summary(
        clean, target="fare", explanatory="class", criterion=c
    ).delta_aic
    for c in criteria
]
pd.DataFrame({"delta_aic": deltas}, index=criteria).round(2)

criteria = ["aic", "aicc", "bic"] deltas = [ pycatdap.target_summary( clean, target="fare", explanatory="class", criterion=c ).delta_aic for c in criteria ] pd.DataFrame({"delta_aic": deltas}, index=criteria).round(2)

9.2 Falling back to the contingency-table view¶

If you specifically need the categorical view of a continuous target (counts / row_prop / col_prop / Pearson residuals), pass target_bins= to discretize Y first. The function then routes through the original categorical path and returns a TargetSummary.

target_bins accepts an int (equal-width K), a sequence of explicit boundaries, or one of "quantile" (default q = 4), "equal_width", "fd" (Freedman-Diaconis).

In [ ]:

fallback = pycatdap.target_summary(
    clean, target="fare", explanatory="class", target_bins="quantile"
)
print(type(fallback).__name__, "delta_aic =", round(fallback.delta_aic, 2))
fallback.counts

fallback = pycatdap.target_summary( clean, target="fare", explanatory="class", target_bins="quantile" ) print(type(fallback).__name__, "delta_aic =", round(fallback.delta_aic, 2)) fallback.counts

Note: the regression-mode delta_aic and the contingency-mode delta_aic are NOT directly comparable — they're computed under different likelihood families (Gaussian vs multinomial). Pick one mode per analysis and compare features within it.

Summary¶

Step	Function	What it showed
Categorical × Categorical	target_summary(df, "survived", "sex")	Four perspectives + ΔAIC = −267
Multi-level	target_summary(df, "survived", "class")	Monotone class effect, ΔAIC = −91
Continuous explanatory	target_summary(clean, "survived", "age")	AIC-optimal binning, ΔAIC = −42
Binning comparison	bins=None vs bins=4 vs bins=[...]	Equal-width hides signal that AIC-optimal recovers
Ranking	target_summary per column, sort by delta_aic	fare and sex dominate survived
Export	.to_html(), .to_plotly_json(), backend="plotly"	Shareable / web-ready output
Continuous target	target_summary(clean, "fare", "class")	RegressionTargetSummary with Gaussian ΔAIC, R², and per-X-bin {count, target_mean, target_std}
Criterion choice	criterion="bic" \| "aic" \| "aicc"	Same R²; different penalty (BIC default, Yao 1988)
Contingency fallback	target_bins="quantile" \| int \| ...	Routes through the categorical path on a discretized Y

Next: For multivariate subset discovery, see notebook 04 (CATDAP-02 + nvar). For regression-error analysis (#18), residual_by_category will reuse the regression-AIC machinery via target=residual_column.