Real-world EDA on the seaborn Titanic dataset¶

This tutorial answers a common adoption question: how do I apply pycatdap to my own messy data? It walks through an end-to-end workflow on the 891-passenger Titanic dataset from seaborn (different from the bundled R contingency-table form): real missing values, mixed dtypes, and a binary target.

By the end of this notebook you'll know how to:

1. Make missingness the first thing you see
2. Use describe() to plan column-level treatment
3. Build a clean working subset and explore it visually
4. Run catdap1 / catdap2 on the cleaned data with a mix of pooled and categorical variables
5. Save a shareable HTML report and a Plotly JSON spec for web handoff

Prerequisite¶

This notebook fetches the Titanic dataset via seaborn, which is not a hard dependency of pycatdap. Install it via the tutorial extra:

pip install 'pycatdap[tutorial]'

In [ ]:

import matplotlib.pyplot as plt
import seaborn as sns

import pycatdap

print(f"pycatdap version: {pycatdap.__version__}")
df = sns.load_dataset("titanic")
print(f"shape: {df.shape}")
df.head()

import matplotlib.pyplot as plt import seaborn as sns import pycatdap print(f"pycatdap version: {pycatdap.__version__}") df = sns.load_dataset("titanic") print(f"shape: {df.shape}") df.head()

The columns mix dtypes — int64, float64, object, category, and bool — which is typical of CSV-derived datasets in the wild.

In [ ]:

df.dtypes

df.dtypes

1. Missingness first¶

Before anything else, look at where the missing values are. plot_missing() gives you that at a glance. The Plotly backend is convenient because you can hover to read exact counts.

In [ ]:

fig = pycatdap.plot_missing(df, backend="plotly")
fig.show()

fig = pycatdap.plot_missing(df, backend="plotly") fig.show()

Two columns are problematic: deck is missing for ~77% of passengers and age for ~20%. embarked / embark_town have a couple of missing rows.

2. Summary table¶

describe() confirms kind detection and surfaces missing rates in one row per variable.

In [ ]:

summary = pycatdap.describe(df)
summary.summary[["kind", "n_obs", "n_missing", "missing_rate", "n_unique"]]

summary = pycatdap.describe(df) summary.summary[["kind", "n_obs", "n_missing", "missing_rate", "n_unique"]]

A few practical observations:

• survived is the response (binary 0/1, no missing)
• alive is the same information as survived — must be dropped to avoid leakage
• deck is too sparse to keep — drop it
• age is recoverable via imputation (median for a first pass)
• embarked/embark_town have 2 missing rows — drop them
• class (category) and pclass (int) are aliases — keep one

3. Clean the data¶

We follow a minimal recipe: drop the leaky alive column, drop the too-sparse deck, impute age with the median, and drop the rows where embarked is missing.

In [ ]:

clean = df.drop(columns=["alive", "deck"]).copy()
clean["age"] = clean["age"].fillna(clean["age"].median())
clean = clean.dropna(subset=["embarked", "embark_town"])
print(f"cleaned shape: {clean.shape}")
pycatdap.describe(clean).summary[["kind", "n_missing", "n_unique"]]

clean = df.drop(columns=["alive", "deck"]).copy() clean["age"] = clean["age"].fillna(clean["age"].median()) clean = clean.dropna(subset=["embarked", "embark_town"]) print(f"cleaned shape: {clean.shape}") pycatdap.describe(clean).summary[["kind", "n_missing", "n_unique"]]

4. Visualize representative variables¶

plot_variable infers kind from dtype, so the same call works for continuous (age, fare) and categorical (sex, class, embarked) columns.

In [ ]:

vars_to_show = ["age", "fare", "sex", "class", "embarked"]
fig, axes = plt.subplots(1, 5, figsize=(20, 3))
for ax, col in zip(axes, vars_to_show, strict=True):
    pycatdap.plot_variable(clean, col, ax=ax)
fig.tight_layout()
plt.show()

vars_to_show = ["age", "fare", "sex", "class", "embarked"] fig, axes = plt.subplots(1, 5, figsize=(20, 3)) for ax, col in zip(axes, vars_to_show, strict=True): pycatdap.plot_variable(clean, col, ax=ax) fig.tight_layout() plt.show()

5. CATDAP-01 on the cleaned subset¶

For the first pass we pick a small subset of explanatory variables that represent the available signal: demographic (sex, age), travel context (class, fare, embarked).

In [ ]:

subset_cols = ["survived", "sex", "class", "age", "fare", "embarked"]
work = clean[subset_cols].copy()
r1 = pycatdap.catdap1(work, response_names=["survived"])
r1.aic.loc["survived"].sort_values()

subset_cols = ["survived", "sex", "class", "age", "fare", "embarked"] work = clean[subset_cols].copy() r1 = pycatdap.catdap1(work, response_names=["survived"]) r1.aic.loc["survived"].sort_values()

sex and class dominate, as expected from the historical "women and children first" account. fare ranks high because it correlates strongly with class. Note that age looks worse than the null model when treated as raw categorical — every unique age is its own bucket, which over-parameterizes. CATDAP-02 with pooling fixes this.

6. CATDAP-02 with mixed pooling¶

Mix pool=0 (search for AIC-optimal bins) for continuous variables with pool=2 (treat as categorical) for the rest. The accuracy keyword sets the minimum bin width — here 1 year for age and $5 for fare.

In [ ]:

# Column order: survived, sex, class, age, fare, embarked
r2 = pycatdap.catdap2(
    work,
    pool=[2, 2, 2, 0, 0, 2],
    response_name="survived",
    accuracy=[0.0, 0.0, 0.0, 1.0, 5.0, 0.0],
    nvar=3,
)
print(f"base AIC: {r2.base_aic:.2f}")
r2.aic

# Column order: survived, sex, class, age, fare, embarked r2 = pycatdap.catdap2( work, pool=[2, 2, 2, 0, 0, 2], response_name="survived", accuracy=[0.0, 0.0, 0.0, 1.0, 5.0, 0.0], nvar=3, ) print(f"base AIC: {r2.base_aic:.2f}") r2.aic

In [ ]:

# AIC-optimal cuts for the two continuous variables
{name: [round(x, 2) for x in cuts] for name, cuts in r2.intervals.items()}

# AIC-optimal cuts for the two continuous variables {name: [round(x, 2) for x in cuts] for name, cuts in r2.intervals.items()}

Once pooled, age becomes informative (ΔAIC ≈ -51) instead of harmful. The cuts cluster around childhood, early adulthood, and old age — natural breakpoints for survival on a passenger ship.

7. Best subset¶

r2.subsets is sorted by AIC. The best 2-variable combination beats the best single variable by a clear margin.

In [ ]:

for s in r2.subsets:
    print(f"  n_vars={s.n_vars}  AIC={s.aic:>8.2f}  vars={s.variables}")

for s in r2.subsets: print(f" n_vars={s.n_vars} AIC={s.aic:>8.2f} vars={s.variables}")

8. Drill into the top pair with target_summary¶

CATDAP-01 ranked sex and class highest. To inspect a single (target, explanatory) pair in depth — counts, row / column proportions, Pearson residuals, and ΔAIC, plus the AIC-optimal binning when the explanatory is continuous — use target_summary. This also exercises the cleanup-before-EDA promise of this notebook: the cleaned age column (median-imputed) is ready to be binned.

In [ ]:

pair_sex = pycatdap.target_summary(clean, target="survived", explanatory="sex")
print(f"sex   delta_aic = {pair_sex.delta_aic:.3f}")
pair_sex.col_prop.round(3)

pair_sex = pycatdap.target_summary(clean, target="survived", explanatory="sex") print(f"sex delta_aic = {pair_sex.delta_aic:.3f}") pair_sex.col_prop.round(3)

In [ ]:

pair_age = pycatdap.target_summary(clean, target="survived", explanatory="age")
print(f"age   delta_aic = {pair_age.delta_aic:.3f}")
print(f"      n_bins    = {len(pair_age.intervals) + 1}")
print(f"      cuts head = {[round(b, 2) for b in pair_age.intervals[:6]]}")

pair_age = pycatdap.target_summary(clean, target="survived", explanatory="age") print(f"age delta_aic = {pair_age.delta_aic:.3f}") print(f" n_bins = {len(pair_age.intervals) + 1}") print(f" cuts head = {[round(b, 2) for b in pair_age.intervals[:6]]}")

sex produces a strong ΔAIC. The continuous age column is auto-binned via AIC-optimal pooling; the cut points capture the survival differences between infants, children, young adults, and the elderly. The full target-pair workflow (multi-feature ranking, equal-width vs AIC binning comparison, Plotly export, and the planned continuous-target support) is covered in notebook 06.

(sex, class) is the best 2-variable subset. Adding more variables yields only marginal improvement — useful when justifying a parsimonious model.

9. Shareable HTML report¶

Every result object has .to_html() for a standalone, dependency-free HTML file you can email or attach to a ticket.

In [ ]:

html = summary.to_html()
print(f"HTML report: {len(html)} bytes")
# To save locally:
# summary.to_html("titanic_describe.html")

html = summary.to_html() print(f"HTML report: {len(html)} bytes") # To save locally: # summary.to_html("titanic_describe.html")

10. Plotly JSON spec for web handoff¶

For LizyStudio (or any react-plotly.js frontend), each result exposes a .to_plotly_json() method that returns a complete Plotly figure spec — no server-side rendering needed.

In [ ]:

spec = r2.to_plotly_json()
print("keys:", list(spec.keys()))
print("data type:", spec["data"][0]["type"])
print("title:", spec["layout"]["title"])

spec = r2.to_plotly_json() print("keys:", list(spec.keys())) print("data type:", spec["data"][0]["type"]) print("title:", spec["layout"]["title"])

Summary¶

Step	Function	What it showed
Missingness	plot_missing(df, backend="plotly")	Drop deck, impute age
Inventory	describe(df).summary	Kind detection + missing rates
Cleanup	pandas	Drop leaky/sparse cols, impute, drop bad rows
Univariate	plot_variable(...) grid	Distribution of mixed dtypes
Pairwise	catdap1(work, response_names=["survived"])	Sex & class dominate
Pooled	catdap2(..., pool=[2,2,2,0,0,2])	age becomes useful once pooled
Subset	r2.subsets	Best 2-var = (sex, class)
Target pair	target_summary(clean, "survived", col)	Per-feature ΔAIC + AIC binning for age
Export	.to_html(), .to_plotly_json()	Shareable report + web handoff

Next steps¶

• 03. AIC-optimal binning on iris dives deeper into the pooling search and shows the cut points overlaid on the data.
• 02. EDA on Titanic (bundled long-form) compares this messy-data workflow against the canonical R datasets::Titanic contingency table.
• 06. Target-pair analysis generalizes the target-pair drill-down with multi-feature ranking, AIC vs equal-width binning comparison, and a Plotly-backed export.