← Back to Projects
Machine LearningShipped

Loan Default Risk (CIS 5450)

Lending Club default model for CIS 5450: clean the data, kill leakage, then rank risk under a messy class balance.

Course project on ~1M Lending Club loans. I filtered to paid vs charged-off, threw out post-outcome cheat columns like recoveries and total_pymnt, and only fitted transforms after the train/test split. Models went from logistic regression up to tuned XGBoost around 0.7175 AUC. Tuning barely moved the needle once the columns were honest. Accuracy was a trap with an 80/20 split, so I cared about AUC, PR, and the fact that a 0.5 threshold is not a business decision.

Engineering highlights

  • Dropped post-outcome leakage before any model saw the data
  • Split first; impute, winsorize, encode, and SMOTE only on train
  • Tuned XGBoost ~0.7175 AUC; spent more time on features than hyperparams
Date
Spring 2026
Focus
ML Pipelines
Build stage
Course project complete
Disciplines
Data Cleaning · EDA · Modeling · Evaluation
Source code
cis5450tabular-mlleakagexgboostcredit-risk
Cover
CIS 5450 loan default risk cover
Full write-up below. The hero is the short version. This is the build story: what I designed, what broke, and why I made the calls I did.

Motivation

Why I started this

What pulled me in, and what I wanted to get better at.

Why I built it

I wanted a tabular ML project where the scary part was not the model library. Leakage and bad metrics can make you look smart while being wrong.

What interested me

XGBoost is not the story. Keeping recoveries and total_pymnt out of the feature set is. Same for fitting transforms only after the split.

What I wanted to learn

CIS 5450 was practice owning the whole notebook path: cleaning, EDA, honest split, model ladder, and threshold thinking that is not just accuracy.

System Overview

How the system fits together

Lending Club loans from roughly 2007 to 2018. Predict charged-off vs paid. Drop leakage, explore the class balance, split, then fit everything on train only. Models go from logistic regression up to tuned XGBoost around 0.7175 AUC.

01

Label & leakage control

Keep only definitive outcomes; remove post-default fields such as recoveries and total_pymnt.

02

EDA → feature design

Plots and tests motivate which affordability and risk signals enter the model.

03

Leakage-safe preprocessing

Split first; fit frequency encoding, impute, winsorize, K-Means, and scale on train only.

04

Model ladder

LR, ElasticNet, tree, RF, XGBoost; compare with AUC-ROC and PR thinking.

05

Decision analysis

Threshold sweeps translate probabilities into asymmetric FN/FP cost tradeoffs.

Data flow

Raw loan.csv → outcome filter + leakage drop → EDA → stratified 80/20 split → train-fitted transforms → models → AUC/PR + cost simulation.

Control flow

No transform is fitted before the split. SMOTE touches train only. Test remains naturally imbalanced so reported AUC reflects realistic ranking.

Project cover
CIS 5450 loan default risk cover

Cover card for the pipeline: kill leakage, split first, then model.

Engineering Breakdown

Broken down by discipline

Each block covers the goal, the design, what broke, what changed, and what shipped.

01Data Contract & Leakage Control

Data Contract & Leakage Control

Goal

Define a prediction-time feature set that a lender could have known at origination.

Design

Filter to paid vs charged-off. Drop high-missingness columns. Remove identifiers and post-outcome fields (recoveries, total_pymnt). Keep borrower credit history, loan terms, and categorical descriptors that exist before final outcome.

Challenges

  • Several 'strong' columns are cheat codes recorded after default.
  • Current/late loans would inject label noise if kept.
  • Grade and interest rate are valid yet partly encode prior underwriting; that must be stated, not hidden.

Iterations

  • Broad column dump.
  • Missingness cull.
  • Explicit leakage audit before modeling.

Final implementation

A compact, origination-plausible feature matrix with leakage columns removed before any model sees the data.

02EDA & Feature Engineering

EDA & Feature Engineering

Goal

Let distributional evidence decide which features and interactions deserve model capacity.

Design

Class balance, KDEs, outliers, correlations, and default rate by grade each map to a downstream choice: SMOTE/class weight, winsorization, regularization/ensembles, and engineered affordability features such as payment_to_income and dti_int_interaction. Hypothesis tests back DTI, interest rate, and revolving utilization differences.

Challenges

  • Accuracy is a trap under ~80/20 imbalance.
  • Skewed income and balances dominate naive distance methods without train-only winsorization.
  • Weak standalone separators can still help in combination.

Iterations

  • Raw EDA on lightly cleaned frames.
  • Engineered affordability/risk interactions after plots.
  • Train-only frequency encoding for employment titles.

Final implementation

An EDA-motivated feature set plus statistical tests that justify the strongest numeric risk signals.

03Modeling & Validation

Modeling & Validation

Goal

Compare linear and tree ensembles on held-out ranking performance without leaking fit statistics.

Design

Stratified split; train-only preprocess; SMOTE for linear models; class weighting / scale_pos_weight for trees. Ladder: unregularized-style LR baseline, ElasticNet, decision tree, random forest, XGBoost, then light tuning. Feature-importance pruning used as a parsimony diagnostic. Threshold and cost sweeps show why 0.5 is not a business policy.

Challenges

  • Linear models plateau near ~0.706 AUC; gains come from capacity and features, not more L2 alone.
  • Tuned XGBoost reaches about 0.7175 AUC with only a marginal lift over default XGBoost (~0.7167), suggesting a signal ceiling.
  • Cost-optimal thresholds on the test set are illustrative; production needs a validation-set policy.

Iterations

  • Baseline logistic regression.
  • Regularized linear and single tree.
  • Bagging and boosting.
  • Threshold / cost simulation after selecting the best ranker.

Final implementation

Tuned XGBoost retained as the primary ranker, with LR coefficients and RF importances agreeing on interest rate, grade structure, DTI, and revolving utilization as core signals.

Model ladder
Model ladder and leakage-safe pipeline summary

AUC progression and leakage-safe preprocessing summarized for review.

Key Design Decisions

Calls I actually made

What else was on the table, what I picked, and why it still made sense once the hardware was real.

01

AUC-ROC and PR over accuracy

The problem

Which metrics match the business question under imbalance?

Alternatives considered

  • Accuracy
  • AUC-ROC + PR/F1 family

Tradeoffs

Accuracy rewards predicting paid. Ranking and PR expose default capture.

Why I chose this

AUC-ROC for model comparison; PR and threshold analysis for decisions.

02

Split before fitting transforms

The problem

When may imputation, winsorization, encoding, and clustering see the data?

Alternatives considered

  • Fit on full data then split
  • Split then train-only fits

Tradeoffs

Full-data fits quietly leak distributional information into test.

Why I chose this

Stratified split first; every fitted statistic is train-only.

03

SMOTE only for linear training

The problem

How should class imbalance be handled across model families?

Alternatives considered

  • SMOTE everywhere
  • Class weights only
  • SMOTE for linear; weights for trees

Tradeoffs

SMOTE on trees can distort leaf statistics; linear models benefit more from balanced gradients.

Why I chose this

SMOTE for LR/ElasticNet train sets; trees use weighting mechanisms; test never resampled.

04

Treat 0.5 as non-sacred

The problem

How should probabilities become approve/deny?

Alternatives considered

  • Fixed 0.5
  • PR-informed threshold
  • Cost simulation

Tradeoffs

Fixed 0.5 ignores asymmetric FN/FP dollars. Test-set sweeps are illustrative, not deployment gospel.

Why I chose this

Show cost-aware threshold thinking; call out validation-set selection for real deployment.

Evolution

How it got here

Bench bring-up, CAD fits, soldering, and the demos in between. Not just the final photo.

  1. Stage 1

    Usable labels

    Definitive outcomes only; leakage columns removed.

    No media for this milestone yet.

  2. Stage 2

    EDA-driven features

    Plots and tests decide what enters the matrix.

    No media for this milestone yet.

  3. Stage 3

    Leakage-safe preprocess

    Split-first pipeline with train-only fits and SMOTE scope rules.

    No media for this milestone yet.

  4. Stage 4

    Model ladder + decisions

    Ensembles, ~0.7175 AUC tuned XGBoost, PR/cost threshold analysis.

    No media for this milestone yet.

Results & Validation

What held up

What worked in the end, what I can show for it, and where it's still limited.

Ranking performance

Notebook model progression / CV vs test comparison

Linear models ~0.706 AUC; default XGBoost ~0.7167; tuned XGBoost ~0.7175 on held-out test.

Statistical grounding

Hypothesis testing section

DTI, interest rate, and revolving utilization differ by default status with p < 0.0005 and positive bootstrap CIs.

Decision framing

Threshold / cost-benefit section

Cost simulation shows asymmetric FN/FP dollars can push an operating threshold away from 0.5 (illustrative test-set sweep).

Limitations

  • Course notebook analysis, not a deployed underwriting system.
  • Grade and interest rate embed prior underwriting; fairness audits would be required for real use.
  • Cost-optimal threshold reported from a test-set simulation should be re-selected on validation data before deployment.

Reflection

Looking back

What surprised me, what I'd redo, and questions I'm still chewing on.

What surprised me

  • Hyperparameter tuning barely moved AUC once features and leakage control were right.
  • The most valuable engineering was column discipline, not model novelty.

What I would redesign

  • Hold out a true validation split for threshold policy.
  • Stronger fairness analysis on grade/rate proxies.

Future improvements

  • Calibration plots and profitability curves on a time-based split.
  • Model monitoring story for drift in int_rate and utilization.

Questions that emerged

  • How much of tabular 'ML gains' are actually leakage removal?
  • When should unsupervised segments be features versus just EDA tools?