Better Models — Overfitting, Cross-Validation, Pipelines, Scaling and Hyperparameter Tuning

A model that memorises its training data is like a student who memorised last year's paper: perfect on that, lost on a new one. This lesson is about making models that generalise.

1. Seeing overfitting

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(7)
n = 400
X = pd.DataFrame({
    "cgpa": rng.normal(7.5, 0.9, n).clip(5, 10).round(2),
    "internships": rng.integers(0, 3, n),
    "projects": rng.integers(0, 5, n),
    "aptitude": rng.normal(65, 12, n).clip(20, 100).round(),
})
score = 1.4 * (X["cgpa"] - 7.5) + 0.8 * X["internships"] + 0.35 * X["projects"] + 0.05 * (X["aptitude"] - 65) + rng.normal(0, 0.8, n)
y = (score > 0.3).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

depths = range(1, 16)
train_acc, test_acc = [], []
for d in depths:
    tree = DecisionTreeClassifier(max_depth=d, random_state=0).fit(X_train, y_train)
    train_acc.append(tree.score(X_train, y_train))
    test_acc.append(tree.score(X_test, y_test))

fig, ax = plt.subplots(figsize=(7, 3.8))
ax.plot(depths, train_acc, marker="o", label="training accuracy")
ax.plot(depths, test_acc, marker="o", label="test accuracy")
ax.set_xlabel("max_depth of the tree (model complexity)")
ax.set_ylabel("Accuracy")
ax.set_title("Deeper trees memorise the training data — test accuracy stops improving")
ax.legend()
plt.savefig("overfitting-depth.svg")

• Underfitting (left): too simple, poor on both sets.
• Overfitting (right): near-perfect on training data, worse on the test set — the model learned the noise.
• The best complexity is where test accuracy is highest.

2. Cross-validation: a more honest score

One train/test split can be lucky or unlucky. k-fold cross-validation splits the data into k parts, trains k times, each time testing on a different part, and averages:

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(7)
n = 400
X = pd.DataFrame({
    "cgpa": rng.normal(7.5, 0.9, n).clip(5, 10).round(2),
    "internships": rng.integers(0, 3, n),
    "projects": rng.integers(0, 5, n),
    "aptitude": rng.normal(65, 12, n).clip(20, 100).round(),
})
score = 1.4 * (X["cgpa"] - 7.5) + 0.8 * X["internships"] + 0.35 * X["projects"] + 0.05 * (X["aptitude"] - 65) + rng.normal(0, 0.8, n)
y = (score > 0.3).astype(int)

for depth in (2, 4, 12):
    scores = cross_val_score(DecisionTreeClassifier(max_depth=depth, random_state=0), X, y, cv=5)
    print(f"depth {depth:2d}: folds {np.round(scores, 2)}  mean {scores.mean():.3f}")

depth  2: folds [0.79 0.79 0.74 0.78 0.78]  mean 0.772
depth  4: folds [0.84 0.81 0.75 0.86 0.81]  mean 0.815
depth 12: folds [0.78 0.82 0.78 0.76 0.78]  mean 0.782

Compare the mean across depths, and look at how much the folds vary — a big spread means the score is unstable.

3. Pipelines: preprocessing without leakage

Real data has text categories (branch) and numbers on different scales. A Pipeline chains preprocessing and the model, so the same steps run in training and prediction — and preprocessing learns only from the training folds (no data leakage):

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(7)
n = 400
X = pd.DataFrame({
    "cgpa": rng.normal(7.5, 0.9, n).clip(5, 10).round(2),
    "internships": rng.integers(0, 3, n),
    "projects": rng.integers(0, 5, n),
    "aptitude": rng.normal(65, 12, n).clip(20, 100).round(),
    "branch": rng.choice(["CSE", "IT", "ECE", "ME"], n),
})
bonus = X["branch"].map({"CSE": 0.4, "IT": 0.3, "ECE": 0.0, "ME": -0.3})
score = 1.4 * (X["cgpa"] - 7.5) + 0.8 * X["internships"] + 0.35 * X["projects"] + 0.05 * (X["aptitude"] - 65) + bonus + rng.normal(0, 0.8, n)
y = (score > 0.3).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

prepare = ColumnTransformer([
    ("numbers", StandardScaler(), ["cgpa", "internships", "projects", "aptitude"]),   # mean 0, sd 1
    ("branch", OneHotEncoder(handle_unknown="ignore"), ["branch"]),                     # CSE → [1,0,0,0] …
])
pipe = Pipeline([("prepare", prepare), ("model", LogisticRegression(max_iter=1000))])

search = GridSearchCV(pipe, {"model__C": [0.01, 0.1, 1, 10]}, cv=5)   # try 4 settings × 5 folds
search.fit(X_train, y_train)
print("best C:", search.best_params_["model__C"])
print("cross-validated accuracy:", round(search.best_score_, 3))
print("test accuracy:", round(search.score(X_test, y_test), 3))

best C: 0.1
cross-validated accuracy: 0.86
test accuracy: 0.86

• StandardScaler puts numbers on the same scale — important for many models (logistic regression with regularisation, k-NN, SVM, neural networks); trees don't need it.
• OneHotEncoder turns categories into 0/1 columns; handle_unknown="ignore" survives a new branch at prediction time.
• Hyperparameters (like C, max_depth) are settings you choose; GridSearchCV tries them with cross-validation on the training set only. The test set is used once, at the end.

4. Data leakage — the silent killer

Leakage = information from the test set (or the future) sneaking into training. Examples: scaling the whole dataset before splitting; a feature like "offer_letter_received" when predicting placement; random splits on time-series data. Leaky models look brilliant in the notebook and fail in production. Pipelines and honest feature choices prevent most of it.

Common mistakes

• Tuning on the test set (then it's no longer a fair exam).
• Preprocessing before the split.
• Comparing models on one lucky split instead of cross-validation.
• Chasing 0.5% accuracy while ignoring leakage and data quality.

Quick recap

• Overfitting: great on training, worse on test; control complexity.
• Cross-validation averages several splits for an honest estimate.
• Pipeline = preprocessing + model, fit on training data only; GridSearchCV tunes hyperparameters.
• Guard against data leakage; touch the test set once.