Skills101

AdvancedPython for Data Science & Machine Learning · Lesson 20 of 21 · Updated ·

Regularisation — Bias vs Variance, Ridge (L2), Lasso (L1) and Choosing the Penalty

A model that scores 99% on its training data and fails on new data is useless. The "Better Models" lesson showed this with decision trees. This lesson gives you the main tool for fixing it in linear models and neural networks: regularisation, which means adding a penalty for large weights to the loss. You will build Ridge and Lasso yourself with NumPy, and see exactly what the penalty does.

1. Too simple, too flexible, just right

Data: 12 students, the hours they studied per day in exam week, and their mock test score. Up to a point more study helps; after that, tiredness pulls the score down, so the true pattern is a curve. (The data is simulated from such a curve plus random noise, so we know the right answer.)

We fit polynomials of degree 1 (a straight line), 3 and 9, and check each on 50 new students the model never saw, called the validation set.

import numpy as np

rng = np.random.default_rng(1)
def true_score(h):
    return 40 + 8 * h - 0.6 * h ** 2

hours = np.sort(rng.uniform(0, 10, 12))
score = true_score(hours) + rng.normal(0, 3, 12)
val_hours = np.linspace(0, 10, 50)
val_score = true_score(val_hours) + rng.normal(0, 3, 50)

def features(h, degree):                      # z, z², z³ … with z = scaled hours
    z = (h - 5) / 3
    return np.column_stack([z ** k for k in range(1, degree + 1)])

def fit(X, y, lam=0.0):                       # least squares, plus an optional L2 penalty
    Xm, ym = X.mean(axis=0), y.mean()
    Xc = X - Xm
    w = np.linalg.solve(Xc.T @ Xc + lam * np.eye(X.shape[1]), Xc.T @ (y - ym))
    return w, ym - Xm @ w

print("degree  train MSE  validation MSE  largest |weight|")
for d in (1, 3, 9):
    w, b = fit(features(hours, d), score)
    train = np.mean((features(hours, d) @ w + b - score) ** 2)
    val = np.mean((features(val_hours, d) @ w + b - val_score) ** 2)
    print(f"{d:4d}  {train:9.1f}  {val:14.1f}  {np.abs(w).max():12.1f}")
degree  train MSE  validation MSE  largest |weight|
   1       21.5            32.8           6.5
   3        0.9             8.6           4.7
   9        0.2           173.2          89.0

Three fits to the same 12 students: a straight line misses the curve, a degree-3 curve follows the true pattern, and a degree-9 curve passes through every point but swings wildly between them

  • Degree 1 underfits. A straight line cannot bend, so it is wrong on both training and new data. This error from a too-simple model is called bias.
  • Degree 9 overfits. It almost passes through every training point (train MSE 0.2) but is terrible on new students (validation MSE 173). It learned the noise. This sensitivity to the exact training sample is called variance.
  • Degree 3 is about right: low error on both.

This is the bias–variance trade-off: more flexible models reduce bias but increase variance. Notice the last column too. The overfitted model needs huge weights to make its sharp swings.

2. The idea: make large weights expensive

Regularisation adds a penalty to the loss:

  • Ridge (L2): loss = MSE + λ × (sum of squared weights)
  • Lasso (L1): loss = MSE + λ × (sum of absolute weights)

λ (lambda, called alpha in scikit-learn) is the penalty strength. λ = 0 is ordinary regression. A larger λ forces weights towards zero, so the model becomes smoother and less flexible. The bias b is normally not penalised.

For Ridge there is a direct formula: w = (XᵀX + λI)⁻¹ Xᵀy, where I is the identity matrix. That is the lam * np.eye(...) term in fit() above, so we can try it straight away.

3. Ridge rescues the degree-9 model

import numpy as np

rng = np.random.default_rng(1)
def true_score(h):
    return 40 + 8 * h - 0.6 * h ** 2

hours = np.sort(rng.uniform(0, 10, 12))
score = true_score(hours) + rng.normal(0, 3, 12)
val_hours = np.linspace(0, 10, 50)
val_score = true_score(val_hours) + rng.normal(0, 3, 50)

def features(h, degree):
    z = (h - 5) / 3
    return np.column_stack([z ** k for k in range(1, degree + 1)])

def fit(X, y, lam=0.0):
    Xm, ym = X.mean(axis=0), y.mean()
    Xc = X - Xm
    w = np.linalg.solve(Xc.T @ Xc + lam * np.eye(X.shape[1]), Xc.T @ (y - ym))
    return w, ym - Xm @ w

print("lambda  train MSE  validation MSE  largest |weight|")
for lam in (0, 0.1, 1, 10, 100):
    w, b = fit(features(hours, 9), score, lam)
    train = np.mean((features(hours, 9) @ w + b - score) ** 2)
    val = np.mean((features(val_hours, 9) @ w + b - val_score) ** 2)
    print(f"{lam:6}  {train:9.1f}  {val:14.1f}  {np.abs(w).max():12.1f}")
lambda  train MSE  validation MSE  largest |weight|
     0        0.2           173.2          89.0
   0.1        0.8            15.1           4.0
     1        1.5            11.8           2.4
    10        4.6            12.5           0.9
   100       11.3            22.7           0.2

Training and validation error of the degree-9 model as the Ridge penalty grows: training error rises steadily, validation error falls sharply and then rises again, with the best value around lambda equal to 1

Read the table from top to bottom:

  1. λ = 0: the overfitted model, weights up to 89.
  2. λ = 1: weights shrink to at most 2.4 and validation error drops from 173 to about 12. Same 9 features, far better model.
  3. λ = 100: too much penalty. The weights are squashed almost to zero, the curve becomes nearly flat, and the model underfits again.

Training error always rises as λ grows, because the penalty stops the model from chasing the training points. Validation error falls and then rises. You pick the λ at the bottom of that U, using validation data or cross-validation, never the test set. The test set is kept for one final, honest check.

4. Lasso: shrinking some weights all the way to zero

Lasso's absolute-value penalty has a special effect: it sets unhelpful weights to exactly zero. That makes it an automatic feature selector.

Below, a placement package (in LPA, simulated) truly depends on CGPA, aptitude and coding scores only. We also add three useless columns. Lasso is fitted with coordinate descent: update one weight at a time, and use "soft thresholding" to shrink it by α, stopping at zero.

import numpy as np

rng = np.random.default_rng(5)
n = 60
names = ["cgpa", "aptitude", "coding", "height_cm", "roll_no_digit", "hostel_floor"]
X = np.column_stack([rng.uniform(6, 10, n), rng.uniform(30, 100, n), rng.uniform(0, 100, n),
                     rng.uniform(150, 190, n), rng.integers(0, 10, n), rng.integers(0, 5, n)]).astype(float)
package = 2 + 0.5 * X[:, 0] + 0.03 * X[:, 1] + 0.04 * X[:, 2] + rng.normal(0, 0.5, n)

Z = (X - X.mean(axis=0)) / X.std(axis=0)      # Lasso needs scaled features
y = package - package.mean()

def lasso(Z, y, alpha, rounds=200):
    w = np.zeros(Z.shape[1])
    for _ in range(rounds):
        for j in range(Z.shape[1]):
            residual = y - Z @ w + Z[:, j] * w[j]      # error ignoring feature j
            rho = np.mean(Z[:, j] * residual)
            w[j] = np.sign(rho) * max(abs(rho) - alpha, 0.0)   # soft thresholding
    return w

plain = np.linalg.lstsq(Z, y, rcond=None)[0]
print(f"{'feature':>14}  plain  alpha=0.1  alpha=0.3")
w1, w3 = lasso(Z, y, 0.1), lasso(Z, y, 0.3)
for i, name in enumerate(names):
    print(f"{name:>14}  {plain[i]:5.2f}  {w1[i] + 0:9.2f}  {w3[i] + 0:9.2f}")
       feature  plain  alpha=0.1  alpha=0.3
          cgpa   0.65       0.51       0.27
      aptitude   0.65       0.53       0.34
        coding   1.05       0.92       0.69
     height_cm   0.12       0.00       0.00
 roll_no_digit  -0.02       0.00       0.00
  hostel_floor  -0.03       0.00       0.00

Ordinary regression gives every column some weight, even roll numbers and hostel floors, because it fits small accidental patterns in the noise. With α = 0.1, Lasso sets all three useless features to exactly 0 and keeps the three real ones. With α = 0.3 the real weights shrink too: too strong a penalty starts to hurt.

5. Which one to use?

MethodPenaltyEffect on weightsUse when
Ridge (L2)λ × Σw²shrinks all weights smoothly, rarely to zeromany useful, related features
Lasso (L1)λ × Σ|w|sets some weights exactly to zeroyou suspect many features are useless; want a simpler model
Elastic Neta mix of bothsome zeros, stable with related featuresmany correlated features

In scikit-learn these are Ridge(alpha=…), Lasso(alpha=…) and ElasticNet(alpha=…, l1_ratio=…). LogisticRegression is regularised by default with an L2 penalty, controlled by C, which is the inverse of the strength: a smaller C means a stronger penalty. Neural networks use the same idea under the name weight decay.

Common mistakes

  • Regularising without scaling features. The penalty treats all weights alike, so a feature measured in large units gets a tiny weight and is barely penalised, while others are punished heavily.
  • Choosing λ (or α) by looking at the test set. Use validation data or cross-validation.
  • Thinking a bigger penalty is always safer. Too much penalty causes underfitting.
  • Confusing scikit-learn's C (bigger = weaker penalty) with alpha (bigger = stronger penalty).
  • Using Lasso's zero weights as proof that a feature "has no effect". With correlated features Lasso may keep one and drop the other almost at random.

Quick recap

  • Underfitting = high bias (too simple); overfitting = high variance (too flexible, learns noise).
  • Regularisation adds a penalty on weight size: Ridge uses squares (L2), Lasso uses absolute values (L1).
  • Larger λ → smaller weights → smoother model; too large → underfitting.
  • Lasso sets useless weights to exactly zero, so it also selects features.
  • Choose λ with validation data or cross-validation, and always scale features first.

Practice what you learned

📝 Quiz: Regularisation (Ridge and Lasso)5 questions · instant answers & explanations · +10 XP · counts towards your free certificateTake the quiz →