# Logistic Regression from Scratch — Sigmoid, Log Loss, Gradient Descent and the Decision Threshold

Source: https://skills101.in/learn/data-science-ml/ml-logistic-regression-from-scratch — Skills101 free tutorial (Python for Data Science & Machine Learning). Updated 2026-11-21.

In the classification lesson, scikit-learn's `LogisticRegression` predicted whether a student gets placed. In the last lesson you wrote gradient descent for a straight line. Put the two together and you can build logistic regression yourself in about 20 lines. Doing it once makes three interview favourites easy to explain: the **sigmoid**, **log loss**, and the **decision threshold**.

## 1. The problem: predicting yes or no

Data: the score of 12 students in a mock placement test (out of 100) and whether they cleared the real company test (1 = cleared, 0 = not). This is practice data.

We want a **probability**: "a student with 62 marks has a 70% chance of clearing". A straight line `w × score + b` cannot give that. It goes below 0 and above 1, and a probability of 1.3 or −0.2 makes no sense.

```python
import numpy as np

score = np.array([35, 42, 48, 50, 55, 58, 62, 66, 70, 75, 82, 90], dtype=float)
cleared = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1], dtype=float)

w, b = np.polyfit(score, cleared, 1)          # an ordinary straight line
for s in (20, 60, 110):
    print(f"score {s:3d}: line says {w * s + b:+.2f}")
```

```text
score  20: line says -0.31
score  60: line says +0.48
score 110: line says +1.47
```

The line gives values outside 0 to 1. We need a function that squeezes any number into that range.

## 2. The sigmoid function

The **sigmoid** (also called the logistic function) is `σ(z) = 1 / (1 + e^(−z))`.

- Large positive `z` → close to 1.
- Large negative `z` → close to 0.
- `z = 0` → exactly 0.5.

```python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in (-6, -2, 0, 2, 6):
    print(f"z = {z:+d}  ->  sigmoid = {sigmoid(z):.3f}")
```

```text
z = -6  ->  sigmoid = 0.002
z = -2  ->  sigmoid = 0.119
z = +0  ->  sigmoid = 0.500
z = +2  ->  sigmoid = 0.881
z = +6  ->  sigmoid = 0.998
```

Logistic regression is just: **probability = sigmoid(w × x + b)**. The straight line is still inside; the sigmoid bends it into an S-shape between 0 and 1. The value `z = w × x + b` is called the **log-odds** (or **logit**).

![S-shaped sigmoid curve fitted to 12 students: students who did not clear sit at probability 0 on the left, students who cleared sit at 1 on the right, and the curve crosses 0.5 near a mock score of 60](https://skills101.in/learn-img/data-science-ml/sigmoid-fit.svg)

## 3. Log loss: the right way to measure a wrong probability

For regression we used mean squared error. For probabilities we use **log loss** (also called **binary cross-entropy**):

- If the true answer is 1, the loss is `−log(p)`.
- If the true answer is 0, the loss is `−log(1 − p)`.

So the loss is small when you give the true answer a high probability, and it grows **without limit** when you are confidently wrong.

```python
import numpy as np

def log_loss_one(y, p):
    return -(y * np.log(p) + (1 - y) * np.log(1 - p))

print("Student actually cleared (y = 1):")
for p in (0.99, 0.7, 0.5, 0.1, 0.01):
    print(f"  model said p = {p:4}: log loss = {log_loss_one(1, p):.3f}   squared error = {(1 - p) ** 2:.3f}")
```

```text
Student actually cleared (y = 1):
  model said p = 0.99: log loss = 0.010   squared error = 0.000
  model said p =  0.7: log loss = 0.357   squared error = 0.090
  model said p =  0.5: log loss = 0.693   squared error = 0.250
  model said p =  0.1: log loss = 2.303   squared error = 0.810
  model said p = 0.01: log loss = 4.605   squared error = 0.980
```

Compare the last row. A model that said "1% chance" for a student who cleared is badly wrong. Squared error caps that mistake at about 1, but log loss gives 4.6, a much stronger push to fix it. Log loss also gives a smooth bowl-shaped surface for gradient descent, so training is reliable.

![Log loss against the predicted probability for a student who really cleared: the loss is near 0 when p is close to 1 and rises steeply as p moves towards 0](https://skills101.in/learn-img/data-science-ml/log-loss-curve.svg)

## 4. Training with gradient descent

The gradient of log loss has a neat form, the same shape as for linear regression:

- ∂loss/∂w = mean((p − y) × x)
- ∂loss/∂b = mean(p − y)

We **standardise** the score first (subtract the mean, divide by the standard deviation), because the last lesson showed that scaled features train much faster.

```python
import numpy as np

score = np.array([35, 42, 48, 50, 55, 58, 62, 66, 70, 75, 82, 90], dtype=float)
cleared = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1], dtype=float)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

mean, std = score.mean(), score.std()
x = (score - mean) / std

w, b, lr = 0.0, 0.0, 0.5
for epoch in range(1001):
    p = sigmoid(w * x + b)
    loss = -np.mean(cleared * np.log(p) + (1 - cleared) * np.log(1 - p))
    w -= lr * np.mean((p - cleared) * x)
    b -= lr * np.mean(p - cleared)
    if epoch in (0, 50, 1000):
        print(f"epoch {epoch:4d}: log loss = {loss:.4f}")

print(f"w = {w:.3f}, b = {b:.3f}  (on the standardised score)")
for s in (45, 60, 80):
    print(f"mock score {s}: P(clear) = {sigmoid(w * (s - mean) / std + b):.2f}")
```

```text
epoch    0: log loss = 0.6931
epoch   50: log loss = 0.4622
epoch 1000: log loss = 0.4613
w = 1.873, b = 0.055  (on the standardised score)
mock score 45: P(clear) = 0.13
mock score 60: P(clear) = 0.48
mock score 80: P(clear) = 0.91
```

At epoch 0 every prediction is 0.5, so the loss is log(2) ≈ 0.693, the loss of pure guessing. Training brings it down, and the model now gives sensible probabilities: low for 45 marks, high for 80.

## 5. From probability to decision: the threshold

The model outputs a probability. To act, you pick a **threshold**: predict "will clear" if p ≥ threshold. The default 0.5 is not always right.

```python
import numpy as np

score = np.array([35, 42, 48, 50, 55, 58, 62, 66, 70, 75, 82, 90], dtype=float)
cleared = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1])

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

mean, std = score.mean(), score.std()
x = (score - mean) / std
w = b = 0.0
for _ in range(1001):
    p = sigmoid(w * x + b)
    w -= 0.5 * np.mean((p - cleared) * x)
    b -= 0.5 * np.mean(p - cleared)
p = sigmoid(w * x + b)

print("threshold  predicted-yes  precision  recall")
for t in (0.3, 0.5, 0.7):
    pred = (p >= t).astype(int)
    tp = int(((pred == 1) & (cleared == 1)).sum())
    precision = tp / pred.sum()
    recall = tp / cleared.sum()
    print(f"   {t}          {pred.sum():2d}          {precision:.2f}      {recall:.2f}")
```

```text
threshold  predicted-yes  precision  recall
   0.3           8          0.62      0.83
   0.5           6          0.83      0.83
   0.7           4          0.75      0.50
```

- **Lower threshold** → more students flagged "will clear" → higher **recall** (you miss fewer real successes) but lower **precision** (more false alarms).
- **Higher threshold** → fewer, surer predictions → usually higher precision, lower recall.

With only 12 students the numbers jump around: at 0.7 precision actually dips to 0.75, because one of the four flagged students (70 marks) did not clear. On real data with thousands of rows the trade-off is much smoother.

Choose the threshold by the cost of each mistake. Suppose a college uses the model to give a few scarce company-referral slots to students likely to clear: a false alarm wastes a slot, so a higher threshold makes sense. If instead it is used to **find** students likely to clear and invite them to a free advanced workshop, missing a good student is the bigger loss, so a lower threshold makes sense. The threshold is a business decision, not a property of the model.

## 6. Reading the weights as odds

**Odds** = p / (1 − p). A probability of 0.8 is odds of 4 (4 to 1). In logistic regression, increasing a feature by 1 unit **multiplies the odds by e^w**.

In our model the feature is the standardised score, so one unit is one standard deviation (about 15.7 marks):

```python
import numpy as np

score = np.array([35, 42, 48, 50, 55, 58, 62, 66, 70, 75, 82, 90], dtype=float)
cleared = np.array([0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1], dtype=float)
mean, std = score.mean(), score.std()
x = (score - mean) / std
w = b = 0.0
for _ in range(1001):
    p = 1 / (1 + np.exp(-(w * x + b)))
    w -= 0.5 * np.mean((p - cleared) * x)
    b -= 0.5 * np.mean(p - cleared)

print(f"one standard deviation = {std:.1f} marks")
print(f"odds multiply by e^w = {np.exp(w):.2f} for each extra {std:.1f} marks")
```

```text
one standard deviation = 15.7 marks
odds multiply by e^w = 6.51 for each extra 15.7 marks
```

This is why logistic regression is popular in banking and healthcare: each weight has a clear meaning that you can explain to a manager.

## Common mistakes

- Using a straight line (linear regression) for yes/no targets and treating its output as a probability.
- Using mean squared error instead of log loss to train a classifier.
- Always using 0.5 as the threshold without thinking about the cost of each kind of mistake.
- Forgetting to scale features before gradient descent.
- Reading a weight as "adds w to the probability". It multiplies the **odds** by e^w; the change in probability depends on where you start.
- Taking `log(0)` when a probability becomes exactly 0 or 1. Libraries clip p to a tiny range such as [1e-15, 1 − 1e-15] to avoid this.

## Quick recap

- Logistic regression: p = sigmoid(w·x + b); the sigmoid maps any number into (0, 1).
- Train it by minimising log loss with gradient descent; the gradient is mean((p − y) × x).
- Log loss punishes confident wrong answers very strongly.
- The threshold turns probabilities into decisions and trades precision against recall.
- e^w is the odds ratio for a one-unit increase in a feature.


Practice quiz and free certificate: https://skills101.in/learn/data-science-ml/ml-logistic-regression-from-scratch