Testing AI-Written Code — Tests First, Edge Cases, Property Checks and Letting AI Write Tests Safely

Reviewing finds many bugs; running finds the rest. With AI writing code faster than you can read it, tests are how you keep up. The good news: AI is also excellent at helping you write tests — if you use it correctly.

1. Test-first with AI

1. You turn the requirement into tests (examples with expected answers).
2. The AI writes the implementation.
3. Run the tests.
4. Failures go back with exact details: input, expected, actual.
5. Accept only when everything passes — and you understand the code.

Tests written before the code describe what you want. Tests written by the same AI after its own code often just describe what the code happens to do.

2. Edge-case tables

For any function, brainstorm inputs across these categories:

An AI-style first attempt and the tests that catch it:

def split_bill_ai(total_paise, people):
    share = total_paise // people
    return [share] * people

def split_bill_fixed(total_paise, people):
    if people <= 0:
        raise ValueError("people must be at least 1")
    if total_paise < 0:
        raise ValueError("amount can't be negative")
    share, extra = divmod(total_paise, people)
    return [share + 1 if i < extra else share for i in range(people)]

cases = [(90000, 3), (10000, 3), (0, 4), (100000000, 7)]
for total, people in cases:
    ai, fixed = split_bill_ai(total, people), split_bill_fixed(total, people)
    print(f"₹{total/100:,.2f} / {people}: AI loses {total - sum(ai)} paise | fixed sums to total: {sum(fixed) == total}")

for bad in [(1000, 0), (-500, 2)]:
    try:
        split_bill_fixed(*bad)
    except ValueError as e:
        print("rejected", bad, "->", e)

₹900.00 / 3: AI loses 0 paise | fixed sums to total: True
₹100.00 / 3: AI loses 1 paise | fixed sums to total: True
₹0.00 / 4: AI loses 0 paise | fixed sums to total: True
₹1,000,000.00 / 7: AI loses 2 paise | fixed sums to total: True
rejected (1000, 0) -> people must be at least 1
rejected (-500, 2) -> amount can't be negative

The AI version "works" on ₹900 / 3 but quietly loses paise whenever the amount doesn't divide evenly, and crashes with an unhelpful ZeroDivisionError for 0 people. The edge-case table found both in seconds.

3. Randomised testing against a brute-force reference

For algorithms (DSA problems especially), a powerful trick: write a slow but obviously correct version, then compare it with the fast AI version on hundreds of random inputs. (This is how Skills101 verifies every coding problem's reference solution.)

Task: maximum sum of any contiguous subarray (Kadane's algorithm). Suppose the AI gives this:

import random

def max_subarray_ai(nums):            # AI version: subtly wrong
    best = cur = 0
    for x in nums:
        cur = max(0, cur + x)
        best = max(best, cur)
    return best

def max_subarray_brute(nums):          # obviously correct: try every subarray
    return max(sum(nums[i:j + 1]) for i in range(len(nums)) for j in range(i, len(nums)))

random.seed(7)
for trial in range(500):
    nums = [random.randint(-10, 10) for _ in range(random.randint(1, 8))]
    if max_subarray_ai(nums) != max_subarray_brute(nums):
        print("MISMATCH on", nums, "ai =", max_subarray_ai(nums), "correct =", max_subarray_brute(nums))
        break
else:
    print("all 500 random tests passed")

MISMATCH on [-3, -8] ai = 0 correct = -3

The bug: when all numbers are negative, the answer is the largest (least negative) number, but the AI version returns 0 — an empty subarray that the problem doesn't allow. The random tester found an input you might never have written by hand. Fixed version and re-check:

import random

def max_subarray_fixed(nums):
    best = cur = nums[0]
    for x in nums[1:]:
        cur = max(x, cur + x)
        best = max(best, cur)
    return best

def max_subarray_brute(nums):
    return max(sum(nums[i:j + 1]) for i in range(len(nums)) for j in range(i, len(nums)))

random.seed(7)
ok = all(max_subarray_fixed(n) == max_subarray_brute(n)
         for n in ([random.randint(-10, 10) for _ in range(random.randint(1, 8))] for _ in range(500)))
print("fixed version passes 500 random tests:", ok)

fixed version passes 500 random tests: True

Libraries like Hypothesis (Python) and fast-check (JavaScript) do this "property-based testing" professionally — they even shrink a failing input to the smallest example.

4. Letting AI write tests — safely

AI is great at suggesting tests. Use prompts like:

• "List edge cases for this function as a table: input, expected output, why it matters. Don't write the implementation."
• "Write pytest tests for these 8 cases. Use exact expected values I gave you."
• "What inputs would break this code?"

Then check the expected values yourself. An AI that writes both code and tests can make the same wrong assumption in both, and everything passes. Your job is to be the independent judge of what "correct" means.

5. Make tests part of the workflow

• Run tests locally before every commit; run them in CI on every push (Cloud & DevOps track, lesson 5).
• When a bug is found, first add a test that reproduces it, then fix it — it can never silently return.
• For agentic tools, tell them: "Run the test suite after changes and show me the output." Then read the output yourself.

Common mistakes

• Only testing the example in the prompt.
• Letting the AI change the tests to make them pass.
• Expected values copied from the code's own output ("it printed 42, so 42 is correct").
• No negative tests (invalid inputs should fail cleanly).

Quick recap

• Tests first: they define correct; the AI's code must meet them.
• Use edge-case tables: normal, boundary, uneven, invalid, large.
• Random testing vs a brute-force reference finds bugs you'd never think of.
• AI can suggest tests; you verify expected values. Tests run in CI; every bug gets a test.