Skills101

AdvancedPython Programming · Lesson 18 of 30 · Updated ·

Iterators and Generators in Python — iter, next and yield

You have used for loops on lists, strings, dictionaries and files. How can one loop work on so many different types? The answer is the iterator protocol. Once you understand it, you can write generators: functions that produce values one at a time, using almost no memory. Generators are a popular interview topic.

1. Iterable and iterator

Two words that sound alike but mean different things:

  • An iterable is anything you can loop over: a list, string, tuple, dict, set, file, range. It can give you an iterator when asked.
  • An iterator is the object that actually walks through the values. It remembers where it is, and gives the next value each time you ask.

Two built-in functions connect them:

  • iter(iterable) returns a new iterator.
  • next(iterator) returns the next value, or raises StopIteration when there are no values left.
marks = [72, 85, 90]
it = iter(marks)
print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("no more values")
72
85
90
no more values

2. What a for loop really does

The loop for m in marks: is roughly this code:

marks = [72, 85, 90]
it = iter(marks)
while True:
    try:
        m = next(it)
    except StopIteration:
        break
    print(m)
72
85
90

So any object that supports iter() and next() works in a for loop, in sum(), list(), max(), in and unpacking.

An iterator is used up after one pass. A list is not — each for loop over a list asks for a fresh iterator.

it = iter([1, 2, 3])
print(list(it))
print(list(it))     # already used up
[1, 2, 3]
[]

3. Writing your own iterator class

A class becomes an iterator when it has two methods:

  • __iter__(self) — returns the iterator (here, the object itself).
  • __next__(self) — returns the next value or raises StopIteration.
class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(3):
    print(n, end=" ")
print("Lift off!")
3 2 1 Lift off!

This works, but it is a lot of code for a simple idea. Generators make it much shorter.

4. Generator functions and yield

A function that contains the keyword yield is a generator function. Calling it does not run the body. It returns a generator object, which is an iterator. Each next() runs the body until the next yield, hands out that value, and then pauses with all local variables saved.

def countdown(start):
    print("starting")
    while start > 0:
        yield start
        start -= 1
    print("finished")

g = countdown(3)
print(type(g).__name__)
print(next(g))
print(next(g))
print(list(g))      # takes the remaining values
generator
starting
3
2
finished
[1]

Timeline of countdown(3): each next() call runs the body until a yield, hands out 3, then 2, then 1, and pauses in between; the final call runs to the end and raises StopIteration

Notice: "starting" is printed only at the first next(), not when countdown(3) was called. When the function ends, the generator raises StopIteration for you.

returnyield
Gives backone value, oncemany values, one at a time
After itfunction is finishedfunction pauses, can resume
Calling the function givesthe valuea generator object

5. Generator expressions and memory

A generator expression looks like a list comprehension, but with round brackets. It produces values lazily: only when they are asked for.

squares_list = [n * n for n in range(5)]
squares_gen = (n * n for n in range(5))
print(squares_list)
print(squares_gen.__class__.__name__)
print(sum(n * n for n in range(1, 1_000_001)))
[0, 1, 4, 9, 16]
generator
333333833333500000

The list comprehension builds all values in memory at once. The generator holds only the current value. For a million numbers, sum(n * n for n in ...) never stores a million-item list. When a generator expression is the only argument, you can drop the extra brackets, as shown.

This laziness also lets you work with sequences that never end:

def ticket_numbers(start):
    n = start
    while True:          # infinite, but safe: values are made on demand
        yield f"T{n}"
        n += 1

gen = ticket_numbers(101)
for _ in range(3):
    print(next(gen))
T101
T102
T103

6. Pipelines and yield from

Generators can feed each other, like stages in a factory. Each stage handles one item at a time.

def read_lines():
    yield from ["Asha,92", "Ravi,38", "Neha,75", "Imran,29"]

def parse(lines):
    for line in lines:
        name, mark = line.split(",")
        yield name, int(mark)

def failed(records):
    for name, mark in records:
        if mark < 40:
            yield name

print(list(failed(parse(read_lines()))))
['Ravi', 'Imran']

yield from iterable hands out every value of another iterable, one by one. It is shorter than a for loop with yield inside. In a real program read_lines could read a huge log file line by line, and the pipeline would still use very little memory.

Common mistakes

  • Expecting a generator to run when you call the function. It only runs when you iterate.
  • Looping over the same generator twice and getting nothing the second time.
  • Calling len() on a generator. Generators have no length; convert to a list first if you really need one.
  • Using [...] when you only need to loop once over a very large result. Use (...) instead.
  • Putting return value in a generator and expecting the loop to receive it. A loop only sees yielded values.

Quick recap

  • Iterable: can be looped over. Iterator: remembers its position and has __next__.
  • iter() makes an iterator; next() gets a value; StopIteration ends the loop.
  • A function with yield returns a generator, which pauses at each yield.
  • Generator expressions (x for x in ...) are lazy and save memory.
  • Generators are single-use; chain them into pipelines; yield from delegates.

Practice what you learned

📝 Quiz: Iterators and Generators5 questions · instant answers & explanations · +10 XP · counts towards your free certificateTake the quiz →