Iterators, Iterables & Generators — How for Really Works

Every for loop you've written (Tutorial 8) and every comprehension and generator expression (Tutorial 14) rests on one small protocol. The thesis: for is syntactic sugar — under it, iter() produces an iterator, next() pulls one item at a time, and a StopIteration signal ends the loop — and yield is the easy way to write your own iterators that produce values lazily, one at a time, without ever building a list.

This is the deep pass of loops and generator expressions. Every output below was produced by running the code.

Table of Contents

1. The iteration protocol
2. Iterable vs iterator
3. Generators with yield
4. Laziness and infinite streams
5. Generator pipelines
6. Mental-model summary
7. Gotcha checklist

1. The iteration protocol

A for loop doesn't have magic access to a sequence's contents. It calls two functions: iter(obj) to get an iterator, then next(iterator) repeatedly to pull items. You can do this by hand.

Output:

10
20
30

Analysis. iter(nums) produced an iterator over the list; each next(it) returned the next item. A for loop does exactly this — call iter, then next over and over — and stops when the iterator signals it's done. The diagram shows the full conversation, including the final next() that ends it.

Intuition. Mechanism. The protocol is two dunder methods: iter(x) calls x.__iter__() to get an iterator, and next(it) calls it.__next__() to get the next value. When there's nothing left, __next__ raises StopIteration — the agreed "I'm done" signal that for catches silently to end the loop.

Concrete bite. Call next past the end yourself and you see the signal a for loop normally hides:

1
Traceback (most recent call last):
  File "/w/main.py", line 3, in <module>
    print(next(it))   # nothing left to give
          ~~~~^^^^
StopIteration

The first next returns 1; the second finds the iterator empty and raises StopIteration. Inside a for loop you never see this — the loop catches it and stops — but by hand it's a real, uncaught exception.

2. Iterable vs iterator

Two related words, one crucial difference. An iterable is anything you can iterate (a list, a string) — it can produce a fresh iterator on demand. An iterator is the one-shot cursor that actually walks through, and it gets used up.

Output:

[1, 2, 3]
[1, 2, 3]
[1, 2, 3]
[]

Analysis. The list nums is an iterable: list(nums) works twice, each time starting fresh. But it = iter(nums) is an iterator: the first list(it) drains it completely, so the second list(it) finds it empty and returns []. Same data, but the iterator is a consumable cursor over it.

Intuition. Mechanism. An iterable's __iter__ returns a new iterator each call (so you can loop it again and again). An iterator's __iter__ returns itself — it is its own cursor — and once its items are pulled, it's exhausted for good.

Concrete bite. You can see the self-return directly:

True
False

iter(it) is it is True — an iterator handed back itself. iter(nums) is nums is False — a list handed back a separate iterator object. That's why a list survives repeated looping and a raw iterator doesn't.

3. Generators with yield

Writing the __iter__/__next__ pair by hand is tedious. A generator function — any def containing yield — does it for you: each yield produces a value and pauses, resuming where it left off on the next pull.

Output:

3
2
1
[3, 2, 1]

Analysis. countdown looks like a normal function but yield makes it a generator. Iterating it runs the body until a yield, hands back that value, and freezes the function's state; the next pull resumes right after the yield. So it produces 3, 2, 1 lazily, and list() collects them all.

Intuition. Mechanism. Calling a generator function doesn't run its body — it returns a generator object (an iterator). The body runs only as items are pulled, pausing at each yield with all local state intact and resuming on the next next().

Concrete bite. That "runs only when pulled" is visible — the body doesn't execute at call time:

created
running the body
1

gen() printed nothing — it just built the generator. Only next(g) ran the body (running the body) up to the yield. The "created" line printing before "running the body" proves the laziness.

4. Laziness and infinite streams

Because a generator produces items only on demand, it can represent a stream with no end — something impossible for a list.

Output:

[1, 2, 3, 4, 5]

Analysis. naturals() is an infinite generator — while True never stops yielding. That's fine because nothing is computed until pulled: itertools.islice(..., 5) pulls exactly five values and stops. (Calling list(naturals()) with no limit would never return — it would try to build an infinite list. Don't run that.)

Intuition. Mechanism. Laziness means an infinite generator costs almost nothing until consumed — it holds only its current state, not its (unbounded) output. You bound it at consumption time with tools like itertools.islice or itertools.takewhile, not at definition time.

Concrete bite. The memory difference between eager and lazy is dramatic and measurable:

Output (exact sizes are CPython-specific):

800984 bytes (list)
192 bytes (generator)

The list holds 100,000 integers — ~800 KB. The generator holds ~192 bytes regardless of how many values it will eventually produce, because it stores its recipe, not its results.

5. Generator pipelines

Generators compose. One can consume another, building a pipeline where data flows through stages, one item at a time, computed only when the end pulls.

Output:

pipeline built
squaring 1
1
squaring 2
4

Analysis. Building pipe printed nothing — squares hasn't run. "pipeline built" prints first; only the first next(pipe) runs the body far enough to yield the first square (squaring 1, then 1). The second next resumes and produces the next (squaring 2, then 4). Work happens per pull, never ahead of demand.

Intuition. Mechanism. Each generator in a chain pulls from the one before it on demand, so the whole pipeline is lazy end-to-end: nothing flows until the final consumer asks, and then exactly one item threads through all stages. No stage builds an intermediate list.

Concrete bite. The interleaved output is the proof: squaring 1 appears between the two print(next(...)) calls, not all at once up front. If squares eagerly built a list, you'd see squaring 1, squaring 2, squaring 3 immediately at construction — instead you see each only when pulled. Demand drives computation.

6. Mental-model summary

7. Gotcha checklist

Your Turn

Before you move on, check your understanding with the coach — explain the idea, apply it, weigh the trade-offs, then defend your reasoning.