The Standard-Library Tour — collections, itertools, functools

Python's slogan is "batteries included," and the batteries that pay off most for everyday code live in three modules. The thesis: collections, itertools, and functools give you specialised containers and composable building blocks that replace whole patterns of hand-written code — often turning a ten-line loop into a one-liner that's faster and clearer. Knowing they exist is half the battle; this chapter is a tour of the highest-leverage tools and the one gotcha each carries.

This builds on dicts & sets, iterators, and decorators. Every output below was produced by running the code.

Table of Contents

1. Counter and defaultdict
2. deque
3. itertools
4. functools
5. namedtuple
6. Mental-model summary
7. Gotcha checklist

1. Counter and defaultdict

collections.Counter tallies occurrences; defaultdict supplies a default for missing keys. Both eliminate the boilerplate around the counting idiom from Tier 2.

Output:

Counter({'a': 3, 'n': 2, 'b': 1})
[('a', 3)]
{'a': [1, 2]}

Analysis. Counter("banana") tallied every character in one call — replacing the whole d.get(ch, 0) + 1 loop — and .most_common(1) returned the top entry. defaultdict(list) let d["a"].append(1) work even though "a" didn't exist yet: the first access auto-created an empty list. Both are dict subclasses, so everything you know about dicts still applies.

Intuition. Mechanism. Counter is a dict mapping items to counts, built in C; its repr orders by descending count. defaultdict(factory) overrides the missing-key behaviour: on a miss it calls factory() (here list()), inserts the result, and returns it — so the key now exists.

Concrete bite. That auto-creation has a flip side worth knowing: a Counter returns 0 for a missing key without inserting it, but merely reading a missing key from a defaultdict creates it. With Counter:

0

c["z"] is 0 — convenient, and (unlike defaultdict) it doesn't add "z" to the counter. The trap is the reverse: some_defaultdict[missing] silently grows the dict, so checking membership with [] on a defaultdict pollutes it.

2. deque

collections.deque ("deck") is a double-ended queue: O(1) append and pop at both ends, unlike a list, where front operations are O(n) (Tutorial 12).

Output:

deque([0, 1, 2, 3, 4])
0
deque([1, 2, 3, 4])

Analysis. appendleft/popleft add and remove at the front in O(1) — the operations a list does in O(n) because it must shift every element. deque is the right structure for queues, sliding windows, and breadth-first search, where you push and pop at the ends.

Intuition. Mechanism. A deque is a doubly-linked structure of blocks, so both ends are cheap; the trade is that random access by index is O(n) (it has to walk), whereas a list indexes in O(1). Different structure, different fast operations.

Concrete bite. A handy feature with a sharp edge: a maxlen deque silently drops items from the far end when it overflows:

deque([2, 3], maxlen=2)
deque([3, 4], maxlen=2)

Creating it with maxlen=2 immediately discarded the oldest (1 → deque([2, 3])), and each append drops another from the left (deque([3, 4])). Perfect for "last N items," but if you didn't expect the dropping, data vanishes silently.

3. itertools

itertools provides lazy, composable iterators (Tutorial 17) for chaining, slicing, and combinatorics — building blocks you combine instead of writing loops.

Output:

[1, 2, 3, 4]
[10, 11, 12]
[('a', 'b'), ('a', 'c'), ('b', 'c')]
['a2', 'b3', 'c1']

Analysis. chain flattens several iterables into one stream; islice takes a slice of any iterator (here the first 3 of the infinite count(10)); combinations yields all 2-element combos; groupby groups consecutive equal items. All are lazy — they yield on demand — which is why islice can safely slice an infinite count.

Intuition. Mechanism. These return iterators, computing each item only when pulled. groupby in particular scans linearly and starts a new group whenever the key changes — it groups consecutive runs, not all equal items globally.

Concrete bite. That "consecutive" rule is the classic groupby trap — unsorted input produces split groups:

a 2
b 2
a 1

The string "aabba" has three as total, but groupby reports two a-groups (a 2 and a later a 1) because the as aren't all adjacent — the bb splits them. To group globally, sort first (groupby(sorted(data))).

4. functools

functools operates on functions. The headliners: reduce (fold a sequence to one value), lru_cache (memoise — a decorator), and partial (pre-fill arguments).

Output:

24
832040
CacheInfo(hits=28, misses=31, maxsize=128, currsize=31)
20

Analysis. reduce folded [1,2,3,4] with * to 24. @lru_cache made the naively-exponential fib linear by caching results — cache_info() shows 31 unique computations (misses) and 28 cache hits, so each fib(n) was computed once. partial(f, 2) produced a new function with the first argument fixed at 2, so double(10) is 2 * 10.

Intuition. Mechanism. reduce(f, seq) applies f cumulatively: f(f(f(1,2),3),4). lru_cache wraps the function in a memo dict keyed by the arguments (which must be hashable). partial returns a new callable that remembers some arguments.

Concrete bite. reduce on an empty sequence with no initial value has nothing to return, so it raises:

Traceback (most recent call last):
  File "/w/main.py", line 2, in <module>
    print(functools.reduce(lambda a, b: a + b, []))
          ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: reduce() of empty iterable with no initial value

With no items and no seed, there's no value to produce — TypeError. Always pass an initial value (reduce(f, seq, 0)) when the sequence might be empty.

5. namedtuple

collections.namedtuple builds a lightweight, immutable class with named fields — a tuple you can read by name, ideal for small records before reaching for a full class or dataclass.

Output:

1 2
Point(x=1, y=2)
1
1 2

Analysis. Point is a new type with fields x and y. An instance reads by name (p.x), prints with a clear repr (Point(x=1, y=2)), and — because it is a tuple — also indexes (p[0]) and unpacks (x, y = p). You get readability and tuple behaviour for almost no code.

Intuition. Mechanism. namedtuple generates a subclass of tuple with property accessors for the field names. So it has all tuple powers (indexing, unpacking, hashable, immutable) plus name access — and the immutability comes straight from tuple.

Concrete bite. Being a tuple, it's immutable — you cannot assign to a field:

Traceback (most recent call last):
  File "/w/main.py", line 4, in <module>
    p.x = 2   # namedtuples are immutable
    ^^^
AttributeError: can't set attribute

p.x = 2 raises AttributeError — a namedtuple is read-only, like the tuple it is. To "change" one, build a new instance (p._replace(x=2)).

6. Mental-model summary

7. Gotcha checklist

Your Turn

Before you move on, check your understanding with the coach — explain the idea, apply it, weigh the trade-offs, then defend your reasoning.