Generators: stream, don't load

The most common way data code dies is not a bug in the logic. It is running out of memory. Someone writes rows = f.readlines(), it works on the 10 MB sample, and then it meets the 10 GB file.

The fix is a different shape of code: process one row at a time and never hold them all. Python has a built-in tool for that shape. It is called a generator.

Where the memory goes

Start with what a list costs. From the last lesson: a list holds an 8-byte pointer per item, plus the objects themselves. Build a list of a million squares and all million exist at once.

A generator does not build anything up front. It holds the recipe and the current position, and produces the next value only when asked.

import sys

n = 1_000_000
as_list = [i * i for i in range(n)]      # square brackets: build it all now
as_gen  = (i * i for i in range(n))      # round brackets: build on demand

print("list object:     ", f"{sys.getsizeof(as_list):>12,}", "bytes")
print("generator object:", f"{sys.getsizeof(as_gen):>12,}", "bytes")
print("same total?      ", sum(as_list) == sum(as_gen))

About 8 MB for the list’s pointer array alone, and that is before counting the million int objects it points to. The generator is around 200 bytes, and it stays that size at a billion items. Both give the same sum.

What a generator is

A function with yield in it is a generator function. Calling it does not run the body. It returns a generator object, paused at the top. Each time you ask for the next value, with next() or a for loop, the body runs until the next yield, hands back that value, and pauses again. Local variables survive the pause.

So the work is pulled, one item at a time, by whoever is consuming it. That is what makes pipelines possible.

Files already stream

You have been using a generator-like object all along. A file opened in Python is an iterator over lines. Underneath, it does not read one line from the disk at a time, which would be slow. It reads a buffer, typically 8 KB, splits out the lines in it, and hands them to you one by one. When the buffer runs out, it reads the next 8 KB.

So for line in f: costs one buffer of memory, whatever the file size. f.readlines() and f.read() throw that away and pull the whole file into memory at once. The streaming version was already there. Most memory bugs come from reaching past it.

A pipeline of generators

Here is the shape you will use for real files: one small generator per step, each pulling from the one before.

import io

RAW = """order_id,status,amount
1,delivered,120.50
2,shipped,35.00
3,delivered,80.25
4,cancelled,15.00
"""

def read_lines(f):
  for line in f:                       # a file object already streams lines
      print("  read  ", line.strip())
      yield line

def parse(lines):
  header = next(lines).strip().split(",")
  for line in lines:
      row = dict(zip(header, line.strip().split(",")))
      print("  parse ", row["order_id"])
      yield row

def only_delivered(rows):
  for row in rows:
      if row["status"] == "delivered":
          yield row

f = io.StringIO(RAW)                     # stands in for open("orders.csv")
for row in only_delivered(parse(read_lines(f))):
  print("USE", row["order_id"], row["amount"])

Read the output carefully. It is not “read everything, then parse everything”. It is read one, parse one, use one, read the next. Only one row is in flight at a time. Swap io.StringIO for open("orders.csv") and this handles a 50 GB file in the same few kilobytes.

This is the same model as the scan in the SQL track: the engine pulls rows through a chain of operators, scan into filter into project. Generators let you build that chain yourself.

Batches: streaming with fewer round trips

One row at a time is right for memory, but wrong for anything with a per-call cost: a database insert, an HTTP request, an embedding API. 10,000 rows as 10,000 inserts is slow. 10,000 rows as 10 inserts of 1,000 is fast.

So stream rows, but hand them off in fixed-size batches. itertools.islice takes the next n items from any iterator:

from itertools import islice

def batched(rows, size):
  it = iter(rows)
  while True:
      batch = list(islice(it, size))   # at most 'size' rows in memory
      if not batch:
          return
      yield batch

rows = ({"id": i} for i in range(1, 2_501))   # 2,500 rows, never all in memory
for n, batch in enumerate(batched(rows, 1_000), start=1):
  print(f"batch {n}: {len(batch)} rows, ids {batch[0]['id']}..{batch[-1]['id']}")

Three batches: 1,000, 1,000 and 500. Peak memory is one batch, whatever the input size. Python 3.12 added itertools.batched, which does the same job and returns tuples.

When not to stream

Some operations need every row before they can answer:

• Sorting. The smallest item might be the last one.
• Exact distinct counts. You must remember every value you have seen.
• Grouping by a key with many values. The dict grows with the number of keys.

For those, you either accept the memory, or you push the work into a tool built to spill to disk, like DuckDB. Streaming still helps on the way in: filter and drop columns while streaming, and the sort sees far less data.

Ship it

Build a three-stage pipeline in one file, stream.py:

1. read_lines(path) yields lines from a CSV file.
2. parse(lines) yields one dict per row, using the header.
3. batched(rows, 500) yields lists of at most 500 rows.

At the end, loop over the batches and print the number of batches and the total row count. Test it on content/datasets/ridelog/events.csv from this repo. It has 3,146 rows, so you should see 7 batches.

You have shipped it when no line of your code ever holds the whole file. Search your file for readlines, read() and list( around the file object. None should be there.