Python generators are implemented as suspended frame objects (PyFrameObject) that maintain their execution state between invocations. When send(value) is called, CPython's internal gen_send_ex() function pushes this value onto the generator's value stack, which the yield expression then pops and returns to the caller. This differs from the initial next() call, which implicitly sends None to prime the generator from its initial state (where f_lasti == -1) to the first yield expression. If send() is called with a non-None value before the generator has yielded for the first time, CPython raises a TypeError because the generator frame lacks a stack position to receive the value. This architectural distinction ensures that bidirectional communication only begins after the generator has reached its first suspension point.
We needed to implement a backpressure-aware data pipeline for processing high-frequency market data feeds, where downstream consumers could dynamically signal upstream producers to throttle or resume data flow without dropping messages or exhausting memory.
One considered approach used threading with bounded queue.Queue instances between pipeline stages. While this provided familiar blocking semantics and thread safety, it suffered from severe GIL contention and context-switching overhead, consuming 15% CPU purely for coordination at high throughput while adding unpredictable latency spikes.
Another alternative involved migrating to asyncio coroutines and async/await syntax. This would have eliminated GIL contention but required a complete rewrite of our synchronous numerical analytics library into async-compatible form, creating a viral refactoring that would touch thousands of lines of business logic and introduce compatibility issues with legacy C extensions.
We ultimately selected a generator-based cooperative multitasking approach using send() to transmit "demand credits" upstream. This solution avoided GIL overhead entirely, required no library rewrites since generators work in synchronous code, and provided explicit control flow through demand = (yield data_chunk) patterns that allowed downstream consumers to pause upstream production immediately by sending zero values.
The result was a 40% reduction in memory usage compared to the queue approach, latency stabilized below 5 milliseconds, and the codebase remained readable with explicit yield points marking suspension boundaries.
Why does calling send() with a non-None value on a newly created generator raise TypeError, and how does this constraint enforce the generator protocol?
When a generator is first created, its frame pointer f_lasti is -1, indicating no bytecode has executed. The CPython interpreter checks if the generator is unstarted when send() is invoked; if the sent value is not None, it raises TypeError because the yield expression hasn't been reached yet to provide a stack slot for the value. This enforcement ensures that the generator initialization logic runs to completion before bidirectional communication begins, maintaining the invariant that values flow into the generator only at explicit yield suspension points.
How does generator.close() ensure that cleanup code within the generator executes, and what distinguishes the GeneratorExit exception from regular exceptions?
The close() method sends a GeneratorExit exception into the generator at its current suspension point by calling throw(GeneratorExit). GeneratorExit inherits from BaseException rather than Exception to prevent it from being caught by generic except Exception handlers that might swallow it improperly. If the generator catches GeneratorExit and re-raises it or exits normally, close() returns silently; however, if the generator yields a value in response to GeneratorExit, CPython raises RuntimeError because a closing generator must not produce new values. This mechanism guarantees that finally blocks and context managers within the generator body execute even during forced termination.
What mechanism allows yield from to handle sent values transparently across nested generators, and how does this differ from manual delegation using a loop with send()?
The yield from syntax delegates not just iteration but the full generator protocol to a sub-generator. When the outer generator executes yield from subgen(), CPython transforms the caller's send(value) into a direct send to the sub-generator until it raises StopIteration (whose value becomes the result of the yield from expression). This differs from manual delegation where a loop like for x in subgen(): yield x cannot intercept values sent into the outer generator to forward them to the inner one. The yield from construct essentially flattens the call stack, allowing bidirectional data flow through arbitrarily deep generator nesting without boilerplate forwarding code, while maintaining proper exception propagation and closing semantics.