The evolution from generator-based coroutines to the modern async/await syntax represents a significant paradigm shift in Python’s approach to asynchronous programming. While generators provided a clever and powerful foundation, they were ultimately a repurposing of a feature designed for iteration, not a purpose-built solution for concurrency. The async/await syntax, introduced in Python 3.5, offered a cleaner, more explicit, and less error-prone model.

The Generator-Based Coroutine Pattern

Before async/await, coroutines were implemented using generators. A function containing a yield expression could be paused, and its execution could be controlled via .send(), .throw(), and .close() methods. This made it possible to build event loops that managed many such generators concurrently.

def old_style_coroutine():
    """A generator-based coroutine."""
    print("Coroutine started")
    # Yield point where execution pauses.
    # The value sent in via .send() is assigned to 'received'
    received = yield 'Please send something'
    print(f"Coroutine received: {received}")
    yield 'Goodbye'

# Using the coroutine
coro = old_style_coroutine()

# Prime the coroutine: advance execution to the first yield
first_yield_value = next(coro)
print(f"First yield: {first_yield_value}")

# Send a value into the paused coroutine, advancing it to the next yield
try:
    second_yield_value = coro.send("Hello from caller")
    print(f"Second yield: {second_yield_value}")
except StopIteration:
    print("Coroutine finished")

The key to understanding this pattern is the initial “priming” step with next(). This advances the generator from its initial state to the first yield expression, where it pauses and yields control back to the caller. Only after this priming can .send() be used. Forgetting to prime the coroutine is a common pitfall, resulting in a TypeError because you’re trying to send a value to a generator that hasn’t started.

Limitations and the Need for a New Syntax

While ingenious, this pattern had several drawbacks:

  1. Ambiguity: It was impossible to tell by looking at a function definition if it was intended to be a generator for iteration or a coroutine for concurrency. This required careful naming conventions (e.g., prefixing with co_).
  2. Error-Prone: The requirement to prime the coroutine was a frequent source of bugs.
  3. Incompatibility with Future Patterns: Mixing generator-coroutines with other awaitable objects (like Futures) was clumsy. The yield from syntax introduced in PEP 380 helped delegate to sub-generators but was still syntactically distinct from what was needed for a robust concurrency model.
  4. No Native Support: The event loop and the entire asynchronous ecosystem had to be built around this generator hack, not a language-supported feature.

The Introduction of Native Coroutines

Python 3.5 addressed these issues head-on with the async and await keywords. An async def function defines a native coroutine. This is a fundamental change: a native coroutine is not a generator.

import asyncio

# A native coroutine
async def simple_native_coroutine():
    print("Native coroutine started")
    # Use 'await' to pause execution until the awaitable completes
    await asyncio.sleep(1)
    print("Native coroutine resumed")
    return "Done"

# Running the native coroutine
result = asyncio.run(simple_native_coroutine())
print(f"Result: {result}")

The differences are critical:

  • Explicit Definition: async def immediately signals the function’s purpose as a coroutine.
  • No Priming Needed: The event loop (or asyncio.run()) knows how to properly initialize and drive a native coroutine.
  • await replaces yield: The await keyword is used to pause execution until an awaitable object (like another coroutine, a task, or a Future) completes. It is semantically clearer than yield.
  • return is Meaningful: Native coroutines can use return with a value, which is delivered via the StopIteration exception in the old model but is now properly propagated as the coroutine’s result.

Bridging the Gap: async generators and yield from

The transition wasn’t instantaneous. Legacy codebases were full of generator-based coroutines. To ensure interoperability during the transition, Python provided a bridge. A generator-based coroutine could be made compatible with the new event loop by decorating it with @asyncio.coroutine, and yield from was used to delegate to other generator-coroutines or Futures.

import asyncio

@asyncio.coroutine  # This decorator marks it as an "old-style" coroutine for asyncio
def old_style_with_asyncio():
    print("Old style coroutine in asyncio")
    # 'yield from' is used to await a Future (like asyncio.sleep)
    yield from asyncio.sleep(1)
    print("Done")
    return "Legacy Result"

# This still works, but the decorator is deprecated
asyncio.run(old_style_with_asyncio())

It’s crucial to note that @asyncio.coroutine and yield from are deprecated since Python 3.8 and removed in Python 3.11. They should not be used in new code. Their existence is a historical artifact of the transition. The yield keyword is still valid inside an async def function, but its use creates an asynchronous generator (using async for and async with), which is a distinct concept from a coroutine and is used for producing a sequence of values asynchronously.

Best Practices and Modern Usage

  1. Always Prefer Native Coroutines: For all new development, exclusively use async def and await. They are clearer, safer, and are the standard supported by all modern asyncio tools and libraries.
  2. Understand the Awaitable Chain: Any function that calls an await must itself be defined with async def. This creates a chain of coroutines that the event loop manages. You cannot await inside a regular function.
  3. Use the High-Level asyncio API: Prefer asyncio.run() to run your main coroutine instead of manually managing the event loop. It handles loop creation, execution, and cleanup correctly.
  4. Legacy Code Migration: When updating old code, replace @asyncio.coroutine with async def and replace yield from with await. This is typically a straightforward mechanical substitution.