PEP 828: Switch to yield from as the choice of syntax (#5047)

Author: ZeroIntensity

peps/pep-0828.rst

@@ -15,8 +15,7 @@ Abstract
 ========
 
 This PEP introduces support for :keyword:`yield from <yield>` in an
-:ref:`asynchronous generator function <asynchronous-generator-functions>`
-through a new ``async yield from`` construct:
+:ref:`asynchronous generator function <asynchronous-generator-functions>`:
 
 .. code-block:: python
 
@@ -26,7 +25,7 @@ through a new ``async yield from`` construct:
         return 3
 
     async def main():
-        result = async yield from agenerator()
+        result = yield from agenerator()
         assert result == 3
 
 
@@ -39,8 +38,7 @@ This is not to be confused with an :term:`asynchronous generator iterator`,
 which is the object *returned* by an asynchronous generator.
 
 This PEP also uses the term "subgenerator" to refer to a generator, synchronous
-or asynchronous, that is used inside of a ``yield from`` or ``async yield from``
-expression.
+or asynchronous, that is used inside a ``yield from``.
 
 
 Motivation
@@ -104,38 +102,18 @@ item. This comes with a few drawbacks:
 Specification
 =============
 
-
-Syntax
-------
-
-
 Compiler changes
-^^^^^^^^^^^^^^^^
+----------------
 
 The compiler will no longer emit a :exc:`SyntaxError` for
-:keyword:`return` statements inside asynchronous generators.
-
-
-Grammar changes
-^^^^^^^^^^^^^^^
-
-The ``yield_expr`` and ``simple_stmt`` rules need to be updated for the new
-``async yield from`` syntax:
-
-.. code-block:: peg
-
-    yield_expr[expr_ty]:
-        | 'async' 'yield' 'from' a=expression
-
-    simple_stmt[stmt_ty] (memo):
-        | &('yield' | 'async') yield_stmt
+:keyword:`return` or ``yield from`` statements inside asynchronous generators.
 
 
 Changes to ``StopAsyncIteration``
 ---------------------------------
 
 The :class:`StopAsyncIteration` exception will gain a new ``value`` attribute
-to be used as the result of ``async yield from`` expressions.
+to be used as the result of ``yield from`` expressions in asynchronous generators.
 
 This attribute can be supplied by passing a positional argument to
 ``StopAsyncIteration``. For example:
@@ -156,18 +134,18 @@ If no argument is supplied, ``value`` will be ``None``.
 In the body of an asynchronous generator function, the statement
 ``return expression`` is roughly equivalent to
 ``raise StopAsyncIteration(expression)``. However, similar to implicit
-``StopIteration`` exceptions raised inside of synchronous generators,
+``StopIteration`` exceptions raised inside synchronous generators,
 the exception cannot be caught in the body of the asynchronous generator.
 
 
-``async yield from`` semantics
-------------------------------
+``yield from`` semantics in an asynchronous generator
+-----------------------------------------------------
 
-The statement
+In an asynchronous generator, the statement
 
 .. code-block:: python
 
-   RESULT = async yield from EXPR
+   RESULT = yield from EXPR
 
 is roughly equivalent to the following:
 
@@ -216,26 +194,24 @@ Rationale
 =========
 
 
-Relation to ``yield from``
---------------------------
+Relation to synchronous generators
+----------------------------------
 
-This PEP aims to be very similar to the semantics of ``yield from``, with the
-exception that asynchronous generator methods are used instead of synchronous
+This PEP aims to be very similar to the semantics of synchronous ``yield from``,
+with the exception that asynchronous generator methods are used instead of synchronous
 generator methods when delegating. This is a very intuitive design and furthers
 symmetry with synchronous generators.
 
 
-Choice of ``async yield from`` as the syntax
---------------------------------------------
+Choice of ``yield from`` as the syntax
+--------------------------------------
 
-This PEP uses ``async yield from`` as the syntax to ensure that the behavior
-of the syntax is immediately clear to the user.
-
-However, it is acknowledged that this is somewhat verbose. There is not any
-great solution to this problem; see :ref:`pep-828-rejected-ideas` for
-discussion about proposed alternatives. In short, ``async yield from`` was
-chosen as the best choice of syntax because, while verbose, it is very clear
-and readable.
+This PEP uses ``yield from`` as the syntax, but it is acknowledged that
+this is somewhat controversial, as this is an asynchronous context switch
+without an explicit ``async`` or ``await`` marker; see
+:ref:`pep-828-rejected-ideas`. In short, ``yield from`` was chosen
+as the best choice of syntax because it is symmetric with synchronous
+generators and easy to type.
 
 
 Backwards Compatibility
@@ -244,7 +220,7 @@ Backwards Compatibility
 This PEP introduces a backwards-compatible syntax change.
 
 The addition of the ``value`` attribute to :exc:`StopAsyncIteration` is a
-minor semantic change to an existing builtin exception, but is unlikely
+minor semantic change to an existing built-in exception, but is unlikely
 to affect existing code in practice, as it mirrors the existing ``value``
 attribute on :exc:`StopIteration` and does not affect any other behavior
 on ``StopAsyncIteration`` or the asynchronous iterator protocol.
@@ -262,10 +238,8 @@ How to Teach This
 The details of this proposal will be located in Python's canonical
 documentation, as with all other language constructs. However, this PEP
 intends to be very intuitive; users should be able to naturally reach
-for ``async yield from`` given their own background knowledge about generators
-in Python. This can be encouraged further by suggesting
-``async yield from`` in the error message when a user attempts to use
-``yield from`` in an asynchronous generator.
+for ``yield from`` given their own background knowledge about generators
+in Python.
 
 
 Reference Implementation
@@ -281,74 +255,74 @@ Rejected Ideas
 ==============
 
 
-Using ``yield from`` to delegate to asynchronous generators
------------------------------------------------------------
+Using ``async yield from`` as the syntax
+----------------------------------------
 
-Due to the verbosity of ``async yield from``, it was proposed to overload
-the existing ``yield from`` syntax to perform asynchronous subgenerator
-delegation when used inside of an asynchronous generator.
+There are two primary counterarguments against this proposal as it is currently
+written:
 
-For example:
+1. It adds behavior that does asynchronous work without having ``async`` or
+   ``await`` as its first non-whitespace token.
+2. It changes the meaning of ``yield from`` depending on the type of generator
+   it is used in.
 
-.. code-block:: python
+The solution to these issues was to introduce a new ``async yield from``
+statement that would perform asynchronous subgenerator delegation. However,
+this new syntax came with its own set of problems.
 
-    async def asubgenerator():
-        yield 1
-        yield 2
+First and foremost, ``async yield from`` is very verbose. There are no other
+syntax constructs in Python that use three keywords in a row.
 
-    async def agenerator():
-        yield from asubgenerator()
+Second, there is no ``async yield`` as a counterpart; CPython already emits
+significantly different bytecode for ``yield`` statements inside asynchronous
+generators, so it's not far-fetched for ``yield from`` to do the same.
 
+Third, ``yield`` (as well as ``yield from``) is, technically speaking, already
+an asynchronous context switch, because the generator will be suspended and can
+only be resumed via an ``await`` (on :meth:`~agen.asend`) from the caller.
 
-This has the benefit of being more concise than ``async yield from``, but also
-has a few downsides.
+Finally, plain ``yield from`` is more symmetric and intuitive to users of
+Python. To visualize, take these two examples:
 
-Most importantly, this makes the asynchronous context switches necessary for
-delegation implicit, which has no precedent in Python; all syntax that may
-execute an ``await`` is prefixed with ``async``. It has been argued that one
-of the upsides of ``async``/``await`` over threads is the explicit switch
-points, so hiding ``await``\ s behind a ``yield from`` hurts this benefit.
+.. code-block:: python
 
-Second, many were uncomfortable with ``yield from`` being context-dependent.
-It felt like a potential footgun for ``yield from`` to mean something
-different based on the type of generator it was used in. In practice, this
-may come up in a scenario where one wants to convert a synchronous generator
-into an asynchronous generator.
+    def gen():
+        yield 1
 
-For example, imagine a developer is writing a function for streaming data
-to the caller:
+    async def agen():
+        yield 1
 
 .. code-block:: python
 
-    def stream_data():
-        yield ...
-        yield from something_else()
-        yield ...
+    def gen():
+        yield from subgen()
 
-Now, imagine that the developer wants to add an ``await`` call somewhere in
-this function; the ``yield from something_else()`` statement would suddenly
-become a runtime :class:`TypeError` (as opposed to a compile-time
-:class:`SyntaxError`). With the current proposal, the existence of
-``async yield from`` (which would ideally be included in the error message)
-would make it much clearer that ``something_else`` must also be asynchronous
-in order to delegate to it.
+    async def agen():
+        yield from asubgen()
+
+
+``async yield from`` would break this symmetry.
 
-Finally, this would preclude the introduction of support for synchronous
-subgenerator delegation inside asynchronous generators (see
-:ref:`pep-828-synchronous-delegation`), because the ``yield from``
-syntax would already be overloaded. However, the author of this proposal
-does acknowledge that the issues with synchronous subdelegation may preclude
-the introduction of this anyway -- it is not entirely clear whether the issues
-are solvable given time.
+However, all in all, the difference between ``yield from`` and
+``async yield from`` is primarily up to the taste of the reader. Even the
+author of this PEP is not completely convinced for one way or the other.
+During discussion, the difference between ``yield from`` and ``async yield from``
+was compared to `deciding what 0^0 should be in mathematics
+<https://en.wikipedia.org/wiki/Zero_to_the_power_of_zero>`_; it depends on the
+context and the point of view. The alternative was to not add support for
+delegation to asynchronous subgenerators, which was a lose-lose scenario for
+all parties involved. A decision had to be made, and it was clear that consensus
+was never going to be reached.
 
 
 ``async from``, ``await from``, and similar spellings
------------------------------------------------------
+*****************************************************
 
-As an alternate solution to the verbosity of ``async yield from``, some have
+As an alternative solution to the verbosity of ``async yield from``, some have
 suggested using spellings such as ``async from`` in order to cut down on the
 verbosity. Unfortunately, changes in the spelling will likely hurt the
-readability of the syntax as a whole.
+readability of the syntax as a whole, and thus would not improve the argument
+for ``async yield from``.
 
 The benefit of ``async yield from`` is that it specifies each of the three
 important parts without introducing new keywords. In particular:
@@ -367,10 +341,10 @@ exists.
 Allowing delegation to synchronous subgenerators
 ------------------------------------------------
 
-In an earlier revision of this proposal, the synchronous ``yield from``
-construct was allowed in an asynchronous generator, which would delegate to a
-synchronous generator from an asynchronous one. This had a number of hidden
-issues.
+In an earlier revision of this proposal, ``yield from`` in an asynchronous
+generator would delegate to a synchronous generator (and ``async yield from``
+was the counterpart for asynchronous subgenerator delegation).
+This had a number of hidden issues.
 
 In particular, the mixing of asynchronous frames with synchronous frames had a
 layer of complexity unfit for Python. In the implementation, there would have
@@ -379,7 +353,7 @@ asynchronous generator methods: :meth:`~agen.asend` to :meth:`~generator.send`,
 :meth:`~agen.athrow` to :meth:`~generator.throw`, and :meth:`~agen.aclose`
 to :meth:`~generator.close`.
 
-It's trivial for anyone that needs to delegate to a subgenerator to write
+It's trivial for anyone who needs to delegate to a subgenerator to write
 the wrapper class to upgrade a synchronous :class:`~collections.abc.Iterable`
 or :class:`~collections.abc.Generator` to an
 async one before calling ``async yield from``.
@@ -427,8 +401,8 @@ async one before calling ``async yield from``.
 
 
     async def agen():
-        async yield from AsAsyncIterator([1, 2, 3])
-        async yield from AsAsyncGenerator(subgen())
+        yield from AsAsyncIterator([1, 2, 3])
+        yield from AsAsyncGenerator(subgen())
 
 
 To quote Brandt Bucher (paraphrased):
@@ -457,6 +431,11 @@ collecting outside opinions about the design and implementation.
 Change History
 ==============
 
+- 18-Jul-2026
+
+    - Switched from ``async yield from`` to ``yield from`` as the choice of
+      syntax.
+
 - 26-May-2026
 
     - Removed support for delegating to a synchronous subgenerator (via