ignore lines inside {}

Author: methane

peps/pep-0822.rst

@@ -155,32 +155,40 @@ Therefore, escape sequences such as ``\x20`` and line-continuation
 backslashes are not treated as whitespace, and you cannot use ``\t`` in
 indentations.
 
-For ``df`` and ``dt`` strings, physical lines of the literal include lines
-inside replacement fields (``{...}``), including lines inside nested string
-literals and multiline format specs.
-Therefore, such lines also affect the longest common indentation of the outer
-literal.
-
-Although lines inside replacement fields participate in calculating the
-longest common indentation, the outer d-string does not remove indentation
-from the contents of those replacement fields. Dedentation is applied only to
-the constant text parts of the outer ``df`` or ``dt`` string. In particular,
-the following retain their source indentation:
+For ``df`` and ``dt`` strings, a physical line participates in the longest
+common indentation calculation only if it starts while the tokenizer is in a
+constant text part of that d-string. A line that starts inside a replacement
+field (``{...}``) does not participate in the calculation, even if the
+replacement field ends and constant text resumes later on the same line.
+This includes lines that start inside nested string literals and multiline
+format specs in a replacement field.
+
+A line that starts in a constant text part still participates when its first
+non-whitespace character opens a replacement field. Its leading whitespace is
+part of the outer d-string and constrains how much indentation can be removed
+from that line.
+
+Dedentation is applied only to the constant text parts of the outer ``df`` or
+``dt`` string. Text inside replacement fields retains its source indentation.
+In particular, the following are not dedented:
 
 * The raw source text of ``df`` debug expressions (``{expr=}``).
 * The captured expression text of ``dt`` interpolations.
 * Multiline string literals in replacement expressions.
 * Multiline format specs.
 
-This distinction lets replacement-field lines constrain how much indentation
-is removed from the surrounding constant text without rewriting Python source
-or format-spec text inside the field.
+Ignoring lines that start inside replacement fields prevents the formatting of
+an expression from changing the value of the surrounding constant text. It
+also ensures that leading whitespace which affects the resulting value is
+represented in the string-part tokens rather than in otherwise insignificant
+spacing between expression tokens.
 
 If a replacement field expression itself contains another d-string literal,
 the nested d-string is supported and is dedented independently according to
-its own physical lines. Those same physical lines also participate in the
-outer d-string's longest-common-indentation calculation. The outer d-string
-does not otherwise dedent the nested literal's contents.
+its own physical lines. Those same physical lines do not participate in the
+outer d-string's longest-common-indentation calculation because they start
+inside its replacement field. The outer d-string does not otherwise dedent the
+nested literal's contents.
 
 
 Examples
@@ -293,8 +301,8 @@ In the following examples, spaces are visualized as ``.`` and tabs as ``--->``.
    print(s.strings) # ('Hello,.', '!\n')
    print(s.values)  # ('World',)
 
-   # In df/dt strings, lines inside replacement fields participate in
-   # the outer longest common indentation calculation.
+   # A line that starts outside a replacement field participates in the outer
+   # longest common indentation calculation, even when it contains a field.
    s = df"""
    ..Hello
    ....{42}
@@ -303,23 +311,33 @@ In the following examples, spaces are visualized as ``.`` and tabs as ``--->``.
 
    s = df"""
    ..Hello
-   .{42}
+   .{42} World
    .."""
-   print(repr(s))  # '.Hello\n42\n'
+   print(repr(s))  # '.Hello\n42 World\n'
 
    s = df"""
    ....Hello {1 +
    ...2}
    ...."""
-   print(repr(s))  # '.Hello 3\n'
+   print(repr(s))  # 'Hello 3\n'
 
    # The source text of a multiline debug expression is not dedented,
-   # although its physical lines affect the outer common indentation.
+   # and a line that starts inside the expression does not affect the outer
+   # common indentation.
    s = df"""
    ....{1 +
    ..1=}
    ...."""
-   print(repr(s))  # '..1 +\n..1=2\n'
+   print(repr(s))  # '1 +\n..1=2\n'
+
+   # A line that starts inside a replacement field is ignored even when the
+   # field ends and constant text resumes on that line.
+   s = df"""
+   ....hello {
+   ........1
+   } world
+   ...."""
+   print(repr(s))  # 'hello 1 world\n'
 
    # A nested d-string in a replacement field is dedented independently.
    s = df"""
@@ -331,16 +349,15 @@ In the following examples, spaces are visualized as ``.`` and tabs as ``--->``.
    .."""
    print(repr(s))  # 'outer\n..nested\nline\n\n'
 
-   # Lines in a nested d-string also constrain the outer indentation.
-   # Here, the unindented "bar" and "baz" lines make the outer common
-   # indentation empty, while the nested d-string is evaluated independently.
+   # Lines in a nested d-string do not constrain the outer indentation. The
+   # nested d-string is evaluated independently.
    s = df"""
    ....foo {d"""
    bar
    baz
    ........"""} spam
    ...."""
-   print(repr(s))  # '....foo bar\nbaz\n spam\n'
+   print(repr(s))  # 'foo bar\nbaz\n spam\n'
 
 
 How to Teach This
@@ -763,6 +780,25 @@ It makes the dedent rules simpler, and the difference from
 specification to users.
 
 
+Dedent inside replacement fields
+--------------------------------
+
+Dedenting expressions inside replacement fields provides too little benefit to
+justify the added parser complexity.
+Considering not only the complexity of Python's own parser but also the ease
+of implementing tools that analyze Python source code, it makes more sense to
+dedent only the constant parts.
+
+For example, the dedent amount of a df-string is the same as that of the
+df-string obtained by replacing all replacement fields with ``{""}``.
+This makes it easier for tools such as :func:`tokenize.tokenize` to calculate
+the dedent amount when they encounter a df-string.
+
+Similarly, in Ruby's ``<~`` heredocs and in Julia's and CoffeeScript's
+``"""`` strings, the contents of replacement fields are not dedented and do
+not contribute to the dedent amount.
+
+
 Copyright
 =========