gh-153569: move f-string state to source spans - #156484

Draft

pablogsal wants to merge 7 commits into

python:mainfrom

pablogsal:gh-153569-tokenizer-validation-tools-clean

Commits on Aug 28, 2026

gh-153569: bound tokenizer input storage
pablogsal
committed
600a445
gh-153569: return tokenizer tokens as source spans
pablogsal
committed
c10fd82
gh-153569: simplify tokenizer buffer relocation
pablogsal
committed
8c4469b
gh-153569: remove unused tokenizer cursor support
pablogsal
committed
19a959a
gh-153569: store f-string state as source spans
pablogsal
committed
9252554
gh-153569: track f-string comments as source spans
pablogsal
committed
e5f9a6c
gh-153569: simplify f-string span handling
pablogsal
committed
962c0c5