reduce the amount of panics in `{TokenStream, Literal}::from_str` calls by cyrgani · Pull Request #147859 · rust-lang/rust

cyrgani · 2025-10-18T20:13:31Z

Before this PR, calling TokenStream::from_str or Literal::from_str with an invalid argument would always cause a compile error, even if the TokenStream is not used afterwards at all.
This PR changes this so it returns a LexError instead in some cases.

This is very theoretically a breaking change, but the doc comment on the impl already says

/// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to
/// change these errors into `LexError`s later.

Fixes some cases of #58736.

rustbot · 2025-10-18T20:13:34Z

rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead.

cc @rust-lang/rust-analyzer

rustbot · 2025-10-18T20:13:36Z

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

cyrgani · 2025-10-18T21:47:52Z

@rustbot author

rustbot · 2025-10-18T21:47:56Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

cyrgani · 2025-10-19T10:12:43Z

@rustbot ready

tests/ui/proc-macro/invalid-punct-ident-4.rs

cyrgani · 2025-10-23T14:03:39Z

This does not fix #58736 entirely yet:

TokenStream::from_str("0b2");

is still a compile error because it internally calls literal_from_str, which then emits the diagnostic.

davidtwco · 2025-10-25T11:33:55Z

I'm not familiar enough with this part of the compiler

r? compiler

add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.

JonathanBrouwer · 2026-02-17T20:24:51Z

@craterbot check

craterbot · 2026-02-17T20:25:16Z

👌 Experiment pr-147859 created and queued.
🤖 Automatically detected try build 0269aec
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2026-02-19T16:16:01Z

🚧 Experiment pr-147859 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2026-02-21T11:54:41Z

🎉 Experiment pr-147859 is completed!
📊 1 regressed and 4 fixed (818761 total)
📊 2549 spurious results on the retry-regressed-list.txt, consider a retry¹ if this is a significant amount.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

re-run the experiment with crates=https://crater-reports.s3.amazonaws.com/pr-147859/retry-regressed-list.txt ↩

JonathanBrouwer · 2026-02-21T14:09:23Z

@joshtriplett Crater run has only spurious results

petrochenkov · 2026-02-22T08:03:29Z

Great.
@bors r+

rust-bors · 2026-02-22T08:03:32Z

📌 Commit f1c80f5 has been approved by petrochenkov

It is now in the queue for this repository.

JonathanBrouwer · 2026-02-22T08:07:15Z

@bors r=petrochenkov,JonathanBrouwer rollup

rust-bors · 2026-02-22T08:07:17Z

📌 Commit f1c80f5 has been approved by petrochenkov,JonathanBrouwer

It is now in the queue for this repository.

…, r=petrochenkov,JonathanBrouwer reduce the amount of panics in `{TokenStream, Literal}::from_str` calls *[View all comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/rust/pull/147859)* Before this PR, calling `TokenStream::from_str` or `Literal::from_str` with an invalid argument would always cause a compile error, even if the `TokenStream` is not used afterwards at all. This PR changes this so it returns a `LexError` instead in some cases. This is very theoretically a breaking change, but the doc comment on the impl already says ``` /// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to /// change these errors into `LexError`s later. ``` Fixes some cases of rust-lang#58736.

Rollup of 15 pull requests Successful merges: - #150468 (rustc_target: callconv: powerpc64: Use the ABI set in target options instead of guessing) - #151628 (Fix ICE in const eval of packed SIMD types with non-power-of-two element counts) - #151871 (don't use env with infer vars) - #152591 (Simplify internals of `{Rc,Arc}::default`) - #152865 (Fixed ByteStr not padding within its Display trait when no specific alignment is mentioned) - #152908 (Enable rust.remap-debuginfo in the dist profile) - #147859 (reduce the amount of panics in `{TokenStream, Literal}::from_str` calls) - #152705 (Test(lib/win/proc): Skip `raw_attributes` doctest under Win7) - #152767 (fix typo in `carryless_mul` macro invocation) - #152837 (fix(codegen): Use `body_codegen_attrs` For Caller In `adjust_target_feature_sig`) - #152871 (Fix warnings in rs{begin,end}.rs files) - #152879 (Remove `impl IntoQueryParam<P> for &'a P`.) - #152933 (Start migration for `LintDiagnostic` items by adding API and migrating `LinkerOutput` lint) - #152937 (remove unneeded reboxing) - #152953 (Fix typo in armv7a-vex-v5.md)

…, r=petrochenkov,JonathanBrouwer reduce the amount of panics in `{TokenStream, Literal}::from_str` calls *[View all comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/rust/pull/147859)* Before this PR, calling `TokenStream::from_str` or `Literal::from_str` with an invalid argument would always cause a compile error, even if the `TokenStream` is not used afterwards at all. This PR changes this so it returns a `LexError` instead in some cases. This is very theoretically a breaking change, but the doc comment on the impl already says ``` /// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to /// change these errors into `LexError`s later. ``` Fixes some cases of rust-lang#58736.

…uwer Rollup of 14 pull requests Successful merges: - #150468 (rustc_target: callconv: powerpc64: Use the ABI set in target options instead of guessing) - #151628 (Fix ICE in const eval of packed SIMD types with non-power-of-two element counts) - #151871 (don't use env with infer vars) - #152591 (Simplify internals of `{Rc,Arc}::default`) - #152865 (Fixed ByteStr not padding within its Display trait when no specific alignment is mentioned) - #147859 (reduce the amount of panics in `{TokenStream, Literal}::from_str` calls) - #152705 (Test(lib/win/proc): Skip `raw_attributes` doctest under Win7) - #152767 (fix typo in `carryless_mul` macro invocation) - #152837 (fix(codegen): Use `body_codegen_attrs` For Caller In `adjust_target_feature_sig`) - #152871 (Fix warnings in rs{begin,end}.rs files) - #152879 (Remove `impl IntoQueryParam<P> for &'a P`.) - #152933 (Start migration for `LintDiagnostic` items by adding API and migrating `LinkerOutput` lint) - #152937 (remove unneeded reboxing) - #152953 (Fix typo in armv7a-vex-v5.md)

Rollup merge of #147859 - cyrgani:nonfatal-tokenstream-parse, r=petrochenkov,JonathanBrouwer reduce the amount of panics in `{TokenStream, Literal}::from_str` calls *[View all comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/rust/pull/147859)* Before this PR, calling `TokenStream::from_str` or `Literal::from_str` with an invalid argument would always cause a compile error, even if the `TokenStream` is not used afterwards at all. This PR changes this so it returns a `LexError` instead in some cases. This is very theoretically a breaking change, but the doc comment on the impl already says ``` /// NOTE: some errors may cause panics instead of returning `LexError`. We reserve the right to /// change these errors into `LexError`s later. ``` Fixes some cases of #58736.

…uwer Rollup of 14 pull requests Successful merges: - rust-lang/rust#150468 (rustc_target: callconv: powerpc64: Use the ABI set in target options instead of guessing) - rust-lang/rust#151628 (Fix ICE in const eval of packed SIMD types with non-power-of-two element counts) - rust-lang/rust#151871 (don't use env with infer vars) - rust-lang/rust#152591 (Simplify internals of `{Rc,Arc}::default`) - rust-lang/rust#152865 (Fixed ByteStr not padding within its Display trait when no specific alignment is mentioned) - rust-lang/rust#147859 (reduce the amount of panics in `{TokenStream, Literal}::from_str` calls) - rust-lang/rust#152705 (Test(lib/win/proc): Skip `raw_attributes` doctest under Win7) - rust-lang/rust#152767 (fix typo in `carryless_mul` macro invocation) - rust-lang/rust#152837 (fix(codegen): Use `body_codegen_attrs` For Caller In `adjust_target_feature_sig`) - rust-lang/rust#152871 (Fix warnings in rs{begin,end}.rs files) - rust-lang/rust#152879 (Remove `impl IntoQueryParam<P> for &'a P`.) - rust-lang/rust#152933 (Start migration for `LintDiagnostic` items by adding API and migrating `LinkerOutput` lint) - rust-lang/rust#152937 (remove unneeded reboxing) - rust-lang/rust#152953 (Fix typo in armv7a-vex-v5.md)

rustbot assigned davidtwco Oct 18, 2025

This comment has been minimized.

Sign in to view

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 18, 2025

cyrgani force-pushed the nonfatal-tokenstream-parse branch from e2c0009 to 80b67bb Compare October 19, 2025 09:03

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 19, 2025

bjorn3 reviewed Oct 20, 2025

View reviewed changes

tests/ui/proc-macro/invalid-punct-ident-4.rs Show resolved Hide resolved

cyrgani force-pushed the nonfatal-tokenstream-parse branch from ba098bc to 0274d78 Compare October 21, 2025 14:28

rustbot assigned wesleywiser and unassigned davidtwco Oct 25, 2025

cyrgani force-pushed the nonfatal-tokenstream-parse branch from 0274d78 to 13d3d06 Compare October 29, 2025 22:19

This comment has been minimized.

Sign in to view

cyrgani changed the title ~~replace TokenStream::from_str panics with LexErrors~~ reduce the amount of panics in {TokenStream, Literal}::from_str calls Oct 29, 2025

cyrgani mentioned this pull request Nov 4, 2025

add larger test for proc_macro FromStr implementations #148505

Merged

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-t-libs-api Status: Awaiting decision from T-libs-api labels Feb 17, 2026

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Feb 21, 2026

petrochenkov removed the I-libs-api-nominated Nominated for discussion during a libs-api team meeting. label Feb 22, 2026

rust-bors bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 22, 2026

jhpratt mentioned this pull request Feb 22, 2026

Rollup of 15 pull requests #152960

Closed

This was referenced Feb 22, 2026

Rollup of 13 pull requests #152964

Closed

Rollup of 14 pull requests #152965

Merged

rust-bors bot merged commit 66cccfd into rust-lang:main Feb 22, 2026
12 checks passed

rustbot added this to the 1.95.0 milestone Feb 22, 2026

cyrgani deleted the nonfatal-tokenstream-parse branch February 22, 2026 20:55

Uh oh!

Comments

Conversation

cyrgani commented Oct 18, 2025 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

cyrgani commented Oct 18, 2025

Uh oh!

rustbot commented Oct 18, 2025

Uh oh!

cyrgani commented Oct 19, 2025

Uh oh!

Uh oh!

cyrgani commented Oct 23, 2025

Uh oh!

davidtwco commented Oct 25, 2025

Uh oh!

This comment has been minimized.

JonathanBrouwer commented Feb 17, 2026

Uh oh!

craterbot commented Feb 17, 2026

Uh oh!

craterbot commented Feb 19, 2026

Uh oh!

craterbot commented Feb 21, 2026

Footnotes

Uh oh!

JonathanBrouwer commented Feb 21, 2026

Uh oh!

petrochenkov commented Feb 22, 2026

Uh oh!

rust-bors bot commented Feb 22, 2026

Uh oh!

JonathanBrouwer commented Feb 22, 2026

Uh oh!

rust-bors bot commented Feb 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

cyrgani commented Oct 18, 2025 •

edited by rustbot

Loading