Prevent macro expansion hang from exponential token growth

[vincenzopalazzo]

Summary

Fixes #95698

Recursive macro_rules! macros can produce exponentially growing token streams that hang the compiler. While the recursion depth limit (default 128) eventually catches infinite recursion, a macro that doubles its output per expansion would produce 2^128 tokens before the depth limit fires — the compiler hangs trying to parse this astronomical input through parse_tt.

This PR adds a new configurable macro_token_limit that caps the number of top-level tokens allowed as input to a single macro_rules! expansion:

• Default: 2^20 (~1 million) tokens
• Configurable via #![macro_token_limit = "N"], following the same pattern as #![recursion_limit = "N"]
• Only applies to macro_rules! (not proc macros)

Why 2^20 as the default?

Based on research across real-world Rust codebases:

Source	Largest legitimate macro invocation
rust-analyzer full analysis of its codebase	~30,672 tokens
Typical large derive/dispatch macros	5,000–100,000 tokens
Extreme case (rust-analyzer issue #10855)	~997,309 tokens
rust-analyzer's own token limit	2,097,152 (2^21)

The 2^20 default provides ~10x headroom over typical large macros. In the pathological case (exponential doubling from 3 tokens), the limit fires at recursion depth ~19 — producing an error in ~200ms instead of hanging indefinitely. The #![macro_token_limit] attribute provides an escape hatch for the rare legitimate case that needs more.

Reproducer (previously hung the compiler)

macro_rules! from_cow_impls {
    ($( $from: ty ),+ $(,)? ) => {
        from_cow_impls!($($from, Cow::from),+);
    };
    ($( $from: ty, $normalizer: expr ),+ $(,)? ) => {};
}

from_cow_impls!(u8, u16);

Now produces:

error: macro expansion token limit reached while expanding `from_cow_impls!`

Changes

File	Change
rustc_span/src/symbol.rs	Add macro_token_limit symbol
rustc_feature/src/builtin_attrs.rs	Register as ungated crate-level attribute
rustc_hir/src/attrs/data_structures.rs	Add MacroTokenLimit to AttributeKind
rustc_hir/src/attrs/encode_cross_crate.rs	Mark as No cross-crate encoding
rustc_attr_parsing/src/attributes/crate_level.rs	Add MacroTokenLimitParser
rustc_attr_parsing/src/context.rs	Register the parser
rustc_interface/src/limits.rs	Add get_macro_token_limit() with default
rustc_interface/src/passes.rs	Read attribute pre-expansion, pass to config
rustc_expand/src/expand.rs	Add macro_token_limit field to ExpansionConfig
rustc_expand/src/mbe/macro_rules.rs	Check token count in expand_macro
rustc_expand/src/errors.rs	New MacroInputTooLarge diagnostic
rustc_passes/src/check_attr.rs	Handle new attribute variant

Test plan

• New regression test tests/ui/macros/issue-95698-exponential-token-growth.rs
• x.py test tests/ui/macros/issue-95698-exponential-token-growth.rs passes (blessed)
• x.py check compiler/rustc_expand passes
• Full CI

[rustbot]

Some changes occurred in compiler/rustc_hir/src/attrs

cc @jdonszelmann, @JonathanBrouwer

Some changes occurred in compiler/rustc_attr_parsing

cc @jdonszelmann, @JonathanBrouwer

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann, @JonathanBrouwer

[rustbot]

r? @jackh726

rustbot has assigned @jackh726.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 69 candidates
• Random selection from 10 candidates

[rustbot]

⚠️ Warning ⚠️

• There are issue links (such as #123) in the commit messages of the following commits.
  Please move them to the PR description, to avoid spamming the issues with references to the commit, and so this bot can automatically canonicalize them to avoid issues with subtree.
  • 03aa980

[rust-log-analyzer]

The job aarch64-gnu-llvm-21-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

##[endgroup]
Executing "/scripts/stage_2_test_set1.sh"
+ /scripts/stage_2_test_set1.sh
+ '[' 1 == 1 ']'
+ echo 'PR_CI_JOB set; skipping tidy'
+ SKIP_TIDY='--skip tidy'
+ ../x.py --stage 2 test --skip tidy --skip compiler --skip src
PR_CI_JOB set; skipping tidy
##[group]Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
---
To only update this specific test, also pass `--test-args cfg/suggest-alternative-name-on-target.rs`

error: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/cfg/suggest-alternative-name-on-target" "-A" "unused" "-W" "unused_attributes" "-A" "internal_features" "-A" "incomplete_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers"
stdout: none
--- stderr -------------------------------
error: unexpected `cfg` condition value: `arm`
##[error]  --> /checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs:5:7
   |
---
LL | #![deny(unexpected_cfgs)]
   |         ^^^^^^^^^^^^^^^
help: `arm` is an expected value for `target_arch`
   |
LL - #[cfg(target_abi = "arm")]
LL + #[cfg(target_arch = "arm")]
   |

error: unexpected `cfg` condition value: `gnu`
##[error]  --> /checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs:12:7
   |
LL | #[cfg(target_arch = "gnu")]
   |       ^^^^^^^^^^^^^^^^^^^
   |
   = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg.html> for more information about checking conditional configuration
help: `gnu` is an expected value for `target_env`
   |
LL - #[cfg(target_arch = "gnu")]
LL + #[cfg(target_env = "gnu")]
   |

error: unexpected `cfg` condition value: `openbsd`
##[error]  --> /checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs:19:7
   |

[rust-bors]

☔ The latest upstream changes (presumably #154958) made this pull request unmergeable. Please resolve the merge conflicts.

[jackh726]

CI isn't passing. And there needs to be many more tests.

In general, I think this needs lang approval, since it's a new attribute.

View changes since this review

[rust-bors]

☔ The latest upstream changes (presumably #155416) made this pull request unmergeable. Please resolve the merge conflicts.