Skip to content

fix: reject and rebuild stale rules caches - #3154

Open
MP-GOWTHAM wants to merge 1 commit into
mandiant:masterfrom
MP-GOWTHAM:fix/stale-rules-cache-invalidation
Open

fix: reject and rebuild stale rules caches#3154
MP-GOWTHAM wants to merge 1 commit into
mandiant:masterfrom
MP-GOWTHAM:fix/stale-rules-cache-invalidation

Conversation

@MP-GOWTHAM

Copy link
Copy Markdown

Fixes #2961

A ruleset loaded from a rules cache written by an older build may contain feature indexes with a different schema, which crashed at match time with:

AttributeError: '_RuleFeatureIndex' object has no attribute 'bytes_prefix_index'

Changes

  • Bump the cache format version (VERSION \x00\x00\x00\x01 -> \x00\x00\x00\x02), so caches written by the previous format are rejected and rebuilt.
  • Validate the loaded ruleset schema on cache load: RuleSet._validate_feature_index() checks the precomputed feature indexes against the fields the current matching code expects. Any mismatch is treated as a cache miss — the invalid entry is deleted and the ruleset is rebuilt, instead of crashing mid-analysis.
  • Harden RuleCache.load: the magic/version/type/id checks no longer rely on assert (which is stripped under python -O); they raise AssertionError explicitly so the invalid-cache path always triggers.

Tests

  • test_ruleset_cache_old_format_version — a cache with the previous format version is rejected and deleted.
  • test_ruleset_cache_stale_feature_index_schema — a cache whose feature indexes are missing bytes_prefix_index is rejected and deleted instead of crashing at match time.

Verification

  • pytest tests/test_rule_cache.py tests/test_rules.py tests/test_match.py tests/test_rules_insn_scope.py: 56 passed, 2 xfailed
  • ruff check and ruff format --check (with .github/ruff.toml): clean

A ruleset loaded from a cache written by an older build may contain feature indexes with a different schema, crashing at match time with AttributeError (missing bytes_prefix_index, mandiant#2961). Bump the cache format version so previous-format caches are rejected, validate the loaded ruleset's precomputed indexes against the current schema, and treat any mismatch as a cache miss (the invalid entry is deleted and the ruleset rebuilt).
@williballenthin

Copy link
Copy Markdown
Collaborator

@MP-GOWTHAM would you please explain the sequence of events that led to you encountering this issue? i'm surprised that the caching logic doesn't already notice this.

@MP-GOWTHAM

Copy link
Copy Markdown
Author

@williballenthin Good question — I found this via the issue tracker rather than hitting it in the wild, so here's the sequence that produces it, and why the existing guards don't catch it:

  1. A user runs a build of capa (same __version__ string, e.g. a dev build or a point release) with rules caching enabled. cache_ruleset() writes a capa-<id>.cache with the current format version, pickling the whole RuleSet — including _feature_indexes_by_scopes, which the code itself flags as "unstable and may change before the next major release".

  2. The user later updates capa to a newer commit that still reports the same version string and has unchanged rule content. compute_cache_identifier() produces the same id (it hashes only the version string + rule contents), so load_cached_ruleset() finds the cache, RuleCache.load() passes, and the old pickled ruleset is returned.

  3. The pickle loads fine because the old _RuleFeatureIndex instances are just objects missing the newer fields. The crash only surfaces later, at match time: _match() accesses feature_index.bytes_prefix_index and raises AttributeError.

Why the existing logic doesn't notice this:

  • The cache identifier deliberately ignores code state beyond the version string, so within a release (where the version string is stable but the schema can change) a stale cache is indistinguishable from a fresh one by id.
  • The mtime heuristic in is_cache_newer_than_rule_code() is only consulted on the CLI path (main.py), not by capa.rules.get_rules() — library/script callers bypass it entirely — and it compares mtimes, which can be defeated by checkout/tooling quirks.
  • Nothing validated the shape of the unpickled ruleset at load time.

So the fix makes load-time schema validation the source of truth (any mismatch = cache miss + rebuild), with the format version bump as the immediate invalidation for caches written by the previous format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stale cache can crash with _RuleFeatureIndex.bytes_prefix_index AttributeError

2 participants