Skip to content

fix(build): give PIT the same SLF4J exclusion as Surefire, and cover FlashAttn - #414

Merged
bernardladenthin merged 1 commit into
mainfrom
claude/fix-pit-slf4j-provider-clash
Sep 5, 2026
Merged

fix(build): give PIT the same SLF4J exclusion as Surefire, and cover FlashAttn#414
bernardladenthin merged 1 commit into
mainfrom
claude/fix-pit-slf4j-provider-clash

Conversation

@bernardladenthin

Copy link
Copy Markdown
Owner

Summary

Fixes the Java Tests Ubuntu failure in run 33861339600. Two failures were stacked there and the first was hiding the second.

1. PIT aborted with "5 tests did not pass without mutation … requires a green suite" — on a suite Surefire had just reported green (1729 tests, 0 failures). Cause: two SLF4J providers on the test classpath. slf4j-simple arrives at runtime scope (it is the binding shipped in the fat jar), logback-classic at test scope (LogCaptor requires logback specifically); SLF4J's ServiceLoader picks one arbitrarily, and every LogCaptor assertion fails when it picks slf4j-simple. The log says it outright:

MINION : SLF4J(W): Class path contains multiple SLF4J providers.
MINION : SLF4J(W): Found provider [org.slf4j.simple.SimpleServiceProvider@…]
MINION : SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@…]

#411 added the binding and the Surefire classpathDependencyExcludes for exactly this — but PIT builds its own classpath and never reads Surefire's configuration, so the mutation run still saw both. pitest-maven has the same parameter, explicitly modelled on Surefire's ("List of classpath entries, formatted as groupId:artifactId…"); it is now set to match. Nothing about the artifact changes: the jar and the fat jar still ship slf4j-simple.

That it went unseen is its own small lesson: every publish run between #411 and now was cancelled, so run 897 is the first to complete since the binding changed.

2. With that cleared, PIT actually ran and surfaced the survivor that run 887 had already reported before #411 ever landed — Mutation score of 99 is below threshold of 100. net.ladenthin.llama.args.* is on the PIT target list at threshold 100, and FlashAttn (added in #408) was the one enum in that package with no test, so getArgValue()'s return argValue had no killer. Adds FlashAttnTest in the shape every sibling enum already uses. Those values are a wire contract, not labels: since llama.cpp b10273 --flash-attn takes a mandatory value, so a wrong or empty token makes the parser eat the following argv entry and the load fails naming a flag the caller never set.

Test plan

  • Failure reproduced locally on the parent commit — same five test names (ChatResponseParserTest.testParseResponse_emitsTimingLine, CompletionResponseParserTest.testParseCompletionResult_emitsTimingLine, and three in TimingsLoggerTest)
  • With the PIT exclusion: no multiple SLF4J providers line, the five tests pass, and the pre-existing survivor becomes visible (99%)
  • With FlashAttnTest added: PIT 319/319 mutations killed (100%), BUILD SUCCESS
  • mvn clean verify green — 1486 tests, 0 failures
  • CI is green on this branch
  • CLAUDE.md updated — the PIT half of the rule is now named next to the existing "spotbugs/spotless bind to verify" trap, including why srcmorph solves the same clash by excluding at the dependency instead (there both providers arrive transitively)

⚠️ What this does NOT fix — macOS 15

The same run also fails Java Tests macOS 15 arm64 (Metal) and (no Metal). That is a separate, bump-introduced regression and is not touched here. What I established:

  • Both were green at b10731 (run 887) and are red at b10797 → introduced by feat!: upgrade llama.cpp from b10731 to b10797 #413.
  • The passing macOS job is macOS 14, and the sharper discriminator is not the OS version: the two failing jobs are exactly the two built with -DGGML_NATIVE=OFF (macOS 14 is the host-native build). Ubuntu and both Windows jobs run these same tests fine.
  • Two failures, both when a second LlamaModel is opened while one is already resident:
    • LlamaModelTest.testSpeculativeDecoding — the draft model (AMD-Llama-135m-code.Q2_K) throws ~30 ms into llama_model_load: E llama_model_load: error loading model: vector. "vector" is libc++'s what() for std::out_of_range/std::length_error, which is why the message is that terse and why it is macOS-shaped.
    • MemoryManagementTest.testPromptCacheCompleteMissAfterWarmup — a second codellama-7b load fails in common_init_from_params after the model itself loaded, with no llama_model_load error line. The Metal job also logs ggml_metal_log_allocated_size: warning: current allocated size is greater than the recommended max working set size, so runner memory pressure is real — but it was green on the same runners at b10731.

I cannot reproduce either on Linux, and the job log has no further native detail. The cheapest decisive next step is a bisect: pin back to b10792 on a probe branch and let CI say whether the five b10792→b10797 commits caused it (#28323 touches llama-model-loader.cpp and llama-hparams.*, which is the suspicious one). Say the word and I'll push that probe — or, if you would rather not spend a run, pinning back to b10792 outright is a safe landing spot, at the cost of the s390x q5_1 fix.

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
  • My commits follow Conventional Commits
  • No security-sensitive changes

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH


Generated by Claude Code

…FlashAttn

Two separate failures were stacked on the `Java Tests Ubuntu` job, and the
first was hiding the second.

1. PIT aborted with "5 tests did not pass without mutation ... requires a green
   suite" — on a suite Surefire had just reported green (1729 tests, 0
   failures). Cause: two SLF4J providers on the test classpath. slf4j-simple
   arrives at runtime scope because it is the binding shipped in the fat jar,
   logback-classic at test scope because LogCaptor requires logback
   specifically; SLF4J's ServiceLoader picks one arbitrarily and every LogCaptor
   assertion fails when it picks slf4j-simple. #411 added the binding and the
   Surefire `classpathDependencyExcludes` for exactly this, but PIT builds its
   OWN classpath and never reads Surefire's configuration, so the mutation run
   still saw both. pitest-maven has the same parameter, modelled on Surefire's;
   it is now set to match. Nothing about the artifact changes.

   It went unseen because every publish run between #411 and now was cancelled;
   run 897 is the first to complete since.

2. With that cleared, PIT actually ran and surfaced the survivor that run 887
   had already reported before #411 ever landed: "Mutation score of 99 is below
   threshold of 100". `net.ladenthin.llama.args.*` is on the PIT target list at
   threshold 100, and FlashAttn — added in #408 — was the one enum in that
   package with no test, so `getArgValue()`'s `return argValue` had no killer.
   Adds FlashAttnTest in the shape every sibling enum already uses. The values
   are a wire contract, not labels: since llama.cpp b10273 `--flash-attn` takes
   a mandatory value, so a wrong or empty token makes the parser eat the next
   argv entry and the load fails naming a flag the caller never set.

Verified locally, both directions: the failure reproduces on the parent commit
with the exact same five test names, and with the fix PIT reports 319/319
mutations killed (100%) and `mvn clean verify` is green at 1486 tests.

CLAUDE.md now names the PIT half of the rule explicitly, next to the existing
"spotbugs/spotless bind to verify" trap, including why srcmorph solves the same
clash by excluding at the dependency instead (there both providers arrive
transitively).

This does NOT address the macOS 15 failures in the same run; those are a
separate, bump-introduced regression and are reported alongside this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH
@claude

claude Bot commented Sep 5, 2026

Copy link
Copy Markdown

Review Summary

This PR fixes two stacked test failures in the Java Tests Ubuntu job with a clear, focused approach. Both changes are correct and follow established patterns.

✅ Issues Fixed

1. PIT SLF4J Provider Clash — Excellent diagnosis. The root cause is spot-on: PIT builds its own classpath and doesn't read Surefire's configuration, so the Surefire-only exclusion left both SLF4J providers on the PIT classpath. The fix—mirroring the Surefire exclusion in pitest-maven configuration—is the right approach. The CLAUDE.md documentation clearly explains this non-obvious trap and why it matters.

2. FlashAttn Test Coverage — The new test follows the established pattern perfectly, extending AbstractCliArgEnumTest and providing all three enum constants with correct arg values and enum count. The docstring accurately emphasizes that these are wire contracts, not cosmetic labels, which is an important constraint since llama.cpp b10273 made the flag value-mandatory.

Code Quality

  • FlashAttnTest.java: Consistent with SamplerTest, LazyModeTest, and other enum tests in the same package. All three values (on, off, auto) are correct and the enum count (3) is accurate.
  • pom.xml: The classpathDependencyExcludes block is properly indented, placed in the correct plugin config, and includes clear inline documentation explaining why it's needed in pitest-maven separately from Surefire.
  • CLAUDE.md: Enhanced with concrete details on the root cause, the failure mechanism, and a cross-repo note on an alternative approach (srcmorph). This is valuable for anyone maintaining the build in the future.

Verification Notes

✅ Test pattern consistency verified against SamplerTest and other enum tests
✅ FlashAttn enum values match test data (on/off/auto)
✅ Enum count correct (3 constants)
✅ Configuration follows pitest-maven's documented parameter format
✅ Documentation explains why this isn't redundant (not just that it's needed)

Scope

The PR body appropriately notes that macOS 15 failures are separate (a bump-introduced regression). This PR stays focused on the PIT/mutation testing surface, which is the right scope boundary.

No issues found. Ready to merge.

@sonarqubecloud

sonarqubecloud Bot commented Sep 5, 2026

Copy link
Copy Markdown

@bernardladenthin
bernardladenthin merged commit c6b0c35 into main Sep 5, 2026
14 of 17 checks passed
@bernardladenthin
bernardladenthin deleted the claude/fix-pit-slf4j-provider-clash branch September 5, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants