feat(ground-truth): wire real CI, read op params from GGUF metadata - #989
Merged
Merged
Conversation
Ground-truth validation existed as working Kotlin code but was dormant in practice: no CI workflow actually ran it (TESTING.md's example block was never a real file), and every test silently skips when the sibling ../skainet-ground-truth checkout or generated GGUF files aren't present — so a broken pipeline showed green, not red. ## Op parameters come from GGUF metadata now, not guessed English text gt/pytorch/io/writer.py already writes op.padding/op.stride/op.groups/etc. as real GGUF metadata (op_params passed to @eXecutable), but GroundTruthLoader never read it — GroundTruthIntegrationTest instead string-matched the human-readable description ("padded" -> padding(1), "strided" -> stride(2)) to guess parameters, silently wrong for any case whose description didn't happen to contain the right word. - GroundTruthLoader.extractOpParams/decodeFieldValue: parse every op.* field into GroundTruthTestCase.rawOpParams (Int/Float/List, scalar or array). - GroundTruthTestCase.resolvedParams(): maps rawOpParams onto OperationParams' named fields. - GroundTruthValidator.validate/assertValid and the validateWith/ assertValidWith extensions now default params to testCase.resolvedParams() instead of an empty OperationParams() -- the JVM-side convenience overloads (GroundTruthLoader.kt) take OperationParams? = null instead, so "not specified" can still fall through to the per-test-case default rather than silently overriding it. - Deleted GroundTruthIntegrationTest's inferConv2dParams/ inferOperationParams entirely. Verified against real data: rebuilt the ground-truth Docker image, regenerated all 7 test suites, and found the fix immediately surfaced a real data gap it was designed to catch -- three TS-001 conv2d cases (UC-001's second function, UC-002, UC-003) set stride=2/padding=1 on the actual torch.nn.Conv2d call but never passed them via op_params, so nothing recorded that parameter at all. Fixed in skainet-ground-truth (separate commit there). TS-001 went from 3/6 to 6/6 passing; TS-003 (flatten, already had start_dim/end_dim recorded) was 5/5 before and after, confirming the metadata path was already correct there. ## CI: make dormancy visible, then fix it - GroundTruthConfig.requireAvailable / TestAssumptions.kt: with -PrequireGroundTruth=true, missing ground truth is check()-failure, not Assume.assumeTrue skip. Verified both directions by removing the local results dir: without the flag the suite still skips cleanly (dev convenience preserved); with it, all four ground-truth tests fail loudly instead. - New .github/workflows/ground-truth.yml: checks out skainet-ground-truth as a sibling (path: ../skainet-ground-truth, matching what build.gradle.kts already expected), builds the Docker image, generates GGUF fixtures, runs jvmTest -PrequireGroundTruth=true. Uses --no-configuration-cache throughout -- CI's ci-gradle.properties enables the config cache by default, and buildGroundTruthDocker/ generateGroundTruth/cleanGroundTruth/listGroundTruth all hit a pre-existing (not introduced here, reproduced on unmodified develop) "cannot serialize Gradle script object references" config-cache incompatibility that would otherwise fail every run. - build.gradle.kts: groundTruthProjectDir is now overridable via -PgroundTruthSourceDir (defaults to today's sibling-checkout convenience) -- the numcrux-readiness piece: SKaiNET's consumer no longer hardcodes where the fixtures physically live. Verified end-to-end: ./gradlew --no-configuration-cache buildGroundTruthDocker generateGroundTruth jvmTest -PrequireGroundTruth=true, the exact command sequence the new workflow runs, exits 0 against real Docker-generated fixtures. Follow-ups (Python-side coverage expansion, gradient validation decision, numcrux transfer) tracked in #984 and its sub-issues -- not attempted here, out of scope for "make the existing pipeline real."
This was referenced Aug 13, 2026
actions/checkout's `path: ../skainet-ground-truth` failed hard on the first real run: "Repository path '/home/runner/work/SKaiNET/ skainet-ground-truth' is not under '/home/runner/work/SKaiNET/SKaiNET'" -- the action explicitly rejects any path resolving outside the checkout's own workspace, so a true sibling checkout via `path: ..` was never actually possible, not just untested. Fixed by checking out to a plain subdirectory (path: skainet-ground-truth, inside SKaiNET's own workspace) and pointing every Gradle invocation at it explicitly via -PgroundTruthSourceDir=$GITHUB_WORKSPACE/ skainet-ground-truth/pytorch -- the override groundTruthProjectDir already supports. Local dev is unaffected: the sibling-checkout default stays in place for anyone who already has skainet-ground-truth cloned next to SKaiNET. Verified locally: -PgroundTruthSourceDir pointed at the same skainet-ground-truth checkout used throughout this PR's earlier testing, confirmed listGroundTruth finds the real generated fixtures through the override.
aharakal
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ground-truth validation existed as working Kotlin code but was dormant in practice:
TESTING.md's.github/workflows/ground-truth.ymlexample block was never a real file.
Assume.assumeTrue) when the sibling../skainet-ground-truthcheckout or generated GGUF files aren't present — thedefault state of any fresh clone or CI runner. Green CI proved nothing.
(
desc.contains("padded") -> padding(1)), even though the Python writer alreadyserializes them as real
op.*GGUF metadata that was simply never read.Changes
Metadata-driven op params (the correctness fix):
GroundTruthLoader.extractOpParams/decodeFieldValue: parse everyop.*GGUF fieldinto
GroundTruthTestCase.rawOpParams.GroundTruthTestCase.resolvedParams(): maps that ontoOperationParams' typed fields.GroundTruthValidator'svalidate/assertValid(and thevalidateWith/assertValidWithextensions) now defaultparamstotestCase.resolvedParams()instead of an empty
OperationParams(). The JVM convenience overloads takeOperationParams? = nullinstead, so "not specified" falls through to theper-test-case default rather than silently overriding it.
GroundTruthIntegrationTest'sinferConv2dParams/inferOperationParamsstring-matching entirely.
Verified against real Docker-generated data — and this immediately surfaced a real gap
in the ground-truth fixtures themselves (three
TS-001conv2d cases hadstride/paddingbaked into the PyTorch call but never recorded inop_params), fixedin the companion
skainet-ground-truthPR:SKaiNET-developers/skainet-ground-truth#3.
With that fix,
TS-001goes from 3/6 to 6/6 passing.CI: make dormancy visible, then fix it:
GroundTruthConfig.requireAvailable/TestAssumptions.kt:-PrequireGroundTruth=trueturns missing ground truth into a hard
check()failure instead of a skip. Verifiedboth directions.
.github/workflows/ground-truth.yml: checks outskainet-ground-truthas asibling (matching what
build.gradle.ktsalready expected), builds the Docker image,generates GGUF fixtures, runs
jvmTest -PrequireGroundTruth=true.--no-configuration-cachethroughout — CI'sci-gradle.propertiesenables theconfig cache by default, and the ground-truth Gradle tasks hit a pre-existing
(reproduced identically on unmodified
develop, not introduced here) "cannotserialize Gradle script object references" config-cache incompatibility.
build.gradle.kts: the fixture source directory is now overridable via-PgroundTruthSourceDir(defaults to today's sibling-checkout convenience) — thenumcrux-readiness piece, so SKaiNET's consumer doesn't hardcode where the fixtures
physically live.
Test plan
./gradlew :skainet-test:skainet-test-groundtruth:compileKotlinJvm :skainet-test:skainet-test-groundtruth:compileTestKotlinJvm— clean compilethe actual validation tests against them
-PrequireGroundTruth=truegenuinely fails (not skips) with theresults directory absent, and skips cleanly without the flag
(
--no-configuration-cache buildGroundTruthDocker generateGroundTruth jvmTest -PrequireGroundTruth=true) end-to-end — exits 0Follow-ups (Python-side op coverage expansion, a decision on gradient validation which
is currently fully unimplemented despite the "gradienttracer" name, and the numcrux
transfer itself) are tracked in #984 and its sub-issues — out of scope here.