Resolve local alleles correctly and emit the missing VCF 4.5 accessors

[ecrum19]

Why

The VCF Core coverage assessment builds its witnesses with the vocabulary repository's own materializer (scripts/vcf_examples.py). This converter is a separate program. Both target VCF Core, but nothing had checked that a query and expected answer written against one also holds over the other.

Replaying the assessment's existing queries and its independently authored expected answers over converter output found 15 requirements where the two disagree. Nine are fixed here.

The correctness defect

vcfc:forAllele on a Number=LA/LR item was resolved positionally, as if the list indexed the record's ALT column. It indexes the sample's local allele subset:

##FORMAT=<ID=LAD,Number=LR,...>
chr1 1 . G A,C,T,<*> . . . GT:LAA:LAD:LPL  2/2:2,4:20,30,10:...

LAA=2,4 makes the local set {REF, ALT2, ALT4}, so 20,30,10 belongs to global alleles 0, 2, 4. The converter attached them to 0, 1, 2:

Global allele	Correct	Was
0	20	20
1	(absent)	30
2	30	10

Wrong values, not missing ones, with nothing in the graph to signal it. An item whose sample declares no LAA now gets no vcfc:forAllele at all — a wrong link is worse than none.

Related: the LocalAlleleSet carried vcfc:localAlleleIndex on the shared allele resource, which the vocabulary deprecates because two samples can give the same ALT different local positions. It now emits LocalAlleleMembership with localIndex and localAllele. LAA is read before the per-key loop, since a FORMAT string may legally list it after a field that depends on it.

Accessors that were never emitted

• ID decomposition — hasIdentifier, identifierValue, componentIndex
• FILTER decomposition — filterStatus, hasFilterCode, filterCodeValue, and declaredByFilter citing each code's ##FILTER line
• fieldIndex on INFO entries, and record-level vcfc:FormatKey resources (IRI shape follows the class's own iriTemplate)
• phaseIndicator per allele call — which also lets the converter emit vcfc:MixedPhasing, previously never used: it chose Phased or Unphased for the whole genotype, which misdescribes one whose indicators disagree

Base-modification aliases

VCF 4.5 reserves these keys in two spellings — M[0-9]+[ACGTUN] and named aliases such as M5mC for M27551C. Only the numeric form was recognised, so an aliased key built no BaseModification carrier at all. The 30 aliases now resolve to the same ChEBI residue while each key keeps its written spelling in its IRI. features-v4.5 emits three carriers where it emitted none.

Oracle

validation_runner gains matching expectations, derived from the source VCF rather than from the emitter, so the census stays an independent check. typed_value_kind now states the typed-value rule once for both INFO and FORMAT — the previous INFO mirror used bare int()/float() and would have counted a BCF-reserved integer and a non-finite float that the emitter deliberately skips.

Validation harness

The cross-engine query agreement job failed on a commit that changed only a docstring, reporting a timeout that had not happened:

comunica could not read /work/.../expanded.nt within 3600s
(raise --comunica-warmup-timeout, or use --engine qlever):
<urlopen error [Errno 111] Connection refused>

It had waited milliseconds. timeout= on urlopen is the socket timeout — how long to wait for a response — and a refused connection comes back from the kernel at once, so the budget never applied.

_await_bind already loops and tolerates refusal; _await_warm made exactly one attempt. That asymmetry was the bug: comunica answers the bind probe and then hands the socket to its worker, so the next connection can be refused for a moment while the process is perfectly healthy. The probe now retries until the budget is genuinely spent, and the reported duration is measured rather than read back from the flag.

Retrying must not become tolerance for a dead engine, so two conditions still fail immediately: the server process exiting, which keeps its exit code and log tail, and the endpoint answering with an error — an HTTPError means comunica replied, and a source it cannot parse would answer every probe identically, so retrying would spin the whole budget before reporting what it knew on the first attempt.

Four unit tests cover those paths (0.3s, all fakes — no Docker, no network). HdtEngine shares the mixin and was exposed to the same failure. The flake is timing-dependent rather than rare: the image floats its base and its apt nodejs/python3, so a rebuild can shift the timing that exposes it.

Release

Bumped to v3.0.3 with scripts/release.py, which synchronises all five version markers; release.py --check-tag v3.0.3 passes, which is the gate CI runs before publishing.

The conda sha256 is deliberately back to its placeholder: the v3.0.3 source archive does not exist until the tag is pushed, and retaining v3.0.2's checksum would be invalid. Populate it after tagging with python3 scripts/release.py 3.0.3 --fetch-conda-sha256.

Documentation

docs/vcf-coverage.md listed Parsed FILTER, Parsed FORMAT keys, Parsed record IDs, Local-allele membership and Mixed phasing as entirely unemitted — precisely the five families this branch emits. That table said the opposite of what the converter does, so it is rewritten from the source rather than from this description:

• Five of the ten families are now emitted; two are complete, and one term each remains in the other three — keyPattern, aliasOf and allelePhaseSet are still not derived, and the table says so per family.
• The ID/FILTER/FORMAT-key decompositions are emitted by append_record_detail_rdf ungated, so they are recorded as present in both representation profiles and both INFO representations — not expanded-only.
• The new resources are counted by q09/q10 but their values are not compared, which places them in the existing "counted but not compared" gap rather than a new one. No mutation targets them yet.
• docs/conversion.md separates the two local-allele facts: LAA resolution is FORMAT-side, and an INFO Number=LA/LR field has no sample, so its items are materialized but carry no forAllele.

Two methodology points this work turned up are now recorded rather than left in the commit log:

• A parser's re-serialization is not the file. str(variant) restores a trailing FORMAT field the source omitted, so the census counted cells the file does not contain — blind to exactly the case VCF Core models on purpose. docs/validation-methodology.md generalises the rule beside the earlier raw_header instance of it.
• The validator ships inside the image. docs/validation.md now warns that an image older than the wrapper reports the new predicates as unexpected, so a MISMATCH on Q9/Q10 is not read as a conversion defect.

Also updated: architecture.md, limitations.md, sample-representation-guide.md, roadmap.md, rml-mappings.md, README.md, and a vcf_rdfizer.py docstring that still said "Three groups". Two stale test counts (415) and two link anchors that never resolved are fixed — both predate this branch.

Results

• Cross-producer divergence over the 10-fixture subset: 178 checks → 40, 15 requirements → 6
• Semantic round-trip: 35/35 records recovered through structured properties only
• Full suite: 707 tests, OK (measured at branch head; the 647 previously quoted here was taken mid-branch, before the last two test commits)
• Mutation score regenerated, not assumed: 96/113 (85%), unchanged — the documented figure still holds

Remaining six: R11 (a datatype the vocabulary deliberately leaves open — guidance added on the vocabulary side), R21, R27, R29 and R61 (accessors not yet built), R89 (partial repeat decomposition).

Note

src/validation/ and vcf_rdfizer_vocab.py are both baked into the image, so ecrum19/vcf-rdfizer:latest needs rebuilding before an end-to-end -m validation run reflects the new expectations. This is now documented in docs/validation.md rather than living only here.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.81145% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/validation/validation_runner.py	86.58%	11 Missing ⚠️
test/test_validation_engines_unit.py	98.70%	1 Missing ⚠️
test/test_vcf45_accessors_unit.py	99.49%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!