Resolve local alleles correctly and emit the missing VCF 4.5 accessors

CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 message: "If you use VCF-RDFizer in your research, please cite it using the metadata below."
 title: "VCF-RDFizer"
 type: software
-version: "3.0.2"
+version: "3.0.3"
 authors:
   - name: "VCF-RDFizer maintainers"
 repository-code: "https://github.com/ecrum19/VCF-RDFizer"

README.md

@@ -347,9 +347,12 @@ preserves the per-sample vocabulary model:
 - every represented FORMAT slot is a `vcfc:FormatFieldValue`, linked to its
   declaration with `vcfc:declaredBy`;
 - `GT` is parsed into a `vcfc:Genotype` with an explicit `vcfc:phasingStatus`
-  and one `vcfc:GenotypeAlleleCall` per position, and `FT`, `PS`/`PSL`, `LAA`,
-  the copy-number and haplotype keys, and the `M`/`DPM`/`ADM` base-modification
-  families get their own resources;
+  — `vcfc:MixedPhasing` when its positions disagree — and one
+  `vcfc:GenotypeAlleleCall` per position carrying its own `vcfc:phaseIndicator`,
+  and `FT`, `PS`/`PSL`, `LAA` (with per-sample `vcfc:LocalAlleleMembership`
+  ordinals), the copy-number and haplotype keys, and the `M`/`DPM`/`ADM`
+  base-modification families in both their numeric and aliased spellings get
+  their own resources;
 - the VCF file declares `vcfc:representationProfile vcfc:ExpandedRepresentation`.
 
 With the default rules, these triples are appended directly from `records.tsv`;
@@ -1080,7 +1083,7 @@ Safe termination:
 
 If you use VCF-RDFizer in a publication, please cite:
 
-VCF-RDFizer maintainers. (2026). *VCF-RDFizer* (Version 3.0.2) [Computer software]. GitHub. https://github.com/ecrum19/VCF-RDFizer
+VCF-RDFizer maintainers. (2026). *VCF-RDFizer* (Version 3.0.3) [Computer software]. GitHub. https://github.com/ecrum19/VCF-RDFizer
 
 BibTeX:
 
@@ -1089,7 +1092,7 @@ BibTeX:
   author  = {{VCF-RDFizer maintainers}},
   title   = {VCF-RDFizer},
   year    = {2026},
-  version = {3.0.2},
+  version = {3.0.3},
   url     = {https://github.com/ecrum19/VCF-RDFizer},
   note    = {Computer software}
 }

conda-recipe/README.md

@@ -7,11 +7,11 @@ do not submit this package to `staged-recipes`.
 
 ## Before submitting to conda-forge
 
-1. Commit the version bump, then create and push a Git tag (for example `v3.0.2`).
+1. Commit the version bump, then create and push a Git tag (for example `v3.0.3`).
 2. Download the source tarball and compute sha256:
    ```bash
    curl -L -o vcf-rdfizer.tar.gz \
-     https://github.com/ecrum19/VCF-RDFizer/archive/refs/tags/v3.0.2.tar.gz
+     https://github.com/ecrum19/VCF-RDFizer/archive/refs/tags/v3.0.3.tar.gz
    shasum -a 256 vcf-rdfizer.tar.gz
    ```
 3. Replace `version` and `sha256` in the feedstock's `recipe/meta.yaml`.

conda-recipe/meta.yaml

@@ -1,13 +1,13 @@
 {% set name = "vcf-rdfizer" %}
-{% set version = "3.0.2" %}
+{% set version = "3.0.3" %}
 
 package:
   name: {{ name|lower }}
   version: {{ version }}
 
 source:
   url: https://github.com/ecrum19/VCF-RDFizer/archive/refs/tags/v{{ version }}.tar.gz
-  sha256: 46c08df157269ec1e3b1c4666a20e37a7b1709b9aa8366265a2bae91d10ae637
+  sha256: REPLACE_WITH_GITHUB_TARBALL_SHA256
 
 build:
   noarch: python

docs/architecture.md

@@ -89,9 +89,9 @@ whose size is the product of its dimensions.
 
 | Emitter | Emits | Why not RML |
 | --- | --- | --- |
-| `emit_sample_representation` | Expanded: `SampleCall`/`FormatFieldValue`/`Genotype`/`PhaseSet`/`LocalAlleleSet`/`BaseModification`. Condensed: `SampleSet`/`CohortCallMatrix`/`FormatValueVector`. `SampleSet` and `VCFSample` in both. | RML would first have to materialize a helper table of variants × samples (× FORMAT keys) — the largest intermediate the pipeline can produce |
+| `emit_sample_representation` | Expanded: `SampleCall`/`FormatFieldValue`/`Genotype`/`PhaseSet`/`LocalAlleleSet`/`LocalAlleleMembership`/`BaseModification`. Condensed: `SampleSet`/`CohortCallMatrix`/`FormatValueVector`. `SampleSet` and `VCFSample` in both. | RML would first have to materialize a helper table of variants × samples (× FORMAT keys) — the largest intermediate the pipeline can produce |
 | `append_header_representation_rdf` | Header-line subclasses, the ordered `HeaderAttribute` resources every structured line needs, and the INFO/FORMAT/FILTER/ALT/contig/META/SAMPLE/PEDIGREE declaration properties | RML cannot choose a class per row, nor decompose one angle-bracketed cell into an ordered set of resources |
-| `emit_record_detail` | `ID`, `ALT`, `QUAL`, `FILTER`, `infoRaw` (each may be the missing token); the allele layer (`ReferenceAllele`/`AltAllele`, breakends, symbolic types); structured `InfoFieldValue` with its `FieldValueItem` decomposition; and the SV carriers | RML cannot type the missing token per row, cannot switch a literal's datatype on a declared `Type`, and the allele and value-item layers would need variants × alleles and variants × INFO-keys helper tables |
+| `emit_record_detail` | `ID`, `ALT`, `QUAL`, `FILTER`, `infoRaw` (each may be the missing token); their decompositions (`RecordIdentifier`, `filterStatus` with its `FilterCode` resources, the ordered `FormatKey` list); the allele layer (`ReferenceAllele`/`AltAllele`, breakends, symbolic types); structured `InfoFieldValue` with its `FieldValueItem` decomposition; and the SV carriers | RML cannot type the missing token per row, cannot switch a literal's datatype on a declared `Type`, and the allele, decomposition and value-item layers would need variants × alleles, variants × components and variants × INFO-keys helper tables |
 
 The exception count did not change with the move to the VCF Core vocabulary, but
 what each emitter covers grew considerably: the vocabulary models the VCF 4.5

docs/conversion.md

@@ -129,6 +129,23 @@ Float lexical space:
 | `.` or empty | `"."^^vcfc:Null` |
 | anything else | a plain string literal, deliberately kept rather than dropped, and reported by the SHACL layer |
 
+Three of those cells are also **decomposed**, in both INFO representations,
+because a single cell carries a list the vocabulary models as resources:
+
+- **ID.** Each semicolon-separated component becomes a
+  `vcfc:RecordIdentifier` at `…#record/{ROW_ID}/id/{N}`, linked with
+  `vcfc:hasIdentifier` and carrying `identifierValue` and `componentIndex`.
+  The raw `vcfc:recordId` stays beside it.
+- **FILTER.** The call gets a `vcfc:filterStatus` of `vcfc:FiltersPassed`
+  (`PASS`), `vcfc:FiltersNotApplied` (`.`) or `vcfc:FiltersFailed`. In the
+  failed case each code also becomes a `vcfc:FilterCode` with
+  `filterCodeValue` and `componentIndex`, joined to its `##FILTER` declaration
+  with `vcfc:declaredByFilter` so a consumer can read the description without
+  re-reading the header.
+- **FORMAT.** The key list becomes ordered `vcfc:FormatKey` resources carrying
+  `vcfc:fieldIndex` and `vcfc:declaredBy`, beside the raw `vcfc:formatRaw`.
+  The IRI shape follows the class's own `iriTemplate`.
+
 `--info-representation structured` (the default) additionally emits:
 
 - **The allele layer.** `REF` and each `ALT` item become an ordered
@@ -143,7 +160,8 @@ Float lexical space:
   `CHROM` names, with `vcfc:chromosome`.
 - **Structured INFO.** One `vcfc:InfoFieldValue` per record and key at
   `…#call/{ROW_ID}/info/{KEY}`, linked to the `##INFO` declaration through
-  `vcfc:declaredBy`. A single-valued field whose declared `Type` is `Integer` or
+  `vcfc:declaredBy` and carrying `vcfc:fieldIndex`, its position in the INFO
+  column. A single-valued field whose declared `Type` is `Integer` or
   `Float` also gets a typed `fieldValueInteger` / `fieldValueDecimal`; a `Flag`
   gets `vcfc:fieldValueBoolean true`.
 - **Value items.** A field whose declared `Number` gives its positions meaning —
@@ -154,6 +172,11 @@ Float lexical space:
   …) record the width with `vcfc:tupleArity` so a consumer can regroup them.
   This closes the multi-valued-INFO gap that earlier versions recorded in
   `limitations.md`.
+
+  The local-allele codes `LA`/`LR`/`LG` are a special case: they index a
+  *sample's* allele subset, and an INFO field has no sample. Their items are
+  therefore still materialized and indexed, but carry no `vcfc:forAllele` —
+  see the genotype layer below, where the subset exists.
 - **The SV carriers.** `SVLEN`, `SVCLAIM`, `IMPRECISE`, `NOVEL`, `END`, `EVENT`
   with `EVENTTYPE`, the confidence intervals (`CIPOS`/`CIEND` through FALDO
   `InRangePosition`, the rest as `vcfc:ConfidenceInterval`), and the
@@ -224,11 +247,45 @@ instead of relying on repeated `sampleId` literals.
 **Expanded mode additionally parses the per-sample values** into the vocabulary's
 genotype layer: `GT` becomes a `vcfc:Genotype` with `genotypeString`, `ploidy`,
 `phasingStatus` and one `vcfc:GenotypeAlleleCall` per position (each either
-pointing at the allele it called or flagged `isNoCall`); `FT` becomes
-`sampleFilter`; `PS`/`PSL`/`PSO`/`PSQ` become a `vcfc:PhaseSet`; `LAA` becomes a
-`vcfc:LocalAlleleSet`; `CN`/`CNQ`/`CNL`/`CNP` and `HAP`/`AHAP` become their
-copy-number and haplotype properties; and the pattern-defined `M`/`DPM`/`ADM`
-keys become `vcfc:BaseModification` resources pointing at a ChEBI residue.
+pointing at the allele it called or flagged `isNoCall`, and each carrying its
+own `vcfc:phaseIndicator`); `FT` becomes `sampleFilter`; `PS`/`PSL`/`PSO`/`PSQ`
+become a `vcfc:PhaseSet`; `LAA` becomes a `vcfc:LocalAlleleSet`;
+`CN`/`CNQ`/`CNL`/`CNP` and `HAP`/`AHAP` become their copy-number and haplotype
+properties; and the pattern-defined `M`/`DPM`/`ADM` keys become
+`vcfc:BaseModification` resources pointing at a ChEBI residue.
+
+Four details of that layer are worth stating explicitly:
+
+- **Local-allele fields are resolved through `LAA`, not positionally.** A
+  `Number=LA`/`LR` value list indexes the **sample's own local allele subset**,
+  not the record's ALT column. With `LAA=2,4` the local set is
+  {REF, ALT2, ALT4}, so the three values of an `LR` field belong to global
+  alleles 0, 2 and 4 — not 0, 1, 2. Each `vcfc:FieldValueItem` is joined with
+  `vcfc:forAllele` through that mapping. A sample that declares no `LAA` has no
+  local ordering to resolve against, so its items keep `vcfc:valueIndex` and get
+  no `vcfc:forAllele` at all: a wrong link is worse than an absent one, because
+  nothing in the graph would signal it. `LAA` is read before the per-key loop,
+  since a FORMAT string may legally list it after a field that depends on it.
+- **Phasing is per position, not per genotype.** `vcfc:phasingStatus` is
+  `vcfc:Phased` or `vcfc:Unphased` only when every position agrees. A genotype
+  whose indicators disagree — `0|1/2` — gets `vcfc:MixedPhasing`, and the
+  per-call `vcfc:phaseIndicator` carries the precise reading. A single verdict
+  would misdescribe it.
+- **Local-allele position belongs to the membership.** Alongside
+  `vcfc:hasLocalAllele`, each member of a `vcfc:LocalAlleleSet` gets a
+  `vcfc:LocalAlleleMembership` with `vcfc:localAllele` and `vcfc:localIndex`.
+  The ordinal cannot live on the allele resource, which is shared across
+  samples: two samples may give the same ALT different local positions. The
+  vocabulary deprecates `vcfc:localAlleleIndex` for that reason, and the
+  converter no longer emits it.
+- **Base-modification aliases resolve.** VCF 4.5 reserves each modification key
+  in two spellings — `M[0-9]+[ACGTUN]` and a named alias such as `M5mC` for
+  `M27551C` — across all three families, 30 aliased keys in total. Both
+  spellings build the same carrier against the same ChEBI residue, while the
+  key keeps its written spelling in the resource IRI so the graph traces back
+  to the source column. A `Number=M` field is decomposed over the bases the
+  genotype's called alleles actually carry, in GT order, since that is the
+  order the specification defines it over.
 
 **Condensed mode deliberately stops at the vector.** Decomposing GT per sample is
 exactly the per-sample materialization the condensed profile exists to avoid.

docs/limitations.md

@@ -80,7 +80,10 @@ declared assembly is recorded but not used for anything.
 target to the VCF Core vocabulary closed three gaps that used to be listed here:
 symbolic ALTs, breakends and `*` are now classified and parsed; `Number=A/R/G/P`
 values are decomposed into per-allele `vcfc:FieldValueItem` resources; and `GT`
-is parsed into ordered allele calls with an explicit phasing status. See
+is parsed into ordered allele calls with an explicit phasing status. The `ID`,
+`FILTER` and `FORMAT` columns followed: each is now decomposed into ordered
+resources beside its raw literal, and a genotype whose positions disagree about
+phasing is `vcfc:MixedPhasing` rather than forced to a single verdict. See
 [`conversion.md`](conversion.md#4-stage-three--the-wrappers-own-emitters).
 
 What remains true:

docs/rml-mappings.md

@@ -247,7 +247,7 @@ published vocabulary is worth knowing about: the ontology declares
 shape constrains them with `sh:datatype xsd:integer`, and a SHACL datatype
 constraint compares the datatype IRI exactly. This converter emits
 `xsd:integer`, which satisfies both. See
-[`validation-migration-notes.md`](validation-migration-notes.md#3-ordinals-are-now-xsdinteger-not-xsdpositiveinteger).
+[`validation-migration-notes.md`](validation-migration-notes.md#5-ordinals-are-xsdinteger).
 
 ---
 

docs/roadmap.md

@@ -33,7 +33,7 @@ a record with `ALT=.` satisfies both the shape and `vcfc:missingValuePolicy`.
 
 ### ~~1. The validation suite has not been migrated~~ — **done**
 
-The suite passes: 415 tests, mutation score 96/113 (85%) across 60 mutations.
+The suite passes: 707 tests, mutation score 96/113 (85%) across 60 mutations.
 The runner and the wrapper now share one vocabulary module instead of mirroring
 each other, and the census derives every new resource family from the VCF. Ten
 of the eighteen new mutations are recorded gaps, each with what would close it.

docs/sample-representation-guide.md

@@ -231,8 +231,8 @@ a decoder or an application-level vector function. If you need to query
 genotypes directly, that is the argument for expanded mode.
 
 Everything *not* per-sample is identical in both profiles: the header layer, the
-allele layer, structured INFO with its value items, and the SV carriers are all
-emitted the same way.
+allele layer, structured INFO with its value items, the SV carriers, and the
+record-level ID/FILTER/FORMAT-key decompositions are all emitted the same way.
 
 ### Condensed growth
 

docs/validation-methodology.md

@@ -146,6 +146,19 @@ never declared it - making the oracle expect a `FilterDefinition` the graph
 could not contain. The oracle now reads the header block from the file itself,
 which is the same text the conversion reads.
 
+The same normalisation reaches the data lines, and cost a second fix. The census
+read each record from `str(variant)`, which restores a trailing FORMAT field the
+file omitted: a sample written `0` under `GT:DP` comes back as `0:.`, so the
+oracle counted a cell the file does not contain. That made the validator blind
+to exactly the case VCF Core models on purpose - an emitter that *invented* the
+dropped field would have looked correct. The census now reads the data lines
+from the file's own text too, alongside the cyvcf2 reader, and asserts the two
+stay aligned rather than silently miscounting if one skips a line.
+
+The rule generalises: **a parser's re-serialization is not the file.** Anything
+the oracle counts should be read from the source text, not from what a library
+renders back.
+
 **When adding a query**, prefer datatype-*family* checks and lexical
 comparisons over anything that assumes a store's internal representation, and
 run the agreement script before trusting it.

docs/validation-migration-notes.md

@@ -7,7 +7,7 @@ and follows each input's declared VCF version. The validation suite has been
 migrated with it. This file records what changed, what it found, and what is
 still open.
 
-**Status: the suite passes.** 415 tests, no failures. The mutation score is
+**Status: the suite passes.** 707 tests, no failures. The mutation score is
 96/113 (85%) across 60 mutations, each listed with its expected detector.
 
 ---
@@ -87,6 +87,33 @@ The fixture's `XSD_POSITIVE_INTEGER` is gone. VCF Core's
 datatype with the bound applied numerically, so the ontology range and the
 shapes no longer disagree.
 
+### 6. The census counts the file's cells, not htslib's rendering of them
+
+`parse_vcf` read each data row from `str(variant)`. htslib normalises what it
+re-serializes: a sample written `0` under `GT:DP` comes back as `0:.`, so the
+census counted a FORMAT cell the file does not contain. That made the validator
+blind to precisely the case VCF Core models deliberately — an emitter that
+*invented* the dropped field would have looked correct.
+
+`read_vcf_data_columns` now reads the data lines from the file's own text,
+alongside the cyvcf2 reader, the same way `read_vcf_header_text` already read
+the header. The two iterators are asserted to stay aligned: a short read means
+the reader skipped a line, and a silently wrong census is worse than an error.
+
+### 7. The record decompositions are derived, not mirrored
+
+`expected_census` gained the ID, FILTER and FORMAT-key decompositions, the
+per-call `vcfc:phaseIndicator`, and `vcfc:fieldIndex` on both INFO entries and
+FORMAT keys. `record_decomposition_counts` derives the ID and FILTER component
+counts from the source columns — including the rule that `PASS` and the missing
+token are statuses rather than failure codes, so they contribute no
+`vcfc:FilterCode` — rather than asking the emitter what it produced.
+
+`typed_value_kind` now states the typed-value rule once for INFO and FORMAT
+both. The previous INFO mirror used bare `int()`/`float()`, so it would have
+counted a BCF-reserved integer and a non-finite float that the emitter
+deliberately skips.
+
 ## What the suite found
 
 Running it against the migrated emitters surfaced a real conversion defect that

docs/validation.md

@@ -97,6 +97,29 @@ vcf-rdfizer --mode full -i cohort.vcf.gz --validate \
   --validation-query-timeout 14400 --validation-time-budget 86400 -o ./out
 ```
 
+### A startup probe that fails on a healthy engine
+
+Comunica's endpoint is probed twice at startup: `_await_bind` waits for the
+port to answer at all, then `_await_warm` proves the endpoint can actually read
+the source, so that readiness is charged to setup rather than billed to the
+first query.
+
+The warm-up probe used to be a single request. Comunica answers the bind probe
+and then hands the socket to its worker, so the next connection can be refused
+for a moment while the process is perfectly healthy — and a refused connection
+comes back from the kernel immediately, not after the warm-up budget. The result
+was an engine that failed in milliseconds while reporting
+
+```
+comunica could not read /work/…/expanded.nt within 3600s
+```
+
+a timeout it had never waited for. The probe now retries until the budget is
+genuinely spent, still failing at once if the server process exits or if the
+endpoint *answers* with an error, and the duration in the message is measured
+rather than read back from the flag. If you see that message now, the wait was
+real: raise `--comunica-warmup-timeout`, or use `--engine qlever`.
+
 ### A decode that reports success and writes garbage
 
 Every decode into N-Triples is now checked structurally before anything reads
@@ -155,6 +178,15 @@ vcf-rdfizer --mode full \
 For standalone validation, the input VCF and the RDF artifact must originate
 from the same conversion.
 
+> **The validator ships inside the image.** `src/validation/` is copied into
+> the Docker image at build time, so `--mode validation` runs the image's
+> expectations, not the working tree's. A release that teaches the converter a
+> new resource family teaches the runner to expect it in the same commit — but
+> an image built before that commit will report the new predicates as
+> unexpected. Pull the image matching your wrapper version (`docker pull
+> ecrum19/vcf-rdfizer:latest`, or pin with `--image-version`) before reading a
+> `MISMATCH` on Q9 or Q10 as a conversion defect.
+
 ## Which artifact is validated
 
 `--rdf` accepts any artifact the pipeline produces: