Add an indexed regional-access arm for the retrieval comparison

[ecrum19]

Why

The whole-file retrieval comparison (13_query_cost) pits SPARQL against cyvcf2 scanning the VCF, because none of Q1–Q13 is coordinate-restricted. That is fair for whole-file questions, but the paper's conclusion that SPARQL wins on selective questions is then shown only against the one VCF access mode nobody uses for them. Real VCF work seeks through a bgzip + tabix/CSI index. This adds that arm.

What

• src/validation/regional_runner.py asks five region-restricted questions (R1 record count, R2 allele shapes, R3 Ti/Tv, R4 FILTER distribution, R5 per-sample genotype classes, expanded only). It asks each one of every access path:

  Arm	Access path
  cyvcf2-scan	whole-file iteration with a POS filter (baseline)
  cyvcf2-indexed	VCF(path)(region) over the bgzipped, indexed copy
  bcftools-indexed	bcftools query -r
  qlever, comunica, hdt, cottas	the same question as SPARQL

  It reuses validation_runner's engines and classification helpers, so a regional answer is classified by the same code as a whole-file one.

• Queries in src/validation/queries/regional/{common,expanded}/. {{CHROM}}, {{START}} and {{END}} are substituted per window, with an injection guard.

• Dockerfile adds the tabix package (bgzip, tabix), which Debian's bcftools package does not ship.

• Docs: docs/validation.md, "Indexed regional access".

Design decisions

• A window means POS, not overlap. An index seek returns records whose span overlaps the region. Every VCF arm keeps the seek, then drops records whose POS falls outside the window, so all arms agree on indels that straddle a window edge.
• Windows are anchored on real records, drawn from a fixed seed and written to windows.json.
• The scan arm is timed on fewer windows, because its cost does not depend on the window. The equality reference still covers every window. --thin-arms gives any other arm the same treatment; the default is the scan arm only.
• Equality first. Every arm is compared against a cyvcf2 reference before any timing. Disagreements go to mismatches.json and the runner exits 1.

Tested on vcf-bench-1, inside an image built from this branch

The image has htslib/tabix 1.22.1, bcftools 1.22 and cyvcf2 0.34.0. Running there found four defects, all fixed in this PR:

1. An unsorted VCF could not be indexed (c3815a4). test-1k interleaves its contigs, so tabix and the bcftools index fallback both failed with "Chromosome blocks not continuous". The original index test would have caught this, but it skips off-image and had never run. The input is now coordinate-sorted with bcftools sort when indexing reports it unsorted. The sort is charged to the VCF side's one-time cost as its own sortSeconds. Any other index failure still raises.

2. Three of four engines failed on the harness's actual input (0ca2ff6). With the .nt.gz that 14_regional_access.sh reuses from 13_query_cost, qlever-index exited 2, Comunica returned HTTP 400, and the COTTAS builder raised KeyError: '.gz', which killed the whole run. The artifact is now materialized as N-Triples once, before any engine, with validation_runner.materialize_ntriples, the step validation has always taken. It is timed as setup (rdfMaterialization).

3. Engine options were silently ignored (0ca2ff6). The runner passed its own flag names (qlever_memory_gb, …) where the engines read memory_gb, port and startup_timeout, and it never offered artifact_path/artifact_format. Engines now get the names they read, plus the native artifact, and any engine start-up failure is recorded against that engine rather than ending the run.

4. A plain-gzip VCF was compressed twice (6ecd528). The benchmark's derived slices (e.g. HG005_GRCh38_r100000.vcf.gz) are plain gzip, not BGZF. bgzip -c on one wrapped the gzip bytes in BGZF, and tabix then failed with "Failed to parse TBX_VCF". A gzip input that is not BGZF is now decompressed while being BGZF-compressed. A failing external tool is also reported as an error exit naming the command, rather than a traceback.

Smoke run (test-1k, windows 1 kb–10 Mb, 2 per size, 1 replicate):

Arms	--rdf	Executions	Failed	Disagreed
all 7	.nt.gz	260	0	0
hdt, qlever	.hdt	80	0	0
cottas, comunica	.cottas	80	0	0

With the native .cottas, COTTAS setup falls from 15.6 s to 6.1 s, because it now queries the supplied file instead of rebuilding it.

Through the harness (ecrum19/vcf-rdfizer-testing#1), on the inputs 13_query_cost already converted, 2 windows per size, 1 replicate:

Scale	Arms	Executions	Failed	Disagreed
small (test-10k, unsorted, so sorted first)	all 7	200	0	0
slice (HG005_GRCh38_r100000, plain gzip)	VCF arms + qlever	140	0	0

Median seconds per question, 10 Mb windows: bcftools-indexed 0.005 (small) / 0.042 (slice), cyvcf2-indexed 0.009 / 0.130, cyvcf2-scan 0.025 / 0.668, qlever 0.024 / 0.045; on small, Comunica 1.08, COTTAS 0.96, HDT 1.11. These are smoke numbers, not results.

Comunica on the 17M-triple slice crashed at start-up. Its 10 s ASK bind probe timed out while the server was still reading the file, and the fallback probe's redirect killed the server. That is validation_runner._await_bind, not this runner, and is left for a separate fix; the investigation runs the heavy engines at small scale only, as 13_query_cost does.

Unit tests

The runner is a standalone performance investigation, so its tests are not in CI. They are test/test_regional_runner.py and test/test_regional_runner_driver.py, named to miss the test_*_unit.py pattern, like cross_engine_agreement.py. test/README.md says how to run them. CI's discovery drops from 935 to 860 tests.

• New test/test_regional_runner_driver.py (52 tests, commit eba4371 plus additions) covers:
  • BGZF detection, TBI vs CSI selection, and the tool fallbacks, against real bgzip/tabix/bcftools
  • sorting, and the indexed arms against a real index
  • the driver end to end: outputs, the disagreement exit, failing arms, refused inputs, all three VCF arms
  • engine handling through a stand-in engine: agreement, failure, unreadable output, start-up failure, .nt.gz decode, option names
• Inside the image on bench-1: 73 regional tests at 6ecd528, all pass, none skipped (75 locally at the branch head, with the --thin-arms tests). Coverage of regional_runner.py went from 49% to 97%; the rest is import guards and missing-tool branches.
• Locally, full suite: 930 tests, 929 pass. The one failure is test_shacl_default_unit's vendored-vs-sibling-vocabulary-checkout comparison, which also fails on an untouched main with the same local vocabulary checkout; it is unrelated.

Follow-up

The harness (14_regional_access.sh, bm_run_raw, collect_metrics.py / datasets.py regional) is ecrum19/vcf-rdfizer-testing#1. It needs an image built from this branch.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!