Preprint ids in ArticleIdentifiers and publication_status in header and storage (preprint-ids decision 4)

[jdkent]

Implements decision 4 of the preprint-identifiers design (reports/preprint-ids-design.md §5, work-graph item study-schema-preprint-ids). User request, 2026-10-16: "I want preprint status to be clear ... on study_schema as well, which will require a separate PR."

Schema change

Paper-parse header (header.yaml, 0.6.0 → 0.7.0). ArticleIdentifiers:

• doi, pmid and pmcid are now described as the version of record's, and never a preprint server's DOI or PubMed's or PMC's record of a preprint.
• New preprint_dois: a list of PreprintDoi {doi (required), server: PreprintServer (required)}. Each separately registered version (Research Square /vN, OSF _vN) is its own entry, kept exactly as registered.
• New preprint_pmid and preprint_pmcid: PubMed's record of the preprint (publication type Preprint) and PMC's copy of it.
• New publication_status: PublicationStatus, required. It is preprint when a preprint_ field is set and doi, pmid and pmcid are all empty, and published otherwise. The producer states it.

parse_enums.yaml gains PreprintServer with 31 values: biorxiv, medrxiv, arxiv, researchsquare, ssrn, preprints_org, authorea, techrxiv, chemrxiv, jmir_preprints, plus one value per OSF-hosted provider:

• osf_preprints 10.31219, arabixiv 31221, metaarxiv 31222, engrxiv 31224, focus_archive 31225
• frenxiv 31226, ina_rxiv 31227, lawarxiv 31228, lis_scholarship_archive 31229, marxiv 31230
• mindrxiv 31231, nutrixiv 31232, paleorxiv 31233, psyarxiv 31234, socarxiv 31235
• sportrxiv 31236, africarxiv 31730, mediarxiv 33767, bodoarxiv 34055, edarxiv 35542, biohackrxiv 37044

Each provider name comes from the Crossref group-title of works under its prefix, checked 2026-10-16. The ingestion registry (preprint-registry) should name servers with exactly these values.

Storage (0.7.0 → 0.8.0).

• Study.doi is described as the version of record's DOI.
• Study.publication_status (mvp, deterministic, optional) is new. The extraction-to-storage map gets a copy derivation from ArticleIdentifiers.publication_status, and storage-parameter-priorities.yaml gets publication_status: n/a.
• study_type is unchanged and keeps PubMed's Preprint verbatim.

Bibliography (including license and the separate licence proposal from ingestion #120), Correction and Reference are not touched.

Where this differs from the design, and why

1. PublicationStatus lives in storage's analysis_enums.yaml, not parse_enums.yaml. One vocabulary has to serve both the header and Study. The repo already does this for CoordinateRole and AnchorKind: the parse schema imports storage's analysis_enums, and storage imports nothing from the parse schema. Defining it in parse_enums would mean storage importing every parse enum. Defining it beside StudyType in storage study.yaml would mean the parse header importing the whole storage Study tree. header.yaml now imports ../neuroimaging-study-storage/analysis_enums, as coordinates.yaml already did. A test asserts that the two generated enums have the same values.
2. Study.publication_status is optional in storage. The design marks only the header field required. Storage records built from 0.6.0 headers have nothing to copy. Making it required is a one-line change if wanted.
3. No value for 10.32943. The design's OSF regex lists 10.32943 (Center for Open Science), but Crossref has 0 works under it, so it has no provider name. The registry should drop it or name it; this vocabulary adds the value when it does.
4. The design's arXiv, SSRN and related notes (DOI patterns, Crossref's journal-article type for SSRN) stay in the ingestion registry. The enum descriptions only give the server and its DOI prefix, so the patterns are not kept in two places.

Mappability

Every new storage field has a parse analog: Study.publication_status ← ArticleIdentifiers.publication_status, and Study.doi ← ArticleIdentifiers.doi (unchanged). The preprint ids have no storage field. Neurostore keeps them on the base study (neurostore-preprint-base-study).

Release and compatibility

Adding a required header field breaks the paper-parse contract. A 0.6.0 header no longer validates against these models, and layouts refuses a schema_version mismatch.

Downstream order, per design §5:

1. This PR, merged to main, alongside the ingestion preprint-registry PR, which is independent of the contract and should use the PreprintServer values above.
2. neurostore (neurostore-preprint-base-study, after neurostore#1843): pin this main sha. Accept 0.7.0 headers. Match base studies on the version-of-record ids and then on the preprint ids. Promote the version-of-record ids. Store publication_status. This must deploy before ingestion writes 0.7.0 headers, otherwise uploads fail loudly (they do not corrupt anything).
3. ingestion (preprint-ids-catalog): pin this sha. paper_parse._identifiers writes the new fields and publication_status. identifiers.json becomes ArticleIdentifiers.
4. ingestion identity stage.
5. pondie (pondie-publication-status): pin this sha and copy Study.publication_status from the header. Its submodule and checks still use the pre-Ship the LinkML source with the package #37 layout (see Tests). It can start any time after step 1.

The extraction schema is unchanged (0.5.0). Study.publication_status is deterministic, so the projection drops it.

Tests

The suite ran in fresh venvs on beast, installed with .[test,generate,layouts,parquet] jsonschema setuptools:

• python tools/generate_models.py --check is clean.
• pytest: 150 passed on Python 3.13 and 3.11, against 139 passed at base 8e15224, with no failures or skips.
• New tests/test_identifiers.py (11 tests):
  • both valid examples validate through ParsedPaper and the parsed-paper JSON Schema, and their status agrees with the documented rule;
  • both invalid examples are refused by the model and by the JSON Schema, each naming its field;
  • a server outside the vocabulary is refused;
  • header and storage share one PublicationStatus;
  • storage accepts both statuses and refuses an unknown one.

Mutations: each was applied to the YAML, followed by a regeneration and the full suite. Every one was killed:

mutation	failing tests
header publication_status not required	1
PreprintDoi.server not required	1
preprint_dois as plain strings	5
preprint_pmid removed	23, plus 4 errors
psyarxiv removed	2
storage publication_status renamed	2
storage PublicationStatus gains in_press	2
preprint_only.json says published	2

pondie schema checks were run with pondie main d722f69 and PONDIE_SCHEMA_DIR set to a copy of this branch's src/study_schema/linkml plus storage-parameter-priorities.yaml, then diffed against the same run at base:

• generate: the projected extraction tree is byte-identical to the base projection. The only changes are in the summary: 23 fields dropped as deterministic instead of 22, and PublicationStatus joins the enums no extracted field reaches.
• extraction_to_storage_map: 36 derivations instead of 35, and no new problems.
• field_provenance: 318 fields instead of 317.
• linkml and storage_parameter_priorities: no new findings.

Some pondie checks already fail at base, for reasons this PR does not touch:

• generate --check reports the committed extraction tree out of date with pondie main.
• The map check lists 5 deterministic fields with no derivation.
• 16 fields are missing priorities.

Examples:

• x7Ex4mpleA01: the three headers move to 0.7.0, with publication_status: published and empty preprint fields. They still round-trip byte for byte.
• examples/identifiers/preprint_only.json: bioRxiv 10.1101/2025.05.13.653896, with PubMed's preprint record 40462995 and PMC12132522 (catalog article qzfriham4plz).
• examples/identifiers/published_with_preprint.json: Dev Sci 10.1111/desc.12933, PMID 31863619, with PsyArXiv 10.31234/osf.io/5m87x (uqzotozf6jhu).
• examples/identifiers/invalid/: two files, one missing publication_status and one with a preprint DOI that has no server.

tools/benchmark_parquet.py now passes publication_status when it builds headers. The README gains a "preprint is not its version of record" bullet and describes the new examples.

🤖 Generated with Claude Code

[jdkent]

LGTM