Licence per copy, one shared bibliography; derived redistribution (study-schema-license)

[jdkent]

Roadmap item study-schema-license. It implements the licence contract (license-contract-design.md, with the user decisions of 2026-10-16). Since 2eff0d9 it is re-scoped to option 1 of parse-storage-unification.md, which the user approved after reviewing this PR: the parse and storage share one bibliography, defined once. Built on main 6464de7, which includes #40.

What changes

A licence belongs to a copy. A copy has four facts:

• route: pmc, publisher, repository, preprint_server or unrecorded.
• version: vor, am or submitted.
• license.
• access, for a copy we hold only: public, tdm_grant or unrecorded.

The copies:

• Copy is abstract, with HeldCopy (adds access) and ReportedCopy (adds reported_by) beneath it. All three are parse-only.
• ParsedPaper.text_copy is required. It is the copy the text was read from.
• ParsedPaper.reported_copies lists the copies a source names that we don't hold.
• publisher-tdm is no longer a licence id. A text-mining grant is access: tdm_grant, recorded beside the copy's own licence.

License, renamed after review. It has four fields:

• license_id (LicenseId). The values are OpenAlex's licence slugs: cc-by, cc-by-nc and so on, plus all-rights-reserved and other. SPDX names its licence identifier licenseId, but the values here are not SPDX ids, which need a version that sources rarely state.
• determination (LicenseDetermination): stated or assumed. This is SPDX's declared/concluded split written as a field.
• stated_by (MetadataSource).
• stated_as.

It lives in neuroimaging-study-storage/license.yaml, along with MetadataSource and OaStatus.

One bibliography.

• A new storage module, neuroimaging-study-storage/bibliography.yaml, holds Bibliography, ArticleIdentifiers, PreprintDoi and Correction. The vocabularies PublicationStatus, PreprintServer, CorrectionKind and StudyType move there with them.
• The parse's paper.yaml and header.yaml import this module, the same way they already import storage enums. Storage never imports the parse.
• Bibliography.license is the required License of the version of record, or of the preprint while publication_status is preprint.
• Bibliography gains oa_status and abstract_provider, and drops open_access. provider takes the MetadataSource vocabulary.
• neurostore_base_study_id moves from ArticleIdentifiers to ArtifactHeader. It is the upload pointer, not an identifier of the paper.
• Every shared attribute is marked [mvp, deterministic]. The two copies that had drifted are reconciled:
  • publication_types takes any_of [StudyType, string];
  • publication_year takes minimum_value: 1600.

Storage Study no longer has title, authors, journal, doi, publication_status, publication_year, language or study_type, and the earlier license and oa_status of this PR are gone too. Instead it gains:

• bibliography: Bibliography, which is required and replaces the title requirement;
• identifiers: ArticleIdentifiers.

Both are the parse's objects, unchanged. The licence is therefore Study.bibliography.license, the same class with no copy rule.

Storage no longer requires a publication type. study_type was required; Bibliography.publication_types is optional, as it always was in the parse. That lets a preprint-only paper, which has no PubMed types, be stored.

Nothing builds a storage Study today. pondie imports no storage model, and neurostore's ingester reads the ParsedPaper dicts directly (_bibliography, _set_retraction). The two map entries and the required bibliography are the contract for a mapper when one is written.

Language and publication types now come only from the parsed paper. Ingestion's paper_parse._bibliography writes them from PubMed; pondie's PubMed fill is deleted; neurostore persists neither. A criterion on either (English only, no reviews; 6 of the benchmark's 16 meta-analyses state one) is answered by joining the record to its parse through parse_id. If neurostore must query them, that belongs on the neurostore item. StudyType lists known types only; any other PubMed type is kept verbatim (tested).

The map. The 14 copy entries become 2 whole-object identity copies: Study.bibliography ← ParsedPaper.bibliography and Study.identifiers ← ArtifactHeader.identifiers. The License.*, Study.license and Study.oa_status entries are deleted, as are the 8 flat bibliographic entries. Map version is 0.7.0.

The extraction record carries no bibliographic fact. I deleted:

• the language and study_type add_slots;
• this PR's publication_status, license and oa_status add_slots;
• the add_class License.

The neuroimaging-study-extraction/study.yaml change matches pondie's generator output (pondie #21 at 3da9c43) semantically. All 10 module YAMLs agree with it once parsed. The root YAML differs only in the existing description gap.

Priorities. Study.bibliography and Study.identifiers are n/a, and so are the fields of the shared classes Bibliography, ArticleIdentifiers, PreprintDoi, Correction and License.

Redistribution is derived, not stored. study_schema.licensing has one function per question:

• Text: text_may_be_redistributed(text_copy). It takes the HeldCopy, so the access is always weighed with the licence.
• Abstract: abstract_may_be_redistributed(bibliography.license).

Text and abstract licences differ for about 275 of 389 sampled PMC copies, so they stay two questions.

Both functions read the LicenseId.redistributable and CopyAccess.redistribution annotations, and both fail closed: they return False and never raise. That covers:

• no licence or no copy;
• an unknown licence id or access;
• a held copy with no access;
• a redistributable licence that is not stated by a named source in a non-empty statement.

Two policy choices are built in. Counting every CC licence as redistributable, NC and ND included, is this project's policy reading, not legal advice. other, which holds Elsevier's open-access user licence, is refused on purpose.

Empty strings. stated_as and abstract take pattern: '\S', so a missing abstract or statement is null, never "".

Rules (enforced by the JSON Schema, and null-safe):

• an assumed licence is all rights reserved and names no source;
• an abstract, and only an abstract, names its provider;
• a copy names a preprint server exactly when it is on one.

The pydantic models don't evaluate LinkML rules. Ingestion's writer and neurostore's ingester must validate parses against study_schema.jsonschema at their boundaries.

Release shape

One breaking release:

• paper-parse 0.7.0 → 0.8.0;
• storage 0.8.0 → 0.9.0 (the flat bibliographic fields are removed);
• extraction 0.5.0 → 0.6.0 (language and study_type are removed, and the record forbids unknown fields);
• map 0.6.0 → 0.7.0.

The package version is unchanged, as with every earlier bump. Models and JSON Schemas are regenerated with tools/generate_models.py.

Old parses can't be read. A 0.7.0 parsed_paper fails under 0.8.0 because text_copy and Bibliography.license are now required, and nothing migrates it. Old storage Studies and records with language/study_type fail too. Release order:

1. Merge this PR.
2. Deploy neurostore with the pin bump and migration.
3. Ingestion download-copy.
4. The reshaped ingestion #120 (not its METADATA_VERSION 2).
5. The ingestion parse writes 0.8.0.
6. The re-upload (license-rerun), which also strips language/study_type from the 1,817 run records outside git.
7. Deploy pondie's pin bump once the parses on disk are 0.8.0.

Existing tests that changed

These two tests in tests/test_identifiers.py encoded the replaced flat shape:

• test_storage_study_carries_the_publication_status;
• test_storage_refuses_an_unknown_publication_status.

They now build a Study with bibliography and identifiers, and assert on Study.identifiers.publication_status, with the same assertions. The user's approval of option 1 covers this move. No other existing test was edited.

Some example files also changed:

• x7Ex4mpleA01: the header pointer, reported_copies, the renamed licence keys.
• The extraction record: the 5 bibliographic keys are dropped.
• examples/identifiers: the pointer key is dropped.
• examples/licenses/*: the new shape.

Tests

• In a fresh beast venv (.[test,generate,layouts,parquet] jsonschema setuptools, Python 3.12), generate --check is clean and 211 tests pass at 4e5c0e2 (209 at 2eff0d9). Main gives 149 passed and 1 skipped (the wheel test, which needs setuptools).

• pondie's extraction_to_storage_map, storage_parameter_priorities and field_provenance checks (PR Drop the per-point sign from the coordinate parse; derive it from the values #21 head) report nothing new compared with main.

  • The map holds 30 derivations, against 36 on main.
  • 344 fields are classified.
  • The older Study.language missing-priority gap is gone.

• After the re-review, the generated-model comparisons (which could only fail if the generator stopped inlining imports) are replaced by checks on the YAML:

  • each shared class and enum is defined in exactly one storage module;
  • the parse imports bibliography, and storage imports nothing from the parse;
  • each map copy names a source and target field that exist and hold the same class.

  An unlisted publication type is accepted in the parse and in storage. Each new test failed under its mutation: publication_types closed to StudyType; a map source_field renamed; OaStatus redefined in the parse.

• New in this re-scope:

  • storage Study.bibliography and Study.identifiers validate the parse's dumps unchanged, in both pydantic and JSON Schema;
  • the shared classes have the same fields in all three contracts;
  • the record carries no bibliographic field;
  • storage requires bibliography;
  • a year below 1600 is refused in the parse and in storage;
  • neurostore_base_study_id sits on the header and is refused in identifiers.

• Every one of these mutations makes at least one test fail:

  • Study.bibliography made optional;
  • the year bound removed;
  • Bibliography.license made optional;
  • the pointer moved back into identifiers;
  • reported_copies moved back onto Bibliography;
  • the record regaining Study.language;
  • licensing reading basis or code;
  • the text function ignoring access;
  • the abstract rule removed;
  • the stated_as pattern removed.

• The earlier mutations (redistributable set, access annotation, each rule half, text_copy required, publisher-tdm) are still covered by the same tests.

What the consumers must change

pondie #21 (pin bump; replaces the separate pondie-license item, which has nothing left to copy):

• Delete pondie/extraction/pubmed.py, _fill_from_pubmed (stages.py:685), its option (extraction/models.py:257) and the mention in record/fix/shape.py:412. The record no longer carries language or study_type.
• Update the two committed fixtures (tests/fixtures/payloads/2abntY3hQSyq, tests/fixtures/coordinate_parse/gx2zzUbbydf9), and delete or rework tests/test_pubmed_study_type.py and tests/test_record_formatting.py.
• The generator needs no change.
• No pondie code builds a storage Study, so nothing there reads Study.bibliography. A future mapper would build it from the parse as storage.Bibliography.model_validate(parse_bib.model_dump()).

Ingestion:

• download-copy: add summary.copy (route, version, access, url) on each download row from one DOWNLOAD_COPY registry. It replaces article_version. Add url_attempts[].license, and backfill with a backup.
• #120 reshape:
  • License takes license_id and determination, and has no version.
  • Drop publisher-tdm; _TDM matches become access: tdm_grant.
  • Elsevier's OA user licence becomes other.
  • LicenseSource becomes MetadataSource, including pmc_cloud.
  • Read each held copy's own statement by that download's own source.
  • Record abstract_provider, and oa_status instead of open_access.
  • Write null, not "", for a missing abstract.
  • The OpenAlex client maps OpenAlex license to our license_id.
• The parse:
  • text_copy and reported_copies go on ParsedPaper, with a no_copy_record exit reason.
  • _header writes neurostore_base_study_id, and _identifiers stops writing it.
  • Write the full License.
  • Validate against the JSON Schema before writing.
  • Count catalog metadata with a year below 1600 before re-running.

Neurostore (after #1840 and the preprint base-study PR):

• Bump the pin, and deploy before ingestion writes 0.8.0.
• _resolve_base_study reads header.neurostore_base_study_id.
• _bibliography reads the same fields, now typed by the shared model.
• BaseStudy gains license (JSONB, bibliography.license dumped whole), oa_status and abstract_provider, through a migration, set on every upload.
• Show license_id, stated_as and the determination ("stated by Crossref" / "assumed: nobody stated a licence").
• Validate uploads against the JSON Schema.
• Gate served text with text_may_be_redistributed(text_copy) and abstracts with abstract_may_be_redistributed(bibliography.license). Whether to hide the refused abstracts is still undecided (see the note in #1845).

Open questions

None. Two were settled in review:

• The extraction schema doesn't need the licence at all now.
• No pydantic copies of the rules; the JSON Schema is the boundary validator.

🤖 Generated with Claude Code

[jdkent]

code and basis are not descriptive enough names, and I have some questions, suggestions, perhaps a larger refactor as parsedpaper starts to share more with the studyschema itself, should they be merged into one object?

[jdkent]

Where else would license come from? I don't think this field is necessary

[jdkent]

(Claude) Agreed. This entry only exists because storage Study carries its own license. If the licence stays on the bibliography (your next comment), both this map entry and the storage copy go.

🤖 Generated with Claude Code

[jdkent]

(Claude) Done in 2eff0d9. Study.license and this entry are gone. So are Study.oa_status and the four License.* entries. The map now has two whole-object copies, Study.bibliography ← ParsedPaper.bibliography and Study.identifiers ← ArtifactHeader.identifiers, in place of the 14 field copies.

🤖 Generated with Claude Code

[jdkent]

License shouldn't have to be copied, if it belongs in bibliography then it should stay in the bib

[jdkent]

Should parsed paper be a strict subset of the study schema? Allowing these two representations to exist will lead to having to maintain logic to translate them

[jdkent]

(Claude) Agreed: one licence, on the bibliography. The copy came from storage Study having no bibliography of its own. How storage refers to it instead depends on your broader question in the next thread; I'll answer that there and fix both together.

🤖 Generated with Claude Code

[jdkent]

(Claude) Today they are two schemas on purpose: the parse is what was read from one paper, storage is the curated study after resolution and merging. extraction-to-storage.map.yaml joins them, and every shared field needs a copy rule there, which is the maintenance cost you describe. Options:

1. Parse as a strict subset: storage imports the parse's shared classes (Bibliography, ArticleIdentifiers, Copy) instead of redefining them, and keeps only storage-only fields of its own. The shared parts then need no translation.
2. One merged object: a parse is a partially filled storage Study.
3. Status quo.

I'm writing these up with what each costs in study_schema, pondie, ingestion and neurostore, and will link it here before changing #41. The naming question (code, basis) goes in the same write-up.

🤖 Generated with Claude Code

[jdkent]

(Claude) Done in 2eff0d9. The licence now lives only in Bibliography.license, plus each copy's own Copy.license. Storage reads it as Study.bibliography.license: the same class, held whole, with no copy rule. Study.license, Study.oa_status, their priorities and the three extraction add_slots plus add_class License are removed.

🤖 Generated with Claude Code

[jdkent]

(Claude) Done in 2eff0d9, as option 1 of the write-up you approved.

• Bibliography, ArticleIdentifiers, PreprintDoi and Correction (and their vocabularies) are now defined once, in a new storage module neuroimaging-study-storage/bibliography.yaml. The parse imports it.
• Storage Study drops its flat title, authors, journal, doi, publication_status, publication_year, language and study_type. In their place it holds bibliography and identifiers unchanged, so nothing is translated.
• The two fields that had drifted are reconciled in the shared class: publication_types is any_of [StudyType, string], and publication_year has a minimum of 1600.
• reported_copies moved onto ParsedPaper, so copies stay parse-only. neurostore_base_study_id moved to the header.
• The extraction record carries no bibliographic fact any more, so pondie Drop the per-point sign from the coordinate parse; derive it from the values #21 deletes its PubMed fill.
• The two test_identifiers.py storage tests moved onto Study.identifiers with the same assertions.

A fully merged object (option 2) was not done, because the parse is mostly provenance and history that storage should not hold.

On naming: code is renamed license_id (SPDX's licenseId; the values are OpenAlex's licence slugs), and basis is renamed determination (stated / assumed).

🤖 Generated with Claude Code

[jdkent]

Does the text license and the abstract license differ in practice? Is this a meaningful separation?

[jdkent]

(Claude) Yes, for a large share, because a licence belongs to a copy, not to the paper.

• In a sample of 389 PMC copies in our catalog, 275 were author manuscripts licensed for text mining only, while the published version is often CC BY or CC BY-NC. For example, PMC10518222 is a mining-only manuscript of a CC BY-NC article: its abstract may be redistributed, its full text may not.
• Likewise, about 47k Elsevier and 10k Wiley texts were fetched with text-mining keys. Even when the article is open access, the bytes we hold came under a mining-only grant.

So the abstract follows the paper's licence and the text follows the copy we hold, and in practice they differ often.

🤖 Generated with Claude Code

[jdkent]

(Claude) Kept as two questions, now with signatures that can't be mixed up: text_may_be_redistributed(text_copy) takes the held copy, so its access is always weighed, and abstract_may_be_redistributed(bibliography.license) takes the paper's licence. Both refuse anything malformed. A CC BY copy obtained under a TDM grant is refused for the text, while its licence still permits the abstract (tested).

🤖 Generated with Claude Code

[jdkent]

(Claude) On naming: agreed, code and basis don't say enough. Candidates: code → license_id (the normalised licence, e.g. cc-by-nc), and basis → determination (stated / assumed). On merging the parse and the study schema: see my reply in the study.yaml thread. I'm writing up the options and their downstream cost before changing this PR.

🤖 Generated with Claude Code

[jdkent]

(Claude) A write-up of the parse-versus-storage options, as promised in the study.yaml thread. Nothing in this PR changes until you pick one.

What the schemas share today. Only bibliographic data. No class is defined in both schemas, and about 10 Study slots repeat bibliography fields. In this PR's map, 12 of the 14 copy entries are plain copies. Two have already drifted, and nothing checks them:

• study_type ← publication_types, renamed and with a wider range;
• publication_year, whose 1600 minimum exists only in storage.

No consumer builds those storage fields from the map. neurostore reads the parse directly, and pondie fetches language and study type from PubMed itself.

Recommendation: option 1, the parse as a strict subset of the study schema.

• Bibliography and ArticleIdentifiers are defined once, in a storage module. The parse imports them, as it already imports 31 storage enums.
• The licence lives only in Bibliography.license. A stored study refers to it as Study.bibliography.license, with no copy.

The other options:

• Option 2, one merged object: a parse would no longer be immutable, revision history would break, and it would add about 185 derivations.
• Option 3, the status quo: the drift stays, and so does the second route to the same data.

How this PR would change under option 1:

• delete Study.license and Study.oa_status, their 6 map entries, the licence priorities, the 3 add_slots and the add_class License;
• add Study.bibliography and Study.identifiers;
• move reported_copies onto ParsedPaper, and neurostore_base_study_id into the header.

It all rides this PR's single breaking release and the one planned re-upload.

Names:

• code → license_id, after SPDX licenseId;
• basis → determination, with values stated and assumed, after SPDX's declared versus concluded licence.

The values stay OpenAlex-style slugs such as cc-by-nc.

One change to existing tests: two tests on main (tests/test_identifiers.py:85 and :93) would move onto Study.identifiers. Their assertions don't change.

Shall I re-scope this PR to option 1?

🤖 Generated with Claude Code

[jdkent]

LGTM