Licence per copy, one shared bibliography; derived redistribution (study-schema-license)

README.md

@@ -185,11 +185,44 @@ What each side may rely on:
   the version of record. A preprint server's DOIs, each with its `PreprintServer`, and
   PubMed's and PMC's records of the preprint are kept beside them in `preprint_dois`,
   `preprint_pmid` and `preprint_pmcid`. `publication_status` states which the paper has
-  (`published` or `preprint`), and pondie copies it to `Study.publication_status`.
-- **Bibliography is copied, never extracted.** pondie fills the record's `Study.title`,
-  `authors`, `doi`, `journal`, `publication_year`, `language` and `study_type` from
-  `ParsedPaper.bibliography` and the header's identifiers (`transform: copy` in
-  `extraction-to-storage.map.yaml`), and skips a paper whose `is_meta_analysis` is true.
+  (`published` or `preprint`). The upload pointer `neurostore_base_study_id` is the
+  header's own, not an identifier of the paper.
+- **A licence belongs to a copy.** A copy is four facts: where it is hosted (`route`), which
+  version it is (`version`: `vor`, `am`, `submitted`), its own `license`, and, for a copy we
+  hold, how we obtained it (`access`: `public`, `tdm_grant`, `unrecorded`).
+  `ParsedPaper.text_copy` is the copy the text was read from; `ParsedPaper.reported_copies`
+  are copies a source lists and we do not hold. Copies are the parse's alone: storage holds
+  no copy. `Bibliography.license` is the paper's
+  licence: the version of record's, or the preprint's while `publication_status` is
+  `preprint`. A licence names its `license_id` (OpenAlex's licence slugs, `cc-by` and the
+  rest) and its `determination`: `stated` by a named source, or `assumed` all rights reserved when
+  every source that could state one answered and none did. A text-mining grant is an
+  access, not a licence: a PMC author manuscript under `TDM` terms is `pmc` / `am` /
+  `tdm_grant` with its own all-rights-reserved licence, and the article's CC licence sits
+  in `Bibliography.license` beside it.
+- **Whether a text may be passed on is derived, never stored.**
+  `study_schema.licensing` reads the `redistributable` annotation of `LicenseId` and the
+  `redistribution` annotation of `CopyAccess`, one function per question. The text follows
+  its copy: `text_may_be_redistributed(text_copy)` takes the held copy, so its access is
+  always weighed. The abstract follows the paper:
+  `abstract_may_be_redistributed(bibliography.license)`, and
+  `Bibliography.abstract_provider` names where the abstract came from. Anything a function
+  does not recognise as permitted -- no licence, an unknown licence id or access, a
+  redistributable licence nobody stated -- is refused.
+  Counting every CC licence as redistributable, NC and ND included, is this project's
+  policy, not legal advice; `other`, which holds Elsevier's open-access user licence, is
+  refused on purpose. A missing abstract or licence statement is null, never `""`.
+- **The bibliography is defined once, and never extracted.** `Bibliography` (title,
+  authors, journal, year, abstract, PubMed types, the paper's licence, corrections) and
+  `ArticleIdentifiers` live in the storage schema's `bibliography` module, and the parse
+  imports them. The storage `Study` holds the parsed paper's two objects unchanged, as
+  `Study.bibliography` and `Study.identifiers` (two whole-object `copy` entries in
+  `extraction-to-storage.map.yaml`), so no field has a second definition or a translation
+  rule. The extraction record carries none of them: a model is never asked for a
+  bibliographic fact, and pondie skips a paper whose `is_meta_analysis` is true. Language
+  and publication types come only from the parsed paper, which ingestion fills from
+  PubMed, so a criterion on either (English only, no reviews) is answered by joining the
+  record to its parse through `parse_id`; neurostore persists neither.
 - **Keys come from what was extracted, not where it was read.** An analysis is
   `analysis#<h>`, where `h` hashes its normalized name, its space and its sorted points
   (coordinates and values); `ParsedAnalysis.key` gives the recipe. The same analysis read
@@ -328,7 +361,14 @@ result the text states. The tests read them through every model, JSON Schema and
 the checks must find nothing wrong with them. `examples/identifiers/` holds header
 identifiers for a preprint-only paper and for a published paper with a preprint DOI, and
 under `invalid/` two the models refuse: one without `publication_status`, one with a preprint
-DOI that names no server.
+DOI that names no server. The example paper's text is an open PMC copy. `examples/licenses/`
+holds the copy and bibliography of a text-mining-only PMC manuscript of a CC BY-NC article,
+a preprint-only paper, and a paper nobody states a licence for, and under `invalid/` the
+shapes the models or the JSON Schema refuse, `publisher-tdm` as a licence id among them.
+The JSON Schema enforces the schema's LinkML rules (an assumed licence names no source, an
+abstract names its provider, a copy names a preprint server exactly when it is on one); the
+pydantic models do not evaluate rules, so a producer and a consumer of a parse validate it
+against `study_schema.jsonschema` at their boundary.
 
 **Storing coordinates.** The models say what a parse means, not how a corpus of them is
 kept. `study_schema.parquet` stores coordinate parses as three Parquet tables -- `parses`,

examples/corpus/x7Ex4mpleA01/parse/coordinate_parse.json

@@ -1,7 +1,7 @@
 {
   "header": {
     "artifact_kind": "coordinate_parse",
-    "schema_version": "0.7.0",
+    "schema_version": "0.8.0",
     "article_id": "art-000001",
     "identifiers": {
       "pmid": "00000000",
@@ -10,8 +10,7 @@
       "preprint_dois": null,
       "preprint_pmid": null,
       "preprint_pmcid": null,
-      "publication_status": "published",
-      "neurostore_base_study_id": "x7Ex4mpleA01"
+      "publication_status": "published"
     },
     "producer": {
       "name": "ns-pond-ingestion-workflow",
@@ -21,7 +20,8 @@
       "prompt_version": null
     },
     "inputs": null,
-    "created_at": "2026-10-09T12:00:00Z"
+    "created_at": "2026-10-09T12:00:00Z",
+    "neurostore_base_study_id": "x7Ex4mpleA01"
   },
   "parse_id": "c106973d3c1eeda6baffcf4413499931f42e026ce7eeb22ed6c3ea4114cbd9fe",
   "text_sha256": "a44901241d8884e85cfead36b324ccb4dba1b0213436ed2a8d216cb8c3803da5",

examples/corpus/x7Ex4mpleA01/parse/parsed_paper.json

@@ -1,7 +1,7 @@
 {
   "header": {
     "artifact_kind": "parsed_paper",
-    "schema_version": "0.7.0",
+    "schema_version": "0.8.0",
     "article_id": "art-000001",
     "identifiers": {
       "pmid": "00000000",
@@ -10,8 +10,7 @@
       "preprint_dois": null,
       "preprint_pmid": null,
       "preprint_pmcid": null,
-      "publication_status": "published",
-      "neurostore_base_study_id": "x7Ex4mpleA01"
+      "publication_status": "published"
     },
     "producer": {
       "name": "ns-pond-ingestion-workflow",
@@ -21,10 +20,25 @@
       "prompt_version": null
     },
     "inputs": null,
-    "created_at": "2026-10-09T12:00:00Z"
+    "created_at": "2026-10-09T12:00:00Z",
+    "neurostore_base_study_id": "x7Ex4mpleA01"
   },
   "source": "pubget",
   "source_reason": "the only render fetched",
+  "text_copy": {
+    "route": "pmc",
+    "version": "vor",
+    "license": {
+      "license_id": "cc-by",
+      "determination": "stated",
+      "stated_by": "jats",
+      "stated_as": "https://creativecommons.org/licenses/by/4.0/"
+    },
+    "url": null,
+    "preprint_server": null,
+    "access": "public"
+  },
+  "reported_copies": null,
   "text_path": "processed/pubget/text.txt",
   "text_sha256": "a44901241d8884e85cfead36b324ccb4dba1b0213436ed2a8d216cb8c3803da5",
   "text_length": null,
@@ -40,17 +54,23 @@
     "issue": null,
     "pages": null,
     "abstract": null,
+    "abstract_provider": null,
     "language": [
       "eng"
     ],
     "publication_types": [
       "Journal Article"
     ],
     "keywords": null,
-    "license": null,
-    "open_access": true,
+    "license": {
+      "license_id": "cc-by",
+      "determination": "stated",
+      "stated_by": "jats",
+      "stated_as": "https://creativecommons.org/licenses/by/4.0/"
+    },
+    "oa_status": "gold",
     "corrections": null,
-    "provider": "example"
+    "provider": "pubmed"
   },
   "is_meta_analysis": false,
   "is_meta_analysis_basis": null,

examples/identifiers/invalid/missing_publication_status.json

@@ -9,6 +9,5 @@
     }
   ],
   "preprint_pmid": null,
-  "preprint_pmcid": null,
-  "neurostore_base_study_id": null
+  "preprint_pmcid": null
 }

examples/identifiers/invalid/preprint_doi_without_server.json

@@ -9,6 +9,5 @@
   ],
   "preprint_pmid": null,
   "preprint_pmcid": null,
-  "publication_status": "published",
-  "neurostore_base_study_id": null
+  "publication_status": "published"
 }

examples/identifiers/preprint_only.json

@@ -10,6 +10,5 @@
   ],
   "preprint_pmid": "40462995",
   "preprint_pmcid": "PMC12132522",
-  "publication_status": "preprint",
-  "neurostore_base_study_id": null
+  "publication_status": "preprint"
 }

examples/identifiers/published_with_preprint.json

@@ -10,6 +10,5 @@
   ],
   "preprint_pmid": null,
   "preprint_pmcid": null,
-  "publication_status": "published",
-  "neurostore_base_study_id": null
+  "publication_status": "published"
 }

examples/licenses/assumed_all_rights_reserved.json

@@ -0,0 +1,32 @@
+{
+  "identifiers": {
+    "pmid": "15016991",
+    "pmcid": null,
+    "doi": "10.1126/science.1089506",
+    "preprint_dois": null,
+    "preprint_pmid": null,
+    "preprint_pmcid": null,
+    "publication_status": "published"
+  },
+  "text_copy": {
+    "route": "publisher",
+    "version": "vor",
+    "access": "public",
+    "url": "https://doi.org/10.1126/science.1089506",
+    "license": {
+      "license_id": "all-rights-reserved",
+      "determination": "assumed"
+    }
+  },
+  "bibliography": {
+    "title": "Intersubject synchronization of cortical activity during natural vision",
+    "journal": "Science",
+    "publication_year": 2004,
+    "license": {
+      "license_id": "all-rights-reserved",
+      "determination": "assumed"
+    },
+    "oa_status": "closed",
+    "provider": "pubmed"
+  }
+}

examples/licenses/invalid/abstract_without_provider.json

@@ -0,0 +1,51 @@
+{
+  "identifiers": {
+    "pmid": "35820077",
+    "pmcid": "PMC10518222",
+    "doi": "10.1002/alz.12697",
+    "preprint_dois": null,
+    "preprint_pmid": null,
+    "preprint_pmcid": null,
+    "publication_status": "published"
+  },
+  "text_copy": {
+    "route": "pmc",
+    "version": "am",
+    "access": "tdm_grant",
+    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/",
+    "license": {
+      "license_id": "all-rights-reserved",
+      "determination": "stated",
+      "stated_by": "pmc_cloud",
+      "stated_as": "TDM"
+    }
+  },
+  "reported_copies": [
+    {
+      "route": "publisher",
+      "version": "vor",
+      "reported_by": "openalex",
+      "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697",
+      "license": {
+        "license_id": "cc-by-nc",
+        "determination": "stated",
+        "stated_by": "openalex",
+        "stated_as": "cc-by-nc"
+      }
+    }
+  ],
+  "bibliography": {
+    "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity",
+    "journal": "Alzheimer's & Dementia",
+    "publication_year": 2022,
+    "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.",
+    "license": {
+      "license_id": "cc-by-nc",
+      "determination": "stated",
+      "stated_by": "crossref",
+      "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/"
+    },
+    "oa_status": "hybrid",
+    "provider": "semantic_scholar"
+  }
+}

examples/licenses/invalid/assumed_with_stated_by.json

@@ -0,0 +1,51 @@
+{
+  "identifiers": {
+    "pmid": "35820077",
+    "pmcid": "PMC10518222",
+    "doi": "10.1002/alz.12697",
+    "preprint_dois": null,
+    "preprint_pmid": null,
+    "preprint_pmcid": null,
+    "publication_status": "published"
+  },
+  "text_copy": {
+    "route": "pmc",
+    "version": "am",
+    "access": "tdm_grant",
+    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/",
+    "license": {
+      "license_id": "all-rights-reserved",
+      "determination": "stated",
+      "stated_by": "pmc_cloud",
+      "stated_as": "TDM"
+    }
+  },
+  "reported_copies": [
+    {
+      "route": "publisher",
+      "version": "vor",
+      "reported_by": "openalex",
+      "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697",
+      "license": {
+        "license_id": "cc-by-nc",
+        "determination": "stated",
+        "stated_by": "openalex",
+        "stated_as": "cc-by-nc"
+      }
+    }
+  ],
+  "bibliography": {
+    "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity",
+    "journal": "Alzheimer's & Dementia",
+    "publication_year": 2022,
+    "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.",
+    "abstract_provider": "pubmed",
+    "license": {
+      "license_id": "all-rights-reserved",
+      "determination": "assumed",
+      "stated_by": "crossref"
+    },
+    "oa_status": "hybrid",
+    "provider": "semantic_scholar"
+  }
+}

examples/licenses/invalid/held_copy_without_access.json

@@ -0,0 +1,51 @@
+{
+  "identifiers": {
+    "pmid": "35820077",
+    "pmcid": "PMC10518222",
+    "doi": "10.1002/alz.12697",
+    "preprint_dois": null,
+    "preprint_pmid": null,
+    "preprint_pmcid": null,
+    "publication_status": "published"
+  },
+  "text_copy": {
+    "route": "pmc",
+    "version": "am",
+    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/",
+    "license": {
+      "license_id": "all-rights-reserved",
+      "determination": "stated",
+      "stated_by": "pmc_cloud",
+      "stated_as": "TDM"
+    }
+  },
+  "reported_copies": [
+    {
+      "route": "publisher",
+      "version": "vor",
+      "reported_by": "openalex",
+      "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697",
+      "license": {
+        "license_id": "cc-by-nc",
+        "determination": "stated",
+        "stated_by": "openalex",
+        "stated_as": "cc-by-nc"
+      }
+    }
+  ],
+  "bibliography": {
+    "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity",
+    "journal": "Alzheimer's & Dementia",
+    "publication_year": 2022,
+    "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.",
+    "abstract_provider": "pubmed",
+    "license": {
+      "license_id": "cc-by-nc",
+      "determination": "stated",
+      "stated_by": "crossref",
+      "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/"
+    },
+    "oa_status": "hybrid",
+    "provider": "semantic_scholar"
+  }
+}