diff --git a/README.md b/README.md index 87236fe..a1a9713 100644 --- a/README.md +++ b/README.md @@ -185,11 +185,44 @@ What each side may rely on: the version of record. A preprint server's DOIs, each with its `PreprintServer`, and PubMed's and PMC's records of the preprint are kept beside them in `preprint_dois`, `preprint_pmid` and `preprint_pmcid`. `publication_status` states which the paper has - (`published` or `preprint`), and pondie copies it to `Study.publication_status`. -- **Bibliography is copied, never extracted.** pondie fills the record's `Study.title`, - `authors`, `doi`, `journal`, `publication_year`, `language` and `study_type` from - `ParsedPaper.bibliography` and the header's identifiers (`transform: copy` in - `extraction-to-storage.map.yaml`), and skips a paper whose `is_meta_analysis` is true. + (`published` or `preprint`). The upload pointer `neurostore_base_study_id` is the + header's own, not an identifier of the paper. +- **A licence belongs to a copy.** A copy is four facts: where it is hosted (`route`), which + version it is (`version`: `vor`, `am`, `submitted`), its own `license`, and, for a copy we + hold, how we obtained it (`access`: `public`, `tdm_grant`, `unrecorded`). + `ParsedPaper.text_copy` is the copy the text was read from; `ParsedPaper.reported_copies` + are copies a source lists and we do not hold. Copies are the parse's alone: storage holds + no copy. `Bibliography.license` is the paper's + licence: the version of record's, or the preprint's while `publication_status` is + `preprint`. A licence names its `license_id` (OpenAlex's licence slugs, `cc-by` and the + rest) and its `determination`: `stated` by a named source, or `assumed` all rights reserved when + every source that could state one answered and none did. A text-mining grant is an + access, not a licence: a PMC author manuscript under `TDM` terms is `pmc` / `am` / + `tdm_grant` with its own all-rights-reserved licence, and the article's CC licence sits + in `Bibliography.license` beside it. +- **Whether a text may be passed on is derived, never stored.** + `study_schema.licensing` reads the `redistributable` annotation of `LicenseId` and the + `redistribution` annotation of `CopyAccess`, one function per question. The text follows + its copy: `text_may_be_redistributed(text_copy)` takes the held copy, so its access is + always weighed. The abstract follows the paper: + `abstract_may_be_redistributed(bibliography.license)`, and + `Bibliography.abstract_provider` names where the abstract came from. Anything a function + does not recognise as permitted -- no licence, an unknown licence id or access, a + redistributable licence nobody stated -- is refused. + Counting every CC licence as redistributable, NC and ND included, is this project's + policy, not legal advice; `other`, which holds Elsevier's open-access user licence, is + refused on purpose. A missing abstract or licence statement is null, never `""`. +- **The bibliography is defined once, and never extracted.** `Bibliography` (title, + authors, journal, year, abstract, PubMed types, the paper's licence, corrections) and + `ArticleIdentifiers` live in the storage schema's `bibliography` module, and the parse + imports them. The storage `Study` holds the parsed paper's two objects unchanged, as + `Study.bibliography` and `Study.identifiers` (two whole-object `copy` entries in + `extraction-to-storage.map.yaml`), so no field has a second definition or a translation + rule. The extraction record carries none of them: a model is never asked for a + bibliographic fact, and pondie skips a paper whose `is_meta_analysis` is true. Language + and publication types come only from the parsed paper, which ingestion fills from + PubMed, so a criterion on either (English only, no reviews) is answered by joining the + record to its parse through `parse_id`; neurostore persists neither. - **Keys come from what was extracted, not where it was read.** An analysis is `analysis#`, where `h` hashes its normalized name, its space and its sorted points (coordinates and values); `ParsedAnalysis.key` gives the recipe. The same analysis read @@ -328,7 +361,14 @@ result the text states. The tests read them through every model, JSON Schema and the checks must find nothing wrong with them. `examples/identifiers/` holds header identifiers for a preprint-only paper and for a published paper with a preprint DOI, and under `invalid/` two the models refuse: one without `publication_status`, one with a preprint -DOI that names no server. +DOI that names no server. The example paper's text is an open PMC copy. `examples/licenses/` +holds the copy and bibliography of a text-mining-only PMC manuscript of a CC BY-NC article, +a preprint-only paper, and a paper nobody states a licence for, and under `invalid/` the +shapes the models or the JSON Schema refuse, `publisher-tdm` as a licence id among them. +The JSON Schema enforces the schema's LinkML rules (an assumed licence names no source, an +abstract names its provider, a copy names a preprint server exactly when it is on one); the +pydantic models do not evaluate rules, so a producer and a consumer of a parse validate it +against `study_schema.jsonschema` at their boundary. **Storing coordinates.** The models say what a parse means, not how a corpus of them is kept. `study_schema.parquet` stores coordinate parses as three Parquet tables -- `parses`, diff --git a/examples/corpus/x7Ex4mpleA01/parse/coordinate_parse.json b/examples/corpus/x7Ex4mpleA01/parse/coordinate_parse.json index 54becbd..4e67b15 100644 --- a/examples/corpus/x7Ex4mpleA01/parse/coordinate_parse.json +++ b/examples/corpus/x7Ex4mpleA01/parse/coordinate_parse.json @@ -1,7 +1,7 @@ { "header": { "artifact_kind": "coordinate_parse", - "schema_version": "0.7.0", + "schema_version": "0.8.0", "article_id": "art-000001", "identifiers": { "pmid": "00000000", @@ -10,8 +10,7 @@ "preprint_dois": null, "preprint_pmid": null, "preprint_pmcid": null, - "publication_status": "published", - "neurostore_base_study_id": "x7Ex4mpleA01" + "publication_status": "published" }, "producer": { "name": "ns-pond-ingestion-workflow", @@ -21,7 +20,8 @@ "prompt_version": null }, "inputs": null, - "created_at": "2026-10-09T12:00:00Z" + "created_at": "2026-10-09T12:00:00Z", + "neurostore_base_study_id": "x7Ex4mpleA01" }, "parse_id": "c106973d3c1eeda6baffcf4413499931f42e026ce7eeb22ed6c3ea4114cbd9fe", "text_sha256": "a44901241d8884e85cfead36b324ccb4dba1b0213436ed2a8d216cb8c3803da5", diff --git a/examples/corpus/x7Ex4mpleA01/parse/parsed_paper.json b/examples/corpus/x7Ex4mpleA01/parse/parsed_paper.json index dd66844..fa5fd7f 100644 --- a/examples/corpus/x7Ex4mpleA01/parse/parsed_paper.json +++ b/examples/corpus/x7Ex4mpleA01/parse/parsed_paper.json @@ -1,7 +1,7 @@ { "header": { "artifact_kind": "parsed_paper", - "schema_version": "0.7.0", + "schema_version": "0.8.0", "article_id": "art-000001", "identifiers": { "pmid": "00000000", @@ -10,8 +10,7 @@ "preprint_dois": null, "preprint_pmid": null, "preprint_pmcid": null, - "publication_status": "published", - "neurostore_base_study_id": "x7Ex4mpleA01" + "publication_status": "published" }, "producer": { "name": "ns-pond-ingestion-workflow", @@ -21,10 +20,25 @@ "prompt_version": null }, "inputs": null, - "created_at": "2026-10-09T12:00:00Z" + "created_at": "2026-10-09T12:00:00Z", + "neurostore_base_study_id": "x7Ex4mpleA01" }, "source": "pubget", "source_reason": "the only render fetched", + "text_copy": { + "route": "pmc", + "version": "vor", + "license": { + "license_id": "cc-by", + "determination": "stated", + "stated_by": "jats", + "stated_as": "https://creativecommons.org/licenses/by/4.0/" + }, + "url": null, + "preprint_server": null, + "access": "public" + }, + "reported_copies": null, "text_path": "processed/pubget/text.txt", "text_sha256": "a44901241d8884e85cfead36b324ccb4dba1b0213436ed2a8d216cb8c3803da5", "text_length": null, @@ -40,6 +54,7 @@ "issue": null, "pages": null, "abstract": null, + "abstract_provider": null, "language": [ "eng" ], @@ -47,10 +62,15 @@ "Journal Article" ], "keywords": null, - "license": null, - "open_access": true, + "license": { + "license_id": "cc-by", + "determination": "stated", + "stated_by": "jats", + "stated_as": "https://creativecommons.org/licenses/by/4.0/" + }, + "oa_status": "gold", "corrections": null, - "provider": "example" + "provider": "pubmed" }, "is_meta_analysis": false, "is_meta_analysis_basis": null, diff --git a/examples/identifiers/invalid/missing_publication_status.json b/examples/identifiers/invalid/missing_publication_status.json index deef6cc..e74134b 100644 --- a/examples/identifiers/invalid/missing_publication_status.json +++ b/examples/identifiers/invalid/missing_publication_status.json @@ -9,6 +9,5 @@ } ], "preprint_pmid": null, - "preprint_pmcid": null, - "neurostore_base_study_id": null + "preprint_pmcid": null } diff --git a/examples/identifiers/invalid/preprint_doi_without_server.json b/examples/identifiers/invalid/preprint_doi_without_server.json index fdd3c95..904eb84 100644 --- a/examples/identifiers/invalid/preprint_doi_without_server.json +++ b/examples/identifiers/invalid/preprint_doi_without_server.json @@ -9,6 +9,5 @@ ], "preprint_pmid": null, "preprint_pmcid": null, - "publication_status": "published", - "neurostore_base_study_id": null + "publication_status": "published" } diff --git a/examples/identifiers/preprint_only.json b/examples/identifiers/preprint_only.json index d9c5c01..92f6776 100644 --- a/examples/identifiers/preprint_only.json +++ b/examples/identifiers/preprint_only.json @@ -10,6 +10,5 @@ ], "preprint_pmid": "40462995", "preprint_pmcid": "PMC12132522", - "publication_status": "preprint", - "neurostore_base_study_id": null + "publication_status": "preprint" } diff --git a/examples/identifiers/published_with_preprint.json b/examples/identifiers/published_with_preprint.json index 58fb921..20b73a0 100644 --- a/examples/identifiers/published_with_preprint.json +++ b/examples/identifiers/published_with_preprint.json @@ -10,6 +10,5 @@ ], "preprint_pmid": null, "preprint_pmcid": null, - "publication_status": "published", - "neurostore_base_study_id": null + "publication_status": "published" } diff --git a/examples/licenses/assumed_all_rights_reserved.json b/examples/licenses/assumed_all_rights_reserved.json new file mode 100644 index 0000000..db21805 --- /dev/null +++ b/examples/licenses/assumed_all_rights_reserved.json @@ -0,0 +1,32 @@ +{ + "identifiers": { + "pmid": "15016991", + "pmcid": null, + "doi": "10.1126/science.1089506", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "publisher", + "version": "vor", + "access": "public", + "url": "https://doi.org/10.1126/science.1089506", + "license": { + "license_id": "all-rights-reserved", + "determination": "assumed" + } + }, + "bibliography": { + "title": "Intersubject synchronization of cortical activity during natural vision", + "journal": "Science", + "publication_year": 2004, + "license": { + "license_id": "all-rights-reserved", + "determination": "assumed" + }, + "oa_status": "closed", + "provider": "pubmed" + } +} diff --git a/examples/licenses/invalid/abstract_without_provider.json b/examples/licenses/invalid/abstract_without_provider.json new file mode 100644 index 0000000..c578a8b --- /dev/null +++ b/examples/licenses/invalid/abstract_without_provider.json @@ -0,0 +1,51 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/invalid/assumed_with_stated_by.json b/examples/licenses/invalid/assumed_with_stated_by.json new file mode 100644 index 0000000..0abfd9d --- /dev/null +++ b/examples/licenses/invalid/assumed_with_stated_by.json @@ -0,0 +1,51 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": { + "license_id": "all-rights-reserved", + "determination": "assumed", + "stated_by": "crossref" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/invalid/held_copy_without_access.json b/examples/licenses/invalid/held_copy_without_access.json new file mode 100644 index 0000000..cd6bb83 --- /dev/null +++ b/examples/licenses/invalid/held_copy_without_access.json @@ -0,0 +1,51 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/invalid/license_as_string.json b/examples/licenses/invalid/license_as_string.json new file mode 100644 index 0000000..aff773d --- /dev/null +++ b/examples/licenses/invalid/license_as_string.json @@ -0,0 +1,47 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": "cc-by-nc", + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/invalid/preprint_copy_without_server.json b/examples/licenses/invalid/preprint_copy_without_server.json new file mode 100644 index 0000000..69feddd --- /dev/null +++ b/examples/licenses/invalid/preprint_copy_without_server.json @@ -0,0 +1,51 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "preprint_server", + "version": "submitted", + "reported_by": "crossref", + "license": { + "license_id": "cc-by", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by/4.0/" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/invalid/publisher_tdm_code.json b/examples/licenses/invalid/publisher_tdm_code.json new file mode 100644 index 0000000..f7fe94f --- /dev/null +++ b/examples/licenses/invalid/publisher_tdm_code.json @@ -0,0 +1,52 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "publisher-tdm", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/invalid/server_named_off_a_preprint_server.json b/examples/licenses/invalid/server_named_off_a_preprint_server.json new file mode 100644 index 0000000..f9bfb1a --- /dev/null +++ b/examples/licenses/invalid/server_named_off_a_preprint_server.json @@ -0,0 +1,53 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + }, + "preprint_server": "biorxiv" + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/licenses/preprint_only.json b/examples/licenses/preprint_only.json new file mode 100644 index 0000000..94f0702 --- /dev/null +++ b/examples/licenses/preprint_only.json @@ -0,0 +1,56 @@ +{ + "identifiers": { + "pmid": null, + "pmcid": null, + "doi": null, + "preprint_dois": [ + { + "doi": "10.1101/2025.05.13.653896", + "server": "biorxiv" + } + ], + "preprint_pmid": "40462995", + "preprint_pmcid": "PMC12132522", + "publication_status": "preprint" + }, + "text_copy": { + "route": "pmc", + "version": "submitted", + "access": "public", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC12132522/", + "license": { + "license_id": "cc-by-nc-nd", + "determination": "stated", + "stated_by": "jats", + "stated_as": "https://creativecommons.org/licenses/by-nc-nd/4.0/" + } + }, + "reported_copies": [ + { + "route": "preprint_server", + "preprint_server": "biorxiv", + "version": "submitted", + "reported_by": "crossref", + "url": "https://doi.org/10.1101/2025.05.13.653896", + "license": { + "license_id": "cc-by-nc-nd", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc-nd/4.0/" + } + } + ], + "bibliography": { + "title": "Direct interoceptive input to the insular cortex shapes learned feeding behavior", + "journal": "bioRxiv", + "publication_year": 2025, + "license": { + "license_id": "cc-by-nc-nd", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc-nd/4.0/" + }, + "oa_status": "green", + "provider": "pubmed" + } +} diff --git a/examples/licenses/tdm_manuscript_of_open_article.json b/examples/licenses/tdm_manuscript_of_open_article.json new file mode 100644 index 0000000..b3ff395 --- /dev/null +++ b/examples/licenses/tdm_manuscript_of_open_article.json @@ -0,0 +1,52 @@ +{ + "identifiers": { + "pmid": "35820077", + "pmcid": "PMC10518222", + "doi": "10.1002/alz.12697", + "preprint_dois": null, + "preprint_pmid": null, + "preprint_pmcid": null, + "publication_status": "published" + }, + "text_copy": { + "route": "pmc", + "version": "am", + "access": "tdm_grant", + "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10518222/", + "license": { + "license_id": "all-rights-reserved", + "determination": "stated", + "stated_by": "pmc_cloud", + "stated_as": "TDM" + } + }, + "reported_copies": [ + { + "route": "publisher", + "version": "vor", + "reported_by": "openalex", + "url": "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/alz.12697", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "openalex", + "stated_as": "cc-by-nc" + } + } + ], + "bibliography": { + "title": "Comparative analytical performance of multiple plasma Aβ42 and Aβ40 assays and their ability to predict positron emission tomography amyloid positivity", + "journal": "Alzheimer's & Dementia", + "publication_year": 2022, + "abstract": "This report details the approach taken to providing a dataset allowing for analyses on the performance of recently developed assays of amyloid beta (Aβ) peptides in plasma and the extent to which they improve the prediction of amyloid positivity.", + "abstract_provider": "pubmed", + "license": { + "license_id": "cc-by-nc", + "determination": "stated", + "stated_by": "crossref", + "stated_as": "http://creativecommons.org/licenses/by-nc/4.0/" + }, + "oa_status": "hybrid", + "provider": "semantic_scholar" + } +} diff --git a/examples/run/records/x7Ex4mpleA01.extraction.json b/examples/run/records/x7Ex4mpleA01.extraction.json index 83a5e31..ee12f17 100644 --- a/examples/run/records/x7Ex4mpleA01.extraction.json +++ b/examples/run/records/x7Ex4mpleA01.extraction.json @@ -26,7 +26,5 @@ "result_reporting": null, "tables": null, "coordinate_sets": null, - "external_datasets": null, - "language": null, - "study_type": null + "external_datasets": null } diff --git a/examples/run/revisions/x7Ex4mpleA01.coordinate_parse.json b/examples/run/revisions/x7Ex4mpleA01.coordinate_parse.json index 398e4f4..69cc88c 100644 --- a/examples/run/revisions/x7Ex4mpleA01.coordinate_parse.json +++ b/examples/run/revisions/x7Ex4mpleA01.coordinate_parse.json @@ -1,7 +1,7 @@ { "header": { "artifact_kind": "coordinate_parse", - "schema_version": "0.7.0", + "schema_version": "0.8.0", "article_id": "art-000001", "identifiers": { "pmid": "00000000", @@ -10,8 +10,7 @@ "preprint_dois": null, "preprint_pmid": null, "preprint_pmcid": null, - "publication_status": "published", - "neurostore_base_study_id": "x7Ex4mpleA01" + "publication_status": "published" }, "producer": { "name": "pondie", @@ -32,7 +31,8 @@ "producer_version": null } ], - "created_at": "2026-10-09T12:00:00Z" + "created_at": "2026-10-09T12:00:00Z", + "neurostore_base_study_id": "x7Ex4mpleA01" }, "parse_id": "2a78a79c87463fac0a9109fba0cc54d844446b3b29c14893747aefe95f25a087", "text_sha256": "a44901241d8884e85cfead36b324ccb4dba1b0213436ed2a8d216cb8c3803da5", diff --git a/extraction-readme.md b/extraction-readme.md index cb5fa4a..59f7fb7 100644 --- a/extraction-readme.md +++ b/extraction-readme.md @@ -21,8 +21,8 @@ around — so recording one means inventing a cohort. Skip the paper instead. The gate runs on PubMed metadata **before any text is read**, in two parts: -1. **Publication type.** Skip when `Study.study_type` (the PubMed publication types, fetched - from E-utilities by PMID) includes `Meta-Analysis`. +1. **Publication type.** Skip when `Bibliography.publication_types` (the PubMed publication + types, fetched from E-utilities by PMID) includes `Meta-Analysis`. 2. **Title.** Skip when the title contains "meta-analysis", case-insensitive. Both are needed. Measured against 200 corpus papers whose own methods say they ran a @@ -647,8 +647,7 @@ asserts that correspondence in both directions, so this table cannot quietly fal | Storage field | How the mapper fills it | |---|---| -| `Study.title`, `.authors`, `.journal`, `.doi`, `.publication_year` | `api_lookup` — bibliographic facts the E-utilities API states, which is not something a model should be asked to read off the page | -| `Study.study_type` | `api_lookup` — PubMed publication types by PMID, stored verbatim and multivalued | +| `Study.bibliography`, `Study.identifiers` | `copy` — the parsed paper's `Bibliography` and header identifiers, held whole: bibliographic facts the metadata lookup states, which is not something a model should be asked to read off the page | | `Analysis.id`, and every other `id` | `generate` — must be a deterministic function of the source location, so re-extraction reproduces it and externally stored evidence still joins. Recipe: study, table or figure identifier, analysis label slug or index. The extraction record's `local_id` goes in the audit log and every reference to it resolves through the same table | | `Analysis.statistical_maps` | `api_lookup` — the maps NeuroVault holds for the study. A paper that shares maps says so in a data-availability sentence, and the sentence is not the map | | `Table.source_path` | `generate` — the repository path of the raw table file | diff --git a/schema-tutorial.md b/schema-tutorial.md index 3083f81..f7b2a52 100644 --- a/schema-tutorial.md +++ b/schema-tutorial.md @@ -194,7 +194,7 @@ something — see §2.4. **`unique_keys`** says a combination may not repeat wit | What it is | The normalized, query-optimized target: typed values bound to vocabularies, no provenance | The immutable record of one extraction: the same shape with every value wrapped in its evidence | | Who writes it | The mapper, from an extraction record | A language model reading the paper, plus deterministic passes | | Hand-edited? | Never — re-running the mapper regenerates it | Never — produce a new version or a separate correction record | -| Size | 45 classes, 37 vocabularies, 301 fields, across 11 modules | Generated: the same classes plus the evidence types | +| Size | 52 classes, 57 vocabularies, 365 fields, across 13 modules | Generated: the same classes plus the evidence types | The extraction schema is **generated** from storage by `gen_extraction_schema.py`, so the two cannot drift. The whole of the projection: @@ -350,8 +350,8 @@ Task-internal structure is *not* here. What participants did inside the scanner | Field | Holds | Not to be confused with | |---|---|---| -| `title`, `authors`, `journal`, `doi`, `publication_year` | Bibliographic facts | Nothing — all `deterministic`, from the E-utilities API rather than the page | -| `study_type` | PubMed publication types, verbatim and multivalued | A design label of your own. `Meta-Analysis` here is also a *gate*: the paper is skipped | +| `bibliography` | The parsed paper's `Bibliography`, whole: title, authors, journal, year, abstract, PubMed publication types (`publication_types`), the paper's licence | Nothing — all `deterministic`, from the metadata lookup rather than the page. `Meta-Analysis` among the publication types is also a *gate*: the paper is skipped | +| `identifiers` | The parsed paper's header identifiers, whole: DOI, PMID, PMCID, preprint identifiers, `publication_status` | — | | `description` | Brief summary of aims and scope | `StudyDesign.description`, which is the design narrative | | `hypothesis` | The study's own prediction, quoted, one entry per prediction | An aim. See below | | `design` | The inlined `StudyDesign` | — | @@ -1604,7 +1604,7 @@ Two checks run on PubMed metadata **before any text is read**, and both exist be meta-analysis has no participant sample, no acquisition and no measure of its own — the three things an `Analysis` is built around, so recording one means inventing a cohort. -1. Skip when `Study.study_type` includes `Meta-Analysis`. +1. Skip when `Bibliography.publication_types` includes `Meta-Analysis`. 2. Skip when the title contains "meta-analysis", case-insensitively. Both are needed: the type label is precise but missing from unindexed records, and the title gate @@ -1745,12 +1745,12 @@ Worth knowing before you go hunting for a slot. The full list with evidence is i ## Appendix A. Class inventory -45 classes. "Declared in" is where the record lives; everything else references it by identifier. The +52 classes. "Declared in" is where the record lives; everything else references it by identifier. The required column leaves out `id`, which every class that has one requires; "Fields" counts it. | Class | Declared in | Required fields | Fields | |---|---|---|---| -| `Study` | the record root | `title`, `study_type` | 23 | +| `Study` | the record root | `bibliography` | 22 | | `StudyDesign` | `Study.design` (inlined) | — | 6 | | `Arm` | `StudyDesign.arms` | `name`, `arm_kind` | 5 | | `Timepoint` | `StudyDesign.timepoints` | `name`, `relation_to_intervention` | 5 | @@ -1791,13 +1791,19 @@ required column leaves out `id`, which every class that has one requires; "Field | `Table` | `Study.tables` | — | 11 | | `StatisticalMap` | `Analysis.statistical_maps` | `map_type` | 7 | | `ExternalDataset` | `Study.external_datasets` | — | 2 | +| `Bibliography` | `Study.bibliography` (inlined), shared with the parsed paper | `title`, `license` | 16 | +| `ArticleIdentifiers` | `Study.identifiers` (inlined), shared with the parsed paper's header | `publication_status` | 7 | +| `PreprintDoi` | `ArticleIdentifiers.preprint_dois` | `doi`, `server` | 2 | +| `Correction` | `Bibliography.corrections` | `kind` | 3 | +| `License` | `Bibliography.license` (inlined) | `license_id`, `determination` | 4 | ## Appendix B. Where everything is -**The schema.** `neuroimaging-study-storage.yaml` plus eleven modules under -[neuroimaging-study-storage/](src/study_schema/linkml/neuroimaging-study-storage/): `subsets`, `design`, `study`, `group`, -`task`, `assessment`, `region`, `acquisition`, `analysis_enums`, `analysis_details`, `analysis`. -`analysis_enums` holds 27 of the 38 vocabularies, separately so that the analysis and +**The schema.** `neuroimaging-study-storage.yaml` plus thirteen modules under +[neuroimaging-study-storage/](src/study_schema/linkml/neuroimaging-study-storage/): `subsets`, `design`, `study`, `license`, +`bibliography`, `group`, `task`, `assessment`, `region`, `acquisition`, `analysis_enums`, +`analysis_details`, `analysis`. +`analysis_enums` holds 30 of the 57 vocabularies, separately so that the analysis and analysis-details modules can both bind to them without either importing the other. [neuroimaging-study-extraction/](src/study_schema/linkml/neuroimaging-study-extraction/) is the generated projection and is never hand-edited. diff --git a/src/study_schema/jsonschema/coordinate-parse.schema.json b/src/study_schema/jsonschema/coordinate-parse.schema.json index 0f90fce..1302a2d 100644 --- a/src/study_schema/jsonschema/coordinate-parse.schema.json +++ b/src/study_schema/jsonschema/coordinate-parse.schema.json @@ -102,13 +102,6 @@ "null" ] }, - "neurostore_base_study_id": { - "description": "Set once the paper has been uploaded; neurostore's BaseStudy id.", - "type": [ - "string", - "null" - ] - }, "pmcid": { "description": "PubMed Central's copy of the version of record. Never PMC's copy of a preprint, which is `preprint_pmcid`.", "type": [ @@ -189,6 +182,13 @@ "null" ] }, + "neurostore_base_study_id": { + "description": "Set once the paper has been uploaded; neurostore's BaseStudy id. An upload pointer, not a bibliographic identifier, so it sits here and not in `identifiers`.", + "type": [ + "string", + "null" + ] + }, "producer": { "$ref": "#/$defs/Producer" }, @@ -232,14 +232,54 @@ }, "Bibliography": { "additionalProperties": false, - "description": "The paper's bibliographic facts, as the metadata lookup returned them. Downstream they are copied, never extracted: pondie fills the record's Study.title, authors, doi, journal, publication_year, language and study_type from here, and neurostore fills BaseStudy from here.", + "description": "The paper's bibliographic facts, as the metadata lookup returned them. Defined once: the parsed paper carries it, and the storage Study holds the same object as `Study.bibliography`, so nothing is copied field by field and nothing is extracted.", + "else": { + "properties": { + "abstract_provider": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + } + } + }, + "if": { + "properties": { + "abstract": { + "anyOf": [ + { + "type": "string" + } + ] + } + }, + "required": [ + "abstract" + ] + }, "properties": { "abstract": { + "description": "Absent (null) when no source gave one; never an empty string.", + "pattern": "\\S", "type": [ "string", "null" ] }, + "abstract_provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The source whose abstract `abstract` is. It may differ from `provider`. The abstract is the publisher's abstract of the version `license` describes, so its licence is `license`." + }, "authors": { "description": "Author names in published order.", "items": { @@ -282,6 +322,7 @@ ] }, "language": { + "description": "Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as.\nAt least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.", "items": { "type": "string" }, @@ -291,15 +332,17 @@ ] }, "license": { - "type": [ - "string", - "null" - ] + "$ref": "#/$defs/License", + "description": "The paper's licence: the version of record's, or the latest preprint's while the header's `publication_status` is `preprint`. Not the licence of a copy of another version, which that copy carries." }, - "open_access": { - "type": [ - "boolean", - "null" + "oa_status": { + "anyOf": [ + { + "$ref": "#/$defs/OaStatus" + }, + { + "type": "null" + } ] }, "pages": { @@ -309,16 +352,27 @@ ] }, "provider": { - "description": "The metadata service the facts came from, such as pubmed or europepmc.", - "type": [ - "string", - "null" - ] + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The metadata service the facts came from." }, "publication_types": { - "description": "PubMed publication types, verbatim. The storage schema's Study.study_type.", + "description": "PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels. `StudyType` lists the known types only; any other type PubMed returns is kept verbatim.", "items": { - "type": "string" + "anyOf": [ + { + "$ref": "#/$defs/StudyType" + }, + { + "type": "string" + } + ] }, "type": [ "array", @@ -326,6 +380,8 @@ ] }, "publication_year": { + "description": "Year in which the study was first published or made publicly available.", + "minimum": 1600, "type": [ "integer", "null" @@ -342,8 +398,23 @@ } }, "required": [ - "title" - ], + "title", + "license" + ], + "then": { + "properties": { + "abstract_provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + }, + "required": [ + "abstract_provider" + ] + }, "title": "Bibliography", "type": "object" }, @@ -559,6 +630,121 @@ "title": "CoordinateRole", "type": "string" }, + "Copy": { + "additionalProperties": false, + "description": "One copy of a paper, where it is, which version it is, and its licence.", + "else": { + "properties": { + "preprint_server": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + } + } + }, + "if": { + "properties": { + "route": { + "const": "preprint_server" + } + }, + "required": [ + "route" + ] + }, + "properties": { + "license": { + "$ref": "#/$defs/License", + "description": "This copy's own licence. A manuscript or preprint can be licensed differently from the version of record." + }, + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + }, + { + "type": "null" + } + ] + }, + "route": { + "$ref": "#/$defs/CopyRoute" + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "version": { + "anyOf": [ + { + "$ref": "#/$defs/CopyVersion" + }, + { + "type": "null" + } + ], + "description": "Absent when no source says which version the copy is." + } + }, + "required": [ + "route", + "license" + ], + "then": { + "properties": { + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + }, + "required": [ + "preprint_server" + ] + }, + "title": "Copy", + "type": "object" + }, + "CopyAccess": { + "description": "How we obtained a copy we hold. The `redistribution` annotation says whether the access itself allows passing the copy on: `by_license` leaves that to the copy's licence, `never` forbids it whatever the licence says.", + "enum": [ + "public", + "tdm_grant", + "unrecorded" + ], + "title": "CopyAccess", + "type": "string" + }, + "CopyRoute": { + "description": "Where a copy of a paper is hosted.", + "enum": [ + "pmc", + "publisher", + "repository", + "preprint_server", + "unrecorded" + ], + "title": "CopyRoute", + "type": "string" + }, + "CopyVersion": { + "description": "Which version of the paper a copy is, in the NISO Journal Article Versions terms. OpenAlex's `publishedVersion`, `acceptedVersion` and `submittedVersion` are `vor`, `am` and `submitted`.", + "enum": [ + "vor", + "am", + "submitted" + ], + "title": "CopyVersion", + "type": "string" + }, "Correction": { "additionalProperties": false, "description": "One notice attached to a paper.", @@ -705,6 +891,93 @@ "title": "FunctionalForm", "type": "string" }, + "HeldCopy": { + "additionalProperties": false, + "description": "A copy we downloaded.", + "else": { + "properties": { + "preprint_server": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + } + } + }, + "if": { + "properties": { + "route": { + "const": "preprint_server" + } + }, + "required": [ + "route" + ] + }, + "properties": { + "access": { + "$ref": "#/$defs/CopyAccess" + }, + "license": { + "$ref": "#/$defs/License", + "description": "This copy's own licence. A manuscript or preprint can be licensed differently from the version of record." + }, + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + }, + { + "type": "null" + } + ] + }, + "route": { + "$ref": "#/$defs/CopyRoute" + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "version": { + "anyOf": [ + { + "$ref": "#/$defs/CopyVersion" + }, + { + "type": "null" + } + ], + "description": "Absent when no source says which version the copy is." + } + }, + "required": [ + "access", + "route", + "license" + ], + "then": { + "properties": { + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + }, + "required": [ + "preprint_server" + ] + }, + "title": "HeldCopy", + "type": "object" + }, "InferenceTarget": { "description": "What the connectivity analysis drew inference about. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -741,6 +1014,127 @@ "title": "InputRef", "type": "object" }, + "License": { + "additionalProperties": false, + "description": "A licence as one source stated it, or all rights reserved assumed because no source stated one.", + "else": { + "properties": { + "license_id": { + "const": "all-rights-reserved" + }, + "stated_as": { + "not": { + "anyOf": [ + { + "type": "string" + } + ] + } + }, + "stated_by": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + } + } + }, + "if": { + "properties": { + "determination": { + "const": "stated" + } + }, + "required": [ + "determination" + ] + }, + "properties": { + "determination": { + "$ref": "#/$defs/LicenseDetermination" + }, + "license_id": { + "$ref": "#/$defs/LicenseId", + "description": "Which licence, as OpenAlex's licence slugs name them (`cc-by`, `cc-by-nc`, ...). Not an SPDX License Identifier: sources rarely state the licence's version, and SPDX identifiers require one." + }, + "stated_as": { + "description": "The statement exactly as the source gave it: a licence URL, a licence name, or a copyright line. What `other` holds, and what attribution quotes. Never empty: a source that gave no statement did not state a licence.", + "pattern": "\\S", + "type": [ + "string", + "null" + ] + }, + "stated_by": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The source whose statement this is." + } + }, + "required": [ + "license_id", + "determination" + ], + "then": { + "properties": { + "stated_as": { + "anyOf": [ + { + "type": "string" + } + ] + }, + "stated_by": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + }, + "required": [ + "stated_by", + "stated_as" + ] + }, + "title": "License", + "type": "object" + }, + "LicenseDetermination": { + "description": "Whether a source stated the licence.", + "enum": [ + "stated", + "assumed" + ], + "title": "LicenseDetermination", + "type": "string" + }, + "LicenseId": { + "description": "The terms a licence grants. A value annotated `redistributable` lets the text it covers be passed on verbatim with attribution: every Creative Commons licence, including ND, because converting a format to text is a change CC 4.0 2(a)(4) allows, and NC, because neurostore's use is non-commercial. That reading of ND and NC is this project's policy, not legal advice, and no lawyer has confirmed it.", + "enum": [ + "cc-by", + "cc-by-sa", + "cc-by-nc", + "cc-by-nc-sa", + "cc-by-nd", + "cc-by-nc-nd", + "cc0", + "public-domain", + "all-rights-reserved", + "other" + ], + "title": "LicenseId", + "type": "string" + }, "MapQuantity": { "description": "What the values of a component or latent-variable map represent. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -803,6 +1197,21 @@ "title": "MetaAnalysisBasis", "type": "string" }, + "MetadataSource": { + "description": "Who stated a bibliographic fact.", + "enum": [ + "jats", + "elsevier", + "pmc_cloud", + "pubmed", + "crossref", + "openalex", + "semantic_scholar", + "europepmc" + ], + "title": "MetadataSource", + "type": "string" + }, "ModelFamily": { "description": "Statistical model family used to estimate the effect. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -834,6 +1243,19 @@ "title": "NullHypothesis", "type": "string" }, + "OaStatus": { + "description": "How a paper is open to read, in OpenAlex's terms (https://help.openalex.org/hc/en-us/articles/24347035046295-Open-Access-OA).", + "enum": [ + "gold", + "diamond", + "hybrid", + "bronze", + "green", + "closed" + ], + "title": "OaStatus", + "type": "string" + }, "OmittedAnalysis": { "additionalProperties": false, "description": "A text or figure analysis the producer read but did not keep, the text counterpart of a `TableReading` with a reason.", @@ -1120,6 +1542,16 @@ "null" ] }, + "reported_copies": { + "description": "Copies of the paper that a source lists and that we do not hold, each with its own licence. Where an open copy is, when `bibliography.oa_status` says one exists.", + "items": { + "$ref": "#/$defs/ReportedCopy" + }, + "type": [ + "array", + "null" + ] + }, "sections": { "items": { "$ref": "#/$defs/Section" @@ -1165,6 +1597,10 @@ "null" ] }, + "text_copy": { + "$ref": "#/$defs/HeldCopy", + "description": "The copy the text was read from. Its licence and access, not the paper's licence, decide whether the text may be passed on." + }, "text_length": { "minimum": 0, "type": [ @@ -1184,6 +1620,7 @@ "required": [ "header", "source", + "text_copy", "text_path", "text_sha256", "bibliography", @@ -1665,6 +2102,93 @@ "title": "ReferenceProvider", "type": "string" }, + "ReportedCopy": { + "additionalProperties": false, + "description": "A copy a metadata source lists and we do not hold.", + "else": { + "properties": { + "preprint_server": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + } + } + }, + "if": { + "properties": { + "route": { + "const": "preprint_server" + } + }, + "required": [ + "route" + ] + }, + "properties": { + "license": { + "$ref": "#/$defs/License", + "description": "This copy's own licence. A manuscript or preprint can be licensed differently from the version of record." + }, + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + }, + { + "type": "null" + } + ] + }, + "reported_by": { + "$ref": "#/$defs/MetadataSource" + }, + "route": { + "$ref": "#/$defs/CopyRoute" + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "version": { + "anyOf": [ + { + "$ref": "#/$defs/CopyVersion" + }, + { + "type": "null" + } + ], + "description": "Absent when no source says which version the copy is." + } + }, + "required": [ + "reported_by", + "route", + "license" + ], + "then": { + "properties": { + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + }, + "required": [ + "preprint_server" + ] + }, + "title": "ReportedCopy", + "type": "object" + }, "ReportedSpace": { "description": "The coordinate space a point is in, normalized to the values NiMARE transforms between by `study_schema.spaces.normalize_space`. Absent when no space is stated, or when the wording names both MNI and TAL; never defaulted to MNI: a rat study stored as MNI is a wrong coordinate, and a missing one is an honest gap. What the paper printed stays in `space_as_printed`.", "enum": [ @@ -1865,6 +2389,29 @@ "title": "StatisticKind", "type": "string" }, + "StudyType": { + "description": "PubMed publication types, spelled as the E-utilities API returns them so that a value needs no translation. Only the design-relevant types are listed; PubMed also returns funding and indexing types, and the vocabulary is open, so any unlisted type passes through verbatim.", + "enum": [ + "Journal Article", + "Meta-Analysis", + "Systematic Review", + "Review", + "Randomized Controlled Trial", + "Clinical Trial", + "Observational Study", + "Comparative Study", + "Multicenter Study", + "Validation Study", + "Case Reports", + "Editorial", + "Letter", + "Comment", + "Preprint", + "Retracted Publication" + ], + "title": "StudyType", + "type": "string" + }, "Supplement": { "additionalProperties": false, "description": "One item of supplementary material.", @@ -2292,5 +2839,5 @@ ], "title": "neuroimaging-paper-parse", "type": "object", - "version": "0.7.0" + "version": "0.8.0" } diff --git a/src/study_schema/jsonschema/extraction-record.schema.json b/src/study_schema/jsonschema/extraction-record.schema.json index d31d305..19e5bdd 100644 --- a/src/study_schema/jsonschema/extraction-record.schema.json +++ b/src/study_schema/jsonschema/extraction-record.schema.json @@ -6984,16 +6984,6 @@ "null" ] }, - "language": { - "description": "Languages of this article, copied from the parsed paper's `Bibliography.language` alongside `study_type`, not read off the paper: three-letter codes, `eng` and the rest.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] - }, "local_id": { "description": "Stable identifier for the study record, such as a DOI-based or repository-local identifier.", "type": "string" @@ -7057,16 +7047,6 @@ "null" ] }, - "study_type": { - "description": "PubMed publication types for this record, verbatim, copied from the parsed paper's `Bibliography.publication_types`, not read off the paper: it is what the indexers said about the article, which is what a publication-type exclusion asks about.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] - }, "tables": { "description": "Publication tables from which results or analysis metadata were extracted.", "items": { @@ -7745,16 +7725,6 @@ "null" ] }, - "language": { - "description": "Languages of this article, copied from the parsed paper's `Bibliography.language` alongside `study_type`, not read off the paper: three-letter codes, `eng` and the rest.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] - }, "local_id": { "description": "Stable identifier for the study record, such as a DOI-based or repository-local identifier.", "type": "string" @@ -7818,16 +7788,6 @@ "null" ] }, - "study_type": { - "description": "PubMed publication types for this record, verbatim, copied from the parsed paper's `Bibliography.publication_types`, not read off the paper: it is what the indexers said about the article, which is what a publication-type exclusion asks about.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] - }, "tables": { "description": "Publication tables from which results or analysis metadata were extracted.", "items": { @@ -7855,5 +7815,5 @@ ], "title": "neuroimaging-study-extraction", "type": "object", - "version": "0.5.0" + "version": "0.6.0" } diff --git a/src/study_schema/jsonschema/parsed-paper.schema.json b/src/study_schema/jsonschema/parsed-paper.schema.json index 1e642d3..9d1635f 100644 --- a/src/study_schema/jsonschema/parsed-paper.schema.json +++ b/src/study_schema/jsonschema/parsed-paper.schema.json @@ -102,13 +102,6 @@ "null" ] }, - "neurostore_base_study_id": { - "description": "Set once the paper has been uploaded; neurostore's BaseStudy id.", - "type": [ - "string", - "null" - ] - }, "pmcid": { "description": "PubMed Central's copy of the version of record. Never PMC's copy of a preprint, which is `preprint_pmcid`.", "type": [ @@ -189,6 +182,13 @@ "null" ] }, + "neurostore_base_study_id": { + "description": "Set once the paper has been uploaded; neurostore's BaseStudy id. An upload pointer, not a bibliographic identifier, so it sits here and not in `identifiers`.", + "type": [ + "string", + "null" + ] + }, "producer": { "$ref": "#/$defs/Producer" }, @@ -232,14 +232,54 @@ }, "Bibliography": { "additionalProperties": false, - "description": "The paper's bibliographic facts, as the metadata lookup returned them. Downstream they are copied, never extracted: pondie fills the record's Study.title, authors, doi, journal, publication_year, language and study_type from here, and neurostore fills BaseStudy from here.", + "description": "The paper's bibliographic facts, as the metadata lookup returned them. Defined once: the parsed paper carries it, and the storage Study holds the same object as `Study.bibliography`, so nothing is copied field by field and nothing is extracted.", + "else": { + "properties": { + "abstract_provider": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + } + } + }, + "if": { + "properties": { + "abstract": { + "anyOf": [ + { + "type": "string" + } + ] + } + }, + "required": [ + "abstract" + ] + }, "properties": { "abstract": { + "description": "Absent (null) when no source gave one; never an empty string.", + "pattern": "\\S", "type": [ "string", "null" ] }, + "abstract_provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The source whose abstract `abstract` is. It may differ from `provider`. The abstract is the publisher's abstract of the version `license` describes, so its licence is `license`." + }, "authors": { "description": "Author names in published order.", "items": { @@ -282,6 +322,7 @@ ] }, "language": { + "description": "Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as.\nAt least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.", "items": { "type": "string" }, @@ -291,15 +332,17 @@ ] }, "license": { - "type": [ - "string", - "null" - ] + "$ref": "#/$defs/License", + "description": "The paper's licence: the version of record's, or the latest preprint's while the header's `publication_status` is `preprint`. Not the licence of a copy of another version, which that copy carries." }, - "open_access": { - "type": [ - "boolean", - "null" + "oa_status": { + "anyOf": [ + { + "$ref": "#/$defs/OaStatus" + }, + { + "type": "null" + } ] }, "pages": { @@ -309,16 +352,27 @@ ] }, "provider": { - "description": "The metadata service the facts came from, such as pubmed or europepmc.", - "type": [ - "string", - "null" - ] + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The metadata service the facts came from." }, "publication_types": { - "description": "PubMed publication types, verbatim. The storage schema's Study.study_type.", + "description": "PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels. `StudyType` lists the known types only; any other type PubMed returns is kept verbatim.", "items": { - "type": "string" + "anyOf": [ + { + "$ref": "#/$defs/StudyType" + }, + { + "type": "string" + } + ] }, "type": [ "array", @@ -326,6 +380,8 @@ ] }, "publication_year": { + "description": "Year in which the study was first published or made publicly available.", + "minimum": 1600, "type": [ "integer", "null" @@ -342,8 +398,23 @@ } }, "required": [ - "title" - ], + "title", + "license" + ], + "then": { + "properties": { + "abstract_provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + }, + "required": [ + "abstract_provider" + ] + }, "title": "Bibliography", "type": "object" }, @@ -559,6 +630,121 @@ "title": "CoordinateRole", "type": "string" }, + "Copy": { + "additionalProperties": false, + "description": "One copy of a paper, where it is, which version it is, and its licence.", + "else": { + "properties": { + "preprint_server": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + } + } + }, + "if": { + "properties": { + "route": { + "const": "preprint_server" + } + }, + "required": [ + "route" + ] + }, + "properties": { + "license": { + "$ref": "#/$defs/License", + "description": "This copy's own licence. A manuscript or preprint can be licensed differently from the version of record." + }, + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + }, + { + "type": "null" + } + ] + }, + "route": { + "$ref": "#/$defs/CopyRoute" + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "version": { + "anyOf": [ + { + "$ref": "#/$defs/CopyVersion" + }, + { + "type": "null" + } + ], + "description": "Absent when no source says which version the copy is." + } + }, + "required": [ + "route", + "license" + ], + "then": { + "properties": { + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + }, + "required": [ + "preprint_server" + ] + }, + "title": "Copy", + "type": "object" + }, + "CopyAccess": { + "description": "How we obtained a copy we hold. The `redistribution` annotation says whether the access itself allows passing the copy on: `by_license` leaves that to the copy's licence, `never` forbids it whatever the licence says.", + "enum": [ + "public", + "tdm_grant", + "unrecorded" + ], + "title": "CopyAccess", + "type": "string" + }, + "CopyRoute": { + "description": "Where a copy of a paper is hosted.", + "enum": [ + "pmc", + "publisher", + "repository", + "preprint_server", + "unrecorded" + ], + "title": "CopyRoute", + "type": "string" + }, + "CopyVersion": { + "description": "Which version of the paper a copy is, in the NISO Journal Article Versions terms. OpenAlex's `publishedVersion`, `acceptedVersion` and `submittedVersion` are `vor`, `am` and `submitted`.", + "enum": [ + "vor", + "am", + "submitted" + ], + "title": "CopyVersion", + "type": "string" + }, "Correction": { "additionalProperties": false, "description": "One notice attached to a paper.", @@ -705,6 +891,93 @@ "title": "FunctionalForm", "type": "string" }, + "HeldCopy": { + "additionalProperties": false, + "description": "A copy we downloaded.", + "else": { + "properties": { + "preprint_server": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + } + } + }, + "if": { + "properties": { + "route": { + "const": "preprint_server" + } + }, + "required": [ + "route" + ] + }, + "properties": { + "access": { + "$ref": "#/$defs/CopyAccess" + }, + "license": { + "$ref": "#/$defs/License", + "description": "This copy's own licence. A manuscript or preprint can be licensed differently from the version of record." + }, + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + }, + { + "type": "null" + } + ] + }, + "route": { + "$ref": "#/$defs/CopyRoute" + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "version": { + "anyOf": [ + { + "$ref": "#/$defs/CopyVersion" + }, + { + "type": "null" + } + ], + "description": "Absent when no source says which version the copy is." + } + }, + "required": [ + "access", + "route", + "license" + ], + "then": { + "properties": { + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + }, + "required": [ + "preprint_server" + ] + }, + "title": "HeldCopy", + "type": "object" + }, "InferenceTarget": { "description": "What the connectivity analysis drew inference about. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -741,6 +1014,127 @@ "title": "InputRef", "type": "object" }, + "License": { + "additionalProperties": false, + "description": "A licence as one source stated it, or all rights reserved assumed because no source stated one.", + "else": { + "properties": { + "license_id": { + "const": "all-rights-reserved" + }, + "stated_as": { + "not": { + "anyOf": [ + { + "type": "string" + } + ] + } + }, + "stated_by": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + } + } + }, + "if": { + "properties": { + "determination": { + "const": "stated" + } + }, + "required": [ + "determination" + ] + }, + "properties": { + "determination": { + "$ref": "#/$defs/LicenseDetermination" + }, + "license_id": { + "$ref": "#/$defs/LicenseId", + "description": "Which licence, as OpenAlex's licence slugs name them (`cc-by`, `cc-by-nc`, ...). Not an SPDX License Identifier: sources rarely state the licence's version, and SPDX identifiers require one." + }, + "stated_as": { + "description": "The statement exactly as the source gave it: a licence URL, a licence name, or a copyright line. What `other` holds, and what attribution quotes. Never empty: a source that gave no statement did not state a licence.", + "pattern": "\\S", + "type": [ + "string", + "null" + ] + }, + "stated_by": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The source whose statement this is." + } + }, + "required": [ + "license_id", + "determination" + ], + "then": { + "properties": { + "stated_as": { + "anyOf": [ + { + "type": "string" + } + ] + }, + "stated_by": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + }, + "required": [ + "stated_by", + "stated_as" + ] + }, + "title": "License", + "type": "object" + }, + "LicenseDetermination": { + "description": "Whether a source stated the licence.", + "enum": [ + "stated", + "assumed" + ], + "title": "LicenseDetermination", + "type": "string" + }, + "LicenseId": { + "description": "The terms a licence grants. A value annotated `redistributable` lets the text it covers be passed on verbatim with attribution: every Creative Commons licence, including ND, because converting a format to text is a change CC 4.0 2(a)(4) allows, and NC, because neurostore's use is non-commercial. That reading of ND and NC is this project's policy, not legal advice, and no lawyer has confirmed it.", + "enum": [ + "cc-by", + "cc-by-sa", + "cc-by-nc", + "cc-by-nc-sa", + "cc-by-nd", + "cc-by-nc-nd", + "cc0", + "public-domain", + "all-rights-reserved", + "other" + ], + "title": "LicenseId", + "type": "string" + }, "MapQuantity": { "description": "What the values of a component or latent-variable map represent. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -803,6 +1197,21 @@ "title": "MetaAnalysisBasis", "type": "string" }, + "MetadataSource": { + "description": "Who stated a bibliographic fact.", + "enum": [ + "jats", + "elsevier", + "pmc_cloud", + "pubmed", + "crossref", + "openalex", + "semantic_scholar", + "europepmc" + ], + "title": "MetadataSource", + "type": "string" + }, "ModelFamily": { "description": "Statistical model family used to estimate the effect. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -834,6 +1243,19 @@ "title": "NullHypothesis", "type": "string" }, + "OaStatus": { + "description": "How a paper is open to read, in OpenAlex's terms (https://help.openalex.org/hc/en-us/articles/24347035046295-Open-Access-OA).", + "enum": [ + "gold", + "diamond", + "hybrid", + "bronze", + "green", + "closed" + ], + "title": "OaStatus", + "type": "string" + }, "OmittedAnalysis": { "additionalProperties": false, "description": "A text or figure analysis the producer read but did not keep, the text counterpart of a `TableReading` with a reason.", @@ -1120,6 +1542,16 @@ "null" ] }, + "reported_copies": { + "description": "Copies of the paper that a source lists and that we do not hold, each with its own licence. Where an open copy is, when `bibliography.oa_status` says one exists.", + "items": { + "$ref": "#/$defs/ReportedCopy" + }, + "type": [ + "array", + "null" + ] + }, "sections": { "items": { "$ref": "#/$defs/Section" @@ -1165,6 +1597,10 @@ "null" ] }, + "text_copy": { + "$ref": "#/$defs/HeldCopy", + "description": "The copy the text was read from. Its licence and access, not the paper's licence, decide whether the text may be passed on." + }, "text_length": { "minimum": 0, "type": [ @@ -1184,6 +1620,7 @@ "required": [ "header", "source", + "text_copy", "text_path", "text_sha256", "bibliography", @@ -1665,6 +2102,93 @@ "title": "ReferenceProvider", "type": "string" }, + "ReportedCopy": { + "additionalProperties": false, + "description": "A copy a metadata source lists and we do not hold.", + "else": { + "properties": { + "preprint_server": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + } + } + }, + "if": { + "properties": { + "route": { + "const": "preprint_server" + } + }, + "required": [ + "route" + ] + }, + "properties": { + "license": { + "$ref": "#/$defs/License", + "description": "This copy's own licence. A manuscript or preprint can be licensed differently from the version of record." + }, + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + }, + { + "type": "null" + } + ] + }, + "reported_by": { + "$ref": "#/$defs/MetadataSource" + }, + "route": { + "$ref": "#/$defs/CopyRoute" + }, + "url": { + "type": [ + "string", + "null" + ] + }, + "version": { + "anyOf": [ + { + "$ref": "#/$defs/CopyVersion" + }, + { + "type": "null" + } + ], + "description": "Absent when no source says which version the copy is." + } + }, + "required": [ + "reported_by", + "route", + "license" + ], + "then": { + "properties": { + "preprint_server": { + "anyOf": [ + { + "$ref": "#/$defs/PreprintServer" + } + ] + } + }, + "required": [ + "preprint_server" + ] + }, + "title": "ReportedCopy", + "type": "object" + }, "ReportedSpace": { "description": "The coordinate space a point is in, normalized to the values NiMARE transforms between by `study_schema.spaces.normalize_space`. Absent when no space is stated, or when the wording names both MNI and TAL; never defaulted to MNI: a rat study stored as MNI is a wrong coordinate, and a missing one is an honest gap. What the paper printed stays in `space_as_printed`.", "enum": [ @@ -1865,6 +2389,29 @@ "title": "StatisticKind", "type": "string" }, + "StudyType": { + "description": "PubMed publication types, spelled as the E-utilities API returns them so that a value needs no translation. Only the design-relevant types are listed; PubMed also returns funding and indexing types, and the vocabulary is open, so any unlisted type passes through verbatim.", + "enum": [ + "Journal Article", + "Meta-Analysis", + "Systematic Review", + "Review", + "Randomized Controlled Trial", + "Clinical Trial", + "Observational Study", + "Comparative Study", + "Multicenter Study", + "Validation Study", + "Case Reports", + "Editorial", + "Letter", + "Comment", + "Preprint", + "Retracted Publication" + ], + "title": "StudyType", + "type": "string" + }, "Supplement": { "additionalProperties": false, "description": "One item of supplementary material.", @@ -2256,6 +2803,16 @@ "null" ] }, + "reported_copies": { + "description": "Copies of the paper that a source lists and that we do not hold, each with its own licence. Where an open copy is, when `bibliography.oa_status` says one exists.", + "items": { + "$ref": "#/$defs/ReportedCopy" + }, + "type": [ + "array", + "null" + ] + }, "sections": { "items": { "$ref": "#/$defs/Section" @@ -2301,6 +2858,10 @@ "null" ] }, + "text_copy": { + "$ref": "#/$defs/HeldCopy", + "description": "The copy the text was read from. Its licence and access, not the paper's licence, decide whether the text may be passed on." + }, "text_length": { "minimum": 0, "type": [ @@ -2320,6 +2881,7 @@ "required": [ "header", "source", + "text_copy", "text_path", "text_sha256", "bibliography", @@ -2327,5 +2889,5 @@ ], "title": "neuroimaging-paper-parse", "type": "object", - "version": "0.7.0" + "version": "0.8.0" } diff --git a/src/study_schema/jsonschema/storage-study.schema.json b/src/study_schema/jsonschema/storage-study.schema.json index d0dd44c..2c769b8 100644 --- a/src/study_schema/jsonschema/storage-study.schema.json +++ b/src/study_schema/jsonschema/storage-study.schema.json @@ -513,6 +513,66 @@ "title": "ArmKind", "type": "string" }, + "ArticleIdentifiers": { + "additionalProperties": false, + "description": "Every identifier known for the paper when the artifact was written. Carried whole, so no consumer has to choose between them or look one up. `doi`, `pmid` and `pmcid` name the version of record; a preprint's identifiers are kept beside them in the `preprint_` fields, so a paper and its preprint are one article.", + "properties": { + "doi": { + "description": "The DOI of the version of record. Never a preprint server's DOI, which is in `preprint_dois`.", + "type": [ + "string", + "null" + ] + }, + "pmcid": { + "description": "PubMed Central's copy of the version of record. Never PMC's copy of a preprint, which is `preprint_pmcid`.", + "type": [ + "string", + "null" + ] + }, + "pmid": { + "description": "PubMed's record of the version of record. Never PubMed's record of a preprint, which is `preprint_pmid`.", + "type": [ + "string", + "null" + ] + }, + "preprint_dois": { + "description": "Every DOI a preprint server registered for the paper. A server that registers each version separately, such as Research Square's `/v2`, gives one entry per version, each kept as registered.", + "items": { + "$ref": "#/$defs/PreprintDoi" + }, + "type": [ + "array", + "null" + ] + }, + "preprint_pmcid": { + "description": "PubMed Central's copy of the preprint.", + "type": [ + "string", + "null" + ] + }, + "preprint_pmid": { + "description": "PubMed's record of the preprint, which PubMed types `Preprint`.", + "type": [ + "string", + "null" + ] + }, + "publication_status": { + "$ref": "#/$defs/PublicationStatus", + "description": "`preprint` when a `preprint_` field is set and `doi`, `pmid` and `pmcid` are all empty, `published` otherwise. The producer states it so that no consumer re-derives it." + } + }, + "required": [ + "publication_status" + ], + "title": "ArticleIdentifiers", + "type": "object" + }, "Assessment": { "additionalProperties": false, "description": "A behavioral, cognitive, biological, clinical, or questionnaire-based instrument or procedure that produces participant-level measurements independently of the neuroimaging acquisition or experimental conditions.", @@ -559,6 +619,194 @@ "title": "AssignmentStructure", "type": "string" }, + "Bibliography": { + "additionalProperties": false, + "description": "The paper's bibliographic facts, as the metadata lookup returned them. Defined once: the parsed paper carries it, and the storage Study holds the same object as `Study.bibliography`, so nothing is copied field by field and nothing is extracted.", + "else": { + "properties": { + "abstract_provider": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + } + } + }, + "if": { + "properties": { + "abstract": { + "anyOf": [ + { + "type": "string" + } + ] + } + }, + "required": [ + "abstract" + ] + }, + "properties": { + "abstract": { + "description": "Absent (null) when no source gave one; never an empty string.", + "pattern": "\\S", + "type": [ + "string", + "null" + ] + }, + "abstract_provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The source whose abstract `abstract` is. It may differ from `provider`. The abstract is the publisher's abstract of the version `license` describes, so its licence is `license`." + }, + "authors": { + "description": "Author names in published order.", + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ] + }, + "corrections": { + "description": "Notices attached to the paper -- errata, retractions, expressions of concern -- as the indexer lists them. A retracted paper is still parsed and stored, marked by its `retraction` notice, and left out of studysets by default; a filter can ask for it explicitly.", + "items": { + "$ref": "#/$defs/Correction" + }, + "type": [ + "array", + "null" + ] + }, + "issue": { + "type": [ + "string", + "null" + ] + }, + "journal": { + "type": [ + "string", + "null" + ] + }, + "keywords": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ] + }, + "language": { + "description": "Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as.\nAt least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.", + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ] + }, + "license": { + "$ref": "#/$defs/License", + "description": "The paper's licence: the version of record's, or the latest preprint's while the header's `publication_status` is `preprint`. Not the licence of a copy of another version, which that copy carries." + }, + "oa_status": { + "anyOf": [ + { + "$ref": "#/$defs/OaStatus" + }, + { + "type": "null" + } + ] + }, + "pages": { + "type": [ + "string", + "null" + ] + }, + "provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The metadata service the facts came from." + }, + "publication_types": { + "description": "PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels. `StudyType` lists the known types only; any other type PubMed returns is kept verbatim.", + "items": { + "anyOf": [ + { + "$ref": "#/$defs/StudyType" + }, + { + "type": "string" + } + ] + }, + "type": [ + "array", + "null" + ] + }, + "publication_year": { + "description": "Year in which the study was first published or made publicly available.", + "minimum": 1600, + "type": [ + "integer", + "null" + ] + }, + "title": { + "type": "string" + }, + "volume": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "title", + "license" + ], + "then": { + "properties": { + "abstract_provider": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + }, + "required": [ + "abstract_provider" + ] + }, + "title": "Bibliography", + "type": "object" + }, "Blinding": { "description": "Who was kept unaware of the assignment. Open vocabulary: write the source's own wording in the field when nothing here fits.", "enum": [ @@ -1079,6 +1327,51 @@ "title": "CoordinateSet", "type": "object" }, + "Correction": { + "additionalProperties": false, + "description": "One notice attached to a paper.", + "properties": { + "doi": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "anyOf": [ + { + "$ref": "#/$defs/CorrectionKind" + }, + { + "type": "string" + } + ] + }, + "pmid": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "kind" + ], + "title": "Correction", + "type": "object" + }, + "CorrectionKind": { + "description": "A notice attached to a paper by its publisher or indexer.", + "enum": [ + "erratum", + "retraction", + "expression_of_concern", + "comment", + "update" + ], + "title": "CorrectionKind", + "type": "string" + }, "DecodingClass": { "additionalProperties": false, "description": "One class discriminated by a decoding analysis.", @@ -2366,6 +2659,127 @@ "title": "LatentDecompositionDetails", "type": "object" }, + "License": { + "additionalProperties": false, + "description": "A licence as one source stated it, or all rights reserved assumed because no source stated one.", + "else": { + "properties": { + "license_id": { + "const": "all-rights-reserved" + }, + "stated_as": { + "not": { + "anyOf": [ + { + "type": "string" + } + ] + } + }, + "stated_by": { + "not": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + } + } + }, + "if": { + "properties": { + "determination": { + "const": "stated" + } + }, + "required": [ + "determination" + ] + }, + "properties": { + "determination": { + "$ref": "#/$defs/LicenseDetermination" + }, + "license_id": { + "$ref": "#/$defs/LicenseId", + "description": "Which licence, as OpenAlex's licence slugs name them (`cc-by`, `cc-by-nc`, ...). Not an SPDX License Identifier: sources rarely state the licence's version, and SPDX identifiers require one." + }, + "stated_as": { + "description": "The statement exactly as the source gave it: a licence URL, a licence name, or a copyright line. What `other` holds, and what attribution quotes. Never empty: a source that gave no statement did not state a licence.", + "pattern": "\\S", + "type": [ + "string", + "null" + ] + }, + "stated_by": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + }, + { + "type": "null" + } + ], + "description": "The source whose statement this is." + } + }, + "required": [ + "license_id", + "determination" + ], + "then": { + "properties": { + "stated_as": { + "anyOf": [ + { + "type": "string" + } + ] + }, + "stated_by": { + "anyOf": [ + { + "$ref": "#/$defs/MetadataSource" + } + ] + } + }, + "required": [ + "stated_by", + "stated_as" + ] + }, + "title": "License", + "type": "object" + }, + "LicenseDetermination": { + "description": "Whether a source stated the licence.", + "enum": [ + "stated", + "assumed" + ], + "title": "LicenseDetermination", + "type": "string" + }, + "LicenseId": { + "description": "The terms a licence grants. A value annotated `redistributable` lets the text it covers be passed on verbatim with attribution: every Creative Commons licence, including ND, because converting a format to text is a change CC 4.0 2(a)(4) allows, and NC, because neurostore's use is non-commercial. That reading of ND and NC is this project's policy, not legal advice, and no lawyer has confirmed it.", + "enum": [ + "cc-by", + "cc-by-sa", + "cc-by-nc", + "cc-by-nc-sa", + "cc-by-nd", + "cc-by-nc-nd", + "cc0", + "public-domain", + "all-rights-reserved", + "other" + ], + "title": "LicenseId", + "type": "string" + }, "MRI": { "additionalProperties": false, "description": "An MRI acquisition of any kind -- functional, structural, or diffusion, as modality records. Its own attributes are the ones commonly useful for study selection, subgrouping, or sensitivity analyses in coordinate- or image-based meta-analysis.", @@ -2638,6 +3052,21 @@ "title": "Mediation", "type": "object" }, + "MetadataSource": { + "description": "Who stated a bibliographic fact.", + "enum": [ + "jats", + "elsevier", + "pmc_cloud", + "pubmed", + "crossref", + "openalex", + "semantic_scholar", + "europepmc" + ], + "title": "MetadataSource", + "type": "string" + }, "Modality": { "description": "Imaging or recording modality of one Acquisition, and through `instantiates` the class in this module holding that modality's parameters. fMRI, sMRI, dMRI, and MRI all instantiate MRI; EEG, fNIRS, and PET instantiate their namesakes; MEG, SPECT, and other instantiate OtherModality. `instantiates` carries a CURIE against this module's own `neurostudy` prefix, so `neurostudy:MRI` resolves to the MRI class below.", "enum": [ @@ -2941,6 +3370,19 @@ "title": "NullHypothesis", "type": "string" }, + "OaStatus": { + "description": "How a paper is open to read, in OpenAlex's terms (https://help.openalex.org/hc/en-us/articles/24347035046295-Open-Access-OA).", + "enum": [ + "gold", + "diamond", + "hybrid", + "bronze", + "green", + "closed" + ], + "title": "OaStatus", + "type": "string" + }, "OtherAnalysisDetails": { "additionalProperties": false, "description": "Payload for a method outside the first-class families. Available for any analysis, so an unusual design never has to delete true terms in order to validate.", @@ -3246,6 +3688,63 @@ "title": "PolaritySemantics", "type": "string" }, + "PreprintDoi": { + "additionalProperties": false, + "description": "A DOI a preprint server registered, with the server that registered it.", + "properties": { + "doi": { + "type": "string" + }, + "server": { + "$ref": "#/$defs/PreprintServer", + "description": "Named by the producer from the DOI, so that no consumer needs a table of preprint servers' DOI patterns." + } + }, + "required": [ + "doi", + "server" + ], + "title": "PreprintDoi", + "type": "object" + }, + "PreprintServer": { + "description": "The preprint server that registered a DOI. A server hosted on the Open Science Framework is named by its provider, which has a DOI prefix of its own. A server is added here when a producer learns to recognise its DOIs.", + "enum": [ + "biorxiv", + "medrxiv", + "arxiv", + "researchsquare", + "osf_preprints", + "arabixiv", + "metaarxiv", + "engrxiv", + "focus_archive", + "frenxiv", + "ina_rxiv", + "lawarxiv", + "lis_scholarship_archive", + "marxiv", + "mindrxiv", + "nutrixiv", + "paleorxiv", + "psyarxiv", + "socarxiv", + "sportrxiv", + "africarxiv", + "mediarxiv", + "bodoarxiv", + "edarxiv", + "biohackrxiv", + "ssrn", + "preprints_org", + "authorea", + "techrxiv", + "chemrxiv", + "jmir_preprints" + ], + "title": "PreprintServer", + "type": "string" + }, "Preprocessing": { "additionalProperties": false, "description": "An ordered preprocessing pipeline applied to acquired data before or as part of statistical modeling. This can be expanded with COBIDAS.", @@ -3738,15 +4237,9 @@ "null" ] }, - "authors": { - "description": "Authors of the study or publication, in published order.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] + "bibliography": { + "$ref": "#/$defs/Bibliography", + "description": "The paper's bibliographic facts, the parsed paper's `Bibliography` unchanged: title, authors, journal, year, abstract, PubMed types, the paper's licence." }, "coordinate_sets": { "description": "Every group of coordinates the paper reports, in its tables or its text, with what each is for.", @@ -3786,13 +4279,6 @@ "null" ] }, - "doi": { - "description": "Digital Object Identifier of the version of record, without requiring a particular URL form. Never a preprint server's DOI.", - "type": [ - "string", - "null" - ] - }, "external_datasets": { "description": "Named external datasets used, cited, or made available by the study.", "items": { @@ -3827,6 +4313,17 @@ "description": "Stable identifier for the study record, such as a DOI-based or repository-local identifier.", "type": "string" }, + "identifiers": { + "anyOf": [ + { + "$ref": "#/$defs/ArticleIdentifiers" + }, + { + "type": "null" + } + ], + "description": "Every identifier known for the paper, the parsed paper's header identifiers unchanged." + }, "inference_settings": { "description": "Thresholding and correction schemes used by one or more analyses. Declared here once and referenced by identifier.", "items": { @@ -3837,23 +4334,6 @@ "null" ] }, - "journal": { - "description": "Journal or other publication venue.", - "type": [ - "string", - "null" - ] - }, - "language": { - "description": "Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest. Filled from the same `esummary` call that fills `study_type`, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as.\nAt least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion, and it was unexpressible. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] - }, "measures": { "description": "The scientific quantities the study measured, used by one or more analyses. Declared here once and referenced by identifier.", "items": { @@ -3895,25 +4375,6 @@ "null" ] }, - "publication_status": { - "anyOf": [ - { - "$ref": "#/$defs/PublicationStatus" - }, - { - "type": "null" - } - ], - "description": "Whether the version of record is known or only the preprint, copied from the parsed paper's header identifiers. `study_type` is independent of it: it keeps PubMed's publication types verbatim, `Preprint` included." - }, - "publication_year": { - "description": "Year in which the study was first published or made publicly available.", - "minimum": 1600, - "type": [ - "integer", - "null" - ] - }, "regions": { "description": "Brain regions the study delimited and then used -- as connectivity seeds or targets, as the search space an analysis was restricted to, as the levels of a factor comparing regions, or as the signal a model term carries. Declared here once and referenced by identifier wherever used.", "items": { @@ -3935,20 +4396,6 @@ ], "description": "Whether the paper reports inferential analyses of its own. An empty `analyses` list is otherwise ambiguous: a methods or data paper that tests nothing and a paper the extractor read badly look the same. With this set, `no_analyses_reported` asserts the emptiness, with the sentences that show it as evidence.\nSays nothing about whether the analyses found anything; that is each `Analysis.outcome`, summarised in `outcome_summary`." }, - "study_type": { - "description": "PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels.", - "items": { - "anyOf": [ - { - "$ref": "#/$defs/StudyType" - }, - { - "type": "string" - } - ] - }, - "type": "array" - }, "tables": { "description": "Publication tables from which results or analysis metadata were extracted.", "items": { @@ -3968,16 +4415,11 @@ "array", "null" ] - }, - "title": { - "description": "Full title of the study or publication.", - "type": "string" } }, "required": [ "id", - "title", - "study_type" + "bibliography" ], "title": "Study", "type": "object" @@ -4548,15 +4990,9 @@ "null" ] }, - "authors": { - "description": "Authors of the study or publication, in published order.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] + "bibliography": { + "$ref": "#/$defs/Bibliography", + "description": "The paper's bibliographic facts, the parsed paper's `Bibliography` unchanged: title, authors, journal, year, abstract, PubMed types, the paper's licence." }, "coordinate_sets": { "description": "Every group of coordinates the paper reports, in its tables or its text, with what each is for.", @@ -4596,13 +5032,6 @@ "null" ] }, - "doi": { - "description": "Digital Object Identifier of the version of record, without requiring a particular URL form. Never a preprint server's DOI.", - "type": [ - "string", - "null" - ] - }, "external_datasets": { "description": "Named external datasets used, cited, or made available by the study.", "items": { @@ -4637,6 +5066,17 @@ "description": "Stable identifier for the study record, such as a DOI-based or repository-local identifier.", "type": "string" }, + "identifiers": { + "anyOf": [ + { + "$ref": "#/$defs/ArticleIdentifiers" + }, + { + "type": "null" + } + ], + "description": "Every identifier known for the paper, the parsed paper's header identifiers unchanged." + }, "inference_settings": { "description": "Thresholding and correction schemes used by one or more analyses. Declared here once and referenced by identifier.", "items": { @@ -4647,23 +5087,6 @@ "null" ] }, - "journal": { - "description": "Journal or other publication venue.", - "type": [ - "string", - "null" - ] - }, - "language": { - "description": "Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest. Filled from the same `esummary` call that fills `study_type`, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as.\nAt least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion, and it was unexpressible. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.", - "items": { - "type": "string" - }, - "type": [ - "array", - "null" - ] - }, "measures": { "description": "The scientific quantities the study measured, used by one or more analyses. Declared here once and referenced by identifier.", "items": { @@ -4705,25 +5128,6 @@ "null" ] }, - "publication_status": { - "anyOf": [ - { - "$ref": "#/$defs/PublicationStatus" - }, - { - "type": "null" - } - ], - "description": "Whether the version of record is known or only the preprint, copied from the parsed paper's header identifiers. `study_type` is independent of it: it keeps PubMed's publication types verbatim, `Preprint` included." - }, - "publication_year": { - "description": "Year in which the study was first published or made publicly available.", - "minimum": 1600, - "type": [ - "integer", - "null" - ] - }, "regions": { "description": "Brain regions the study delimited and then used -- as connectivity seeds or targets, as the search space an analysis was restricted to, as the levels of a factor comparing regions, or as the signal a model term carries. Declared here once and referenced by identifier wherever used.", "items": { @@ -4745,20 +5149,6 @@ ], "description": "Whether the paper reports inferential analyses of its own. An empty `analyses` list is otherwise ambiguous: a methods or data paper that tests nothing and a paper the extractor read badly look the same. With this set, `no_analyses_reported` asserts the emptiness, with the sentences that show it as evidence.\nSays nothing about whether the analyses found anything; that is each `Analysis.outcome`, summarised in `outcome_summary`." }, - "study_type": { - "description": "PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels.", - "items": { - "anyOf": [ - { - "$ref": "#/$defs/StudyType" - }, - { - "type": "string" - } - ] - }, - "type": "array" - }, "tables": { "description": "Publication tables from which results or analysis metadata were extracted.", "items": { @@ -4778,18 +5168,13 @@ "array", "null" ] - }, - "title": { - "description": "Full title of the study or publication.", - "type": "string" } }, "required": [ "id", - "title", - "study_type" + "bibliography" ], "title": "neuroimaging-study-storage", "type": "object", - "version": "0.8.0" + "version": "0.9.0" } diff --git a/src/study_schema/licensing.py b/src/study_schema/licensing.py new file mode 100644 index 0000000..9af5111 --- /dev/null +++ b/src/study_schema/licensing.py @@ -0,0 +1,87 @@ +"""Whether the text or the abstract of a parsed paper may be passed on. + +Nothing in a parsed paper or a record says this; it is derived here from the facts they +hold. ``REDISTRIBUTABLE_LICENSES`` is the LicenseId values annotated ``redistributable`` in +neuroimaging-study-storage/license.yaml, and ``REDISTRIBUTABLE_ACCESS`` the CopyAccess values +annotated ``redistribution: by_license`` in neuroimaging-paper-parse/parse_enums.yaml; +tests/test_licensing.py keeps them equal. + +Two questions, one function each, so that neither can be asked with the other's facts: + +- The parsed text: ``text_may_be_redistributed(paper.text_copy)``. It takes the held copy, + so the access it was obtained under is always weighed with its licence. +- The abstract: ``abstract_may_be_redistributed(paper.bibliography.license)``. The indexers + carry the publisher's abstract of the version that licence describes. + +The reading is a policy, not legal advice: every CC licence counts as redistributable, +NC and ND included, and ``other`` never does, so Elsevier's open-access user licence is +refused on purpose. The JSON Schema is the validator of a parse or record; these functions +assume nothing it checks, and answer False for anything they do not recognise as +permitted. +""" + +from typing import Any, Mapping, Optional + +REDISTRIBUTABLE_LICENSES = frozenset( + { + "cc-by", + "cc-by-sa", + "cc-by-nc", + "cc-by-nc-sa", + "cc-by-nd", + "cc-by-nc-nd", + "cc0", + "public-domain", + } +) +REDISTRIBUTABLE_ACCESS = frozenset({"public"}) + + +def _value(item: Any) -> Optional[str]: + return getattr(item, "value", item) + + +def _field(license: Any, name: str) -> Any: + if isinstance(license, Mapping): + return license.get(name) + return getattr(license, name, None) + + +def _license_permits(license: Any) -> bool: + """A redistributable licence id, stated by a named source in a non-empty statement.""" + if license is None or _value(_field(license, "license_id")) not in REDISTRIBUTABLE_LICENSES: + return False + stated_as = _field(license, "stated_as") + return ( + _value(_field(license, "determination")) == "stated" + and bool(_field(license, "stated_by")) + and isinstance(stated_as, str) + and bool(stated_as.strip()) + ) + + +def abstract_may_be_redistributed(paper_license: Any) -> bool: + """True when the paper's licence (a License dict or model, ``Bibliography.license``) + lets its abstract be passed on. + + False, never an error, for anything else: no licence, an unknown licence id, or a + redistributable licence id that is not stated by a named source in a non-empty statement, + which the schema's rules forbid (an assumed licence is always all rights reserved). + """ + return _license_permits(paper_license) + + +def text_may_be_redistributed(held_copy: Any) -> bool: + """True when the held copy the text was read from (a HeldCopy dict or model, + ``ParsedPaper.text_copy``) may be passed on: its licence permits it and it was + obtained publicly. + + False, never an error, for anything else: no copy, a copy without an access or with + one outside CopyAccess, or a licence ``abstract_may_be_redistributed`` would refuse. + """ + if held_copy is None: + return False + return ( + _value(_field(held_copy, "access")) in REDISTRIBUTABLE_ACCESS + and _license_permits(_field(held_copy, "license")) + ) diff --git a/src/study_schema/linkml/extraction-deviations.yaml b/src/study_schema/linkml/extraction-deviations.yaml index 2041f48..e8e84ea 100644 --- a/src/study_schema/linkml/extraction-deviations.yaml +++ b/src/study_schema/linkml/extraction-deviations.yaml @@ -29,7 +29,7 @@ version: 0.1.0 # record the mapper writes; this one describes what a model read. header: - version: 0.5.0 + version: 0.6.0 description: >- Immutable, evidence-first structured output from extracting a neuroimaging @@ -341,57 +341,12 @@ deviations: what says so. Without it the two halves look like independent findings, and a meta-analysis that counts both counts one sample twice. - - id: language-is-copied-from-the-parsed-paper-onto-the-record - operation: add_slot - class: study.Study - slot: language - why: >- - The same case as `study_type`, copied from the same place: storage marks it - `deterministic` so the projection drops it, and the value is copied onto the record on - the extraction side. At least six of the benchmark's sixteen meta-analyses state "English - language" as an inclusion criterion and no slot held it, so the criterion could not be - written at all. Measured over 1,804 records the field excludes one paper, which is the - point: a criterion that is stated should be expressible even where it is nearly inert, - and the alternative is a query that silently ignores it. - definition: - range: string - multivalued: true - description: >- - Languages of this article, copied from the parsed paper's `Bibliography.language` - alongside `study_type`, not read off the paper: three-letter codes, `eng` and the - rest. - - - id: publication-type-is-copied-from-the-parsed-paper-onto-the-record - operation: add_slot - class: study.Study - slot: study_type - why: >- - Storage marks this `deterministic`, so the projection drops it, and the same argument - as `mirror_of` applies: the value is copied onto the record on the extraction side, not - at ingestion, so the slot has to exist in the extraction record even though no pass - extracts it. Measured against the queries the benchmark's own meta-analyses ran, six - of sixteen exclude by publication type -- "editorial letters, case-reports, systematic - reviews, meta-analyses" (sleep deprivation), "systematic reviews or meta-analyses" - (social) -- and the criterion is unexpressible today because the slot is `None` on all - 1,817 records with no writer anywhere in the package. - - Native strings and not an `ExtractedValue`: PubMed is not the paper, so there is no - span in the source text to cite and `evidence` would have nothing honest to say. The - same reason `local_id` and the provenance fields are native. - definition: - range: string - multivalued: true - description: >- - PubMed publication types for this record, verbatim, copied from the parsed paper's - `Bibliography.publication_types`, not read off the paper: it is what the indexers - said about the article, which is what a publication-type exclusion asks about. - - id: is-healthy-is-derived-onto-the-record operation: add_slot class: group.Group slot: is_healthy why: >- - The same case as `study_type` and `mirror_of`, and it was already being written + The same case as `mirror_of`, and it was already being written without the declaration: `pondie.normalization.is_healthy.apply` fills the slot from `medical_condition` after extraction, and the validator reported it as an undeclared attribute on every group it touched -- 4,133 groups across the committed corpus. diff --git a/src/study_schema/linkml/extraction-to-storage.map.yaml b/src/study_schema/linkml/extraction-to-storage.map.yaml index 11b2882..55dd9c6 100644 --- a/src/study_schema/linkml/extraction-to-storage.map.yaml +++ b/src/study_schema/linkml/extraction-to-storage.map.yaml @@ -52,7 +52,7 @@ id: https://neurostuff.org/schemas/extraction-to-storage-map name: extraction-to-storage-map title: Neuroimaging Study Extraction-to-Storage Transformation Map -version: 0.6.0 +version: 0.7.0 source_schema: https://neurostuff.org/schemas/neuroimaging-study-extraction target_schema: https://neurostuff.org/schemas/neuroimaging-study-storage @@ -120,38 +120,24 @@ derivations: transform: generate Preprocessing.id: transform: generate - Study.authors: + Study.bibliography: transform: copy source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: Bibliography.authors + source_field: ParsedPaper.bibliography notes: >- - Author list in published order, copied from the parsed paper. Not extracted: a - bibliographic fact the metadata lookup states is not one a model should be asked to - read off the page. - Study.doi: + The same Bibliography class on both sides, held whole: title, authors, journal, year, + abstract, PubMed types and the paper's licence. Not extracted: bibliographic facts the + metadata lookup states are not ones a model should be asked to read off the page. + Study.identifiers: transform: copy source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: ArticleIdentifiers.doi + source_field: ArtifactHeader.identifiers notes: >- - DOI from the parsed paper's header identifiers. Not extracted: a bibliographic fact - the metadata lookup states is not one a model should be asked to read off the page. - Study.publication_status: - transform: copy - source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: ArticleIdentifiers.publication_status - notes: >- - Stated by the producer of the parsed paper from the identifiers it holds. Not - extracted: whether a version of record exists is not something the text says. + The same ArticleIdentifiers class on both sides, held whole, publication_status + included. Study.id: transform: generate notes: Assigned by the ingestion pipeline; typically DOI-derived (doi:10.../record) or a UUID. - Study.journal: - transform: copy - source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: Bibliography.journal - notes: >- - Journal title, copied from the parsed paper. Not extracted: a bibliographic fact the - metadata lookup states is not one a model should be asked to read off the page. Task.paradigm_category: transform: derive source_representation: corpus @@ -184,37 +170,6 @@ derivations: null; no_analyses when result_reporting says the paper reports none here and analyses is empty; undetermined otherwise. Derived rather than extracted so it cannot contradict the outcomes it summarises. - Study.publication_year: - transform: copy - source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: Bibliography.publication_year - notes: >- - Publication year, copied from the parsed paper. Not extracted: a bibliographic fact - the metadata lookup states is not one a model should be asked to read off the page. - Study.language: - transform: copy - source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: Bibliography.language - notes: >- - Article languages, copied from the parsed paper. Not extracted: what language an - article is indexed in is a bibliographic fact the metadata lookup states, and the text - a reader is handed is in one language whatever that fact says. - Study.study_type: - transform: copy - source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: Bibliography.publication_types - notes: >- - PubMed publication types, copied verbatim from the parsed paper and multivalued. Not - extracted: the LLM would be guessing a bibliographic fact the metadata lookup states, - and the value gates extraction rather than following it -- a paper whose - ParsedPaper.is_meta_analysis is true is skipped before any text is read. - Study.title: - transform: copy - source_artifact: neuroimaging-paper-parse/ParsedPaper - source_field: Bibliography.title - notes: >- - Article title, copied from the parsed paper. Not extracted: a bibliographic fact the - metadata lookup states is not one a model should be asked to read off the page. Group.is_healthy: transform: derive source_representation: sibling_field diff --git a/src/study_schema/linkml/neuroimaging-paper-parse.yaml b/src/study_schema/linkml/neuroimaging-paper-parse.yaml index 197170a..f538d69 100644 --- a/src/study_schema/linkml/neuroimaging-paper-parse.yaml +++ b/src/study_schema/linkml/neuroimaging-paper-parse.yaml @@ -29,7 +29,7 @@ description: >- record so the stored analyses match the analyses the record describes. license: CC0-1.0 -version: 0.7.0 +version: 0.8.0 prefixes: linkml: https://w3id.org/linkml/ diff --git a/src/study_schema/linkml/neuroimaging-paper-parse/header.yaml b/src/study_schema/linkml/neuroimaging-paper-parse/header.yaml index 6380b54..7ffd70b 100644 --- a/src/study_schema/linkml/neuroimaging-paper-parse/header.yaml +++ b/src/study_schema/linkml/neuroimaging-paper-parse/header.yaml @@ -11,7 +11,7 @@ prefixes: imports: - linkml:types - parse_enums - - ../neuroimaging-study-storage/analysis_enums + - ../neuroimaging-study-storage/bibliography default_prefix: parse default_range: string @@ -48,66 +48,11 @@ classes: description: Every artifact this one was made from, by fingerprint. created_at: range: datetime - - ArticleIdentifiers: - description: >- - Every identifier known for the paper when the artifact was written. Carried whole, so - no consumer has to choose between them or look one up. `doi`, `pmid` and `pmcid` name - the version of record; a preprint's identifiers are kept beside them in the - `preprint_` fields, so a paper and its preprint are one article. - attributes: - pmid: - range: string - description: >- - PubMed's record of the version of record. Never PubMed's record of a preprint, - which is `preprint_pmid`. - pmcid: - range: string - description: >- - PubMed Central's copy of the version of record. Never PMC's copy of a preprint, - which is `preprint_pmcid`. - doi: - range: string - description: >- - The DOI of the version of record. Never a preprint server's DOI, which is in - `preprint_dois`. - preprint_dois: - range: PreprintDoi - multivalued: true - inlined_as_list: true - description: >- - Every DOI a preprint server registered for the paper. A server that registers each - version separately, such as Research Square's `/v2`, gives one entry per version, - each kept as registered. - preprint_pmid: - range: string - description: PubMed's record of the preprint, which PubMed types `Preprint`. - preprint_pmcid: - range: string - description: PubMed Central's copy of the preprint. - publication_status: - range: PublicationStatus - required: true - description: >- - `preprint` when a `preprint_` field is set and `doi`, `pmid` and `pmcid` are all - empty, `published` otherwise. The producer states it so that no consumer re-derives - it. neurostore_base_study_id: range: string - description: Set once the paper has been uploaded; neurostore's BaseStudy id. - - PreprintDoi: - description: A DOI a preprint server registered, with the server that registered it. - attributes: - doi: - range: string - required: true - server: - range: PreprintServer - required: true description: >- - Named by the producer from the DOI, so that no consumer needs a table of preprint - servers' DOI patterns. + Set once the paper has been uploaded; neurostore's BaseStudy id. An upload pointer, + not a bibliographic identifier, so it sits here and not in `identifiers`. Producer: description: The program and stage that wrote an artifact. diff --git a/src/study_schema/linkml/neuroimaging-paper-parse/paper.yaml b/src/study_schema/linkml/neuroimaging-paper-parse/paper.yaml index d936b9f..b949f8c 100644 --- a/src/study_schema/linkml/neuroimaging-paper-parse/paper.yaml +++ b/src/study_schema/linkml/neuroimaging-paper-parse/paper.yaml @@ -11,6 +11,8 @@ imports: - linkml:types - parse_enums - header + - ../neuroimaging-study-storage/license + - ../neuroimaging-study-storage/bibliography default_prefix: parse default_range: string @@ -41,6 +43,20 @@ classes: Why this source's text was chosen over the others downloaded, such as "most tables" or "first in download order". Every stage reads this one text, so the choice is made once and recorded. + text_copy: + range: HeldCopy + inlined: true + required: true + description: >- + The copy the text was read from. Its licence and access, not the paper's licence, + decide whether the text may be passed on. + reported_copies: + range: ReportedCopy + multivalued: true + inlined_as_list: true + description: >- + Copies of the paper that a source lists and that we do not hold, each with its + own licence. Where an open copy is, when `bibliography.oa_status` says one exists. text_path: range: string required: true @@ -98,58 +114,61 @@ classes: multivalued: true inlined_as_list: true - Bibliography: - description: >- - The paper's bibliographic facts, as the metadata lookup returned them. Downstream they - are copied, never extracted: pondie fills the record's Study.title, authors, doi, - journal, publication_year, language and study_type from here, and neurostore fills - BaseStudy from here. + Copy: + abstract: true + description: One copy of a paper, where it is, which version it is, and its licence. + rules: + - description: A copy on a preprint server, and only such a copy, names the server. + open_world: true + preconditions: + slot_conditions: + route: + equals_string: preprint_server + postconditions: + slot_conditions: + preprint_server: + value_presence: PRESENT + any_of: + - range: PreprintServer + elseconditions: + slot_conditions: + preprint_server: + none_of: + - range: PreprintServer attributes: - title: - range: string + route: + range: CopyRoute required: true - authors: - range: string - multivalued: true - description: Author names in published order. - journal: - range: string - publication_year: - range: integer - volume: - range: string - issue: - range: string - pages: - range: string - abstract: - range: string - language: - range: string - multivalued: true - publication_types: - range: string - multivalued: true - description: PubMed publication types, verbatim. The storage schema's Study.study_type. - keywords: - range: string - multivalued: true + version: + range: CopyVersion + description: Absent when no source says which version the copy is. license: - range: string - open_access: - range: boolean - corrections: - range: Correction - multivalued: true - inlined_as_list: true + range: License + inlined: true + required: true description: >- - Notices attached to the paper -- errata, retractions, expressions of concern -- as - the indexer lists them. A retracted paper is still parsed and stored, marked by - its `retraction` notice, and left out of studysets by default; a filter can ask for - it explicitly. - provider: + This copy's own licence. A manuscript or preprint can be licensed differently from + the version of record. + url: range: string - description: The metadata service the facts came from, such as pubmed or europepmc. + preprint_server: + range: PreprintServer + + HeldCopy: + is_a: Copy + description: A copy we downloaded. + attributes: + access: + range: CopyAccess + required: true + + ReportedCopy: + is_a: Copy + description: A copy a metadata source lists and we do not hold. + attributes: + reported_by: + range: MetadataSource + required: true Reference: description: One entry of a paper's reference list. @@ -213,19 +232,6 @@ classes: maximum_value: 1 description: For a heuristic alignment, how sure; absent for an exact one. - Correction: - description: One notice attached to a paper. - attributes: - kind: - any_of: - - range: CorrectionKind - - range: string - required: true - pmid: - range: string - doi: - range: string - Supplement: description: One item of supplementary material. attributes: diff --git a/src/study_schema/linkml/neuroimaging-paper-parse/parse_enums.yaml b/src/study_schema/linkml/neuroimaging-paper-parse/parse_enums.yaml index 2c5fd81..db0dcc1 100644 --- a/src/study_schema/linkml/neuroimaging-paper-parse/parse_enums.yaml +++ b/src/study_schema/linkml/neuroimaging-paper-parse/parse_enums.yaml @@ -3,9 +3,9 @@ name: neuroimaging-paper-parse-enums title: Neuroimaging Paper Parse Vocabularies description: >- The vocabularies of the parsed paper and the coordinate parse. Coordinate roles, anchor - kinds and publication status are not here: they are the storage schema's - `CoordinateRole`, `AnchorKind` and `PublicationStatus`, imported, so that the parse and - the record write them in the same words. + kinds, licences and the bibliography's vocabularies are not here: they are the storage + schema's `CoordinateRole`, `AnchorKind` and its licence and bibliography modules, + imported, so that the parse and the record write them in the same words. prefixes: linkml: https://w3id.org/linkml/ parse: https://neurostuff.org/schemas/neuroimaging-paper-parse/ @@ -44,74 +44,54 @@ enums: pdf: description: A PDF converted to text. - PreprintServer: + CopyRoute: + description: Where a copy of a paper is hosted. + permissible_values: + pmc: + description: PubMed Central or Europe PMC. + publisher: + description: The publisher's own site or full-text API. + repository: + description: An institutional or subject repository other than PMC. + preprint_server: + description: A preprint server; the copy's `preprint_server` names it. + unrecorded: + description: Where the copy came from was not recorded, such as a PDF supplied by hand. + + CopyVersion: + description: >- + Which version of the paper a copy is, in the NISO Journal Article Versions terms. + OpenAlex's `publishedVersion`, `acceptedVersion` and `submittedVersion` are `vor`, + `am` and `submitted`. + permissible_values: + vor: + description: The version of record, as the journal published it. + am: + description: The accepted manuscript, after peer review and before the publisher's typesetting. + submitted: + description: The version submitted to a journal or posted as a preprint, before peer review. + + CopyAccess: description: >- - The preprint server that registered a DOI. A server hosted on the Open Science - Framework is named by its provider, which has a DOI prefix of its own. A server is - added here when a producer learns to recognise its DOIs. + How we obtained a copy we hold. The `redistribution` annotation says whether the + access itself allows passing the copy on: `by_license` leaves that to the copy's + licence, `never` forbids it whatever the licence says. permissible_values: - biorxiv: - description: bioRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). - medrxiv: - description: medRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). - arxiv: - description: arXiv, DOI prefix 10.48550, the DOI registered for an arXiv id. - researchsquare: - description: Research Square, DOI prefix 10.21203. - osf_preprints: - description: OSF Preprints, DOI prefix 10.31219. - arabixiv: - description: Arabixiv (OSF), DOI prefix 10.31221. - metaarxiv: - description: MetaArXiv (OSF), DOI prefix 10.31222. - engrxiv: - description: engrXiv (OSF), DOI prefix 10.31224. - focus_archive: - description: FocUS Archive (OSF), DOI prefix 10.31225. - frenxiv: - description: Frenxiv (OSF), DOI prefix 10.31226. - ina_rxiv: - description: INA-Rxiv (OSF), DOI prefix 10.31227. - lawarxiv: - description: LawArXiv (OSF), DOI prefix 10.31228. - lis_scholarship_archive: - description: LIS Scholarship Archive (OSF), DOI prefix 10.31229. - marxiv: - description: MarXiv (OSF), DOI prefix 10.31230. - mindrxiv: - description: MindRxiv (OSF), DOI prefix 10.31231. - nutrixiv: - description: NutriXiv (OSF), DOI prefix 10.31232. - paleorxiv: - description: PaleorXiv (OSF), DOI prefix 10.31233. - psyarxiv: - description: PsyArXiv (OSF), DOI prefix 10.31234. - socarxiv: - description: SocArXiv (OSF), DOI prefix 10.31235. - sportrxiv: - description: SportRxiv (OSF), DOI prefix 10.31236. - africarxiv: - description: AfricArXiv (OSF), DOI prefix 10.31730. - mediarxiv: - description: MediArXiv (OSF), DOI prefix 10.33767. - bodoarxiv: - description: BodoArXiv (OSF), DOI prefix 10.34055. - edarxiv: - description: EdArXiv (OSF), DOI prefix 10.35542. - biohackrxiv: - description: BioHackrXiv (OSF), DOI prefix 10.37044. - ssrn: - description: SSRN, DOI prefix 10.2139. - preprints_org: - description: Preprints.org, DOI prefix 10.20944. - authorea: - description: Authorea, DOI prefix 10.22541. - techrxiv: - description: TechRxiv, DOI prefix 10.36227. - chemrxiv: - description: ChemRxiv, DOI prefix 10.26434. - jmir_preprints: - description: JMIR Preprints, DOI prefix 10.2196. + public: + description: Fetched as any reader could fetch it, without our credentials. + annotations: + redistribution: by_license + tdm_grant: + description: >- + Obtained under terms that permit text mining only: a publisher's full-text API + with our key, such as Elsevier's or Wiley's, or a PubMed Central author manuscript + whose record says `TDM`. + annotations: + redistribution: never + unrecorded: + description: How the copy was obtained was not recorded. + annotations: + redistribution: never SectionKind: description: What a span of the text is. @@ -311,15 +291,6 @@ enums: excluded_by_hand: description: A person marked it as not a coordinate table, so it was not read. - CorrectionKind: - description: A notice attached to a paper by its publisher or indexer. - permissible_values: - erratum: - retraction: - expression_of_concern: - comment: - update: - ThresholdLevel: description: The level a threshold applies at. permissible_values: diff --git a/src/study_schema/linkml/neuroimaging-study-extraction.yaml b/src/study_schema/linkml/neuroimaging-study-extraction.yaml index 5232b6e..1cf7662 100644 --- a/src/study_schema/linkml/neuroimaging-study-extraction.yaml +++ b/src/study_schema/linkml/neuroimaging-study-extraction.yaml @@ -15,7 +15,7 @@ description: |- Evidence is present when one or more independently sufficient EvidenceSets were found, or not_found when a value was extracted but no span supports it. For a not_reported attribute, omit value and set evidence.status to not_applicable. Entities refer to each other by document-local local_id; ingestion resolves them and mints the storage identifiers. Do not manually correct an extraction record: create a new extraction version or a separate correction record. Rules this schema cannot state -- which papers are skipped before extraction, how a multivalued extracted fact is shaped, which invariants the structural validator enforces, and what a paper may report that no field holds -- are in extraction-readme.md. Everything this schema does that is not a projection of storage is in extraction-deviations.yaml. license: CC0-1.0 -version: 0.5.0 +version: 0.6.0 see_also: - https://github.com/neurostuff/study_schema/issues/3 - https://github.com/neurostuff/study_schema/issues/5 diff --git a/src/study_schema/linkml/neuroimaging-study-extraction/study.yaml b/src/study_schema/linkml/neuroimaging-study-extraction/study.yaml index 9b028db..ff2d515 100644 --- a/src/study_schema/linkml/neuroimaging-study-extraction/study.yaml +++ b/src/study_schema/linkml/neuroimaging-study-extraction/study.yaml @@ -177,20 +177,6 @@ classes: multivalued: true inlined_as_list: true description: Named external datasets used, cited, or made available by the study. - language: - range: string - multivalued: true - description: >- - Languages of this article, copied from the parsed paper's `Bibliography.language` - alongside `study_type`, not read off the paper: three-letter codes, `eng` and the - rest. - study_type: - range: string - multivalued: true - description: >- - PubMed publication types for this record, verbatim, copied from the parsed paper's - `Bibliography.publication_types`, not read off the paper: it is what the indexers - said about the article, which is what a publication-type exclusion asks about. ExternalDataset: description: A named external dataset, with an optional public location. attributes: diff --git a/src/study_schema/linkml/neuroimaging-study-storage.yaml b/src/study_schema/linkml/neuroimaging-study-storage.yaml index cabba9d..6825267 100644 --- a/src/study_schema/linkml/neuroimaging-study-storage.yaml +++ b/src/study_schema/linkml/neuroimaging-study-storage.yaml @@ -30,7 +30,7 @@ description: >- stage can map and which facets have no field to map. license: CC0-1.0 -version: 0.8.0 +version: 0.9.0 see_also: - https://www.humanbrainmapping.org/files/2016/COBIDASreport.pdf - https://github.com/ohbm/cobidas_schema @@ -58,6 +58,8 @@ imports: - neuroimaging-study-storage/subsets - neuroimaging-study-storage/design - neuroimaging-study-storage/study + - neuroimaging-study-storage/license + - neuroimaging-study-storage/bibliography - neuroimaging-study-storage/group - neuroimaging-study-storage/task - neuroimaging-study-storage/assessment diff --git a/src/study_schema/linkml/neuroimaging-study-storage/analysis_enums.yaml b/src/study_schema/linkml/neuroimaging-study-storage/analysis_enums.yaml index e05222b..db323d3 100644 --- a/src/study_schema/linkml/neuroimaging-study-storage/analysis_enums.yaml +++ b/src/study_schema/linkml/neuroimaging-study-storage/analysis_enums.yaml @@ -1056,16 +1056,3 @@ enums: node: in_subset: [mvp, model_extracted] description: An atlas parcel or network node definition. - - PublicationStatus: - in_subset: [mvp, deterministic] - description: Whether a paper's version of record is known, or only its preprint. - permissible_values: - published: - in_subset: [mvp, deterministic] - description: >- - A DOI, PubMed record or PMC copy of the version of record is known. The paper may - have been a preprint too. - preprint: - in_subset: [mvp, deterministic] - description: Only a preprint's identifiers are known. diff --git a/src/study_schema/linkml/neuroimaging-study-storage/bibliography.yaml b/src/study_schema/linkml/neuroimaging-study-storage/bibliography.yaml new file mode 100644 index 0000000..c129666 --- /dev/null +++ b/src/study_schema/linkml/neuroimaging-study-storage/bibliography.yaml @@ -0,0 +1,399 @@ +id: https://neurostuff.org/schemas/neuroimaging-study-storage/bibliography +name: neuroimaging-study-storage-bibliography +title: Neuroimaging Study Storage Bibliography Module +description: >- + A paper's bibliographic facts and identifiers, defined once. The paper-parse schema imports + this module, so the parsed paper and the storage Study hold the same `Bibliography` and + `ArticleIdentifiers`, and the record never restates one of their fields. +prefixes: + linkml: https://w3id.org/linkml/ + neurostudy: https://neurostuff.org/schemas/neuroimaging-study-storage/ +imports: + - linkml:types + - subsets + - license +default_prefix: neurostudy +default_range: string + +classes: + + Bibliography: + description: >- + The paper's bibliographic facts, as the metadata lookup returned them. Defined once: + the parsed paper carries it, and the storage Study holds the same object as + `Study.bibliography`, so nothing is copied field by field and nothing is extracted. + rules: + # `any_of` / `none_of` over the range, rather than value_presence alone, so that a + # field written as null counts as absent. + - description: An abstract, and only an abstract, says which source it came from. + open_world: true + preconditions: + slot_conditions: + abstract: + any_of: + - range: string + postconditions: + slot_conditions: + abstract_provider: + value_presence: PRESENT + any_of: + - range: MetadataSource + elseconditions: + slot_conditions: + abstract_provider: + none_of: + - range: MetadataSource + attributes: + title: + in_subset: [mvp, deterministic] + range: string + required: true + authors: + in_subset: [mvp, deterministic] + range: string + multivalued: true + description: Author names in published order. + journal: + in_subset: [mvp, deterministic] + range: string + publication_year: + in_subset: [mvp, deterministic] + range: integer + minimum_value: 1600 + description: Year in which the study was first published or made publicly available. + volume: + in_subset: [mvp, deterministic] + range: string + issue: + in_subset: [mvp, deterministic] + range: string + pages: + in_subset: [mvp, deterministic] + range: string + abstract: + in_subset: [mvp, deterministic] + range: string + pattern: '\S' + description: Absent (null) when no source gave one; never an empty string. + abstract_provider: + in_subset: [mvp, deterministic] + range: MetadataSource + description: >- + The source whose abstract `abstract` is. It may differ from `provider`. The + abstract is the publisher's abstract of the version `license` describes, so its + licence is `license`. + language: + in_subset: [mvp, deterministic] + range: string + multivalued: true + description: >- + Languages of the article, as the E-utilities API reports them -- the three-letter + codes PubMed assigns, `eng` and the rest, and not read off the paper: the text a + reader is given is in one language whatever the article is indexed as. + + At least six of the benchmark's sixteen meta-analyses state "English language" as + an inclusion criterion. On the 1,804 records measured it excludes exactly one + paper, so it buys precision rather than recall; the criterion is stated, so the + field states it. + publication_types: + in_subset: [mvp, deterministic] + any_of: + - range: StudyType + - range: string + multivalued: true + description: >- + PubMed publication types for this record, verbatim as the E-utilities API returns + them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because + PubMed assigns several: every article is `Journal Article` and the design of the + study motivates other labels. `StudyType` lists the known types only; any other + type PubMed returns is kept verbatim. + keywords: + in_subset: [mvp, deterministic] + range: string + multivalued: true + license: + in_subset: [mvp, deterministic] + range: License + inlined: true + required: true + description: >- + The paper's licence: the version of record's, or the latest preprint's while the + header's `publication_status` is `preprint`. Not the licence of a copy of another + version, which that copy carries. + oa_status: + in_subset: [mvp, deterministic] + range: OaStatus + corrections: + in_subset: [mvp, deterministic] + range: Correction + multivalued: true + inlined_as_list: true + description: >- + Notices attached to the paper -- errata, retractions, expressions of concern -- as + the indexer lists them. A retracted paper is still parsed and stored, marked by + its `retraction` notice, and left out of studysets by default; a filter can ask for + it explicitly. + provider: + in_subset: [mvp, deterministic] + range: MetadataSource + description: The metadata service the facts came from. + + ArticleIdentifiers: + description: >- + Every identifier known for the paper when the artifact was written. Carried whole, so + no consumer has to choose between them or look one up. `doi`, `pmid` and `pmcid` name + the version of record; a preprint's identifiers are kept beside them in the + `preprint_` fields, so a paper and its preprint are one article. + attributes: + pmid: + in_subset: [mvp, deterministic] + range: string + description: >- + PubMed's record of the version of record. Never PubMed's record of a preprint, + which is `preprint_pmid`. + pmcid: + in_subset: [mvp, deterministic] + range: string + description: >- + PubMed Central's copy of the version of record. Never PMC's copy of a preprint, + which is `preprint_pmcid`. + doi: + in_subset: [mvp, deterministic] + range: string + description: >- + The DOI of the version of record. Never a preprint server's DOI, which is in + `preprint_dois`. + preprint_dois: + in_subset: [mvp, deterministic] + range: PreprintDoi + multivalued: true + inlined_as_list: true + description: >- + Every DOI a preprint server registered for the paper. A server that registers each + version separately, such as Research Square's `/v2`, gives one entry per version, + each kept as registered. + preprint_pmid: + in_subset: [mvp, deterministic] + range: string + description: PubMed's record of the preprint, which PubMed types `Preprint`. + preprint_pmcid: + in_subset: [mvp, deterministic] + range: string + description: PubMed Central's copy of the preprint. + publication_status: + in_subset: [mvp, deterministic] + range: PublicationStatus + required: true + description: >- + `preprint` when a `preprint_` field is set and `doi`, `pmid` and `pmcid` are all + empty, `published` otherwise. The producer states it so that no consumer re-derives + it. + + PreprintDoi: + description: A DOI a preprint server registered, with the server that registered it. + attributes: + doi: + in_subset: [mvp, deterministic] + range: string + required: true + server: + in_subset: [mvp, deterministic] + range: PreprintServer + required: true + description: >- + Named by the producer from the DOI, so that no consumer needs a table of preprint + servers' DOI patterns. + + Correction: + description: One notice attached to a paper. + attributes: + kind: + in_subset: [mvp, deterministic] + any_of: + - range: CorrectionKind + - range: string + required: true + pmid: + in_subset: [mvp, deterministic] + range: string + doi: + in_subset: [mvp, deterministic] + range: string + +enums: + + PublicationStatus: + in_subset: [mvp, deterministic] + description: Whether a paper's version of record is known, or only its preprint. + permissible_values: + published: + in_subset: [mvp, deterministic] + description: >- + A DOI, PubMed record or PMC copy of the version of record is known. The paper may + have been a preprint too. + preprint: + in_subset: [mvp, deterministic] + description: Only a preprint's identifiers are known. + + PreprintServer: + in_subset: [mvp, deterministic] + description: >- + The preprint server that registered a DOI. A server hosted on the Open Science + Framework is named by its provider, which has a DOI prefix of its own. A server is + added here when a producer learns to recognise its DOIs. + permissible_values: + biorxiv: + in_subset: [mvp, deterministic] + description: bioRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). + medrxiv: + in_subset: [mvp, deterministic] + description: medRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). + arxiv: + in_subset: [mvp, deterministic] + description: arXiv, DOI prefix 10.48550, the DOI registered for an arXiv id. + researchsquare: + in_subset: [mvp, deterministic] + description: Research Square, DOI prefix 10.21203. + osf_preprints: + in_subset: [mvp, deterministic] + description: OSF Preprints, DOI prefix 10.31219. + arabixiv: + in_subset: [mvp, deterministic] + description: Arabixiv (OSF), DOI prefix 10.31221. + metaarxiv: + in_subset: [mvp, deterministic] + description: MetaArXiv (OSF), DOI prefix 10.31222. + engrxiv: + in_subset: [mvp, deterministic] + description: engrXiv (OSF), DOI prefix 10.31224. + focus_archive: + in_subset: [mvp, deterministic] + description: FocUS Archive (OSF), DOI prefix 10.31225. + frenxiv: + in_subset: [mvp, deterministic] + description: Frenxiv (OSF), DOI prefix 10.31226. + ina_rxiv: + in_subset: [mvp, deterministic] + description: INA-Rxiv (OSF), DOI prefix 10.31227. + lawarxiv: + in_subset: [mvp, deterministic] + description: LawArXiv (OSF), DOI prefix 10.31228. + lis_scholarship_archive: + in_subset: [mvp, deterministic] + description: LIS Scholarship Archive (OSF), DOI prefix 10.31229. + marxiv: + in_subset: [mvp, deterministic] + description: MarXiv (OSF), DOI prefix 10.31230. + mindrxiv: + in_subset: [mvp, deterministic] + description: MindRxiv (OSF), DOI prefix 10.31231. + nutrixiv: + in_subset: [mvp, deterministic] + description: NutriXiv (OSF), DOI prefix 10.31232. + paleorxiv: + in_subset: [mvp, deterministic] + description: PaleorXiv (OSF), DOI prefix 10.31233. + psyarxiv: + in_subset: [mvp, deterministic] + description: PsyArXiv (OSF), DOI prefix 10.31234. + socarxiv: + in_subset: [mvp, deterministic] + description: SocArXiv (OSF), DOI prefix 10.31235. + sportrxiv: + in_subset: [mvp, deterministic] + description: SportRxiv (OSF), DOI prefix 10.31236. + africarxiv: + in_subset: [mvp, deterministic] + description: AfricArXiv (OSF), DOI prefix 10.31730. + mediarxiv: + in_subset: [mvp, deterministic] + description: MediArXiv (OSF), DOI prefix 10.33767. + bodoarxiv: + in_subset: [mvp, deterministic] + description: BodoArXiv (OSF), DOI prefix 10.34055. + edarxiv: + in_subset: [mvp, deterministic] + description: EdArXiv (OSF), DOI prefix 10.35542. + biohackrxiv: + in_subset: [mvp, deterministic] + description: BioHackrXiv (OSF), DOI prefix 10.37044. + ssrn: + in_subset: [mvp, deterministic] + description: SSRN, DOI prefix 10.2139. + preprints_org: + in_subset: [mvp, deterministic] + description: Preprints.org, DOI prefix 10.20944. + authorea: + in_subset: [mvp, deterministic] + description: Authorea, DOI prefix 10.22541. + techrxiv: + in_subset: [mvp, deterministic] + description: TechRxiv, DOI prefix 10.36227. + chemrxiv: + in_subset: [mvp, deterministic] + description: ChemRxiv, DOI prefix 10.26434. + jmir_preprints: + in_subset: [mvp, deterministic] + description: JMIR Preprints, DOI prefix 10.2196. + + CorrectionKind: + in_subset: [mvp, deterministic] + description: A notice attached to a paper by its publisher or indexer. + permissible_values: + erratum: + in_subset: [mvp, deterministic] + retraction: + in_subset: [mvp, deterministic] + expression_of_concern: + in_subset: [mvp, deterministic] + comment: + in_subset: [mvp, deterministic] + update: + in_subset: [mvp, deterministic] + + StudyType: + in_subset: [mvp, deterministic] + description: >- + PubMed publication types, spelled as the E-utilities API returns them so that a value + needs no translation. Only the design-relevant types are listed; PubMed also returns + funding and indexing types, and the vocabulary is open, so any unlisted type passes + through verbatim. + permissible_values: + "Journal Article": + in_subset: [mvp, deterministic] + description: Assigned to every article; carries no design information on its own. + "Meta-Analysis": + in_subset: [mvp, deterministic] + description: >- + A quantitative synthesis of published results. + "Systematic Review": + in_subset: [mvp, deterministic] + description: A structured synthesis with a search protocol but no pooled effect estimate. + "Review": + in_subset: [mvp, deterministic] + description: A review without a systematic search protocol. + "Randomized Controlled Trial": + in_subset: [mvp, deterministic] + "Clinical Trial": + in_subset: [mvp, deterministic] + "Observational Study": + in_subset: [mvp, deterministic] + "Comparative Study": + in_subset: [mvp, deterministic] + "Multicenter Study": + in_subset: [mvp, deterministic] + "Validation Study": + in_subset: [mvp, deterministic] + "Case Reports": + in_subset: [mvp, deterministic] + description: PubMed's spelling is plural. + "Editorial": + in_subset: [mvp, deterministic] + "Letter": + in_subset: [mvp, deterministic] + "Comment": + in_subset: [mvp, deterministic] + "Preprint": + in_subset: [mvp, deterministic] + "Retracted Publication": + in_subset: [mvp, deterministic] diff --git a/src/study_schema/linkml/neuroimaging-study-storage/license.yaml b/src/study_schema/linkml/neuroimaging-study-storage/license.yaml new file mode 100644 index 0000000..86c6a94 --- /dev/null +++ b/src/study_schema/linkml/neuroimaging-study-storage/license.yaml @@ -0,0 +1,201 @@ +id: https://neurostuff.org/schemas/neuroimaging-study-storage/license +name: neuroimaging-study-storage-license +title: Neuroimaging Study Storage License Module +description: >- + The licence of a paper or of one copy of it, and who stated it. The paper-parse schema + imports this module, so the parsed paper and the storage Study hold the same `License`. + + Whether a licence lets a text or an abstract be redistributed is not stored. It is + `study_schema.licensing`'s `text_may_be_redistributed` and + `abstract_may_be_redistributed`, which read the `redistributable` annotation of + `LicenseId` here and the `redistribution` annotation of the paper-parse `CopyAccess`, so a + change of policy is a release of this package rather than a re-run. +prefixes: + linkml: https://w3id.org/linkml/ + neurostudy: https://neurostuff.org/schemas/neuroimaging-study-storage/ +imports: + - linkml:types + - subsets +default_prefix: neurostudy +default_range: string + +classes: + + License: + description: >- + A licence as one source stated it, or all rights reserved assumed because no source + stated one. + rules: + # `any_of` / `none_of` over the range, rather than value_presence alone, so that a + # field written as null counts as absent. + - description: >- + A stated licence says who stated it and how. An assumed one is all rights reserved, + and nobody stated it. + open_world: true + preconditions: + slot_conditions: + determination: + equals_string: stated + postconditions: + slot_conditions: + stated_by: + value_presence: PRESENT + any_of: + - range: MetadataSource + stated_as: + value_presence: PRESENT + any_of: + - range: string + elseconditions: + slot_conditions: + license_id: + equals_string: all-rights-reserved + stated_by: + none_of: + - range: MetadataSource + stated_as: + none_of: + - range: string + attributes: + license_id: + in_subset: [mvp, deterministic] + range: LicenseId + required: true + description: >- + Which licence, as OpenAlex's licence slugs name them (`cc-by`, `cc-by-nc`, ...). + Not an SPDX License Identifier: sources rarely state the licence's version, + and SPDX identifiers require one. + determination: + in_subset: [mvp, deterministic] + range: LicenseDetermination + required: true + stated_by: + in_subset: [mvp, deterministic] + range: MetadataSource + description: The source whose statement this is. + stated_as: + in_subset: [mvp, deterministic] + range: string + pattern: '\S' + description: >- + The statement exactly as the source gave it: a licence URL, a licence name, or a + copyright line. What `other` holds, and what attribution quotes. Never empty: a + source that gave no statement did not state a licence. + +enums: + + LicenseId: + in_subset: [mvp, deterministic] + description: >- + The terms a licence grants. A value annotated `redistributable` lets the text it + covers be passed on verbatim with attribution: every Creative Commons licence, + including ND, because converting a format to text is a change CC 4.0 2(a)(4) allows, + and NC, because neurostore's use is non-commercial. That reading of ND and NC is this + project's policy, not legal advice, and no lawyer has confirmed it. + permissible_values: + cc-by: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + cc-by-sa: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + cc-by-nc: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + cc-by-nc-sa: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + cc-by-nd: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + cc-by-nc-nd: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + cc0: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + public-domain: + in_subset: [mvp, deterministic] + annotations: + redistributable: true + all-rights-reserved: + in_subset: [mvp, deterministic] + description: >- + A copyright statement with no licence, or terms that allow text mining only, such + as PubMed Central's `TDM` for an author manuscript. + other: + in_subset: [mvp, deterministic] + description: >- + A stated licence outside this list, such as a publisher's own user licence or + OpenAlex's `publisher-specific-oa`; `stated_as` holds it. Not redistributable, on + purpose: Elsevier's open-access user licence is `other`, because its terms are not + a CC licence's. + + LicenseDetermination: + in_subset: [mvp, deterministic] + description: Whether a source stated the licence. + permissible_values: + stated: + in_subset: [mvp, deterministic] + description: A source stated it; `stated_by` says which. + assumed: + in_subset: [mvp, deterministic] + description: >- + Every source that could state a licence answered and none did, so all rights are + reserved. + + MetadataSource: + in_subset: [mvp, deterministic] + description: Who stated a bibliographic fact. + permissible_values: + jats: + in_subset: [mvp, deterministic] + description: A held copy's JATS XML, such as its ``. + elsevier: + in_subset: [mvp, deterministic] + description: A held copy's Elsevier full-text XML, its `coredata`. + pmc_cloud: + in_subset: [mvp, deterministic] + description: PubMed Central's cloud record of a held copy. + pubmed: + in_subset: [mvp, deterministic] + crossref: + in_subset: [mvp, deterministic] + openalex: + in_subset: [mvp, deterministic] + semantic_scholar: + in_subset: [mvp, deterministic] + europepmc: + in_subset: [mvp, deterministic] + + OaStatus: + in_subset: [mvp, deterministic] + description: >- + How a paper is open to read, in OpenAlex's terms + (https://help.openalex.org/hc/en-us/articles/24347035046295-Open-Access-OA). + permissible_values: + gold: + in_subset: [mvp, deterministic] + description: Published openly licensed in an open-access journal. + diamond: + in_subset: [mvp, deterministic] + description: Gold, in a journal that charges authors no fee. + hybrid: + in_subset: [mvp, deterministic] + description: Published openly licensed in a subscription journal. + bronze: + in_subset: [mvp, deterministic] + description: Free to read at the publisher, without an open licence. + green: + in_subset: [mvp, deterministic] + description: Closed at the publisher, with a free copy in a repository. + closed: + in_subset: [mvp, deterministic] + description: No free copy is known. diff --git a/src/study_schema/linkml/neuroimaging-study-storage/study.yaml b/src/study_schema/linkml/neuroimaging-study-storage/study.yaml index 51d0f60..5deb8db 100644 --- a/src/study_schema/linkml/neuroimaging-study-storage/study.yaml +++ b/src/study_schema/linkml/neuroimaging-study-storage/study.yaml @@ -14,6 +14,7 @@ imports: - region - acquisition - analysis + - bibliography default_prefix: neurostudy default_range: string @@ -34,11 +35,21 @@ classes: required: true description: Stable identifier for the study record, such as a DOI-based or repository-local identifier. - title: + bibliography: in_subset: [mvp, deterministic] - range: string + range: Bibliography + inlined: true required: true - description: Full title of the study or publication. + description: >- + The paper's bibliographic facts, the parsed paper's `Bibliography` unchanged: + title, authors, journal, year, abstract, PubMed types, the paper's licence. + identifiers: + in_subset: [mvp, deterministic] + range: ArticleIdentifiers + inlined: true + description: >- + Every identifier known for the paper, the parsed paper's header identifiers + unchanged. description: in_subset: [mvp, model_extracted] range: string @@ -72,60 +83,6 @@ classes: The study's design outside its tasks: how participants were assigned, the arms they were assigned to, and the occasions data were collected on. The narrative design description is this record's own `description`. - authors: - in_subset: [mvp, deterministic] - range: string - multivalued: true - description: Authors of the study or publication, in published order. - journal: - in_subset: [mvp, deterministic] - range: string - description: Journal or other publication venue. - doi: - in_subset: [mvp, deterministic] - range: string - description: >- - Digital Object Identifier of the version of record, without requiring a particular - URL form. Never a preprint server's DOI. - publication_status: - in_subset: [mvp, deterministic] - range: PublicationStatus - description: >- - Whether the version of record is known or only the preprint, copied from the - parsed paper's header identifiers. `study_type` is independent of it: it keeps - PubMed's publication types verbatim, `Preprint` included. - publication_year: - in_subset: [mvp, deterministic] - range: integer - minimum_value: 1600 - description: Year in which the study was first published or made publicly available. - language: - in_subset: [mvp, deterministic] - range: string - multivalued: true - description: >- - Languages of the article, as the E-utilities API reports them -- the three-letter - codes PubMed assigns, `eng` and the rest. Filled from the same `esummary` call - that fills `study_type`, and not read off the paper: the text a reader is given - is in one language whatever the article is indexed as. - - At least six of the benchmark's sixteen meta-analyses state "English language" as - an inclusion criterion, and it was unexpressible. On the 1,804 records measured it - excludes exactly one paper, so it buys precision rather than recall; the criterion - is stated, so the field states it. - - study_type: - in_subset: [mvp, deterministic] - any_of: - - range: StudyType - - range: string - multivalued: true - required: true - description: >- - PubMed publication types for this record, verbatim as the E-utilities API returns - them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because - PubMed assigns several: every article is `Journal Article` and the design of the study - motivates other labels. groups: in_subset: [mvp, model_extracted] range: Group @@ -306,50 +263,3 @@ enums: description: >- No analysis reports a significant effect and at least one outcome was not reported, or `result_reporting` was not reported. - - StudyType: - in_subset: [mvp, deterministic] - description: >- - PubMed publication types, spelled as the E-utilities API returns them so that a value - needs no translation. Only the design-relevant types are listed; PubMed also returns - funding and indexing types, and the vocabulary is open, so any unlisted type passes - through verbatim. - permissible_values: - "Journal Article": - in_subset: [mvp, deterministic] - description: Assigned to every article; carries no design information on its own. - "Meta-Analysis": - in_subset: [mvp, deterministic] - description: >- - A quantitative synthesis of published results. - "Systematic Review": - in_subset: [mvp, deterministic] - description: A structured synthesis with a search protocol but no pooled effect estimate. - "Review": - in_subset: [mvp, deterministic] - description: A review without a systematic search protocol. - "Randomized Controlled Trial": - in_subset: [mvp, deterministic] - "Clinical Trial": - in_subset: [mvp, deterministic] - "Observational Study": - in_subset: [mvp, deterministic] - "Comparative Study": - in_subset: [mvp, deterministic] - "Multicenter Study": - in_subset: [mvp, deterministic] - "Validation Study": - in_subset: [mvp, deterministic] - "Case Reports": - in_subset: [mvp, deterministic] - description: PubMed's spelling is plural. - "Editorial": - in_subset: [mvp, deterministic] - "Letter": - in_subset: [mvp, deterministic] - "Comment": - in_subset: [mvp, deterministic] - "Preprint": - in_subset: [mvp, deterministic] - "Retracted Publication": - in_subset: [mvp, deterministic] diff --git a/src/study_schema/models/extraction.py b/src/study_schema/models/extraction.py index fa7214d..5cb31f4 100644 --- a/src/study_schema/models/extraction.py +++ b/src/study_schema/models/extraction.py @@ -32,7 +32,7 @@ metamodel_version = "1.12.0" -version = "0.5.0" +version = "0.6.0" class ConfiguredBaseModel(BaseModel): @@ -2601,8 +2601,6 @@ class Study(ConfiguredBaseModel): tables: Optional[list[Table]] = Field(default=None, description="""Publication tables from which results or analysis metadata were extracted.""") coordinate_sets: Optional[list[CoordinateSet]] = Field(default=None, description="""Every group of coordinates the paper reports, in its tables or its text, with what each is for.""") external_datasets: Optional[list[ExternalDataset]] = Field(default=None, description="""Named external datasets used, cited, or made available by the study.""") - language: Optional[list[str]] = Field(default=None, description="""Languages of this article, copied from the parsed paper's `Bibliography.language` alongside `study_type`, not read off the paper: three-letter codes, `eng` and the rest.""") - study_type: Optional[list[str]] = Field(default=None, description="""PubMed publication types for this record, verbatim, copied from the parsed paper's `Bibliography.publication_types`, not read off the paper: it is what the indexers said about the article, which is what a publication-type exclusion asks about.""") @field_validator('stable_id') def pattern_stable_id(cls, v): diff --git a/src/study_schema/models/paper_parse.py b/src/study_schema/models/paper_parse.py index b641f75..94f88a6 100644 --- a/src/study_schema/models/paper_parse.py +++ b/src/study_schema/models/paper_parse.py @@ -32,7 +32,7 @@ metamodel_version = "1.12.0" -version = "0.7.0" +version = "0.8.0" class ConfiguredBaseModel(BaseModel): @@ -118,133 +118,65 @@ class DocumentSource(str, Enum): """ -class PreprintServer(str, Enum): - """ - The preprint server that registered a DOI. A server hosted on the Open Science Framework is named by its provider, which has a DOI prefix of its own. A server is added here when a producer learns to recognise its DOIs. - """ - biorxiv = "biorxiv" - """ - bioRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). - """ - medrxiv = "medrxiv" - """ - medRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). - """ - arxiv = "arxiv" - """ - arXiv, DOI prefix 10.48550, the DOI registered for an arXiv id. - """ - researchsquare = "researchsquare" - """ - Research Square, DOI prefix 10.21203. - """ - osf_preprints = "osf_preprints" - """ - OSF Preprints, DOI prefix 10.31219. - """ - arabixiv = "arabixiv" - """ - Arabixiv (OSF), DOI prefix 10.31221. - """ - metaarxiv = "metaarxiv" - """ - MetaArXiv (OSF), DOI prefix 10.31222. - """ - engrxiv = "engrxiv" - """ - engrXiv (OSF), DOI prefix 10.31224. - """ - focus_archive = "focus_archive" - """ - FocUS Archive (OSF), DOI prefix 10.31225. - """ - frenxiv = "frenxiv" - """ - Frenxiv (OSF), DOI prefix 10.31226. - """ - ina_rxiv = "ina_rxiv" - """ - INA-Rxiv (OSF), DOI prefix 10.31227. - """ - lawarxiv = "lawarxiv" - """ - LawArXiv (OSF), DOI prefix 10.31228. - """ - lis_scholarship_archive = "lis_scholarship_archive" - """ - LIS Scholarship Archive (OSF), DOI prefix 10.31229. - """ - marxiv = "marxiv" +class CopyRoute(str, Enum): """ - MarXiv (OSF), DOI prefix 10.31230. - """ - mindrxiv = "mindrxiv" - """ - MindRxiv (OSF), DOI prefix 10.31231. - """ - nutrixiv = "nutrixiv" - """ - NutriXiv (OSF), DOI prefix 10.31232. - """ - paleorxiv = "paleorxiv" - """ - PaleorXiv (OSF), DOI prefix 10.31233. - """ - psyarxiv = "psyarxiv" + Where a copy of a paper is hosted. """ - PsyArXiv (OSF), DOI prefix 10.31234. + pmc = "pmc" """ - socarxiv = "socarxiv" + PubMed Central or Europe PMC. """ - SocArXiv (OSF), DOI prefix 10.31235. + publisher = "publisher" """ - sportrxiv = "sportrxiv" - """ - SportRxiv (OSF), DOI prefix 10.31236. + The publisher's own site or full-text API. """ - africarxiv = "africarxiv" + repository = "repository" """ - AfricArXiv (OSF), DOI prefix 10.31730. + An institutional or subject repository other than PMC. """ - mediarxiv = "mediarxiv" + preprint_server = "preprint_server" """ - MediArXiv (OSF), DOI prefix 10.33767. + A preprint server; the copy's `preprint_server` names it. """ - bodoarxiv = "bodoarxiv" + unrecorded = "unrecorded" """ - BodoArXiv (OSF), DOI prefix 10.34055. + Where the copy came from was not recorded, such as a PDF supplied by hand. """ - edarxiv = "edarxiv" + + +class CopyVersion(str, Enum): """ - EdArXiv (OSF), DOI prefix 10.35542. + Which version of the paper a copy is, in the NISO Journal Article Versions terms. OpenAlex's `publishedVersion`, `acceptedVersion` and `submittedVersion` are `vor`, `am` and `submitted`. """ - biohackrxiv = "biohackrxiv" + vor = "vor" """ - BioHackrXiv (OSF), DOI prefix 10.37044. + The version of record, as the journal published it. """ - ssrn = "ssrn" + am = "am" """ - SSRN, DOI prefix 10.2139. + The accepted manuscript, after peer review and before the publisher's typesetting. """ - preprints_org = "preprints_org" + submitted = "submitted" """ - Preprints.org, DOI prefix 10.20944. + The version submitted to a journal or posted as a preprint, before peer review. """ - authorea = "authorea" + + +class CopyAccess(str, Enum): """ - Authorea, DOI prefix 10.22541. + How we obtained a copy we hold. The `redistribution` annotation says whether the access itself allows passing the copy on: `by_license` leaves that to the copy's licence, `never` forbids it whatever the licence says. """ - techrxiv = "techrxiv" + public = "public" """ - TechRxiv, DOI prefix 10.36227. + Fetched as any reader could fetch it, without our credentials. """ - chemrxiv = "chemrxiv" + tdm_grant = "tdm_grant" """ - ChemRxiv, DOI prefix 10.26434. + Obtained under terms that permit text mining only: a publisher's full-text API with our key, such as Elsevier's or Wiley's, or a PubMed Central author manuscript whose record says `TDM`. """ - jmir_preprints = "jmir_preprints" + unrecorded = "unrecorded" """ - JMIR Preprints, DOI prefix 10.2196. + How the copy was obtained was not recorded. """ @@ -488,17 +420,6 @@ class TableReadingKind(str, Enum): """ -class CorrectionKind(str, Enum): - """ - A notice attached to a paper by its publisher or indexer. - """ - erratum = "erratum" - retraction = "retraction" - expression_of_concern = "expression_of_concern" - comment = "comment" - update = "update" - - class ThresholdLevel(str, Enum): """ The level a threshold applies at. @@ -610,6 +531,287 @@ class CitationAlignment(str, Enum): """ +class LicenseId(str, Enum): + """ + The terms a licence grants. A value annotated `redistributable` lets the text it covers be passed on verbatim with attribution: every Creative Commons licence, including ND, because converting a format to text is a change CC 4.0 2(a)(4) allows, and NC, because neurostore's use is non-commercial. That reading of ND and NC is this project's policy, not legal advice, and no lawyer has confirmed it. + """ + cc_by = "cc-by" + cc_by_sa = "cc-by-sa" + cc_by_nc = "cc-by-nc" + cc_by_nc_sa = "cc-by-nc-sa" + cc_by_nd = "cc-by-nd" + cc_by_nc_nd = "cc-by-nc-nd" + cc0 = "cc0" + public_domain = "public-domain" + all_rights_reserved = "all-rights-reserved" + """ + A copyright statement with no licence, or terms that allow text mining only, such as PubMed Central's `TDM` for an author manuscript. + """ + other = "other" + """ + A stated licence outside this list, such as a publisher's own user licence or OpenAlex's `publisher-specific-oa`; `stated_as` holds it. Not redistributable, on purpose: Elsevier's open-access user licence is `other`, because its terms are not a CC licence's. + """ + + +class LicenseDetermination(str, Enum): + """ + Whether a source stated the licence. + """ + stated = "stated" + """ + A source stated it; `stated_by` says which. + """ + assumed = "assumed" + """ + Every source that could state a licence answered and none did, so all rights are reserved. + """ + + +class MetadataSource(str, Enum): + """ + Who stated a bibliographic fact. + """ + jats = "jats" + """ + A held copy's JATS XML, such as its ``. + """ + elsevier = "elsevier" + """ + A held copy's Elsevier full-text XML, its `coredata`. + """ + pmc_cloud = "pmc_cloud" + """ + PubMed Central's cloud record of a held copy. + """ + pubmed = "pubmed" + crossref = "crossref" + openalex = "openalex" + semantic_scholar = "semantic_scholar" + europepmc = "europepmc" + + +class OaStatus(str, Enum): + """ + How a paper is open to read, in OpenAlex's terms (https://help.openalex.org/hc/en-us/articles/24347035046295-Open-Access-OA). + """ + gold = "gold" + """ + Published openly licensed in an open-access journal. + """ + diamond = "diamond" + """ + Gold, in a journal that charges authors no fee. + """ + hybrid = "hybrid" + """ + Published openly licensed in a subscription journal. + """ + bronze = "bronze" + """ + Free to read at the publisher, without an open licence. + """ + green = "green" + """ + Closed at the publisher, with a free copy in a repository. + """ + closed = "closed" + """ + No free copy is known. + """ + + +class PublicationStatus(str, Enum): + """ + Whether a paper's version of record is known, or only its preprint. + """ + published = "published" + """ + A DOI, PubMed record or PMC copy of the version of record is known. The paper may have been a preprint too. + """ + preprint = "preprint" + """ + Only a preprint's identifiers are known. + """ + + +class PreprintServer(str, Enum): + """ + The preprint server that registered a DOI. A server hosted on the Open Science Framework is named by its provider, which has a DOI prefix of its own. A server is added here when a producer learns to recognise its DOIs. + """ + biorxiv = "biorxiv" + """ + bioRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). + """ + medrxiv = "medrxiv" + """ + medRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). + """ + arxiv = "arxiv" + """ + arXiv, DOI prefix 10.48550, the DOI registered for an arXiv id. + """ + researchsquare = "researchsquare" + """ + Research Square, DOI prefix 10.21203. + """ + osf_preprints = "osf_preprints" + """ + OSF Preprints, DOI prefix 10.31219. + """ + arabixiv = "arabixiv" + """ + Arabixiv (OSF), DOI prefix 10.31221. + """ + metaarxiv = "metaarxiv" + """ + MetaArXiv (OSF), DOI prefix 10.31222. + """ + engrxiv = "engrxiv" + """ + engrXiv (OSF), DOI prefix 10.31224. + """ + focus_archive = "focus_archive" + """ + FocUS Archive (OSF), DOI prefix 10.31225. + """ + frenxiv = "frenxiv" + """ + Frenxiv (OSF), DOI prefix 10.31226. + """ + ina_rxiv = "ina_rxiv" + """ + INA-Rxiv (OSF), DOI prefix 10.31227. + """ + lawarxiv = "lawarxiv" + """ + LawArXiv (OSF), DOI prefix 10.31228. + """ + lis_scholarship_archive = "lis_scholarship_archive" + """ + LIS Scholarship Archive (OSF), DOI prefix 10.31229. + """ + marxiv = "marxiv" + """ + MarXiv (OSF), DOI prefix 10.31230. + """ + mindrxiv = "mindrxiv" + """ + MindRxiv (OSF), DOI prefix 10.31231. + """ + nutrixiv = "nutrixiv" + """ + NutriXiv (OSF), DOI prefix 10.31232. + """ + paleorxiv = "paleorxiv" + """ + PaleorXiv (OSF), DOI prefix 10.31233. + """ + psyarxiv = "psyarxiv" + """ + PsyArXiv (OSF), DOI prefix 10.31234. + """ + socarxiv = "socarxiv" + """ + SocArXiv (OSF), DOI prefix 10.31235. + """ + sportrxiv = "sportrxiv" + """ + SportRxiv (OSF), DOI prefix 10.31236. + """ + africarxiv = "africarxiv" + """ + AfricArXiv (OSF), DOI prefix 10.31730. + """ + mediarxiv = "mediarxiv" + """ + MediArXiv (OSF), DOI prefix 10.33767. + """ + bodoarxiv = "bodoarxiv" + """ + BodoArXiv (OSF), DOI prefix 10.34055. + """ + edarxiv = "edarxiv" + """ + EdArXiv (OSF), DOI prefix 10.35542. + """ + biohackrxiv = "biohackrxiv" + """ + BioHackrXiv (OSF), DOI prefix 10.37044. + """ + ssrn = "ssrn" + """ + SSRN, DOI prefix 10.2139. + """ + preprints_org = "preprints_org" + """ + Preprints.org, DOI prefix 10.20944. + """ + authorea = "authorea" + """ + Authorea, DOI prefix 10.22541. + """ + techrxiv = "techrxiv" + """ + TechRxiv, DOI prefix 10.36227. + """ + chemrxiv = "chemrxiv" + """ + ChemRxiv, DOI prefix 10.26434. + """ + jmir_preprints = "jmir_preprints" + """ + JMIR Preprints, DOI prefix 10.2196. + """ + + +class CorrectionKind(str, Enum): + """ + A notice attached to a paper by its publisher or indexer. + """ + erratum = "erratum" + retraction = "retraction" + expression_of_concern = "expression_of_concern" + comment = "comment" + update = "update" + + +class StudyType(str, Enum): + """ + PubMed publication types, spelled as the E-utilities API returns them so that a value needs no translation. Only the design-relevant types are listed; PubMed also returns funding and indexing types, and the vocabulary is open, so any unlisted type passes through verbatim. + """ + Journal_Article = "Journal Article" + """ + Assigned to every article; carries no design information on its own. + """ + Meta_Analysis = "Meta-Analysis" + """ + A quantitative synthesis of published results. + """ + Systematic_Review = "Systematic Review" + """ + A structured synthesis with a search protocol but no pooled effect estimate. + """ + Review = "Review" + """ + A review without a systematic search protocol. + """ + Randomized_Controlled_Trial = "Randomized Controlled Trial" + Clinical_Trial = "Clinical Trial" + Observational_Study = "Observational Study" + Comparative_Study = "Comparative Study" + Multicenter_Study = "Multicenter Study" + Validation_Study = "Validation Study" + Case_Reports = "Case Reports" + """ + PubMed's spelling is plural. + """ + Editorial = "Editorial" + Letter = "Letter" + Comment = "Comment" + Preprint = "Preprint" + Retracted_Publication = "Retracted Publication" + + class AnalysisType(str, Enum): """ The method that produced the statistical map, independent of what was tested about it. A between-group comparison of connectivity, decoding accuracy, or component loadings keeps the method here and records the comparison in Effect. Bound only by ConjunctionComponent.analysis_type; an Analysis states its method by which AnalysisDetails subclass it fills. Open vocabulary: write the source's own wording in the field when nothing here fits. @@ -1413,20 +1615,6 @@ class AnchorKind(str, Enum): """ -class PublicationStatus(str, Enum): - """ - Whether a paper's version of record is known, or only its preprint. - """ - published = "published" - """ - A DOI, PubMed record or PMC copy of the version of record is known. The paper may have been a preprint too. - """ - preprint = "preprint" - """ - Only a preprint's identifiers are known. - """ - - class ArtifactHeader(ConfiguredBaseModel): """ @@ -1439,28 +1627,7 @@ class ArtifactHeader(ConfiguredBaseModel): producer: Producer = Field(default=...) inputs: Optional[list[InputRef]] = Field(default=None, description="""Every artifact this one was made from, by fingerprint.""") created_at: Optional[datetime ] = Field(default=None) - - -class ArticleIdentifiers(ConfiguredBaseModel): - """ - Every identifier known for the paper when the artifact was written. Carried whole, so no consumer has to choose between them or look one up. `doi`, `pmid` and `pmcid` name the version of record; a preprint's identifiers are kept beside them in the `preprint_` fields, so a paper and its preprint are one article. - """ - pmid: Optional[str] = Field(default=None, description="""PubMed's record of the version of record. Never PubMed's record of a preprint, which is `preprint_pmid`.""") - pmcid: Optional[str] = Field(default=None, description="""PubMed Central's copy of the version of record. Never PMC's copy of a preprint, which is `preprint_pmcid`.""") - doi: Optional[str] = Field(default=None, description="""The DOI of the version of record. Never a preprint server's DOI, which is in `preprint_dois`.""") - preprint_dois: Optional[list[PreprintDoi]] = Field(default=None, description="""Every DOI a preprint server registered for the paper. A server that registers each version separately, such as Research Square's `/v2`, gives one entry per version, each kept as registered.""") - preprint_pmid: Optional[str] = Field(default=None, description="""PubMed's record of the preprint, which PubMed types `Preprint`.""") - preprint_pmcid: Optional[str] = Field(default=None, description="""PubMed Central's copy of the preprint.""") - publication_status: PublicationStatus = Field(default=..., description="""`preprint` when a `preprint_` field is set and `doi`, `pmid` and `pmcid` are all empty, `published` otherwise. The producer states it so that no consumer re-derives it.""") - neurostore_base_study_id: Optional[str] = Field(default=None, description="""Set once the paper has been uploaded; neurostore's BaseStudy id.""") - - -class PreprintDoi(ConfiguredBaseModel): - """ - A DOI a preprint server registered, with the server that registered it. - """ - doi: str = Field(default=...) - server: PreprintServer = Field(default=..., description="""Named by the producer from the DOI, so that no consumer needs a table of preprint servers' DOI patterns.""") + neurostore_base_study_id: Optional[str] = Field(default=None, description="""Set once the paper has been uploaded; neurostore's BaseStudy id. An upload pointer, not a bibliographic identifier, so it sits here and not in `identifiers`.""") class Producer(ConfiguredBaseModel): @@ -1492,6 +1659,95 @@ class TextSpan(ConfiguredBaseModel): text: Optional[str] = Field(default=None, description="""The characters in the range, kept so a reader can check the offsets.""") +class License(ConfiguredBaseModel): + """ + A licence as one source stated it, or all rights reserved assumed because no source stated one. + """ + license_id: LicenseId = Field(default=..., description="""Which licence, as OpenAlex's licence slugs name them (`cc-by`, `cc-by-nc`, ...). Not an SPDX License Identifier: sources rarely state the licence's version, and SPDX identifiers require one.""") + determination: LicenseDetermination = Field(default=...) + stated_by: Optional[MetadataSource] = Field(default=None, description="""The source whose statement this is.""") + stated_as: Optional[str] = Field(default=None, description="""The statement exactly as the source gave it: a licence URL, a licence name, or a copyright line. What `other` holds, and what attribution quotes. Never empty: a source that gave no statement did not state a licence.""") + + @field_validator('stated_as') + def pattern_stated_as(cls, v): + pattern=re.compile(r"\S") + if isinstance(v, list): + for element in v: + if isinstance(element, str) and not pattern.search(element): + err_msg = f"Invalid stated_as format: {element}" + raise ValueError(err_msg) + elif isinstance(v, str) and not pattern.search(v): + err_msg = f"Invalid stated_as format: {v}" + raise ValueError(err_msg) + return v + + +class Bibliography(ConfiguredBaseModel): + """ + The paper's bibliographic facts, as the metadata lookup returned them. Defined once: the parsed paper carries it, and the storage Study holds the same object as `Study.bibliography`, so nothing is copied field by field and nothing is extracted. + """ + title: str = Field(default=...) + authors: Optional[list[str]] = Field(default=None, description="""Author names in published order.""") + journal: Optional[str] = Field(default=None) + publication_year: Optional[int] = Field(default=None, description="""Year in which the study was first published or made publicly available.""", ge=1600) + volume: Optional[str] = Field(default=None) + issue: Optional[str] = Field(default=None) + pages: Optional[str] = Field(default=None) + abstract: Optional[str] = Field(default=None, description="""Absent (null) when no source gave one; never an empty string.""") + abstract_provider: Optional[MetadataSource] = Field(default=None, description="""The source whose abstract `abstract` is. It may differ from `provider`. The abstract is the publisher's abstract of the version `license` describes, so its licence is `license`.""") + language: Optional[list[str]] = Field(default=None, description="""Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as. +At least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.""") + publication_types: Optional[list[Union[StudyType, str]]] = Field(default=None, description="""PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels. `StudyType` lists the known types only; any other type PubMed returns is kept verbatim.""") + keywords: Optional[list[str]] = Field(default=None) + license: License = Field(default=..., description="""The paper's licence: the version of record's, or the latest preprint's while the header's `publication_status` is `preprint`. Not the licence of a copy of another version, which that copy carries.""") + oa_status: Optional[OaStatus] = Field(default=None) + corrections: Optional[list[Correction]] = Field(default=None, description="""Notices attached to the paper -- errata, retractions, expressions of concern -- as the indexer lists them. A retracted paper is still parsed and stored, marked by its `retraction` notice, and left out of studysets by default; a filter can ask for it explicitly.""") + provider: Optional[MetadataSource] = Field(default=None, description="""The metadata service the facts came from.""") + + @field_validator('abstract') + def pattern_abstract(cls, v): + pattern=re.compile(r"\S") + if isinstance(v, list): + for element in v: + if isinstance(element, str) and not pattern.search(element): + err_msg = f"Invalid abstract format: {element}" + raise ValueError(err_msg) + elif isinstance(v, str) and not pattern.search(v): + err_msg = f"Invalid abstract format: {v}" + raise ValueError(err_msg) + return v + + +class ArticleIdentifiers(ConfiguredBaseModel): + """ + Every identifier known for the paper when the artifact was written. Carried whole, so no consumer has to choose between them or look one up. `doi`, `pmid` and `pmcid` name the version of record; a preprint's identifiers are kept beside them in the `preprint_` fields, so a paper and its preprint are one article. + """ + pmid: Optional[str] = Field(default=None, description="""PubMed's record of the version of record. Never PubMed's record of a preprint, which is `preprint_pmid`.""") + pmcid: Optional[str] = Field(default=None, description="""PubMed Central's copy of the version of record. Never PMC's copy of a preprint, which is `preprint_pmcid`.""") + doi: Optional[str] = Field(default=None, description="""The DOI of the version of record. Never a preprint server's DOI, which is in `preprint_dois`.""") + preprint_dois: Optional[list[PreprintDoi]] = Field(default=None, description="""Every DOI a preprint server registered for the paper. A server that registers each version separately, such as Research Square's `/v2`, gives one entry per version, each kept as registered.""") + preprint_pmid: Optional[str] = Field(default=None, description="""PubMed's record of the preprint, which PubMed types `Preprint`.""") + preprint_pmcid: Optional[str] = Field(default=None, description="""PubMed Central's copy of the preprint.""") + publication_status: PublicationStatus = Field(default=..., description="""`preprint` when a `preprint_` field is set and `doi`, `pmid` and `pmcid` are all empty, `published` otherwise. The producer states it so that no consumer re-derives it.""") + + +class PreprintDoi(ConfiguredBaseModel): + """ + A DOI a preprint server registered, with the server that registered it. + """ + doi: str = Field(default=...) + server: PreprintServer = Field(default=..., description="""Named by the producer from the DOI, so that no consumer needs a table of preprint servers' DOI patterns.""") + + +class Correction(ConfiguredBaseModel): + """ + One notice attached to a paper. + """ + kind: Union[CorrectionKind, str] = Field(default=...) + pmid: Optional[str] = Field(default=None) + doi: Optional[str] = Field(default=None) + + class ParsedPaper(ConfiguredBaseModel): """ One paper as text, tables and bibliographic facts. The same shape for every download source. @@ -1500,6 +1756,8 @@ class ParsedPaper(ConfiguredBaseModel): header: ArtifactHeader = Field(default=...) source: Union[DocumentSource, str] = Field(default=...) source_reason: Optional[str] = Field(default=None, description="""Why this source's text was chosen over the others downloaded, such as \"most tables\" or \"first in download order\". Every stage reads this one text, so the choice is made once and recorded.""") + text_copy: HeldCopy = Field(default=..., description="""The copy the text was read from. Its licence and access, not the paper's licence, decide whether the text may be passed on.""") + reported_copies: Optional[list[ReportedCopy]] = Field(default=None, description="""Copies of the paper that a source lists and that we do not hold, each with its own licence. Where an open copy is, when `bibliography.oa_status` says one exists.""") text_path: str = Field(default=..., description="""The text file's path, relative to the paper's bundle directory.""") text_sha256: str = Field(default=..., description="""sha256 of the text file's UTF-8 bytes. This artifact's fingerprint.""") text_length: Optional[int] = Field(default=None, ge=0) @@ -1513,25 +1771,39 @@ class ParsedPaper(ConfiguredBaseModel): tables: Optional[list[ParsedTable]] = Field(default=None) -class Bibliography(ConfiguredBaseModel): +class Copy(ConfiguredBaseModel): """ - The paper's bibliographic facts, as the metadata lookup returned them. Downstream they are copied, never extracted: pondie fills the record's Study.title, authors, doi, journal, publication_year, language and study_type from here, and neurostore fills BaseStudy from here. + One copy of a paper, where it is, which version it is, and its licence. """ - title: str = Field(default=...) - authors: Optional[list[str]] = Field(default=None, description="""Author names in published order.""") - journal: Optional[str] = Field(default=None) - publication_year: Optional[int] = Field(default=None) - volume: Optional[str] = Field(default=None) - issue: Optional[str] = Field(default=None) - pages: Optional[str] = Field(default=None) - abstract: Optional[str] = Field(default=None) - language: Optional[list[str]] = Field(default=None) - publication_types: Optional[list[str]] = Field(default=None, description="""PubMed publication types, verbatim. The storage schema's Study.study_type.""") - keywords: Optional[list[str]] = Field(default=None) - license: Optional[str] = Field(default=None) - open_access: Optional[bool] = Field(default=None) - corrections: Optional[list[Correction]] = Field(default=None, description="""Notices attached to the paper -- errata, retractions, expressions of concern -- as the indexer lists them. A retracted paper is still parsed and stored, marked by its `retraction` notice, and left out of studysets by default; a filter can ask for it explicitly.""") - provider: Optional[str] = Field(default=None, description="""The metadata service the facts came from, such as pubmed or europepmc.""") + route: CopyRoute = Field(default=...) + version: Optional[CopyVersion] = Field(default=None, description="""Absent when no source says which version the copy is.""") + license: License = Field(default=..., description="""This copy's own licence. A manuscript or preprint can be licensed differently from the version of record.""") + url: Optional[str] = Field(default=None) + preprint_server: Optional[PreprintServer] = Field(default=None) + + +class HeldCopy(Copy): + """ + A copy we downloaded. + """ + access: CopyAccess = Field(default=...) + route: CopyRoute = Field(default=...) + version: Optional[CopyVersion] = Field(default=None, description="""Absent when no source says which version the copy is.""") + license: License = Field(default=..., description="""This copy's own licence. A manuscript or preprint can be licensed differently from the version of record.""") + url: Optional[str] = Field(default=None) + preprint_server: Optional[PreprintServer] = Field(default=None) + + +class ReportedCopy(Copy): + """ + A copy a metadata source lists and we do not hold. + """ + reported_by: MetadataSource = Field(default=...) + route: CopyRoute = Field(default=...) + version: Optional[CopyVersion] = Field(default=None, description="""Absent when no source says which version the copy is.""") + license: License = Field(default=..., description="""This copy's own licence. A manuscript or preprint can be licensed differently from the version of record.""") + url: Optional[str] = Field(default=None) + preprint_server: Optional[PreprintServer] = Field(default=None) class Reference(ConfiguredBaseModel): @@ -1559,15 +1831,6 @@ class Citation(ConfiguredBaseModel): confidence: Optional[float] = Field(default=None, description="""For a heuristic alignment, how sure; absent for an exact one.""", ge=0, le=1) -class Correction(ConfiguredBaseModel): - """ - One notice attached to a paper. - """ - kind: Union[CorrectionKind, str] = Field(default=...) - pmid: Optional[str] = Field(default=None) - doi: Optional[str] = Field(default=None) - - class Supplement(ConfiguredBaseModel): """ One item of supplementary material. @@ -1792,16 +2055,20 @@ class AnalysisVerdict(ConfiguredBaseModel): # Model rebuild # see https://pydantic-docs.helpmanual.io/usage/models/#rebuilding-a-model ArtifactHeader.model_rebuild() -ArticleIdentifiers.model_rebuild() -PreprintDoi.model_rebuild() Producer.model_rebuild() InputRef.model_rebuild() TextSpan.model_rebuild() -ParsedPaper.model_rebuild() +License.model_rebuild() Bibliography.model_rebuild() +ArticleIdentifiers.model_rebuild() +PreprintDoi.model_rebuild() +Correction.model_rebuild() +ParsedPaper.model_rebuild() +Copy.model_rebuild() +HeldCopy.model_rebuild() +ReportedCopy.model_rebuild() Reference.model_rebuild() Citation.model_rebuild() -Correction.model_rebuild() Supplement.model_rebuild() Section.model_rebuild() ParsedTable.model_rebuild() diff --git a/src/study_schema/models/storage.py b/src/study_schema/models/storage.py index 33bfce6..0dd3fe5 100644 --- a/src/study_schema/models/storage.py +++ b/src/study_schema/models/storage.py @@ -32,7 +32,7 @@ metamodel_version = "1.12.0" -version = "0.8.0" +version = "0.9.0" class ConfiguredBaseModel(BaseModel): @@ -525,46 +525,161 @@ class Modality(str, Enum): """ -class ResultReporting(str, Enum): - """ - Whether a paper reports inferential analyses of its own. - """ - analyses_reported = "analyses_reported" +class PublicationStatus(str, Enum): """ - The paper reports at least one inferential test of its own. `analyses` lists them, null and coordinate-free ones included. + Whether a paper's version of record is known, or only its preprint. """ - no_analyses_reported = "no_analyses_reported" + published = "published" """ - The paper reports no inferential test of its own: a methods, protocol, data, or purely descriptive paper, a review, a commentary. `analyses` is empty by assertion. + A DOI, PubMed record or PMC copy of the version of record is known. The paper may have been a preprint too. """ - analyses_reported_elsewhere = "analyses_reported_elsewhere" + preprint = "preprint" """ - The paper's tests exist but are not in the text read: deferred to a companion paper, or to a supplement that was not available. `analyses` lists only what the text itself reports. + Only a preprint's identifiers are known. """ -class OutcomeSummary(str, Enum): +class PreprintServer(str, Enum): """ - What a study's analyses found, taken together. + The preprint server that registered a DOI. A server hosted on the Open Science Framework is named by its provider, which has a DOI prefix of its own. A server is added here when a producer learns to recognise its DOIs. """ - some_significant_effects = "some_significant_effects" + biorxiv = "biorxiv" """ - At least one analysis has `outcome: significant_effect`. + bioRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). """ - only_null_effects = "only_null_effects" + medrxiv = "medrxiv" """ - Every analysis has `outcome: no_significant_effect`. The study tested and found nothing; it has no coordinates for that reason, not for want of extraction. + medRxiv, DOI prefix 10.1101 and, since 2025-11, 10.64898 (openRxiv). """ - no_analyses = "no_analyses" + arxiv = "arxiv" """ - `result_reporting` is `no_analyses_reported` or `analyses_reported_elsewhere`, and `analyses` is empty. + arXiv, DOI prefix 10.48550, the DOI registered for an arXiv id. """ - undetermined = "undetermined" + researchsquare = "researchsquare" """ - No analysis reports a significant effect and at least one outcome was not reported, or `result_reporting` was not reported. + Research Square, DOI prefix 10.21203. + """ + osf_preprints = "osf_preprints" + """ + OSF Preprints, DOI prefix 10.31219. + """ + arabixiv = "arabixiv" + """ + Arabixiv (OSF), DOI prefix 10.31221. + """ + metaarxiv = "metaarxiv" + """ + MetaArXiv (OSF), DOI prefix 10.31222. + """ + engrxiv = "engrxiv" + """ + engrXiv (OSF), DOI prefix 10.31224. + """ + focus_archive = "focus_archive" + """ + FocUS Archive (OSF), DOI prefix 10.31225. + """ + frenxiv = "frenxiv" + """ + Frenxiv (OSF), DOI prefix 10.31226. + """ + ina_rxiv = "ina_rxiv" + """ + INA-Rxiv (OSF), DOI prefix 10.31227. + """ + lawarxiv = "lawarxiv" + """ + LawArXiv (OSF), DOI prefix 10.31228. + """ + lis_scholarship_archive = "lis_scholarship_archive" + """ + LIS Scholarship Archive (OSF), DOI prefix 10.31229. + """ + marxiv = "marxiv" + """ + MarXiv (OSF), DOI prefix 10.31230. + """ + mindrxiv = "mindrxiv" + """ + MindRxiv (OSF), DOI prefix 10.31231. + """ + nutrixiv = "nutrixiv" + """ + NutriXiv (OSF), DOI prefix 10.31232. + """ + paleorxiv = "paleorxiv" + """ + PaleorXiv (OSF), DOI prefix 10.31233. + """ + psyarxiv = "psyarxiv" + """ + PsyArXiv (OSF), DOI prefix 10.31234. + """ + socarxiv = "socarxiv" + """ + SocArXiv (OSF), DOI prefix 10.31235. + """ + sportrxiv = "sportrxiv" + """ + SportRxiv (OSF), DOI prefix 10.31236. + """ + africarxiv = "africarxiv" + """ + AfricArXiv (OSF), DOI prefix 10.31730. + """ + mediarxiv = "mediarxiv" + """ + MediArXiv (OSF), DOI prefix 10.33767. + """ + bodoarxiv = "bodoarxiv" + """ + BodoArXiv (OSF), DOI prefix 10.34055. + """ + edarxiv = "edarxiv" + """ + EdArXiv (OSF), DOI prefix 10.35542. + """ + biohackrxiv = "biohackrxiv" + """ + BioHackrXiv (OSF), DOI prefix 10.37044. + """ + ssrn = "ssrn" + """ + SSRN, DOI prefix 10.2139. + """ + preprints_org = "preprints_org" + """ + Preprints.org, DOI prefix 10.20944. + """ + authorea = "authorea" + """ + Authorea, DOI prefix 10.22541. + """ + techrxiv = "techrxiv" + """ + TechRxiv, DOI prefix 10.36227. + """ + chemrxiv = "chemrxiv" + """ + ChemRxiv, DOI prefix 10.26434. + """ + jmir_preprints = "jmir_preprints" + """ + JMIR Preprints, DOI prefix 10.2196. """ +class CorrectionKind(str, Enum): + """ + A notice attached to a paper by its publisher or indexer. + """ + erratum = "erratum" + retraction = "retraction" + expression_of_concern = "expression_of_concern" + comment = "comment" + update = "update" + + class StudyType(str, Enum): """ PubMed publication types, spelled as the E-utilities API returns them so that a value needs no translation. Only the design-relevant types are listed; PubMed also returns funding and indexing types, and the vocabulary is open, so any unlisted type passes through verbatim. @@ -602,6 +717,135 @@ class StudyType(str, Enum): Retracted_Publication = "Retracted Publication" +class ResultReporting(str, Enum): + """ + Whether a paper reports inferential analyses of its own. + """ + analyses_reported = "analyses_reported" + """ + The paper reports at least one inferential test of its own. `analyses` lists them, null and coordinate-free ones included. + """ + no_analyses_reported = "no_analyses_reported" + """ + The paper reports no inferential test of its own: a methods, protocol, data, or purely descriptive paper, a review, a commentary. `analyses` is empty by assertion. + """ + analyses_reported_elsewhere = "analyses_reported_elsewhere" + """ + The paper's tests exist but are not in the text read: deferred to a companion paper, or to a supplement that was not available. `analyses` lists only what the text itself reports. + """ + + +class OutcomeSummary(str, Enum): + """ + What a study's analyses found, taken together. + """ + some_significant_effects = "some_significant_effects" + """ + At least one analysis has `outcome: significant_effect`. + """ + only_null_effects = "only_null_effects" + """ + Every analysis has `outcome: no_significant_effect`. The study tested and found nothing; it has no coordinates for that reason, not for want of extraction. + """ + no_analyses = "no_analyses" + """ + `result_reporting` is `no_analyses_reported` or `analyses_reported_elsewhere`, and `analyses` is empty. + """ + undetermined = "undetermined" + """ + No analysis reports a significant effect and at least one outcome was not reported, or `result_reporting` was not reported. + """ + + +class LicenseId(str, Enum): + """ + The terms a licence grants. A value annotated `redistributable` lets the text it covers be passed on verbatim with attribution: every Creative Commons licence, including ND, because converting a format to text is a change CC 4.0 2(a)(4) allows, and NC, because neurostore's use is non-commercial. That reading of ND and NC is this project's policy, not legal advice, and no lawyer has confirmed it. + """ + cc_by = "cc-by" + cc_by_sa = "cc-by-sa" + cc_by_nc = "cc-by-nc" + cc_by_nc_sa = "cc-by-nc-sa" + cc_by_nd = "cc-by-nd" + cc_by_nc_nd = "cc-by-nc-nd" + cc0 = "cc0" + public_domain = "public-domain" + all_rights_reserved = "all-rights-reserved" + """ + A copyright statement with no licence, or terms that allow text mining only, such as PubMed Central's `TDM` for an author manuscript. + """ + other = "other" + """ + A stated licence outside this list, such as a publisher's own user licence or OpenAlex's `publisher-specific-oa`; `stated_as` holds it. Not redistributable, on purpose: Elsevier's open-access user licence is `other`, because its terms are not a CC licence's. + """ + + +class LicenseDetermination(str, Enum): + """ + Whether a source stated the licence. + """ + stated = "stated" + """ + A source stated it; `stated_by` says which. + """ + assumed = "assumed" + """ + Every source that could state a licence answered and none did, so all rights are reserved. + """ + + +class MetadataSource(str, Enum): + """ + Who stated a bibliographic fact. + """ + jats = "jats" + """ + A held copy's JATS XML, such as its ``. + """ + elsevier = "elsevier" + """ + A held copy's Elsevier full-text XML, its `coredata`. + """ + pmc_cloud = "pmc_cloud" + """ + PubMed Central's cloud record of a held copy. + """ + pubmed = "pubmed" + crossref = "crossref" + openalex = "openalex" + semantic_scholar = "semantic_scholar" + europepmc = "europepmc" + + +class OaStatus(str, Enum): + """ + How a paper is open to read, in OpenAlex's terms (https://help.openalex.org/hc/en-us/articles/24347035046295-Open-Access-OA). + """ + gold = "gold" + """ + Published openly licensed in an open-access journal. + """ + diamond = "diamond" + """ + Gold, in a journal that charges authors no fee. + """ + hybrid = "hybrid" + """ + Published openly licensed in a subscription journal. + """ + bronze = "bronze" + """ + Free to read at the publisher, without an open licence. + """ + green = "green" + """ + Closed at the publisher, with a free copy in a repository. + """ + closed = "closed" + """ + No free copy is known. + """ + + class AnalysisType(str, Enum): """ The method that produced the statistical map, independent of what was tested about it. A between-group comparison of connectivity, decoding accuracy, or component loadings keeps the method here and records the comparison in Effect. Bound only by ConjunctionComponent.analysis_type; an Analysis states its method by which AnalysisDetails subclass it fills. Open vocabulary: write the source's own wording in the field when nothing here fits. @@ -1405,20 +1649,6 @@ class AnchorKind(str, Enum): """ -class PublicationStatus(str, Enum): - """ - Whether a paper's version of record is known, or only its preprint. - """ - published = "published" - """ - A DOI, PubMed record or PMC copy of the version of record is known. The paper may have been a preprint too. - """ - preprint = "preprint" - """ - Only a preprint's identifiers are known. - """ - - class StudyDesign(ConfiguredBaseModel): """ @@ -2005,25 +2235,84 @@ class CoordinateSet(ConfiguredBaseModel): description: Optional[str] = Field(default=None, description="""What the set is, in the source's words, when the role alone does not say.""") +class Bibliography(ConfiguredBaseModel): + """ + The paper's bibliographic facts, as the metadata lookup returned them. Defined once: the parsed paper carries it, and the storage Study holds the same object as `Study.bibliography`, so nothing is copied field by field and nothing is extracted. + """ + title: str = Field(default=...) + authors: Optional[list[str]] = Field(default=None, description="""Author names in published order.""") + journal: Optional[str] = Field(default=None) + publication_year: Optional[int] = Field(default=None, description="""Year in which the study was first published or made publicly available.""", ge=1600) + volume: Optional[str] = Field(default=None) + issue: Optional[str] = Field(default=None) + pages: Optional[str] = Field(default=None) + abstract: Optional[str] = Field(default=None, description="""Absent (null) when no source gave one; never an empty string.""") + abstract_provider: Optional[MetadataSource] = Field(default=None, description="""The source whose abstract `abstract` is. It may differ from `provider`. The abstract is the publisher's abstract of the version `license` describes, so its licence is `license`.""") + language: Optional[list[str]] = Field(default=None, description="""Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as. +At least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.""") + publication_types: Optional[list[Union[StudyType, str]]] = Field(default=None, description="""PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels. `StudyType` lists the known types only; any other type PubMed returns is kept verbatim.""") + keywords: Optional[list[str]] = Field(default=None) + license: License = Field(default=..., description="""The paper's licence: the version of record's, or the latest preprint's while the header's `publication_status` is `preprint`. Not the licence of a copy of another version, which that copy carries.""") + oa_status: Optional[OaStatus] = Field(default=None) + corrections: Optional[list[Correction]] = Field(default=None, description="""Notices attached to the paper -- errata, retractions, expressions of concern -- as the indexer lists them. A retracted paper is still parsed and stored, marked by its `retraction` notice, and left out of studysets by default; a filter can ask for it explicitly.""") + provider: Optional[MetadataSource] = Field(default=None, description="""The metadata service the facts came from.""") + + @field_validator('abstract') + def pattern_abstract(cls, v): + pattern=re.compile(r"\S") + if isinstance(v, list): + for element in v: + if isinstance(element, str) and not pattern.search(element): + err_msg = f"Invalid abstract format: {element}" + raise ValueError(err_msg) + elif isinstance(v, str) and not pattern.search(v): + err_msg = f"Invalid abstract format: {v}" + raise ValueError(err_msg) + return v + + +class ArticleIdentifiers(ConfiguredBaseModel): + """ + Every identifier known for the paper when the artifact was written. Carried whole, so no consumer has to choose between them or look one up. `doi`, `pmid` and `pmcid` name the version of record; a preprint's identifiers are kept beside them in the `preprint_` fields, so a paper and its preprint are one article. + """ + pmid: Optional[str] = Field(default=None, description="""PubMed's record of the version of record. Never PubMed's record of a preprint, which is `preprint_pmid`.""") + pmcid: Optional[str] = Field(default=None, description="""PubMed Central's copy of the version of record. Never PMC's copy of a preprint, which is `preprint_pmcid`.""") + doi: Optional[str] = Field(default=None, description="""The DOI of the version of record. Never a preprint server's DOI, which is in `preprint_dois`.""") + preprint_dois: Optional[list[PreprintDoi]] = Field(default=None, description="""Every DOI a preprint server registered for the paper. A server that registers each version separately, such as Research Square's `/v2`, gives one entry per version, each kept as registered.""") + preprint_pmid: Optional[str] = Field(default=None, description="""PubMed's record of the preprint, which PubMed types `Preprint`.""") + preprint_pmcid: Optional[str] = Field(default=None, description="""PubMed Central's copy of the preprint.""") + publication_status: PublicationStatus = Field(default=..., description="""`preprint` when a `preprint_` field is set and `doi`, `pmid` and `pmcid` are all empty, `published` otherwise. The producer states it so that no consumer re-derives it.""") + + +class PreprintDoi(ConfiguredBaseModel): + """ + A DOI a preprint server registered, with the server that registered it. + """ + doi: str = Field(default=...) + server: PreprintServer = Field(default=..., description="""Named by the producer from the DOI, so that no consumer needs a table of preprint servers' DOI patterns.""") + + +class Correction(ConfiguredBaseModel): + """ + One notice attached to a paper. + """ + kind: Union[CorrectionKind, str] = Field(default=...) + pmid: Optional[str] = Field(default=None) + doi: Optional[str] = Field(default=None) + + class Study(ConfiguredBaseModel): """ The publication-level record that contains the groups, tasks, design, acquisitions, preprocessing pipelines, model estimations, analyses, and reported result tables belonging to one study. Conditions belong to the task they are part of, not to the study. """ id: str = Field(default=..., description="""Stable identifier for the study record, such as a DOI-based or repository-local identifier.""") - title: str = Field(default=..., description="""Full title of the study or publication.""") + bibliography: Bibliography = Field(default=..., description="""The paper's bibliographic facts, the parsed paper's `Bibliography` unchanged: title, authors, journal, year, abstract, PubMed types, the paper's licence.""") + identifiers: Optional[ArticleIdentifiers] = Field(default=None, description="""Every identifier known for the paper, the parsed paper's header identifiers unchanged.""") description: Optional[str] = Field(default=None, description="""Brief free-text summary of the study aims and scope.""") hypothesis: Optional[list[str]] = Field(default=None, description="""The study's own statement of what it expected to find, quoted from the paper -- usually the closing paragraph of the Introduction, in the form \"we hypothesized that\" or \"we predicted that\". Verbatim but for citation markers, one entry per prediction, so a study stating several gets several. A prediction, not a purpose. \"The aim of this study was to examine whether connectivity differs between patients and controls\" says what was tested and takes no value here; \"we expected connectivity to be lower in patients\" says what was expected and does. Recording an aim as a hypothesis asserts a direction the study never committed to, which is precisely what a reader consults this field to check. Absent when the paper states no prediction, which is common: never paraphrase an aim, an abstract sentence, or a finding into one. Where the only statement is in the Discussion -- \"consistent with our hypothesis that connectivity would be lower\" -- quote that clause, which is still the study's account of what it predicted rather than `Analysis.interpretations`, the source's account of what it found.""") design: Optional[StudyDesign] = Field(default=None, description="""The study's design outside its tasks: how participants were assigned, the arms they were assigned to, and the occasions data were collected on. The narrative design description is this record's own `description`.""") - authors: Optional[list[str]] = Field(default=None, description="""Authors of the study or publication, in published order.""") - journal: Optional[str] = Field(default=None, description="""Journal or other publication venue.""") - doi: Optional[str] = Field(default=None, description="""Digital Object Identifier of the version of record, without requiring a particular URL form. Never a preprint server's DOI.""") - publication_status: Optional[PublicationStatus] = Field(default=None, description="""Whether the version of record is known or only the preprint, copied from the parsed paper's header identifiers. `study_type` is independent of it: it keeps PubMed's publication types verbatim, `Preprint` included.""") - publication_year: Optional[int] = Field(default=None, description="""Year in which the study was first published or made publicly available.""", ge=1600) - language: Optional[list[str]] = Field(default=None, description="""Languages of the article, as the E-utilities API reports them -- the three-letter codes PubMed assigns, `eng` and the rest. Filled from the same `esummary` call that fills `study_type`, and not read off the paper: the text a reader is given is in one language whatever the article is indexed as. -At least six of the benchmark's sixteen meta-analyses state \"English language\" as an inclusion criterion, and it was unexpressible. On the 1,804 records measured it excludes exactly one paper, so it buys precision rather than recall; the criterion is stated, so the field states it.""") - study_type: list[Union[StudyType, str]] = Field(default=..., description="""PubMed publication types for this record, verbatim as the E-utilities API returns them (https://pubmed.ncbi.nlm.nih.gov/help/#publication-types). Multivalued because PubMed assigns several: every article is `Journal Article` and the design of the study motivates other labels.""") groups: Optional[list[Group]] = Field(default=None, description="""Participant or demographic groups.""") tasks: Optional[list[Task]] = Field(default=None, description="""Task, resting-state, stimulation, passive-viewing, or other meaningful participant paradigms associated with study acquisitions. Each holds the conditions it was made of.""") assessments: Optional[list[Assessment]] = Field(default=None, description="""Measures of participants that were taken independently of an acquisition.""") @@ -2051,6 +2340,29 @@ class ExternalDataset(ConfiguredBaseModel): url: Optional[str] = Field(default=None, description="""Optional landing-page, repository, or direct-access URL for the dataset.""") +class License(ConfiguredBaseModel): + """ + A licence as one source stated it, or all rights reserved assumed because no source stated one. + """ + license_id: LicenseId = Field(default=..., description="""Which licence, as OpenAlex's licence slugs name them (`cc-by`, `cc-by-nc`, ...). Not an SPDX License Identifier: sources rarely state the licence's version, and SPDX identifiers require one.""") + determination: LicenseDetermination = Field(default=...) + stated_by: Optional[MetadataSource] = Field(default=None, description="""The source whose statement this is.""") + stated_as: Optional[str] = Field(default=None, description="""The statement exactly as the source gave it: a licence URL, a licence name, or a copyright line. What `other` holds, and what attribution quotes. Never empty: a source that gave no statement did not state a licence.""") + + @field_validator('stated_as') + def pattern_stated_as(cls, v): + pattern=re.compile(r"\S") + if isinstance(v, list): + for element in v: + if isinstance(element, str) and not pattern.search(element): + err_msg = f"Invalid stated_as format: {element}" + raise ValueError(err_msg) + elif isinstance(v, str) and not pattern.search(v): + err_msg = f"Invalid stated_as format: {v}" + raise ValueError(err_msg) + return v + + class AnalysisDetails(ConfiguredBaseModel): """ Base class for the method-specific payload of one Analysis, instantiated only through its subclasses. Analysis.details is required, so exactly one subclass always applies. @@ -2226,8 +2538,13 @@ class NotStructurableDetails(AnalysisDetails): StatisticalMap.model_rebuild() Table.model_rebuild() CoordinateSet.model_rebuild() +Bibliography.model_rebuild() +ArticleIdentifiers.model_rebuild() +PreprintDoi.model_rebuild() +Correction.model_rebuild() Study.model_rebuild() ExternalDataset.model_rebuild() +License.model_rebuild() AnalysisDetails.model_rebuild() MassUnivariateDetails.model_rebuild() DecodingDetails.model_rebuild() diff --git a/storage-parameter-priorities.yaml b/storage-parameter-priorities.yaml index ef6bf3f..cccedb7 100644 --- a/storage-parameter-priorities.yaml +++ b/storage-parameter-priorities.yaml @@ -22,17 +22,12 @@ Timepoint: Study: id: n/a - title: n/a + bibliography: n/a + identifiers: n/a description: n/a hypothesis: 1 design: n/a - authors: n/a - journal: n/a - doi: n/a - publication_status: n/a - publication_year: n/a external_datasets: 1 - study_type: 0 groups: n/a tasks: n/a assessments: n/a @@ -49,6 +44,48 @@ Study: tables: n/a coordinate_sets: n/a +License: + license_id: n/a + determination: n/a + stated_by: n/a + stated_as: n/a + +Bibliography: + title: n/a + authors: n/a + journal: n/a + publication_year: n/a + volume: n/a + issue: n/a + pages: n/a + abstract: n/a + abstract_provider: n/a + language: n/a + publication_types: n/a + keywords: n/a + license: n/a + oa_status: n/a + corrections: n/a + provider: n/a + +ArticleIdentifiers: + pmid: n/a + pmcid: n/a + doi: n/a + preprint_dois: n/a + preprint_pmid: n/a + preprint_pmcid: n/a + publication_status: n/a + +PreprintDoi: + doi: n/a + server: n/a + +Correction: + kind: n/a + pmid: n/a + doi: n/a + ExternalDataset: name: 1 url: 1 diff --git a/tests/test_identifiers.py b/tests/test_identifiers.py index ae86435..3b75d06 100644 --- a/tests/test_identifiers.py +++ b/tests/test_identifiers.py @@ -81,20 +81,72 @@ def test_header_and_storage_share_one_publication_status_vocabulary(): } == {"published", "preprint"} +def _study_with(identifiers: dict) -> dict: + bibliography = { + "title": "An example", + "publication_types": ["Preprint"], + "license": {"license_id": "all-rights-reserved", "determination": "assumed"}, + } + return {"id": "x7Ex4mpleA01", "bibliography": bibliography, "identifiers": identifiers} + + @pytest.mark.parametrize("status", ["published", "preprint"]) def test_storage_study_carries_the_publication_status(status): validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") - study = {"id": "x7Ex4mpleA01", "title": "An example", "study_type": ["Preprint"], "publication_status": status} + study = _study_with({"publication_status": status}) - assert storage.Study.model_validate(study).publication_status == status + assert storage.Study.model_validate(study).identifiers.publication_status == status validator.validate(study, jsonschema.load("storage-study")) def test_storage_refuses_an_unknown_publication_status(): validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") - study = {"id": "x7Ex4mpleA01", "title": "An example", "study_type": ["Preprint"], "publication_status": "in_press"} + study = _study_with({"publication_status": "in_press"}) with pytest.raises(ValidationError, match="publication_status"): storage.Study.model_validate(study) with pytest.raises(validator.ValidationError): validator.validate(study, jsonschema.load("storage-study")) + + +def test_the_upload_pointer_is_the_headers_not_an_identifier(example_paths): + paper = load(example_paths["parsed_paper"]) + assert paper_parse.ParsedPaper.model_validate(paper).header.neurostore_base_study_id == "x7Ex4mpleA01" + paper["header"]["identifiers"]["neurostore_base_study_id"] = "x7Ex4mpleA01" + with pytest.raises(ValidationError, match="neurostore_base_study_id"): + paper_parse.ParsedPaper.model_validate(paper) + + +@pytest.mark.parametrize("year, valid", [(1600, True), (2026, True), (1599, False)]) +def test_a_publication_year_before_1600_is_refused(year, valid, example_paths): + # The parse and the storage Study share the bounds: one Bibliography. + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = load(example_paths["parsed_paper"]) + paper["bibliography"]["publication_year"] = year + study = {"id": "x7Ex4mpleA01", "bibliography": paper["bibliography"]} + for model, document, schema in ( + (paper_parse.ParsedPaper, paper, "parsed-paper"), + (storage.Study, study, "storage-study"), + ): + if valid: + model.model_validate(document) + validator.validate(document, jsonschema.load(schema)) + else: + with pytest.raises(ValidationError, match="publication_year"): + model.model_validate(document) + with pytest.raises(validator.ValidationError): + validator.validate(document, jsonschema.load(schema)) + + +def test_an_unlisted_publication_type_is_kept_verbatim(example_paths): + # StudyType lists known types only; PubMed's vocabulary is open. + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + types = ["Journal Article", "Research Support, N.I.H., Extramural"] + paper = load(example_paths["parsed_paper"]) + paper["bibliography"]["publication_types"] = types + study = {"id": "x7Ex4mpleA01", "bibliography": paper["bibliography"]} + + assert paper_parse.ParsedPaper.model_validate(paper).bibliography.publication_types == types + assert storage.Study.model_validate(study).bibliography.publication_types == types + validator.validate(paper, jsonschema.load("parsed-paper")) + validator.validate(study, jsonschema.load("storage-study")) diff --git a/tests/test_licensing.py b/tests/test_licensing.py new file mode 100644 index 0000000..f19c2a9 --- /dev/null +++ b/tests/test_licensing.py @@ -0,0 +1,352 @@ +"""A licence belongs to a copy; whether a text may be passed on is derived from the copy's +licence and how we obtained it, never stored.""" + +import pytest +import yaml +from pydantic import ValidationError + +from conftest import EXAMPLES, load +from study_schema import jsonschema, linkml +from study_schema.licensing import ( + REDISTRIBUTABLE_ACCESS, + REDISTRIBUTABLE_LICENSES, + abstract_may_be_redistributed, + text_may_be_redistributed, +) +from study_schema.models import extraction, paper_parse, storage + +LICENSES = EXAMPLES / "licenses" +LINKML = linkml.STORAGE.parent +# name -> (text may be passed on, abstract may be passed on) +VALID = { + "tdm_manuscript_of_open_article": (False, True), + "preprint_only": (True, True), + "assumed_all_rights_reserved": (False, False), +} +# Refused by the models and the JSON Schema alike. +INVALID = { + "publisher_tdm_code": "license_id", + "license_as_string": "license", + "held_copy_without_access": "access", +} +# LinkML rules: the JSON Schema states them as if/then/else; the pydantic models do not +# evaluate rules. +INVALID_BY_RULE = [ + "assumed_with_stated_by", + "abstract_without_provider", + "preprint_copy_without_server", + "server_named_off_a_preprint_server", +] + +CC_BY = {"license_id": "cc-by", "determination": "stated", "stated_by": "jats", "stated_as": "https://creativecommons.org/licenses/by/4.0/"} +ARR = {"license_id": "all-rights-reserved", "determination": "assumed"} + + +def _values(path, enum): + return yaml.safe_load(path.read_text())["enums"][enum]["permissible_values"] + + +def _paper_with(example: dict, example_paths) -> dict: + paper = load(example_paths["parsed_paper"]) + paper["header"]["identifiers"] = example["identifiers"] + paper["text_copy"] = example["text_copy"] + paper["reported_copies"] = example.get("reported_copies") + paper["bibliography"].update(example["bibliography"]) + return paper + + +def test_the_sets_match_the_schema_annotations(): + codes = _values(LINKML / "neuroimaging-study-storage" / "license.yaml", "LicenseId") + access = _values(LINKML / "neuroimaging-paper-parse" / "parse_enums.yaml", "CopyAccess") + assert {c for c, spec in codes.items() if (spec.get("annotations") or {}).get("redistributable")} == REDISTRIBUTABLE_LICENSES + assert {a for a, spec in access.items() if spec["annotations"]["redistribution"] == "by_license"} == REDISTRIBUTABLE_ACCESS + assert {spec["annotations"]["redistribution"] for spec in access.values()} == {"by_license", "never"} + + +@pytest.mark.parametrize( + "license, access, expected", + [ + (CC_BY, "public", True), + (CC_BY, "tdm_grant", False), # a CC-licensed Elsevier article fetched with our key + (CC_BY, "unrecorded", False), # a supplied PDF nobody recorded the origin of + (CC_BY, None, True), # the abstract of a CC-licensed paper + ({**CC_BY, "license_id": "cc-by-nc-nd"}, "public", True), + ({**CC_BY, "license_id": "other", "stated_as": "Elsevier user license"}, "public", False), + ({**CC_BY, "license_id": "all-rights-reserved", "stated_as": "© 2005 Elsevier Inc."}, "public", False), + (ARR, "public", False), + (ARR, None, False), + ], +) +def test_redistribution(license, access, expected): + if access is None: + assert abstract_may_be_redistributed(license) is expected + assert abstract_may_be_redistributed(storage.License.model_validate(license)) is expected + else: + copy = {"route": "pmc", "license": license, "access": access} + assert text_may_be_redistributed(copy) is expected + assert text_may_be_redistributed(paper_parse.HeldCopy.model_validate(copy)) is expected + + +def test_a_cc_by_copy_obtained_under_a_tdm_grant_is_not_redistributed(): + copy = paper_parse.HeldCopy.model_validate({"route": "publisher", "version": "vor", "access": "tdm_grant", "license": CC_BY}) + assert abstract_may_be_redistributed(copy.license) + assert not text_may_be_redistributed(copy) + + +def test_every_code_and_access_is_decided(): + for code in paper_parse.LicenseId: + license = {**CC_BY, "license_id": code.value} + for access in paper_parse.CopyAccess: + expected = code.value in REDISTRIBUTABLE_LICENSES and access.value == "public" + copy = {"route": "pmc", "license": license, "access": access} + assert text_may_be_redistributed(copy) is expected + assert abstract_may_be_redistributed(license) is (code.value in REDISTRIBUTABLE_LICENSES) + + +@pytest.mark.parametrize( + "license, access", + [ + (None, None), # no licence at all + (CC_BY, "bogus"), # an access outside CopyAccess + (CC_BY, ""), + ({**CC_BY, "license_id": "cc-by-4.0"}, "public"), # an id outside LicenseId + ({"license_id": "cc-by", "determination": "assumed"}, None), # the rules make an assumed licence ARR + ({**CC_BY, "determination": "assumed"}, "public"), + ({**CC_BY, "determination": "inferred"}, "public"), + ({**CC_BY, "stated_by": None}, "public"), # stated by nobody + ({**CC_BY, "stated_as": ""}, "public"), # stated in no words + ({**CC_BY, "stated_as": " "}, None), + ({"determination": "stated", "stated_by": "jats", "stated_as": "x"}, "public"), # no licence id + ], +) +def test_malformed_input_is_never_permitted(license, access): + if access is None: + assert abstract_may_be_redistributed(license) is False + assert text_may_be_redistributed({"route": "pmc", "license": license, "access": access}) is False + + +@pytest.mark.parametrize( + "copy", + [ + None, + {"route": "pmc", "license": CC_BY}, # a held copy without its access + {"route": "pmc", "license": CC_BY, "access": None}, + {"route": "pmc", "access": "public"}, # without its licence + ], +) +def test_a_malformed_held_copy_is_never_permitted(copy): + assert text_may_be_redistributed(copy) is False + + +@pytest.mark.parametrize("name, expected", VALID.items()) +def test_licence_examples_validate_and_decide(name, expected, example_paths): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = _paper_with(load(LICENSES / f"{name}.json"), example_paths) + + parsed = paper_parse.ParsedPaper.model_validate(paper) + validator.validate(paper, jsonschema.load("parsed-paper")) + # Written by the models, nulls and all, the file still validates. + validator.validate(parsed.model_dump(mode="json"), jsonschema.load("parsed-paper")) + text = text_may_be_redistributed(parsed.text_copy) + abstract = abstract_may_be_redistributed(parsed.bibliography.license) + assert (text, abstract) == expected + + +def test_the_open_pmc_example_may_be_passed_on(example_paths): + paper = paper_parse.ParsedPaper.model_validate(load(example_paths["parsed_paper"])) + assert (paper.text_copy.route, paper.text_copy.access) == ("pmc", "public") + assert text_may_be_redistributed(paper.text_copy) + assert abstract_may_be_redistributed(paper.bibliography.license) + + +def test_a_tdm_manuscript_keeps_the_articles_own_licence_beside_it(): + # PMC10518222: the held copy is a text-mining-only NIH manuscript of a CC BY-NC article. + example = load(LICENSES / "tdm_manuscript_of_open_article.json") + copy = paper_parse.HeldCopy.model_validate(example["text_copy"]) + bibliography = example["bibliography"] + assert (copy.version, copy.access, copy.license.license_id) == ("am", "tdm_grant", "all-rights-reserved") + assert bibliography["license"]["license_id"] == "cc-by-nc" + assert bibliography["abstract_provider"] != bibliography["provider"] + + +def test_a_preprint_only_paper_takes_the_preprints_licence(): + example = load(LICENSES / "preprint_only.json") + assert example["identifiers"]["publication_status"] == "preprint" + [server_copy] = example["reported_copies"] + assert (server_copy["route"], server_copy["preprint_server"]) == ("preprint_server", "biorxiv") + assert example["bibliography"]["license"]["license_id"] == server_copy["license"]["license_id"] + + +@pytest.mark.parametrize("name, field", INVALID.items()) +def test_invalid_licence_examples_are_refused(name, field, example_paths): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = _paper_with(load(LICENSES / "invalid" / f"{name}.json"), example_paths) + + with pytest.raises(ValidationError, match=field): + paper_parse.ParsedPaper.model_validate(paper) + with pytest.raises(validator.ValidationError): + validator.validate(paper, jsonschema.load("parsed-paper")) + + +def test_a_paper_without_its_text_copy_is_refused(example_paths): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = load(example_paths["parsed_paper"]) + del paper["text_copy"] + with pytest.raises(ValidationError, match="text_copy"): + paper_parse.ParsedPaper.model_validate(paper) + with pytest.raises(validator.ValidationError, match="text_copy"): + validator.validate(paper, jsonschema.load("parsed-paper")) + + +def test_a_paper_without_a_bibliography_licence_is_refused(example_paths): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = load(example_paths["parsed_paper"]) + del paper["bibliography"]["license"] + with pytest.raises(ValidationError, match="license"): + paper_parse.ParsedPaper.model_validate(paper) + with pytest.raises(validator.ValidationError, match="license"): + validator.validate(paper, jsonschema.load("parsed-paper")) + + +@pytest.mark.parametrize( + "where, value", + [ + (("bibliography", "abstract"), ""), + (("bibliography", "abstract"), " \n"), + (("bibliography", "license", "stated_as"), ""), + (("text_copy", "license", "stated_as"), " "), + ], +) +def test_an_empty_statement_or_abstract_is_refused(where, value, example_paths): + # A missing abstract or statement is null or absent, never an empty string. + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = load(example_paths["parsed_paper"]) + holder = paper + for key in where[:-1]: + holder = holder[key] + holder[where[-1]] = value + if where[-1] == "abstract": + paper["bibliography"]["abstract_provider"] = "pubmed" + with pytest.raises(ValidationError, match=where[-1]): + paper_parse.ParsedPaper.model_validate(paper) + with pytest.raises(validator.ValidationError): + validator.validate(paper, jsonschema.load("parsed-paper")) + + +@pytest.mark.parametrize("name", INVALID_BY_RULE) +def test_the_json_schema_refuses_what_the_rules_forbid(name, example_paths): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + paper = _paper_with(load(LICENSES / "invalid" / f"{name}.json"), example_paths) + with pytest.raises(validator.ValidationError): + validator.validate(paper, jsonschema.load("parsed-paper")) + + +def test_a_field_written_as_null_counts_as_absent(example_paths): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + schema = jsonschema.load("parsed-paper") + paper = _paper_with(load(LICENSES / "assumed_all_rights_reserved.json"), example_paths) + paper["bibliography"]["license"] = {**ARR, "stated_by": None, "stated_as": None} + validator.validate(paper, schema) + paper["bibliography"]["license"] = {**CC_BY, "stated_by": None} + with pytest.raises(validator.ValidationError): + validator.validate(paper, schema) + paper["bibliography"]["license"] = dict(CC_BY) + paper["bibliography"]["abstract"] = "An abstract." + paper["bibliography"]["abstract_provider"] = None + with pytest.raises(validator.ValidationError): + validator.validate(paper, schema) + + +SHARED_CLASSES = ("Bibliography", "ArticleIdentifiers", "PreprintDoi", "Correction", "License") +SHARED_ENUMS = ( + "LicenseId", "LicenseDetermination", "MetadataSource", "OaStatus", + "PublicationStatus", "PreprintServer", "CorrectionKind", "StudyType", +) + + +def _definitions(kind): + """Element name -> every YAML module (relative path) that defines it.""" + where = {} + for path in sorted(LINKML.rglob("*.yaml")): + for name in (yaml.safe_load(path.read_text()) or {}).get(kind) or {}: + where.setdefault(name, []).append(path.relative_to(LINKML).as_posix()) + return where + + +def test_the_shared_bibliography_is_defined_once_in_storage(): + classes, enums = _definitions("classes"), _definitions("enums") + for name in SHARED_CLASSES: + assert len(classes[name]) == 1 and classes[name][0].startswith("neuroimaging-study-storage/"), (name, classes[name]) + for name in SHARED_ENUMS: + assert len(enums[name]) == 1 and enums[name][0].startswith("neuroimaging-study-storage/"), (name, enums[name]) + codes = _values(LINKML / "neuroimaging-study-storage" / "license.yaml", "LicenseId") + assert "publisher-tdm" not in codes + + +def test_the_parse_imports_the_bibliography_and_storage_never_imports_the_parse(): + imports = lambda path: yaml.safe_load(path.read_text()).get("imports") or [] + for module in ("paper", "header"): + assert "../neuroimaging-study-storage/bibliography" in imports(LINKML / "neuroimaging-paper-parse" / f"{module}.yaml") + for path in (LINKML / "neuroimaging-study-storage").glob("*.yaml"): + assert not any("paper-parse" in item for item in imports(path)), path + + +def _class_of(field): + """The class a model field holds, through Optional.""" + import typing + annotation = field.annotation + args = [a for a in typing.get_args(annotation) if a is not type(None)] + return (args[0] if args else annotation).__name__ + + +def test_every_copy_in_the_map_names_a_field_that_exists_with_the_same_class(): + derivations = yaml.safe_load((LINKML / "extraction-to-storage.map.yaml").read_text())["derivations"] + copies = {target: d for target, d in derivations.items() if d.get("transform") == "copy"} + assert set(copies) == {"Study.bibliography", "Study.identifiers"} + for target, derivation in copies.items(): + assert derivation["source_artifact"] == "neuroimaging-paper-parse/ParsedPaper" + source_class, source_field = derivation["source_field"].split(".") + target_class, target_field = target.split(".") + source = getattr(paper_parse, source_class).model_fields[source_field] + stored = getattr(storage, target_class).model_fields[target_field] + assert _class_of(source) == _class_of(stored), target + + +@pytest.mark.parametrize("name", VALID) +def test_the_storage_study_holds_the_parsed_papers_bibliography_unchanged(name, example_paths): + # One class on both sides: the parse's dump is a storage Study's bibliography as it is. + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + parsed = paper_parse.ParsedPaper.model_validate(_paper_with(load(LICENSES / f"{name}.json"), example_paths)) + study = { + "id": "x7Ex4mpleA01", + "bibliography": parsed.bibliography.model_dump(mode="json"), + "identifiers": parsed.header.identifiers.model_dump(mode="json"), + } + + stored = storage.Study.model_validate(study) + validator.validate(study, jsonschema.load("storage-study")) + assert stored.bibliography.model_dump(mode="json") == study["bibliography"] + assert stored.identifiers.model_dump(mode="json") == study["identifiers"] + assert abstract_may_be_redistributed(stored.bibliography.license) is VALID[name][1] + + +def test_the_record_carries_no_bibliographic_fact(): + assert not {"bibliography", "identifiers", "title", "language", "study_type", "license"} & set(extraction.Study.model_fields) + + +def test_storage_refuses_a_licence_outside_the_contract(): + validator = pytest.importorskip("jsonschema", reason="optional: JSON Schema validation") + def study(license): + return {"id": "x7Ex4mpleA01", "bibliography": {"title": "An example", "license": license}} + tdm_code = study({"license_id": "publisher-tdm", "determination": "stated", "stated_by": "jats", "stated_as": "TDM"}) + with pytest.raises(ValidationError, match="license_id"): + storage.Study.model_validate(tdm_code) + with pytest.raises(validator.ValidationError): + validator.validate(tdm_code, jsonschema.load("storage-study")) + with pytest.raises(validator.ValidationError): + validator.validate(study({**ARR, "stated_by": "crossref"}), jsonschema.load("storage-study")) + + +def test_storage_requires_the_bibliography(): + with pytest.raises(ValidationError, match="bibliography"): + storage.Study.model_validate({"id": "x7Ex4mpleA01"})