Cmr latest version by lavanya3k · Pull Request #355 · NASA-IMPACT/pyQuARC

lavanya3k · 2025-10-02T17:55:19Z

Description of the code changes: During the pyQuARC run, we want to ensure that the collection ID retrieved from the CMR query uses the latest version of the schema (e.g., umm-c). There were instances in which multiple versions of the collection ID existed in the CMR, particularly CDDIS. With this code fix, pyQuARC runs for the latest or recent version of the collection.

Example: C1000000003-CDDIS (umm-c).

Expected output: Schema version is displayed at the top of the results

FBayat021 · 2025-10-06T15:19:35Z

@lavanya3k I tested on the following Concept IDs:

C3173447237-NSIDC_CPRD
C3274606314-NSIDC_CPRD
C1977859380-GHRC_DAAC (umm-c native 1.18.3)

And it pulled the latest version in 1.18.4! Great work! Just to confirm, this does cross-validate against Schema version 1.18.4/latest version available? (It's not just adjusting the schema specification value to the latest, right?)

Also, can we fix the output message here so it says Collection instead of ollection?

Let @slesaad test this too!

lavanya3k · 2025-10-06T15:31:13Z

@fb0023 - Thanks for testing the PR. The above fix will retrieve the latest version of the collection ID (e.g., 1.18.4) and validate it against umm-c 1.18.4 in pyQuARC. I'm not sure if this addresses the CMR validation completely, which was requested by the ESDIS team. I am going to let @slesaad and you take over, and let me know if anything is missing with the request.
I will fix the typo with the 'Collection' shortly.

lavanya3k · 2025-10-07T19:24:28Z

Here is the screenshot after retesting. The typo did not occur while testing.
Included code changes in main.py for error_prompt, and updated to the latest version of lxml for Python 3.10+.

slesaad · 2025-10-07T19:26:04Z

pyQuARC/main.py

+    def _get_latest_version(self, concept_id):
+        """
+        Fetches the latest revision version for a given concept_id from CMR
+
+        Args:
+            concept_id (str): The concept ID to query
+
+        Returns:
+            str: The latest revision number, or None if not found
+        """
+        try:
+            # Construct the CMR metadata URL for the concept
+            url = f"{self.cmr_host}/search/concepts/{concept_id}.umm_json"
+            headers = get_headers()
+            response = requests.get(url, headers=headers)
+
+            if response.status_code == 200:
+                # Extract revision-id from response headers
+                revision_id = response.headers.get('CMR-Revision-Id')
+                return revision_id
+            else:
+                print(f"Warning: Could not fetch latest version for {concept_id}. Using default.")
+                return None
+        except Exception as e:
+            print(f"Error fetching latest version for {concept_id}: {str(e)}")
+            return None
+
+    def _get_collection_version(self, concept_id):
+        """
+        Fetch the MetadataSpecification.Version of a collection from CMR.
+        Args:
+            concept_id (str): The concept ID to query.
+
+        Returns:
+            str: The collection's MetadataSpecification.Version, or None if not found.
+        """
+        try:
+            url = f"{self.cmr_host}/search/concepts/{concept_id}.umm_json"
+            headers = get_headers()
+            response = requests.get(url, headers=headers)
+
+            if response.status_code == 200:
+                data = response.json()
+                # UMM collections have MetadataSpecification.Version
+                version = data.get("MetadataSpecification", {}).get("Version")
+                return version
+            else:
+                print(f"Warning: Could not fetch metadata for {concept_id}.")
+                return None
+        except Exception as e:
+            print(f"Error fetching collection version for {concept_id}: {str(e)}")
+            return None
+


The two methods have a lot in common - can you DRY (dont repeat yourself) it? also not sure what's the difference between these two versions and it looks like you're just printing the collection_version, but not using it elsewhere, do you even need that?

Agreed. Please take a look at the updated code changes.

slesaad · 2025-10-07T19:26:18Z

pyQuARC/main.py

@@ -194,24 +260,19 @@
                    )
                    continue
                content = content.encode()
-                cmr_response = self._validate_with_cmr(concept_id, content)
                validation_errors, pyquarc_errors = checker.run(content)
                self.errors.append(
                    {
                        "concept_id": concept_id,
                        "errors": validation_errors,
-                        "cmr_validation": {
-                            "errors": cmr_response.json().get("errors", []),
-                            # TODO: show warnings
-                            "warnings": cmr_response.json().get("warnings", [])
-                        },


why did we remove cmr validation?

Do you suggest keeping the cmr_validation. Please feel free to edit the code.

We have a ticket that's asking for cmr validation - why remove?

slesaad · 2025-10-15T19:18:34Z

pyQuARC/main.py

+            info_type (str): Type of information to fetch.
+                        Options: "revision" or "metadata_version".
+


i don't think this is implemented??

It is only the metadata version, and not the revision.

slesaad · 2025-10-15T19:22:10Z

pyQuARC/main.py

+
        Returns:
-            str: The collection's MetadataSpecification.Version, or None if not found.
+            str: The requested info (revision ID or MetadataSpecification.Version), or None if not found.


Suggested change

str: The requested info (revision ID or MetadataSpecification.Version), or None if not found.

dict: {"revision_id": str | None, "metadata_version": str | None } A dict of Revision ID and Metadata Version of the collection

slesaad · 2025-10-15T19:28:13Z

pyQuARC/main.py

+            if response.status_code != 200:
+                print(f"Warning: Could not fetch data for {concept_id}. Status: {response.status_code}")
+                return {"revision_id": None, "metadata_version": None}
+
+            data = response.json() if response.content else {}
+            return {
+                "revision_id": response.headers.get("CMR-Revision-Id"),
+                "metadata_version": data.get("MetadataSpecification", {}).get("Version"),
+            }
+
        except Exception as e:
-            print(f"Error fetching collection version for {concept_id}: {str(e)}")
-            return None
+            # Unified error handling — return dict even on failure
+            print(f"Error fetching collection info for {concept_id}: {str(e)}")


Rewrite this method with something like this:

failure_return_value = {"revision_id": None, "metadata_version": None} try: url = f"{self.cmr_host}/search/concepts/{concept_id}.umm_json" headers = get_headers() response = requests.get(url, headers=headers) response.raise_for_status() data = response.json() if response.content else {} return { "revision_id": response.headers.get("CMR-Revision-Id"), "metadata_version": data.get("MetadataSpecification", {}).get("Version"), } except Exception as e: # Unified error handling — return dict even on failure print(f"Error fetching collection info for {concept_id}: {str(e)}") return failure_return_value

slesaad · 2025-10-15T19:31:43Z

pyQuARC/main.py

+                if version_to_use:
+                    print(f"Using latest revision {version_to_use} for {concept_id}")
+                if metadata_version:
+                    print(f"Collection {concept_id} schema version: {metadata_version}")


Let's remove these print statements

lavanya3k · 2025-10-16T20:38:06Z

Tested in the local branch, and it works:

Expected output:

CMR latest version #355

aca4e40

lavanya3k changed the base branch from master to dev October 2, 2025 17:59

lavanya3k mentioned this pull request Oct 2, 2025

pyQuARC API Call Enhancement #329

Closed

FBayat021 requested review from FBayat021 and slesaad October 6, 2025 14:23

Cmr latest version #355 - included changes in main.py

6495730

slesaad reviewed Oct 7, 2025

View reviewed changes

Code changes for _get_collection_version -LA

40e5202

slesaad reviewed Oct 15, 2025

View reviewed changes

slesaad added 2 commits October 16, 2025 15:20

Refactor and undo cmr_validation removal

ed5a941

Readd concept id printing

24d9909

slesaad approved these changes Oct 16, 2025

View reviewed changes

lavanya3k merged commit fcba356 into dev Oct 16, 2025
1 check passed

This was referenced Oct 17, 2025

Release v1.3.0 #364

Merged

Automate schema in rule_mapping #341

Closed

Update Schema Version #317

Closed

		info_type (str): Type of information to fetch.
		Options: "revision" or "metadata_version".

	str: The requested info (revision ID or MetadataSpecification.Version), or None if not found.
	dict: {"revision_id": str \| None, "metadata_version": str \| None } A dict of Revision ID and Metadata Version of the collection

Conversation

lavanya3k commented Oct 2, 2025

Uh oh!

FBayat021 commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lavanya3k commented Oct 6, 2025

Uh oh!

lavanya3k commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lavanya3k commented Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FBayat021 commented Oct 6, 2025 •

edited

Loading

lavanya3k commented Oct 7, 2025 •

edited

Loading