Conversation
Add view_NGWMN_WaterLevels, view_NGWMN_WellConstruction, and view_NGWMN_Lithology as PostgreSQL views over the new Ocotillo tables, replicating the legacy AMPAPI (NM_Aquifer) view definitions so NGWMN exports no longer depend on the static NMA_view_NGWMN_* copy tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Point the /ngwmn endpoint queries at view_NGWMN_WaterLevels, view_NGWMN_WellConstruction, and view_NGWMN_Lithology instead of the legacy NMA_view_NGWMN_* copy tables, and add endpoint tests covering the XML output, BGS depth math, timezone date handling, measurement method and accuracy mapping, and private-record exclusion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Aggregate raw transducer observations into one row per well, parameter, day, and QC status, replacing the legacy NMA_WaterLevelsContinuous_Pressure_Daily table as the daily rollup. Includes a unique index so the view supports REFRESH MATERIALIZED VIEW CONCURRENTLY via the refresh CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the legacy NMA_WaterLevelsContinuous_Pressure_Daily query in make_waterlevels_response with the transducer_daily_data materialized view, and simplify the merge/XML code to use the view's (point_id, date_measured, depth_to_water_bgs) rows instead of the 19-column positional legacy layout. The merge rule is unchanged: on overlapping dates the manual reading wins when shallower, and only one record is emitted per date. Guard the depth comparison against NULLs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
view_NGWMN_WaterLevels, view_NGWMN_WellConstruction, and view_NGWMN_Lithology become NGWMN_WaterLevels, NGWMN_WellConstruction, and NGWMN_Lithology. The migration has not shipped, so it is edited in place rather than adding a rename migration. References to the legacy AMPAPI view names and CSV source tables keep the view_ prefix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jirhiker
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR moves the /ngwmn/* export endpoints off legacy static copy tables and onto live database views/materialized views backed by the new Ocotillo data model, including a new daily transducer aggregation MV used by the water-level export.
Changes:
- Added
NGWMN_WaterLevels,NGWMN_WellConstruction, andNGWMN_LithologySQL views sourced from the Ocotillo tables. - Added
transducer_daily_datamaterialized view (with indexes) to provide daily transducer aggregates and support concurrent refresh. - Updated the NGWMN service helper to query the new views and simplified manual/transducer merge logic; added new endpoint and MV tests.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
alembic/versions/u8v9w0x1y2z3_add_ngwmn_views_from_ocotillo_model.py |
Introduces NGWMN_* views over the new Ocotillo schema. |
alembic/versions/v0w1x2y3z4a5_add_transducer_daily_data_materialized_view.py |
Adds transducer daily aggregate materialized view + indexes for concurrent refresh. |
services/ngwmn_helper.py |
Switches NGWMN endpoints to the new views and updates manual/transducer merge. |
tests/test_ngwmn_endpoints.py |
Adds coverage for XML responses, BGS math, timezone handling, privacy exclusion, and merge behavior. |
tests/test_transducer_daily_data.py |
Adds coverage for MV aggregation correctness and concurrent refresh capability. |
.gitignore |
Ignores Claude Code local settings/worktrees. |
Add read-only ORM mappings for the NGWMN_* views and the transducer_daily_data materialized view on a separate declarative base, kept out of Base.metadata so Alembic autogenerate does not try to create tables for them. The NGWMN response builders now query these models column-wise instead of executing raw SQL strings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The NGWMN views filtered only on observation.release_status, so a private thing (or private screen/casing/lithology rows) could still be exported. Require release_status='public' on the thing, field event, field activity, sample, and observation in NGWMN_WaterLevels; on the thing, well_screen, and well_casing_material rows in NGWMN_WellConstruction; and on the thing and formation association in NGWMN_Lithology. The transducer daily query in make_waterlevels_response now joins thing and requires it to be public as well. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7e7c5d18a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Moves the
/ngwmnendpoints off the static legacy copy tables and onto live views over the new Ocotillo data model.NGWMN_WaterLevels,NGWMN_WellConstruction,NGWMN_Lithologyviews (migrationu8v9w0x1y2z3): reproduce the original AMPAPI (NM_Aquifer)view_NGWMN_*definitions, but sourced fromobservation/sample/field_activity/field_event/thing,well_screen/well_casing_material, andthing_geologic_formation_association/geologic_formation.transducer_daily_datamaterialized view (migrationv0w1x2y3z4a5): daily aggregates of raw transducer observations per well, parameter, day, and QC status. ReplacesNMA_WaterLevelsContinuous_Pressure_Dailyas the daily rollup. Has a unique index soREFRESH MATERIALIZED VIEW CONCURRENTLYworks via the refresh CLI.services/ngwmn_helper.py: all three NGWMN queries now read the new views. The manual/transducer merge code was simplified from the 19-column positional legacy layout to the view's(point_id, date_measured, depth_to_water_bgs)rows; merge semantics are unchanged (manual reading wins on a shared date when shallower, one record per date), with a new NULL guard on the depth comparison..gitignore: ignore.claude/settings.local.jsonand.claude/worktrees/.Notes for review
DepthToWaterBGSis computed asobservation.value - measuring_point_height, because the transfer stored depth below the measuring point invalue.DateMeasuredinverts the transfer's timezone logic: rows stored at 00:00 UTC (no time measured) keep the UTC date; others shift back to America/Denver.NGWMN_Lithology.LithologyandTERMboth carry the resolved lithology term (the legacy abbreviation code was not migrated);StratSourceis NULL (never transferred).CasingDescriptionis rebuilt as a comma-separated list ofwell_casing_materialterms; the legacy free text was not migrated.transducer_daily_data(oco refresh-pygeoapi-materialized-views --view transducer_daily_data).NMA_view_NGWMN_*tables and their transfer scripts are untouched; references with theview_prefix that remain intransfers/point at legacy CSV sources, which is intentional.Testing
tests/test_ngwmn_endpoints.py: XML output for all three endpoints, BGS math, timezone date handling, method/accuracy mapping, private-record exclusion, and the manual/transducer merge (manual wins, transducer-only, transducer wins).tests/test_transducer_daily_data.py: aggregation correctness and concurrent refresh.tests/test_ogc.pyfailures are pre-existing onstaging(verified on a clean tree).🤖 Generated with Claude Code