feat(transfers): add NM_Wells 1:1 staging mirror + ref-table lexicon loader#686
Draft
jirhiker wants to merge 22 commits into
Draft
feat(transfers): add NM_Wells 1:1 staging mirror + ref-table lexicon loader#686jirhiker wants to merge 22 commits into
jirhiker wants to merge 22 commits into
Conversation
Phase 1 of the NM_Wells -> Ocotillo migration: faithful column-for-column staging mirror of the legacy NM_Wells SQL Server DB, plus loaders. The transform into the Ocotillo model (Phase 2) is documented inline but not built. - db/nmw_legacy.py: 17 NMW_* mirror models (5 Main, 7 Geothermal, 5 DST), source column names preserved, per-column Phase-2 transform-target notes. Main columns from the planning workbook field map; Geothermal/DST columns, lengths and PKs taken directly from the SQL-dump DDL. - alembic: two migrations (Main; Geothermal+DST) chained off current head, bodies generated from model metadata. Single head. - transfers/nmw_mirror_transfer.py: data-driven CSV -> NMW_* loader with type coercion (NaN/NaT -> None, rowversion dropped), chunked ON CONFLICT upsert. Gated by TRANSFER_NMW_MIRROR (default off; separate source DB). - transfers/reference_lexicon_transfer.py: loads all 49 ref_* lookups into the lexicon (category per table), idempotent like init_lexicon; registered as a foundational transfer. - db/__init__.py, transfers/transfer.py, .env.example: wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Phase 1 of the NM_Wells → Ocotillo migration: introduces a 1:1 staging-mirror schema for NM_Wells legacy tables, plus transfer loaders to (a) populate the staging mirror from CSV exports and (b) load ref_* reference tables into the lexicon as foundational data.
Changes:
- Add NM_Wells legacy staging mirror ORM models and Alembic migrations for Main + Geothermal/DST tables.
- Add a generic, chunked mirror loader (
transfer_nmw_mirror) and gate it behindTRANSFER_NMW_MIRROR(default off). - Add a foundational “reference → lexicon” loader (
transfer_reference_tables) and run it as part of Phase 1 foundational transfers.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| transfers/transfer.py | Wires in new foundational reference→lexicon transfer and optional NM_Wells mirror load; adjusts foundational parallelism. |
| transfers/reference_lexicon_transfer.py | New transfer to ingest legacy ref_* lookups into lexicon categories/terms. |
| transfers/nmw_mirror_transfer.py | New generic CSV→NMW_* staging mirror loader with type coercion + chunked idempotent inserts. |
| db/nmw_legacy.py | New SQLAlchemy models for 1:1 NM_Wells staging mirror tables. |
| db/init.py | Exposes NM_Wells legacy models via db package import. |
| alembic/versions/u7v8w9x0y1z2_nmw_legacy_staging_mirror_tables.py | Migration creating the 5 “Main” staging mirror tables. |
| alembic/versions/v8w9x0y1z2a3_nmw_geothermal_dst_mirror_tables.py | Migration creating Geothermal + DST staging mirror tables. |
| .env.example | Adds TRANSFER_NMW_MIRROR toggle (default false). |
- nmw_mirror_transfer: parse DateTime values with pd.to_datetime(errors=coerce) since read_csv does not parse_dates (avoids driver-dependent insert failures). - db/nmw_legacy: fix attribute typos (dst_operator, recov_column, resistivity) while preserving the legacy DB column names; fix latitude_dd27 comment typo. - reference_lexicon_transfer: correct stale exclusion comment (ref_date_drilled is included; only ref_nm_quads is excluded). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jirhiker
changed the title
The SSMA_TimeStamp column is a SQL Server rowversion artifact with no value as staging data (the loader already skipped it). Remove it from the NMW_* mirror models and both migrations; drop the now-unused LargeBinary import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Confirmed source PKs from the NM_Wells SQL dump DDL: - WellHeaders/WellRecords/WellSamples have declared PRIMARY KEY constraints (WellDataID / RecrdSetID / SamplSetID) matching the models. - WellLocations and WellZDatum declare no PK, only unique indexes on OBJECTID and GlobalID. Switch WellZDatum PK from GlobalID to OBJECTID for consistency with WellLocations and safety (OBJECTID identity is never NULL; the GlobalID unique index permits one NULL). Update the migration accordingly. Remove the TODO(verify) note; PKs are now confirmed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add transfers/nmw_sql_dump.py: streams INSERT [dbo].[tbl_*] (...) VALUES (...)
statements out of a SQL Server data-dump .sql file, yielding {column: value}
dicts. Handles N'...' / escaped '', embedded commas/parens, CAST(expr AS type),
multi-row VALUES, 0x binary -> None, and UTF-16/UTF-8 (BOM auto-detect).
Refactor transfer_nmw_mirror to be source-agnostic: when NMW_SQL_DUMP points at
a .sql data dump it loads from there, otherwise falls back to per-table CSVs.
Same model-driven type coercion and chunked ON CONFLICT upsert for both.
Note: the provided NMWells.sql is schema-only; NMW_SQL_DUMP expects a separate
data dump containing INSERT statements.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er.py Move the NM_Wells (geothermal) orchestration out of transfers/transfer.py into a new standalone transfers/transfer_geothermal.py. Revert all NM_Wells wiring from transfer.py and mark that module deprecated (module docstring + DeprecationWarning in transfer_all) so new migrations get their own orchestrator. transfer_geothermal.py runs the reference->lexicon load (TRANSFER_GEOTHERMAL_REFERENCE) and the NMW_* mirror load (TRANSFER_NMW_MIRROR); both default on. Run: python -m transfers.transfer_geothermal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
reference_lexicon_transfer now selects its row source the same way as nmw_mirror_transfer: a SQL Server data dump when NMW_SQL_DUMP is set (parsed by nmw_sql_dump.iter_table_rows), otherwise per-table CSV. _pick_columns operates on a column-name list and rows are processed as dicts so both sources share one path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add LEXICON_REF_BY_COLUMN mapping every coded mirror attribute to its ref_* source table (which reference_lexicon_transfer loads as a lexicon category whose rows become terms). These 40 attributes will become lexicon_term FKs / enums in the Phase-2 transform. Add LEXICON_CANDIDATES_NO_REF for 8 coded columns that have no ref_* table and will need a new category/enum (DrillFluid, TestType, Operation, etc.). Validated: every column + ref table exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the dead `category = table[4:]` line and fix the stale docstring; the category is nmw_<table> (e.g. nmw_ref_states). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ile) Two pygeoapi point layers over the NMW_* staging mirror, geometry from NMW_WellLocations Lat/Long_dd83: - ogc_geothermal_wells_bht: one feature per geothermal well with bottom-hole temperature data (NMW_GtBhtData), aggregate BHT stats. - ogc_geothermal_wells_temperature_profile: one feature per geothermal well with a downhole temperature-vs-depth series (NMW_GtTempDepths) as an ordered JSON array. Wells link via gt_*.SamplSetID -> NMW_WellSamples -> NMW_WellRecords -> NMW_WellLocations. Guards required tables; drops views on downgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The temperature-vs-depth profile view scans/groups NMW_GtTempDepths (~370k source rows) and builds a per-well JSON series — too heavy to recompute per pygeoapi request. Convert it to a MATERIALIZED view with a unique index on well_data_id (enables REFRESH CONCURRENTLY) and a GiST index on geom. The BHT view stays a regular view (small source). REFRESH after a data reload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pygeoapi point layer ogc_geothermal_wells_heat_flow: one feature per geothermal well with summary heat-flow determinations (NMW_GtSumHeatFlow) - aggregate heat flow, thermal gradient, thermal conductivity and quality. Geometry from NMW_WellLocations; linked via NMW_GtSumHeatFlow.RecrdSetID -> NMW_WellRecords. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pygeoapi point layer ogc_geothermal_wells_interval_heat_flow from NMW_GtHeatFlow (per-interval values: Q heat flow, gradient, Kpr conductivity, Ka diffusivity), one feature per well. Distinct from ogc_geothermal_wells_heat_flow (summary, NMW_GtSumHeatFlow). Linked via IntrvlGUID -> NMW_WsIntervals -> NMW_WellSamples -> NMW_WellRecords -> NMW_WellLocations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Rename ogc_geothermal_wells_heat_flow -> ogc_geothermal_wells_summary_heat_flow. - Add a `measurements` JSON series to both heat-flow views: one element per determination/interval (depth range, heat flow, gradient, conductivity, etc.), ordered by depth, alongside the existing per-well aggregates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When NMW_SQL_DUMP is set, the mirror now parses the dump with sqlparse (nmw_sql_dump.write_table_csv) into a CSV per table, then bulk-loads each via Postgres COPY ... FROM STDIN (truncate + COPY; Postgres casts text -> types) — far faster than row-by-row ORM inserts. CSV dir defaults to a temp dir (override NMW_CSV_DIR). The CSV-exports fallback (no dump) keeps the row-insert path. Adds sqlparse dependency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add refresh_materialized_views (REFRESH the geothermal materialized views, currently ogc_geothermal_wells_temperature_profile; skip any not present). The transfer_geothermal orchestrator calls it after the NMW_* mirror load so the materialized view reflects the freshly loaded data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er-93c916 # Conflicts: # pyproject.toml
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve requirements.txt conflict by regenerating from the merged uv.lock (uv export). Brings staging fixes incl. the CLI test update. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 of the NM_Wells → Ocotillo migration: a 1:1 staging-mirror schema for the legacy NM_Wells SQL Server database, loaders that populate it (from a SQL dump or CSV), the
ref_*→ lexicon loader, a standalone orchestrator, and pygeoapi OGC views over the geothermal data.Two-phase by design:
NMW_*mirror tables (faithful, column-for-column), following thedb/nma_legacy.py(NM_Aquifer) convention.What's included
Mirror schema —
db/nmw_legacy.py+ 2 migrations17 of 22 "Migrate First" source tables,
NMW_prefix, original column names preserved, each column annotated with its Phase-2 Ocotillo target:SSMA_TimeStamprowversion dropped.u7v8w9x0y1z2(Main),v8w9x0y1z2a3(Geothermal/DST); bodies generated from model metadata.Loaders
transfers/nmw_mirror_transfer.py— data-driven CSV/SQL →NMW_*loader; model-driven type coercion (NULL/NaN/NaT → None, rowversion dropped), chunkedON CONFLICT DO NOTHINGupsert.transfers/nmw_sql_dump.py— streamsINSERT [dbo].[tbl_*] (...) VALUES (...)from a SQL Server data dump (handlesN'...'/escaped'', embedded commas/parens,CAST, multi-row VALUES,0xbinary, UTF-16/UTF-8 BOM).transfers/reference_lexicon_transfer.py— loads all 49ref_*lookups into the lexicon (one category per table; idempotent likeinit_lexicon).NMW_SQL_DUMPis set, else per-table CSV.Orchestration
transfers/transfer_geothermal.py— standalone orchestrator (python -m transfers.transfer_geothermal): runs the reference→lexicon load then the mirror load. FlagsTRANSFER_GEOTHERMAL_REFERENCE,TRANSFER_NMW_MIRROR,TRANSFER_LIMIT,NMW_SQL_DUMP.transfers/transfer.py(the legacy NM_Aquifer driver) is deprecated — module docstring +DeprecationWarning; no NM_Wells wiring added to it.Lexicon flagging (for Phase 2)
db/nmw_legacy.pyexposesLEXICON_REF_BY_COLUMN(40 coded attributes → theirref_*source table; will becomelexicon_termFKs / enums) andLEXICON_CANDIDATES_NO_REF(8 coded columns with no ref table → need a new category/enum). Validated: every column + ref table exists.OGC views (pygeoapi) — 5 geothermal layers
ogc_geothermal_wells_bht— wells with bottom-hole-temperature data (NMW_GtBhtData), aggregate BHT stats.ogc_geothermal_wells_temperature_profile— materialized + indexed (uniquewell_data_id, GiSTgeom); downhole temp-vs-depth series (NMW_GtTempDepths, ~370k source rows) as an ordered JSON array.ogc_geothermal_wells_summary_heat_flow— summary heat-flow determinations (NMW_GtSumHeatFlow): aggregates + ameasurementsJSON series.ogc_geothermal_wells_interval_heat_flow— per-interval heat-flow values (NMW_GtHeatFlow): aggregates + ameasurementsJSON series.NMW_WellLocationsLat/Long_dd83; required-table guards; follow the existingogc_*migration pattern.w9x0y1z2a3b4(BHT + profile),x0y1z2a3b4c5(summary heat flow),y1z2a3b4c5d6(interval heat flow).Design notes
NMWells.sqlis schema-only (no rows) — it seeded the models/migrations;NMW_SQL_DUMPshould point at a separate data dump (or use CSV exports).REFRESH MATERIALIZED VIEWafter a data reload.Not in this PR
tbl_sources+ 4 Subsurface Library tables.Testing
y1z2a3b4c5d6); pre-commit (black/flake8) green.🤖 Generated with Claude Code