From ed50baab6780498a585af0456d90a15c9f05676a Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 10:44:34 -0600 Subject: [PATCH 01/13] refactor: update README --- README.md | 69 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 53 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index d0a8c4980..e4d51cab1 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ NMSampleLocations is a FastAPI-based backend service designed to manage geospati - ๐Ÿ”Ž Filtering by location, date, type, and more - ๐Ÿ“ฆ PostgreSQL + PostGIS database backend - ๐Ÿ” Optional authentication and role-based access -- ๐Ÿงพ Interactive API documentation via Swagger and ReDoc +- ๐Ÿงพ Interactive API documentation via OpenAPI and ReDoc --- @@ -31,40 +31,77 @@ NMSampleLocations is a FastAPI-based backend service designed to manage geospati - Python 3.11+ - PostgreSQL with PostGIS extension - [`uv`](https://github.com/astral-sh/uv) package manager +- Docker Desktop 4+ -### Installation +### Installation & Development Setup + +#### 1. Clone the repository ```bash -# Clone the repository git clone https://github.com/DataIntegrationGroup/NMSampleLocations.git cd NMSampleLocations +``` + +#### 2. Set up virtual environment and install depdencies + +Mac/Linux +```bash +uv venv +source .venv/bin/activate # (Mac/Linux) +uv pip install -r requirements.txt +``` -# Set up virtual environment and install dependencies +Windows +```bash uv venv -source .venv/bin/activate +source .venv/Scripts/activate # (Windows) uv pip install -r requirements.txt +``` -# Set up pre-commit hooks +#### 3. Setup pre-commit hookes + +```bash pre-commit install +``` +#### 4. Setup environment variables + +```bash # Set up environment variables cp .env.example .env # Edit `.env` to configure database connection and app settings +``` + +#### 5. Build Docker images and start services -# Run database migrations -alembic upgrade head +Builds the images for the server, database, and app and starts the services. -# Start the development server -uvicorn app.main:app --reload +```bash +docker compose up --build # -d for silent/detached build ``` +Notes: +- To access the app run `docker exec -it nmsamplelocations-app-1 bash` +- Pytest can be run through the command line outside of the app. + ### ๐Ÿงญ Project Structure ```text app/ -โ”œโ”€โ”€ api/ # Route declarations -โ”œโ”€โ”€ core/ # Settings and application config -โ”œโ”€โ”€ db/ # Database models, sessions, migrations -โ”œโ”€โ”€ schemas/ # Pydantic data models -โ”œโ”€โ”€ services/ # Business logic and helpers -โ””โ”€โ”€ main.py # FastAPI entry point +โ”œโ”€โ”€ .env # Environment variables +โ”œโ”€โ”€ .pre-commit-config.yaml # pre-commit hook configuration file +โ”œโ”€โ”€ constants.py # Static variables used throughout the code +โ”œโ”€โ”€ docker-compose.yml # Docker compose file to build database and start server +โ”œโ”€โ”€ entrypoint.sh # Used by Docker to run database migrations and start server +โ”œโ”€โ”€ main.py # FastAPI entry point +| +โ”œโ”€โ”€ alembic/ # Alembic configuration and migration scripts +โ”œโ”€โ”€ api/ # Route declarations +โ”œโ”€โ”€ core/ # Settings and application config +โ”œโ”€โ”€ db/ # Database models, sessions, migrations +โ”œโ”€โ”€ docker/ # Custom Docker files +โ”œโ”€โ”€ migrations/ # Scripts to migrate data from NM_Aquifer to current db schema +โ”œโ”€โ”€ schemas/ # Pydantic data models +โ”œโ”€โ”€ services/ # Reusable database interactions +โ”œโ”€โ”€ services/ # Business logic and helpers +โ””โ”€โ”€ tests/ # Code tests ``` \ No newline at end of file From d36a17ff82874d37a3d649dbc30d84c17f09020b Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 14:20:16 -0600 Subject: [PATCH 02/13] docs: update README --- README.md | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index e4d51cab1..55d3b4dcc 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,11 @@ NMSampleLocations is a FastAPI-based backend service designed to manage geospati ### Prerequisites - Python 3.11+ -- PostgreSQL with PostGIS extension - [`uv`](https://github.com/astral-sh/uv) package manager -- Docker Desktop 4+ +- Docker Desktop 4+ if wanting to host server/database locally with containers +- PostgreSQL with PostGIS extension if wanting to host server/database locally without containers -### Installation & Development Setup +### Installation & Setup #### 1. Clone the repository @@ -67,22 +67,38 @@ pre-commit install #### 4. Setup environment variables ```bash -# Set up environment variables -cp .env.example .env # Edit `.env` to configure database connection and app settings +cp .env.example .env +``` + +#### 5. Database and server + +##### 5a. With PostgreSQL with PostGIS extension installed locally + +```bash +# Run database migrations +alembic upgrade head + +# Start the development server +uvicorn app.main:app --reload ``` -#### 5. Build Docker images and start services +Notes: +- Requires PostgreSQL and the PostGIS extensions to be installed locally. + +##### 5b. With Docker Builds the images for the server, database, and app and starts the services. ```bash -docker compose up --build # -d for silent/detached build +# include -d flag for silent/detached build +docker compose up --build ``` Notes: - To access the app run `docker exec -it nmsamplelocations-app-1 bash` - Pytest can be run through the command line outside of the app. +- Requires Docker Desktop be installed. ### ๐Ÿงญ Project Structure ```text From 0152bfd7bde5578363c3bc27b6fa40180b3d63a7 Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 14:38:35 -0600 Subject: [PATCH 03/13] refactor: make README options tables --- README.md | 47 ++++++++++------------------------------------- 1 file changed, 10 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 55d3b4dcc..b97e83f85 100644 --- a/README.md +++ b/README.md @@ -44,19 +44,11 @@ cd NMSampleLocations #### 2. Set up virtual environment and install depdencies -Mac/Linux -```bash -uv venv -source .venv/bin/activate # (Mac/Linux) -uv pip install -r requirements.txt -``` -Windows -```bash -uv venv -source .venv/Scripts/activate # (Windows) -uv pip install -r requirements.txt -``` +| Mac/Linux | Windows | +| --------- | ------- | +| 
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
| 
uv venv
source .venv/Scripts/activate
uv pip install -r requirements.txt
+ #### 3. Setup pre-commit hookes @@ -73,32 +65,13 @@ cp .env.example .env #### 5. Database and server -##### 5a. With PostgreSQL with PostGIS extension installed locally - -```bash -# Run database migrations -alembic upgrade head - -# Start the development server -uvicorn app.main:app --reload -``` - -Notes: -- Requires PostgreSQL and the PostGIS extensions to be installed locally. - -##### 5b. With Docker - -Builds the images for the server, database, and app and starts the services. - -```bash -# include -d flag for silent/detached build -docker compose up --build -``` +| PostgreSQL + PostGIS install locally | Docker| +| -------------------- | ----- | +|
#run database migrations
alembic upgrade head

# start development server
uvicorn app.main:app --reload
| 
# include -d flag for slient/detached build
docker compose up --build



| +| Requires PostgreSQL and PostGIS extensions to be installed locally | Requires Docker Desktop to be installed | +| | To access the app run `docker exec -it nmsamplelocations-app-1 bash` | +| | Pytest can be run through the command line outside of the app | -Notes: -- To access the app run `docker exec -it nmsamplelocations-app-1 bash` -- Pytest can be run through the command line outside of the app. -- Requires Docker Desktop be installed. ### ๐Ÿงญ Project Structure ```text From 4aaa2dbec31fe2ac6e4035adac42f9d7c1d2c38e Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 14:40:22 -0600 Subject: [PATCH 04/13] fix: correct typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b97e83f85..d4f8c41ed 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ cp .env.example .env | PostgreSQL + PostGIS install locally | Docker| | -------------------- | ----- | -|
#run database migrations
alembic upgrade head

# start development server
uvicorn app.main:app --reload
| 
# include -d flag for slient/detached build
docker compose up --build



| +|
#run database migrations
alembic upgrade head

# start development server
uvicorn app.main:app --reload
| 
# include -d flag for silent/detached build
docker compose up --build



| | Requires PostgreSQL and PostGIS extensions to be installed locally | Requires Docker Desktop to be installed | | | To access the app run `docker exec -it nmsamplelocations-app-1 bash` | | | Pytest can be run through the command line outside of the app | From e3bdaff9db89aaae3bbb49c8ba86076bd3949130 Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 15:25:49 -0600 Subject: [PATCH 05/13] refactor: update README from PR feedback --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d4f8c41ed..3cc395cad 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ app/ โ”œโ”€โ”€ alembic/ # Alembic configuration and migration scripts โ”œโ”€โ”€ api/ # Route declarations โ”œโ”€โ”€ core/ # Settings and application config -โ”œโ”€โ”€ db/ # Database models, sessions, migrations +โ”œโ”€โ”€ db/ # Database models, sessions, and engine โ”œโ”€โ”€ docker/ # Custom Docker files โ”œโ”€โ”€ migrations/ # Scripts to migrate data from NM_Aquifer to current db schema โ”œโ”€โ”€ schemas/ # Pydantic data models From 851d53ad64ec79d5fe03858028db75c65d6cc400 Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 15:28:24 -0600 Subject: [PATCH 06/13] refactor: update README --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 3cc395cad..50c0c5bdb 100644 --- a/README.md +++ b/README.md @@ -85,12 +85,11 @@ app/ | โ”œโ”€โ”€ alembic/ # Alembic configuration and migration scripts โ”œโ”€โ”€ api/ # Route declarations -โ”œโ”€โ”€ core/ # Settings and application config +โ”œโ”€โ”€ core/ # Settings, application config, and dependencies โ”œโ”€โ”€ db/ # Database models, sessions, and engine โ”œโ”€โ”€ docker/ # Custom Docker files โ”œโ”€โ”€ migrations/ # Scripts to migrate data from NM_Aquifer to current db schema โ”œโ”€โ”€ schemas/ # Pydantic data models -โ”œโ”€โ”€ services/ # Reusable database interactions -โ”œโ”€โ”€ services/ # Business logic and helpers +โ”œโ”€โ”€ services/ # Reusable business logic, helpers, and database interactions โ””โ”€โ”€ tests/ # Code tests ``` \ No newline at end of file From e9e48a4c5ba30eb44c1069fc82b051ea212d420f Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 15:40:04 -0600 Subject: [PATCH 07/13] refactor: rename migrations transfer for clarity --- README.md | 4 ++-- {migration => transfer}/data/location.csv | 0 {migration => transfer}/data/water_levels.csv | 0 {migration => transfer}/data/welldata.csv | 0 migration/migration.py => transfer/transfer.py | 1 - migration/migration2.py => transfer/transfer2.py | 2 +- 6 files changed, 3 insertions(+), 4 deletions(-) rename {migration => transfer}/data/location.csv (100%) rename {migration => transfer}/data/water_levels.csv (100%) rename {migration => transfer}/data/welldata.csv (100%) rename migration/migration.py => transfer/transfer.py (99%) rename migration/migration2.py => transfer/transfer2.py (99%) diff --git a/README.md b/README.md index 50c0c5bdb..04390bbe0 100644 --- a/README.md +++ b/README.md @@ -88,8 +88,8 @@ app/ โ”œโ”€โ”€ core/ # Settings, application config, and dependencies โ”œโ”€โ”€ db/ # Database models, sessions, and engine โ”œโ”€โ”€ docker/ # Custom Docker files -โ”œโ”€โ”€ migrations/ # Scripts to migrate data from NM_Aquifer to current db schema โ”œโ”€โ”€ schemas/ # Pydantic data models โ”œโ”€โ”€ services/ # Reusable business logic, helpers, and database interactions -โ””โ”€โ”€ tests/ # Code tests +โ”œโ”€โ”€ tests/ # Code tests +โ””โ”€โ”€ transfer/ # Scripts to transfer data from NM_Aquifer to current db schema ``` \ No newline at end of file diff --git a/migration/data/location.csv b/transfer/data/location.csv similarity index 100% rename from migration/data/location.csv rename to transfer/data/location.csv diff --git a/migration/data/water_levels.csv b/transfer/data/water_levels.csv similarity index 100% rename from migration/data/water_levels.csv rename to transfer/data/water_levels.csv diff --git a/migration/data/welldata.csv b/transfer/data/welldata.csv similarity index 100% rename from migration/data/welldata.csv rename to transfer/data/welldata.csv diff --git a/migration/migration.py b/transfer/transfer.py similarity index 99% rename from migration/migration.py rename to transfer/transfer.py index 82802157f..9c0741f41 100644 --- a/migration/migration.py +++ b/transfer/transfer.py @@ -20,7 +20,6 @@ import pyproj from shapely import Point from shapely.ops import transform -from sqlalchemy.exc import ProgrammingError from db import * from db.location import Location diff --git a/migration/migration2.py b/transfer/transfer2.py similarity index 99% rename from migration/migration2.py rename to transfer/transfer2.py index 2db797d0a..6f16863bf 100644 --- a/migration/migration2.py +++ b/transfer/transfer2.py @@ -111,7 +111,7 @@ def migrate_water_levels(session, limit=800): obs.series = series obs.observation_timestamp = datetime.fromisoformat(row.DateMeasured) # print("rw", row.DateMeasured, row.TimeMeasured) - gwl_obs = GroundwaterLevelObservation() + gwl_obs = Observation() gwl_obs.observation = obs gwl_obs.depth_to_water = row.DepthToWater gwl_obs.measuring_point_height = row.MPHeight From 04663875eec7a6d33d93da1b68a2c07f229ad581 Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Fri, 8 Aug 2025 15:42:24 -0600 Subject: [PATCH 08/13] refactor: call /transfer /transfers to keep consistent style --- README.md | 2 +- {transfer => transfers}/data/location.csv | 0 {transfer => transfers}/data/water_levels.csv | 0 {transfer => transfers}/data/welldata.csv | 0 {transfer => transfers}/transfer.py | 0 {transfer => transfers}/transfer2.py | 0 6 files changed, 1 insertion(+), 1 deletion(-) rename {transfer => transfers}/data/location.csv (100%) rename {transfer => transfers}/data/water_levels.csv (100%) rename {transfer => transfers}/data/welldata.csv (100%) rename {transfer => transfers}/transfer.py (100%) rename {transfer => transfers}/transfer2.py (100%) diff --git a/README.md b/README.md index 04390bbe0..06f0678a4 100644 --- a/README.md +++ b/README.md @@ -91,5 +91,5 @@ app/ โ”œโ”€โ”€ schemas/ # Pydantic data models โ”œโ”€โ”€ services/ # Reusable business logic, helpers, and database interactions โ”œโ”€โ”€ tests/ # Code tests -โ””โ”€โ”€ transfer/ # Scripts to transfer data from NM_Aquifer to current db schema +โ””โ”€โ”€ transfers/ # Scripts to transfer data from NM_Aquifer to current db schema ``` \ No newline at end of file diff --git a/transfer/data/location.csv b/transfers/data/location.csv similarity index 100% rename from transfer/data/location.csv rename to transfers/data/location.csv diff --git a/transfer/data/water_levels.csv b/transfers/data/water_levels.csv similarity index 100% rename from transfer/data/water_levels.csv rename to transfers/data/water_levels.csv diff --git a/transfer/data/welldata.csv b/transfers/data/welldata.csv similarity index 100% rename from transfer/data/welldata.csv rename to transfers/data/welldata.csv diff --git a/transfer/transfer.py b/transfers/transfer.py similarity index 100% rename from transfer/transfer.py rename to transfers/transfer.py diff --git a/transfer/transfer2.py b/transfers/transfer2.py similarity index 100% rename from transfer/transfer2.py rename to transfers/transfer2.py From 23e4929c8da299fa880daf1ca07bd5d7035bf01d Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Mon, 11 Aug 2025 09:39:54 -0600 Subject: [PATCH 09/13] fix: change
to
for rendering in GitHub --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 06f0678a4..8b7422457 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ cd NMSampleLocations | Mac/Linux | Windows | | --------- | ------- | -| 
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
| 
uv venv
source .venv/Scripts/activate
uv pip install -r requirements.txt
+| 
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
| 
uv venv
source .venv/Scripts/activate
uv pip install -r requirements.txt
#### 3. Setup pre-commit hookes @@ -67,7 +67,7 @@ cp .env.example .env | PostgreSQL + PostGIS install locally | Docker| | -------------------- | ----- | -|
#run database migrations
alembic upgrade head

# start development server
uvicorn app.main:app --reload
| 
# include -d flag for silent/detached build
docker compose up --build



| +|
#run database migrations
alembic upgrade head

# start development server
uvicorn app.main:app --reload
| 
# include -d flag for silent/detached build
docker compose up --build



| | Requires PostgreSQL and PostGIS extensions to be installed locally | Requires Docker Desktop to be installed | | | To access the app run `docker exec -it nmsamplelocations-app-1 bash` | | | Pytest can be run through the command line outside of the app | From 6885070781104dcd75f6bcd912daee2ae8475b3e Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Mon, 11 Aug 2025 09:41:38 -0600 Subject: [PATCH 10/13] refactor: update schemas documentation in README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8b7422457..add3922e6 100644 --- a/README.md +++ b/README.md @@ -88,8 +88,8 @@ app/ โ”œโ”€โ”€ core/ # Settings, application config, and dependencies โ”œโ”€โ”€ db/ # Database models, sessions, and engine โ”œโ”€โ”€ docker/ # Custom Docker files -โ”œโ”€โ”€ schemas/ # Pydantic data models +โ”œโ”€โ”€ schemas/ # Pydantic schemas and validations โ”œโ”€โ”€ services/ # Reusable business logic, helpers, and database interactions โ”œโ”€โ”€ tests/ # Code tests -โ””โ”€โ”€ transfers/ # Scripts to transfer data from NM_Aquifer to current db schema +โ””โ”€โ”€ transfers/ # Scripts to transfer data from NM_Aquifer to current db schema ``` \ No newline at end of file From 1f2ec93a9f257e9ed891770998f2069de663c720 Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Mon, 11 Aug 2025 10:21:34 -0600 Subject: [PATCH 11/13] fix: use HTML tables instead of markdown to display newlines in codeblocks --- README.md | 96 +++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 87 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index add3922e6..ba286b9ba 100644 --- a/README.md +++ b/README.md @@ -45,9 +45,36 @@ cd NMSampleLocations #### 2. Set up virtual environment and install depdencies -| Mac/Linux | Windows | -| --------- | ------- | -| 
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
| 
uv venv
source .venv/Scripts/activate
uv pip install -r requirements.txt
+ + + + + + + + + +
+ Mac/Linux + + Windows +
+ +```bash +uv venv +source .venv/bin/activate +uv sync --locked +``` + + + +```bash +uv venv +uv .venv/Scripts/activate +uv sync --locked +``` + +
#### 3. Setup pre-commit hookes @@ -65,12 +92,63 @@ cp .env.example .env #### 5. Database and server -| PostgreSQL + PostGIS install locally | Docker| -| -------------------- | ----- | -|
#run database migrations
alembic upgrade head

# start development server
uvicorn app.main:app --reload
| 
# include -d flag for silent/detached build
docker compose up --build



| -| Requires PostgreSQL and PostGIS extensions to be installed locally | Requires Docker Desktop to be installed | -| | To access the app run `docker exec -it nmsamplelocations-app-1 bash` | -| | Pytest can be run through the command line outside of the app | + + + + + + + + + + + + + + + + + + + + + + +
+ PostgreSQL + PostGIS installed locally + + Docker +
+ +```bash +#run database migrations +alembic upgrade head + +# start development server +uvicorn app.main:app --reload +``` + + + +```bash +# include -d flag for silent/detached build +docker compose up --build + + + +``` + +
+Requires PostgreSQL and PostGIS extensions to be installed locally + +Requires Docker Desktop to be installed locally +
+ +To access the app run `docker exec -it nmsamplelocations-app-1 bash` +
+ +Pytest can be run through the command line outside of the app +
### ๐Ÿงญ Project Structure From 24f5b4f3402e814cfe2244ba3932068e45455672 Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Mon, 11 Aug 2025 10:36:36 -0600 Subject: [PATCH 12/13] fix: udpate docker verbiage for correctness --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ba286b9ba..159c0e68c 100644 --- a/README.md +++ b/README.md @@ -138,14 +138,14 @@ Requires Docker Desktop to be installed locally -To access the app run `docker exec -it nmsamplelocations-app-1 bash` +Run docker exec -it nmsamplelocations-app-1 bash to open a shell inside the running app container. -Pytest can be run through the command line outside of the app +After the database container is running, you can run tests with Pytest from your local command line (not necessarily inside the app container). From a734a8cecb38155a896d4b86cbe53d063b356a0e Mon Sep 17 00:00:00 2001 From: Jacob Brown Date: Mon, 11 Aug 2025 10:38:02 -0600 Subject: [PATCH 13/13] fix: correct typo --- README.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/README.md b/README.md index 159c0e68c..c7efc102c 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ uv sync --locked ```bash uv venv -uv .venv/Scripts/activate +source .venv/Scripts/activate uv sync --locked ``` @@ -119,9 +119,6 @@ uvicorn app.main:app --reload ```bash # include -d flag for silent/detached build docker compose up --build - - - ```