Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
# echo "POSTGRES_URL=$(az keyvault secret show --vault-name kv-hyf-data --name postgres-url --query value -o tsv)" >> .env
# echo "AZURE_STORAGE_CONNECTION_STRING=$(az keyvault secret show --vault-name kv-hyf-data --name storage-connection-string --query value -o tsv)" >> .env

POSTGRES_URL=postgresql://<user>:<password>@<server>.postgres.database.azure.com:5432/<db>?sslmode=require
AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net
export POSTGRES_URL=postgresql://<user>:<password>@<server>.postgres.database.azure.com:5432/<db>?sslmode=require
export AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net
# Your personal schema on the shared Postgres server — replace with your GitHub handle.
# All tables are created here so they don't collide with other students.
DB_SCHEMA=dev_<your_github_handle>
LOG_LEVEL=INFO
export DB_SCHEMA=dev_<your_github_handle>
export LOG_LEVEL=INFO
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,15 @@ uv run pytest tests/ -v

```bash
# Build for linux/amd64 (required by Azure Container Apps) and push to ACR
docker build --platform linux/amd64 -t hyfregistry.azurecr.io/my-pipeline:1.0 .
docker push hyfregistry.azurecr.io/my-pipeline:1.0
docker build --platform linux/amd64 -t hyfregistry.azurecr.io/my-pipeline:latest .
docker push hyfregistry.azurecr.io/my-pipeline:latest

# Create Container App Job (runs daily at 06:00 UTC)
az containerapp job create \
--name my-pipeline-job \
--resource-group rg-hyf-data \
--environment env-hyf-data \
--image hyfregistry.azurecr.io/my-pipeline:1.0 \
--image hyfregistry.azurecr.io/my-pipeline:latest \
--registry-server hyfregistry.azurecr.io \
--trigger-type Schedule \
--cron-expression "0 6 * * *" \
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ dependencies = [
"azure-storage-blob==12.25.1",
"requests==2.32.3",
"pandas==2.2.3",
"python-dotenv>=1.0",
]

[dependency-groups]
Expand Down
4 changes: 4 additions & 0 deletions src/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
import os
import sys

from dotenv import load_dotenv

load_dotenv()

import pandas as pd
from pydantic import ValidationError

Expand Down
11 changes: 11 additions & 0 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading