Skip to content

feat: add event-driven cloud processing architecture - #5

Merged
cr-shah merged 1 commit into
mainfrom
feat/event-driven-cloud-pipeline
Sep 23, 2026
Merged

cr-shah merged 1 commit into
mainfrom
feat/event-driven-cloud-pipeline

Conversation

@cr-shah

@cr-shah cr-shah commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • add S3/local raw-event storage, Lambda ingestion, versioned SQS messages, retry and dead-letter handling
  • add an idempotent container worker that reuses existing validation, feature, scoring, and expected-loss logic
  • separate PostgreSQL operational state from Snowflake analytical history with incremental MERGE loading
  • add Prefect batch flows, Terraform infrastructure, secure configuration, structured logging, and extensive deployment documentation

Reliability and security

  • PostgreSQL event and transaction uniqueness prevents duplicate financial and analyst-review records
  • Snowflake MERGE makes analytical retries idempotent
  • failed SQS messages remain available for retry and DLQ routing
  • S3 versioning, encryption, public-access blocking, least-privilege IAM, ignored state/secrets, and Docker context exclusions are included

Verification

  • 73 tests passed with 66% aggregate coverage
  • Ruff lint and formatting passed
  • GitHub Actions CI passed
  • dependency lock and installed-package checks passed
  • Lambda deployment package built successfully
  • Terraform HCL and Docker Compose YAML parsed successfully
  • staged secret scan found no credentials
  • fresh 12,000-row demo generation matched checked-in artifacts byte for byte

@cr-shah cr-shah added the enhancement New feature or request label Sep 23, 2026
@cr-shah
cr-shah merged commit 1957887 into main Sep 23, 2026
2 checks passed

This branch was successfully deployed

1 active deployment
github-pages — 9a3feaf5 Deployed Sep 25, 2026 by cr-shah via deploy #2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant