Provider directory bulk export example

README.md

@@ -45,6 +45,7 @@ A collection of examples on top of Aidbox FHIR platform
 - [Topic-Based Subscription to Kafka](aidbox-features/aidbox-subscriptions-to-kafka/)
 - [Aidbox with read-only replica](aidbox-features/aidbox-with-ro-replica/)
 - [AWS S3 Aidbox integration](aidbox-features/aws-s3-aidbox-integration/)
+- [Provider Directory Bulk Export](aidbox-features/provider-directory-bulk-export/)
 - [Init bundle with environment template](aidbox-features/init-bundle-env-template/)
 - [OpenTelemetry](aidbox-features/OpenTelemetry/)
 - [Organization-Based Access Control (OrgBAC) Practitioner Application](aidbox-features/orgbac-practitioner-application/)

aidbox-features/provider-directory-bulk-export/.env.example

@@ -0,0 +1 @@
+AIDBOX_LICENSE=<your-aidbox-license-jwt>

aidbox-features/provider-directory-bulk-export/.gitignore

@@ -0,0 +1,3 @@
+.env
+output/
+node_modules/

aidbox-features/provider-directory-bulk-export/README.md

@@ -0,0 +1,52 @@
+---
+features: [Bulk API, $export, Plan-Net, Provider Directory, S3, MinIO, Presigned URLs]
+languages: [TypeScript]
+---
+# Provider directory bulk export
+
+A minimal TypeScript provider-directory publishing pipeline on Aidbox. The flow:
+
+1. **Export**: kick off a system-level FHIR [`$export`](https://docs.aidbox.app/api/bulk-api/export) with `_typeFilter`, poll to completion, download the gzipped NDJSON.
+2. **Scope filter**: walk references from the kept parents and drop children nothing points at.
+3. **Bundle**: partition into FHIR collection `Bundle` files plus an `index.json` manifest.
+4. **Publish**: upload through Aidbox-presigned URLs to S3-compatible storage (local [MinIO](https://github.com/minio/minio) here). The script holds no bucket credentials.
+
+This is the shape behind CMS provider-directory feeds (Medicare Plan Finder): a crawler starts from `index.json` and downloads the bundles it lists.
+
+## Layout
+
+| File | Purpose |
+|---|---|
+| `docker-compose.yml` | Aidbox, PostgreSQL, MinIO (buckets auto-created), and an `export` service that runs the pipeline with [Bun](https://bun.sh). |
+| `init-bundle.json` | An `AwsAccount` for MinIO, the `provider-export` client with a least-privilege `AccessPolicy`, and a small Plan-Net dataset (one plan, two networks, practitioners in and out of network). |
+| `src/main.ts` | Scope config and orchestration of the four steps. |
+| `src/aidbox.ts` | Token mint, `$export` kickoff and polling, presigned uploads. |
+| `src/filter.ts` | Two-pass scope filter. |
+| `src/bundle.ts` | Bundle partitioning and `index.json`. |
+
+## Run
+
+1. Add an Aidbox license to `.env` (obtain one at [aidbox.app](https://aidbox.app)):
+
+   ```bash
+   cp .env.example .env
+   ```
+
+2. Start the stack. `curl http://localhost:8888/health` returns `200` when ready.
+
+   ```bash
+   docker compose up -d
+   ```
+
+3. Run the pipeline:
+
+   ```bash
+   # "export" is the compose service that runs `bun src/main.ts`. --rm drops the container after
+   docker compose run --rm export
+   ```
+
+   It prints progress and the manifest URL `http://localhost:9000/provider-directory-publish/index.json`. Bundles are also written to `./output`.
+
+## Adapting
+
+Scope IDs live in `src/main.ts`, sample data and the storage account in `init-bundle.json`. Swap MinIO for S3, GCP, or Azure through the `AwsAccount`.

aidbox-features/provider-directory-bulk-export/docker-compose.yml

@@ -0,0 +1,87 @@
+volumes:
+  aidbox_pg_data: {}
+  minio_data: {}
+
+services:
+  aidbox_db:
+    image: healthsamurai/aidboxdb:17
+    volumes:
+      - aidbox_pg_data:/var/lib/postgresql/data:delegated
+    environment:
+      POSTGRES_USER: aidbox
+      POSTGRES_PORT: "5432"
+      POSTGRES_DB: aidbox
+      POSTGRES_PASSWORD: jcRsTCRWqi
+
+  aidbox:
+    image: healthsamurai/aidboxone:edge
+    pull_policy: always
+    depends_on:
+      - aidbox_db
+      - createbucket
+    ports:
+      - 8888:8888
+    volumes:
+      - ./init-bundle.json:/init-bundle/init-bundle.json:ro
+    environment:
+      BOX_LICENSE: ${AIDBOX_LICENSE}
+      AIDBOX_PORT: 8888
+      AIDBOX_BASE_URL: http://aidbox:8888
+      AIDBOX_ADMIN_PASSWORD: PsCnpDmHZF
+      AIDBOX_CLIENT_SECRET: secret
+      PGHOST: aidbox_db
+      PGPORT: "5432"
+      PGUSER: aidbox
+      PGPASSWORD: jcRsTCRWqi
+      PGDATABASE: aidbox
+      AIDBOX_FHIR_PACKAGES: hl7.fhir.r4.core#4.0.1:hl7.fhir.us.davinci-pdex-plan-net#1.1.0
+      AIDBOX_FHIR_SCHEMA_VALIDATION: true
+      AIDBOX_CORRECT_AIDBOX_FORMAT: true
+      BOX_FHIR_TERMINOLOGY_ENGINE: hybrid
+      BOX_FHIR_TERMINOLOGY_ENGINE_HYBRID_EXTERNAL_TX_SERVER: https://tx.health-samurai.io/fhir
+      AIDBOX_COMPLIANCE: enabled
+      BOX_INIT_BUNDLE: file:///init-bundle/init-bundle.json
+      BOX_FHIR_BULK_STORAGE_PROVIDER: aws
+      BOX_FHIR_BULK_STORAGE_AWS_ACCOUNT: minio
+      BOX_FHIR_BULK_STORAGE_AWS_BUCKET: provider-directory-export
+
+  minio:
+    image: minio/minio:RELEASE.2025-04-22T22-12-26Z-cpuv1
+    command: server /data --console-address ":9001"
+    ports:
+      - "9000:9000"
+      - "9001:9001"
+    volumes:
+      - minio_data:/data
+    environment:
+      MINIO_ROOT_USER: minioadmin
+      MINIO_ROOT_PASSWORD: minioadmin
+
+  createbucket:
+    image: minio/mc:RELEASE.2025-04-16T18-13-26Z-cpuv1
+    depends_on:
+      - minio
+    entrypoint: >
+      /bin/sh -c "
+      until mc alias set local http://minio:9000 minioadmin minioadmin; do sleep 1; done;
+      mc mb --ignore-existing local/provider-directory-export;
+      mc mb --ignore-existing local/provider-directory-publish;
+      mc anonymous set download local/provider-directory-publish;
+      exit 0"
+
+  export:
+    image: oven/bun:1
+    profiles: ["export"]
+    depends_on:
+      - aidbox
+    working_dir: /app
+    volumes:
+      - .:/app
+    environment:
+      AIDBOX_URL: http://aidbox:8888
+      CLIENT_ID: provider-export
+      CLIENT_SECRET: provider-export-secret
+      PUBLISH_ACCOUNT: minio
+      PUBLISH_BUCKET: provider-directory-publish
+      PUBLIC_BASE: http://localhost:9000/provider-directory-publish
+    command: bun src/main.ts