Skip to content

Add quiltx cross-account bucket bootstrap flow#33

Open
drernie wants to merge 5 commits intomainfrom
codex/add-bucket-chokepoint
Open

Add quiltx cross-account bucket bootstrap flow#33
drernie wants to merge 5 commits intomainfrom
codex/add-bucket-chokepoint

Conversation

@drernie
Copy link
Copy Markdown
Member

@drernie drernie commented Apr 19, 2026

Role in the chokepoint

The bootstrap step that creates the named IAM role in the control account and installs the bucket-scoped inline policy, so that deployment#2372's narrowed templates have a role to reference. This branch also switches the bootstrap to consume the registry-issued ExternalId instead of generating one client-side, and now applies the full two-principal bucket policy contract. See meta#84.

Scope expanded. This branch now consumes the registry-managed ExternalId contract and the registry-surfaced Athena role ARN instead of treating the bucket policy as a single-principal concern. See the scope-update comment, meta/20-actually-finish-all.md § Progress, and meta/22g-two-principal-fix.md.

What this PR completes

  • New quiltx command to create/reuse QuiltDataAccessRole with an inline policy scoped to the named bucket (quiltx/bucket.py, quiltx/tools/bucket.py).
  • Bootstrap flow now reads the registry-issued ExternalId from the bucket registration / bucket config response (surfaced by enterprise#1037) and reuses it for the bootstrap-role trust policy instead of generating one client-side. First-time registration in a new external account and repeat registration in the same external account both converge on the registry-managed value.
  • Bucket add/update/query GraphQL calls now fetch athenaAccessRoleArn, and the final bucket/SNS policy principals are the deduped union of explicit --principal values, external_role_arn, and the stack-owned Athena role.
  • New bucket bootstrap does an initial apply, registers with the catalog, then re-applies the bucket/SNS policy when the returned Athena role ARN expands the principal set.
  • Already-registered cross-account buckets now idempotently re-apply the two-principal bucket/SNS policy on re-bootstrap instead of leaving the old single-principal policy in place.
  • Registers the bucket via the admin API with external_role_arn set.
  • configure_bucket_notifications still runs under bucket-owner credentials, unchanged.

Reference: proj/251208-add-bucket/10-cross-account-grants.md §role-scoped trust, proj/251208-add-bucket/20-actually-finish-all.md.

How to verify

  • Unit tests on the new role-bootstrap and bucket-registration code (tests/test_bucket.py) cover new registration, already-registered cross-account re-bootstrap, and SNS reuse/create flows under the delegated library path.
  • End-to-end: run quiltx against a fresh data account; confirm role is created, inline policy matches the bucket scope, ExternalId in the trust policy matches the value the registry returns, bucket registers successfully, and the resulting bucket policy names both the data-access role and QuiltAthenaAccessRole.
  • Repeat registration in the same external data account reuses the same registry-issued ExternalId and converges the bucket/SNS policies onto the two-principal shape.

Risk & rollback

Coupled to enterprise#1037 (registry is now the source of truth for ExternalId and athenaAccessRoleArn) and deployment#2372 (named role contract). Safe to revert alone — the bootstrap is idempotent and does not delete existing roles on rollback. However, deployment#2372's generated templates expect this role to exist, and the bootstrap now expects the registry to supply the bucket-policy principals; if this PR is reverted after the upstream PRs merge, new bucket registrations will fall back to the stale single-principal shape until re-applied.

drernie and others added 4 commits April 18, 2026 17:26
Read the registry-issued ExternalId from bucket registration / bucket
config responses and reuse it for bootstrap-role, instead of generating
it ad hoc client-side.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@drernie
Copy link
Copy Markdown
Member Author

drernie commented Apr 20, 2026

Scope update — follow-on work added to this branch

The Athena runtime rewiring and registry-side bucket-aware S3 plumbing that was previously planned as a separate follow-on stack has been added onto this same branch rather than split out. Per-repo progress is tracked in meta/251208-add-bucket/20-actually-finish-all.md § Progress; intentionally deferred items are in 21-deferred-work.md (legacy direct-Athena cleanup, deeper pkgpush/s3hash redesign, live stack/unstable validation).

CI should be re-run green against this expanded scope before re-review. See meta#84 for the combined commit request.

@drernie
Copy link
Copy Markdown
Member Author

drernie commented Apr 20, 2026

Testing status (this PR)

Pulled from the actual diff; see meta/21-deferred-work.md § Testing status for the project-wide picture.

Tested — unit tests updated or added on this branch:

  • tests/test_bucket.py — registry-issued ExternalId is read from bucket registration / bucket config responses and consumed by the bootstrap-role flow instead of being generated client-side.

Not covered by new tests in this PR — reasoned about, not validated:

  • None identified; the code changes here are narrowly exercised by the updated tests.

Deferred validation (not on this branch or any branch):

  • End-to-end registration against a fresh external account against stack/unstable.
  • Repeat registration in the same external account reusing the same ExternalId.
  • Confirmation that the final bucket-owner contract matches what the docs claim.

See 20-actually-finish-all.md § quiltx PR evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant