From 2ca16f5af93224f0c57bb8bcafe24cc8d23a76ff Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 13:46:50 +0100 Subject: [PATCH 01/44] docs: record edge rebuild plan and gateway identity proposal --- AGENTS.md | 42 +++++ README.md | 12 ++ docs/decisions/0001-gateway-identity.md | 94 +++++++++++ docs/decisions/README.md | 12 ++ docs/plan.md | 200 ++++++++++++++++++++++++ 5 files changed, 360 insertions(+) create mode 100644 AGENTS.md create mode 100644 README.md create mode 100644 docs/decisions/0001-gateway-identity.md create mode 100644 docs/decisions/README.md create mode 100644 docs/plan.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..366a4a6 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,42 @@ +# Working agreement + +Build a new Go module from first principles. Beakley is reference material for +requirements and failure cases, not a source to copy and update. + +The planned root package is `edge`. Target Go 1.27 from the start and use +`encoding/json/v2` directly for JSON handled by this module. + +Read [the implementation plan](docs/plan.md) and relevant +[decision records](docs/decisions/README.md) before work. Preserve the distinction +between accepted direction, proposed API details, and implemented behavior. + +## Decisions and user review + +When a substantive question, design decision, or issue arises: + +1. Inspect our code and tests and consult the relevant official documentation. + Route AWS knowledge lookups through `aws-proxy-public`; do not use account + APIs for documentation research. Use official Go documentation and the + supplied Go skills when applicable. +2. Record the evidence, alternatives, recommendation, reasoning, and consequences + in a decision record. Label unverified assumptions and open questions. +3. Present the question or issue and the recommendation to the user for review. +4. Await the user's approval or redirection before implementing the dependent + behavior. Continue useful independent work within the accepted plan. +5. Record the user's resolution and update the plan before implementing it. + +Do not interpret silence as approval. Routine execution within an approved +decision does not require another approval. Do not silently settle public API, +trust-boundary, compatibility, or error-policy decisions. + +## Implementation and validation + +- Work on a working branch, not `main`. +- Keep transport, identity, authentication, and authorization responsibilities + distinct. Keep `identity` free of AWS dependencies. +- Preserve denied-by-default authorization and validate identity invariants. +- Test external contracts rather than mirroring implementation details. +- Use recorded/synthetic events and local test servers; do not deploy resources + or invoke live AWS services merely to run unit tests. +- Keep documentation and decision status accurate as work progresses. +- Do not copy Beakley source or tests into this repository. diff --git a/README.md b/README.md new file mode 100644 index 0000000..4834fbc --- /dev/null +++ b/README.md @@ -0,0 +1,12 @@ +# go-lambda-edge + +A planned Go 1.27 module for serving AWS API Gateway requests through ordinary +`net/http` handlers in AWS Lambda, with companion identity, authentication, and +authorization packages. + +The root package will be named `edge`. This project is being designed and built +from scratch. No runtime implementation is present yet. + +- [Implementation plan](docs/plan.md) +- [Design decisions](docs/decisions/README.md) +- [Repository working agreement](AGENTS.md) diff --git a/docs/decisions/0001-gateway-identity.md b/docs/decisions/0001-gateway-identity.md new file mode 100644 index 0000000..a31be62 --- /dev/null +++ b/docs/decisions/0001-gateway-identity.md @@ -0,0 +1,94 @@ +# 0001: Explicit gateway identity + +Status: proposed; awaiting user review of the concrete contract. + +## Context and user direction + +The user accepted the overall rebuild plan, then refined the initial review's +gateway-default recommendation: no gateway assertions by default feels more +natural, with a positive variadic option such as `WithGateway` to enable them. + +This record distinguishes receiving API Gateway traffic from using gateway +assertions to establish application identity. + +## Evidence + +- AWS documents that a JWT authorizer validates the token from its configured + identity source, then passes claims to the integration. The event's claims are + the asserted result; comparing selected claims does not authenticate a second + raw token from another source. + [JWT authorizer contract](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-jwt-authorizer.html) +- AWS documents IAM authorization as route configuration requiring SigV4 and + `execute-api` permission. Choosing whether this module consumes that assertion + does not alter the deployed route's authentication. + [IAM authorization](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control-iam.html) +- Beakley's `lambda/adapter.go` attaches an identity even when anonymous; + `authn/middleware.go` skips verification whenever the context slot is present. + That coupling prevents a local verifier from naturally following the adapter + on a route with no gateway-authenticated caller. +- Our repository currently has no runtime code. No existing API constrains the + new option name or default. + +## Recommendation for review + +Use the provisional variadic option `WithGatewayIdentity(enabled bool)` with a +default of false. The positive opt-in call reads `WithGatewayIdentity(true)`; +the boolean permits configuration helpers to explicitly disable it without a +second negatively named option. + +Proposed behavior: + +| Configuration | Result | +| --- | --- | +| No identity option | Translate the HTTP event; do not derive identity from gateway assertions, parse credentials, or fetch keys | +| `WithGatewayIdentity(true)` | Derive identity from supported native gateway JWT/Cognito and IAM assertions | +| Local bearer middleware explicitly configured | Verify the supplied token under caller-provided issuer/client policy | + +The option should cover both IAM and JWT/Cognito assertions. Custom-authorizer +contexts require an explicit mapper; their schema must not be guessed. + +The base adapter should not insert an anonymous identity solely to mark itself +as having run. Reading absent identity still yields the anonymous zero value, +and protected routes deny it. Invocation metadata can remain available separately. +This does not promise to preserve an identity from an unrelated invocation or +settle precedence between multiple configured producers. + +Gateway mode should use asserted event claims without supplementing permissions +from an unverified header token. It assumes the deployment restricts invocation +to the intended trusted integration. It does not independently verify SigV4 or +JWT signatures, discover an issuer, or configure AWS authorization. + +## Alternatives + +- `WithGateway`: concise, but suggests enabling the transport, which is already + the adapter's purpose. +- `NoGateway`: encodes a trust-default that the user's follow-up rejects. +- `WithGatewayIdentity()` without an argument: concise, but provides no direct + way for reusable option configuration to turn the policy back off. +- Automatically verify JWTs by default: cannot safely choose trusted issuers, + clients, audiences, and networking policy from untrusted request data. +- Opt in for JWT only while always consuming IAM: possible, but gives the same + gateway trust boundary two different defaults. + +## Consequences and remaining decisions + +- Applications must explicitly select how identity is established. An adapter + without authentication can serve public routes; it does not imply that all + routes are protected or that gateway authentication was disabled in AWS. +- Exact option mechanics, constructor validation, and error handling will be + reviewed with the adapter API. +- Local-versus-gateway precedence, re-verification, conflicting credentials, + and custom-authorizer mapping need further decisions before implementation. +- Claims unavailable at full fidelity from the gateway remain unavailable through + that source. Local verification is the path to trusted raw-token fidelity. + +## Review requested + +Approve or redirect the proposed name/shape and the policy that gateway identity +is opt-in for both IAM and JWT/Cognito, while the default performs only transport +adaptation and leaves authentication to explicit composition. + +## Resolution + +Pending. The user's default-direction preference is recorded above; the concrete +API and behavior in this proposal have not yet been approved. diff --git a/docs/decisions/README.md b/docs/decisions/README.md new file mode 100644 index 0000000..c707cb1 --- /dev/null +++ b/docs/decisions/README.md @@ -0,0 +1,12 @@ +# Design decisions + +Statuses: proposed, accepted, superseded, or rejected. A proposed record is not +authorization to implement the behavior it describes. + +Each substantive decision records the question, relevant code and official +sources, alternatives, recommendation, consequences, and the user's resolution. +Keep related decisions small enough to review and implement independently. + +| Record | Status | Question | +| --- | --- | --- | +| [0001: Gateway identity](0001-gateway-identity.md) | Proposed | How should gateway identity be enabled, and which assertions does it cover? | diff --git a/docs/plan.md b/docs/plan.md new file mode 100644 index 0000000..8605c47 --- /dev/null +++ b/docs/plan.md @@ -0,0 +1,200 @@ +# Edge implementation plan + +Status: accepted overall direction; individual contracts require review. + +## Objective + +Build `github.com/asteroid-computing/go-lambda-edge`, with root package `edge`, +as a new Go 1.27 module. Adapt AWS API Gateway Lambda proxy events to ordinary +`net/http` handlers and provide complementary identity, authentication, +authorization, and consumer-test packages. + +Beakley was reviewed at `/Volumes/home/Developer/spmt-20260905/beakley`. +Its requirements and observed failures inform this design. This work is not a +port, source copy, or compatibility-preserving update of Beakley. + +## Accepted direction + +- Set the module's Go baseline to 1.27.0. +- Use `encoding/json/v2` directly for our event, response, JWKS, and claim JSON + handling. Do not route our codec through `encoding/json`'s compatibility API. + Check third-party parser behavior separately during dependency selection. +- Keep ordinary `http.Handler` as the application boundary. +- Own event serialization at a raw Lambda invocation boundary implementing + `Invoke(context.Context, []byte) ([]byte, error)`. +- Support REST API proxy events, HTTP API payload 1.0, and HTTP API payload 2.0. + API product and payload version are distinct concepts. +- Start with buffered responses. Streaming is a separate future capability. +- Keep payload dispatch and the response writer private initially. +- Preserve separate IAM/JWT caller representations and an anonymous zero value; + enforce invariants rather than relying on comments about a union. +- Separate token scopes, Cognito groups, and application-resolved grants. +- Preserve context-aware authorization and distinguish denial from dependency + failure. +- Preserve full-fidelity claims when the selected source provides them. Do not + claim to recover arrays or original numeric values from flattened data. +- Research substantive decisions and bring recommendations to the user before + implementing dependent behavior. See `AGENTS.md`. + +The user's follow-up supersedes the review's initial gateway-default proposal: +prefer no gateway-derived identity by default, with an explicit positive option +to enable it. Exact option naming, IAM scope, and composition remain proposed in +[decision 0001](decisions/0001-gateway-identity.md). + +## Planned package boundaries + +| Package | Responsibility | +| --- | --- | +| `edge` (root) | Invocation adapter, payload decoding, HTTP translation, gateway metadata, optional gateway identity extraction | +| `identity` | Validated callers, claim representation, context transport; standard library only | +| `authn` | Bearer authentication, explicitly named Cognito verification, JWKS caching | +| `authz` | Decisions, rules, combinators, middleware, application-principal resolution | +| `edgetest` | Consumer identity fixtures and gateway event helpers | + +Avoid a broad root-package facade that reexports every companion type. Keep AWS +event types out of transport-independent packages. Reassess any additional +exported package against a concrete consumer need; version-reporting machinery +does not need to precede a working adapter. + +## Milestones and acceptance criteria + +### 0. Persist the design and working agreement + +- [x] Create a working branch from the initial `main` commit. +- [x] Record accepted direction and review findings. +- [x] Establish decision records and review workflow. +- [ ] Resolve decision 0001 before implementing authentication defaults. + +### 1. Specify the transport contracts + +Research and present a contract covering: + +- Raw invocation API, constructor/options, configuration validation, and errors. +- Discrimination between all three supported event cases; reject unsupported + versions and malformed shapes without fallback to a different format. +- Strict JSON v2 handling of duplicate names and invalid UTF-8, with tolerance + for additional AWS fields. +- V1 decoded paths versus V2 raw paths, escaped segments, query representation, + custom-domain mappings, and information that AWS does not preserve. +- Headers and multivalue precedence per key, cookies, host, protocol, + `RequestURI`, source address, body decoding/length/cleanup, and context lifetime. +- Status commitment, header snapshots, implicit responses, content-type + detection, HEAD/bodyless statuses, informational responses, trailers, and + accurately advertised optional `ResponseWriter` capabilities. +- Binary response selection and REST API deployment requirements; response-size + limits, including base64 and JSON-envelope overhead. +- Separation of HTTP application outcomes from invocation/translation faults. + +Acceptance: a reviewed contract and fixture matrix exist before the corresponding +behavior is implemented. Resolve smaller decisions incrementally rather than +presenting an entire frozen public API at once. + +### 2. Implement and prove the HTTP adapter + +- Create the Go module and minimal public adapter after its contract is reviewed. +- Implement the adapter from scratch using JSON v2 directly. +- Test raw invocation through the actual pinned AWS Lambda Go SDK boundary. +- Exercise documented event fixtures and adversarial/malformed input. +- Compare relevant request/response behavior with real `net/http` serving, + documenting unavoidable transport differences. +- Fuzz decoding and path/query conversion; add targeted allocation benchmarks. +- Run appropriate tests, race detection, vet, and Lambda-target build checks. + +Acceptance: the same application handler behaves as specified across all three +event cases; no auth or codec behavior depends on accidental SDK dispatch. + +### 3. Identity and gateway producers + +- Review caller constructors, immutable/owned data, validation, anonymous state, + and context-presence semantics. +- Preserve issuer plus subject for JWT application identity; do not treat a + bare subject as globally unique or assume every M2M caller has a user subject. +- Keep audience, app client, token use, and provenance distinct. +- Implement the approved opt-in gateway policy for native JWT/Cognito and IAM. +- Review custom-authorizer mapping for both payload families, including context, + error returns, provenance, and conflicts between identity sources. +- Expose only the gateway metadata handlers actually need, separately from + authenticated identity. Do not synthesize cryptographic guarantees from it. + +Acceptance: invalid/ambiguous identities cannot authorize; gateway and token +claims cannot be silently mixed; unavailable claim fidelity remains explicit. + +### 4. Cognito verification and JWKS resilience + +- Review a verifier requiring explicit issuer/client restrictions; accepting all + pool clients must be a deliberate policy. +- Keep access-token `client_id`, ID-token audience, and access-token resource + audience validation separate. Preserve access-token-only defaults unless + deliberately widened. +- Review algorithm/key selection, expiry/not-before/issued-at behavior, leeway, + malformed claims, and the chosen JWT dependency's JSON behavior. +- Separate cache freshness, unknown-key refresh, and failure backoff. +- Bound network duration and JWKS response size. Define stale-key behavior, + rotation, concurrency, waiter cancellation, and HTTP client ownership. +- Use deterministic cache/rotation/outage tests; consider Go 1.27 test networking + and `testing/synctest` where useful. + +Acceptance: malformed credentials fail closed, failures do not create fetch +storms, and dependency outages remain distinguishable from invalid credentials. + +### 5. Authorization and consumer ergonomics + +- Implement reviewed rules/combinators, scope/group/grant separation, principal + resolution, and explicit authentication precedence. +- Review missing/invalid credential responses (ordinarily 401), authenticated + denial (403), and dependency unavailability (503), including bearer challenges + and behavior on public routes. +- Define ARN matching precisely, including separators, partitions, account + boundaries, and STS role sessions. Do not approximate IAM policy evaluation. +- Validate nil/empty authorizers and invalid identities consistently. +- Provide consumer fixtures, runnable examples, and limited structured logging + that excludes raw credentials and unnecessary profile claims. + +Acceptance: examples work both in Lambda and ordinary HTTP applications, while +authorization decisions remain inspectable and deny by default. + +## Findings carried forward from the Beakley review + +These are requirements to test independently, not instructions to preserve old +implementation structure. + +1. Unverified header JWTs were reconciled using only `sub` and `client_id`. + A probe admitted altered groups with an invalid signature when modeling a + gateway/adapter identity-source mismatch. This is conditional on that mismatch, + not a demonstrated bypass of a correctly aligned native JWT authorizer. +2. Repeated final `WriteHeader` calls replaced the committed status, and header + changes after commitment appeared in output. Both reproduced. +3. V1 path `/a?b#c` was interpreted as path, query, and fragment rather than as + the supplied path. Reproduced. +4. V1 helpers discarded single-value maps wholesale when multivalue maps were + nonempty. Host lookup and request metadata also need defined semantics. +5. A caller with both IAM and JWT arms could satisfy both rule conditions. + Reproduced; validation was not consistently enforced. +6. Cold JWKS failures bypassed the refetch floor: three immediate calls produced + three network requests. Reproduced. Outage and cancellation behavior need + explicit contracts beyond the existing successful-cache tests. +7. An attached anonymous identity prevented subsequent bearer verification. + Existing tests codified that behavior; the new composition policy must be + deliberate rather than inherited. +8. `path.Match` permits wildcards across colons, contrary to the old ARN-matching + comment. Synthetic IAM-role examples are insufficient for role-session cases. +9. Client ID and audience were conflated, and verifier construction allowed any + client in the issuer's pool unless explicitly restricted. + +Validation performed during review: the old suite passed with `-race` in a +temporary copy after changing only its Go directive to 1.27. The original 1.26 +directive triggered Go 1.27's standard-library version checks for JSON v2. The +SDK serialization-boundary probe passed on Go 1.27. No live AWS integration was +deployed or exercised, and the Beakley repository was not edited. + +## Official references + +- [Go 1.27 release notes](https://go.dev/doc/go1.27) +- [JSON v2](https://pkg.go.dev/encoding/json/v2) +- [HTTP ResponseWriter](https://pkg.go.dev/net/http#ResponseWriter) +- [AWS Lambda Go Handler](https://pkg.go.dev/github.com/aws/aws-lambda-go/lambda#Handler) +- [API Gateway HTTP API payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) +- [Gateway JWT validation](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-jwt-authorizer.html) +- [Gateway IAM authorization](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control-iam.html) +- [Cognito resource binding](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-define-resource-servers.html) +- [REST API binary media](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings.html) From efd1ad7b4ea6b504b246e8ee193f491d03170732 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 13:51:00 +0100 Subject: [PATCH 02/44] docs: accept gateway identity policy and propose adapter boundary --- docs/decisions/0001-gateway-identity.md | 29 +++--- docs/decisions/0002-adapter-boundary.md | 122 ++++++++++++++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 15 ++- 4 files changed, 148 insertions(+), 21 deletions(-) create mode 100644 docs/decisions/0002-adapter-boundary.md diff --git a/docs/decisions/0001-gateway-identity.md b/docs/decisions/0001-gateway-identity.md index a31be62..0a2b37a 100644 --- a/docs/decisions/0001-gateway-identity.md +++ b/docs/decisions/0001-gateway-identity.md @@ -1,6 +1,6 @@ # 0001: Explicit gateway identity -Status: proposed; awaiting user review of the concrete contract. +Status: accepted by the user. ## Context and user direction @@ -29,14 +29,14 @@ assertions to establish application identity. - Our repository currently has no runtime code. No existing API constrains the new option name or default. -## Recommendation for review +## Decision -Use the provisional variadic option `WithGatewayIdentity(enabled bool)` with a +Use the variadic option `WithGatewayIdentity(enabled bool)` with a default of false. The positive opt-in call reads `WithGatewayIdentity(true)`; the boolean permits configuration helpers to explicitly disable it without a second negatively named option. -Proposed behavior: +Accepted behavior: | Configuration | Result | | --- | --- | @@ -44,16 +44,16 @@ Proposed behavior: | `WithGatewayIdentity(true)` | Derive identity from supported native gateway JWT/Cognito and IAM assertions | | Local bearer middleware explicitly configured | Verify the supplied token under caller-provided issuer/client policy | -The option should cover both IAM and JWT/Cognito assertions. Custom-authorizer +The option covers both IAM and JWT/Cognito assertions. Custom-authorizer contexts require an explicit mapper; their schema must not be guessed. -The base adapter should not insert an anonymous identity solely to mark itself +The base adapter must not insert an anonymous identity solely to mark itself as having run. Reading absent identity still yields the anonymous zero value, and protected routes deny it. Invocation metadata can remain available separately. This does not promise to preserve an identity from an unrelated invocation or settle precedence between multiple configured producers. -Gateway mode should use asserted event claims without supplementing permissions +Gateway mode uses asserted event claims without supplementing permissions from an unverified header token. It assumes the deployment restricts invocation to the intended trusted integration. It does not independently verify SigV4 or JWT signatures, discover an issuer, or configure AWS authorization. @@ -82,13 +82,12 @@ JWT signatures, discover an issuer, or configure AWS authorization. - Claims unavailable at full fidelity from the gateway remain unavailable through that source. Local verification is the path to trusted raw-token fidelity. -## Review requested - -Approve or redirect the proposed name/shape and the policy that gateway identity -is opt-in for both IAM and JWT/Cognito, while the default performs only transport -adaptation and leaves authentication to explicit composition. - ## Resolution -Pending. The user's default-direction preference is recorded above; the concrete -API and behavior in this proposal have not yet been approved. +The user replied "approved" to the concrete proposal: `WithGatewayIdentity(true)` +opts in to both IAM and JWT/Cognito assertions, the default performs transport +adaptation without establishing identity or fetching keys, and local verification +is explicitly composed. This approves the decision above, not the remaining +constructor, precedence, mapper, or error-policy decisions. + +Implementation status: not yet implemented. The transport API is the next review. diff --git a/docs/decisions/0002-adapter-boundary.md b/docs/decisions/0002-adapter-boundary.md new file mode 100644 index 0000000..5364359 --- /dev/null +++ b/docs/decisions/0002-adapter-boundary.md @@ -0,0 +1,122 @@ +# 0002: Adapter construction and invocation boundary + +Status: proposed; awaiting user review. + +## Question + +Should the adapter validate configuration at construction, and should failures to +translate an invocation return a Lambda error or a synthesized HTTP response? + +The raw `Invoke` boundary and variadic gateway-identity option are accepted +direction. Constructor errors, option application, and invocation-failure policy +still need decisions before implementation. + +## Evidence + +- The official AWS Lambda Go SDK defines `Handler` as + `Invoke(context.Context, []byte) ([]byte, error)`. Its v1.54.0 implementation + recognizes this interface before reflected function dispatch and passes raw + bytes directly. The ordinary function path decodes and encodes through + `encoding/json`. + [Handler interface](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.54.0/lambda#Handler) + [SDK handler source](https://github.com/aws/aws-lambda-go/blob/v1.54.0/lambda/handler.go) +- Consequently, the adapter object should be passed to `lambda.Start(adapter)`. + Passing `adapter.Invoke` as a function loses interface dispatch and subjects + `[]byte` parameters/results to ordinary JSON serialization semantics. +- AWS documents function errors as a generic API Gateway 502 response; invocation + requests rejected by the Lambda API produce 500. API Gateway does not retry + Lambda invocations. Returning a valid proxy response instead lets the function + control the HTTP status and body, and is a handled outcome rather than a Lambda + function error. + [API Gateway/Lambda error behavior](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway-errors.html) +- Beakley's adapter synthesizes a 500 response and returns a nil error for + extraction/conversion failures. This obscures the distinction between an + application HTTP response and failure of the adapter itself. +- Our repository currently contains plans and decision records only. There is + no implementation or compatibility commitment to preserve. + +The SDK version above was inspected as evidence, not selected as the new module's +dependency version. That selection and its checks belong to implementation setup. + +## Recommendation + +### Public shape + +Proposed signatures, not implemented API: + +```go +func New(handler http.Handler, opts ...Option) (*Adapter, error) +func WithGatewayIdentity(enabled bool) Option +func (a *Adapter) Invoke(ctx context.Context, payload []byte) ([]byte, error) +``` + +`New` validates the handler and configuration without I/O. Missing handlers, +invalid options, and invalid configuration return an error before the process +begins serving. Do not implicitly use `http.DefaultServeMux` or turn a missing +handler into a production fallback response. + +Options are applied in argument order, with the last assignment winning for +scalar settings such as gateway identity. Reject nil options. The adapter owns +its resulting configuration, exposes no mutating setters, and uses fresh request +state for each invocation. The supplied handler remains caller-owned and must be +safe for concurrent use if invocations are made concurrently. + +Application startup handles constructor failure in `main`, then calls +`lambda.Start(adapter)`. The library does not terminate the process. No `MustNew` +or separate library-owned start function is needed initially. + +### Failure ownership + +| Outcome | Proposed behavior | +| --- | --- | +| Invalid adapter configuration | `New` returns an error; startup caller decides how to report/terminate | +| Invalid JSON, unsupported event, or missing required envelope structure | `Invoke` returns an error without invoking the HTTP handler | +| Failure to convert the event, such as invalid base64 | `Invoke` returns an error without invoking the HTTP handler | +| Failure to encode a valid gateway response | `Invoke` returns an error; no partial payload is usable | +| Handler/middleware writes an HTTP response, including 4xx/5xx | `Invoke` returns the encoded response with nil error | + +In particular, malformed application JSON inside an otherwise valid request body +is the HTTP handler's concern. It is not an invalid Lambda event. The adapter +must not interpret application payloads to decide whether an invocation failed. + +Return contextual errors rather than logging and returning them. Error strings +must not include the raw event, authorization header, JWT, or application body. +An HTTP client normally receives the gateway's generic 502 for adapter failures; +operators retain the Lambda failure signal and diagnostic error. Direct callers +of `Invoke` also receive the error. + +This policy covers transport decoding/conversion/encoding failures. It does not +yet decide authentication outcomes, malformed authorizer contexts, context +cancellation timing, handler panics, unsupported response capabilities, or the +public sentinel/type taxonomy. Those require the relevant contracts. + +## Alternatives and tradeoffs + +- Return only `*Adapter` from `New`: a shorter startup line, but configuration + errors must become panics or surface during an invocation. An explicit error + keeps validation at the point the caller still controls startup. +- Return synthesized HTTP 500 responses for transport faults: gives clients a + controlled response body and status, but records the Lambda invocation as + handled. Before classifying the event, the correct response format is also + unknown. Prefer a consistent error path for adapter failures. +- Return Lambda errors for every HTTP 4xx/5xx: loses the handler's intended + response and misclassifies ordinary application outcomes as invocation faults. +- Export event unions or typed proxy methods now: adds public compatibility + obligations without improving the ordinary handler or Lambda startup path. + +## Validation after approval + +- Constructor cases: absent handler, nil option, invalid configuration, explicit + gateway enable/disable, and duplicate scalar options. +- SDK-boundary tests pass an adapter object through `lambda.NewHandler(...).Invoke` + and verify object-shaped responses without base64/double encoding. +- Malformed/unsupported envelopes never reach the application handler. +- Application HTTP error statuses and bodies survive the round trip with nil + invocation error; transport faults return errors with no sensitive payloads. + +These tests will follow the reviewed payload and response contracts; this +decision does not authorize inventing those contracts during implementation. + +## Resolution + +Pending user approval or redirection. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index c707cb1..9dc2b2f 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -9,4 +9,5 @@ Keep related decisions small enough to review and implement independently. | Record | Status | Question | | --- | --- | --- | -| [0001: Gateway identity](0001-gateway-identity.md) | Proposed | How should gateway identity be enabled, and which assertions does it cover? | +| [0001: Gateway identity](0001-gateway-identity.md) | Accepted | How should gateway identity be enabled, and which assertions does it cover? | +| [0002: Adapter boundary](0002-adapter-boundary.md) | Proposed | How should construction fail, and which failures become Lambda invocation errors? | diff --git a/docs/plan.md b/docs/plan.md index 8605c47..ec43f3f 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -36,10 +36,11 @@ port, source copy, or compatibility-preserving update of Beakley. - Research substantive decisions and bring recommendations to the user before implementing dependent behavior. See `AGENTS.md`. -The user's follow-up supersedes the review's initial gateway-default proposal: -prefer no gateway-derived identity by default, with an explicit positive option -to enable it. Exact option naming, IAM scope, and composition remain proposed in -[decision 0001](decisions/0001-gateway-identity.md). +The user approved [decision 0001](decisions/0001-gateway-identity.md), superseding +the review's initial gateway-default proposal: no gateway-derived identity by +default; `WithGatewayIdentity(true)` opts in for both IAM and JWT/Cognito. Local +verification is explicitly composed. Constructor details and precedence between +multiple identity producers remain separate decisions. ## Planned package boundaries @@ -63,10 +64,14 @@ does not need to precede a working adapter. - [x] Create a working branch from the initial `main` commit. - [x] Record accepted direction and review findings. - [x] Establish decision records and review workflow. -- [ ] Resolve decision 0001 before implementing authentication defaults. +- [x] Resolve decision 0001 before implementing authentication defaults. ### 1. Specify the transport contracts +Current review: [decision 0002](decisions/0002-adapter-boundary.md) proposes +construction, option application, and invocation-failure handling. It is pending; +the gateway-identity approval does not settle these contracts. + Research and present a contract covering: - Raw invocation API, constructor/options, configuration validation, and errors. From e3a0125881d55fa34c17066108f63574aa673f23 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 14:03:22 +0100 Subject: [PATCH 03/44] feat(edge): add validated adapter construction --- README.md | 8 +- adapter.go | 70 ++++++++++++ adapter_test.go | 44 +++++++ docs/decisions/0001-gateway-identity.md | 4 +- docs/decisions/0002-adapter-boundary.md | 30 +++-- docs/decisions/0003-event-decoding.md | 145 ++++++++++++++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 14 ++- go.mod | 3 + 9 files changed, 303 insertions(+), 18 deletions(-) create mode 100644 adapter.go create mode 100644 adapter_test.go create mode 100644 docs/decisions/0003-event-decoding.md create mode 100644 go.mod diff --git a/README.md b/README.md index 4834fbc..2507529 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ # go-lambda-edge -A planned Go 1.27 module for serving AWS API Gateway requests through ordinary +A Go 1.27 module being built to serve AWS API Gateway requests through ordinary `net/http` handlers in AWS Lambda, with companion identity, authentication, and authorization packages. -The root package will be named `edge`. This project is being designed and built -from scratch. No runtime implementation is present yet. +The root package is `edge`. This project is being designed and built from scratch. +The constructor and options foundation is implemented. Event decoding, invocation, +HTTP translation, and identity extraction are not implemented yet; the adapter +cannot yet be registered as a Lambda handler. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) diff --git a/adapter.go b/adapter.go new file mode 100644 index 0000000..3f2620f --- /dev/null +++ b/adapter.go @@ -0,0 +1,70 @@ +// Package edge provides the configuration foundation for adapting AWS API Gateway +// events to net/http handlers. Invocation and HTTP translation are not yet +// implemented. +package edge + +import ( + "errors" + "fmt" + "net/http" + "reflect" +) + +// Adapter holds an HTTP handler and its API Gateway adaptation configuration. +// Construct an Adapter with [New]; its zero value is not usable. +// Configuration is fixed at construction. Invocation is not yet implemented. +type Adapter struct { + handler http.Handler + config config +} + +type config struct { + gatewayIdentity bool +} + +// Option configures an [Adapter] during construction. +// Use the option functions provided by this package. A nil Option is invalid. +// Options apply in argument order; the last assignment to a scalar setting wins. +type Option func(*config) + +// WithGatewayIdentity configures whether gateway assertions establish identity. +// The default is false. When enabled, the policy covers native IAM and +// JWT/Cognito assertions; it does not configure authorization in AWS or enable +// local token verification. Custom-authorizer contexts need an explicit mapper. +// This option currently records policy; invocation and extraction are not yet +// implemented. +func WithGatewayIdentity(enabled bool) Option { + return func(c *config) { + c.gatewayIdentity = enabled + } +} + +// New validates handler and opts and returns an Adapter without performing I/O +// or invoking handler. Nil handlers, including typed nil handlers, and nil +// options are rejected. On error, the returned Adapter is nil. +// +// The caller retains ownership of handler and is responsible for its concurrency +// safety when sharing it. New does not use [http.DefaultServeMux]. +func New(handler http.Handler, opts ...Option) (*Adapter, error) { + if handler == nil { + return nil, errors.New("edge: nil HTTP handler") + } + // An interface containing a typed nil is non-nil. Check nil-capable kinds + // before accepting a handler that would defer a startup fault to invocation. + v := reflect.ValueOf(handler) + switch v.Kind() { + case reflect.Chan, reflect.Func, reflect.Map, reflect.Pointer, reflect.Slice: + if v.IsNil() { + return nil, errors.New("edge: nil HTTP handler") + } + } + + var cfg config + for i, opt := range opts { + if opt == nil { + return nil, fmt.Errorf("edge: nil option at index %d", i) + } + opt(&cfg) + } + return &Adapter{handler: handler, config: cfg}, nil +} diff --git a/adapter_test.go b/adapter_test.go new file mode 100644 index 0000000..9d4412f --- /dev/null +++ b/adapter_test.go @@ -0,0 +1,44 @@ +package edge_test + +import ( + "net/http" + "testing" + + "github.com/asteroid-computing/go-lambda-edge" +) + +func TestNewRejectsNilHandlers(t *testing.T) { + var mux *http.ServeMux + for _, tt := range []struct { + name string + handler http.Handler + }{ + {name: "nil_interface"}, + {name: "nil_function", handler: http.HandlerFunc(nil)}, + {name: "nil_pointer", handler: mux}, + } { + t.Run(tt.name, func(t *testing.T) { + adapter, err := edge.New(tt.handler) + if err == nil || adapter != nil { + t.Errorf("New(%T) = (%v, %v), want (nil, error)", tt.handler, adapter, err) + } + }) + } +} + +func TestNewRejectsNilOption(t *testing.T) { + adapter, err := edge.New(http.NewServeMux(), edge.WithGatewayIdentity(true), nil) + if err == nil || adapter != nil { + t.Errorf("New(mux, option, nil) = (%v, %v), want (nil, error)", adapter, err) + } +} + +func TestNewDoesNotInvokeHandler(t *testing.T) { + handler := http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + t.Fatal("New invoked the HTTP handler") + }) + adapter, err := edge.New(handler, edge.WithGatewayIdentity(true)) + if err != nil || adapter == nil { + t.Errorf("New(handler, option) = (%v, %v), want (non-nil, nil)", adapter, err) + } +} diff --git a/docs/decisions/0001-gateway-identity.md b/docs/decisions/0001-gateway-identity.md index 0a2b37a..97a50ed 100644 --- a/docs/decisions/0001-gateway-identity.md +++ b/docs/decisions/0001-gateway-identity.md @@ -90,4 +90,6 @@ adaptation without establishing identity or fetching keys, and local verificatio is explicitly composed. This approves the decision above, not the remaining constructor, precedence, mapper, or error-policy decisions. -Implementation status: not yet implemented. The transport API is the next review. +Implementation status: `WithGatewayIdentity` now records the approved setting in +constructor-owned configuration. Invocation and identity extraction are not yet +implemented; the option cannot establish identity until those stages are built. diff --git a/docs/decisions/0002-adapter-boundary.md b/docs/decisions/0002-adapter-boundary.md index 5364359..20aa658 100644 --- a/docs/decisions/0002-adapter-boundary.md +++ b/docs/decisions/0002-adapter-boundary.md @@ -1,15 +1,15 @@ # 0002: Adapter construction and invocation boundary -Status: proposed; awaiting user review. +Status: accepted by the user. ## Question Should the adapter validate configuration at construction, and should failures to translate an invocation return a Lambda error or a synthesized HTTP response? -The raw `Invoke` boundary and variadic gateway-identity option are accepted -direction. Constructor errors, option application, and invocation-failure policy -still need decisions before implementation. +This decision settles constructor errors, option application, and transport +invocation-failure policy alongside the previously accepted raw `Invoke` boundary +and variadic gateway-identity option. ## Evidence @@ -38,11 +38,11 @@ still need decisions before implementation. The SDK version above was inspected as evidence, not selected as the new module's dependency version. That selection and its checks belong to implementation setup. -## Recommendation +## Decision ### Public shape -Proposed signatures, not implemented API: +Accepted signatures (implementation status is recorded below): ```go func New(handler http.Handler, opts ...Option) (*Adapter, error) @@ -67,7 +67,7 @@ or separate library-owned start function is needed initially. ### Failure ownership -| Outcome | Proposed behavior | +| Outcome | Accepted behavior | | --- | --- | | Invalid adapter configuration | `New` returns an error; startup caller decides how to report/terminate | | Invalid JSON, unsupported event, or missing required envelope structure | `Invoke` returns an error without invoking the HTTP handler | @@ -104,7 +104,7 @@ public sentinel/type taxonomy. Those require the relevant contracts. - Export event unions or typed proxy methods now: adds public compatibility obligations without improving the ordinary handler or Lambda startup path. -## Validation after approval +## Validation - Constructor cases: absent handler, nil option, invalid configuration, explicit gateway enable/disable, and duplicate scalar options. @@ -119,4 +119,16 @@ decision does not authorize inventing those contracts during implementation. ## Resolution -Pending user approval or redirection. +The user replied "approved" to the constructor and failure-handling contract: +`New` returns `(*Adapter, error)`, options apply in order with last scalar assignment +winning, the adapter object is passed to `lambda.Start`, transport failures return +invocation errors, and handler HTTP responses retain their status/body with nil +invocation error. + +Implementation status: constructor/options foundation is implemented with no +external dependencies. Tests cover nil/typed-nil handlers, nil options, and +construction without handler invocation. `go test -race ./...`, `go vet ./...`, +and Linux arm64/amd64 package builds pass on Go 1.27.1; formatting is clean. + +`Invoke`, identity behavior, duplicate-option behavior as observed during an +invocation, and integration tests follow the remaining payload/HTTP contracts. diff --git a/docs/decisions/0003-event-decoding.md b/docs/decisions/0003-event-decoding.md new file mode 100644 index 0000000..a1d8de9 --- /dev/null +++ b/docs/decisions/0003-event-decoding.md @@ -0,0 +1,145 @@ +# 0003: Private wire types and explicit event decoding + +Status: proposed; awaiting user review. + +## Question + +Which event shapes does the decoder accept, and should our JSON representation +use the AWS SDK event structs or private types owned by this module? + +## Evidence + +- AWS's REST proxy example has `httpMethod`, `path`, and `requestContext`, with + an API ID in the context and no top-level version discriminator. Its examples + also show null optional fields. + [REST proxy input](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) + [AWS HTTP request example](https://aws.amazon.com/blogs/developer/handling-arbitrary-http-requests-in-amazon-api-gateway/) +- HTTP APIs support payload versions `1.0` and `2.0`. V1 carries method/path at + the top level; V2 uses `requestContext.http.method` and `rawPath`. The versions + differ in headers, queries, cookies, and custom-domain mapping visibility. + [HTTP API payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) +- The inspected AWS Go SDK v1.54.0 uses non-pointer scalar event fields and + `map[string]string` for V2 authorizer JWT claims. Those types are convenient + consumers of the wire format, but alone cannot distinguish an omitted field + from a zero value or preserve a non-string claim value should one arrive. + [SDK event types](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.54.0/events#APIGatewayV2HTTPRequest) + [SDK JWT claims](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.54.0/events#APIGatewayV2HTTPRequestContextAuthorizerJWTDescription) +- JSON v2 requires a single JSON value, rejects invalid UTF-8 and duplicate + object names by default, and matches struct member names case-sensitively. + JSON null generally decodes as a Go zero value; unmarshaling into `any` decodes + numbers as float64. Presence/type validation therefore needs explicit handling. + [JSON v2 documentation](https://pkg.go.dev/encoding/json/v2) + [Unmarshal](https://pkg.go.dev/encoding/json/v2#Unmarshal) +- Lambda Function URLs use the same request/response schema as API Gateway V2. + Shape validation cannot authenticate the invoking service. + [Function URL payloads](https://docs.aws.amazon.com/lambda/latest/dg/urls-invocation.html) +- Our implementation currently contains only the constructor/options foundation. + No existing decoder or wire schema constrains this choice. The accepted plan + already requires direct JSON v2 and rejection rather than V2-to-V1 fallback. + +## Recommendation + +### Wire representation + +Use private, purpose-built request/response types, decoded and encoded directly +with `encoding/json/v2`. Model only fields needed by the reviewed transport and +identity contracts. Keep presence information where validation needs it. + +Carry authorizer data internally as `jsontext.Value` until the selected identity +producer interprets it. Preserve exactly the JSON representation AWS supplies; +this does not assert that AWS supplies full-fidelity JWT claims or authorize +guessing an array from a flattened string. Avoid float64 conversion of arbitrary +claim numbers at the transport boundary. + +The AWS Go SDK remains the runtime integration in consuming applications and a +compatibility-test target. Its event structs need not control our internal +decoder or become exported API. Dependency-version selection remains separate. + +### Version dispatch and minimum structure + +| Discriminator | Canonical method/path | Interpretation | +| --- | --- | --- | +| `version` absent | `httpMethod`, `path` | REST proxy envelope | +| `version` exactly `"1.0"` | `httpMethod`, `path` | HTTP API payload 1.0 envelope | +| `version` exactly `"2.0"` | `requestContext.http.method`, `rawPath` | Payload 2.0 envelope | +| Any other present value, including null, empty string, or a number | None | Reject | + +For every accepted envelope require a non-null object `requestContext`, a +nonempty string `requestContext.apiId`, and the canonical method/path fields as +nonempty strings. V2 also requires the `requestContext.http` object. These are +proposed library validity checks based on the documented envelopes, not a claim +that AWS publishes a formal required-field schema for all variants. + +Do not fall back to a different format after a decode/validation error. Do not +invent `GET` or `/` when a canonical method/path is missing. V2's reported HTTP +context path does not replace a missing `rawPath`. Method syntax and exact path +escaping are part of the subsequent HTTP translation contract. + +The API ID helps reject unrelated event shapes such as ordinary ALB envelopes; +it is not proof of origin. We do not promise Function URL support merely because +a V2-shaped event passes validation, nor reject one based on guessed hostnames or +API ID spelling. Gateway identity remains the explicit deployment trust decision +accepted in decision 0001. + +### Strictness and forward compatibility + +- Accept one top-level JSON object. Reject null, arrays, scalars, invalid UTF-8, + duplicate object names, and trailing non-whitespace content. +- Use exact AWS member names and require correct types for consumed fields. + Do not enable legacy case-insensitive matching or coercions. +- Permit unknown members for forward compatibility. Unknown data still must be + valid JSON; this does not relax syntax checking inside authorizer data. +- Handle documented absent/null optional fields intentionally. In particular, + absent/null body means empty bytes; absent/null header and query maps mean no + entries. A missing `isBase64Encoded` means false; a present value must be a JSON + boolean. Further fields will be specified when their consumers are designed. +- Preserve opaque authorizer data during transport decoding. Do not establish + identity, interpret claim semantics, or validate a custom-authorizer schema + when no identity producer was selected. +- Return failures through the invocation-error contract accepted in decision + 0002. Do not include the input payload or credential values in errors. + +### Fixture matrix + +| Case | Expected result | +| --- | --- | +| Documented REST envelope, no version | Select REST/V1 translation | +| HTTP API `1.0` envelope | Select HTTP API/V1 translation | +| HTTP API `2.0` envelope | Select V2 translation | +| Future version, null/empty/non-string version | Reject without fallback | +| Malformed V2 plus otherwise usable V1 fields | Reject V2; never reinterpret as V1 | +| Missing/null/wrong-type context, API ID, method, or canonical path | Reject | +| Unrelated SQS/S3/ALB events; empty object | Reject | +| Extra unknown fields | Accept if the selected envelope remains valid | +| Duplicate names, including in nested opaque data; invalid UTF-8 | Reject | +| Multiple JSON values or non-object root | Reject | +| Null optional body/maps | Accept as absent/empty | +| Opaque authorizer string, array, object, or numeric claims | Preserve wire values pending identity interpretation | +| Method/body carrying application-invalid data | Leave application validation to the appropriate later layer | + +Write fixtures from documented contracts and focused synthetic cases, not by +copying Beakley's tests. Include an SDK-generated representative event as an +additional compatibility check rather than treating SDK serialization as the +sole oracle. Run fuzzing against the decoder after implementation. + +## Alternatives and tradeoffs + +- Reuse SDK event types directly: less schema code to maintain, but additional + presence validation and opaque-data extraction are still needed. A hybrid + decoder would then carry two representations of important parts of the input. +- Use `map[string]any` throughout: flexible but scatters type assertions and + exposes numeric precision loss. Keep flexibility only where the schema is + genuinely opaque. +- Guess a version from whichever decode succeeds: accepts malformed events in + the wrong format and contradicts the approved rejection policy. +- Reject all unknown fields: catches misspellings, but makes additive AWS fields + a breaking change for deployed applications. + +Private types create schema-maintenance work. Limit it to fields we consume, +link their source documentation, and maintain independent wire fixtures so the +types do not become a self-confirming specification. + +## Resolution + +Pending user approval or redirection. No decoder is implemented under this +proposal yet. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 9dc2b2f..71dbab1 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -10,4 +10,5 @@ Keep related decisions small enough to review and implement independently. | Record | Status | Question | | --- | --- | --- | | [0001: Gateway identity](0001-gateway-identity.md) | Accepted | How should gateway identity be enabled, and which assertions does it cover? | -| [0002: Adapter boundary](0002-adapter-boundary.md) | Proposed | How should construction fail, and which failures become Lambda invocation errors? | +| [0002: Adapter boundary](0002-adapter-boundary.md) | Accepted | How should construction fail, and which failures become Lambda invocation errors? | +| [0003: Event decoding](0003-event-decoding.md) | Proposed | Which event shapes are accepted, and should the wire types be private? | diff --git a/docs/plan.md b/docs/plan.md index ec43f3f..d0d9441 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -68,9 +68,14 @@ does not need to precede a working adapter. ### 1. Specify the transport contracts -Current review: [decision 0002](decisions/0002-adapter-boundary.md) proposes -construction, option application, and invocation-failure handling. It is pending; -the gateway-identity approval does not settle these contracts. +Accepted: [decision 0002](decisions/0002-adapter-boundary.md) settles construction, +option application, and transport invocation-failure handling. Its independent +constructor/options foundation is implemented while the event/HTTP contracts are +reviewed. Constructor tests pass with race detection, vet and formatting checks +pass, and package builds pass for Linux arm64 and amd64. + +Current review: [decision 0003](decisions/0003-event-decoding.md) proposes private +wire types, version dispatch, structural validation, and a decoder fixture matrix. Research and present a contract covering: @@ -96,7 +101,8 @@ presenting an entire frozen public API at once. ### 2. Implement and prove the HTTP adapter -- Create the Go module and minimal public adapter after its contract is reviewed. +- [x] Create the Go 1.27 module and constructor/options foundation after review. +- [ ] Implement raw invocation after the remaining transport contracts are reviewed. - Implement the adapter from scratch using JSON v2 directly. - Test raw invocation through the actual pinned AWS Lambda Go SDK boundary. - Exercise documented event fixtures and adversarial/malformed input. diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..85057cf --- /dev/null +++ b/go.mod @@ -0,0 +1,3 @@ +module github.com/asteroid-computing/go-lambda-edge + +go 1.27.0 From fc65284e2a6d94385290c337caff0054f70a7f67 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 14:39:15 +0100 Subject: [PATCH 04/44] docs: accept typed event entry points and AWS type reuse --- docs/decisions/0002-adapter-boundary.md | 5 +- docs/decisions/0003-event-decoding.md | 74 ++++++++++++++----- docs/decisions/0004-typed-events.md | 97 +++++++++++++++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 14 +++- 5 files changed, 170 insertions(+), 23 deletions(-) create mode 100644 docs/decisions/0004-typed-events.md diff --git a/docs/decisions/0002-adapter-boundary.md b/docs/decisions/0002-adapter-boundary.md index 20aa658..d1af293 100644 --- a/docs/decisions/0002-adapter-boundary.md +++ b/docs/decisions/0002-adapter-boundary.md @@ -101,8 +101,9 @@ public sentinel/type taxonomy. Those require the relevant contracts. unknown. Prefer a consistent error path for adapter failures. - Return Lambda errors for every HTTP 4xx/5xx: loses the handler's intended response and misclassifies ordinary application outcomes as invocation faults. -- Export event unions or typed proxy methods now: adds public compatibility - obligations without improving the ordinary handler or Lambda startup path. +- Export event unions or typed proxy methods at the time of this decision: + deferred pending a concrete consumer need. The user's subsequent request and + acceptance of [decision 0004](0004-typed-events.md) add HandleV1 and HandleV2. ## Validation diff --git a/docs/decisions/0003-event-decoding.md b/docs/decisions/0003-event-decoding.md index a1d8de9..d673178 100644 --- a/docs/decisions/0003-event-decoding.md +++ b/docs/decisions/0003-event-decoding.md @@ -1,6 +1,6 @@ -# 0003: Private wire types and explicit event decoding +# 0003: AWS event types and explicit event decoding -Status: proposed; awaiting user review. +Status: proposed; AWS type reuse accepted, detailed decoding policy pending. ## Question @@ -41,19 +41,36 @@ use the AWS SDK event structs or private types owned by this module? ### Wire representation -Use private, purpose-built request/response types, decoded and encoded directly -with `encoding/json/v2`. Model only fields needed by the reviewed transport and -identity contracts. Keep presence information where validation needs it. +Reuse AWS Lambda Go request/response structs wherever they meet the reviewed +contract, decoded and encoded directly with `encoding/json/v2`. The user accepted +this direction after questioning the original private-schema proposal. SDK type +reuse does not require using the SDK's JSON decoder. Keep a small explicit +version discriminator and targeted presence/type validation where needed; +do not duplicate complete AWS schemas just to select the event format. -Carry authorizer data internally as `jsontext.Value` until the selected identity -producer interprets it. Preserve exactly the JSON representation AWS supplies; +Proposed exception: carry authorizer data internally as `jsontext.Value` until +the selected identity producer interprets it. Preserve exactly the JSON +representation AWS supplies; this does not assert that AWS supplies full-fidelity JWT claims or authorize guessing an array from a flattened string. Avoid float64 conversion of arbitrary claim numbers at the transport boundary. -The AWS Go SDK remains the runtime integration in consuming applications and a -compatibility-test target. Its event structs need not control our internal -decoder or become exported API. Dependency-version selection remains separate. +Authorizer extraction and SDK decoding must be designed together so an SDK claim +field cannot prematurely reject data intended to remain opaque. The exact +exception mechanism remains pending; authorizer fidelity alone does not justify +private copies of the entire envelope. + +The AWS Lambda Go library remains the runtime integration and supplies event +types. Its reflected function path decodes into the handler's declared parameter +type; an `any` parameter receives generic JSON values, not automatically selected +AWS event structs. Our raw boundary selects the version before decoding the +concrete event. Dependency-version selection remains separate. +[SDK handler implementation](https://github.com/aws/aws-lambda-go/blob/v1.54.0/lambda/handler.go) +[AWS Go handler documentation](https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html) + +The user also requested a route for already typed events that avoids JSON work +inside edge. See [decision 0004](0004-typed-events.md) for the accepted signatures +and the limits of validation after another component has decoded the payload. ### Version dispatch and minimum structure @@ -81,6 +98,22 @@ a V2-shaped event passes validation, nor reject one based on guessed hostnames o API ID spelling. Gateway identity remains the explicit deployment trust decision accepted in decision 0001. +### Already typed events + +The selected method determines the payload family without automatic dispatch. +Proposed semantic checks still require nonempty API ID, method, and canonical +path. HandleV2 accepts Version equal to "2.0" or empty: an omitted Go string +does not make a caller-created V2 struct ambiguous, because its type and method +already select V2. Reject any other nonempty Version as contradictory input. +HandleV1 has no version field to inspect. Neither method can validate original +JSON shape, member presence, or syntax. Do not serialize structs to imitate +raw-wire validation. + +This is a proposed ergonomic exception for explicitly typed input, not a change +to raw Invoke: a present empty version in JSON remains invalid there. Matching +raw and typed valid events share translation; their available validation evidence +differs as accepted in decision 0004. + ### Strictness and forward compatibility - Accept one top-level JSON object. Reject null, arrays, scalars, invalid UTF-8, @@ -116,6 +149,9 @@ accepted in decision 0001. | Null optional body/maps | Accept as absent/empty | | Opaque authorizer string, array, object, or numeric claims | Preserve wire values pending identity interpretation | | Method/body carrying application-invalid data | Leave application validation to the appropriate later layer | +| Typed V2 with empty or "2.0" Version and valid required values | Select V2 directly | +| Typed V2 with other nonempty Version | Reject contradictory input | +| Typed V1/V2 missing API ID, canonical method, or path | Reject before HTTP handling | Write fixtures from documented contracts and focused synthetic cases, not by copying Beakley's tests. Include an SDK-generated representative event as an @@ -124,9 +160,8 @@ sole oracle. Run fuzzing against the decoder after implementation. ## Alternatives and tradeoffs -- Reuse SDK event types directly: less schema code to maintain, but additional - presence validation and opaque-data extraction are still needed. A hybrid - decoder would then carry two representations of important parts of the input. +- Private copies of all event types: more schema maintenance than needed for + automatic dispatch. Prefer SDK types with narrowly justified exceptions. - Use `map[string]any` throughout: flexible but scatters type assertions and exposes numeric precision loss. Keep flexibility only where the schema is genuinely opaque. @@ -135,11 +170,14 @@ sole oracle. Run fuzzing against the decoder after implementation. - Reject all unknown fields: catches misspellings, but makes additive AWS fields a breaking change for deployed applications. -Private types create schema-maintenance work. Limit it to fields we consume, -link their source documentation, and maintain independent wire fixtures so the -types do not become a self-confirming specification. +SDK structs alone do not provide the proposed presence checks or opaque-claim +contract. Limit supplementary representations to those needs and maintain +independent wire fixtures so SDK serialization is not the sole specification. ## Resolution -Pending user approval or redirection. No decoder is implemented under this -proposal yet. +The user accepted AWS type reuse and requested typed-event entry points. This +supersedes the original recommendation for entirely private wire types. The +detailed version/structure/strictness rules above remain proposed; agreement to +type reuse does not approve every earlier validation rule. No decoder is +implemented yet. diff --git a/docs/decisions/0004-typed-events.md b/docs/decisions/0004-typed-events.md new file mode 100644 index 0000000..c8adb88 --- /dev/null +++ b/docs/decisions/0004-typed-events.md @@ -0,0 +1,97 @@ +# 0004: Typed event entry points + +Status: accepted by the user; not implemented yet. + +## Question + +How should callers with an AWS event struct bypass edge's JSON envelope work, +while retaining the same HTTP and gateway-identity behavior? + +## Evidence + +- The user requested an option to skip JSON when the caller already has typed + events and accepted reuse of AWS event structs in decision 0003. +- Our Adapter currently only holds a handler and immutable configuration. There + is no invocation implementation to preserve or duplicate. +- AWS permits a handler with context and a JSON-compatible input, returning a + JSON-compatible response and error. Its runtime chooses the input type from + the function signature and owns decoding/encoding for that function. + [AWS handler signatures](https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html) + [SDK implementation inspected at v1.54.0](https://github.com/aws/aws-lambda-go/blob/v1.54.0/lambda/handler.go) +- AWS supplies matching request and response types for both payload families. + V1 represents REST proxy and HTTP API payload 1.0; the V1 request struct has no + version discriminator. The V2 JWT claims representation is map[string]string. + [AWS event types](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.54.0/events) + +## Decision + +Expose two methods on the same configured Adapter: + +```go +func (a *Adapter) HandleV1(ctx context.Context, event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) +func (a *Adapter) HandleV2(ctx context.Context, event events.APIGatewayV2HTTPRequest) (events.APIGatewayV2HTTPResponse, error) +``` + +The user approved these names, exact signatures, and the boundary contract below. +They are not implemented yet; shared HTTP translation still needs review. + +The ordinary raw registration remains lambda.Start(adapter). A caller that +chooses a fixed payload family can register lambda.Start(adapter.HandleV1) or +lambda.Start(adapter.HandleV2), or pass an already decoded struct directly. +These examples describe the planned API, not working functionality today. + +Use shared internal HTTP translation, response capture, and identity policy for +both raw and typed entry points. Do not marshal a typed event and call Invoke; +do not encode a response just to decode it into an AWS struct. Typed methods +skip envelope decoding, automatic format selection, and response serialization +inside edge. They still perform HTTP conversion, including base64 body handling. + +All JSON handled by edge continues to use encoding/json/v2 directly. Registering +a typed function with the inspected SDK delegates envelope decoding/encoding to +the SDK's encoding/json path outside edge. Applications requiring direct v2 +control of the Lambda envelope should use the raw Adapter registration. + +Keep semantic validation and transport-error behavior consistent where the +typed values retain the necessary information. On a conversion error, return a +zero response and a non-nil error. Ordinary application HTTP errors remain +responses with nil Go error, following decision 0002. + +Typed values cannot prove that original JSON had unique keys, exact spelling, +correct field presence, or preserved arbitrary numbers. In particular, V1 has +no Version field to validate. Do not promise raw-wire validation equivalence or +recover values lost by a caller's decoder. Identity extraction must respect the +fidelity of the supplied source and must never supplement it with unverified +header JWT claims. Detailed authorizer representations remain under review. + +## Alternatives and tradeoffs + +- A WithSkipJSON boolean cannot change a Go method's parameter or result type; + a distinct callable boundary is needed for typed input and output. +- One any-based method with a type switch can accept both event types but loses + the compile-time request/response pairing. Registering that method directly + with Lambda would receive generic JSON values instead of AWS structs. +- Separate constructors or adapter types could encode the same distinction but + add configuration surface without a demonstrated need. + +AWS event types become a deliberate public dependency of edge. Keep them out of +transport-independent identity/authentication/authorization packages. Future SDK +upgrades must preserve the typed API and pass boundary compatibility tests. + +## Validation plan + +- Compile and exercise each typed method through the pinned SDK handler wrapper. +- Compare raw and typed HTTP results using independently constructed equivalent + events for both formats, including cookies, multivalue headers, binary bodies, + application error responses, and conversion failures. +- Verify reviewed gateway-identity defaults and opt-in behavior on both paths. +- Test and document cases where original wire information is unavailable after + upstream decoding, without claiming identical raw-wire rejection guarantees. + +## Resolution + +The user approved the proposed methods and boundary contract. This extends the +earlier decision 0002 deferral of typed proxy methods in response to a concrete +consumer requirement; its raw invocation contract remains accepted. Record the +remaining decoding and HTTP translation decisions before implementing behavior +that depends on them. Do not add nonfunctional public method stubs merely to +claim the accepted API is implemented. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 71dbab1..3e13e70 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -11,4 +11,5 @@ Keep related decisions small enough to review and implement independently. | --- | --- | --- | | [0001: Gateway identity](0001-gateway-identity.md) | Accepted | How should gateway identity be enabled, and which assertions does it cover? | | [0002: Adapter boundary](0002-adapter-boundary.md) | Accepted | How should construction fail, and which failures become Lambda invocation errors? | -| [0003: Event decoding](0003-event-decoding.md) | Proposed | Which event shapes are accepted, and should the wire types be private? | +| [0003: Event decoding](0003-event-decoding.md) | Proposed; AWS type reuse accepted | How should SDK types, dispatch, and wire validation work together? | +| [0004: Typed events](0004-typed-events.md) | Accepted | How can already typed events bypass edge's envelope JSON work? | diff --git a/docs/plan.md b/docs/plan.md index d0d9441..f89e377 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -22,6 +22,11 @@ port, source copy, or compatibility-preserving update of Beakley. - Keep ordinary `http.Handler` as the application boundary. - Own event serialization at a raw Lambda invocation boundary implementing `Invoke(context.Context, []byte) ([]byte, error)`. +- Reuse AWS Lambda Go event/response structs where they meet the contract; + reserve custom representations for demonstrated limitations. +- Support already typed events without an envelope JSON round trip inside edge. + Decision 0004 accepts HandleV1 and HandleV2 with matching AWS request/response + types, sharing translation and identity policy with raw invocation. - Support REST API proxy events, HTTP API payload 1.0, and HTTP API payload 2.0. API product and payload version are distinct concepts. - Start with buffered responses. Streaming is a separate future capability. @@ -74,8 +79,11 @@ constructor/options foundation is implemented while the event/HTTP contracts are reviewed. Constructor tests pass with race detection, vet and formatting checks pass, and package builds pass for Linux arm64 and amd64. -Current review: [decision 0003](decisions/0003-event-decoding.md) proposes private -wire types, version dispatch, structural validation, and a decoder fixture matrix. +Current review: [decision 0003](decisions/0003-event-decoding.md) records accepted +AWS type reuse and proposes version dispatch, structural validation, and a decoder +fixture matrix. [Decision 0004](decisions/0004-typed-events.md) accepts typed +methods that skip envelope JSON work inside edge. These entry points await the +remaining decoding and shared HTTP translation contracts before implementation. Research and present a contract covering: @@ -103,6 +111,8 @@ presenting an entire frozen public API at once. - [x] Create the Go 1.27 module and constructor/options foundation after review. - [ ] Implement raw invocation after the remaining transport contracts are reviewed. +- [ ] Implement the accepted typed entry points after shared HTTP translation + review, documenting upstream codec ownership. - Implement the adapter from scratch using JSON v2 directly. - Test raw invocation through the actual pinned AWS Lambda Go SDK boundary. - Exercise documented event fixtures and adversarial/malformed input. From 40d9181f6b72b8fc4c3deab508d2a64c3f3ee32d Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 14:53:23 +0100 Subject: [PATCH 05/44] feat(edge): decode gateway events with JSON v2 and AWS types --- README.md | 7 +- decode.go | 135 ++++++++++ decode_test.go | 243 ++++++++++++++++++ docs/decisions/0003-event-decoding.md | 42 ++- .../0005-http-request-translation.md | 162 ++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 15 +- go.mod | 2 + go.sum | 10 + testdata/events/README.md | 16 ++ testdata/events/http-v1.json | 22 ++ testdata/events/http-v2.json | 20 ++ testdata/events/rest.json | 19 ++ 13 files changed, 674 insertions(+), 22 deletions(-) create mode 100644 decode.go create mode 100644 decode_test.go create mode 100644 docs/decisions/0005-http-request-translation.md create mode 100644 go.sum create mode 100644 testdata/events/README.md create mode 100644 testdata/events/http-v1.json create mode 100644 testdata/events/http-v2.json create mode 100644 testdata/events/rest.json diff --git a/README.md b/README.md index 2507529..5a241eb 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ A Go 1.27 module being built to serve AWS API Gateway requests through ordinary authorization packages. The root package is `edge`. This project is being designed and built from scratch. -The constructor and options foundation is implemented. Event decoding, invocation, -HTTP translation, and identity extraction are not implemented yet; the adapter -cannot yet be registered as a Lambda handler. +The constructor, options, and private event decoder are implemented. The decoder +uses AWS Lambda Go v1.55.0 event structs and encoding/json/v2 directly, retaining +authorizer JSON separately. Invocation, HTTP translation, and identity extraction +are not implemented yet; the adapter cannot yet be registered as a Lambda handler. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) diff --git a/decode.go b/decode.go new file mode 100644 index 0000000..681411a --- /dev/null +++ b/decode.go @@ -0,0 +1,135 @@ +package edge + +import ( + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + + "github.com/aws/aws-lambda-go/events" +) + +// decodedEvent contains exactly one request on success. Authorizer JSON belongs +// to this value, independently of the input buffer; it is not decoded into the +// SDK's string/float64 claim representations. No identity is established here. +type decodedEvent struct { + version string + v1 *events.APIGatewayProxyRequest + v2 *events.APIGatewayV2HTTPRequest + authorizer jsontext.Value +} + +func decodeEvent(payload []byte) (decodedEvent, error) { + // This small first pass validates all JSON syntax, including unknown and + // opaque fields, and retains presence where SDK scalar zero values cannot. + var envelope *struct { + Version jsontext.Value `json:"version"` + IsBase64Encoded jsontext.Value `json:"isBase64Encoded"` + } + if err := json.Unmarshal(payload, &envelope); err != nil { + // JSON errors can contain object names and input values. Do not expose + // them, even through an unwrap chain, at this credential-bearing boundary. + return decodedEvent{}, errors.New("edge: invalid event JSON") + } + if envelope == nil { + return decodedEvent{}, errors.New("edge: event must be a JSON object") + } + var version string + if len(envelope.Version) != 0 { + if err := json.Unmarshal(envelope.Version, &version); err != nil { + return decodedEvent{}, errors.New("edge: invalid payload version") + } + if version != "1.0" && version != "2.0" { + return decodedEvent{}, errors.New("edge: unsupported payload version") + } + } + if len(envelope.IsBase64Encoded) != 0 && envelope.IsBase64Encoded.Kind() != 't' && envelope.IsBase64Encoded.Kind() != 'f' { + return decodedEvent{}, errors.New("edge: isBase64Encoded must be a boolean") + } + + if version == "2.0" { + var wire struct { + events.APIGatewayV2HTTPRequest + RequestContext struct { + events.APIGatewayV2HTTPRequestContext + Authorizer jsontext.Value `json:"authorizer"` + } `json:"requestContext"` + } + if err := json.Unmarshal(payload, &wire, eventStringOptions); err != nil { + return decodedEvent{}, errors.New("edge: invalid v2 event fields") + } + event := wire.APIGatewayV2HTTPRequest + event.RequestContext = wire.RequestContext.APIGatewayV2HTTPRequestContext + if err := validateV2(event); err != nil { + return decodedEvent{}, err + } + return decodedEvent{version: version, v2: &event, authorizer: wire.RequestContext.Authorizer}, nil + } + + var wire struct { + events.APIGatewayProxyRequest + RequestContext struct { + events.APIGatewayProxyRequestContext + Authorizer jsontext.Value `json:"authorizer"` + } `json:"requestContext"` + } + if err := json.Unmarshal(payload, &wire, eventStringOptions); err != nil { + return decodedEvent{}, errors.New("edge: invalid v1 event fields") + } + event := wire.APIGatewayProxyRequest + event.RequestContext = wire.RequestContext.APIGatewayProxyRequestContext + if err := validateV1(event); err != nil { + return decodedEvent{}, err + } + return decodedEvent{version: version, v1: &event, authorizer: wire.RequestContext.Authorizer}, nil +} + +func validateV1(event events.APIGatewayProxyRequest) error { + if event.RequestContext.APIID == "" { + return errors.New("edge: missing requestContext.apiId") + } + if event.HTTPMethod == "" { + return errors.New("edge: missing httpMethod") + } + if event.Path == "" { + return errors.New("edge: missing path") + } + return nil +} + +func validateV2(event events.APIGatewayV2HTTPRequest) error { + if event.Version != "" && event.Version != "2.0" { + return errors.New("edge: conflicting v2 payload version") + } + if event.RequestContext.APIID == "" { + return errors.New("edge: missing requestContext.apiId") + } + if event.RequestContext.HTTP.Method == "" { + return errors.New("edge: missing requestContext.http.method") + } + if event.RawPath == "" { + return errors.New("edge: missing rawPath") + } + return nil +} + +// Optional collections may be null, but their string elements must be strings. +// Scope this rule to collections: AWS also sends null optional scalar metadata +// and body fields. Options and unmarshaler lists are immutable after creation. +var eventStringOptions = json.WithUnmarshalers(json.JoinUnmarshalers( + json.UnmarshalFunc(func(data []byte, out *map[string]string) error { + return json.Unmarshal(data, out, nonNullStringOptions) + }), + json.UnmarshalFunc(func(data []byte, out *map[string][]string) error { + return json.Unmarshal(data, out, nonNullStringOptions) + }), + json.UnmarshalFunc(func(data []byte, out *[]string) error { + return json.Unmarshal(data, out, nonNullStringOptions) + }), +)) + +var nonNullStringOptions = json.WithUnmarshalers(json.UnmarshalFunc(func(data []byte, out *string) error { + if jsontext.Value(data).Kind() != '"' { + return errors.New("expected a JSON string") + } + return json.Unmarshal(data, out) +})) diff --git a/decode_test.go b/decode_test.go new file mode 100644 index 0000000..6ac3356 --- /dev/null +++ b/decode_test.go @@ -0,0 +1,243 @@ +package edge + +import ( + "bytes" + "encoding/json/jsontext" + "encoding/json/v2" + "os" + "slices" + "strings" + "testing" + + "github.com/aws/aws-lambda-go/events" +) + +// Decoder tests exercise the accepted wire contract directly while the public +// Invoke and typed HTTP entry points await their translation contract. +const minimalV1 = `{"httpMethod":"GET","path":"/","requestContext":{"apiId":"example"}}` +const minimalV2 = `{"version":"2.0","rawPath":"/","requestContext":{"apiId":"example","http":{"method":"GET"}}}` + +func TestDecodeDocumentedFixtures(t *testing.T) { + for _, tt := range []struct { + file string + version string + path string + method string + body string + }{ + {file: "rest", path: "/a?b#c", method: "POST", body: "{application-invalid-json"}, + {file: "http-v1", version: "1.0", path: "/my/path", method: "GET"}, + {file: "http-v2", version: "2.0", path: "/my/%2Fpath", method: "POST", body: "AP8="}, + } { + t.Run(tt.file, func(t *testing.T) { + payload, err := os.ReadFile("testdata/events/" + tt.file + ".json") + if err != nil { + t.Fatal(err) + } + got, err := decodeEvent(payload) + if err != nil { + t.Fatalf("decodeEvent(%s) = %v", tt.file, err) + } + if got.version != tt.version { + t.Errorf("version = %q, want %q", got.version, tt.version) + } + var path, method, body string + if tt.version == "2.0" { + if got.v2 == nil || got.v1 != nil { + t.Fatal("decodeEvent(v2) did not select exactly the v2 event") + } + path, method, body = got.v2.RawPath, got.v2.RequestContext.HTTP.Method, got.v2.Body + if got.v2.RawQueryString != "parameter1=value1¶meter1=value2&literal=%2C" || !slices.Equal(got.v2.Cookies, []string{"a=1", "b=2"}) || !got.v2.IsBase64Encoded { + t.Error("decodeEvent(v2) changed raw query, cookies, or binary flag") + } + if got.v2.RequestContext.Authorizer != nil { + t.Error("decodeEvent(v2) prematurely interpreted authorizer JSON") + } + } else { + if got.v1 == nil || got.v2 != nil { + t.Fatal("decodeEvent(v1) did not select exactly the v1 event") + } + path, method, body = got.v1.Path, got.v1.HTTPMethod, got.v1.Body + if got.v1.RequestContext.Authorizer != nil { + t.Error("decodeEvent(v1) prematurely interpreted authorizer JSON") + } + } + if path != tt.path || method != tt.method || body != tt.body { + t.Errorf("path/method/body = (%q, %q, %q), want (%q, %q, %q)", path, method, body, tt.path, tt.method, tt.body) + } + }) + } +} + +func TestDecodeRejectsInvalidEnvelopes(t *testing.T) { + for name, payload := range map[string]string{ + "empty": "", + "null": "null", + "array": "[]", + "string": `"event"`, + "number": "12", + "empty_object": "{}", + "trailing_value": minimalV1 + " {}", + "trailing_garbage": minimalV1 + " broken", + "sqs": `{"Records":[{"eventSource":"aws:sqs"}]}`, + "alb": `{"httpMethod":"GET","path":"/","requestContext":{"elb":{}}}`, + "null_context": strings.Replace(minimalV1, `{"apiId":"example"}`, "null", 1), + "array_context": strings.Replace(minimalV1, `{"apiId":"example"}`, "[]", 1), + "missing_context": `{"httpMethod":"GET","path":"/"}`, + "empty_api_id": strings.Replace(minimalV1, `"example"`, `""`, 1), + "null_api_id": strings.Replace(minimalV1, `"example"`, "null", 1), + "numeric_api_id": strings.Replace(minimalV1, `"example"`, "1", 1), + "missing_method": strings.Replace(minimalV1, `"httpMethod":"GET",`, "", 1), + "null_method": strings.Replace(minimalV1, `"GET"`, "null", 1), + "empty_method": strings.Replace(minimalV1, `"GET"`, `""`, 1), + "wrong_case_method": strings.Replace(minimalV1, "httpMethod", "HTTPMethod", 1), + "null_path": strings.Replace(minimalV1, `"path":"/"`, `"path":null`, 1), + "missing_path": strings.Replace(minimalV1, `"path":"/",`, "", 1), + "null_version": strings.Replace(minimalV2, `"2.0"`, "null", 1), + "numeric_version": strings.Replace(minimalV2, `"2.0"`, "2.0", 1), + "empty_version": strings.Replace(minimalV2, `"2.0"`, `""`, 1), + "future_version": strings.Replace(minimalV2, `"2.0"`, `"3.0"`, 1), + "v2_missing_http": strings.Replace(minimalV2, `,"http":{"method":"GET"}`, "", 1), + "v2_null_http": strings.Replace(minimalV2, `{"method":"GET"}`, "null", 1), + "v2_missing_raw_path": strings.Replace(minimalV2, `"rawPath":"/",`, "", 1), + "v2_no_path_fallback": strings.Replace(minimalV2, `"rawPath":"/"`, `"path":"/","httpMethod":"GET"`, 1), + "duplicate_method": strings.Replace(minimalV1, `"GET"`, `"GET","httpMethod":"POST"`, 1), + "escaped_duplicate": strings.Replace(minimalV1, `"GET"`, `"GET","http\u004dethod":"POST"`, 1), + "unknown_duplicate": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","future":{"x":1,"x":2}`, 1), + "null_binary_flag": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","isBase64Encoded":null`, 1), + "string_binary_flag": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","isBase64Encoded":"true"`, 1), + "numeric_body": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","body":1`, 1), + "array_headers": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","headers":[]`, 1), + "null_header_value": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","headers":{"x":null}`, 1), + "null_multivalue_item": strings.Replace(minimalV1, `"path":"/"`, `"path":"/","multiValueHeaders":{"x":[null]}`, 1), + "null_cookie_item": strings.Replace(minimalV2, `"rawPath":"/"`, `"rawPath":"/","cookies":[null]`, 1), + "invalid_utf8": strings.Replace(minimalV1, "example", "\xff", 1), + "authorizer_duplicate": strings.Replace(minimalV1, `"apiId":"example"`, `"apiId":"example","authorizer":{"secret":1,"secret":2}`, 1), + "authorizer_bad_utf8": strings.Replace(minimalV1, `"apiId":"example"`, "\"apiId\":\"example\",\"authorizer\":\"\xff\"", 1), + } { + t.Run(name, func(t *testing.T) { + got, err := decodeEvent([]byte(payload)) + if err == nil { + t.Fatal("decodeEvent(invalid envelope) succeeded") + } + if got.v1 != nil || got.v2 != nil || got.version != "" || got.authorizer != nil { + t.Error("decodeEvent(invalid envelope) returned partial data") + } + }) + } +} + +func TestDecodePreservesOpaqueAuthorizer(t *testing.T) { + for _, base := range []string{minimalV1, minimalV2} { + for _, authorizer := range []string{`null`, `"opaque"`, `[1,"two"]`, `9007199254740993`, `{"jwt":{"claims":{"groups":["a","b"],"n":9007199254740993,"exponent":1e999}}}`} { + payload := []byte(strings.Replace(base, `"apiId":"example"`, `"apiId":"example","authorizer":`+authorizer, 1)) + got, err := decodeEvent(payload) + if err != nil { + t.Fatalf("decodeEvent(authorizer=%s) = %v", authorizer, err) + } + clear(payload) + if string(got.authorizer) != authorizer { + t.Errorf("authorizer after clearing input = %s, want %s", got.authorizer, authorizer) + } + } + } +} + +func TestDecodeDoesNotExposeInputInErrors(t *testing.T) { + const secret = "credential-marker-not-for-errors" + for _, payload := range []string{ + `{"` + secret + `":1,"` + secret + `":2}`, + strings.Replace(minimalV2, `"2.0"`, `"`+secret+`"`, 1), + strings.Replace(minimalV1, `"path":"/"`, `"path":"/","headers":{"`+secret+`":123}`, 1), + strings.Replace(minimalV1, `"GET"`, `{"`+secret+`":true}`, 1), + } { + _, err := decodeEvent([]byte(payload)) + if err == nil { + t.Fatal("decodeEvent(invalid secret-bearing input) succeeded") + } + if strings.Contains(err.Error(), secret) { + t.Errorf("decodeEvent error leaked input: %v", err) + } + } +} + +func TestDecodeNullOptionalFieldsAndIndependentMaps(t *testing.T) { + payload := strings.Replace(minimalV1, `"path":"/"`, `"path":"/","body":null,"headers":{"x":"single","single":"keep"},"multiValueHeaders":{"x":["a","b"]},"queryStringParameters":null,"multiValueQueryStringParameters":null`, 1) + got, err := decodeEvent([]byte(payload)) + if err != nil { + t.Fatal(err) + } + if got.v1.Body != "" || got.v1.IsBase64Encoded || got.v1.Headers["single"] != "keep" || got.v1.Headers["x"] != "single" || !slices.Equal(got.v1.MultiValueHeaders["x"], []string{"a", "b"}) { + t.Error("decodeEvent changed optional defaults or merged single/multivalue headers before HTTP translation") + } +} + +func TestTypedValidationAndSDKCompatibility(t *testing.T) { + v1 := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}} + v2 := events.APIGatewayV2HTTPRequest{RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}} + if err := validateV1(v1); err != nil { + t.Errorf("validateV1(valid) = %v", err) + } + for _, version := range []string{"", "2.0", "1.0", "3.0"} { + v2.Version = version + err := validateV2(v2) + wantErr := version != "" && version != "2.0" + if (err != nil) != wantErr { + t.Errorf("validateV2(version=%q) = %v, want error %v", version, err, wantErr) + } + } + v2.Version = "2.0" + for _, event := range []any{v1, v2} { + payload, err := json.Marshal(event) + if err != nil { + t.Fatal(err) + } + if _, err := decodeEvent(payload); err != nil { + t.Errorf("decodeEvent(SDK %T) = %v", event, err) + } + } + v1.RequestContext.APIID = "" + v2.RequestContext.APIID = "" + if validateV1(v1) == nil || validateV2(v2) == nil { + t.Error("typed validation accepted missing API ID") + } +} + +func FuzzDecodeEvent(f *testing.F) { + f.Add([]byte(minimalV1)) + f.Add([]byte(minimalV2)) + f.Add([]byte(`{"version":"2.0","version":"1.0"}`)) + for _, file := range []string{"rest", "http-v1", "http-v2"} { + payload, err := os.ReadFile("testdata/events/" + file + ".json") + if err != nil { + f.Fatal(err) + } + f.Add(payload) + } + f.Fuzz(func(t *testing.T, payload []byte) { + before := bytes.Clone(payload) + got, err := decodeEvent(payload) + if !bytes.Equal(payload, before) { + t.Fatal("decodeEvent mutated input") + } + if err != nil { + if got.v1 != nil || got.v2 != nil || got.authorizer != nil || got.version != "" { + t.Fatal("decodeEvent returned partial data with error") + } + return + } + var object map[string]jsontext.Value + if err := json.Unmarshal(payload, &object); err != nil || object == nil { + t.Fatalf("decodeEvent accepted a non-object or invalid JSON: %v", err) + } + if (got.v1 == nil) == (got.v2 == nil) { + t.Fatal("decodeEvent did not select exactly one event format") + } + if got.v2 != nil && (got.v2.Version != "2.0" || got.v2.RawPath == "" || got.v2.RequestContext.HTTP.Method == "" || got.v2.RequestContext.APIID == "") { + t.Fatal("decodeEvent accepted v2 without its required values") + } + if got.v1 != nil && (got.v1.Path == "" || got.v1.HTTPMethod == "" || got.v1.RequestContext.APIID == "") { + t.Fatal("decodeEvent accepted v1 without its required values") + } + }) +} diff --git a/docs/decisions/0003-event-decoding.md b/docs/decisions/0003-event-decoding.md index d673178..6cbc726 100644 --- a/docs/decisions/0003-event-decoding.md +++ b/docs/decisions/0003-event-decoding.md @@ -1,6 +1,6 @@ # 0003: AWS event types and explicit event decoding -Status: proposed; AWS type reuse accepted, detailed decoding policy pending. +Status: accepted by the user; private decoder and typed validators implemented. ## Question @@ -33,11 +33,11 @@ use the AWS SDK event structs or private types owned by this module? - Lambda Function URLs use the same request/response schema as API Gateway V2. Shape validation cannot authenticate the invoking service. [Function URL payloads](https://docs.aws.amazon.com/lambda/latest/dg/urls-invocation.html) -- Our implementation currently contains only the constructor/options foundation. +- At the time of this decision, implementation contained only the constructor/options foundation. No existing decoder or wire schema constrains this choice. The accepted plan already requires direct JSON v2 and rejection rather than V2-to-V1 fallback. -## Recommendation +## Decision ### Wire representation @@ -48,7 +48,7 @@ reuse does not require using the SDK's JSON decoder. Keep a small explicit version discriminator and targeted presence/type validation where needed; do not duplicate complete AWS schemas just to select the event format. -Proposed exception: carry authorizer data internally as `jsontext.Value` until +Carry authorizer data internally as `jsontext.Value` until the selected identity producer interprets it. Preserve exactly the JSON representation AWS supplies; this does not assert that AWS supplies full-fidelity JWT claims or authorize @@ -57,7 +57,7 @@ claim numbers at the transport boundary. Authorizer extraction and SDK decoding must be designed together so an SDK claim field cannot prematurely reject data intended to remain opaque. The exact -exception mechanism remains pending; authorizer fidelity alone does not justify +exception mechanism is an implementation detail; authorizer fidelity alone does not justify private copies of the entire envelope. The AWS Lambda Go library remains the runtime integration and supplies event @@ -84,7 +84,7 @@ and the limits of validation after another component has decoded the payload. For every accepted envelope require a non-null object `requestContext`, a nonempty string `requestContext.apiId`, and the canonical method/path fields as nonempty strings. V2 also requires the `requestContext.http` object. These are -proposed library validity checks based on the documented envelopes, not a claim +accepted library validity checks based on the documented envelopes, not a claim that AWS publishes a formal required-field schema for all variants. Do not fall back to a different format after a decode/validation error. Do not @@ -101,7 +101,7 @@ accepted in decision 0001. ### Already typed events The selected method determines the payload family without automatic dispatch. -Proposed semantic checks still require nonempty API ID, method, and canonical +Semantic checks still require nonempty API ID, method, and canonical path. HandleV2 accepts Version equal to "2.0" or empty: an omitted Go string does not make a caller-created V2 struct ambiguous, because its type and method already select V2. Reject any other nonempty Version as contradictory input. @@ -109,7 +109,7 @@ HandleV1 has no version field to inspect. Neither method can validate original JSON shape, member presence, or syntax. Do not serialize structs to imitate raw-wire validation. -This is a proposed ergonomic exception for explicitly typed input, not a change +This is an accepted ergonomic exception for explicitly typed input, not a change to raw Invoke: a present empty version in JSON remains invalid there. Matching raw and typed valid events share translation; their available validation evidence differs as accepted in decision 0004. @@ -170,14 +170,28 @@ sole oracle. Run fuzzing against the decoder after implementation. - Reject all unknown fields: catches misspellings, but makes additive AWS fields a breaking change for deployed applications. -SDK structs alone do not provide the proposed presence checks or opaque-claim +SDK structs alone do not provide the accepted presence checks or opaque-claim contract. Limit supplementary representations to those needs and maintain independent wire fixtures so SDK serialization is not the sole specification. ## Resolution -The user accepted AWS type reuse and requested typed-event entry points. This -supersedes the original recommendation for entirely private wire types. The -detailed version/structure/strictness rules above remain proposed; agreement to -type reuse does not approve every earlier validation rule. No decoder is -implemented yet. +The user accepted AWS type reuse, the typed entry points, and subsequently the +remaining decoding contract in full: dispatch, structural checks, strict JSON, +opaque authorizer data, and the typed V2 empty-version exception. This supersedes +the original recommendation for entirely private wire types. + +The private decoder and typed validators are implemented. Pin AWS Lambda Go +v1.55.0, verified against the +Go module proxy on 2026-09-14. Its API Gateway event definitions are unchanged +from the inspected v1.54.0, and its reflected handler still uses encoding/json. +Use SDK embedding with a shadowed request context/authorizer field to retain +opaque data without copying the complete schema. SDK dependency selection and +this private decoding mechanism do not alter the accepted public API. + +Validation: documented-shape synthetic fixtures, malformed/unsupported envelopes, +opaque authorizer ownership and precision, error redaction, and SDK serialization +compatibility pass. Race tests, vet, and Linux arm64/amd64 builds pass. A 15-second +fuzz run completed 1,553,008 executions without a failure. Public Invoke and typed +HTTP entry points still await the HTTP translation and response contracts; these +checks do not claim end-to-end Lambda adaptation is implemented. diff --git a/docs/decisions/0005-http-request-translation.md b/docs/decisions/0005-http-request-translation.md new file mode 100644 index 0000000..573c623 --- /dev/null +++ b/docs/decisions/0005-http-request-translation.md @@ -0,0 +1,162 @@ +# 0005: Translating gateway requests into net/http + +Status: proposed; awaiting user review. + +## Question + +What should the HTTP handler observe across raw and typed API Gateway inputs, +including information that API Gateway has already decoded, combined, or lost? + +## Evidence + +- AWS REST events contain decoded parameter maps, and AWS does not preserve + request-parameter ordering. Its documented header merge rule deduplicates a + key/value present in both single and multivalue maps, rather than discarding + all single-value entries whenever any multivalue entries exist. + [REST proxy inputs](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) +- HTTP API V2 supplies rawPath and rawQueryString. Its flattened query/header + maps combine duplicate values with commas. RawPath omits the custom-domain + API mapping prefix. V2 also has a dedicated cookies array. + [HTTP API payload differences](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) +- Go's URL.Path is decoded; URL.RawPath can preserve the encoded spelling. + Incoming HTTP requests normally have no URL scheme or authority. The host is + in Request.Host and absent from Request.Header. Body is always non-nil, and + the server owns its cleanup. RemoteAddr has no required format, although the + ordinary HTTP server uses IP:port. TLS describes the connection received by + the server, not an assertion about an upstream connection. + [Go URL](https://pkg.go.dev/net/url#URL) + [Go Request](https://pkg.go.dev/net/http#Request) +- Go's base64 decoder can return partial data with an error. That partial data + must not reach the HTTP handler under decision 0002. + [DecodeString](https://pkg.go.dev/encoding/base64#Encoding.DecodeString) +- The new decoder preserves SDK transport fields and authorizer JSON separately. + It does not yet construct HTTP requests. Beakley's decoded-path reparsing and + wholesale single-map loss are independently reproduced requirements to avoid. + +## Recommendation + +### Paths and queries + +- V1: assign the supplied decoded path to URL.Path directly. Generate an escaped + request target from that value; do not parse it as a URL or unescape it again. + Literal question marks, hashes, and percent signs remain path data. +- V2: decode rawPath once into URL.Path and retain its encoded form in RawPath + when valid. Reject invalid path escapes and raw path delimiters that would + change the request target. Do not substitute requestContext.http.path. +- Require an absolute path beginning with a slash. Do not clean dot segments, + collapse repeated slashes, strip a stage, or invent a custom-domain prefix. + ServeMux retains responsibility for its own redirect/routing behavior. +- V1 query names are case-sensitive. Preserve multivalue occurrences and their + order, then append a single-map value only when that exact value is absent + from the multivalue list for the same key. Include keys found only in the + single map. This applies AWS's header merge principle consistently to queries; + handling contradictory synthetic query maps is an edge policy, not an AWS + guarantee. Encode through url.Values.Encode; original encoding/key order is + unavailable and cannot be reconstructed. +- V2: use rawQueryString unchanged, including repeated parameters and encoding. + Never reconstruct it from the comma-flattened map or split values on commas. + Leave query parsing and its errors to the handler's ordinary net/url APIs. +- Populate RequestURI from the escaped path and query. V1 necessarily receives + a reconstructed target. A lost empty trailing question mark or API mapping + prefix cannot be recovered. Never populate Fragment, User, or Opaque. +- Leave URL.Scheme and URL.Host empty, as for an ordinary origin-form server + request. Do not derive them from forwarding headers. + +### Headers, cookies, and host + +- Canonicalize header names with net/http's rules and validate HTTP field names + and values. Reject invalid field names and prohibited control bytes. Preserve + ordinary value text and commas without generic splitting or trimming. +- V1: merge by case-insensitive name and exact value as described for queries. + Preserve repeated values already in the multivalue list. Across differently + cased map keys, use sorted source keys for deterministic traversal because + original cross-key order is unavailable. Do not discard unrelated single-map + values. This also specifies deterministic handling of caller-created structs. +- V2: retain each flattened header value as supplied. If cookies is nonempty, + it supplies the Cookie header, joined with "; "; otherwise retain an existing + Cookie header. Dedicated cookies take precedence to avoid duplicate cookies + from contradictory typed/synthetic inputs. Do not comma-split cookie values. +- Move the selected Host header into Request.Host and remove it from Header. + Reject multiple selected host values or malformed authority syntax. If there + is no Host header, use requestContext.domainName; if both are missing, leave + Host empty. Do not trust X-Forwarded-Host to change the target host. +- Leave gateway pathParameters and routeKey separate from Go ServeMux state. + Do not synthesize Request.Pattern or populate PathValue from an unrelated + gateway route. A future metadata accessor remains a separate API decision. + +### Method, body, and invocation metadata + +- Preserve method casing; validate it as an HTTP token. No empty-method-to-GET + fallback. Unsupported application methods remain the handler's concern. +- Decode a base64 body with the standard padded encoding when flagged. Return + a conversion error on malformed base64 and discard all partial decoded data. + Otherwise use the supplied string bytes without parsing application JSON, + decompressing content, or applying a second text encoding. +- Set ContentLength to the actual decoded byte length. Reconcile a supplied + Content-Length header to that length, and remove Transfer-Encoding and Trailer + declarations because the Lambda envelope has already framed the complete body + and supplies no streaming trailers. Leave TransferEncoding and Trailer empty. + These are transport normalization choices for review, not claims of original + client framing fidelity. +- Supply http.NoBody for an empty body and a fresh readable body otherwise. + Own request headers, query value slices, and body state; do not mutate or alias + mutable request data supplied by typed callers. Application middleware can + change the request without changing the source event. +- Use the invocation context as the parent of a request context, and cancel the + child after ServeHTTP returns. Close the adapter-created body and clean up + multipart temporary files created through standard request parsing. The + gateway does not provide ordinary client-disconnect notification through this + context. Do not invent such a signal. +- Preserve the reported HTTP protocol and parsed major/minor values. Default + an absent value to HTTP/1.1 as an explicitly documented library fallback; + reject a present malformed protocol. Do not infer protocol from headers. +- Set RemoteAddr to requestContext's source IP as supplied (empty when absent). + Do not add an invented client port or replace it with X-Forwarded-For. This + differs from the usual IP:port form but fits Go's documented field contract. +- Leave TLS nil: Lambda has no client TLS connection to expose. Do not create a + fabricated tls.ConnectionState from gateway metadata. Any later mTLS identity + support needs an explicit gateway assertion contract. + +### Shared behavior and remaining decisions + +Raw and typed entry points use the same conversion functions after their +respective validated input boundaries. Conversion failures use decision 0002's +error policy without credential-bearing details. Gateway identity remains the +separate opt-in producer from decision 0001. + +Response commitment, binary output selection, response limits, panic handling, +and optional ResponseWriter capabilities remain a separate response contract. +Gateway metadata accessors, identity interpretation, and cancellation's effect on +the final invocation result remain separate reviews. Passing cancellation to the +handler does not by itself settle whether to discard a completed response. + +## Alternatives and consequences + +- Parsing a V1 path as a complete URL changes literal path delimiters and can + route to a different resource. Assigning URL.Path avoids that bug. +- Using V2's flattened query map loses distinctions between repeats and literal + commas. RawQueryString preserves the representation AWS still provides. +- Multivalue replacement per whole map loses unrelated fields; replacement per + key also discards a differing single value. The documented key/value merge + rule preserves all supplied values without duplicating the single-map echo. +- Synthesizing https/TLS metadata may help existing redirect middleware, but + creates an inaccurate connection object. Document the server-request shape + and review explicit gateway metadata separately. +- Requiring every context metadata field would reject otherwise usable typed + requests. Defaults above are explicit and limited to fields unnecessary for + unambiguous route selection. + +## Validation plan + +Compare relevant results with Go's HTTP request parsing and a real local HTTP +server, plus independent AWS fixtures. Cover V1 literal delimiters, V2 encoded +slashes and double escaping, Unicode, repeated slashes, malformed path escapes, +query repeats/commas, all header merge cases, cookie precedence, Host promotion, +invalid header bytes, body bytes/base64 errors, protocol handling, source address, +context cancellation, body/multipart cleanup, and caller-owned data isolation. +Exercise the same HTTP assertions through raw and typed entry points when the +response contract makes those public entry points implementable. + +## Resolution + +Pending review. No HTTP request translation is implemented under this proposal. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 3e13e70..68f6cfe 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -11,5 +11,6 @@ Keep related decisions small enough to review and implement independently. | --- | --- | --- | | [0001: Gateway identity](0001-gateway-identity.md) | Accepted | How should gateway identity be enabled, and which assertions does it cover? | | [0002: Adapter boundary](0002-adapter-boundary.md) | Accepted | How should construction fail, and which failures become Lambda invocation errors? | -| [0003: Event decoding](0003-event-decoding.md) | Proposed; AWS type reuse accepted | How should SDK types, dispatch, and wire validation work together? | +| [0003: Event decoding](0003-event-decoding.md) | Accepted | How should SDK types, dispatch, and wire validation work together? | | [0004: Typed events](0004-typed-events.md) | Accepted | How can already typed events bypass edge's envelope JSON work? | +| [0005: HTTP requests](0005-http-request-translation.md) | Proposed | How should raw and typed events become ordinary server requests? | diff --git a/docs/plan.md b/docs/plan.md index f89e377..2a1f869 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -79,11 +79,16 @@ constructor/options foundation is implemented while the event/HTTP contracts are reviewed. Constructor tests pass with race detection, vet and formatting checks pass, and package builds pass for Linux arm64 and amd64. -Current review: [decision 0003](decisions/0003-event-decoding.md) records accepted -AWS type reuse and proposes version dispatch, structural validation, and a decoder -fixture matrix. [Decision 0004](decisions/0004-typed-events.md) accepts typed +Accepted: [decision 0003](decisions/0003-event-decoding.md) settles AWS type reuse, +version dispatch, structural validation, and the decoder fixture matrix. The +private decoder and typed validators are implemented and tested against AWS +Lambda Go v1.55.0. [Decision 0004](decisions/0004-typed-events.md) accepts typed methods that skip envelope JSON work inside edge. These entry points await the -remaining decoding and shared HTTP translation contracts before implementation. +shared HTTP translation contract before implementation. + +Current review: [decision 0005](decisions/0005-http-request-translation.md) proposes +request URLs, header/query merging, cookies, body handling, invocation metadata, +and request ownership. The buffered response contract follows separately. Research and present a contract covering: @@ -110,6 +115,8 @@ presenting an entire frozen public API at once. ### 2. Implement and prove the HTTP adapter - [x] Create the Go 1.27 module and constructor/options foundation after review. +- [x] Implement the private decoder and typed semantic validators with direct + JSON v2, SDK types, opaque authorizer data, fixtures, and fuzzing. - [ ] Implement raw invocation after the remaining transport contracts are reviewed. - [ ] Implement the accepted typed entry points after shared HTTP translation review, documenting upstream codec ownership. diff --git a/go.mod b/go.mod index 85057cf..3830d49 100644 --- a/go.mod +++ b/go.mod @@ -1,3 +1,5 @@ module github.com/asteroid-computing/go-lambda-edge go 1.27.0 + +require github.com/aws/aws-lambda-go v1.55.0 diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..271d53f --- /dev/null +++ b/go.sum @@ -0,0 +1,10 @@ +github.com/aws/aws-lambda-go v1.55.0 h1:D+AUkk49hTldMnv+Ydt+/jdWVXWPsp8hlwY6Enob2FM= +github.com/aws/aws-lambda-go v1.55.0/go.mod h1:V+NzkHNR6vBC8C1PDloqSLE+7jYWFiPvJJFiCiTm8nE= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/testify v1.7.2 h1:4jaiDzPyXQvSd7D0EjG45355tLlV3VOECpq10pLC+8s= +github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/testdata/events/README.md b/testdata/events/README.md new file mode 100644 index 0000000..48a4cc4 --- /dev/null +++ b/testdata/events/README.md @@ -0,0 +1,16 @@ +# Decoder fixtures + +These are synthetic integration events derived from the documented AWS payload +shapes, with focused edge cases added. They are not captured production requests, +SDK-generated golden files, or copies of Beakley tests. + +- `rest.json`: REST proxy shape, decoded path containing delimiters, independent + single/multivalue maps, and application-invalid JSON left as an opaque body. +- `http-v1.json`: HTTP API payload 1.0 and documented null optional fields. +- `http-v2.json`: HTTP API payload 2.0, escaped path, raw repeated query values, + cookies, binary body, native authorizer, and a synthetic future field. + +Sources: + +- [REST proxy integration](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) +- [HTTP API payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) diff --git a/testdata/events/http-v1.json b/testdata/events/http-v1.json new file mode 100644 index 0000000..b172c88 --- /dev/null +++ b/testdata/events/http-v1.json @@ -0,0 +1,22 @@ +{ + "version": "1.0", + "resource": "/my/path", + "path": "/my/path", + "httpMethod": "GET", + "headers": {"header1": "value1"}, + "multiValueHeaders": {"header1": ["value1", "value2"]}, + "queryStringParameters": {"parameter1": "value1"}, + "multiValueQueryStringParameters": {"parameter1": ["value1", "value2"]}, + "requestContext": { + "apiId": "example", + "httpMethod": "GET", + "path": "/my/path", + "protocol": "HTTP/1.1", + "identity": {"sourceIp": "192.0.2.1", "accessKey": null}, + "authorizer": {"claims": null, "scopes": null} + }, + "body": null, + "isBase64Encoded": false, + "pathParameters": null, + "stageVariables": null +} diff --git a/testdata/events/http-v2.json b/testdata/events/http-v2.json new file mode 100644 index 0000000..a60d118 --- /dev/null +++ b/testdata/events/http-v2.json @@ -0,0 +1,20 @@ +{ + "version": "2.0", + "routeKey": "$default", + "rawPath": "/my/%2Fpath", + "rawQueryString": "parameter1=value1¶meter1=value2&literal=%2C", + "cookies": ["a=1", "b=2"], + "headers": {"header1": "value1,value2"}, + "queryStringParameters": {"parameter1": "value1,value2", "literal": ","}, + "requestContext": { + "apiId": "example", + "domainName": "example.execute-api.eu-west-1.amazonaws.com", + "http": {"method": "POST", "path": "/my//path", "protocol": "HTTP/1.1", "sourceIp": "192.0.2.1"}, + "authorizer": {"jwt": {"claims": {"sub": "example"}, "scopes": ["read"]}} + }, + "body": "AP8=", + "isBase64Encoded": true, + "pathParameters": null, + "stageVariables": null, + "futureField": {"nested": [true, 9007199254740993]} +} diff --git a/testdata/events/rest.json b/testdata/events/rest.json new file mode 100644 index 0000000..34112b1 --- /dev/null +++ b/testdata/events/rest.json @@ -0,0 +1,19 @@ +{ + "resource": "/{proxy+}", + "path": "/a?b#c", + "httpMethod": "POST", + "headers": {"Content-Type": "application/json", "X-Single": "one"}, + "multiValueHeaders": {"X-Multi": ["one", "two"]}, + "queryStringParameters": {"single": "one"}, + "multiValueQueryStringParameters": {"multi": ["one", "two"]}, + "requestContext": { + "apiId": "example", + "stage": "test", + "identity": {"sourceIp": "192.0.2.1", "userArn": null}, + "authorizer": {"claims": null, "scopes": null} + }, + "body": "{application-invalid-json", + "isBase64Encoded": false, + "pathParameters": {"proxy": "a?b#c"}, + "stageVariables": null +} From cea8263bea2c92005ed56ef82d023e44c83f610d Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 15:08:53 +0100 Subject: [PATCH 06/44] feat(edge): translate gateway events into HTTP requests --- README.md | 8 +- adapter.go | 4 +- docs/decisions/0003-event-decoding.md | 2 +- docs/decisions/0004-typed-events.md | 3 +- .../0005-http-request-translation.md | 27 +- docs/decisions/0006-buffered-responses.md | 172 ++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 12 +- request.go | 240 +++++++++++ request_test.go | 394 ++++++++++++++++++ 10 files changed, 850 insertions(+), 15 deletions(-) create mode 100644 docs/decisions/0006-buffered-responses.md create mode 100644 request.go create mode 100644 request_test.go diff --git a/README.md b/README.md index 5a241eb..74da4e1 100644 --- a/README.md +++ b/README.md @@ -5,10 +5,12 @@ A Go 1.27 module being built to serve AWS API Gateway requests through ordinary authorization packages. The root package is `edge`. This project is being designed and built from scratch. -The constructor, options, and private event decoder are implemented. The decoder +The constructor, options, private event decoder, and request conversion are implemented. The decoder uses AWS Lambda Go v1.55.0 event structs and encoding/json/v2 directly, retaining -authorizer JSON separately. Invocation, HTTP translation, and identity extraction -are not implemented yet; the adapter cannot yet be registered as a Lambda handler. +authorizer JSON separately. Request conversion covers URLs, headers, body bytes, +and request lifetime. Buffered responses, public invocation methods, and identity +extraction are not implemented yet; the adapter cannot yet be registered as a +Lambda handler. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) diff --git a/adapter.go b/adapter.go index 3f2620f..0a899a6 100644 --- a/adapter.go +++ b/adapter.go @@ -1,5 +1,5 @@ -// Package edge provides the configuration foundation for adapting AWS API Gateway -// events to net/http handlers. Invocation and HTTP translation are not yet +// Package edge provides the foundation for adapting AWS API Gateway events to +// net/http handlers. Public invocation and response encoding are not yet // implemented. package edge diff --git a/docs/decisions/0003-event-decoding.md b/docs/decisions/0003-event-decoding.md index 6cbc726..d1ff965 100644 --- a/docs/decisions/0003-event-decoding.md +++ b/docs/decisions/0003-event-decoding.md @@ -193,5 +193,5 @@ Validation: documented-shape synthetic fixtures, malformed/unsupported envelopes opaque authorizer ownership and precision, error redaction, and SDK serialization compatibility pass. Race tests, vet, and Linux arm64/amd64 builds pass. A 15-second fuzz run completed 1,553,008 executions without a failure. Public Invoke and typed -HTTP entry points still await the HTTP translation and response contracts; these +HTTP entry points still await response encoding and gateway identity production; these checks do not claim end-to-end Lambda adaptation is implemented. diff --git a/docs/decisions/0004-typed-events.md b/docs/decisions/0004-typed-events.md index c8adb88..d8c38da 100644 --- a/docs/decisions/0004-typed-events.md +++ b/docs/decisions/0004-typed-events.md @@ -33,7 +33,8 @@ func (a *Adapter) HandleV2(ctx context.Context, event events.APIGatewayV2HTTPReq ``` The user approved these names, exact signatures, and the boundary contract below. -They are not implemented yet; shared HTTP translation still needs review. +They are not implemented yet; request conversion is complete, while response +encoding and gateway identity production remain under development. The ordinary raw registration remains lambda.Start(adapter). A caller that chooses a fixed payload family can register lambda.Start(adapter.HandleV1) or diff --git a/docs/decisions/0005-http-request-translation.md b/docs/decisions/0005-http-request-translation.md index 573c623..8285f64 100644 --- a/docs/decisions/0005-http-request-translation.md +++ b/docs/decisions/0005-http-request-translation.md @@ -1,6 +1,6 @@ # 0005: Translating gateway requests into net/http -Status: proposed; awaiting user review. +Status: accepted by the user; shared request conversion implemented. ## Question @@ -33,7 +33,7 @@ including information that API Gateway has already decoded, combined, or lost? It does not yet construct HTTP requests. Beakley's decoded-path reparsing and wholesale single-map loss are independently reproduced requirements to avoid. -## Recommendation +## Decision ### Paths and queries @@ -56,6 +56,8 @@ including information that API Gateway has already decoded, combined, or lost? - V2: use rawQueryString unchanged, including repeated parameters and encoding. Never reconstruct it from the comma-flattened map or split values on commas. Leave query parsing and its errors to the handler's ordinary net/url APIs. + Request-line syntax validation rejects literal ASCII whitespace/control bytes; + it does not decode query values or reject malformed query percent escapes. - Populate RequestURI from the escaped path and query. V1 necessarily receives a reconstructed target. A lost empty trailing question mark or API mapping prefix cannot be recovered. Never populate Fragment, User, or Opaque. @@ -96,7 +98,7 @@ including information that API Gateway has already decoded, combined, or lost? Content-Length header to that length, and remove Transfer-Encoding and Trailer declarations because the Lambda envelope has already framed the complete body and supplies no streaming trailers. Leave TransferEncoding and Trailer empty. - These are transport normalization choices for review, not claims of original + These are accepted transport normalization choices, not claims of original client framing fidelity. - Supply http.NoBody for an empty body and a fresh readable body otherwise. Own request headers, query value slices, and body state; do not mutate or alias @@ -159,4 +161,21 @@ response contract makes those public entry points implementable. ## Resolution -Pending review. No HTTP request translation is implemented under this proposal. +The user approved this request-translation contract. Shared V1/V2 conversion and +request-lifetime management are implemented in request.go. Response behavior +remains a separate decision; public invocation methods are not implemented yet. + +Validation passed: request observations compared with a real local HTTP server, +independent request-target parsing, header/query merge and ownership checks, +binary bodies, malformed-input rejection, context cancellation, and body/multipart +cleanup on normal return and panic. Race tests, vet, formatting, and Linux arm64 +and amd64 builds passed. V2 target fuzzing completed 687,712 executions and V1 +query fuzzing completed 113,979 executions without failures, following an earlier +1,054,667-execution path-only run. + +Implementation limit: automatic multipart cleanup reaches forms attached to the +served request. Forms first parsed on middleware-created request copies are not +visible to the adapter, matching the standard Go server's limitation. The +recommendation to document middleware ownership for those forms is included in +[decision 0006](0006-buffered-responses.md) for review. Do not claim cleanup of +every independently created request or multipart reader. diff --git a/docs/decisions/0006-buffered-responses.md b/docs/decisions/0006-buffered-responses.md new file mode 100644 index 0000000..16d9d1c --- /dev/null +++ b/docs/decisions/0006-buffered-responses.md @@ -0,0 +1,172 @@ +# 0006: Buffered HTTP responses and invocation completion + +Status: proposed; awaiting user review. + +## Question + +How should the adapter capture an HTTP response, encode it for each gateway +format, and handle features or failures that a buffered invocation cannot carry? + +## Evidence + +- Go's ResponseWriter commits a final status and headers once, supplies an + implicit 200 on Write, and supports automatic content-type detection. The + ordinary server handles informational responses, trailers, and connection + capabilities that are not represented by a buffered gateway response. + [ResponseWriter](https://pkg.go.dev/net/http#ResponseWriter) + [DetectContentType](https://pkg.go.dev/net/http#DetectContentType) + [ResponseController](https://pkg.go.dev/net/http#ResponseController) +- AWS V1 responses support multivalue headers. V2 has single-string headers and + a cookies array whose entries become separate Set-Cookie fields. Explicit + statusCode avoids V2's inferred-response behavior. + [REST response format](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) + [HTTP API response formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) +- AWS requires base64 encoding for binary REST proxy responses and matching + binary media configuration. The first Accept value affects REST binary + handling; AWS recommends */* when clients such as browsers control that order. + [Binary responses](https://docs.aws.amazon.com/apigateway/latest/developerguide/lambda-proxy-binary-media.html) +- Lambda buffered responses have a 6 MB maximum, regardless of API Gateway's + larger service limit. JSON quoting, headers, cookies, and base64 consume that + payload budget. Streaming is a separate invocation contract. + [Lambda buffered/streaming limits](https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html) +- The inspected Go 1.27.1 server rejects writes to 204/304 with ErrBodyNotAllowed, + omits HEAD bodies, and validates declared content length. Its cleanup removes + multipart files reachable from the request it served. WithContext makes a + request copy; forms parsed later on that copy are not visible to the original. + [Go server source](https://go.dev/src/net/http/server.go) + [WithContext](https://pkg.go.dev/net/http#Request.WithContext) +- Our decoder and request conversion are independent of response capture. + serveHTTP already cancels the request child context and cleans up on both + return and panic. It reports cleanup errors without file paths or input values. + +## Recommendation + +### Writer commitment and ordinary responses + +- Implement a private buffered ResponseWriter shared by all entry points. + First final WriteHeader commits the status and a deep header snapshot; later + calls and header mutations cannot rewrite the committed response. Write commits + 200 when needed, and a handler that writes nothing returns 200 with an empty body. +- Accept final statuses 200 through 599. An invalid first status records an + adapter fault; WriteHeader cannot return an error, so completion must report + the fault. Do not log it or synthesize a different application response. +- Detect an absent Content-Type from up to the first 512 buffered body bytes, + unless Content-Encoding is set. An explicitly present nil/empty header slice + suppresses automatic content type, following net/http's header convention. + Empty bodies do not acquire a sniffed type. Do not generate Date or connection + management headers inside edge; the gateway owns the network response. +- Validate committed header names/values and require encodable UTF-8 header + strings so raw and typed paths cannot disagree through replacement encoding. + Canonicalize header names and retain all values; use deterministic traversal + for differently cased keys supplied by direct Header map assignment. +- Derive Content-Length from the buffered bytes when absent. If explicitly + supplied, require one valid nonnegative decimal length and enforce it: an + overflowing Write returns http.ErrContentLength; a mismatch at completion is + an invocation error. Transport faults remain faults even if the handler ignores + a Write error. Ordinary application 4xx/5xx responses still return nil Go error. +- For HEAD, accept/count writes and retain at most the sniffing prefix, but emit + no body. Honor a supplied representation length; otherwise infer a length only + when the handler actually wrote representation bytes. Do not require HEAD's + declared length to match an omitted representation body. +- For 204/304, emit no body and return http.ErrBodyNotAllowed from Write. That + error alone does not invalidate the committed bodyless response. Suppress + Content-Length for both statuses and Content-Type for 304, matching Go's server. + +### Capabilities the buffered adapter cannot provide + +- Do not implement Flusher, Hijacker, Pusher, CloseNotifier, full-duplex, or + connection-deadline methods. ResponseController capability requests naturally + return an error matching http.ErrNotSupported. Request.Context is the lifecycle + signal; it does not imply a client-disconnect signal. +- Treat explicit 1xx writes, including 101 upgrades, as unsupported invocation + faults. A buffered proxy response cannot convey their separate timing. This + is an intentional compatibility limit, rather than pretending they were sent. +- Reject declared trailers and TrailerPrefix values. Reject upgrades. Remove + connection-specific response headers and fields nominated by Connection; + never claim a chunked transfer encoding for an already buffered payload. + +### Gateway response encoding + +- Return AWS response structs from typed methods. Invoke serializes the selected + response with encoding/json/v2 directly. Do not JSON-round-trip typed events + or responses. Always set an explicit final status and body representation. +- V1 uses MultiValueHeaders for every response header, including each Set-Cookie. + Leave the redundant single-value Headers map empty. +- V2 moves Set-Cookie values into Cookies. Combine other repeated header values + with a comma and space into Headers, without deduplicating or interpreting + commas already inside a value. Leave MultiValueHeaders empty. V2 cannot retain + general header line boundaries; applications needing those must choose V1. +- Emit text directly only when bytes are valid UTF-8, Content-Encoding is absent + or identity, and the media type is textual: text/*, application/json, + application/xml, application/javascript, application/x-www-form-urlencoded, + or a +json/+xml suffix. Otherwise base64 encode the bytes and set + IsBase64Encoded. An empty emitted body is not base64 encoded. Missing/invalid + media type is conservatively binary if it was not resolved by sniffing. +- Binary selection changes only the envelope representation, not the application + Content-Type or byte sequence. REST deployments must configure binary media + types; document the browser Accept caveat in the runnable Lambda example. + +### Bounds and completion + +- Initially cap stored response body bytes at 6,000,000, an intentionally + conservative interpretation of AWS's documented 6 MB limit. Reject a Write + crossing the cap without retaining a partial write, and record a sticky fault. + Check base64 expansion before allocating the encoded body. No new public + size-tuning option is proposed for the first implementation. +- Invoke also checks the final serialized envelope against 6,000,000 bytes, + including JSON overhead. A body below the cap can still fail this final check. + A typed method cannot measure the exact envelope produced by its caller's + serializer without violating the agreed JSON-free typed boundary; document + that the caller/runtime owns that final limit. Apply the common body/expansion + caps on both paths. Header collections are caller-controlled; a body cap is + not a total-process-memory guarantee. +- Propagate handler panics to the Lambda runtime after cleanup; do not convert + them into successful HTTP 500 responses. Direct Go callers retain normal panic + behavior. Do not add logging to the adapter. +- Before returning success, check the invocation parent context. If it is + canceled, return its error and no usable response. Check the parent, not the + child context that serveHTTP cancels during normal cleanup. This deliberately + makes cancellation win over a just-completed HTTP response when already known. +- Cleanup failures are invocation failures even after a handler produced a + response. Keep diagnostics free of file paths, bodies, tokens, and header values. + +### Multipart cleanup limit discovered during implementation + +The request layer removes multipart files attached to the served request. Like +Go's server, it cannot discover a form first parsed on an independently copied +request. Recommend documenting that middleware which makes such a copy owns +cleanup of forms it creates there. Do not preparse multipart bodies: that would +consume bodies before handlers choose parsing limits and would change streaming +read behavior. This limitation is surfaced for review; there is no hidden global +temporary-directory scan or eager parsing workaround in the implementation. + +## Alternatives and consequences + +- httptest.ResponseRecorder is a useful test reference but advertises Flush and + does not enforce every production server behavior. Do not expose it as our + production writer merely to avoid implementing the approved contract. +- Silently dropping informational responses or trailers would let applications + believe unsupported output reached clients. Explicit faults are inspectable. +- Base64 for every response is simpler but adds size overhead and REST deployment + requirements even for ordinary JSON/text responses. UTF-8 alone is insufficient + to decide whether a declared binary representation should be encoded. +- Returning HTTP 500 for all faults obscures the Lambda invocation failure signal + and contradicts decision 0002. Application-written 500 responses remain valid. +- Typed callers retain their no-JSON boundary, at the cost of owning final + serialization limits. The library cannot promise exact codec parity with an + external runtime whose formatting options the caller controls. + +## Validation plan + +Use a real net/http server for commitment, sniffing, HEAD, 204/304, and length +comparisons. Exercise multiple Write calls, post-commit mutation, ignored writer +errors, status/field validation, size boundaries, binary/text round trips, v1 +multivalue headers, v2 cookies, unsupported capability checks, panic cleanup, +parent cancellation, and cleanup failure propagation. Verify raw registration +through lambda.NewHandler(adapter).Invoke and typed method registration through +the SDK wrapper once the remaining gateway-identity path permits public wiring. + +## Resolution + +Pending user approval. No buffered writer or response encoding is implemented +under this proposal yet. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 68f6cfe..a10e0c6 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -13,4 +13,5 @@ Keep related decisions small enough to review and implement independently. | [0002: Adapter boundary](0002-adapter-boundary.md) | Accepted | How should construction fail, and which failures become Lambda invocation errors? | | [0003: Event decoding](0003-event-decoding.md) | Accepted | How should SDK types, dispatch, and wire validation work together? | | [0004: Typed events](0004-typed-events.md) | Accepted | How can already typed events bypass edge's envelope JSON work? | -| [0005: HTTP requests](0005-http-request-translation.md) | Proposed | How should raw and typed events become ordinary server requests? | +| [0005: HTTP requests](0005-http-request-translation.md) | Accepted | How should raw and typed events become ordinary server requests? | +| [0006: Buffered responses](0006-buffered-responses.md) | Proposed | How should HTTP responses, unsupported capabilities, and completion failures behave? | diff --git a/docs/plan.md b/docs/plan.md index 2a1f869..6778fc9 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -84,11 +84,16 @@ version dispatch, structural validation, and the decoder fixture matrix. The private decoder and typed validators are implemented and tested against AWS Lambda Go v1.55.0. [Decision 0004](decisions/0004-typed-events.md) accepts typed methods that skip envelope JSON work inside edge. These entry points await the -shared HTTP translation contract before implementation. +response contract and gateway identity production before public wiring. -Current review: [decision 0005](decisions/0005-http-request-translation.md) proposes +Accepted: [decision 0005](decisions/0005-http-request-translation.md) settles request URLs, header/query merging, cookies, body handling, invocation metadata, -and request ownership. The buffered response contract follows separately. +and request ownership. Shared request conversion is implemented with local HTTP +comparisons, race tests, fuzzing, vet, and Lambda-target builds passing. + +Current review: [decision 0006](decisions/0006-buffered-responses.md) proposes +buffered response commitment/encoding, unsupported features, payload bounds, +invocation completion, and the documented multipart cleanup limit. Research and present a contract covering: @@ -117,6 +122,7 @@ presenting an entire frozen public API at once. - [x] Create the Go 1.27 module and constructor/options foundation after review. - [x] Implement the private decoder and typed semantic validators with direct JSON v2, SDK types, opaque authorizer data, fixtures, and fuzzing. +- [x] Implement shared HTTP request conversion and request-lifetime cleanup. - [ ] Implement raw invocation after the remaining transport contracts are reviewed. - [ ] Implement the accepted typed entry points after shared HTTP translation review, documenting upstream codec ownership. diff --git a/request.go b/request.go new file mode 100644 index 0000000..1ded1ec --- /dev/null +++ b/request.go @@ -0,0 +1,240 @@ +package edge + +import ( + "bytes" + "context" + "encoding/base64" + "errors" + "io" + "maps" + "net/http" + "net/netip" + "net/url" + "slices" + "strconv" + "strings" + + "github.com/aws/aws-lambda-go/events" +) + +func requestV1(ctx context.Context, event events.APIGatewayProxyRequest) (*http.Request, error) { + if err := validateV1(event); err != nil { + return nil, err + } + if !strings.HasPrefix(event.Path, "/") { + return nil, errors.New("edge: path must begin with a slash") + } + query := make(url.Values) + for key, values := range event.MultiValueQueryStringParameters { + query[key] = slices.Clone(values) + } + for key, value := range event.QueryStringParameters { + if !slices.Contains(event.MultiValueQueryStringParameters[key], value) { + query[key] = append(query[key], value) + } + } + header, err := requestHeaders(event.Headers, event.MultiValueHeaders) + if err != nil { + return nil, err + } + u := &url.URL{Path: event.Path, RawQuery: query.Encode()} + return newRequest(ctx, &http.Request{ + Method: event.HTTPMethod, + URL: u, + Header: header, + Host: event.RequestContext.DomainName, + Proto: event.RequestContext.Protocol, + RemoteAddr: event.RequestContext.Identity.SourceIP, + }, event.Body, event.IsBase64Encoded) +} + +func requestV2(ctx context.Context, event events.APIGatewayV2HTTPRequest) (*http.Request, error) { + if err := validateV2(event); err != nil { + return nil, err + } + if !strings.HasPrefix(event.RawPath, "/") { + return nil, errors.New("edge: rawPath must begin with a slash") + } + path, err := url.PathUnescape(event.RawPath) + if err != nil { + return nil, errors.New("edge: invalid rawPath encoding") + } + u := &url.URL{Path: path, RawPath: event.RawPath, RawQuery: event.RawQueryString} + // EscapedPath ignores an invalid RawPath hint. Reject rather than silently + // replacing raw delimiters, whitespace, or an invalid encoded spelling. + if u.EscapedPath() != event.RawPath { + return nil, errors.New("edge: invalid rawPath syntax") + } + // Query decoding remains the handler's job, but a request target cannot + // contain literal ASCII whitespace or control bytes on an HTTP request line. + for i := 0; i < len(event.RawQueryString); i++ { + if event.RawQueryString[i] <= ' ' || event.RawQueryString[i] == 0x7f { + return nil, errors.New("edge: invalid raw query syntax") + } + } + header, err := requestHeaders(event.Headers, nil) + if err != nil { + return nil, err + } + if len(event.Cookies) != 0 { + cookie := strings.Join(event.Cookies, "; ") + if !validHeaderValue(cookie) { + return nil, errors.New("edge: invalid cookie header") + } + header.Set("Cookie", cookie) + } + return newRequest(ctx, &http.Request{ + Method: event.RequestContext.HTTP.Method, + URL: u, + Header: header, + Host: event.RequestContext.DomainName, + Proto: event.RequestContext.HTTP.Protocol, + RemoteAddr: event.RequestContext.HTTP.SourceIP, + }, event.Body, event.IsBase64Encoded) +} + +func requestHeaders(single map[string]string, multi map[string][]string) (http.Header, error) { + header := make(http.Header) + for _, key := range slices.Sorted(maps.Keys(multi)) { + if !validToken(key) { + return nil, errors.New("edge: invalid header name") + } + name := http.CanonicalHeaderKey(key) + for _, value := range multi[key] { + if !validHeaderValue(value) { + return nil, errors.New("edge: invalid header value") + } + header[name] = append(header[name], value) + } + } + // Keep the original multivalue lists for cross-map deduplication. Repeated + // single values under differently cased keys must not deduplicate each other. + multiValues := header.Clone() + for _, key := range slices.Sorted(maps.Keys(single)) { + if !validToken(key) { + return nil, errors.New("edge: invalid header name") + } + value := single[key] + if !validHeaderValue(value) { + return nil, errors.New("edge: invalid header value") + } + name := http.CanonicalHeaderKey(key) + if !slices.Contains(multiValues[name], value) { + header[name] = append(header[name], value) + } + } + return header, nil +} + +func newRequest(ctx context.Context, r *http.Request, body string, binary bool) (*http.Request, error) { + if ctx == nil { + return nil, errors.New("edge: nil invocation context") + } + if !validToken(r.Method) { + return nil, errors.New("edge: invalid HTTP method") + } + host := r.Host + header := r.Header + if hosts, present := header["Host"]; present { + if len(hosts) != 1 || hosts[0] == "" { + return nil, errors.New("edge: invalid Host header") + } + host = hosts[0] + } + if host != "" && !validAuthority(host) { + return nil, errors.New("edge: invalid request host") + } + delete(header, "Host") + if r.Proto == "" { + r.Proto = "HTTP/1.1" + } + major, minor, ok := http.ParseHTTPVersion(r.Proto) + if !ok { + return nil, errors.New("edge: invalid HTTP protocol") + } + var data []byte + if binary { + var err error + data, err = base64.StdEncoding.DecodeString(body) + if err != nil { + return nil, errors.New("edge: invalid base64 request body") + } + } else { + data = []byte(body) + } + if _, present := header["Content-Length"]; present { + header.Set("Content-Length", strconv.Itoa(len(data))) + } + delete(header, "Transfer-Encoding") + delete(header, "Trailer") + r.RequestURI = r.URL.RequestURI() + r.Host = host + r.ProtoMajor, r.ProtoMinor = major, minor + r.Body = http.NoBody + r.ContentLength = int64(len(data)) + if len(data) != 0 { + r.Body = io.NopCloser(bytes.NewReader(data)) + } + return r.WithContext(ctx), nil +} + +// serveHTTP owns the transport-created body and the handler's request lifetime. +// Identity production belongs before this call. Response encoding belongs after +// it. Cleanup runs on panic too; deciding panic/result policy is the invoker's job. +func (a *Adapter) serveHTTP(w http.ResponseWriter, r *http.Request) (err error) { + ctx, cancel := context.WithCancel(r.Context()) + r = r.WithContext(ctx) + body := r.Body + defer func() { + cancel() + if closeErr := body.Close(); closeErr != nil { + err = errors.Join(err, errors.New("edge: request body cleanup failed")) + } + if r.MultipartForm != nil { + if removeErr := r.MultipartForm.RemoveAll(); removeErr != nil { + err = errors.Join(err, errors.New("edge: multipart cleanup failed")) + } + } + }() + a.handler.ServeHTTP(w, r) + return nil +} + +// validToken implements the shared HTTP method/field-name token grammar. +func validToken(s string) bool { + if s == "" { + return false + } + for i := 0; i < len(s); i++ { + c := s[i] + if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || strings.ContainsRune("!#$%&'*+-.^_`|~", rune(c)) { + continue + } + return false + } + return true +} + +func validHeaderValue(s string) bool { + for i := 0; i < len(s); i++ { + if s[i] < ' ' && s[i] != '\t' || s[i] == 0x7f { + return false + } + } + return true +} + +func validAuthority(host string) bool { + if strings.ContainsAny(host, "/\\?#@, \t\r\n") { + return false + } + u, err := url.Parse("http://" + host) + if err != nil || u.Host != host || u.Hostname() == "" || u.User != nil || u.Path != "" || u.RawQuery != "" || u.Fragment != "" { + return false + } + if strings.HasPrefix(host, "[") { + addr, err := netip.ParseAddr(u.Hostname()) + return err == nil && addr.Is6() + } + return !strings.ContainsAny(u.Hostname(), ":[]") +} diff --git a/request_test.go b/request_test.go new file mode 100644 index 0000000..a127365 --- /dev/null +++ b/request_test.go @@ -0,0 +1,394 @@ +package edge + +import ( + "bufio" + "bytes" + "context" + "errors" + "io" + "mime/multipart" + "net/http" + "net/http/httptest" + "os" + "slices" + "strings" + "testing" + + "github.com/aws/aws-lambda-go/events" +) + +func v1RequestEvent() events.APIGatewayProxyRequest { + return events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}} +} + +func v2RequestEvent() events.APIGatewayV2HTTPRequest { + return events.APIGatewayV2HTTPRequest{RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}} +} + +func TestRequestPathsMatchHTTPParsing(t *testing.T) { + for _, tt := range []struct { + name string + v1Path string + raw string + path string + }{ + {name: "delimiters", v1Path: "/a?b#c%", raw: "/a%3Fb%23c%25", path: "/a?b#c%"}, + {name: "encoded_slash", v1Path: "/a/b", raw: "/a%2fb", path: "/a/b"}, + {name: "double_escape", v1Path: "/a%2Fb", raw: "/a%252Fb", path: "/a%2Fb"}, + {name: "unicode", v1Path: "/café", raw: "/caf%C3%A9", path: "/café"}, + {name: "no_cleaning", v1Path: "/a//../b", raw: "/a//../b", path: "/a//../b"}, + {name: "network_path", v1Path: "//host/a", raw: "//host/a", path: "//host/a"}, + } { + t.Run(tt.name, func(t *testing.T) { + v1, v2 := v1RequestEvent(), v2RequestEvent() + v1.Path, v2.RawPath = tt.v1Path, tt.raw + v2.RawQueryString = "x=a&x=b&literal=%2C&space=+" + v2.QueryStringParameters = map[string]string{"wrong": "ignored"} + r1, err := requestV1(t.Context(), v1) + if err != nil { + t.Fatal(err) + } + r2, err := requestV2(t.Context(), v2) + if err != nil { + t.Fatal(err) + } + if r1.URL.Path != tt.v1Path || r1.URL.Fragment != "" || r1.URL.RawQuery != "" { + t.Errorf("requestV1(%q) changed path into another URL component: %v", tt.v1Path, r1.URL) + } + if r2.RequestURI != tt.raw+"?"+v2.RawQueryString || r2.URL.Path != tt.path || r2.URL.RawQuery != v2.RawQueryString || r2.URL.Scheme != "" || r2.URL.Host != "" { + t.Errorf("requestV2(%q) = URI %q, URL %v; want original target and decoded path %q", tt.raw, r2.RequestURI, r2.URL, tt.path) + } + for _, r := range []*http.Request{r1, r2} { + parsed, err := http.ReadRequest(bufio.NewReader(strings.NewReader("GET " + r.RequestURI + " HTTP/1.1\r\nHost: example.com\r\n\r\n"))) + if err != nil { + t.Fatalf("ReadRequest(%q) = %v", r.RequestURI, err) + } + if parsed.URL.Path != r.URL.Path || parsed.URL.RawQuery != r.URL.RawQuery || parsed.URL.Host != r.URL.Host { + t.Errorf("HTTP parser URL %v differs from adapter URL %v", parsed.URL, r.URL) + } + if err := parsed.Body.Close(); err != nil { + t.Error(err) + } + } + }) + } +} + +func TestRequestMergingAndOwnership(t *testing.T) { + event := v1RequestEvent() + event.Headers = map[string]string{"host": "example.com:443", "X-Mixed": "echo", "x-extra": "keep", "x-case": "from-single", "Content-Length": "999", "Transfer-Encoding": "chunked", "Trailer": "Digest"} + event.MultiValueHeaders = map[string][]string{"x-mixed": {"echo", "repeat", "repeat"}, "X-Case": {"upper"}, "x-case": {"lower"}, "Content-Length": {"3"}} + event.QueryStringParameters = map[string]string{"a": "echo", "b": "extra", "Case": "upper", "case": "lower"} + event.MultiValueQueryStringParameters = map[string][]string{"a": {"echo", "repeat", "repeat"}, "b": {"multi"}} + event.Body = "hello" + r, err := requestV1(t.Context(), event) + if err != nil { + t.Fatal(err) + } + if r.Host != "example.com:443" || r.Header.Get("Host") != "" || r.Header.Get("X-Extra") != "keep" { + t.Errorf("host/header promotion lost values: host=%q, headers=%v", r.Host, r.Header) + } + if !slices.Equal(r.Header.Values("X-Mixed"), []string{"echo", "repeat", "repeat"}) || !slices.Equal(r.Header.Values("X-Case"), []string{"upper", "lower", "from-single"}) { + t.Errorf("header merge = %v, want multivalue order and distinct single value", r.Header) + } + query := r.URL.Query() + if !slices.Equal(query["a"], []string{"echo", "repeat", "repeat"}) || !slices.Equal(query["b"], []string{"multi", "extra"}) || query.Get("Case") != "upper" || query.Get("case") != "lower" { + t.Errorf("query merge = %v, want repeated values and case-sensitive keys", query) + } + if r.ContentLength != 5 || r.Header.Get("Content-Length") != "5" || r.Header.Get("Transfer-Encoding") != "" || r.Header.Get("Trailer") != "" || r.TransferEncoding != nil || r.Trailer != nil { + t.Errorf("body framing = length %d, headers %v; want decoded length without transfer framing", r.ContentLength, r.Header) + } + r.Header["X-Mixed"][0] = "changed" + r.Header.Set("X-Extra", "changed") + query["a"][0] = "changed" + if event.MultiValueHeaders["x-mixed"][0] != "echo" || event.Headers["x-extra"] != "keep" || event.MultiValueQueryStringParameters["a"][0] != "echo" || event.Headers["Content-Length"] != "999" { + t.Error("request mutation changed caller-owned event") + } +} + +func TestRequestV2CookiesBodyAndMetadata(t *testing.T) { + event := v2RequestEvent() + event.Headers = map[string]string{"cookie": "ignored=1", "x-list": "a,b", "x-forwarded-for": "untrusted", "x-forwarded-host": "untrusted"} + event.Cookies = []string{"a=1", "b=2"} + event.Body, event.IsBase64Encoded = "AP8=", true + event.RequestContext.DomainName = "example.com" + event.RequestContext.HTTP.SourceIP = "2001:db8::1" + event.RequestContext.HTTP.Protocol = "HTTP/2.0" + r, err := requestV2(t.Context(), event) + if err != nil { + t.Fatal(err) + } + body, err := io.ReadAll(r.Body) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(body, []byte{0, 255}) || r.ContentLength != 2 { + t.Errorf("body = %x, length %d; want 00ff, length 2", body, r.ContentLength) + } + if r.Header.Get("Cookie") != "a=1; b=2" || len(r.Cookies()) != 2 || !slices.Equal(r.Header.Values("X-List"), []string{"a,b"}) { + t.Errorf("cookies/flattened header = %v, want cookies joined and comma value intact", r.Header) + } + if r.Host != "example.com" || r.RemoteAddr != "2001:db8::1" || r.Proto != "HTTP/2.0" || r.ProtoMajor != 2 || r.ProtoMinor != 0 || r.TLS != nil || r.Pattern != "" || r.GetBody != nil { + t.Error("gateway metadata did not produce the specified server request fields") + } + r.Header.Set("Cookie", "changed") + if event.Cookies[0] != "a=1" || event.Headers["cookie"] != "ignored=1" { + t.Error("cookie conversion aliased caller data") + } + event.Cookies = nil + r, err = requestV2(t.Context(), event) + if err != nil || r.Header.Get("Cookie") != "ignored=1" { + t.Fatalf("empty cookies fallback = (%v, %v), want existing Cookie header", r, err) + } +} + +func TestRequestRejectsInvalidTransportValues(t *testing.T) { + for _, tt := range []struct { + name string + change func(*events.APIGatewayV2HTTPRequest) + }{ + {name: "missing_api", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.APIID = "" }}, + {name: "relative_path", change: func(e *events.APIGatewayV2HTTPRequest) { e.RawPath = "relative" }}, + {name: "bad_escape", change: func(e *events.APIGatewayV2HTTPRequest) { e.RawPath = "/%xy" }}, + {name: "query_in_path", change: func(e *events.APIGatewayV2HTTPRequest) { e.RawPath = "/a?b" }}, + {name: "fragment_in_path", change: func(e *events.APIGatewayV2HTTPRequest) { e.RawPath = "/a#b" }}, + {name: "space_in_raw_path", change: func(e *events.APIGatewayV2HTTPRequest) { e.RawPath = "/a b" }}, + {name: "bad_method", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.HTTP.Method = "GET secret" }}, + {name: "bad_protocol", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.HTTP.Protocol = "secret" }}, + {name: "bad_base64", change: func(e *events.APIGatewayV2HTTPRequest) { e.Body, e.IsBase64Encoded = "aGVsbG8=secret", true }}, + {name: "bad_header_name", change: func(e *events.APIGatewayV2HTTPRequest) { e.Headers = map[string]string{"secret:name": "value"} }}, + {name: "bad_header_value", change: func(e *events.APIGatewayV2HTTPRequest) { e.Headers = map[string]string{"x": "secret\r\nx: bad"} }}, + {name: "bad_cookie", change: func(e *events.APIGatewayV2HTTPRequest) { e.Cookies = []string{"secret\x00"} }}, + {name: "multiple_hosts", change: func(e *events.APIGatewayV2HTTPRequest) { e.Headers = map[string]string{"Host": "a", "host": "b"} }}, + {name: "empty_host", change: func(e *events.APIGatewayV2HTTPRequest) { e.Headers = map[string]string{"host": ""} }}, + {name: "host_path", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.DomainName = "example.com/secret" }}, + {name: "host_credentials", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.DomainName = "secret@example.com" }}, + {name: "unbracketed_ipv6", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.DomainName = "2001:db8::1" }}, + {name: "bad_ipv6", change: func(e *events.APIGatewayV2HTTPRequest) { e.RequestContext.DomainName = "[secret]" }}, + } { + t.Run(tt.name, func(t *testing.T) { + event := v2RequestEvent() + tt.change(&event) + r, err := requestV2(t.Context(), event) + if err == nil || r != nil { + t.Fatalf("requestV2(invalid transport) = (%v, %v), want (nil, error)", r, err) + } + if strings.Contains(err.Error(), "secret") { + t.Errorf("conversion error exposed input: %v", err) + } + }) + } +} + +func TestRequestDefaultsAndContext(t *testing.T) { + parent, cancel := context.WithCancel(t.Context()) + defer cancel() + event := v1RequestEvent() + event.HTTPMethod = "custom" + r, err := requestV1(parent, event) + if err != nil { + t.Fatal(err) + } + if r.Method != "custom" || r.Host != "" || r.RemoteAddr != "" || r.Proto != "HTTP/1.1" || r.ProtoMajor != 1 || r.ProtoMinor != 1 || r.Body != http.NoBody || r.ContentLength != 0 { + t.Error("minimal typed event did not get the documented defaults") + } + cancel() + if !errors.Is(r.Context().Err(), context.Canceled) { + t.Errorf("request context error = %v, want canceled", r.Context().Err()) + } + if r, err := requestV1(nil, event); err == nil || r != nil { + t.Errorf("nil context = (%v, %v), want (nil, error)", r, err) + } + event.Path = "relative" + if r, err := requestV1(t.Context(), event); err == nil || r != nil { + t.Errorf("relative v1 path = (%v, %v), want (nil, error)", r, err) + } +} + +func TestRequestMatchesLocalHTTPServer(t *testing.T) { + type observed struct { + method, path, escapedPath, query, target, host, contentType, cookie, body string + length int64 + } + observations := make(chan observed, 2) + handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + data, err := io.ReadAll(r.Body) + if err != nil { + t.Error(err) + w.WriteHeader(http.StatusBadRequest) + return + } + observations <- observed{method: r.Method, path: r.URL.Path, escapedPath: r.URL.EscapedPath(), query: r.URL.RawQuery, target: r.RequestURI, host: r.Host, contentType: r.Header.Get("Content-Type"), cookie: r.Header.Get("Cookie"), body: string(data), length: r.ContentLength} + w.WriteHeader(http.StatusNoContent) + }) + server := httptest.NewServer(handler) + defer server.Close() + clientRequest, err := http.NewRequestWithContext(t.Context(), "POST", server.URL+"/a%2Fb?x=1&x=2", strings.NewReader("hello")) + if err != nil { + t.Fatal(err) + } + clientRequest.Header.Set("Content-Type", "text/plain") + clientRequest.Header.Set("Cookie", "a=1; b=2") + response, err := server.Client().Do(clientRequest) + if err != nil { + t.Fatal(err) + } + if err := response.Body.Close(); err != nil { + t.Fatal(err) + } + if response.StatusCode != http.StatusNoContent { + t.Fatalf("local HTTP server returned %d", response.StatusCode) + } + want := <-observations + event := v2RequestEvent() + event.RawPath, event.RawQueryString = "/a%2Fb", "x=1&x=2" + event.Body = "hello" + event.RequestContext.HTTP.Method = "POST" + event.Headers = map[string]string{"host": clientRequest.URL.Host, "content-type": "text/plain"} + event.Cookies = []string{"a=1", "b=2"} + r, err := requestV2(t.Context(), event) + if err != nil { + t.Fatal(err) + } + adapter, err := New(handler) + if err != nil { + t.Fatal(err) + } + if err := adapter.serveHTTP(httptest.NewRecorder(), r); err != nil { + t.Fatal(err) + } + if got := <-observations; got != want { + t.Errorf("adapter request = %+v, local HTTP request = %+v", got, want) + } +} + +type trackedBody struct { + io.Reader + closed bool +} + +func (b *trackedBody) Close() error { + b.closed = true + return nil +} + +func TestServeHTTPCleansUp(t *testing.T) { + for _, panicHandler := range []bool{false, true} { + var data bytes.Buffer + writer := multipart.NewWriter(&data) + part, err := writer.CreateFormFile("file", "example.txt") + if err != nil { + t.Fatal(err) + } + if _, err := io.WriteString(part, "force this file onto disk"); err != nil { + t.Fatal(err) + } + if err := writer.Close(); err != nil { + t.Fatal(err) + } + r := httptest.NewRequest("POST", "/", &data).WithContext(t.Context()) + r.Header.Set("Content-Type", writer.FormDataContentType()) + body := &trackedBody{Reader: r.Body} + r.Body = body + var handlerContext context.Context + var temporaryPath string + adapter, err := New(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + handlerContext = r.Context() + if err := r.ParseMultipartForm(0); err != nil { + t.Fatal(err) + } + file, err := r.MultipartForm.File["file"][0].Open() + if err != nil { + t.Fatal(err) + } + diskFile, ok := file.(*os.File) + if !ok { + t.Fatal("multipart test did not create a file on disk") + } + temporaryPath = diskFile.Name() + if err := file.Close(); err != nil { + t.Fatal(err) + } + if panicHandler { + panic("test panic") + } + })) + if err != nil { + t.Fatal(err) + } + var recovered any + func() { + defer func() { recovered = recover() }() + if err := adapter.serveHTTP(httptest.NewRecorder(), r); err != nil { + t.Error(err) + } + }() + if (recovered != nil) != panicHandler { + t.Errorf("serveHTTP panic = %v, want panic %v", recovered, panicHandler) + } + if !body.closed || handlerContext.Err() != context.Canceled || t.Context().Err() != nil { + t.Error("serveHTTP did not close body/cancel child, or canceled invocation parent") + } + if _, err := os.Stat(temporaryPath); !errors.Is(err, os.ErrNotExist) { + t.Errorf("multipart file remains after serveHTTP: %v", err) + } + } +} + +func FuzzRequestV2Target(f *testing.F) { + for _, path := range []string{"/", "/a%2Fb", "/a%252Fb", "/a?b", "//host/a", "/%ff", "/café", "/a//../b"} { + f.Add(path, "x=1&x=2") + } + f.Add("/", "x=%zz") + f.Add("/", "x=secret\r\ny=bad") + f.Fuzz(func(t *testing.T, path, query string) { + event := v2RequestEvent() + event.RawPath = path + event.RawQueryString = query + r, err := requestV2(t.Context(), event) + if err != nil { + if r != nil { + t.Fatal("invalid path returned a partial request") + } + return + } + target := path + if query != "" { + target += "?" + query + } + if r.RequestURI != target || r.URL.EscapedPath() != path || r.URL.RawQuery != query || r.URL.Host != "" || r.URL.Fragment != "" { + t.Fatalf("accepted path %q changed target: %v", path, r.URL) + } + parsed, err := http.ReadRequest(bufio.NewReader(strings.NewReader("GET " + target + " HTTP/1.1\r\nHost: example.com\r\n\r\n"))) + if err != nil { + t.Fatalf("accepted path %q cannot be parsed by net/http: %v", path, err) + } + if parsed.URL.Path != r.URL.Path || parsed.URL.RawQuery != query { + t.Fatalf("path %q decoded differently by HTTP parser: %q vs %q", path, parsed.URL.Path, r.URL.Path) + } + if err := parsed.Body.Close(); err != nil { + t.Error(err) + } + }) +} + +func FuzzRequestV1Query(f *testing.F) { + f.Add("x", "a,b", "a b") + f.Add("", "%", "&=") + f.Fuzz(func(t *testing.T, key, single, multi string) { + event := v1RequestEvent() + event.QueryStringParameters = map[string]string{key: single} + event.MultiValueQueryStringParameters = map[string][]string{key: {multi, multi}} + r, err := requestV1(t.Context(), event) + if err != nil { + t.Fatal(err) + } + want := []string{multi, multi} + if single != multi { + want = append(want, single) + } + if got := r.URL.Query()[key]; !slices.Equal(got, want) { + t.Fatalf("query values = %q, want %q", got, want) + } + }) +} From d9404797f9dae5ba028fa43c2928a0c7786c3cae Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 15:53:41 +0100 Subject: [PATCH 07/44] feat(edge): bound response encoding and unify invocation lifetime --- .github/workflows/go.yml | 49 +++++ README.md | 9 +- adapter.go | 3 +- docs/benchmarks.md | 19 ++ docs/decisions/0006-buffered-responses.md | 60 +++++-- docs/decisions/0007-response-metadata.md | 71 ++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 30 +++- docs/reviews/2026-09-14-plan-review.md | 145 +++++++++++++++ invocation.go | 54 ++++++ invocation_test.go | 171 ++++++++++++++++++ request.go | 22 --- request_test.go | 24 ++- response_encoding.go | 99 ++++++++++ response_encoding_test.go | 209 ++++++++++++++++++++++ 15 files changed, 916 insertions(+), 52 deletions(-) create mode 100644 .github/workflows/go.yml create mode 100644 docs/benchmarks.md create mode 100644 docs/decisions/0007-response-metadata.md create mode 100644 docs/reviews/2026-09-14-plan-review.md create mode 100644 invocation.go create mode 100644 invocation_test.go create mode 100644 response_encoding.go create mode 100644 response_encoding_test.go diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml new file mode 100644 index 0000000..8939ae9 --- /dev/null +++ b/.github/workflows/go.yml @@ -0,0 +1,49 @@ +name: Go + +on: + push: + pull_request: + +permissions: + contents: read + +jobs: + test: + runs-on: ubuntu-latest + strategy: + matrix: + go: ['1.27.0', '1.27.x'] + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: actions/setup-go@v7 + with: + go-version: ${{ matrix.go }} + cache-dependency-path: go.sum + - name: Check formatting + run: test -z "$(gofmt -l .)" + - name: Test with race detection + run: go test -race ./... + - name: Vet + run: go vet ./... + + lambda-build: + runs-on: ubuntu-latest + strategy: + matrix: + arch: [arm64, amd64] + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: actions/setup-go@v7 + with: + go-version: '1.27.x' + cache-dependency-path: go.sum + - name: Build for Lambda + env: + GOOS: linux + GOARCH: ${{ matrix.arch }} + CGO_ENABLED: '0' + run: go build ./... diff --git a/README.md b/README.md index 74da4e1..94e2d13 100644 --- a/README.md +++ b/README.md @@ -8,10 +8,13 @@ The root package is `edge`. This project is being designed and built from scratc The constructor, options, private event decoder, and request conversion are implemented. The decoder uses AWS Lambda Go v1.55.0 event structs and encoding/json/v2 directly, retaining authorizer JSON separately. Request conversion covers URLs, headers, body bytes, -and request lifetime. Buffered responses, public invocation methods, and identity -extraction are not implemented yet; the adapter cannot yet be registered as a -Lambda handler. +and request lifetime. A shared invocation scope now covers cleanup and cancellation; +private response encoding provides text/base64 selection and bounded direct JSON +v2 output. The buffered HTTP writer, gateway header conversion, public invocation +methods, and identity extraction are not implemented yet; the adapter cannot yet +be registered as a Lambda handler. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) - [Repository working agreement](AGENTS.md) +- [Initial codec benchmarks](docs/benchmarks.md) diff --git a/adapter.go b/adapter.go index 0a899a6..9da2c0e 100644 --- a/adapter.go +++ b/adapter.go @@ -1,6 +1,5 @@ // Package edge provides the foundation for adapting AWS API Gateway events to -// net/http handlers. Public invocation and response encoding are not yet -// implemented. +// net/http handlers. Public invocation is not yet implemented. package edge import ( diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..aa6f617 --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,19 @@ +# Performance baselines + +Initial response-codec measurements on 2026-09-14: Go 1.27.1, darwin/arm64, +Apple M2. Command: `go test -run '^$' -bench '^BenchmarkMarshalResponse$' +-benchmem -benchtime=200ms`. + +| Case | Time/op | Allocated bytes/op | Allocations/op | +| --- | ---: | ---: | ---: | +| Small JSON body in V2 envelope | 623 ns | 696 | 4 | +| Text body 1 KiB below 6 MiB | 5.71 ms | 18,875,672 | 11 | +| NUL body whose JSON escaping exceeds envelope limit | 5.25 ms | 27,559,434 | 19 | + +These are exploratory local samples, not CI thresholds or Lambda performance +claims. Allocated bytes/op measures cumulative allocations, not peak live memory. +The bounded destination retains at most 6 MiB, but JSON codec temporaries still +allocate. Do not advertise a 6 MiB total-process or peak-memory guarantee. + +Raw/typed adapter comparisons, binary response benchmarks, and peak-memory +measurements follow when the complete response and invocation paths are wired. diff --git a/docs/decisions/0006-buffered-responses.md b/docs/decisions/0006-buffered-responses.md index 16d9d1c..3eacfe0 100644 --- a/docs/decisions/0006-buffered-responses.md +++ b/docs/decisions/0006-buffered-responses.md @@ -1,6 +1,11 @@ # 0006: Buffered HTTP responses and invocation completion -Status: proposed; awaiting user review. +Status: accepted with the 2026-09-14 review refinements; implementation underway. + +Reviewed against the implemented decoder/request layer on 2026-09-14. The +[plan review](../reviews/2026-09-14-plan-review.md) was approved by the user and its +refinements are incorporated below. Exact metadata budgets and the initial V2 +joinable-field set remain separate specifications before dependent implementation. ## Question @@ -60,17 +65,23 @@ format, and handle features or failures that a buffered invocation cannot carry? Canonicalize header names and retain all values; use deterministic traversal for differently cased keys supplied by direct Header map assignment. - Derive Content-Length from the buffered bytes when absent. If explicitly - supplied, require one valid nonnegative decimal length and enforce it: an + present with a nil/empty slice, suppress inference. Otherwise require one valid + nonnegative decimal length and enforce it: an overflowing Write returns http.ErrContentLength; a mismatch at completion is an invocation error. Transport faults remain faults even if the handler ignores a Write error. Ordinary application 4xx/5xx responses still return nil Go error. - For HEAD, accept/count writes and retain at most the sniffing prefix, but emit - no body. Honor a supplied representation length; otherwise infer a length only + no body. Capture the original request method before application code runs. + Honor a supplied representation length; otherwise infer a length only when the handler actually wrote representation bytes. Do not require HEAD's declared length to match an omitted representation body. -- For 204/304, emit no body and return http.ErrBodyNotAllowed from Write. That +- For 204/205/304, emit no body and return http.ErrBodyNotAllowed from a nonempty + Write. A zero-length Write commits an implicit response and returns (0, nil). + That error alone does not invalidate the committed bodyless response. Suppress - Content-Length for both statuses and Content-Type for 304, matching Go's server. + Content-Length for these statuses and Content-Type for 304. Bodyless 205 follows + RFC 9110 and intentionally differs from Go 1.27.1's body-allowed helper. + [205 semantics](https://www.rfc-editor.org/rfc/rfc9110.html#section-15.3.6) ### Capabilities the buffered adapter cannot provide @@ -92,10 +103,12 @@ format, and handle features or failures that a buffered invocation cannot carry? or responses. Always set an explicit final status and body representation. - V1 uses MultiValueHeaders for every response header, including each Set-Cookie. Leave the redundant single-value Headers map empty. -- V2 moves Set-Cookie values into Cookies. Combine other repeated header values - with a comma and space into Headers, without deduplicating or interpreting - commas already inside a value. Leave MultiValueHeaders empty. V2 cannot retain - general header line boundaries; applications needing those must choose V1. +- V2 moves Set-Cookie values into Cookies. Join repeated values with a comma and + space only for a documented, audited set of list-valued fields. Reject repeated + singleton or unknown fields. Preserve order and commas inside individual + values. Leave MultiValueHeaders empty. Specify the initial list separately; + V2 cannot retain general header line boundaries, so applications requiring + them must choose V1. - Emit text directly only when bytes are valid UTF-8, Content-Encoding is absent or identity, and the media type is textual: text/*, application/json, application/xml, application/javascript, application/x-www-form-urlencoded, @@ -108,13 +121,17 @@ format, and handle features or failures that a buffered invocation cannot carry? ### Bounds and completion -- Initially cap stored response body bytes at 6,000,000, an intentionally - conservative interpretation of AWS's documented 6 MB limit. Reject a Write +- Initially cap stored response body bytes at 6,291,456 (6 MiB). AWS explicitly + defines its documented MB unit as 1,024 KB in the + [Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html). + This corrects the draft's earlier 6,000,000-byte interpretation. Reject a Write crossing the cap without retaining a partial write, and record a sticky fault. Check base64 expansion before allocating the encoded body. No new public size-tuning option is proposed for the first implementation. -- Invoke also checks the final serialized envelope against 6,000,000 bytes, - including JSON overhead. A body below the cap can still fail this final check. +- Invoke uses json/v2.MarshalWrite with a bounded destination to enforce + 6,291,456 bytes including JSON overhead while retaining encoded output. + A body below the cap can still fail this check. Preflight committed metadata + bytes and entry counts before copying; specify those exact budgets separately. A typed method cannot measure the exact envelope produced by its caller's serializer without violating the agreed JSON-free typed boundary; document that the caller/runtime owns that final limit. Apply the common body/expansion @@ -123,12 +140,18 @@ format, and handle features or failures that a buffered invocation cannot carry? - Propagate handler panics to the Lambda runtime after cleanup; do not convert them into successful HTTP 500 responses. Direct Go callers retain normal panic behavior. Do not add logging to the adapter. -- Before returning success, check the invocation parent context. If it is +- One invocation scope covers conversion, identity, serving, finalization, and + cleanup. Reject an already-canceled parent before running application code. + Before returning success, check the invocation parent context. If it is canceled, return its error and no usable response. Check the parent, not the - child context that serveHTTP cancels during normal cleanup. This deliberately + child context canceled during normal cleanup. This deliberately makes cancellation win over a just-completed HTTP response when already known. - Cleanup failures are invocation failures even after a handler produced a response. Keep diagnostics free of file paths, bodies, tokens, and header values. + Preserve a primary operation error and join sanitized cleanup failures. Parent + cancellation replaces only an otherwise successful result. Preserve errors.Is + for context and standard HTTP errors; review exported adapter error categories + separately before publishing the invocation methods. ### Multipart cleanup limit discovered during implementation @@ -165,8 +188,11 @@ multivalue headers, v2 cookies, unsupported capability checks, panic cleanup, parent cancellation, and cleanup failure propagation. Verify raw registration through lambda.NewHandler(adapter).Invoke and typed method registration through the SDK wrapper once the remaining gateway-identity path permits public wiring. +Add explicit 205 divergence, zero-length writes, suppressed inferred length, +exact envelope boundaries, metadata bounds, and sequential/concurrent isolation. ## Resolution -Pending user approval. No buffered writer or response encoding is implemented -under this proposal yet. +Accepted by the user on 2026-09-14 with the plan-review enhancements. Implement +settled behavior now; resolve the separately identified metadata, header-list, +error API, and identity details before their dependent implementation. diff --git a/docs/decisions/0007-response-metadata.md b/docs/decisions/0007-response-metadata.md new file mode 100644 index 0000000..7819b56 --- /dev/null +++ b/docs/decisions/0007-response-metadata.md @@ -0,0 +1,71 @@ +# 0007: Response metadata budgets and V2 list fields + +Status: proposed; awaiting user review. + +## Question and evidence + +Decision 0006 and the accepted plan review require bounded metadata snapshots +and an audited list of fields whose repeated values can be joined for V2. They +deliberately left the numeric budgets and initial field set unspecified. + +The existing request helper preserves multiple values and canonicalizes keys; +the new response path must not inherit its unbounded copying or assume every +field is a list. AWS V2 provides a single string per header and a separate Cookies +array; V1 supports multiple values. +[AWS payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) + +HTTP does not define one universal field-size limit. A library metadata budget +is our resource policy, not a claim about gateway quotas. +[HTTP field limits](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.4) + +## Recommendation + +Before sorting, canonicalizing, or cloning a committed Header map, allow at most +64 KiB (65,536 bytes) of names plus values and 1,024 entries. Count each original +map key's bytes once and every value's bytes. Count max(1, len(values)) entries +per original key, including empty slices, so empty values cannot bypass the +entry bound. Include Set-Cookie and fields later stripped from the response. +Check incrementally without overflowing counters. Neither limit is configurable +in the first implementation. + +The budget protects adapter-created copies; it cannot prevent an application +from allocating its own Header map. The separate 6 MiB serialized-envelope +limit still applies to Invoke. These metadata limits apply equally to raw and +typed entry points. Generated Content-Type/Content-Length have small fixed +overhead outside the application snapshot budget. + +For V2, initially allow comma-space joining for exactly these repeated fields: + +| Fields | Source | +| --- | --- | +| Accept-Ranges, Allow, Content-Encoding, Content-Language, Vary, WWW-Authenticate | [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html) | +| Cache-Control | [RFC 9111 section 5.2](https://www.rfc-editor.org/rfc/rfc9111.html#section-5.2) | +| Access-Control-Allow-Headers, Access-Control-Allow-Methods, Access-Control-Expose-Headers | [Fetch field grammar](https://fetch.spec.whatwg.org/#http-new-header-syntax) | + +Each has a list grammar. Preserve order, duplicates, and embedded commas without +parsing field-specific parameters. This authorizes joining, not semantic +validation of every field. A single value for any valid field passes through. +Set-Cookie keeps its separate Cookies treatment. Reject other repeated fields, +including Location, Content-Type, Access-Control-Allow-Origin, and unknown +extension fields; do not silently select the first or last value. Further list +fields can be added after checking their definitions and consumer requirements. + +WWW-Authenticate allows multiple challenges, but some clients handle separate +lines better. V2 cannot promise separate lines; applications requiring them need +V1. Hop-by-hop fields are removed under decision 0006 before V2 joining. + +## Alternatives and consequences + +- Using the full payload budget for metadata permits disproportionate map/slice + overhead. A byte limit alone does not constrain large slices of empty values. +- Much smaller limits could unnecessarily constrain cookies and application + metadata. The proposed 64 KiB/1,024 bounds are library defaults chosen for a + generous finite snapshot, not documented AWS allowances. +- An exhaustive registry or public join callback adds maintenance or API surface + before there is a demonstrated need. The initial set covers common response, + cache, authentication, and CORS fields; other repeated fields fail visibly. + +## Resolution + +Pending user approval. The response snapshot and V2 header conversion depend on +this decision. Bounded envelope encoding and invocation lifetime work do not. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index a10e0c6..f500572 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -14,4 +14,5 @@ Keep related decisions small enough to review and implement independently. | [0003: Event decoding](0003-event-decoding.md) | Accepted | How should SDK types, dispatch, and wire validation work together? | | [0004: Typed events](0004-typed-events.md) | Accepted | How can already typed events bypass edge's envelope JSON work? | | [0005: HTTP requests](0005-http-request-translation.md) | Accepted | How should raw and typed events become ordinary server requests? | -| [0006: Buffered responses](0006-buffered-responses.md) | Proposed | How should HTTP responses, unsupported capabilities, and completion failures behave? | +| [0006: Buffered responses](0006-buffered-responses.md) | Accepted | How should HTTP responses, unsupported capabilities, and completion failures behave? | +| [0007: Response metadata](0007-response-metadata.md) | Proposed | What metadata budgets and initial V2 joinable-field set should apply? | diff --git a/docs/plan.md b/docs/plan.md index 6778fc9..a73dd7b 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -91,10 +91,24 @@ request URLs, header/query merging, cookies, body handling, invocation metadata, and request ownership. Shared request conversion is implemented with local HTTP comparisons, race tests, fuzzing, vet, and Lambda-target builds passing. -Current review: [decision 0006](decisions/0006-buffered-responses.md) proposes +Accepted: [decision 0006](decisions/0006-buffered-responses.md) specifies buffered response commitment/encoding, unsupported features, payload bounds, invocation completion, and the documented multipart cleanup limit. +The accepted [2026-09-14 plan review](reviews/2026-09-14-plan-review.md) requires +response-contract refinements and completing a runnable adapter, including the +minimum gateway identity foundation, before broader authentication work. These +enhancements are incorporated into decision 0006. Concrete metadata/header +policies, exported error categories, and identity APIs remain separate reviews. + +Implemented under decision 0006: bounded direct JSON v2 envelope encoding, +text/base64 body selection, and a shared invocation scope that preserves primary +errors, cleans up on failures/panics, and checks parent cancellation. The HTTP +writer and gateway header conversion await the concrete policy in +[decision 0007](decisions/0007-response-metadata.md). Public wiring still awaits +the minimum identity contracts. [Initial codec benchmarks](benchmarks.md) record +allocation baselines; they do not establish a peak-memory bound. + Research and present a contract covering: - Raw invocation API, constructor/options, configuration validation, and errors. @@ -123,16 +137,30 @@ presenting an entire frozen public API at once. - [x] Implement the private decoder and typed semantic validators with direct JSON v2, SDK types, opaque authorizer data, fixtures, and fuzzing. - [x] Implement shared HTTP request conversion and request-lifetime cleanup. +- [ ] Implement buffered responses and move cleanup into a shared invocation scope. +- [x] Implement bounded envelope/body encoding and shared invocation cleanup. +- [ ] Implement the minimum reviewed identity/context and native gateway producers + needed to honor WithGatewayIdentity(true) before publishing invocation. - [ ] Implement raw invocation after the remaining transport contracts are reviewed. - [ ] Implement the accepted typed entry points after shared HTTP translation review, documenting upstream codec ownership. - Implement the adapter from scratch using JSON v2 directly. - Test raw invocation through the actual pinned AWS Lambda Go SDK boundary. +- Add runnable raw/typed registration examples, sequential/concurrent invocation + isolation checks, and ServeMux, Redirect, ServeContent/Range, compression, and + cookie integration cases. - Exercise documented event fixtures and adversarial/malformed input. - Compare relevant request/response behavior with real `net/http` serving, documenting unavoidable transport differences. - Fuzz decoding and path/query conversion; add targeted allocation benchmarks. - Run appropriate tests, race detection, vet, and Lambda-target build checks. +- Put those checks in Go 1.27 CI and establish raw/typed, binary, and near-limit + allocation baselines before setting performance thresholds. + +CI configuration is present for Go 1.27.0/latest 1.27 patch tests, race detection, +vet, and Lambda-target builds. It has not run on GitHub yet. Local Go 1.27.1 race +tests, vet, formatting checks, and both target builds pass for the initial +encoding/lifetime implementation. Acceptance: the same application handler behaves as specified across all three event cases; no auth or codec behavior depends on accidental SDK dispatch. diff --git a/docs/reviews/2026-09-14-plan-review.md b/docs/reviews/2026-09-14-plan-review.md new file mode 100644 index 0000000..26b9241 --- /dev/null +++ b/docs/reviews/2026-09-14-plan-review.md @@ -0,0 +1,145 @@ +# Plan review after request translation + +Status: accepted enhancements (2026-09-14). + +Reviewed the plan, decisions 0001–0006, and the implemented constructor, decoder, +request conversion, and tests on `work/edge-foundation`. The user approved these +enhancements with “go with your recommended enhancements.” They refine decision +0006 and milestone sequencing, preserving accepted AWS-type reuse, JSON-free +typed entry points, direct JSON v2, and opt-in gateway identity. Exact metadata +budgets, the initial V2 joinable-field set, exported errors, and identity APIs +still require the concrete specifications identified below. + +## 1. Make response limits precise and enforce them during encoding + +AWS defines its MB unit as 1,024 KB. The buffered response limit is therefore +6 MiB, or 6,291,456 bytes, including the serialized envelope. The original +6,000,000-byte draft cap leaves capacity unused; decision 0006 now records the +correct documented unit. A lower application budget could be a future option. +[AWS Lambda quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html) + +Recommend using `encoding/json/v2.MarshalWrite` with a bounded destination for +raw invocation output. Stop retaining encoded output at the limit, rather than +allocating an oversized complete envelope before rejecting it. Keep body-write +and base64-expansion checks. Preflight the committed header/cookie data before +deep copying it: bound retained field bytes and entry counts as well as the body. +Set the exact metadata budget in the response contract before implementation; +do not borrow an unrelated AWS request-header quota. +[MarshalWrite](https://pkg.go.dev/encoding/json/v2#MarshalWrite) + +This bounds retained adapter output, not all codec temporaries or memory already +allocated by an application. Measure peak allocations near the limit. Typed +methods stay JSON-free and continue to delegate the exact final serialized +envelope limit to their caller/runtime. Test exact boundaries, escaped strings, +binary expansion, large cookies, and many empty header entries. + +## 2. Do not combine arbitrary repeated V2 headers + +Decision 0006 currently proposes comma-joining every field except Set-Cookie. +That can change meaning: two Location values cannot safely become one URI. +HTTP field definitions determine whether comma-separated lists are supported; +Set-Cookie requires separate treatment. +[RFC 9110 field ordering](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.3) +[Location](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.2) + +Recommend preserving V1 multivalue output and V2 Cookies as proposed. For V2, +join repeated values only for a documented, audited set of list-valued fields; +reject repeated singleton or unknown fields with an invocation error. A single +value passes through without splitting embedded commas. Preserve value order. +Specify the initial set before coding it; do not add a public policy callback +without a demonstrated consumer need. + +The alternative of taking the first or last value silently loses data. Blind +joining changes semantics. Strict rejection makes V2's limitation visible; +applications can explicitly construct a field value according to its definition, +or choose V1 when separate field values are required. +[AWS HTTP API response formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) + +## 3. Specify a small response behavior matrix before writing the writer + +Retain the existing snapshot, binary encoding, and unsupported-capability +direction. Add these cases to the contract and differential tests: + +- A zero-length Write commits an implicit response but returns `(0, nil)`, + including after 204/304. Only a nonempty forbidden-body write returns + `http.ErrBodyNotAllowed`. +- An explicitly present nil/empty Content-Length slice suppresses inferred + length, just as the draft already specifies for Content-Type. Distinguish this + from an explicit empty string, which is an invalid length. +- Capture the original request method for HEAD response handling so a handler's + mutation of its request cannot change transport finalization. +- Add 205 Reset Content. Recommend suppressing its body and rejecting nonempty + writes with `http.ErrBodyNotAllowed`. RFC 9110 prohibits response content here, + while the inspected Go 1.27.1 server's body-allowed helper does not exclude 205. + Record this as an intentional standards-based difference, not Go parity. + +Use the installed Go 1.27.1 server as the implementation reference and the HTTP +specification to identify deliberate differences. The draft's 200–599 status +range, sticky errors, and unsupported 1xx/trailer behavior are also stricter than +ordinary Go serving; keep those differences explicit in consumer documentation. +[Go server implementation](https://go.dev/src/net/http/server.go) +[Go body/status handling](https://go.dev/src/net/http/transfer.go) +[205 semantics](https://www.rfc-editor.org/rfc/rfc9110.html#section-15.3.6) + +## 4. Give the whole invocation one lifetime + +Currently `serveHTTP` creates the child context. The planned identity step runs +before this helper, so it would not share that scope. Recommend moving ownership +to a shared invocation coordinator covering conversion, identity production, +handler execution, response finalization, and cleanup. Pass that scope through +the existing conversion functions instead of adding a second handler-only scope. + +Reject an already-canceled parent before invoking application code. Continue to +check the parent before returning success; ordinary child cancellation during +cleanup must not turn success into a context error. Ensure identity failures, +writer faults, and panics all reach cleanup. Preserve panic propagation. + +Define simultaneous-error behavior before implementation: retain a primary +conversion/identity/response fault, join sanitized cleanup errors, and let parent +cancellation replace only an otherwise successful result. Preserve `errors.Is` +for context and standard HTTP errors. Review a small set of programmatic adapter +error categories before publishing the invocation methods, so applications need +not match diagnostic strings. Exact exported names remain a separate API detail. + +Retain the documented multipart limitation: a form first parsed on a separate +request copy is that middleware's cleanup responsibility. Eager parsing would +change handler behavior and does not belong in transport conversion. +[Request.WithContext](https://pkg.go.dev/net/http#Request.WithContext) + +## 5. Make the next deliverable a complete adapter + +The private decoder and request conversion already have substantive unit, +differential, race, and fuzz coverage. The next major uncertainty is composition, +not another private conversion layer in isolation. + +Recommend this sequence after the response refinements are resolved: + +1. Implement the buffered writer and response conversion. +2. Review and implement the minimum identity/context and native gateway-producer + contracts needed to honor `WithGatewayIdentity(true)`. Keep the option's + accepted default and do not publish a callable adapter that silently ignores + an enabled option. +3. Wire Invoke, HandleV1, and HandleV2 through the common invocation coordinator. +4. Add runnable raw and typed Lambda registration examples and integration tests + through the pinned SDK's `lambda.NewHandler` boundary. +5. Continue with Cognito/JWKS resilience, broader identity composition, + authorization, and consumer helpers as already planned. + +For the first complete adapter, test sequential reuse and concurrent invocations, +raw/typed HTTP equivalence where input fidelity permits it, cancellation/cleanup, +and absence of cross-invocation body, header, or identity leakage. Exercise real +handlers such as ServeMux, Redirect, ServeContent with Range, compression, and +cookie round trips. Some handlers may expose an intentional transport limit; +document that outcome instead of weakening the contract to make a test pass. + +Add Go 1.27 CI for tests, race detection, vet, and Linux arm64/amd64 builds. Add +allocation benchmarks comparing raw and typed paths, plus binary and near-limit +responses. Establish measured baselines before choosing performance thresholds. +These checks complement the existing fuzzing and make a usable first release +the next concrete acceptance milestone. + +## Resolution + +Accepted by the user on 2026-09-14. Decision 0006 and the implementation plan +incorporate the refinements. Implementation may proceed for settled behavior; +the remaining concrete policy/API details are reviewed separately. diff --git a/invocation.go b/invocation.go new file mode 100644 index 0000000..aa09322 --- /dev/null +++ b/invocation.go @@ -0,0 +1,54 @@ +package edge + +import ( + "context" + "errors" + "io" + "net/http" +) + +// invocation owns the transport-created request and original body. The context +// is passed separately so every stage uses the same explicit lifetime. +type invocation struct { + request *http.Request + body io.ReadCloser +} + +// ownRequest must run immediately after successful conversion, before identity +// or application code can fail or replace the request's Body field. +func (inv *invocation) ownRequest(r *http.Request) { + inv.request = r + inv.body = r.Body +} + +// withInvocation scopes conversion, identity, serving, and finalization. A panic +// passes through after cleanup. The caller must discard its result on error. +func withInvocation(parent context.Context, run func(context.Context, *invocation) error) (err error) { + if parent == nil { + return errors.New("edge: nil invocation context") + } + if err := parent.Err(); err != nil { + return err + } + ctx, cancel := context.WithCancel(parent) + defer cancel() + var inv invocation + defer func() { + if inv.body != nil { + if closeErr := inv.body.Close(); closeErr != nil { + err = errors.Join(err, errors.New("edge: request body cleanup failed")) + } + } + if inv.request != nil && inv.request.MultipartForm != nil { + if removeErr := inv.request.MultipartForm.RemoveAll(); removeErr != nil { + err = errors.Join(err, errors.New("edge: multipart cleanup failed")) + } + } + // Normal child cancellation is not a failed invocation. Preserve an + // existing operation/cleanup error when the parent also canceled. + if err == nil { + err = parent.Err() + } + }() + return run(ctx, &inv) +} diff --git a/invocation_test.go b/invocation_test.go new file mode 100644 index 0000000..f36a540 --- /dev/null +++ b/invocation_test.go @@ -0,0 +1,171 @@ +package edge + +import ( + "context" + "errors" + "io" + "net/http" + "strings" + "testing" + "time" +) + +func TestInvocationRejectsInvalidContextBeforeRunning(t *testing.T) { + canceled, cancel := context.WithCancel(t.Context()) + cancel() + expired, stop := context.WithDeadline(t.Context(), time.Unix(0, 0)) + defer stop() + for _, tt := range []struct { + name string + ctx context.Context + want error + }{ + {name: "nil"}, + {name: "canceled", ctx: canceled, want: context.Canceled}, + {name: "expired", ctx: expired, want: context.DeadlineExceeded}, + } { + t.Run(tt.name, func(t *testing.T) { + ran := false + err := withInvocation(tt.ctx, func(context.Context, *invocation) error { + ran = true + return nil + }) + if ran || err == nil || tt.want != nil && !errors.Is(err, tt.want) { + t.Errorf("withInvocation(%s) ran=%v err=%v, want no run and %v", tt.name, ran, err, tt.want) + } + }) + } +} + +func TestInvocationScopeIncludesAllStages(t *testing.T) { + type contextKey struct{} + parent := context.WithValue(t.Context(), contextKey{}, "invocation value") + var scope context.Context + var original *trackedBody + var replacement *trackedBody + err := withInvocation(parent, func(ctx context.Context, inv *invocation) error { + scope = ctx + // Conversion, identity, handler, and finalization all execute under ctx. + r, err := requestV2(ctx, v2RequestEvent()) + if err != nil { + return err + } + original = &trackedBody{Reader: r.Body} + r.Body = original + inv.ownRequest(r) + if r.Context() != ctx || ctx.Err() != nil || ctx.Value(contextKey{}) != "invocation value" { + t.Error("converted request lost invocation scope/value") + } + replacement = &trackedBody{Reader: strings.NewReader("application-owned")} + r.Body = replacement + if original.closed || ctx.Err() != nil { + t.Error("scope/body closed before finalization") + } + _, err = marshalResponse(struct{ StatusCode int }{StatusCode: http.StatusOK}) + return err + }) + if err != nil { + t.Fatal(err) + } + if scope.Err() != context.Canceled || parent.Err() != nil || !original.closed || replacement.closed { + t.Errorf("completed scope: child=%v parent=%v original closed=%v replacement closed=%v", scope.Err(), parent.Err(), original.closed, replacement.closed) + } +} + +type failingCloseBody struct { + io.Reader + closed bool +} + +func (b *failingCloseBody) Close() error { + b.closed = true + return errors.New("private body credential or path") +} + +func TestInvocationErrorPrecedence(t *testing.T) { + operationErr := errors.New("operation failed") + for _, tt := range []struct { + name string + operation error + cancel bool + failCleanup bool + }{ + {name: "success"}, + {name: "late_cancel", cancel: true}, + {name: "operation_failure", operation: operationErr}, + {name: "operation_and_cancel", operation: operationErr, cancel: true}, + {name: "cleanup_failure", failCleanup: true}, + {name: "cleanup_and_cancel", failCleanup: true, cancel: true}, + {name: "operation_and_cleanup", operation: operationErr, failCleanup: true}, + {name: "all_failures", operation: operationErr, failCleanup: true, cancel: true}, + {name: "standard_http_error", operation: http.ErrContentLength}, + } { + t.Run(tt.name, func(t *testing.T) { + parent, cancel := context.WithCancel(t.Context()) + defer cancel() + var scope context.Context + body := &failingCloseBody{Reader: http.NoBody} + err := withInvocation(parent, func(ctx context.Context, inv *invocation) error { + scope = ctx + if tt.failCleanup { + inv.ownRequest(&http.Request{Body: body}) + } + if tt.cancel { + cancel() + } + return tt.operation + }) + wantError := tt.operation != nil || tt.failCleanup || tt.cancel + if (err != nil) != wantError { + t.Fatalf("withInvocation() = %v, want error %v", err, wantError) + } + if tt.operation != nil && !errors.Is(err, tt.operation) { + t.Errorf("error %v does not preserve operation %v", err, tt.operation) + } + wantCanceled := tt.cancel && tt.operation == nil && !tt.failCleanup + if errors.Is(err, context.Canceled) != wantCanceled { + t.Errorf("error %v matches cancellation=%v, want %v", err, errors.Is(err, context.Canceled), wantCanceled) + } + if tt.failCleanup && (!body.closed || strings.Contains(err.Error(), "private")) { + t.Errorf("cleanup closed=%v err=%v, want sanitized failure", body.closed, err) + } + if scope.Err() != context.Canceled { + t.Errorf("child after return = %v, want canceled", scope.Err()) + } + }) + } +} + +func TestInvocationCleansUpBeforeHandler(t *testing.T) { + identityErr := errors.New("identity failed") + body := &trackedBody{Reader: http.NoBody} + var scope context.Context + err := withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + scope = ctx + inv.ownRequest(&http.Request{Body: body}) + return identityErr + }) + if !errors.Is(err, identityErr) || !body.closed || scope.Err() != context.Canceled { + t.Errorf("pre-handler failure: err=%v closed=%v context=%v", err, body.closed, scope.Err()) + } +} + +func TestInvocationPreservesPanic(t *testing.T) { + panicValue := new("application panic") + body := &failingCloseBody{Reader: http.NoBody} + var scope context.Context + var recovered any + func() { + defer func() { recovered = recover() }() + if err := withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + scope = ctx + inv.ownRequest(&http.Request{Body: body}) + panic(panicValue) + }); err != nil { + t.Errorf("panicking invocation returned %v", err) + } + }() + if recovered != panicValue || !body.closed || scope.Err() != context.Canceled { + t.Errorf("panic result: value=%v closed=%v context=%v", recovered, body.closed, scope.Err()) + } +} diff --git a/request.go b/request.go index 1ded1ec..f5a792a 100644 --- a/request.go +++ b/request.go @@ -178,28 +178,6 @@ func newRequest(ctx context.Context, r *http.Request, body string, binary bool) return r.WithContext(ctx), nil } -// serveHTTP owns the transport-created body and the handler's request lifetime. -// Identity production belongs before this call. Response encoding belongs after -// it. Cleanup runs on panic too; deciding panic/result policy is the invoker's job. -func (a *Adapter) serveHTTP(w http.ResponseWriter, r *http.Request) (err error) { - ctx, cancel := context.WithCancel(r.Context()) - r = r.WithContext(ctx) - body := r.Body - defer func() { - cancel() - if closeErr := body.Close(); closeErr != nil { - err = errors.Join(err, errors.New("edge: request body cleanup failed")) - } - if r.MultipartForm != nil { - if removeErr := r.MultipartForm.RemoveAll(); removeErr != nil { - err = errors.Join(err, errors.New("edge: multipart cleanup failed")) - } - } - }() - a.handler.ServeHTTP(w, r) - return nil -} - // validToken implements the shared HTTP method/field-name token grammar. func validToken(s string) bool { if s == "" { diff --git a/request_test.go b/request_test.go index a127365..8ec64a6 100644 --- a/request_test.go +++ b/request_test.go @@ -254,7 +254,13 @@ func TestRequestMatchesLocalHTTPServer(t *testing.T) { if err != nil { t.Fatal(err) } - if err := adapter.serveHTTP(httptest.NewRecorder(), r); err != nil { + err = withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + r = r.WithContext(ctx) + inv.ownRequest(r) + adapter.handler.ServeHTTP(httptest.NewRecorder(), r) + return nil + }) + if err != nil { t.Fatal(err) } if got := <-observations; got != want { @@ -272,7 +278,7 @@ func (b *trackedBody) Close() error { return nil } -func TestServeHTTPCleansUp(t *testing.T) { +func TestInvocationCleansUpMultipart(t *testing.T) { for _, panicHandler := range []bool{false, true} { var data bytes.Buffer writer := multipart.NewWriter(&data) @@ -319,18 +325,24 @@ func TestServeHTTPCleansUp(t *testing.T) { var recovered any func() { defer func() { recovered = recover() }() - if err := adapter.serveHTTP(httptest.NewRecorder(), r); err != nil { + err := withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + r = r.WithContext(ctx) + inv.ownRequest(r) + adapter.handler.ServeHTTP(httptest.NewRecorder(), r) + return nil + }) + if err != nil { t.Error(err) } }() if (recovered != nil) != panicHandler { - t.Errorf("serveHTTP panic = %v, want panic %v", recovered, panicHandler) + t.Errorf("invocation panic = %v, want panic %v", recovered, panicHandler) } if !body.closed || handlerContext.Err() != context.Canceled || t.Context().Err() != nil { - t.Error("serveHTTP did not close body/cancel child, or canceled invocation parent") + t.Error("invocation did not close body/cancel child, or canceled parent") } if _, err := os.Stat(temporaryPath); !errors.Is(err, os.ErrNotExist) { - t.Errorf("multipart file remains after serveHTTP: %v", err) + t.Errorf("multipart file remains after invocation: %v", err) } } } diff --git a/response_encoding.go b/response_encoding.go new file mode 100644 index 0000000..fa63035 --- /dev/null +++ b/response_encoding.go @@ -0,0 +1,99 @@ +package edge + +import ( + "encoding/base64" + "encoding/json/v2" + "errors" + "mime" + "net/http" + "strings" + "unicode/utf8" +) + +// Lambda's documented MB unit is binary. This is the complete buffered payload +// budget, not an allowance for a body plus an uncounted JSON envelope. +const maxResponseBytes = 6 * 1024 * 1024 + +var errResponseTooLarge = errors.New("edge: response exceeds buffered payload limit") + +// marshalResponse is only for the raw invocation path. Typed entry points leave +// envelope serialization, and its exact final size, to their caller. +func marshalResponse(response any) ([]byte, error) { + var out responseBuffer + if err := json.MarshalWrite(&out, response); err != nil { + if errors.Is(err, errResponseTooLarge) { + return nil, errResponseTooLarge + } + // Codec errors can include application values. Do not expose that chain. + return nil, errors.New("edge: response JSON encoding failed") + } + return out.data, nil +} + +// responseBuffer bounds both length and backing capacity. MarshalWrite can still +// allocate temporary codec storage; this is not a total-memory guarantee. +type responseBuffer struct { + data []byte +} + +func (b *responseBuffer) Write(p []byte) (int, error) { + if len(p) > maxResponseBytes-len(b.data) { + return 0, errResponseTooLarge + } + n := len(b.data) + len(p) + if n > cap(b.data) { + capacity := min(maxResponseBytes, max(n, 2*cap(b.data), 512)) + data := make([]byte, len(b.data), capacity) + copy(data, b.data) + b.data = data + } + b.data = append(b.data, p...) + return len(p), nil +} + +// encodeResponseBody receives the finalized, canonical response headers. It +// changes only the gateway envelope representation, never the content bytes. +func encodeResponseBody(body []byte, header http.Header) (string, bool, error) { + if len(body) > maxResponseBytes { + return "", false, errResponseTooLarge + } + if len(body) == 0 { + return "", false, nil + } + if textualResponse(header) && utf8.Valid(body) { + return string(body), false, nil + } + if base64.StdEncoding.EncodedLen(len(body)) > maxResponseBytes { + return "", false, errResponseTooLarge + } + return base64.StdEncoding.EncodeToString(body), true, nil +} + +func textualResponse(header http.Header) bool { + encodings := header.Values("Content-Encoding") + if len(encodings) != 0 && (len(encodings) != 1 || !strings.EqualFold(strings.TrimSpace(encodings[0]), "identity")) { + return false + } + types := header.Values("Content-Type") + if len(types) != 1 { + return false + } + mediaType, _, err := mime.ParseMediaType(types[0]) + if err != nil { + return false + } + // ParseMediaType also accepts a bare token (useful for Content-Disposition). + // A response media type needs both a type and a subtype. + if !strings.Contains(mediaType, "/") { + return false + } + if strings.HasPrefix(mediaType, "text/") || strings.HasSuffix(mediaType, "+json") || strings.HasSuffix(mediaType, "+xml") { + return true + } + switch mediaType { + case "application/json", "application/xml", "application/javascript", "application/x-www-form-urlencoded": + return true + default: + return false + } +} diff --git a/response_encoding_test.go b/response_encoding_test.go new file mode 100644 index 0000000..9623a9f --- /dev/null +++ b/response_encoding_test.go @@ -0,0 +1,209 @@ +package edge + +import ( + "bytes" + "encoding/base64" + "encoding/json/v2" + "errors" + "net/http" + "slices" + "strings" + "testing" + + "github.com/aws/aws-lambda-go/events" +) + +func TestEncodeResponseBody(t *testing.T) { + for _, tt := range []struct { + name string + body []byte + header http.Header + wantBase bool + }{ + {name: "empty"}, + {name: "empty_binary", header: http.Header{"Content-Type": {"application/octet-stream"}}}, + {name: "text", body: []byte("café"), header: http.Header{"Content-Type": {"text/plain; charset=utf-8"}}}, + {name: "json", body: []byte(`{"ok":true}`), header: http.Header{"Content-Type": {"Application/JSON"}}}, + {name: "json_suffix", body: []byte(`{}`), header: http.Header{"Content-Type": {"application/problem+json"}}}, + {name: "xml", body: []byte(""), header: http.Header{"Content-Type": {"application/xml"}}}, + {name: "xml_suffix", body: []byte(""), header: http.Header{"Content-Type": {"image/svg+xml"}}}, + {name: "javascript", body: []byte("true"), header: http.Header{"Content-Type": {"application/javascript"}}}, + {name: "form", body: []byte("a=b"), header: http.Header{"Content-Type": {"application/x-www-form-urlencoded"}}}, + {name: "identity", body: []byte("hello"), header: http.Header{"Content-Type": {"text/plain"}, "Content-Encoding": {"Identity"}}}, + {name: "suppressed_encoding", body: []byte("hello"), header: http.Header{"Content-Type": {"text/plain"}, "Content-Encoding": nil}}, + {name: "binary", body: []byte{0, 255}, header: http.Header{"Content-Type": {"application/octet-stream"}}, wantBase: true}, + {name: "utf8_binary", body: []byte("hello"), header: http.Header{"Content-Type": {"application/octet-stream"}}, wantBase: true}, + {name: "invalid_utf8", body: []byte{255}, header: http.Header{"Content-Type": {"text/plain"}}, wantBase: true}, + {name: "compressed", body: []byte("hello"), header: http.Header{"Content-Type": {"text/plain"}, "Content-Encoding": {"gzip"}}, wantBase: true}, + {name: "multiple_encodings", body: []byte("hello"), header: http.Header{"Content-Type": {"text/plain"}, "Content-Encoding": {"identity", "gzip"}}, wantBase: true}, + {name: "missing_type", body: []byte("hello"), wantBase: true}, + {name: "suppressed_type", body: []byte("hello"), header: http.Header{"Content-Type": nil}, wantBase: true}, + {name: "invalid_type", body: []byte("hello"), header: http.Header{"Content-Type": {"text/plain; broken"}}, wantBase: true}, + {name: "bare_suffix", body: []byte("hello"), header: http.Header{"Content-Type": {"problem+json"}}, wantBase: true}, + {name: "ambiguous_type", body: []byte("hello"), header: http.Header{"Content-Type": {"text/plain", "application/octet-stream"}}, wantBase: true}, + } { + t.Run(tt.name, func(t *testing.T) { + original := slices.Clone(tt.body) + got, binary, err := encodeResponseBody(tt.body, tt.header) + if err != nil { + t.Fatalf("encodeResponseBody() = %v", err) + } + if binary != tt.wantBase { + t.Errorf("IsBase64Encoded = %v, want %v", binary, tt.wantBase) + } + decoded := []byte(got) + if binary { + decoded, err = base64.StdEncoding.DecodeString(got) + if err != nil { + t.Fatal(err) + } + } + if !bytes.Equal(decoded, original) || !bytes.Equal(tt.body, original) { + t.Errorf("body round trip = %q, input after encoding = %q, want %q", decoded, tt.body, original) + } + }) + } +} + +func TestResponseEncodingLimits(t *testing.T) { + for _, tt := range []struct { + name string + length int + mediaType string + wantErr bool + }{ + {name: "text_at_limit", length: maxResponseBytes, mediaType: "text/plain"}, + {name: "text_over_limit", length: maxResponseBytes + 1, mediaType: "text/plain", wantErr: true}, + {name: "binary_at_limit", length: maxResponseBytes / 4 * 3, mediaType: "application/octet-stream"}, + {name: "binary_expansion_over_limit", length: maxResponseBytes/4*3 + 1, mediaType: "application/octet-stream", wantErr: true}, + } { + t.Run(tt.name, func(t *testing.T) { + body := bytes.Repeat([]byte("x"), tt.length) + got, binary, err := encodeResponseBody(body, http.Header{"Content-Type": {tt.mediaType}}) + if tt.wantErr { + if !errors.Is(err, errResponseTooLarge) || got != "" || binary { + t.Fatalf("over-limit encoding = (%d bytes, %v, %v), want no result and size error", len(got), binary, err) + } + return + } + if err != nil || len(got) != maxResponseBytes { + t.Errorf("at-limit encoding = (%d bytes, %v), want %d bytes", len(got), err, maxResponseBytes) + } + }) + } +} + +func TestMarshalResponseSDKTypes(t *testing.T) { + t.Run("v1", func(t *testing.T) { + want := events.APIGatewayProxyResponse{StatusCode: 201, Body: "hello", MultiValueHeaders: map[string][]string{"Set-Cookie": {"a=1", "b=2"}, "X-Trace": {"one", "two"}}} + data, err := marshalResponse(want) + if err != nil { + t.Fatal(err) + } + var got events.APIGatewayProxyResponse + if err := json.Unmarshal(data, &got); err != nil { + t.Fatal(err) + } + if got.StatusCode != want.StatusCode || got.Body != want.Body || got.IsBase64Encoded || !slices.Equal(got.MultiValueHeaders["Set-Cookie"], want.MultiValueHeaders["Set-Cookie"]) || !slices.Equal(got.MultiValueHeaders["X-Trace"], want.MultiValueHeaders["X-Trace"]) { + t.Errorf("V1 response round trip = %+v, want %+v", got, want) + } + }) + t.Run("v2", func(t *testing.T) { + want := events.APIGatewayV2HTTPResponse{StatusCode: 200, Body: "AP8=", IsBase64Encoded: true, Headers: map[string]string{"Content-Type": "application/octet-stream"}, Cookies: []string{"a=1", "b=2"}} + data, err := marshalResponse(want) + if err != nil { + t.Fatal(err) + } + var got events.APIGatewayV2HTTPResponse + if err := json.Unmarshal(data, &got); err != nil { + t.Fatal(err) + } + if got.StatusCode != want.StatusCode || got.Body != want.Body || !got.IsBase64Encoded || got.Headers["Content-Type"] != want.Headers["Content-Type"] || !slices.Equal(got.Cookies, want.Cookies) { + t.Errorf("V2 response round trip = %+v, want %+v", got, want) + } + }) +} + +func TestMarshalResponseEnvelopeLimit(t *testing.T) { + // Measure the actual codec overhead rather than coupling the test to field + // order, optional-field tags, or whitespace choices in the pinned SDK. + response := events.APIGatewayV2HTTPResponse{StatusCode: 200} + empty, err := marshalResponse(response) + if err != nil { + t.Fatal(err) + } + for _, extra := range []int{-1, 0, 1} { + response.Body = strings.Repeat("x", maxResponseBytes-len(empty)+extra) + got, err := marshalResponse(response) + if extra > 0 { + if !errors.Is(err, errResponseTooLarge) || got != nil { + t.Errorf("oversized envelope = (%d bytes, %v), want nil and size error", len(got), err) + } + continue + } + if err != nil || len(got) != maxResponseBytes+extra { + t.Errorf("envelope boundary %d = (%d bytes, %v)", extra, len(got), err) + } + } + // The body itself is small enough, but JSON escape expansion is not. + response.Body = strings.Repeat("\x00", maxResponseBytes/6) + if got, err := marshalResponse(response); !errors.Is(err, errResponseTooLarge) || got != nil { + t.Errorf("escaped envelope = (%d bytes, %v), want nil and size error", len(got), err) + } + response.Body = "" + response.Cookies = []string{strings.Repeat("x", maxResponseBytes)} + if got, err := marshalResponse(response); !errors.Is(err, errResponseTooLarge) || got != nil { + t.Errorf("cookie envelope = (%d bytes, %v), want nil and size error", len(got), err) + } +} + +func TestMarshalResponseSanitizesCodecErrors(t *testing.T) { + response := events.APIGatewayV2HTTPResponse{StatusCode: 200, Headers: map[string]string{"private-secret": "invalid\xff"}} + got, err := marshalResponse(response) + if err == nil || got != nil { + t.Fatalf("invalid header UTF-8 = (%d bytes, %v), want nil and encoding error", len(got), err) + } + if strings.Contains(err.Error(), "private-secret") || strings.Contains(err.Error(), "invalid") { + t.Errorf("encoding error leaked application data: %v", err) + } +} + +func TestResponseBufferRejectsWholeWrite(t *testing.T) { + var out responseBuffer + first := bytes.Repeat([]byte("x"), maxResponseBytes-1) + if n, err := out.Write(first); n != len(first) || err != nil { + t.Fatalf("initial Write = (%d, %v)", n, err) + } + if n, err := out.Write([]byte("yz")); n != 0 || !errors.Is(err, errResponseTooLarge) { + t.Errorf("crossing Write = (%d, %v), want zero and size error", n, err) + } + if !bytes.Equal(out.data, first) || cap(out.data) > maxResponseBytes { + t.Errorf("failed Write retained data or excessive capacity: len=%d cap=%d", len(out.data), cap(out.data)) + } + if n, err := out.Write([]byte("y")); n != 1 || err != nil || len(out.data) != maxResponseBytes || cap(out.data) > maxResponseBytes { + t.Errorf("exact-limit Write = (%d, %v), len=%d cap=%d", n, err, len(out.data), cap(out.data)) + } +} + +func BenchmarkMarshalResponse(b *testing.B) { + for _, tt := range []struct { + name string + body string + }{ + {name: "small_json", body: `{"ok":true}`}, + {name: "near_limit", body: strings.Repeat("x", maxResponseBytes-1024)}, + {name: "escaped_over_limit", body: strings.Repeat("\x00", maxResponseBytes/6)}, + } { + b.Run(tt.name, func(b *testing.B) { + response := events.APIGatewayV2HTTPResponse{StatusCode: 200, Body: tt.body} + b.ReportAllocs() + b.SetBytes(int64(len(tt.body))) + for b.Loop() { + _, err := marshalResponse(response) + if err != nil && !errors.Is(err, errResponseTooLarge) { + b.Fatal(err) + } + } + }) + } +} From bdb429cdaafb2d76e2b102c97739d16bb81db767 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 16:00:18 +0100 Subject: [PATCH 08/44] docs: qualify metadata policy and propose REST response streaming --- docs/decisions/0007-response-metadata.md | 26 +++- docs/decisions/0008-streaming-architecture.md | 139 ++++++++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 7 + 4 files changed, 170 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/0008-streaming-architecture.md diff --git a/docs/decisions/0007-response-metadata.md b/docs/decisions/0007-response-metadata.md index 7819b56..09db6e1 100644 --- a/docs/decisions/0007-response-metadata.md +++ b/docs/decisions/0007-response-metadata.md @@ -1,6 +1,23 @@ # 0007: Response metadata budgets and V2 list fields -Status: proposed; awaiting user review. +Status: qualified direction accepted; concrete metadata policy remains unresolved. + +## Accepted qualification + +The user accepted the qualified recommendation: keep the documented Lambda +envelope limit and semantics-preserving header translation, but do not freeze +the proposed 64 KiB/1,024-entry limits without stronger justification. The ten +fields below are an initial audit, not an AWS allowlist or an exhaustive set of +valid list-valued fields. Broader compatibility must be considered before fixing +that set as the library contract. No numeric metadata limit or exact joinable +set has been approved or implemented. + +The numeric proposal was a library resource policy, not an AWS quota or a result +of header-allocation benchmarks. Its intended protection was to bound additional +snapshot allocations, not memory already allocated by the application. Current +AWS HTTP API quotas explicitly limit the request line and headers; that does not +establish the same limit for an edge response snapshot. +[HTTP API quotas](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-quotas.html) ## Question and evidence @@ -18,7 +35,7 @@ HTTP does not define one universal field-size limit. A library metadata budget is our resource policy, not a claim about gateway quotas. [HTTP field limits](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.4) -## Recommendation +## Initial proposal retained for discussion (not accepted as written) Before sorting, canonicalizing, or cloning a committed Header map, allow at most 64 KiB (65,536 bytes) of names plus values and 1,024 entries. Count each original @@ -67,5 +84,6 @@ V1. Hop-by-hop fields are removed under decision 0006 before V2 joining. ## Resolution -Pending user approval. The response snapshot and V2 header conversion depend on -this decision. Bounded envelope encoding and invocation lifetime work do not. +The user agreed with the qualification above. Resolve evidence-based resource +budgets and broader field compatibility before implementing those policies. +Bounded envelope encoding and invocation lifetime work do not depend on them. diff --git a/docs/decisions/0008-streaming-architecture.md b/docs/decisions/0008-streaming-architecture.md new file mode 100644 index 0000000..5d960f9 --- /dev/null +++ b/docs/decisions/0008-streaming-architecture.md @@ -0,0 +1,139 @@ +# 0008: Streaming architecture alongside buffered invocation + +Status: proposed architecture for discussion; no streaming implementation. + +## Question + +How can edge support streaming HTTP responses without changing the accepted +buffered/raw/typed contracts or abandoning ordinary net/http handlers? + +## AWS evidence + +- API Gateway currently supports response streaming for REST APIs, including + Regional, private, and edge-optimized endpoints. HTTP APIs and request + streaming are not supported. Integration response transfer mode must be + STREAM. Streams can run for up to 15 minutes, subject to invocation and idle + timeouts; a disconnected client need not stop Lambda execution. + [Streaming support and constraints](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode.html) +- Lambda proxy streaming uses InvokeWithResponseStream and a different + integration URI ending in response-streaming-invocations. Input remains the + REST proxy event. Output is JSON metadata, eight NUL delimiter bytes, then raw + payload bytes. The delimiter must appear within the first 16 KB. Metadata + supports statusCode, headers, multiValueHeaders, and cookies. There is no + base64 body field. These deployment and framing requirements must match. + [Lambda streaming integration contract](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html) +- Lambda documents a 200 MB streamed-response ceiling versus 6 MB buffered, + with bandwidth shaping. Do not blindly reuse the buffered body limit or claim + this is an independently verified end-to-end gateway allowance. An AWS .NET + article states a 10 MB gateway limit, while the gateway streaming guide + describes bandwidth behavior above 10 MB. Resolve that documentation conflict + before promising an exact end-to-end maximum or encoding it as a library limit. + [Lambda streaming](https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html) + [AWS .NET article](https://aws.amazon.com/blogs/developer/announcing-response-streaming-for-net-on-aws-lambda/) + +## SDK and repository evidence + +The pinned aws-lambda-go v1.55.0 already provides +events.APIGatewayProxyStreamingResponse, with status, headers, multivalue headers, +cookies, and an io.Reader body. It implements Read, Close, and ContentType; Read +generates the prefix using encoding/json. Reusing that serializer inside edge +would conflict with our direct JSON v2 requirement. +[Pinned SDK streaming response](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/apigw.go) + +The SDK's function-handler path can return readers. Its runtime loop consumes +and closes the reader and honors its ContentType method. Its legacy/public +Handler.Invoke byte-slice interface instead reads the complete response into +memory. Calling lambda.NewHandler(...).Invoke can check content but cannot prove +incremental streaming. Reader dispatch has JSON-serialization precedence rules; +the exact wrapper must be tested against this pinned implementation. +[Handler dispatch](https://github.com/aws/aws-lambda-go/blob/v1.55.0/lambda/handler.go) +[Runtime loop](https://github.com/aws/aws-lambda-go/blob/v1.55.0/lambda/invoke_loop.go) +[Runtime transport and error trailers](https://github.com/aws/aws-lambda-go/blob/v1.55.0/lambda/runtime_api_client.go) + +Our request conversion can be shared. The buffered response encoder creates a +complete body and envelope, so it cannot implement streaming. withInvocation +currently ends its scope when its callback returns; returning a reader from that +callback would cancel and clean up too early. + +## Recommendation + +Keep both modes in the root edge package with a common http.Handler application +boundary. Add an explicit streaming adapter/entry point rather than changing the +return contract of Invoke, HandleV1, or HandleV2. A boolean option cannot make a +byte-slice result incremental. Constructor/method names and exact raw/typed +streaming signatures remain a separate proposal; NewStreaming is a possible name, +not an approved API. + +Start streaming support with REST Lambda proxy integration only. Do not infer +streaming from payload version, an Accept header, or a call to Flush. The AWS +integration must be configured for it. HTTP API payload 1.0 shares the V1 event +type but does not gain streaming support from that type. Function URLs are a +possible later target requiring their own request/authentication contract. + +Share request conversion, gateway metadata, identity production, and applicable +HTTP header/status validation. Use distinct buffered and streaming writers: + +| Concern | Buffered | Streaming | +| --- | --- | --- | +| Application boundary | http.Handler | http.Handler | +| Response delivery | Complete AWS response/envelope | Metadata prefix followed by raw bytes | +| Body retention | Bounded full body | Small bounded buffering with backpressure | +| Flush | Unsupported | Flush/Error-capable writer; no promise about downstream packet timing | +| Binary bytes | Base64 when required | Raw bytes | +| Content length | Can infer after handler returns | Usually unknown; do not prebuffer to infer | +| Failure | No usable result on transport failure | Depends on whether transmission has started | +| Lifetime | Through synchronous completion | Through stream consumption/closure and producer completion | + +Retain AWS's runtime transport. Implement the small metadata encoder/framing +reader in edge with encoding/json/v2 directly; do not build a separate Runtime +API client. Use a bounded producer/consumer bridge, such as io.Pipe, between +ServeHTTP writes and the runtime's reader. Close must cancel the producer and +unblock pending writes. Prove that no producer survives invocation completion. + +Authenticate and authorize before exposing stream content. Validate the prefix +before emitting it and preserve multivalue headers supported by this format. +Unlike the earlier arbitrary snapshot proposal, the streaming prefix has a +documented AWS framing bound; verify its exact byte boundary in the wire tests. + +Define commitment and failure semantics explicitly. Pre-transmission failures +can remain invocation errors. After transmission begins, headers/status cannot +be replaced and bytes cannot be retracted. Terminate a failed stream and use the +runtime's error path; do not inject an unrequested error document into arbitrary +application data. Application-defined SSE/NDJSON error events remain the +application's decision. Consumer-visible truncation/error behavior needs testing. + +Goroutine panics require a deliberate bridge to the runtime failure mechanism; +the SDK's synchronous handler recovery does not catch a producer panic after +the function returns. Preserve sanitized diagnostics and explicitly review any +necessary difference from buffered panic propagation. Do not advertise arbitrary +HTTP trailers, Hijacker, WebSockets, request streaming, or full duplex merely +because response flushing works. + +## Sequence and proof before public API + +Design the shared boundaries now; finish the first buffered adapter slice, then +implement streaming under an agreed contract. First make a local Runtime API +protocol probe using the pinned SDK. Verify that initial bytes arrive before the +producer completes, along with ContentType handling, raw/typed input capture, +JSON v2 prefix framing, backpressure, Close, cancellation, and late read errors. +Do not use a fully collecting test wrapper as evidence of streaming. + +Then exercise ordinary SSE, NDJSON, and binary handlers with a slow reader and +an early-closing reader. Cover prefix limits, headers/status commitment, HEAD, +bodyless statuses, compression, panic cleanup, and warm-invocation reuse. An +optional deployed test would be needed to establish end-to-end API Gateway +flush timing, interruption behavior, and disputed service limits; none is +authorized or performed by this design review. + +## Alternatives and resolution + +Using the SDK streaming response object is attractive but delegates its internal +prefix JSON to the v1 compatibility API. Returning complete byte slices keeps +the old API but defeats incremental delivery. A separate runtime or mandatory +Lambda Web Adapter adds deployment machinery that the existing SDK transport +does not yet appear to require. + +Recommendation: explicit REST streaming mode within edge, shared HTTP/identity +foundations, direct JSON v2 framing, and AWS SDK transport. Await review of this +architecture and then the concrete streaming API/lifecycle contract. No streaming +methods or placeholder APIs have been added. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index f500572..918295d 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -15,4 +15,5 @@ Keep related decisions small enough to review and implement independently. | [0004: Typed events](0004-typed-events.md) | Accepted | How can already typed events bypass edge's envelope JSON work? | | [0005: HTTP requests](0005-http-request-translation.md) | Accepted | How should raw and typed events become ordinary server requests? | | [0006: Buffered responses](0006-buffered-responses.md) | Accepted | How should HTTP responses, unsupported capabilities, and completion failures behave? | -| [0007: Response metadata](0007-response-metadata.md) | Proposed | What metadata budgets and initial V2 joinable-field set should apply? | +| [0007: Response metadata](0007-response-metadata.md) | Qualified direction accepted; details open | What metadata budgets and V2 joinable-field set should apply? | +| [0008: Streaming architecture](0008-streaming-architecture.md) | Proposed | How should REST response streaming fit alongside the buffered adapter? | diff --git a/docs/plan.md b/docs/plan.md index a73dd7b..5688199 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -101,6 +101,13 @@ minimum gateway identity foundation, before broader authentication work. These enhancements are incorporated into decision 0006. Concrete metadata/header policies, exported error categories, and identity APIs remain separate reviews. +The user accepted the qualification to decision 0007: do not freeze arbitrary +64 KiB/1,024-entry metadata limits or the initial ten-field list without stronger +resource and compatibility justification. Streaming is now being discussed in +[decision 0008](decisions/0008-streaming-architecture.md): explicit REST response +streaming alongside the buffered adapter, sharing HTTP/identity foundations. +This is a proposed architecture, not implemented streaming support. + Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary errors, cleans up on failures/panics, and checks parent cancellation. The HTTP From 02c9f0372eac2c9e8d04a919e76de0e28deaf4a7 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 16:14:03 +0100 Subject: [PATCH 09/44] docs: specify streaming boundaries and probe SDK transport --- docs/decisions/0008-streaming-architecture.md | 12 +- docs/decisions/0009-streaming-boundaries.md | 219 ++++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 7 +- internal/streamprobe/runtime_test.go | 272 ++++++++++++++++++ 5 files changed, 506 insertions(+), 7 deletions(-) create mode 100644 docs/decisions/0009-streaming-boundaries.md create mode 100644 internal/streamprobe/runtime_test.go diff --git a/docs/decisions/0008-streaming-architecture.md b/docs/decisions/0008-streaming-architecture.md index 5d960f9..0c861f9 100644 --- a/docs/decisions/0008-streaming-architecture.md +++ b/docs/decisions/0008-streaming-architecture.md @@ -1,6 +1,6 @@ # 0008: Streaming architecture alongside buffered invocation -Status: proposed architecture for discussion; no streaming implementation. +Status: architecture accepted; concrete boundaries proposed in decision 0009. ## Question @@ -133,7 +133,9 @@ the old API but defeats incremental delivery. A separate runtime or mandatory Lambda Web Adapter adds deployment machinery that the existing SDK transport does not yet appear to require. -Recommendation: explicit REST streaming mode within edge, shared HTTP/identity -foundations, direct JSON v2 framing, and AWS SDK transport. Await review of this -architecture and then the concrete streaming API/lifecycle contract. No streaming -methods or placeholder APIs have been added. +The user agreed with the architecture and requested the detailed design graph. +[Decision 0009](0009-streaming-boundaries.md) proposes concrete APIs, ownership, +commitment, and failure rules, backed by a local SDK Runtime API probe. SDK +transport reuse remains the preferred approach, with a documented wire-header +compatibility question to settle before claiming AWS deployment support. No +streaming methods or placeholder APIs have been added. diff --git a/docs/decisions/0009-streaming-boundaries.md b/docs/decisions/0009-streaming-boundaries.md new file mode 100644 index 0000000..4a0fda5 --- /dev/null +++ b/docs/decisions/0009-streaming-boundaries.md @@ -0,0 +1,219 @@ +# 0009: Streaming entry points, ownership, and failure boundaries + +Status: proposed; high-level architecture accepted in decision 0008. + +## Recommended public boundary + +Keep a distinct adapter in the root edge package. Proposed signatures: + +```go +func NewStreaming(handler http.Handler, opts ...Option) (*StreamingAdapter, error) +func (a *StreamingAdapter) Handle(ctx context.Context, event jsontext.Value) (io.ReadCloser, error) +func (a *StreamingAdapter) HandleV1(ctx context.Context, event events.APIGatewayProxyRequest) (io.ReadCloser, error) +``` + +Register `lambda.Start(adapter.Handle)` or `lambda.Start(adapter.HandleV1)`. +Do not register the streaming adapter object via the byte-returning lambda.Handler +interface. No HandleV2: API Gateway HTTP APIs do not currently stream responses. +REST configuration is a deployment precondition; a typed V1 event alone cannot +prove the invoking API product. Raw Handle rejects the explicitly versioned +HTTP API envelopes supported by the separate buffered adapter. + +Handle receives raw JSON via the standard jsontext.Value capture type and applies +edge's direct JSON v2 decoding and validation. The SDK still owns its initial +capture step. HandleV1 skips request JSON work. Both must encode streaming +metadata with JSON v2; typed streaming cannot eliminate that wire prefix. + +Reuse the existing Option type and immutable configuration, including the +accepted WithGatewayIdentity default. NewStreaming performs the same constructor +validation as New and starts no goroutines or I/O. The caller owns handler +concurrency safety. Do not add Start helpers, stream-specific request types, or +a public ResponseWriter interface. + +The concrete stream remains private and implements Read, Close, ContentType, +and a MarshalJSON rejection method. The latter follows the SDK streaming type's +reader-dispatch pattern: it prevents accidental JSON serialization and lets the +SDK consume the reader. It does not perform JSON encoding. ContentType returns +the Lambda integration media type, independently of the application's HTTP +Content-Type in the prefix. Direct callers must consume and Close the stream. + +## Design graph + +```mermaid +flowchart TD + Deploy[REST integration configured for STREAM] --> Runtime[AWS Lambda Go runtime] + Runtime --> Entry[Handle raw JSON or HandleV1 typed event] + Entry --> Prepare[Validate context and event; convert request; optional gateway identity] + Prepare --> Producer[One producer goroutine: ordinary http.Handler and middleware] + Producer --> Commit[Freeze HTTP status and header snapshot] + Commit --> Prefix[Validate and encode metadata with JSON v2] + Prefix --> Ready[Publish ready signal before any blocking pipe write] + Ready --> Return[Entry point returns private io.ReadCloser] + Producer --> Pipe[Raw body through io.Pipe with backpressure] + Return --> Consumer[SDK reads metadata then body] + Pipe --> Consumer + Consumer --> Gateway[Lambda service then API Gateway then client] + Producer --> Cleanup[Handler completion and request cleanup] + Cleanup --> Terminal[EOF or sanitized terminal stream error] + Terminal --> Consumer + Consumer --> Close[Close: cancel, unblock writes, join producer] + Close --> Next[Only then permit next invocation] +``` + +The graph separates application HTTP commitment from handing a response stream +to the runtime. That handoff is our practical error boundary; do not assume that +the absence of an observed network write makes it safe to return a different +result after handing off the reader. + +## Ownership and lifecycle + +| Owner | Responsibility | +| --- | --- | +| Entry point | Synchronous validation/conversion and optional gateway identity; start producer; wait for ready, failure, or cancellation | +| Producer goroutine | Run handler/middleware; own writer state; finalize body accounting; clean original request body and attached multipart form | +| Private stream | Own immutable prefix, pipe reader, cancellation, terminal read error, and producer completion signal | +| SDK runtime | Consume/close the returned stream and perform Runtime API transport | +| Application | Honor request cancellation and Write/Flush errors; authorize before sending content; own any goroutines it starts | + +Use one producer, not a goroutine per write. Publish a one-shot ready signal +after prefix validation and before blocking on pipe output. Otherwise the +producer can wait for a reader while the entry point waits for the producer. +Do not share the handler's mutable Header map with the consumer. + +Successful EOF must occur only after producer completion and request cleanup, +so cleanup failures can still become terminal read errors. Keep the invocation +scope until consumption finishes or Close completes. Refactor the shared scope +into acquisition/cleanup primitives with the existing synchronous helper as one +owner; do not return a live stream from inside today's withInvocation callback. + +Close is idempotent, safe to call concurrently with Read, and cancels the child +context, closes the pipe's read side to unblock writes, then joins the producer. +Parent cancellation also closes the bridge so a writer blocked by a stalled +consumer can exit. A context.AfterFunc cancellation callback must be stopped or +joined during teardown; stopping it alone does not wait for an active callback. + +Go cannot forcibly terminate a handler that ignores cancellation and write +errors. Close may wait for such a handler; in Lambda the invocation timeout is +the final bound. Do not return early and silently leave that handler running +into a warm invocation. The guarantee is bounded adapter buffering and cleanup +of cooperative handlers, not the ability to kill arbitrary application code. +[io.Pipe](https://pkg.go.dev/io#Pipe) +[context.AfterFunc](https://pkg.go.dev/context#AfterFunc) + +## Writer and framing behavior + +- Share HTTP status/header validation where the contracts match. Preserve REST + multivalue response fields rather than applying V2 comma-joining rules. +- First final WriteHeader, Write, or Flush commits status/header snapshot. + Flush without a prior status commits 200. Subsequent header mutation does not + alter the snapshot. +- Recommend retaining at most a 512-byte initial body prefix for content-type + detection when needed, then using the unbuffered pipe. Publish on filling that + prefix, explicit Flush, or handler completion. If type detection is suppressed + or unnecessary, first nonempty Write can publish immediately. Empty Flush + publishes metadata without inventing a type from absent body bytes. +- Implement Flush and FlushError so ResponseController.Flush can report failure. + Flush drains edge's pending prefix/body bytes toward the SDK. It does not + guarantee client receipt, packet boundaries, or intermediary flush timing. +- Use direct JSON v2 for the bounded prefix, followed by eight NUL bytes, then + raw body bytes. Do not base64 encode the body or buffer it to infer length. + Enforce AWS's documented first-16-KB framing requirement. Verify the precise + byte-boundary interpretation before finalizing its numeric constant; this is + distinct from the unapproved general metadata budgets in decision 0007. +- Honor supplied valid Content-Length and reject writes that exceed it before + forwarding their excess. A short body discovered after handoff is a terminal + stream error. Without a declared length, let the gateway provide framing. +- Capture the original method. Suppress HEAD bodies and 204/205/304 bodies; + retain the accepted forbidden-write conventions. Do not infer a HEAD length + after already publishing its metadata. +- Do not expose Hijacker, full duplex, connection deadlines, or arbitrary HTTP + trailers. Explicit unsupported features follow the accepted fault policy. + Lambda error trailers are runtime signaling, not application HTTP trailers. + +## Failures and observability + +| Failure point | Recommended behavior | +| --- | --- | +| Before producer starts | Return nil stream and sanitized error; no application execution | +| Before handoff | Cancel/join and clean up; return nil stream and error | +| Producer panic before handoff | Join/clean up, then re-panic on entry goroutine so normal SDK recovery applies | +| After handoff | Preserve sent status/bytes; close pipe with sanitized terminal error; no replacement body/status | +| Producer panic after handoff | Recover within producer, clean up, report a sanitized terminal stream error | +| Parent cancellation or consumer Close | Cancel, unblock the pipe, join producer; preserve an existing primary failure | + +The late-panic rule intentionally differs from the SDK's synchronous panic path, +which can terminate the process. Do not introduce os.Exit or an unhandled worker +panic into the library. This difference needs explicit user approval. + +AWS says midstream error trailers are treated as a successful response with error +metadata. Do not claim they necessarily increment Lambda's ordinary invocation +error metric, change an HTTP status, or reach every API Gateway client as useful +trailers. Application protocols such as SSE/NDJSON may define their own error +events; edge must not inject these into arbitrary streams. +[AWS Runtime API streaming errors](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html) + +Recommend an optional application-owned terminal-error reporter so failures +after handoff are observable without library logging: + +```go +func WithStreamErrorReporter(report func(context.Context, error)) Option +``` + +Reject nil callbacks and reject this streaming-only option in New. Invoke it once +for a terminal failure after handoff, including cancellation/panic, after producer +cleanup and before completion. Do not report errors that were returned directly +before handoff. It receives the invocation context (possibly canceled) and only +sanitized errors and must return promptly. If it panics, recover at the reporter +boundary, preserve the original terminal failure, add a sanitized reporter fault, +and do not recursively report. Integrate these faults with the separately reviewed +shared error categories. No callback is required to use streaming; without one the reader +and SDK error metadata are the available signals. + +## Local SDK probe: observed results and compatibility gates + +`internal/streamprobe/runtime_test.go` starts the pinned SDK in a subprocess and +serves a synthetic Runtime API locally. The second stream segment is withheld +until the server receives the first byte. All five cases passed with race +detection on Go 1.27.1: + +- Raw jsontext.Value capture preserved the exact object, duplicate names, and + an integer above 2^53 for later strict validation by edge. +- Typed APIGatewayProxyRequest input reached the typed function correctly. +- SDK delivery was incremental and HTTP/1.1 chunked; the integration ContentType + method was honored through an io.ReadCloser return type. +- Reader Close ran before the next invocation request. +- A late reader error became Lambda runtime error trailers. +- Returning bytes and a non-EOF error together lost those bytes in the SDK's + errorCapturingReader. The edge stream must normalize reads: return any bytes + with nil error, and deliver the saved error on the next nonempty Read. A fifth + probe case verifies this guard preserves the final bytes and the error trailer. + Do not change SDK code in place. + +The SDK did not send Lambda-Runtime-Function-Response-Mode. AWS's custom-runtime +documentation says streaming responses must send this header as `streaming`. +The SDK also provides official streaming response examples, so this observation +alone does not prove the service rejects or buffers its output. It is a concrete +documentation/SDK mismatch, not resolved by a permissive local test server. + +Recommend retaining SDK reuse as the design, but treat this mismatch as a release +gate: seek an authoritative clarification or a suitable SDK fix/version; if +needed, propose a separately approved minimal AWS integration test. Do not patch +http.DefaultTransport globally, alter the module cache, or silently introduce a +custom runtime to bypass the issue. The local probe proves the SDK's wire +behavior, not API Gateway delivery or billing/error metrics. + +## Implementation order and resolution + +1. Review this API, ownership graph, handoff boundary, and late-error/panic policy. +2. Extend scope ownership and implement the private stream bridge, including + byte/error normalization, cancellation, Close, and handshake tests. +3. Implement shared header validation and separate streaming writer/framing. +4. Wire the proposed raw and typed REST entry points with identity policy. +5. Exercise SSE, NDJSON, binary output, early Close, slow reads, and warm reuse; + resolve the AWS runtime-header mismatch and service-limit questions before + advertising deployment support. + +The first complete buffered adapter remains the initial deliverable. The local +SDK probe and design allow streaming requirements to shape shared ownership now. +The detailed API and policy in this record are proposed, not implemented or +approved by the prior agreement to the high-level architecture. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 918295d..5a955a3 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -16,4 +16,5 @@ Keep related decisions small enough to review and implement independently. | [0005: HTTP requests](0005-http-request-translation.md) | Accepted | How should raw and typed events become ordinary server requests? | | [0006: Buffered responses](0006-buffered-responses.md) | Accepted | How should HTTP responses, unsupported capabilities, and completion failures behave? | | [0007: Response metadata](0007-response-metadata.md) | Qualified direction accepted; details open | What metadata budgets and V2 joinable-field set should apply? | -| [0008: Streaming architecture](0008-streaming-architecture.md) | Proposed | How should REST response streaming fit alongside the buffered adapter? | +| [0008: Streaming architecture](0008-streaming-architecture.md) | Accepted | How should REST response streaming fit alongside the buffered adapter? | +| [0009: Streaming boundaries](0009-streaming-boundaries.md) | Proposed | What API, lifecycle, handoff, and error rules should streaming use? | diff --git a/docs/plan.md b/docs/plan.md index 5688199..19d0b33 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -106,7 +106,12 @@ The user accepted the qualification to decision 0007: do not freeze arbitrary resource and compatibility justification. Streaming is now being discussed in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. -This is a proposed architecture, not implemented streaming support. +The high-level architecture is accepted. [Decision 0009](decisions/0009-streaming-boundaries.md) +proposes concrete APIs and an ownership/state graph. A local SDK Runtime API +probe demonstrates incremental delivery, raw/typed capture, closure, and error +trailers; it also exposes a bytes-plus-error reader issue and a missing header +relative to AWS's documented streaming contract. No production streaming API is +implemented, and deployed API Gateway behavior is not verified. Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary diff --git a/internal/streamprobe/runtime_test.go b/internal/streamprobe/runtime_test.go new file mode 100644 index 0000000..7bc7198 --- /dev/null +++ b/internal/streamprobe/runtime_test.go @@ -0,0 +1,272 @@ +// Package streamprobe tests the pinned SDK transport against a local Runtime API. +// It is a design probe, not an implementation of edge's streaming API. +package streamprobe + +import ( + "bytes" + "context" + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "slices" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-lambda-go/lambda" +) + +const eventJSON = `{"httpMethod":"GET","path":"/","requestContext":{"apiId":"example"},"number":9007199254740993,"duplicate":1,"duplicate":2}` +const integrationContentType = "application/vnd.awslambda.http-integration-response" + +// SDK reader dispatch must retain the concrete value's ContentType and Close +// methods even though the function's declared result is io.ReadCloser. +type probeStream struct { + body io.Reader + url string + deferErrors bool + pending error +} + +func (s *probeStream) Read(p []byte) (int, error) { + if len(p) == 0 { + return 0, nil + } + if s.pending != nil { + return 0, s.pending + } + n, err := s.body.Read(p) + if s.deferErrors && n > 0 && err != nil { + s.pending = err + return n, nil + } + return n, err +} + +func (s *probeStream) ContentType() string { return integrationContentType } +func (s *probeStream) MarshalJSON() ([]byte, error) { + return nil, errors.New("stream is not a JSON response") +} +func (s *probeStream) Close() error { + response, err := (&http.Client{Timeout: 3 * time.Second}).Get(s.url + "/closed") + if err != nil { + return err + } + return response.Body.Close() +} + +type gatedTail struct { + ctx context.Context // This reader belongs to exactly one invocation. + url string + mode string + tail *bytes.Reader + once sync.Once + err error +} + +func (r *gatedTail) Read(p []byte) (int, error) { + r.once.Do(func() { + request, err := http.NewRequestWithContext(r.ctx, http.MethodGet, r.url+"/continue", nil) + if err != nil { + r.err = err + return + } + response, err := http.DefaultClient.Do(request) + if err != nil { + r.err = err + return + } + r.err = response.Body.Close() + }) + if r.err != nil { + return 0, r.err + } + n, err := r.tail.Read(p) + if r.mode == "data_and_error" && n > 0 || r.mode == "late_error" && err == io.EOF { + return n, errors.New("synthetic stream failure") + } + return n, err +} + +func TestRuntimeStreamingProbeProcess(t *testing.T) { + mode := os.Getenv("EDGE_STREAM_PROBE") + if mode == "" { + return + } + makeStream := func(ctx context.Context) (io.ReadCloser, error) { + prefix, err := json.Marshal(struct { + StatusCode int `json:"statusCode"` + Headers map[string][]string `json:"multiValueHeaders"` + }{StatusCode: 200, Headers: map[string][]string{"Content-Type": {"text/plain"}}}) + if err != nil { + return nil, err + } + prefix = append(prefix, make([]byte, 8)...) + prefix = append(prefix, "first\n"...) + tail := "last\n" + if mode == "late_error" { + tail = "" + } + url := "http://" + os.Getenv("AWS_LAMBDA_RUNTIME_API") + tailMode := mode + if mode == "guarded_data_and_error" { + tailMode = "data_and_error" + } + return &probeStream{body: io.MultiReader(bytes.NewReader(prefix), &gatedTail{ctx: ctx, url: url, mode: tailMode, tail: bytes.NewReader([]byte(tail))}), url: url, deferErrors: mode == "guarded_data_and_error"}, nil + } + if mode == "typed" { + lambda.Start(func(ctx context.Context, event events.APIGatewayProxyRequest) (io.ReadCloser, error) { + if event.HTTPMethod != "GET" || event.Path != "/" || event.RequestContext.APIID != "example" { + return nil, errors.New("typed input was not decoded") + } + return makeStream(ctx) + }) + return + } + lambda.Start(func(ctx context.Context, event jsontext.Value) (io.ReadCloser, error) { + if string(event) != eventJSON { + return nil, errors.New("raw event capture changed JSON") + } + return makeStream(ctx) + }) +} + +func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { + for _, mode := range []string{"raw", "typed", "late_error", "data_and_error", "guarded_data_and_error"} { + t.Run(mode, func(t *testing.T) { + ctx, cancel := context.WithTimeout(t.Context(), 15*time.Second) + defer cancel() + firstRead := make(chan struct{}, 1) + release := make(chan struct{}) + closed := make(chan struct{}, 1) + next := make(chan struct{}, 1) + type result struct { + body []byte + header http.Header + trailer http.Header + chunked bool + err error + } + results := make(chan result, 1) + var invokes atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/continue": + select { + case <-release: + w.WriteHeader(http.StatusNoContent) + case <-r.Context().Done(): + } + case "/closed": + closed <- struct{}{} + w.WriteHeader(http.StatusNoContent) + case "/2018-06-01/runtime/invocation/next": + if invokes.Add(1) > 1 { + next <- struct{}{} + <-r.Context().Done() + return + } + w.Header().Set("Lambda-Runtime-Aws-Request-Id", "probe") + w.Header().Set("Lambda-Runtime-Deadline-Ms", strconv.FormatInt(time.Now().Add(time.Minute).UnixMilli(), 10)) + if _, err := io.WriteString(w, eventJSON); err != nil { + t.Error(err) + } + case "/2018-06-01/runtime/invocation/probe/response": + // Read only one byte before releasing the second part of the + // stream. A fully buffered implementation cannot pass this. + one := make([]byte, 1) + _, err := io.ReadFull(r.Body, one) + if err != nil { + results <- result{err: err} + return + } + firstRead <- struct{}{} + rest, err := io.ReadAll(r.Body) + results <- result{body: append(one, rest...), header: r.Header.Clone(), trailer: r.Trailer.Clone(), chunked: slices.Contains(r.TransferEncoding, "chunked"), err: err} + w.WriteHeader(http.StatusAccepted) + default: + data, err := io.ReadAll(r.Body) + results <- result{err: errors.New("unexpected Runtime API path " + r.URL.Path + ": " + string(data))} + if err != nil { + t.Error(err) + } + w.WriteHeader(http.StatusAccepted) + } + })) + defer server.Close() + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + cmd := exec.CommandContext(ctx, executable, "-test.run=^TestRuntimeStreamingProbeProcess$") + for _, entry := range os.Environ() { + if !strings.HasPrefix(entry, "AWS_LAMBDA_RUNTIME_API=") && !strings.HasPrefix(entry, "EDGE_STREAM_PROBE=") && !strings.HasPrefix(entry, "_LAMBDA_SERVER_PORT=") { + cmd.Env = append(cmd.Env, entry) + } + } + cmd.Env = append(cmd.Env, "EDGE_STREAM_PROBE="+mode, "AWS_LAMBDA_RUNTIME_API="+server.Listener.Addr().String()) + var output bytes.Buffer + cmd.Stdout, cmd.Stderr = &output, &output + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + defer func() { + cancel() + // The runtime normally loops forever; this test deliberately + // terminates its subprocess after observing the next invocation. + if err := cmd.Wait(); err != nil && ctx.Err() == nil { + t.Errorf("runtime exited: %v; output: %s", err, output.String()) + } + }() + select { + case <-firstRead: + case got := <-results: + t.Fatalf("runtime failed before streaming: %v", got.err) + case <-ctx.Done(): + t.Fatal("no incremental response before deadline") + } + close(release) + var got result + select { + case got = <-results: + case <-ctx.Done(): + t.Fatal("stream did not finish") + } + if got.err != nil { + t.Fatal(got.err) + } + if !got.chunked || got.header.Get("Content-Type") != integrationContentType { + t.Errorf("stream chunked=%v content-type=%q", got.chunked, got.header.Get("Content-Type")) + } + prefix, body, ok := bytes.Cut(got.body, make([]byte, 8)) + if !ok || !jsontext.Value(prefix).IsValid() { + t.Fatalf("invalid integration framing: %q", got.body) + } + wantBody := "first\nlast\n" + wantError := mode == "late_error" || mode == "data_and_error" || mode == "guarded_data_and_error" + if mode == "late_error" || mode == "data_and_error" { + wantBody = "first\n" + } + if string(body) != wantBody || (got.trailer.Get("Lambda-Runtime-Function-Error-Type") != "") != wantError { + t.Errorf("body=%q trailers=%v, want body=%q error=%v", body, got.trailer, wantBody, wantError) + } + t.Logf("mode=%s incremental=true response-mode=%q runtime-error-trailer=%v", mode, got.header.Get("Lambda-Runtime-Function-Response-Mode"), wantError) + for _, notification := range []<-chan struct{}{closed, next} { + select { + case <-notification: + case <-ctx.Done(): + t.Fatal("runtime did not close stream before requesting next invocation") + } + } + }) + } +} From a4726dcd1d6730028988baf80f4ffca38934508c Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 16:37:09 +0100 Subject: [PATCH 10/44] docs: verify latest stable SDK and gateway streaming contract --- docs/decisions/0008-streaming-architecture.md | 23 +++++++++++++------ docs/decisions/0009-streaming-boundaries.md | 12 +++++++++- docs/plan.md | 3 +++ 3 files changed, 30 insertions(+), 8 deletions(-) diff --git a/docs/decisions/0008-streaming-architecture.md b/docs/decisions/0008-streaming-architecture.md index 0c861f9..637e676 100644 --- a/docs/decisions/0008-streaming-architecture.md +++ b/docs/decisions/0008-streaming-architecture.md @@ -22,14 +22,15 @@ buffered/raw/typed contracts or abandoning ordinary net/http handlers? supports statusCode, headers, multiValueHeaders, and cookies. There is no base64 body field. These deployment and framing requirements must match. [Lambda streaming integration contract](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html) -- Lambda documents a 200 MB streamed-response ceiling versus 6 MB buffered, - with bandwidth shaping. Do not blindly reuse the buffered body limit or claim - this is an independently verified end-to-end gateway allowance. An AWS .NET - article states a 10 MB gateway limit, while the gateway streaming guide - describes bandwidth behavior above 10 MB. Resolve that documentation conflict - before promising an exact end-to-end maximum or encoding it as a library limit. +- The full API Gateway streaming guide explicitly states that streaming can + exceed the ordinary 10 MB response limit. Use that service-specific guide over + the conflicting 10 MB statement in an AWS .NET blog article; do not impose a + 10 MB streaming cap. Lambda documents a separate 200 MB streamed-response + ceiling versus 6 MB buffered. Both stages can apply bandwidth shaping; the + gateway describes 2 MB/s after its first 10 MB. No deployed throughput or + boundary measurements have been performed by this project. + [API Gateway streaming guide](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode.html) [Lambda streaming](https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html) - [AWS .NET article](https://aws.amazon.com/blogs/developer/announcing-response-streaming-for-net-on-aws-lambda/) ## SDK and repository evidence @@ -40,6 +41,14 @@ generates the prefix using encoding/json. Reusing that serializer inside edge would conflict with our direct JSON v2 requirement. [Pinned SDK streaming response](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/apigw.go) +On 2026-09-14, both `go list -m -json github.com/aws/aws-lambda-go@latest` and +AWS's GitHub latest-release endpoint resolve to v1.55.0. The release is neither +draft nor prerelease, was published on 2026-08-27, and explicitly includes Go 1.27 +compatibility work. go.mod already requires this version. "Pinned" means the +exact version recorded for reproducible builds, not a policy of remaining on an +older release. Use the latest stable SDK and verify compatibility when updating. +[Latest stable release verified](https://github.com/aws/aws-lambda-go/releases/tag/v1.55.0) + The SDK's function-handler path can return readers. Its runtime loop consumes and closes the reader and honors its ContentType method. Its legacy/public Handler.Invoke byte-slice interface instead reads the complete response into diff --git a/docs/decisions/0009-streaming-boundaries.md b/docs/decisions/0009-streaming-boundaries.md index 4a0fda5..76158e8 100644 --- a/docs/decisions/0009-streaming-boundaries.md +++ b/docs/decisions/0009-streaming-boundaries.md @@ -195,6 +195,16 @@ The SDK also provides official streaming response examples, so this observation alone does not prove the service rejects or buffers its output. It is a concrete documentation/SDK mismatch, not resolved by a permissive local test server. +The SDK is the latest stable release: both the Go module proxy and AWS GitHub +release endpoint were checked on 2026-09-14 and returned v1.55.0. Reading the full +API Gateway guide through AWS MCP confirms its supported combination is STREAM, +InvokeWithResponseStream, and correctly framed metadata/body output. That guide +does not impose this Runtime API header itself. Keep the header question scoped +to the Lambda runtime boundary; it is not evidence that API Gateway streaming +or the SDK's documented streaming response type is unsupported. +[API Gateway supported configurations](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html) +[AWS SDK stable release](https://github.com/aws/aws-lambda-go/releases/tag/v1.55.0) + Recommend retaining SDK reuse as the design, but treat this mismatch as a release gate: seek an authoritative clarification or a suitable SDK fix/version; if needed, propose a separately approved minimal AWS integration test. Do not patch @@ -210,7 +220,7 @@ behavior, not API Gateway delivery or billing/error metrics. 3. Implement shared header validation and separate streaming writer/framing. 4. Wire the proposed raw and typed REST entry points with identity policy. 5. Exercise SSE, NDJSON, binary output, early Close, slow reads, and warm reuse; - resolve the AWS runtime-header mismatch and service-limit questions before + resolve the AWS runtime-header mismatch and exact prefix-boundary questions before advertising deployment support. The first complete buffered adapter remains the initial deliverable. The local diff --git a/docs/plan.md b/docs/plan.md index 19d0b33..75ddadf 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -24,6 +24,9 @@ port, source copy, or compatibility-preserving update of Beakley. `Invoke(context.Context, []byte) ([]byte, error)`. - Reuse AWS Lambda Go event/response structs where they meet the contract; reserve custom representations for demonstrated limitations. +- Use the latest stable aws-lambda-go release, recorded explicitly in go.mod + for reproducibility. On 2026-09-14, the Go module proxy and AWS GitHub releases + both confirmed v1.55.0, already required here; it includes Go 1.27 compatibility. - Support already typed events without an envelope JSON round trip inside edge. Decision 0004 accepts HandleV1 and HandleV2 with matching AWS request/response types, sharing translation and identity policy with raw invocation. From 53a98be507cad9cd7f4c105a297d9237cfb326f1 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 16:55:34 +0100 Subject: [PATCH 11/44] Implement private streaming bridge and invocation ownership --- README.md | 6 + docs/decisions/0009-streaming-boundaries.md | 28 +- docs/decisions/README.md | 2 +- docs/plan.md | 17 +- invocation.go | 45 +- stream.go | 260 +++++++++++ stream_test.go | 459 ++++++++++++++++++++ 7 files changed, 794 insertions(+), 23 deletions(-) create mode 100644 stream.go create mode 100644 stream_test.go diff --git a/README.md b/README.md index 94e2d13..35a0bf8 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,12 @@ v2 output. The buffered HTTP writer, gateway header conversion, public invocatio methods, and identity extraction are not implemented yet; the adapter cannot yet be registered as a Lambda handler. +The accepted streaming design now has a private bridge with incremental delivery, +backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle +is covered by race-enabled synthetic concurrency tests. HTTP streaming framing +and public streaming entry points remain under construction; deployed API Gateway +streaming has not been verified. + - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) - [Repository working agreement](AGENTS.md) diff --git a/docs/decisions/0009-streaming-boundaries.md b/docs/decisions/0009-streaming-boundaries.md index 76158e8..42afc96 100644 --- a/docs/decisions/0009-streaming-boundaries.md +++ b/docs/decisions/0009-streaming-boundaries.md @@ -1,10 +1,10 @@ # 0009: Streaming entry points, ownership, and failure boundaries -Status: proposed; high-level architecture accepted in decision 0008. +Status: accepted by the user on 2026-09-14; implementation in progress. ## Recommended public boundary -Keep a distinct adapter in the root edge package. Proposed signatures: +Keep a distinct adapter in the root edge package. Accepted signatures: ```go func NewStreaming(handler http.Handler, opts ...Option) (*StreamingAdapter, error) @@ -143,7 +143,7 @@ of cooperative handlers, not the ability to kill arbitrary application code. The late-panic rule intentionally differs from the SDK's synchronous panic path, which can terminate the process. Do not introduce os.Exit or an unhandled worker -panic into the library. This difference needs explicit user approval. +panic into the library. The user explicitly approved this difference. AWS says midstream error trailers are treated as a successful response with error metadata. Do not claim they necessarily increment Lambda's ordinary invocation @@ -225,5 +225,23 @@ behavior, not API Gateway delivery or billing/error metrics. The first complete buffered adapter remains the initial deliverable. The local SDK probe and design allow streaming requirements to shape shared ownership now. -The detailed API and policy in this record are proposed, not implemented or -approved by the prior agreement to the high-level architecture. +The user approved the detailed API, ownership, handoff, and failure policies on +2026-09-14 and asked to start with the private bridge. This acceptance does not +resolve the separate metadata-policy or AWS compatibility questions above. + +Implemented on 2026-09-14: shared invocation context/cleanup primitives and the +private bridge in `stream.go`. Publication waits for the entry point to accept +ownership before body writes; the producer cannot finish publication ahead of +that handoff. The bridge copies its prefix and otherwise uses io.Pipe without +body accumulation. The future framing layer must validate and bound that prefix. + +Tests in `stream_test.go` cover incremental reads, backpressure, prefix ownership, +concurrent Close/Read, parent cancellation/deadlines without a consumer, waiting +for cleanup before terminal results, error precedence, both panic boundaries, +reporter panic isolation, and bytes-plus-error normalization. Go 1.27 synctest +bubbles also require their goroutines to exit. Full race tests, the existing SDK +probe, vet, formatting, and Linux arm64/amd64 builds pass on Go 1.27.1. + +This is the ownership/transport foundation. It does not yet implement the HTTP +streaming writer, prefix codec/limit, constructor/options, or public entry points, +and it does not resolve the documented AWS Runtime API header mismatch. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 5a955a3..f6e64f7 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -17,4 +17,4 @@ Keep related decisions small enough to review and implement independently. | [0006: Buffered responses](0006-buffered-responses.md) | Accepted | How should HTTP responses, unsupported capabilities, and completion failures behave? | | [0007: Response metadata](0007-response-metadata.md) | Qualified direction accepted; details open | What metadata budgets and V2 joinable-field set should apply? | | [0008: Streaming architecture](0008-streaming-architecture.md) | Accepted | How should REST response streaming fit alongside the buffered adapter? | -| [0009: Streaming boundaries](0009-streaming-boundaries.md) | Proposed | What API, lifecycle, handoff, and error rules should streaming use? | +| [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | diff --git a/docs/plan.md b/docs/plan.md index 75ddadf..0fc0880 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -110,12 +110,23 @@ resource and compatibility justification. Streaming is now being discussed in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. The high-level architecture is accepted. [Decision 0009](decisions/0009-streaming-boundaries.md) -proposes concrete APIs and an ownership/state graph. A local SDK Runtime API +accepts concrete APIs and an ownership/state graph. The private bridge and shared +invocation ownership primitives are implemented. A local SDK Runtime API probe demonstrates incremental delivery, raw/typed capture, closure, and error trailers; it also exposes a bytes-plus-error reader issue and a missing header relative to AWS's documented streaming contract. No production streaming API is implemented, and deployed API Gateway behavior is not verified. +The private bridge publishes an owned prefix through an explicit handoff before +body writes can block, uses an unbuffered pipe, and transfers cleanup to one +producer. Close and parent cancellation release blocked I/O; terminal reads and +Close join cleanup and error reporting. Early panics are rethrown at entry, late +panics become sanitized terminal errors, and reader results defer errors that +accompany bytes. Synthetic concurrency tests cover these boundaries and reporter +failures. Full race tests (including the existing SDK probe), vet, formatting, +and Linux arm64/amd64 builds pass on Go 1.27.1. HTTP streaming commitment, +sniffing, framing validation, and public entry points remain to be implemented. + Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary errors, cleans up on failures/panics, and checks parent cancellation. The HTTP @@ -152,8 +163,10 @@ presenting an entire frozen public API at once. - [x] Implement the private decoder and typed semantic validators with direct JSON v2, SDK types, opaque authorizer data, fixtures, and fuzzing. - [x] Implement shared HTTP request conversion and request-lifetime cleanup. -- [ ] Implement buffered responses and move cleanup into a shared invocation scope. +- [ ] Implement the buffered HTTP writer and gateway response conversion. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. +- [x] Extract shared invocation ownership and implement the private streaming + bridge, with handoff, backpressure, cancellation, cleanup, and error tests. - [ ] Implement the minimum reviewed identity/context and native gateway producers needed to honor WithGatewayIdentity(true) before publishing invocation. - [ ] Implement raw invocation after the remaining transport contracts are reviewed. diff --git a/invocation.go b/invocation.go index aa09322..435a7f4 100644 --- a/invocation.go +++ b/invocation.go @@ -21,29 +21,44 @@ func (inv *invocation) ownRequest(r *http.Request) { inv.body = r.Body } -// withInvocation scopes conversion, identity, serving, and finalization. A panic -// passes through after cleanup. The caller must discard its result on error. -func withInvocation(parent context.Context, run func(context.Context, *invocation) error) (err error) { +// cleanup runs once under the invocation owner's control, after application +// execution. Transport errors remain primary; cleanup details stay private. +func (inv *invocation) cleanup(err error) error { + if inv.body != nil { + if closeErr := inv.body.Close(); closeErr != nil { + err = errors.Join(err, errors.New("edge: request body cleanup failed")) + } + } + if inv.request != nil && inv.request.MultipartForm != nil { + if removeErr := inv.request.MultipartForm.RemoveAll(); removeErr != nil { + err = errors.Join(err, errors.New("edge: multipart cleanup failed")) + } + } + return err +} + +func invocationContext(parent context.Context) (context.Context, context.CancelFunc, error) { if parent == nil { - return errors.New("edge: nil invocation context") + return nil, nil, errors.New("edge: nil invocation context") } if err := parent.Err(); err != nil { - return err + return nil, nil, err } ctx, cancel := context.WithCancel(parent) + return ctx, cancel, nil +} + +// withInvocation scopes conversion, identity, serving, and finalization. A panic +// passes through after cleanup. The caller must discard its result on error. +func withInvocation(parent context.Context, run func(context.Context, *invocation) error) (err error) { + ctx, cancel, err := invocationContext(parent) + if err != nil { + return err + } defer cancel() var inv invocation defer func() { - if inv.body != nil { - if closeErr := inv.body.Close(); closeErr != nil { - err = errors.Join(err, errors.New("edge: request body cleanup failed")) - } - } - if inv.request != nil && inv.request.MultipartForm != nil { - if removeErr := inv.request.MultipartForm.RemoveAll(); removeErr != nil { - err = errors.Join(err, errors.New("edge: multipart cleanup failed")) - } - } + err = inv.cleanup(err) // Normal child cancellation is not a failed invocation. Preserve an // existing operation/cleanup error when the parent also canceled. if err == nil { diff --git a/stream.go b/stream.go new file mode 100644 index 0000000..eefbc49 --- /dev/null +++ b/stream.go @@ -0,0 +1,260 @@ +package edge + +import ( + "bytes" + "context" + "errors" + "io" + "sync" +) + +var ( + errStreamPanic = errors.New("edge: stream producer panicked") + errStreamReporter = errors.New("edge: stream error reporter panicked") + errStreamUnpublished = errors.New("edge: stream producer ended without publishing a response") +) + +// streamOutput belongs to the producer. publish accepts an already validated, +// bounded wire prefix and completes the handoff before body writes may block. +// Its caller must supply sanitized errors and publish exactly once. +type streamOutput struct { + publish func([]byte) error + body io.Writer +} + +// responseStream separates producer lifetime from the entry point's stack. +// mu protects handoff, interruption, and terminal commitment. done publishes the +// final error/panic after cleanup and reporting. readMu only serializes readers; +// interruption and Close must never acquire it while a Read can be blocked. +type responseStream struct { + reader *io.PipeReader + writer *io.PipeWriter + offer chan []byte + accept chan struct{} + done chan struct{} + closed chan struct{} + cancel context.CancelFunc + + mu sync.Mutex + handedOff bool + settled bool + stopErr error + err error + panicValue any + + readMu sync.Mutex + source io.Reader + pending error + closeOnce sync.Once +} + +// startStream performs preparation synchronously, then transfers request cleanup +// to one producer. Until publication, errors return directly and producer panics +// are rethrown here. The caller must consume and Close a successful result. +// prepare and produce are private transport callbacks, not application hooks. +func startStream(parent context.Context, prepare func(context.Context, *invocation) error, produce func(context.Context, *streamOutput) error, report func(context.Context, error)) (result *responseStream, err error) { + ctx, cancel, err := invocationContext(parent) + if err != nil { + return nil, err + } + var inv invocation + owned := true + defer func() { + if owned { + defer cancel() + err = inv.cleanup(err) + } + }() + if err := prepare(ctx, &inv); err != nil { + return nil, err + } + if err := ctx.Err(); err != nil { + return nil, err + } + reader, writer := io.Pipe() + s := &responseStream{ + reader: reader, + writer: writer, + offer: make(chan []byte), + accept: make(chan struct{}), + done: make(chan struct{}), + closed: make(chan struct{}), + cancel: cancel, + } + watcherDone := make(chan struct{}) + stopWatcher := context.AfterFunc(ctx, func() { + defer close(watcherDone) + s.interrupt(ctx.Err()) + }) + output := &streamOutput{ + body: writer, + publish: func(prefix []byte) error { + select { + case s.offer <- bytes.Clone(prefix): + case <-ctx.Done(): + return ctx.Err() + } + select { + case <-s.accept: + return nil + case <-ctx.Done(): + return ctx.Err() + } + }, + } + owned = false + go func() { + var producerErr error + defer func() { + panicValue := recover() + if panicValue != nil { + producerErr = errors.Join(errStreamPanic, producerErr) + } + // stop does not join an already running AfterFunc callback. + if !stopWatcher() { + <-watcherDone + } + s.finish(ctx, producerErr, panicValue, report) + }() + defer func() { producerErr = inv.cleanup(producerErr) }() + producerErr = produce(ctx, output) + }() + + select { + case prefix := <-s.offer: + s.mu.Lock() + if !s.settled && s.stopErr == nil && ctx.Err() == nil { + // This acknowledgement is the error boundary. The producer cannot + // finish publication or write until the stream has an owner. + s.handedOff = true + s.source = io.MultiReader(bytes.NewReader(prefix), reader) + close(s.accept) + s.mu.Unlock() + return s, nil + } + s.mu.Unlock() + case <-s.done: + case <-ctx.Done(): + } + s.interrupt(ctx.Err()) + <-s.done + cancel() + if s.panicValue != nil { + panic(s.panicValue) + } + return nil, s.err +} + +// interrupt never joins the producer: it is also called by the cancellation +// watcher that the producer itself must join. Closing the reader unblocks both +// directions of the pipe, including writes when no consumer is reading. +func (s *responseStream) interrupt(err error) { + s.mu.Lock() + defer s.mu.Unlock() + if s.settled || s.stopErr != nil { + return + } + if err == nil { + err = context.Canceled + } + s.stopErr = err + s.cancel() + _ = s.reader.CloseWithError(err) +} + +func (s *responseStream) finish(ctx context.Context, err error, panicValue any, report func(context.Context, error)) { + s.mu.Lock() + s.settled = true + // A pipe-close error is only a consequence of interruption. Other producer + // and cleanup faults retain precedence over concurrent cancellation. + if err == nil || err == io.ErrClosedPipe { + if s.stopErr != nil { + err = s.stopErr + } else if ctx.Err() != nil { + err = ctx.Err() + } + } + if !s.handedOff { + s.panicValue = panicValue + if err == nil { + err = errStreamUnpublished + } + } + reportFailure := s.handedOff && err != nil && report != nil + s.mu.Unlock() + if reportFailure { + err = reportStreamError(ctx, report, err) + } + s.err = err + _ = s.writer.CloseWithError(err) + close(s.done) +} + +func reportStreamError(ctx context.Context, report func(context.Context, error), err error) (result error) { + result = err + defer func() { + if recover() != nil { + result = errors.Join(err, errStreamReporter) + } + }() + report(ctx, err) + return result +} + +func (s *responseStream) Read(p []byte) (int, error) { + s.readMu.Lock() + defer s.readMu.Unlock() + if len(p) == 0 { + return 0, nil + } + select { + case <-s.closed: + <-s.done + if s.err != nil { + return 0, s.err + } + return 0, io.ErrClosedPipe + default: + } + if s.pending != nil { + return 0, s.pending + } + n, err := s.source.Read(p) + if err != nil { + // Interruption can close the pipe before cleanup finishes. Never expose + // terminal status until producer cleanup and reporting have completed. + <-s.done + s.cancel() + if s.err != nil { + err = s.err + } + if n > 0 { + // aws-lambda-go v1.55.0 discards n on a non-EOF reader error. + // Retain the terminal error for the next nonempty Read instead. + s.pending = err + return n, nil + } + } + return n, err +} + +func (s *responseStream) Close() error { + s.closeOnce.Do(func() { + close(s.closed) + s.cancel() + s.interrupt(context.Canceled) + <-s.done + _ = s.reader.Close() + }) + return s.err +} + +func (*responseStream) ContentType() string { + return "application/vnd.awslambda.http-integration-response" +} + +// MarshalJSON deliberately declines serialization so the SDK selects its +// io.Reader response path. Edge's framing codec uses JSON v2 separately. +func (*responseStream) MarshalJSON() ([]byte, error) { + return nil, errors.New("edge: streaming response cannot be marshaled as JSON") +} diff --git a/stream_test.go b/stream_test.go new file mode 100644 index 0000000..6dc425a --- /dev/null +++ b/stream_test.go @@ -0,0 +1,459 @@ +package edge + +import ( + "bytes" + "context" + "errors" + "io" + "net/http" + "strings" + "testing" + "testing/synctest" + "time" +) + +func prepareEmptyStream(context.Context, *invocation) error { return nil } + +func TestStreamHandoffBackpressureAndCleanup(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + body := &trackedBody{Reader: http.NoBody} + var scope context.Context + prepare := func(ctx context.Context, inv *invocation) error { + scope = ctx + inv.ownRequest(&http.Request{Body: body}) + return nil + } + prefix := []byte("prefix") + published := make(chan struct{}) + written := make(chan struct{}) + payload := bytes.Repeat([]byte("x"), 128*1024) + s, err := startStream(t.Context(), prepare, func(ctx context.Context, out *streamOutput) error { + if err := out.publish(prefix); err != nil { + return err + } + clear(prefix) // The consumer must own its copy before publish returns. + close(published) + _, err := out.body.Write(payload) + close(written) + return err + }, nil) + if err != nil { + t.Fatal(err) + } + defer s.Close() + <-published + synctest.Wait() + select { + case <-written: + t.Error("producer completed body write without a consumer") + default: + } + if body.closed || scope.Err() != nil { + t.Errorf("after handoff: body closed=%v, context=%v", body.closed, scope.Err()) + } + first := make([]byte, len(prefix)) + if _, err := io.ReadFull(s, first); err != nil || string(first) != "prefix" { + t.Fatalf("initial Read = %q, %v; want owned prefix", first, err) + } + // A small read cannot drain a large write into hidden adapter buffering. + if _, err := io.ReadFull(s, make([]byte, 7)); err != nil { + t.Fatal(err) + } + synctest.Wait() + select { + case <-written: + t.Error("producer completed a large write after only seven body bytes were read") + default: + } + rest, err := io.ReadAll(s) + if err != nil || !bytes.Equal(rest, payload[7:]) { + t.Errorf("remaining body: bytes=%d err=%v, want %d bytes", len(rest), err, len(payload)-7) + } + if !body.closed || scope.Err() != context.Canceled { + t.Errorf("after EOF: body closed=%v, context=%v", body.closed, scope.Err()) + } + if err := s.Close(); err != nil { + t.Errorf("Close after EOF = %v, want nil", err) + } + if s.ContentType() != "application/vnd.awslambda.http-integration-response" { + t.Errorf("ContentType = %q", s.ContentType()) + } + if data, err := s.MarshalJSON(); err == nil || len(data) != 0 { + t.Errorf("MarshalJSON = %q, %v; want rejection", data, err) + } + }) +} + +type streamCloseBody struct { + io.Reader + close func() error +} + +func (b *streamCloseBody) Close() error { return b.close() } + +func TestStreamCloseCancelsUnblocksAndJoins(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + allowCleanup := make(chan struct{}) + var scope context.Context + cleaned := false + reports := 0 + body := &streamCloseBody{Reader: http.NoBody, close: func() error { + <-allowCleanup + cleaned = true + return nil + }} + prepare := func(ctx context.Context, inv *invocation) error { + scope = ctx + inv.ownRequest(&http.Request{Body: body}) + return nil + } + s, err := startStream(t.Context(), prepare, func(ctx context.Context, out *streamOutput) error { + if err := out.publish([]byte("prefix")); err != nil { + return err + } + _, err := out.body.Write([]byte("blocked body")) + return err + }, func(ctx context.Context, err error) { + reports++ + if !cleaned || ctx.Err() != context.Canceled || !errors.Is(err, context.Canceled) { + t.Errorf("report: cleaned=%v context=%v err=%v", cleaned, ctx.Err(), err) + } + }) + if err != nil { + t.Fatal(err) + } + closed := make(chan error, 1) + go func() { closed <- s.Close() }() + synctest.Wait() + if scope.Err() != context.Canceled { + t.Errorf("Close did not cancel request: %v", scope.Err()) + } + select { + case err := <-closed: + t.Fatalf("Close returned before cleanup: %v", err) + default: + } + read := make(chan error, 1) + go func() { + _, err := io.ReadAll(s) + read <- err + }() + synctest.Wait() + select { + case err := <-read: + t.Fatalf("Read exposed terminal error before cleanup: %v", err) + default: + } + close(allowCleanup) + if err := <-closed; !errors.Is(err, context.Canceled) { + t.Errorf("Close = %v, want canceled", err) + } + if err := <-read; !errors.Is(err, context.Canceled) { + t.Errorf("Read = %v, want canceled", err) + } + if err := s.Close(); !errors.Is(err, context.Canceled) || reports != 1 { + t.Errorf("repeated Close = %v, reports=%d; want canceled and one report", err, reports) + } + }) +} + +func TestStreamParentCancellationWithoutConsumer(t *testing.T) { + for _, deadline := range []bool{false, true} { + name := "cancel" + if deadline { + name = "deadline" + } + t.Run(name, func(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + parent, cancel := context.WithTimeout(t.Context(), time.Hour) + defer cancel() + body := &trackedBody{Reader: http.NoBody} + prepare := func(ctx context.Context, inv *invocation) error { + inv.ownRequest(&http.Request{Body: body}) + return nil + } + s, err := startStream(parent, prepare, func(ctx context.Context, out *streamOutput) error { + if err := out.publish([]byte("prefix")); err != nil { + return err + } + _, err := out.body.Write([]byte("blocked")) + return err + }, nil) + if err != nil { + t.Fatal(err) + } + defer s.Close() + if deadline { + time.Sleep(time.Hour) // Virtual time inside the synctest bubble. + } else { + cancel() + } + synctest.Wait() + if !body.closed { + t.Error("parent cancellation did not release blocked producer and clean request") + } + if _, err := io.ReadAll(s); !errors.Is(err, parent.Err()) { + t.Errorf("Read after parent cancellation = %v, want %v", err, parent.Err()) + } + }) + }) + } +} + +func TestStreamCloseReleasesBlockedRead(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + s, err := startStream(t.Context(), prepareEmptyStream, func(ctx context.Context, out *streamOutput) error { + if err := out.publish(nil); err != nil { + return err + } + <-ctx.Done() + return ctx.Err() + }, nil) + if err != nil { + t.Fatal(err) + } + read := make(chan error, 1) + go func() { + _, err := s.Read(make([]byte, 1)) + read <- err + }() + synctest.Wait() + select { + case err := <-read: + t.Fatalf("Read returned before data or cancellation: %v", err) + default: + } + closed := make(chan error, 2) + for range 2 { + go func() { closed <- s.Close() }() + } + for range 2 { + if err := <-closed; !errors.Is(err, context.Canceled) { + t.Errorf("concurrent Close = %v, want canceled", err) + } + } + if err := <-read; !errors.Is(err, context.Canceled) { + t.Errorf("blocked Read = %v, want canceled", err) + } + }) +} + +func TestStreamEOFWaitsForCleanupAndKeepsScopeUntilConsumption(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + allowCleanup := make(chan struct{}) + var scope context.Context + cleaned := false + prepare := func(ctx context.Context, inv *invocation) error { + scope = ctx + inv.ownRequest(&http.Request{Body: &streamCloseBody{Reader: http.NoBody, close: func() error { + <-allowCleanup + cleaned = true + return nil + }}}) + return nil + } + s, err := startStream(t.Context(), prepare, func(ctx context.Context, out *streamOutput) error { + return out.publish([]byte("prefix")) + }, nil) + if err != nil { + t.Fatal(err) + } + defer s.Close() + if _, err := io.ReadFull(s, make([]byte, len("prefix"))); err != nil { + t.Fatal(err) + } + read := make(chan error, 1) + go func() { + _, err := s.Read(make([]byte, 1)) + read <- err + }() + synctest.Wait() + select { + case err := <-read: + t.Fatalf("terminal Read completed before cleanup: %v", err) + default: + } + if scope.Err() != nil { + t.Errorf("scope canceled before completion: %v", scope.Err()) + } + close(allowCleanup) + if err := <-read; err != io.EOF || !cleaned || scope.Err() != context.Canceled { + t.Errorf("terminal Read = %v, cleaned=%v context=%v", err, cleaned, scope.Err()) + } + }) +} + +func TestStreamFailuresBeforeHandoff(t *testing.T) { + failure := errors.New("edge: preparation or production failed") + panicValue := new("private application panic") + for _, stage := range []string{"prepare_error", "prepare_panic", "producer_error", "producer_panic", "no_publication", "canceled"} { + t.Run(stage, func(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + parent, cancel := context.WithCancel(t.Context()) + defer cancel() + body := &trackedBody{Reader: http.NoBody} + var scope context.Context + prepare := func(ctx context.Context, inv *invocation) error { + scope = ctx + inv.ownRequest(&http.Request{Body: body}) + if stage == "prepare_error" { + return failure + } + if stage == "prepare_panic" { + panic(panicValue) + } + return nil + } + var recovered any + var result *responseStream + var err error + reports := 0 + func() { + defer func() { recovered = recover() }() + result, err = startStream(parent, prepare, func(ctx context.Context, out *streamOutput) error { + switch stage { + case "producer_error": + return failure + case "producer_panic": + panic(panicValue) + case "canceled": + cancel() + <-ctx.Done() + return ctx.Err() + } + return nil + }, func(context.Context, error) { reports++ }) + }() + if result != nil || !body.closed || scope.Err() != context.Canceled || reports != 0 { + t.Errorf("failed entry: result=%v closed=%v context=%v reports=%d", result, body.closed, scope.Err(), reports) + } + if strings.HasSuffix(stage, "panic") { + if recovered != panicValue { + t.Errorf("panic = %v, want original value", recovered) + } + return + } + want := failure + if stage == "no_publication" { + want = errStreamUnpublished + } else if stage == "canceled" { + want = context.Canceled + } + if recovered != nil || !errors.Is(err, want) { + t.Errorf("entry error=%v panic=%v, want %v and no panic", err, recovered, want) + } + }) + }) + } +} + +func TestStreamLateFailurePreservesBytesAndReportsAfterCleanup(t *testing.T) { + for _, mode := range []string{"error", "panic", "cleanup_error", "cancel_and_error", "reporter_panic"} { + t.Run(mode, func(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + parent, cancel := context.WithCancel(t.Context()) + defer cancel() + cleaned := false + body := &streamCloseBody{Reader: http.NoBody, close: func() error { + cleaned = true + if mode == "cleanup_error" { + return errors.New("private cleanup path") + } + return nil + }} + prepare := func(ctx context.Context, inv *invocation) error { + inv.ownRequest(&http.Request{Body: body}) + return nil + } + reports := 0 + var reported error + s, err := startStream(parent, prepare, func(ctx context.Context, out *streamOutput) error { + if err := out.publish([]byte("prefix")); err != nil { + return err + } + if _, err := out.body.Write([]byte("final bytes")); err != nil { + return err + } + if mode == "panic" { + panic("private panic data") + } + if mode == "cancel_and_error" { + cancel() + } + if mode == "cleanup_error" { + return nil + } + return http.ErrContentLength + }, func(ctx context.Context, err error) { + reports++ + reported = err + if !cleaned { + t.Error("reported before request cleanup") + } + if mode == "reporter_panic" { + panic("private reporter data") + } + }) + if err != nil { + t.Fatal(err) + } + defer s.Close() + data, err := io.ReadAll(s) + if string(data) != "prefixfinal bytes" || err == nil || reports != 1 || reported == nil { + t.Fatalf("stream = %q, %v; reports=%d (%v)", data, err, reports, reported) + } + if strings.Contains(err.Error(), "private") || strings.Contains(reported.Error(), "private") { + t.Errorf("terminal error exposed private details: %v; reported %v", err, reported) + } + if mode == "panic" && !errors.Is(err, errStreamPanic) { + t.Errorf("late panic = %v, want stream panic error", err) + } + if mode != "panic" && mode != "cleanup_error" && !errors.Is(err, http.ErrContentLength) { + t.Errorf("terminal error lost primary fault: %v", err) + } + if mode == "cancel_and_error" && errors.Is(err, context.Canceled) { + t.Errorf("cancellation replaced primary fault: %v", err) + } + if mode == "reporter_panic" && !errors.Is(err, errStreamReporter) { + t.Errorf("reporter panic was not retained: %v", err) + } + }) + }) + } +} + +type finalBytesReader struct { + err error +} + +func (r *finalBytesReader) Read(p []byte) (int, error) { + return copy(p, "last"), r.err +} + +func TestStreamDefersErrorAccompanyingBytes(t *testing.T) { + for _, terminal := range []error{io.EOF, http.ErrContentLength} { + // Inject a legal Reader result that the current SDK mishandles. This + // protects the runtime-facing Read contract independently of io.Pipe. + done := make(chan struct{}) + close(done) + s := &responseStream{ + source: &finalBytesReader{err: terminal}, + done: done, + closed: make(chan struct{}), + cancel: func() {}, + } + if terminal != io.EOF { + s.err = terminal + } + p := make([]byte, 8) + if n, err := s.Read(p); n != 4 || err != nil || string(p[:n]) != "last" { + t.Fatalf("Read bytes+%v = %q, %v; want last, nil", terminal, p[:n], err) + } + if n, err := s.Read(nil); n != 0 || err != nil { + t.Errorf("empty Read = %d, %v; want 0, nil", n, err) + } + if n, err := s.Read(p); n != 0 || !errors.Is(err, terminal) { + t.Errorf("terminal Read = %d, %v; want 0, %v", n, err, terminal) + } + } +} From 931aa60fea01feb67bafea0f746f7ce7cae8e2f2 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 17:04:00 +0100 Subject: [PATCH 12/44] Propose shared response header policy and streaming prefix limits --- docs/decisions/0007-response-metadata.md | 5 + docs/decisions/0010-response-header-design.md | 236 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 5 +- 4 files changed, 246 insertions(+), 1 deletion(-) create mode 100644 docs/decisions/0010-response-header-design.md diff --git a/docs/decisions/0007-response-metadata.md b/docs/decisions/0007-response-metadata.md index 09db6e1..f2e94cf 100644 --- a/docs/decisions/0007-response-metadata.md +++ b/docs/decisions/0007-response-metadata.md @@ -2,6 +2,11 @@ Status: qualified direction accepted; concrete metadata policy remains unresolved. +The concrete follow-up is [decision 0010](0010-response-header-design.md), proposed +after the streaming bridge implementation. It recommends a broader field audit, +weighted resource accounting, and separate encoded-output limits. Its details +still require user review; this record preserves the earlier qualification. + ## Accepted qualification The user accepted the qualified recommendation: keep the documented Lambda diff --git a/docs/decisions/0010-response-header-design.md b/docs/decisions/0010-response-header-design.md new file mode 100644 index 0000000..e67c063 --- /dev/null +++ b/docs/decisions/0010-response-header-design.md @@ -0,0 +1,236 @@ +# 0010: Shared response headers and transport-specific encoding + +Status: proposed for user review on 2026-09-14. No dependent implementation yet. + +## Recommendation + +Keep the application boundary as http.Header. Capture one private, owned, +canonical multivalue snapshot at HTTP commitment. Validate/filter it using shared +rules, then encode it for the selected gateway response format. Only V2 needs +comma combination; REST streaming shares V1's multivalue representation. + +Replace decision 0007's initial ten-field audit with the broader table below. +Replace its unaccepted 64 KiB/1,024-entry pair with a single weighted resource +budget tied to the existing Lambda envelope ceiling. Keep actual envelope and +stream-prefix checks separate. These are proposed refinements, not changes to +the already implemented bridge. + +## Evidence and limits of the evidence + +AWS public MCP was used to read the complete current gateway response-format, +streaming-integration, REST-quota, and HTTP-API-quota pages. The sources establish: + +| Boundary | What the documentation establishes | +| --- | --- | +| REST / HTTP API payload 1.0 | MultiValueHeaders can carry single and repeated values | +| HTTP API payload 2.0 | Headers contains one string per name; Cookies carries separate Set-Cookie values | +| REST streaming | Metadata supports MultiValueHeaders and Cookies; all extra headers can use MultiValueHeaders | +| Streaming framing | JSON followed by eight NUL bytes; delimiter must appear within the first 16 KB | +| REST quotas | Current table lists 20,480 bytes for combined header names, values, whitespace and terminators; private APIs have an 8,000-byte entry | +| HTTP API quotas | The 10,240-byte entry explicitly covers the request line and headers; it does not establish an equivalent response limit | + +Sources: [HTTP API formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html), +[streaming format](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html), +[REST quotas](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-execution-service-limits-table.html), +[HTTP API quotas](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-quotas.html). + +These are not interchangeable byte counts. Edge does not know every deployment +setting or the final headers added by Gateway/CDN layers. Passing edge's checks +cannot guarantee gateway acceptance. Do not turn the HTTP API request quota into +a response limit, infer private/public endpoint configuration from payload V1, +or promise an exact final HTTP wire size from a Lambda JSON envelope. + +HTTP allows comma combination when the field definition supports it, with +Set-Cookie requiring special handling. The IANA registry provides useful type +and specification references, but registration alone is not permission to join. +Structured Fields Lists and Dictionaries support combination; Items do not +become lists merely because their names are registered. +[RFC 9110 sections 5.2–5.5](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.2), +[IANA field registry](https://www.iana.org/assignments/http-fields/), +[RFC 9651 section 4.2](https://www.rfc-editor.org/rfc/rfc9651.html#section-4.2). + +Go 1.27's new DefaultMaxHeaderValueCount is 500, but it is a request-parser limit. +It is useful evidence that value count matters, not authority for applying 500 +to edge responses. The installed Go 1.27.1 ResponseWriter documentation confirms +header commitment and nil-value suppression. Our request helpers already have +token/control-character checks, but their copying is not resource-bounded and +must not be reused wholesale for response snapshots. +[Go ResponseWriter](https://pkg.go.dev/net/http#ResponseWriter), +[Go request value-count limit](https://pkg.go.dev/net/http#DefaultMaxHeaderValueCount). + +## Shared commitment pipeline + +1. Preflight the application map's resource charge without allocating a sorted + key slice or cloning values. Reject over-budget input before snapshot growth. +2. Recognize Go's TrailerPrefix as an unsupported feature before generic name + validation. Reject invalid token names, invalid UTF-8, CR/LF, NUL, DEL and other control + bytes except HTAB. Reuse small syntax predicates where correct; response + UTF-8 validation remains explicit. Reject invalid data rather than silently + removing it or replacing bytes during JSON encoding. +3. Sort original keys lexically, canonicalize names, concatenate same-name + slices in that deterministic key order, and preserve slice element order. + This keeps the accepted mixed-case policy. A Go map cannot recover insertion + order across differently spelled keys; recommend Header.Add/Set to callers. +4. Normalize only leading/trailing SP and HTAB in values. Preserve interior + whitespace, quoting, commas and parameters. Do not split values on commas, + deduplicate values, rewrite CSP directives, or parse arbitrary field grammars. +5. Preserve the distinction between absence and an explicitly present nil/empty + slice until automatic-header decisions finish. Empty slices emit no field; + a slice containing an empty string represents an actual empty field value. + If differently cased aliases include actual values, those values win over a + nil alias: a suppression marker cannot erase explicit content. +6. Detect unsupported upgrades/trailer declarations before hop-header removal. + Parse Connection's comma-separated tokens case-insensitively; malformed + tokens are faults. Remove Connection-nominated fields and the fixed + connection-specific set listed below. Preserve removal/suppression state so + automatic Content-Type/Content-Length cannot reintroduce a nominated field. +7. Apply the accepted status/method, Content-Type and Content-Length rules to + the resulting snapshot. Buffering can infer length at completion; streaming + cannot infer an unpublished final body length after handoff. Generated fields + must fit the same resource budget and the applicable encoded-output limit. +8. Encode using the transport-specific rules below. Pre-handoff failures remain + invocation errors; no partial headers are published. Unsupported trailers + attempted after commitment remain faults under the accepted late-error policy. + +The fixed removal set follows the categories also handled by Go's ReverseProxy: +Connection, Proxy-Connection, Keep-Alive, Proxy-Authenticate, +Proxy-Authorization, TE, Trailer, Transfer-Encoding and Upgrade. Upgrade/trailer +attempts first produce the accepted unsupported-feature fault; removal must not +hide that fault. This is application-response handling, separate from the Lambda +SDK's own transport headers and error trailers. +[Go ReverseProxy source](https://go.dev/src/net/http/httputil/reverseproxy.go), +[Connection semantics](https://www.rfc-editor.org/rfc/rfc9110.html#section-7.6.1). + +Keep the shared layer concerned with syntax, ownership, transport controls and +representation. Beyond the already accepted Content-Length rules, do not expand +this milestone into a universal semantic validator for every HTTP header. +Applications remain responsible for field-specific validity in every format. + +## Format-specific rules + +| Output | Ordinary fields | Set-Cookie | +| --- | --- | --- | +| Buffered V1, including HTTP API payload 1.0 | MultiValueHeaders; no redundant Headers map | Separate elements in MultiValueHeaders | +| Buffered V2 | One value unchanged; multiple values combined only for the audited set | Move values into Cookies; never comma-combine | +| REST streaming | MultiValueHeaders in JSON v2 metadata; no redundant Headers map | Separate elements in MultiValueHeaders; omit redundant Cookies | + +Preserve the accepted V1 treatment of unknown repeated fields. V2 rejects a +repeated field outside the audited set, even if its values happen to be equal. +Do not silently use the first/last value or deduplicate to avoid an error. +Examples of V2 rejections include Content-Type, Location, ETag, Date and +Access-Control-Allow-Origin. Content-Length retains its stricter shared rules. + +For a known combinable field, concatenate complete values with comma-space. +This is a representation conversion, not validation that each value is legal for +that field. In particular, registration does not repair malformed list syntax, +empty list elements, invalid dictionary members, or invalid singleton-alternative +combinations. Original within-slice order is preserved, including for ordered +fallbacks, challenges and signature dictionaries. + +### Proposed initial V2 combination table + +| Fields | Definition supporting combination | +| --- | --- | +| Accept-Ranges, Allow, Content-Encoding, Content-Language, Vary, WWW-Authenticate | [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html) list grammars | +| Cache-Control | [RFC 9111 section 5.2](https://www.rfc-editor.org/rfc/rfc9111.html#section-5.2) | +| Access-Control-Allow-Headers, Access-Control-Allow-Methods, Access-Control-Expose-Headers | [Fetch CORS grammars](https://fetch.spec.whatwg.org/#http-new-header-syntax) | +| Link | [RFC 8288 section 3](https://www.rfc-editor.org/rfc/rfc8288.html#section-3) | +| Content-Security-Policy, Content-Security-Policy-Report-Only | [CSP sections 3.1–3.2](https://www.w3.org/TR/CSP3/#csp-header) | +| Referrer-Policy | [Referrer Policy section 4.1](https://www.w3.org/TR/referrer-policy/#referrer-policy-header) | +| Server-Timing | [Server Timing section 3](https://www.w3.org/TR/server-timing/#the-server-timing-header-field) | +| Accept-Patch | [RFC 5789 section 3.1](https://www.rfc-editor.org/rfc/rfc5789.html#section-3.1) | +| Accept-CH | [RFC 8942 section 3.1](https://www.rfc-editor.org/rfc/rfc8942.html#section-3.1) structured List | +| Cache-Status, Proxy-Status | [RFC 9211](https://www.rfc-editor.org/rfc/rfc9211.html#section-2), [RFC 9209](https://www.rfc-editor.org/rfc/rfc9209.html#section-2) structured Lists | +| Content-Digest, Repr-Digest, Want-Content-Digest, Want-Repr-Digest | [RFC 9530 sections 2–4](https://www.rfc-editor.org/rfc/rfc9530.html#section-2) structured Dictionaries | +| Signature, Signature-Input | [RFC 9421 sections 4.1–4.2](https://www.rfc-editor.org/rfc/rfc9421.html#section-4.1) structured Dictionaries | + +This is an audited starting set of 25 fields, not an exhaustive HTTP registry or +an AWS allowlist. Signature-field transport does not imply signature verification +or a guarantee that Gateway transformations preserve signed message components. + +Do not add a public joining callback or mutable global registry initially. +Applications with another list-defined extension can explicitly construct one +valid combined value using ordinary Header.Set, or use V1 when separate values +are required. Add fields to the built-in set with a specification reference and +fixtures when needed. Revisit a per-adapter extension option only if middleware +integration demonstrates that the ordinary HTTP boundary is insufficient. + +## Resource accounting and encoded limits + +Recommend a single preflight charge based on the HTTP/2 field-section accounting +model: for each original value, charge len(name) + len(value) + 32 bytes. +Charge a nil/empty slice as one name-plus-32 entry. Count original bytes before +trimming, merging or dropping fields, including cookies and hop headers. +Use overflow-safe incremental subtraction from the remaining budget. +[RFC 9113 section 6.5.2](https://www.rfc-editor.org/rfc/rfc9113.html#section-6.5.2). + +The 32-byte per-line term is established HTTP accounting reused as an edge +resource model. It is not an AWS quota or an exact measurement of Go heap usage. +Counting suppressed entries is an additional edge policy that bounds copying +and sorting even when map values are empty. It also avoids an independent, +arbitrary value-count limit. + +Recommend starting with the existing 6,291,456-byte response ceiling as this +resource budget, on both buffered and streaming paths, without a new public +tuning option in the first implementation. This is intentionally a broad +compatibility ceiling rather than an attempt to duplicate smaller deployment +quotas. It limits copied logical bytes and, because every valid key costs at +least 33 bytes per entry, permits at most 190,650 charged entries. Counting only +name/value bytes would admit arbitrarily large slices of empty strings; this +cannot. The previous paired proposal also bounded count, using a separate knob. + +This is a proposed library policy. A response below an AWS envelope limit can +still exceed the weighted budget. A passing weighted budget does not promise +that a response fits Lambda JSON, API Gateway, a browser, or a low-memory Lambda +configuration. Map overhead, backing strings, encoding temporaries and the +application's existing allocations are not an exact 6 MiB heap guarantee. + +Before finalizing the implementation, measure ordinary headers/cookies, many +empty values, many distinct names, mixed-case collisions and escaped values. +Record cumulative allocations and retained snapshot size separately. If the +high-cardinality cases make this broad ceiling impractical, bring measured +evidence and a lower/configurable policy back for review; do not silently +substitute another small fixed limit. + +Independently enforce the existing complete buffered JSON envelope limit on raw +Invoke. Typed buffered methods still perform no response JSON round trip and +leave the caller's final serialized-envelope limit with that caller/runtime. + +For streaming, recommend a conservative initial limit of **16,000 bytes for the +entire encoded JSON prefix plus all eight delimiter bytes**, checked by bounded +direct JSON v2 encoding before publish. That leaves at most 15,992 JSON bytes, +including status, names, array syntax and escaping. The guide does not explicitly +resolve decimal/binary KB or delimiter-start versus delimiter-end accounting. +Keeping the complete delimiter within 16,000 bytes satisfies the stricter +interpretations; it deliberately gives up up to 384 bytes compared with a +16,384-byte interpretation. Label this as edge's conservative compatibility +policy and relax it only on authoritative clarification or approved deployment +evidence. This resolves the implementation choice without claiming AWS specified +an exact 16,000-byte service boundary. + +Neither the weighted budget nor the exact streaming-prefix cap is a stream body +limit. The streaming writer retains only its accepted sniffing prefix and uses +the already implemented bridge for the body. + +## Alternatives and next implementation + +- Join every repeated V2 field: simplest mechanically, but can alter singleton, + cookie and extension semantics. AWS's representation does not authorize that. +- Preserve every field in every format: impossible with V2's single-string map. +- Fixed 64 KiB plus 1,024 entries: still lacks a demonstrated resource/compatibility + reason for those numbers. A weighted budget replaces both controls. +- Reuse Gateway's smallest quota globally: would conflate request/response, + endpoint configuration, HTTP field bytes and JSON framing bytes. +- Full semantic registry/parser: significantly larger scope than transport + adaptation; unnecessary for ordinary single values and V1 preservation. + +After approval, implement snapshot/preflight and the V1/V2 projections first. +Test mutation after commitment, case collisions, nil/empty distinctions, control +bytes/UTF-8, Connection nominations, cookies, quoted commas, list/dictionary +order, rejected repeated fields and weighted boundaries. Run allocation probes +before settling the resource implementation. Then add bounded stream-prefix +encoding and connect the shared snapshot to the separate HTTP writers. + +Public invocation/identity work and the SDK Runtime API header compatibility +question remain separate. No account APIs or live AWS deployments were used for +this proposal. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index f6e64f7..2d4adaf 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -18,3 +18,4 @@ Keep related decisions small enough to review and implement independently. | [0007: Response metadata](0007-response-metadata.md) | Qualified direction accepted; details open | What metadata budgets and V2 joinable-field set should apply? | | [0008: Streaming architecture](0008-streaming-architecture.md) | Accepted | How should REST response streaming fit alongside the buffered adapter? | | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | +| [0010: Response header design](0010-response-header-design.md) | Proposed | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | diff --git a/docs/plan.md b/docs/plan.md index 0fc0880..88614e1 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -106,7 +106,10 @@ policies, exported error categories, and identity APIs remain separate reviews. The user accepted the qualification to decision 0007: do not freeze arbitrary 64 KiB/1,024-entry metadata limits or the initial ten-field list without stronger -resource and compatibility justification. Streaming is now being discussed in +resource and compatibility justification. [Decision 0010](decisions/0010-response-header-design.md) +now proposes shared snapshots, a 25-field V2 combination audit, weighted resource +accounting, and a conservative exact streaming-prefix limit. These details await +user review; no header-policy implementation has been added. Streaming is specified in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. The high-level architecture is accepted. [Decision 0009](decisions/0009-streaming-boundaries.md) From 722bef56f878ac67df742180da3ec2f2f5c136d8 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 18:01:22 +0100 Subject: [PATCH 13/44] Clarify payload terminology and propose consumer header processing --- docs/decisions/0010-response-header-design.md | 6 + .../0011-request-header-processing.md | 117 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 5 + 4 files changed, 129 insertions(+) create mode 100644 docs/decisions/0011-request-header-processing.md diff --git a/docs/decisions/0010-response-header-design.md b/docs/decisions/0010-response-header-design.md index e67c063..04edff1 100644 --- a/docs/decisions/0010-response-header-design.md +++ b/docs/decisions/0010-response-header-design.md @@ -2,6 +2,12 @@ Status: proposed for user review on 2026-09-14. No dependent implementation yet. +Terminology: V1/V2 in this record mean API Gateway payload formats 1.0/2.0, +not versions of this Go module. The field-combination table concerns outgoing +responses only. It does not restrict custom incoming request headers. +The user's action-dispatch requirement is recorded separately in +[decision 0011](0011-request-header-processing.md). + ## Recommendation Keep the application boundary as http.Header. Capture one private, owned, diff --git a/docs/decisions/0011-request-header-processing.md b/docs/decisions/0011-request-header-processing.md new file mode 100644 index 0000000..1f76571 --- /dev/null +++ b/docs/decisions/0011-request-header-processing.md @@ -0,0 +1,117 @@ +# 0011: Consumer request-header processing and action dispatch + +Status: first-class support required by the user; concrete API proposed for review. + +## Requirement and terminology + +Consumers will implement APIs in which a request header identifies the action +to dispatch. They need a supported interface for processing custom headers. +This is incoming request processing, separate from decision 0010's outgoing +response-header conversion. API Gateway payload format 2.0 is not module v2. + +The existing request conversion preserves custom header names and values. Its +only special handling is the accepted transport normalization, such as moving +Host into Request.Host and reconciling body framing. The response-combination +table must never become an incoming-header allowlist. + +## Evidence + +request.go canonicalizes names and preserves repeated V1 values. API Gateway +payload 2.0 provides flattened header strings: AWS combines repeated values +with commas. Original field-line multiplicity is unavailable in that format. +Reading Header.Values is necessary for V1/ordinary HTTP multiplicity, but not +sufficient to establish that a payload 2.0 value originated in one field line. +[AWS payload differences](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html), +[Go Header.Values](https://pkg.go.dev/net/http#Header.Values). + +Header.Get returns the first associated value. Using it alone to choose a +security-relevant action could hide conflicting values. The repository has no +consumer dispatcher implementation or action-header contract to reuse; a +focused search of Beakley also found no such contract. +[Go Header.Get](https://pkg.go.dev/net/http#Header.Get). + +## Recommended extension boundary + +Expose a small consumer-implemented interface at the ordinary HTTP middleware +boundary, with a function adapter for consumers who prefer functions: + +```go +// Proposed signature, not implemented. +type RequestMiddleware interface { + Wrap(next http.Handler) http.Handler +} +``` + +Header processing can inspect the complete normalized *http.Request, attach +consumer-owned typed context metadata via WithContext, reject a request with an +ordinary HTTP response, or call the next handler. This avoids inventing a second +error-to-HTTP protocol or putting application parsing errors into the Lambda +invocation-failure channel. It also supports headers beyond the action selector. + +Recommend ordinary explicit composition into the handler supplied to New or +NewStreaming, usable identically with an ordinary Go HTTP server. A processor +must preserve the request's context ancestry, honor cancellation, and obey the +usual ResponseWriter lifetime. Middleware construction occurs outside request +execution; the wrapped handler must be safe for the application's concurrency. +Do not add a raw AWS-event hook for this feature. + +This interface is intentionally broader than a callback returning a header map: +an action dispatcher needs validation, request metadata and controlled rejection, +not just header rewriting. Keep the business action registry, payload schemas, +handler resolution and action-specific authorization in the consuming dispatcher. +First-class support means a documented extension contract, strict selection +helpers, consumer examples and a cross-format test matrix; it does not require +edge to own the application's action registry. + +## Action-selection helper and pipeline + +Recommend providing a shared strict single-value header selector for consumers +to use inside their middleware/dispatcher, with inspectable missing/invalid/ +ambiguous outcomes. Its exact exported signature and error types need review +alongside the shared error API. It must not require AWS dependencies or JSON. + +Pending consumer details, recommend: + +- Configure the action header name explicitly; do not hard-code an X-Action + convention. Validate its name at construction. +- Require one nonempty normalized action value. Preserve identifier case. +- Check every represented value case-insensitively by header name. Reject + multiple values, including identical repeats that are still visible. +- For an action grammar that excludes commas, reject any comma-bearing value. + This rejects both joined duplicates in payload 2.0 and literal commas; it does + not recover which one occurred. Do not split a joined value and choose one. +- Do not promise detection of duplicate field lines already deduplicated or lost + by Gateway or an earlier proxy. Validate the available representation. +- Parse the action once and use the same selected value for authorization, + dispatch and consumer logging. Do not read a mutable header again later to + choose a different action from the one authorized. +- Do not infer authentication or authorization from possession of the action + header. Action-specific authorization must precede action execution and any + streaming response publication. + +The consumer pipeline is normalized HTTP request and optional gateway identity, +then consumer header processing/local authentication, action selection, +action-specific authorization and execution. The application can place general +authentication before selection to suit its disclosure policy; the invariant is +that action authorization and execution use the same validated selection. + +Missing/malformed action selections should ordinarily produce an application +400 response, not a Lambda invocation error. The consumer owns response bodies +and unknown-action policy. Authentication/authorization response policy remains +the separate authn/authz contract. No fallback/default action is inferred. + +## Decisions still to review + +- Middleware interface/function-adapter names and whether a composition helper + materially improves ordinary Wrap calls. +- Consumer action-header name and identifier grammar. A clarification was sent; + no concrete consumer format is assumed accepted. +- Strict selection helper signature and inspectable error categories. +- Whether action metadata is entirely consumer-owned or a demonstrated shared + authz/dispatcher need warrants a small transport-independent action type. + +Validate custom headers, missing/empty values, casing, repeated/conflicting and +identical values, flattened comma values, native HTTP behavior, both raw/typed +Gateway payloads, middleware ordering, context propagation, rejected-request +short circuiting and authorization-before-streaming. No production API changes +are made by this record. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 2d4adaf..0cf16a6 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -19,3 +19,4 @@ Keep related decisions small enough to review and implement independently. | [0008: Streaming architecture](0008-streaming-architecture.md) | Accepted | How should REST response streaming fit alongside the buffered adapter? | | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | | [0010: Response header design](0010-response-header-design.md) | Proposed | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | +| [0011: Request header processing](0011-request-header-processing.md) | Requirement accepted; API proposed | How can consumers process custom headers and safely select actions for their own dispatchers? | diff --git a/docs/plan.md b/docs/plan.md index 88614e1..2b93045 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -20,6 +20,11 @@ port, source copy, or compatibility-preserving update of Beakley. handling. Do not route our codec through `encoding/json`'s compatibility API. Check third-party parser behavior separately during dependency selection. - Keep ordinary `http.Handler` as the application boundary. +- Provide first-class consumer request-header processing for header-selected + action dispatchers. Preserve custom incoming headers independently of response + combination rules. [Decision 0011](decisions/0011-request-header-processing.md) + proposes the extension interface and strict selection support; API details + and the consumer action-header contract remain under review. - Own event serialization at a raw Lambda invocation boundary implementing `Invoke(context.Context, []byte) ([]byte, error)`. - Reuse AWS Lambda Go event/response structs where they meet the contract; From f0965f7ef4fae67311ae17efa52d57ad61aabde2 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 14 Sep 2026 22:30:18 +0100 Subject: [PATCH 14/44] Specify recommended action header selector and middleware contracts --- .../0011-request-header-processing.md | 109 ++++++++++++++++-- 1 file changed, 100 insertions(+), 9 deletions(-) diff --git a/docs/decisions/0011-request-header-processing.md b/docs/decisions/0011-request-header-processing.md index 1f76571..0d05d88 100644 --- a/docs/decisions/0011-request-header-processing.md +++ b/docs/decisions/0011-request-header-processing.md @@ -100,15 +100,106 @@ Missing/malformed action selections should ordinarily produce an application and unknown-action policy. Authentication/authorization response policy remains the separate authn/authz contract. No fallback/default action is inferred. -## Decisions still to review - -- Middleware interface/function-adapter names and whether a composition helper - materially improves ordinary Wrap calls. -- Consumer action-header name and identifier grammar. A clarification was sent; - no concrete consumer format is assumed accepted. -- Strict selection helper signature and inspectable error categories. -- Whether action metadata is entirely consumer-owned or a demonstrated shared - authz/dispatcher need warrants a small transport-independent action type. +## Concrete recommendations for the open decisions + +The user asked for recommendations for these remaining choices. The following +replaces the open-ended alternatives with a proposed initial contract; approval +is still pending. + +### Interface and composition + +Keep RequestMiddleware.Wrap and add the function adapter: + +```go +type RequestMiddlewareFunc func(http.Handler) http.Handler +func (f RequestMiddlewareFunc) Wrap(next http.Handler) http.Handler +``` + +Use ordinary explicit composition: outer.Wrap(inner.Wrap(dispatcher)). The +outer processor runs first on the request. Do not add WithRequestMiddleware, +a Chain helper, implicit reordering, or another middleware package initially. +The same composed handler goes to edge.New, edge.NewStreaming, or net/http. +This keeps execution order visible and does not tie header processing to an AWS +constructor. Nil function adapters/next handlers are invalid programmer inputs, +like using an unusable http.HandlerFunc; the adapter is not a fallible factory. +Consumer middleware factories can return their own configuration errors. + +### Action header and identifier grammar + +Require an explicit header name at selector construction. Use Action with a +value such as orders.create in examples, but do not make it a default or reserve +the name globally. Existing consumer names remain supported. Avoid introducing +an X- prefix simply to designate an application-defined field. +[RFC 6648](https://www.rfc-editor.org/rfc/rfc6648.html#section-3). + +For the built-in selector, trim outer SP/HTAB and require a nonempty HTTP token, +preserving case. The existing validToken helper already implements this grammar. +This admits common identifiers such as orders.create, CreateOrder and +orders-create, while excluding commas, embedded whitespace, quoting, colon, +slash and non-ASCII text. HTTP does not mandate this grammar for action headers; +it is our recommended selector contract. A consumer with a different protocol +can implement its own parser behind RequestMiddleware without changing gateway +translation or weakening the standard selector. +[HTTP token grammar](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.6.2). + +### Selector API and errors + +Recommend a reusable immutable selector in the root edge package: + +```go +func NewActionHeader(name string) (*ActionHeader, error) +func (h *ActionHeader) Parse(headers http.Header) (string, error) +``` + +Construction validates/canonicalizes the configured field name. The zero value +and nil receiver are unusable and Parse returns a configuration error for them. +Parse neither modifies headers nor writes a response. It needs no context, +AWS event types, or JSON processing. An instance can be reused concurrently; +callers must not concurrently mutate a Header map being inspected. + +Match header names using ASCII case-insensitive comparison, even when a consumer supplies a Header map +with noncanonical direct assignments. Count represented values across all +matching aliases; do not rely solely on Header.Get or Header.Values. Count a +nil/empty value slice as zero values. Multiple represented values are ambiguous +even when identical, and take precedence over errors in an individual value. + +Export three action-selection sentinels, inspected with errors.Is: + +| Error | Meaning | +| --- | --- | +| ErrActionMissing | No represented value, including an absent field or nil/empty slices | +| ErrActionAmbiguous | More than one represented value, or one containing a comma | +| ErrActionInvalid | Exactly one value that is empty after SP/HTAB trimming or fails token grammar | + +For one comma-bearing value, ambiguity classification precedes token validation; +it deliberately covers both Gateway-combined values and a literal comma, whose +origins cannot be distinguished. Return an empty action on every error. Do not +include the supplied action or other header values in error messages. Invalid +constructor configuration is separate from these request-input categories. + +Keep these application-input errors separate from transport invocation faults. +The parsing helper does not encode HTTP statuses. Recommend 400 for selection +errors and, for the header-dispatched API convention, 400 with a distinct +consumer-owned unknown-action code for a syntactically valid but unregistered +action. The consumer retains its response schema and disclosure policy. General +authentication and action authorization retain their separate authn/authz rules. + +### Action metadata ownership + +Return a string and keep action metadata consumer-owned. Do not add an edge.Action +type, global action context key, or action package at this stage. The dispatcher +can convert the validated string into its own action type, pass it directly to +its authorizer/handler, or attach it through a private typed context key. + +Resolve once, authorize that same selection, and execute it. If selecting a +registered operation produces an immutable operation descriptor, pass that +descriptor through authorization and execution instead of resolving again from +mutable headers. This supports first-class selection without coupling the +transport module to a consumer's action registry or permissions model. + +These recommendations settle what to implement next for this feature if +approved. They do not imply approval of decision 0010's separate response-header +budgets/combination policy or settle the broader identity/error APIs. Validate custom headers, missing/empty values, casing, repeated/conflicting and identical values, flattened comma values, native HTTP behavior, both raw/typed From 3d83d44c4c075f3e693b6317992393989ddcf53f Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 11:16:52 +0100 Subject: [PATCH 15/44] docs: refine header API proposal after Go 1.27 review --- .../0011-request-header-processing.md | 83 +++++++++++++------ docs/plan.md | 4 +- 2 files changed, 61 insertions(+), 26 deletions(-) diff --git a/docs/decisions/0011-request-header-processing.md b/docs/decisions/0011-request-header-processing.md index 0d05d88..3dc7402 100644 --- a/docs/decisions/0011-request-header-processing.md +++ b/docs/decisions/0011-request-header-processing.md @@ -32,14 +32,13 @@ focused search of Beakley also found no such contract. ## Recommended extension boundary -Expose a small consumer-implemented interface at the ordinary HTTP middleware -boundary, with a function adapter for consumers who prefer functions: +Use the standard http.Handler interface as the supported consumer boundary. +Consumers implement handlers or compose middleware with the ordinary function +shape: ```go -// Proposed signature, not implemented. -type RequestMiddleware interface { - Wrap(next http.Handler) http.Handler -} +// Consumer-defined middleware; not an additional exported edge type. +func processHeaders(next http.Handler) http.Handler ``` Header processing can inspect the complete normalized *http.Request, attach @@ -55,7 +54,7 @@ usual ResponseWriter lifetime. Middleware construction occurs outside request execution; the wrapped handler must be safe for the application's concurrency. Do not add a raw AWS-event hook for this feature. -This interface is intentionally broader than a callback returning a header map: +The HTTP handler contract is broader than a callback returning a header map: an action dispatcher needs validation, request metadata and controlled rejection, not just header rewriting. Keep the business action registry, payload schemas, handler resolution and action-specific authorization in the consuming dispatcher. @@ -67,8 +66,8 @@ edge to own the application's action registry. Recommend providing a shared strict single-value header selector for consumers to use inside their middleware/dispatcher, with inspectable missing/invalid/ -ambiguous outcomes. Its exact exported signature and error types need review -alongside the shared error API. It must not require AWS dependencies or JSON. +ambiguous outcomes. The proposed signatures and errors below remain subject to +review alongside the shared error API. Parsing needs no AWS event types or JSON. Pending consumer details, recommend: @@ -108,21 +107,19 @@ is still pending. ### Interface and composition -Keep RequestMiddleware.Wrap and add the function adapter: +Use http.Handler directly, with http.HandlerFunc when adapting a handler +function. Do not export RequestMiddleware or RequestMiddlewareFunc: no edge API +consumes their proposed Wrap method, and the standard interface already supplies +the required extension point. Consumers may define their own processor types +without adopting an additional edge interface. -```go -type RequestMiddlewareFunc func(http.Handler) http.Handler -func (f RequestMiddlewareFunc) Wrap(next http.Handler) http.Handler -``` - -Use ordinary explicit composition: outer.Wrap(inner.Wrap(dispatcher)). The +Use ordinary explicit composition: outer(inner(dispatcher)). The outer processor runs first on the request. Do not add WithRequestMiddleware, a Chain helper, implicit reordering, or another middleware package initially. The same composed handler goes to edge.New, edge.NewStreaming, or net/http. This keeps execution order visible and does not tie header processing to an AWS -constructor. Nil function adapters/next handlers are invalid programmer inputs, -like using an unusable http.HandlerFunc; the adapter is not a fallible factory. -Consumer middleware factories can return their own configuration errors. +constructor. Consumer middleware factories can return their own configuration +errors before composition and follow the ordinary HTTP handler contract. ### Action header and identifier grammar @@ -138,7 +135,7 @@ This admits common identifiers such as orders.create, CreateOrder and orders-create, while excluding commas, embedded whitespace, quoting, colon, slash and non-ASCII text. HTTP does not mandate this grammar for action headers; it is our recommended selector contract. A consumer with a different protocol -can implement its own parser behind RequestMiddleware without changing gateway +can implement its own parser in HTTP middleware without changing gateway translation or weakening the standard selector. [HTTP token grammar](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.6.2). @@ -147,12 +144,14 @@ translation or weakening the standard selector. Recommend a reusable immutable selector in the root edge package: ```go -func NewActionHeader(name string) (*ActionHeader, error) -func (h *ActionHeader) Parse(headers http.Header) (string, error) +func NewActionHeader(name string) (ActionHeader, error) +func (h ActionHeader) Parse(headers http.Header) (string, error) ``` -Construction validates/canonicalizes the configured field name. The zero value -and nil receiver are unusable and Parse returns a configuration error for them. +Construction validates/canonicalizes the configured field name, stored privately. +The selector is a small immutable value: copying it is safe and it needs neither +shared mutable state nor pointer identity. Its zero value has no configured name, +so Parse returns a configuration error rather than inferring a default header. Parse neither modifies headers nor writes a response. It needs no context, AWS event types, or JSON processing. An instance can be reused concurrently; callers must not concurrently mutate a Header map being inspected. @@ -197,6 +196,42 @@ descriptor through authorization and execution instead of resolving again from mutable headers. This supports first-class selection without coupling the transport module to a consumer's action registry or permissions model. +## Go rules and Go 1.27 review — 2026-09-15 + +Reviewed the Go rules skill's core, types, naming, functions, errors, commentary +and testing guidance, the official Go 1.27 release notes, and the current +adapter/request code. This review replaces the earlier proposed Wrap interface +and pointer-returning selector with the recommendations above. The changes follow +the rules on least mechanism, interfaces at their consumption boundary, and +value semantics for small immutable types. They are API design judgments, not +new Go 1.27 requirements. The user-requested consumer interface remains +http.Handler; first-class header support includes the selector, documentation, +examples and contract tests. + +Keep the constructor because explicit startup validation is useful. A standalone +ParseActionHeader(headers, name) helper would repeat configuration validation on +every request. A useful implicit zero value would require choosing a default +header name, which conflicts with the explicit configuration recommendation. +Keep three sentinel errors with errors.Is; they classify failures without +inventing a structured error payload. errors.AsType is useful when callers need +a typed error's data, but is not a replacement for sentinel matching here. + +Go 1.27 supports generic methods, but this selector has no requirement for a +generic method or generic action type. Consumers can convert the returned string +to their own action type. Direct encoding/json/v2 remains the accepted JSON +boundary; the selector and HTTP middleware need no JSON processing. +[Go 1.27 release notes](https://go.dev/doc/go1.27). + +For subsequent contract tests, consider httptest.NewTestServer for ordinary HTTP +integration using its in-memory network, and synctest.Sleep when a streaming +test needs both virtual time advancement and quiescence. Existing tests need no +mechanical rewrite. Go 1.27's Server.MaxHeaderValueCount is an HTTP server request +limit; it neither validates action ambiguity nor bounds Lambda response headers. +It does not replace this selector or settle decision 0010's budgets. +[httptest.NewTestServer](https://pkg.go.dev/net/http/httptest#NewTestServer), +[synctest.Sleep](https://pkg.go.dev/testing/synctest#Sleep), +[http.Server](https://pkg.go.dev/net/http#Server). + These recommendations settle what to implement next for this feature if approved. They do not imply approval of decision 0010's separate response-header budgets/combination policy or settle the broader identity/error APIs. diff --git a/docs/plan.md b/docs/plan.md index 2b93045..964527a 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -23,8 +23,8 @@ port, source copy, or compatibility-preserving update of Beakley. - Provide first-class consumer request-header processing for header-selected action dispatchers. Preserve custom incoming headers independently of response combination rules. [Decision 0011](decisions/0011-request-header-processing.md) - proposes the extension interface and strict selection support; API details - and the consumer action-header contract remain under review. + proposes the standard HTTP extension boundary and strict selection support; + API details and the consumer action-header contract remain under review. - Own event serialization at a raw Lambda invocation boundary implementing `Invoke(context.Context, []byte) ([]byte, error)`. - Reuse AWS Lambda Go event/response structs where they meet the contract; From 78eddbb246b705b00089df1b6ab49333d415454a Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 11:34:27 +0100 Subject: [PATCH 16/44] feat: add strict action header selection for HTTP consumers --- README.md | 20 +++ action.go | 83 +++++++++++ action_example_test.go | 129 +++++++++++++++++ action_gateway_test.go | 136 ++++++++++++++++++ action_test.go | 112 +++++++++++++++ .../0011-request-header-processing.md | 52 +++++-- docs/decisions/README.md | 2 +- docs/plan.md | 7 +- 8 files changed, 523 insertions(+), 18 deletions(-) create mode 100644 action.go create mode 100644 action_example_test.go create mode 100644 action_gateway_test.go create mode 100644 action_test.go diff --git a/README.md b/README.md index 35a0bf8..dfc6d54 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,26 @@ is covered by race-enabled synthetic concurrency tests. HTTP streaming framing and public streaming entry points remain under construction; deployed API Gateway streaming has not been verified. +## Header-selected actions + +`edge.NewActionHeader("Action")` creates an immutable selector whose +`Parse(r.Header)` method returns an action string or an error. Configure the +header name explicitly. Selection preserves case, trims outer spaces/tabs, and +requires a nonempty HTTP token. Repeated values and comma-bearing values are +rejected, including Gateway payload 2.0's comma-combined duplicates. Use +`errors.Is` with `edge.ErrActionMissing`, `edge.ErrActionAmbiguous`, or +`edge.ErrActionInvalid` to distinguish request-input failures. + +Use ordinary `http.Handler` middleware to process this and other custom headers. +The consumer owns its action registry, metadata, authorization and HTTP error +responses. Select once, authorize that selection, then execute it; do not select +again from mutable headers. An action header does not authenticate a caller. +The selector is usable now with ordinary Go HTTP servers, independently of the +unfinished Lambda invocation methods. + +See the [runnable middleware example](action_example_test.go) and +[accepted header contract](docs/decisions/0011-request-header-processing.md). + - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) - [Repository working agreement](AGENTS.md) diff --git a/action.go b/action.go new file mode 100644 index 0000000..d501bdc --- /dev/null +++ b/action.go @@ -0,0 +1,83 @@ +package edge + +import ( + "errors" + "net/http" + "strings" +) + +var ( + // ErrActionMissing means the configured header has no represented values. + // An absent header and a nil or empty value slice are all missing. + ErrActionMissing = errors.New("edge: missing action") + + // ErrActionAmbiguous means the header has multiple represented values or a + // value containing a comma. Gateway payload 2.0 combines duplicates with + // commas; a literal comma cannot be distinguished from combined values. + ErrActionAmbiguous = errors.New("edge: ambiguous action") + + // ErrActionInvalid means the single action value is empty after trimming + // outer spaces and tabs, or does not satisfy the HTTP token grammar. + ErrActionInvalid = errors.New("edge: invalid action") +) + +// ActionHeader selects an action from a configured request header. +// Construct it with [NewActionHeader]; its zero value is unconfigured. +// An ActionHeader is immutable, safe to copy, and safe for concurrent use. +type ActionHeader struct { + name string +} + +// NewActionHeader validates name as an HTTP field name and returns a selector. +// The name is required and matched case-insensitively; no default is inferred. +// On error, the returned selector is the zero value. Configuration errors are +// separate from [ErrActionMissing], [ErrActionAmbiguous] and [ErrActionInvalid]. +func NewActionHeader(name string) (ActionHeader, error) { + if !validToken(name) { + return ActionHeader{}, errors.New("edge: invalid action header name") + } + return ActionHeader{name: http.CanonicalHeaderKey(name)}, nil +} + +// Parse returns the single action in headers, trimming only outer SP/HTAB and +// preserving case. An action must be a nonempty HTTP token (RFC 9110 §5.6.2). +// Parse inspects all case-insensitive aliases, including noncanonical map keys. +// Multiple values are ambiguous even if identical; a comma is also ambiguous. +// Multiplicity and comma checks take precedence over token validation. +// Duplicates already lost by a gateway or earlier proxy cannot be detected. +// +// On failure Parse returns an empty string and [ErrActionMissing], +// [ErrActionAmbiguous] or [ErrActionInvalid], matchable with [errors.Is]. An +// unconfigured selector instead returns a configuration error. Errors do not +// include supplied header values. Parse does not modify headers or write an +// HTTP response. Callers must not mutate headers concurrently with Parse. +// Select once and use that same action for authorization and execution. +func (h ActionHeader) Parse(headers http.Header) (string, error) { + if h.name == "" { + return "", errors.New("edge: unconfigured action header") + } + var value string + found := false + for name, values := range headers { + // CanonicalHeaderKey folds only valid ASCII field names. In particular, + // Unicode case equivalents must not alias the configured HTTP name. + if http.CanonicalHeaderKey(name) != h.name || len(values) == 0 { + continue + } + if found || len(values) > 1 { + return "", ErrActionAmbiguous + } + value, found = values[0], true + } + if !found { + return "", ErrActionMissing + } + if strings.Contains(value, ",") { + return "", ErrActionAmbiguous + } + value = strings.Trim(value, " \t") + if !validToken(value) { + return "", ErrActionInvalid + } + return value, nil +} diff --git a/action_example_test.go b/action_example_test.go new file mode 100644 index 0000000..4b92679 --- /dev/null +++ b/action_example_test.go @@ -0,0 +1,129 @@ +package edge_test + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "slices" + "testing" + + "github.com/asteroid-computing/go-lambda-edge" +) + +// These are consumer-owned middleware and metadata, not edge API types. +type actionKey struct{} + +func processActionHeaders(selector edge.ActionHeader, next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, err := selector.Parse(r.Header) + if err != nil { + http.Error(w, "invalid action selection", http.StatusBadRequest) + return + } + ctx := context.WithValue(r.Context(), actionKey{}, action) + next.ServeHTTP(w, r.WithContext(ctx)) + }) +} + +func ExampleActionHeader() { + selector, err := edge.NewActionHeader("Action") + if err != nil { + fmt.Println(err) + return + } + dispatcher := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, ok := r.Context().Value(actionKey{}).(string) + if !ok { + http.Error(w, "missing action metadata", http.StatusInternalServerError) + return + } + // This example operation is public. Protected operations must authorize + // this same selection before executing or publishing streaming output. + switch action { + case "service.ping": + fmt.Fprintln(w, "pong") + default: + http.Error(w, "unknown action", http.StatusBadRequest) + } + }) + // The composed http.Handler also goes directly to edge.New. + handler := processActionHeaders(selector, dispatcher) + r := httptest.NewRequest(http.MethodPost, "/", nil) + r.Header.Set("Action", "service.ping") + w := httptest.NewRecorder() + handler.ServeHTTP(w, r) + fmt.Println(w.Code) + fmt.Print(w.Body.String()) + // Output: + // 200 + // pong +} + +func TestActionHeaderMiddlewareComposition(t *testing.T) { + selector, err := edge.NewActionHeader("Action") + if err != nil { + t.Fatal(err) + } + for _, tt := range []struct { + name string + values []string + status int + steps []string + }{ + {name: "allowed", values: []string{"read"}, status: http.StatusOK, steps: []string{"extra headers", "authorize read", "execute read"}}, + {name: "denied", values: []string{"delete"}, status: http.StatusForbidden, steps: []string{"extra headers", "authorize delete"}}, + {name: "ambiguous", values: []string{"read", "delete"}, status: http.StatusBadRequest, steps: []string{"extra headers"}}, + {name: "missing", status: http.StatusBadRequest, steps: []string{"extra headers"}}, + } { + t.Run(tt.name, func(t *testing.T) { + var steps []string + type traceKey struct{} + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + r := httptest.NewRequestWithContext(ctx, http.MethodPost, "/", nil) + r.Header["Action"] = tt.values + r.Header.Set("Custom-Trace", "trace-123") + dispatcher := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, ok := r.Context().Value(actionKey{}).(string) + if !ok { + t.Error("dispatcher did not receive selected action") + return + } + if r.Context().Value(traceKey{}) != "trace-123" { + t.Error("action middleware lost earlier custom-header metadata") + } + steps = append(steps, "authorize "+action) + if action != "read" { + http.Error(w, "denied", http.StatusForbidden) + return + } + // Later header changes cannot change the authorized selection. + r.Header.Set("Action", "delete") + cancel() + if r.Context().Err() != context.Canceled { + t.Error("middleware lost parent cancellation") + } + steps = append(steps, "execute "+action) + w.WriteHeader(http.StatusOK) + }) + next := processActionHeaders(selector, dispatcher) + handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + steps = append(steps, "extra headers") + ctx := context.WithValue(r.Context(), traceKey{}, r.Header.Get("Custom-Trace")) + next.ServeHTTP(w, r.WithContext(ctx)) + }) + if _, err := edge.New(handler); err != nil { + t.Fatalf("edge.New rejected composed HTTP handler: %v", err) + } + w := httptest.NewRecorder() + handler.ServeHTTP(w, r) + if w.Code != tt.status || !slices.Equal(steps, tt.steps) { + t.Errorf("composition = status %d, steps %v; want %d, %v", w.Code, steps, tt.status, tt.steps) + } + if r.Context().Value(actionKey{}) != nil || r.Context().Value(traceKey{}) != nil { + t.Error("middleware modified the original request context") + } + }) + } +} diff --git a/action_gateway_test.go b/action_gateway_test.go new file mode 100644 index 0000000..a7ac5a0 --- /dev/null +++ b/action_gateway_test.go @@ -0,0 +1,136 @@ +package edge + +import ( + "encoding/json/v2" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestActionHeaderAcrossRequestFormats(t *testing.T) { + selector, err := NewActionHeader("Action") + if err != nil { + t.Fatal(err) + } + server := httptest.NewTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Custom-Trace") != "keep" { + http.Error(w, "lost custom header", http.StatusInternalServerError) + return + } + action, err := selector.Parse(r.Header) + if err != nil { + http.Error(w, "invalid action selection", http.StatusBadRequest) + return + } + io.WriteString(w, action) + })) + for _, tt := range []struct { + name string + values []string + want string + wantErr error + }{ + {name: "single", values: []string{"Orders.Create"}, want: "Orders.Create"}, + {name: "outer_whitespace", values: []string{" \tOrders.Create\t "}, want: "Orders.Create"}, + {name: "missing", wantErr: ErrActionMissing}, + {name: "empty", values: []string{""}, wantErr: ErrActionInvalid}, + {name: "invalid", values: []string{"orders/create"}, wantErr: ErrActionInvalid}, + {name: "duplicates", values: []string{"read", "delete"}, wantErr: ErrActionAmbiguous}, + {name: "identical_duplicates", values: []string{"read", "read"}, wantErr: ErrActionAmbiguous}, + {name: "literal_comma", values: []string{"read,delete"}, wantErr: ErrActionAmbiguous}, + } { + t.Run(tt.name, func(t *testing.T) { + for _, format := range []string{"typed_v1", "raw_rest", "raw_http_v1", "typed_v2", "raw_http_v2", "native_http"} { + t.Run(format, func(t *testing.T) { + if format == "native_http" { + r, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://example.com/", nil) + if err != nil { + t.Fatal(err) + } + r.Header.Set("Custom-Trace", "keep") + for _, value := range tt.values { + r.Header.Add("Action", value) + } + response, err := server.Client().Do(r) + if err != nil { + t.Fatal(err) + } + defer response.Body.Close() + body, err := io.ReadAll(response.Body) + if err != nil { + t.Fatal(err) + } + wantStatus := http.StatusOK + if tt.wantErr != nil { + wantStatus = http.StatusBadRequest + } + if response.StatusCode != wantStatus || tt.wantErr == nil && string(body) != tt.want { + t.Errorf("HTTP response = %d %q; want status %d, action %q", response.StatusCode, body, wantStatus, tt.want) + } + return + } + r := actionGatewayRequest(t, format, tt.values) + defer r.Body.Close() + got, err := selector.Parse(r.Header) + if got != tt.want || !errors.Is(err, tt.wantErr) { + t.Errorf("Parse translated headers = %q, %v; want %q, %v", got, err, tt.want, tt.wantErr) + } + if r.Header.Get("Custom-Trace") != "keep" { + t.Error("request conversion lost unrelated custom header") + } + }) + } + }) + } +} + +func actionGatewayRequest(t *testing.T, format string, values []string) *http.Request { + t.Helper() + v1, v2 := v1RequestEvent(), v2RequestEvent() + v2.Version = "2.0" + v1.Headers = map[string]string{"Custom-Trace": "keep"} + v2.Headers = map[string]string{"custom-trace": "keep"} + if len(values) != 0 { + // A value mirrored by V1's single and multivalue maps is not an extra + // field line. Actual repeats in the multivalue list must survive. + v1.Headers["action"] = values[0] + v1.MultiValueHeaders = map[string][]string{"ACTION": values} + v2.Headers["action"] = strings.Join(values, ",") + } + var r *http.Request + var err error + switch format { + case "typed_v1": + r, err = requestV1(t.Context(), v1) + case "typed_v2": + r, err = requestV2(t.Context(), v2) + default: + var event any = v1 + if format == "raw_http_v2" { + event = v2 + } + payload, marshalErr := json.Marshal(event) + if marshalErr != nil { + t.Fatal(marshalErr) + } + if format == "raw_http_v1" { + payload = append([]byte(`{"version":"1.0",`), payload[1:]...) + } + decoded, decodeErr := decodeEvent(payload) + if decodeErr != nil { + t.Fatal(decodeErr) + } + if decoded.v2 != nil { + r, err = requestV2(t.Context(), *decoded.v2) + } else { + r, err = requestV1(t.Context(), *decoded.v1) + } + } + if err != nil { + t.Fatal(err) + } + return r +} diff --git a/action_test.go b/action_test.go new file mode 100644 index 0000000..f7f3c89 --- /dev/null +++ b/action_test.go @@ -0,0 +1,112 @@ +package edge_test + +import ( + "errors" + "fmt" + "net/http" + "reflect" + "strings" + "testing" + + "github.com/asteroid-computing/go-lambda-edge" +) + +func TestActionHeaderParse(t *testing.T) { + selector, err := edge.NewActionHeader("aCtIoN") + if err != nil { + t.Fatal(err) + } + for _, tt := range []struct { + name string + headers http.Header + want string + wantErr error + }{ + {name: "nil", wantErr: edge.ErrActionMissing}, + {name: "unrelated", headers: http.Header{"Else": {"orders.create"}}, wantErr: edge.ErrActionMissing}, + {name: "nil_slice", headers: http.Header{"Action": nil}, wantErr: edge.ErrActionMissing}, + {name: "empty_slice", headers: http.Header{"Action": {}}, wantErr: edge.ErrActionMissing}, + {name: "empty", headers: http.Header{"Action": {""}}, wantErr: edge.ErrActionInvalid}, + {name: "whitespace", headers: http.Header{"Action": {" \t "}}, wantErr: edge.ErrActionInvalid}, + {name: "trim_and_preserve_case", headers: http.Header{"Action": {" \tOrders.Create\t "}}, want: "Orders.Create"}, + {name: "direct_assignment", headers: http.Header{"aCtIoN": {"orders.create"}}, want: "orders.create"}, + {name: "empty_alias", headers: http.Header{"ACTION": {}, "action": {"orders.create"}}, want: "orders.create"}, + {name: "conflict", headers: http.Header{"Action": {"read", "delete"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "identical", headers: http.Header{"Action": {"read", "read"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "case_aliases", headers: http.Header{"Action": {"read"}, "ACTION": {"delete"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "identical_aliases", headers: http.Header{"Action": {"read"}, "action": {"read"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "invalid_plus_alias", headers: http.Header{"Action": {""}, "action": {"read"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "multiple_empty", headers: http.Header{"Action": {"", ""}}, wantErr: edge.ErrActionAmbiguous}, + {name: "joined", headers: http.Header{"Action": {"read,delete"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "identical_joined", headers: http.Header{"Action": {"read,read"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "comma_before_invalid", headers: http.Header{"Action": {",\r\n"}}, wantErr: edge.ErrActionAmbiguous}, + {name: "space_in_token", headers: http.Header{"Action": {"orders create"}}, wantErr: edge.ErrActionInvalid}, + {name: "slash", headers: http.Header{"Action": {"orders/create"}}, wantErr: edge.ErrActionInvalid}, + {name: "colon", headers: http.Header{"Action": {"orders:create"}}, wantErr: edge.ErrActionInvalid}, + {name: "quoted", headers: http.Header{"Action": {`"read"`}}, wantErr: edge.ErrActionInvalid}, + {name: "unicode_space", headers: http.Header{"Action": {"\u00a0read"}}, wantErr: edge.ErrActionInvalid}, + {name: "unicode", headers: http.Header{"Action": {"créer"}}, wantErr: edge.ErrActionInvalid}, + {name: "invalid_utf8", headers: http.Header{"Action": {"\xff"}}, wantErr: edge.ErrActionInvalid}, + {name: "newline", headers: http.Header{"Action": {"read\n"}}, wantErr: edge.ErrActionInvalid}, + {name: "nul", headers: http.Header{"Action": {"read\x00"}}, wantErr: edge.ErrActionInvalid}, + {name: "del", headers: http.Header{"Action": {"read\x7f"}}, wantErr: edge.ErrActionInvalid}, + {name: "all_token_punctuation", headers: http.Header{"Action": {"!#$%&'*+-.^_`|~AZaz09"}}, want: "!#$%&'*+-.^_`|~AZaz09"}, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + before := tt.headers.Clone() + got, err := selector.Parse(tt.headers) + if got != tt.want || !errors.Is(err, tt.wantErr) { + t.Errorf("Parse(%v) = %q, %v; want %q, %v", tt.headers, got, err, tt.want, tt.wantErr) + } + if !reflect.DeepEqual(tt.headers, before) { + t.Errorf("Parse mutated headers: got %v, want %v", tt.headers, before) + } + if err != nil && !errors.Is(fmt.Errorf("select action: %w", err), tt.wantErr) { + t.Error("wrapped selection error lost its category") + } + }) + } +} + +func TestActionHeaderConfiguration(t *testing.T) { + for _, name := range []string{"", " Action", "Action ", "Action:Name", "Action\r\n", "Àction", "Action\xff"} { + t.Run(fmt.Sprintf("%q", name), func(t *testing.T) { + h, err := edge.NewActionHeader(name) + if err == nil || h != (edge.ActionHeader{}) { + t.Fatalf("NewActionHeader(%q) = %v, %v; want zero selector and error", name, h, err) + } + _, parseErr := h.Parse(http.Header{"Action": {"read"}}) + if parseErr == nil { + t.Fatal("zero selector Parse succeeded") + } + for _, category := range []error{edge.ErrActionMissing, edge.ErrActionAmbiguous, edge.ErrActionInvalid} { + if errors.Is(err, category) || errors.Is(parseErr, category) { + t.Errorf("configuration error matched input category %v", category) + } + } + }) + } + selector, err := edge.NewActionHeader("Task") + if err != nil { + t.Fatal(err) + } + copy := selector + got, err := copy.Parse(http.Header{"taſk": {"wrong"}, "task": {"right"}, "Action": {"ignored"}}) + if got != "right" || err != nil { + t.Errorf("copied custom selector Parse = %q, %v; want right", got, err) + } +} + +func TestActionHeaderErrorsDoNotDiscloseValues(t *testing.T) { + selector, err := edge.NewActionHeader("Action") + if err != nil { + t.Fatal(err) + } + for _, value := range []string{"secret/action", "secret,action"} { + action, err := selector.Parse(http.Header{"Action": {value}, "Authorization": {"secret-credential"}}) + if action != "" || err == nil || strings.Contains(err.Error(), "secret") { + t.Errorf("Parse returned %q, %v; want empty action and sanitized error", action, err) + } + } +} diff --git a/docs/decisions/0011-request-header-processing.md b/docs/decisions/0011-request-header-processing.md index 3dc7402..9ed3383 100644 --- a/docs/decisions/0011-request-header-processing.md +++ b/docs/decisions/0011-request-header-processing.md @@ -1,6 +1,7 @@ # 0011: Consumer request-header processing and action dispatch -Status: first-class support required by the user; concrete API proposed for review. +Status: accepted on 2026-09-15, including the Go review's standard HTTP boundary +and immutable value selector. Selector and consumer middleware example implemented. ## Requirement and terminology @@ -66,10 +67,10 @@ edge to own the application's action registry. Recommend providing a shared strict single-value header selector for consumers to use inside their middleware/dispatcher, with inspectable missing/invalid/ -ambiguous outcomes. The proposed signatures and errors below remain subject to -review alongside the shared error API. Parsing needs no AWS event types or JSON. +ambiguous outcomes. The signatures and errors below are accepted independently +of the broader transport error API. Parsing needs no AWS event types or JSON. -Pending consumer details, recommend: +The accepted selection contract is: - Configure the action header name explicitly; do not hard-code an X-Action convention. Validate its name at construction. @@ -99,11 +100,10 @@ Missing/malformed action selections should ordinarily produce an application and unknown-action policy. Authentication/authorization response policy remains the separate authn/authz contract. No fallback/default action is inferred. -## Concrete recommendations for the open decisions +## Accepted API and behavior -The user asked for recommendations for these remaining choices. The following -replaces the open-ended alternatives with a proposed initial contract; approval -is still pending. +The user accepted the following contract, including the Go review refinements +recorded below. ### Interface and composition @@ -232,12 +232,34 @@ It does not replace this selector or settle decision 0010's budgets. [synctest.Sleep](https://pkg.go.dev/testing/synctest#Sleep), [http.Server](https://pkg.go.dev/net/http#Server). -These recommendations settle what to implement next for this feature if -approved. They do not imply approval of decision 0010's separate response-header +These accepted recommendations settle the initial implementation for this +feature. They do not imply approval of decision 0010's separate response-header budgets/combination policy or settle the broader identity/error APIs. -Validate custom headers, missing/empty values, casing, repeated/conflicting and -identical values, flattened comma values, native HTTP behavior, both raw/typed -Gateway payloads, middleware ordering, context propagation, rejected-request -short circuiting and authorization-before-streaming. No production API changes -are made by this record. +## Implementation and validation + +action.go implements the immutable value selector and three sentinel errors. +It reuses the HTTP token validator and compares canonical field names, including +noncanonical direct map assignments, without Unicode case folding. Selection +does not mutate the request or encode JSON. + +Tests cover custom header names, missing/empty values, token grammar, casing, +repeated/conflicting and identical values, comma values, sanitized errors, +zero configuration, copying and concurrent selector reuse. The request matrix +covers native HTTP through Go 1.27's in-memory test server, raw REST and HTTP +payload 1.0/2.0 decoding, and already typed V1/V2 request conversion. It preserves +the accepted distinction between V1 values mirrored across maps and actual +repeats within multivalue lists. + +The runnable consumer example uses ordinary http.Handler middleware and a +consumer-owned context key. Composition tests cover extra-header metadata, +ordering, rejected-request short circuiting, context ancestry/cancellation and +using the same selection for authorization and execution after a header change. +They exercise the existing constructor and private request foundations, not +public Lambda invocation, which is still pending. Authorization-before-streaming +integration remains an acceptance test for the future public streaming writer; +this feature adds no streaming API or response-header policy. + +Validation on Go 1.27.1: focused action tests and the runnable example, the full +race suite (including the existing SDK streaming probe), go vet, formatting/diff +checks, and Linux arm64/amd64 builds all pass. No live AWS resources were used. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 0cf16a6..6485ce8 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -19,4 +19,4 @@ Keep related decisions small enough to review and implement independently. | [0008: Streaming architecture](0008-streaming-architecture.md) | Accepted | How should REST response streaming fit alongside the buffered adapter? | | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | | [0010: Response header design](0010-response-header-design.md) | Proposed | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | -| [0011: Request header processing](0011-request-header-processing.md) | Requirement accepted; API proposed | How can consumers process custom headers and safely select actions for their own dispatchers? | +| [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | diff --git a/docs/plan.md b/docs/plan.md index 964527a..0fdcd84 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -23,8 +23,9 @@ port, source copy, or compatibility-preserving update of Beakley. - Provide first-class consumer request-header processing for header-selected action dispatchers. Preserve custom incoming headers independently of response combination rules. [Decision 0011](decisions/0011-request-header-processing.md) - proposes the standard HTTP extension boundary and strict selection support; - API details and the consumer action-header contract remain under review. + accepts the standard HTTP extension boundary and immutable ActionHeader + selector, with explicit configuration and strict missing/ambiguous/invalid + selection errors. The user approved the Go review refinements on 2026-09-15. - Own event serialization at a raw Lambda invocation boundary implementing `Invoke(context.Context, []byte) ([]byte, error)`. - Reuse AWS Lambda Go event/response structs where they meet the contract; @@ -171,6 +172,8 @@ presenting an entire frozen public API at once. - [x] Implement the private decoder and typed semantic validators with direct JSON v2, SDK types, opaque authorizer data, fixtures, and fuzzing. - [x] Implement shared HTTP request conversion and request-lifetime cleanup. +- [x] Implement the accepted action-header selector with ordinary HTTP middleware + examples and native HTTP/raw/typed Gateway request contract tests. - [ ] Implement the buffered HTTP writer and gateway response conversion. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. - [x] Extract shared invocation ownership and implement the private streaming From 16767b337744eaee416482d82db24e5799b1e2f6 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 13:53:58 +0100 Subject: [PATCH 17/44] docs: measure header snapshot costs and refine budget proposal --- docs/benchmarks.md | 62 +++++++ docs/decisions/0010-response-header-design.md | 59 ++++-- docs/plan.md | 4 +- internal/headerprobe/probe_test.go | 172 ++++++++++++++++++ 4 files changed, 280 insertions(+), 17 deletions(-) create mode 100644 internal/headerprobe/probe_test.go diff --git a/docs/benchmarks.md b/docs/benchmarks.md index aa6f617..2fc7165 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -17,3 +17,65 @@ allocate. Do not advertise a 6 MiB total-process or peak-memory guarantee. Raw/typed adapter comparisons, binary response benchmarks, and peak-memory measurements follow when the complete response and invocation paths are wired. + +## Response-header allocation probe + +2026-09-15, Go 1.27.1, darwin/arm64, Apple M2. This is an isolated candidate +algorithm in [internal/headerprobe](../internal/headerprobe/probe_test.go), not +production edge behavior. It performs weighted preflight, preallocated key +sorting, canonicalization, merging and outer SP/HTAB trimming. It deliberately +omits syntax validation, hop-header filtering, automatic headers and transport +projection; the real writer will have additional costs. + +Commands (three samples each): + +```sh +go test ./internal/headerprobe -run 'TestSnapshotMemory|TestPreflightBoundaries' -v -count=3 +go test ./internal/headerprobe -run '^$' -bench '^BenchmarkSnapshot$' -benchmem -benchtime=200ms -count=3 +``` + +Median benchmark results and approximate retained heap after GC: + +| Fixture | Time/op | Cumulative allocation/op | Additional retained heap | +| --- | ---: | ---: | ---: | +| Ordinary headers and two cookies | 487 ns | 576 B | Small; 512–5,832 B observed | +| 190,650 empty values under one name | 1.00 ms | 3,056,017 B | 3,056,016 B | +| 157,286 distinct names, near 6 MiB charge | 41.69 ms | 17,633,168 B | 15,110,032 B | +| 26,214 distinct names, near 1 MiB charge | 4.87 ms | 2,429,490 B | 1,993,632 B | +| 6,553 distinct names, near 256 KiB charge | 1.04 ms | 607,486 B | 498,432 B | +| 157,286 suppressed names, near 6 MiB charge | 35.13 ms | 15,116,592 B | 12,593,456 B | +| Mixed-case aliases, near 6 MiB charge | 1.16 ms | 4,473,267 B | 3,178,912–3,181,320 B | +| One quote-filled value, near 6 MiB charge | 191 ns | 416 B | 416 B | +| Empty values exceeding budget by one entry | 390 µs | 0 B | 0–112 B noise | + +The smaller fixtures use the same algorithm and allocate inputs whose weighted +charge fits the named candidate budget. The prototype still enforces the original +6 MiB ceiling; it does not implement the proposed configurable production option. +Over-budget preflight rejection allocates no snapshot. The quote-filled value +shares immutable string storage with the still-live input; the tiny snapshot +allocation does not include a copy of its bytes. Unbounded JSON v2 encoding of +that snapshot alone produces 12,582,856 bytes, demonstrating why exact encoded +limits remain independent of the weighted charge. + +Memory samples keep both the original header map and snapshot alive across +measurement. Two garbage collections before and after clear previous cases' +sync.Pool buffers. Small deltas remain noisy. Reported retained heap excludes the +application's existing input and is not peak memory, total process memory or RSS. +Encoding happens outside the snapshot memory sample. The suppressed-name case +retains all suppression entries; production may eliminate some once automatic +header decisions finish. Timings are local samples, not Lambda latency claims. + +Preallocating the sorted key slice reduced cumulative allocation for the largest +distinct-name case from roughly 29.35 MB to 17.63 MB; retained map/slice costs +still dominate. This optimization alone does not make a 6 MiB accounting budget +an equivalent heap bound. + +Recommendation for review: use a 256 KiB weighted default and an explicit override +up to 6 MiB, preserving separate encoded limits. The lower default targets modest +additional metadata storage; its exact number is an engineering choice informed +by the measured 0.48 MiB retained/0.58 MiB cumulative high-cardinality case, not +an AWS quota or a proven universal memory bound. See +[decision 0010](decisions/0010-response-header-design.md). + +The isolated probe passes race detection, vet and formatting/diff checks. No +production adapter code changed, and no live AWS services were invoked. diff --git a/docs/decisions/0010-response-header-design.md b/docs/decisions/0010-response-header-design.md index 04edff1..ed0f494 100644 --- a/docs/decisions/0010-response-header-design.md +++ b/docs/decisions/0010-response-header-design.md @@ -1,6 +1,7 @@ # 0010: Shared response headers and transport-specific encoding -Status: proposed for user review on 2026-09-14. No dependent implementation yet. +Status: proposed for user review on 2026-09-14; resource recommendation revised +with allocation evidence on 2026-09-15. No dependent production implementation. Terminology: V1/V2 in this record mean API Gateway payload formats 1.0/2.0, not versions of this Go module. The field-combination table concerns outgoing @@ -17,7 +18,7 @@ comma combination; REST streaming shares V1's multivalue representation. Replace decision 0007's initial ten-field audit with the broader table below. Replace its unaccepted 64 KiB/1,024-entry pair with a single weighted resource -budget tied to the existing Lambda envelope ceiling. Keep actual envelope and +budget with a measured default and explicit override. Keep actual envelope and stream-prefix checks separate. These are proposed refinements, not changes to the already implemented bridge. @@ -176,14 +177,35 @@ Counting suppressed entries is an additional edge policy that bounds copying and sorting even when map values are empty. It also avoids an independent, arbitrary value-count limit. -Recommend starting with the existing 6,291,456-byte response ceiling as this -resource budget, on both buffered and streaming paths, without a new public -tuning option in the first implementation. This is intentionally a broad -compatibility ceiling rather than an attempt to duplicate smaller deployment -quotas. It limits copied logical bytes and, because every valid key costs at -least 33 bytes per entry, permits at most 190,650 charged entries. Counting only -name/value bytes would admit arbitrarily large slices of empty strings; this -cannot. The previous paired proposal also bounded count, using a separate knob. +Following the allocation probe, recommend a **262,144-byte (256 KiB) default** +resource budget on both buffered and streaming paths, with an explicit +WithResponseHeaderBudget(bytes int) Option for applications that need another +budget. Recommend accepting positive values up to the existing 6,291,456-byte +response ceiling, rejecting invalid configuration in New/NewStreaming. Omission +uses the default; an explicit zero does not disable the bound. This follows the +existing options' last-assignment-wins rule and fixed constructor configuration. +The numeric default and option are proposed API/library policy, not AWS quotas. + +The earlier proposal used 6 MiB unconditionally. A prototype with 157,286 distinct +names near that weighted ceiling retained approximately 14.41 MiB for its +snapshot and cumulatively allocated 16.82 MiB per operation, even after +preallocating sorting storage. At a 256 KiB fixture charge, 6,553 distinct names +retained approximately 0.48 MiB and cumulatively allocated 0.58 MiB. A 1 MiB +candidate retained approximately 1.90 MiB. These are measured shapes, not +worst-case proofs. See [the allocation report](../benchmarks.md#response-header-allocation-probe). + +The default aims to keep additional metadata storage modest while leaving an +explicit route for applications with unusual headers. The original 6 MiB remains +the largest opt-in budget; it is no longer the default memory tradeoff for every +application. The 256 KiB choice is an engineering judgment informed by these +measurements, not a uniquely correct threshold derived from AWS documentation. +An override does not change the independent response-envelope or streaming-prefix +limits, and does not guarantee acceptance by API Gateway. + +At the default, every valid key costs at least 33 bytes per charged entry, +allowing at most 7,943 entries; the 6 MiB override permits at most 190,650. +Counting only name/value bytes would admit arbitrarily large slices of empty +strings; the per-entry charge prevents this without a second count setting. This is a proposed library policy. A response below an AWS envelope limit can still exceed the weighted budget. A passing weighted budget does not promise @@ -191,12 +213,13 @@ that a response fits Lambda JSON, API Gateway, a browser, or a low-memory Lambda configuration. Map overhead, backing strings, encoding temporaries and the application's existing allocations are not an exact 6 MiB heap guarantee. -Before finalizing the implementation, measure ordinary headers/cookies, many -empty values, many distinct names, mixed-case collisions and escaped values. -Record cumulative allocations and retained snapshot size separately. If the -high-cardinality cases make this broad ceiling impractical, bring measured -evidence and a lower/configurable policy back for review; do not silently -substitute another small fixed limit. +The isolated, test-only internal/headerprobe package now measures ordinary +headers/cookies, many empty values, many distinct names, suppressed names, +mixed-case collisions, escaped values and rejection before copying. It does not +implement production header validation, filtering or projection. Retained heap +deltas keep the input alive and exclude application allocations; neither these +deltas nor cumulative allocations measure peak memory or RSS. Repeat the probes +against the actual implementation before treating these figures as its baseline. Independently enforce the existing complete buffered JSON envelope limit on raw Invoke. Typed buffered methods still perform no response JSON round trip and @@ -225,6 +248,10 @@ the already implemented bridge for the body. - Preserve every field in every format: impossible with V2's single-string map. - Fixed 64 KiB plus 1,024 entries: still lacks a demonstrated resource/compatibility reason for those numbers. A weighted budget replaces both controls. +- Fixed 6 MiB weighted default: broad compatibility, but the prototype retains + roughly 14.41 MiB for one high-cardinality snapshot before application and + response-encoding allocations. Prefer a smaller default with an explicit + override rather than assigning that tradeoff to every consumer. - Reuse Gateway's smallest quota globally: would conflate request/response, endpoint configuration, HTTP field bytes and JSON framing bytes. - Full semantic registry/parser: significantly larger scope than transport diff --git a/docs/plan.md b/docs/plan.md index 0fdcd84..d81c95a 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -114,7 +114,9 @@ The user accepted the qualification to decision 0007: do not freeze arbitrary 64 KiB/1,024-entry metadata limits or the initial ten-field list without stronger resource and compatibility justification. [Decision 0010](decisions/0010-response-header-design.md) now proposes shared snapshots, a 25-field V2 combination audit, weighted resource -accounting, and a conservative exact streaming-prefix limit. These details await +accounting, and a conservative exact streaming-prefix limit. The 2026-09-15 +allocation probe revises the proposed resource policy to a 256 KiB default with +an explicit positive override up to 6 MiB; this is not yet accepted. These details await user review; no header-policy implementation has been added. Streaming is specified in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. diff --git a/internal/headerprobe/probe_test.go b/internal/headerprobe/probe_test.go new file mode 100644 index 0000000..1a11559 --- /dev/null +++ b/internal/headerprobe/probe_test.go @@ -0,0 +1,172 @@ +// Package headerprobe measures a candidate for decision 0010. Nothing in edge +// imports this package. This is an allocation experiment, not the response +// header implementation: filtering, semantic validation and projection are absent. +package headerprobe + +import ( + "encoding/json/v2" + "fmt" + "net/http" + "runtime" + "slices" + "strings" + "testing" +) + +const budget = 6 * 1024 * 1024 + +// preflight uses incremental subtraction so neither long names nor large value +// slices can overflow a summed charge. Empty slices still cost a name plus 32. +func preflight(h http.Header, remaining int) bool { + for name, values := range h { + for i := range max(1, len(values)) { + for _, n := range []int{len(name), 32} { + if n > remaining { + return false + } + remaining -= n + } + if len(values) != 0 { + if len(values[i]) > remaining { + return false + } + remaining -= len(values[i]) + } + } + } + return true +} + +func snapshot(h http.Header) http.Header { + if !preflight(h, budget) { + return nil + } + keys := make([]string, 0, len(h)) + for key := range h { + keys = append(keys, key) + } + slices.Sort(keys) + out := make(http.Header, len(h)) + for _, key := range keys { + name := http.CanonicalHeaderKey(key) + values := slices.Grow(out[name], len(h[key])) + for _, value := range h[key] { + values = append(values, strings.Trim(value, " \t")) + } + out[name] = values + } + return out +} + +func fixture(name string) http.Header { + ceiling := budget + if strings.HasSuffix(name, "_256k") { + name = strings.TrimSuffix(name, "_256k") + ceiling = 256 * 1024 + } + if strings.HasSuffix(name, "_1m") { + name = strings.TrimSuffix(name, "_1m") + ceiling = 1024 * 1024 + } + switch name { + case "ordinary": + return http.Header{ + "Content-Type": {"application/json"}, + "Cache-Control": {"private", "no-store"}, + "Set-Cookie": {"a=1; Secure; HttpOnly; SameSite=Lax", "b=2; Secure; HttpOnly"}, + "Vary": {"Origin", "Accept-Encoding"}, + } + case "empty_values": + return http.Header{"A": make([]string, budget/33)} + case "distinct_names", "suppressed_names": + h := make(http.Header) + for i := range ceiling / 40 { + key := fmt.Sprintf("H-%06d", i) + if name == "suppressed_names" { + h[key] = nil + } else { + h[key] = []string{""} + } + } + return h + case "case_collisions": + count := budget / (2 * (len("Example") + 32)) + return http.Header{"Example": make([]string, count), "example": make([]string, count)} + case "escaped_value": + return http.Header{"A": {strings.Repeat(`"`, budget-33)}} + case "over_budget": + return http.Header{"A": make([]string, budget/33+1)} + default: + panic("unknown probe fixture") + } +} + +var cases = []string{"ordinary", "empty_values", "distinct_names", "distinct_names_1m", "distinct_names_256k", "suppressed_names", "case_collisions", "escaped_value", "over_budget"} + +func BenchmarkSnapshot(b *testing.B) { + for _, name := range cases { + b.Run(name, func(b *testing.B) { + h := fixture(name) + b.ReportAllocs() + for b.Loop() { + out := snapshot(h) + runtime.KeepAlive(out) + } + }) + } +} + +// TestSnapshotMemory reports incremental live heap after GC, with the input +// deliberately kept alive in both samples. It excludes the application's input +// allocations and is not a peak-memory/RSS measurement. Repeat in a quiet process. +func TestSnapshotMemory(t *testing.T) { + t.Logf("toolchain=%s target=%s/%s", runtime.Version(), runtime.GOOS, runtime.GOARCH) + for _, name := range cases { + t.Run(name, func(t *testing.T) { + h := fixture(name) + // Two collections discard both active and victim sync.Pool entries + // from previous cases, including the JSON encoder's large buffers. + runtime.GC() + runtime.GC() + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + out := snapshot(h) + runtime.GC() + runtime.GC() + runtime.ReadMemStats(&after) + runtime.KeepAlive(h) + runtime.KeepAlive(out) + retained := int64(after.HeapAlloc) - int64(before.HeapAlloc) + // Encoding is intentionally outside the snapshot memory sample. This + // is ordinary JSON v2, not the production bounded response encoder. + encoded, err := json.Marshal(out) + if err != nil { + t.Fatal(err) + } + t.Logf("input_names=%d accepted=%t retained_delta_bytes=%d snapshot_json_bytes=%d", len(h), out != nil, retained, len(encoded)) + }) + } +} + +func TestPreflightBoundaries(t *testing.T) { + for _, tt := range []struct { + name string + h http.Header + max int + want bool + }{ + {name: "nil_slice_at_limit", h: http.Header{"A": nil}, max: 33, want: true}, + {name: "empty_slice_at_limit", h: http.Header{"A": {}}, max: 33, want: true}, + {name: "empty_value_at_limit", h: http.Header{"A": {""}}, max: 33, want: true}, + {name: "name_over_limit", h: http.Header{"AA": nil}, max: 33}, + {name: "value_over_limit", h: http.Header{"A": {"x"}}, max: 33}, + {name: "repeats_at_limit", h: http.Header{"A": {"", ""}}, max: 66, want: true}, + {name: "repeats_over_limit", h: http.Header{"A": {"", ""}}, max: 65}, + } { + t.Run(tt.name, func(t *testing.T) { + if got := preflight(tt.h, tt.max); got != tt.want { + t.Errorf("preflight(%v, %d) = %t, want %t", tt.h, tt.max, got, tt.want) + } + }) + } +} From 93eb51c2006665e80328c473579d2d52a4b4e01e Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 14:15:38 +0100 Subject: [PATCH 18/44] feat: implement bounded response headers and streaming metadata --- README.md | 22 +- adapter.go | 21 +- docs/benchmarks.md | 32 ++ docs/decisions/0006-buffered-responses.md | 8 +- docs/decisions/0007-response-metadata.md | 19 +- docs/decisions/0010-response-header-design.md | 57 +++- docs/decisions/README.md | 4 +- docs/plan.md | 21 +- response_encoding.go | 10 +- response_headers.go | 230 ++++++++++++++ response_headers_bench_test.go | 84 ++++++ response_headers_test.go | 284 ++++++++++++++++++ stream_encoding.go | 33 ++ stream_encoding_test.go | 121 ++++++++ 14 files changed, 903 insertions(+), 43 deletions(-) create mode 100644 response_headers.go create mode 100644 response_headers_bench_test.go create mode 100644 response_headers_test.go create mode 100644 stream_encoding.go create mode 100644 stream_encoding_test.go diff --git a/README.md b/README.md index dfc6d54..0770bd0 100644 --- a/README.md +++ b/README.md @@ -10,14 +10,17 @@ uses AWS Lambda Go v1.55.0 event structs and encoding/json/v2 directly, retainin authorizer JSON separately. Request conversion covers URLs, headers, body bytes, and request lifetime. A shared invocation scope now covers cleanup and cancellation; private response encoding provides text/base64 selection and bounded direct JSON -v2 output. The buffered HTTP writer, gateway header conversion, public invocation -methods, and identity extraction are not implemented yet; the adapter cannot yet +v2 output. Private response-header snapshots and V1/V2 projections now enforce +the accepted syntax, suppression, cookie and field-combination rules. The +buffered HTTP writer, public invocation methods, and identity extraction are not +implemented yet; the adapter cannot yet be registered as a Lambda handler. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle -is covered by race-enabled synthetic concurrency tests. HTTP streaming framing -and public streaming entry points remain under construction; deployed API Gateway +is covered by race-enabled synthetic concurrency tests. Bounded JSON v2 metadata +prefix encoding is implemented. HTTP streaming commitment and public streaming +entry points remain under construction; deployed API Gateway streaming has not been verified. ## Header-selected actions @@ -40,6 +43,17 @@ unfinished Lambda invocation methods. See the [runnable middleware example](action_example_test.go) and [accepted header contract](docs/decisions/0011-request-header-processing.md). +## Response header budget + +Response headers have a 256 KiB weighted resource budget, configurable with +`edge.WithResponseHeaderBudget(bytes)` from 1 byte through 6 MiB. Each original +value costs `len(name) + len(value) + 32`; nil/empty slices still cost a name plus +32, and generated headers consume remaining budget. This is separate from the +complete buffered-envelope limit and the 16,000-byte streaming metadata prefix +(including its delimiter). It is neither an AWS quota nor a total heap bound. +The option validates configuration now; the HTTP writers will apply it at +commitment when invocation is wired. + - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) - [Repository working agreement](AGENTS.md) diff --git a/adapter.go b/adapter.go index 9da2c0e..a64efd2 100644 --- a/adapter.go +++ b/adapter.go @@ -18,7 +18,8 @@ type Adapter struct { } type config struct { - gatewayIdentity bool + gatewayIdentity bool + responseHeaderBudget int } // Option configures an [Adapter] during construction. @@ -38,6 +39,19 @@ func WithGatewayIdentity(enabled bool) Option { } } +// WithResponseHeaderBudget sets the response header resource budget in bytes. +// The default is 256 KiB. New rejects a final configured value outside 1..6 MiB; +// zero does not disable the bound. Each original value costs its name length, +// value length and 32 bytes; nil/empty slices cost their name length and 32. +// Generated headers also consume the budget. This is a library resource policy, +// not an AWS quota or a bound on total heap usage. It does not change the separate +// buffered envelope or streaming metadata limits. Invocation is not yet wired. +func WithResponseHeaderBudget(bytes int) Option { + return func(c *config) { + c.responseHeaderBudget = bytes + } +} + // New validates handler and opts and returns an Adapter without performing I/O // or invoking handler. Nil handlers, including typed nil handlers, and nil // options are rejected. On error, the returned Adapter is nil. @@ -58,12 +72,15 @@ func New(handler http.Handler, opts ...Option) (*Adapter, error) { } } - var cfg config + cfg := config{responseHeaderBudget: defaultResponseHeaderBudget} for i, opt := range opts { if opt == nil { return nil, fmt.Errorf("edge: nil option at index %d", i) } opt(&cfg) } + if cfg.responseHeaderBudget <= 0 || cfg.responseHeaderBudget > maxResponseBytes { + return nil, errors.New("edge: response header budget must be between 1 and 6291456 bytes") + } return &Adapter{handler: handler, config: cfg}, nil } diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 2fc7165..120784c 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -79,3 +79,35 @@ an AWS quota or a proven universal memory bound. See The isolated probe passes race detection, vet and formatting/diff checks. No production adapter code changed, and no live AWS services were invoked. + +## Implemented response-header layer + +After approval of decision 0010, repeated measurements against the actual +snapshot and projection functions on the same Go 1.27.1 / Apple M2 environment: + +```sh +go test -run '^TestResponseHeaderRetainedMemory$' -v -count=3 +go test -run '^$' -bench '^BenchmarkResponseHeaders$' -benchmem -benchtime=200ms -count=3 +``` + +Median cumulative allocations in bytes per operation; projection columns include +snapshot construction and independently owned projection storage: + +| Fixture | Snapshot only | Snapshot + V1 | Snapshot + V2 | Retained snapshot heap | +| --- | ---: | ---: | ---: | ---: | +| Ordinary headers/cookies | 600 | 712 | 1,016 | 728–1,048 | +| 6,553 distinct names near default 256 KiB charge | 607,557 | 1,106,130 | 937,527 | 498,648–498,760 | +| 157,286 distinct names near explicit 6 MiB charge | 17,633,192 | 32,743,176 | 28,129,496 | 15,110,248–15,126,208 | + +The default's high-cardinality retained snapshot remains about 0.48 MiB, matching +the candidate probe. Projections add allocations, especially with the largest +opt-in budget. Suppressed entries are omitted from output; projection maps are +sized for represented fields, not the number of suppression markers. + +These helper benchmarks exclude the HTTP writer, body handling and final JSON +encoding. Compiler escape analysis can also differ once they are called through +the future public invocation path. Retained heap excludes the application input +and projections; it is not peak memory or RSS. Median snapshot timings were +approximately 1.36 µs, 1.27 ms and 52.17 ms respectively, with substantial timing +variation in the largest case. These are exploratory samples, not CI thresholds +or Lambda latency claims. They do not warrant changing the approved default. diff --git a/docs/decisions/0006-buffered-responses.md b/docs/decisions/0006-buffered-responses.md index 3eacfe0..b5af001 100644 --- a/docs/decisions/0006-buffered-responses.md +++ b/docs/decisions/0006-buffered-responses.md @@ -4,8 +4,8 @@ Status: accepted with the 2026-09-14 review refinements; implementation underway Reviewed against the implemented decoder/request layer on 2026-09-14. The [plan review](../reviews/2026-09-14-plan-review.md) was approved by the user and its -refinements are incorporated below. Exact metadata budgets and the initial V2 -joinable-field set remain separate specifications before dependent implementation. +refinements are incorporated below. Decision 0010 now settles the exact metadata +budget and V2 joinable-field set; their private shared layer is implemented. ## Question @@ -106,7 +106,7 @@ format, and handle features or failures that a buffered invocation cannot carry? - V2 moves Set-Cookie values into Cookies. Join repeated values with a comma and space only for a documented, audited set of list-valued fields. Reject repeated singleton or unknown fields. Preserve order and commas inside individual - values. Leave MultiValueHeaders empty. Specify the initial list separately; + values. Leave MultiValueHeaders empty. Decision 0010 specifies the initial list; V2 cannot retain general header line boundaries, so applications requiring them must choose V1. - Emit text directly only when bytes are valid UTF-8, Content-Encoding is absent @@ -131,7 +131,7 @@ format, and handle features or failures that a buffered invocation cannot carry? - Invoke uses json/v2.MarshalWrite with a bounded destination to enforce 6,291,456 bytes including JSON overhead while retaining encoded output. A body below the cap can still fail this check. Preflight committed metadata - bytes and entry counts before copying; specify those exact budgets separately. + using decision 0010's weighted resource charge before copying. A typed method cannot measure the exact envelope produced by its caller's serializer without violating the agreed JSON-free typed boundary; document that the caller/runtime owns that final limit. Apply the common body/expansion diff --git a/docs/decisions/0007-response-metadata.md b/docs/decisions/0007-response-metadata.md index f2e94cf..fa716c0 100644 --- a/docs/decisions/0007-response-metadata.md +++ b/docs/decisions/0007-response-metadata.md @@ -1,11 +1,13 @@ # 0007: Response metadata budgets and V2 list fields -Status: qualified direction accepted; concrete metadata policy remains unresolved. +Status: historical qualification; concrete metadata policy resolved by accepted +decision 0010 on 2026-09-15. The concrete follow-up is [decision 0010](0010-response-header-design.md), proposed after the streaming bridge implementation. It recommends a broader field audit, -weighted resource accounting, and separate encoded-output limits. Its details -still require user review; this record preserves the earlier qualification. +weighted resource accounting, and separate encoded-output limits. The user has +accepted its measured default and override; this record preserves the earlier +qualification and the superseded initial proposal below. ## Accepted qualification @@ -13,9 +15,8 @@ The user accepted the qualified recommendation: keep the documented Lambda envelope limit and semantics-preserving header translation, but do not freeze the proposed 64 KiB/1,024-entry limits without stronger justification. The ten fields below are an initial audit, not an AWS allowlist or an exhaustive set of -valid list-valued fields. Broader compatibility must be considered before fixing -that set as the library contract. No numeric metadata limit or exact joinable -set has been approved or implemented. +valid list-valued fields. Broader compatibility was considered in decision 0010's +accepted 25-field audit and measured resource policy. The numeric proposal was a library resource policy, not an AWS quota or a result of header-allocation benchmarks. Its intended protection was to bound additional @@ -89,6 +90,6 @@ V1. Hop-by-hop fields are removed under decision 0006 before V2 joining. ## Resolution -The user agreed with the qualification above. Resolve evidence-based resource -budgets and broader field compatibility before implementing those policies. -Bounded envelope encoding and invocation lifetime work do not depend on them. +The user agreed with the qualification above, then approved the evidence-based +resource budget and broader field audit in decision 0010. Follow that later +record for the current contract and implementation status. diff --git a/docs/decisions/0010-response-header-design.md b/docs/decisions/0010-response-header-design.md index ed0f494..45c9879 100644 --- a/docs/decisions/0010-response-header-design.md +++ b/docs/decisions/0010-response-header-design.md @@ -1,7 +1,8 @@ # 0010: Shared response headers and transport-specific encoding -Status: proposed for user review on 2026-09-14; resource recommendation revised -with allocation evidence on 2026-09-15. No dependent production implementation. +Status: accepted by the user on 2026-09-15, including the measured 256 KiB +default and explicit override up to 6 MiB. Shared snapshots, projections and +streaming-prefix encoding are implemented; HTTP writer wiring follows. Terminology: V1/V2 in this record mean API Gateway payload formats 1.0/2.0, not versions of this Go module. The field-combination table concerns outgoing @@ -19,7 +20,7 @@ comma combination; REST streaming shares V1's multivalue representation. Replace decision 0007's initial ten-field audit with the broader table below. Replace its unaccepted 64 KiB/1,024-entry pair with a single weighted resource budget with a measured default and explicit override. Keep actual envelope and -stream-prefix checks separate. These are proposed refinements, not changes to +stream-prefix checks separate. These are accepted refinements, not changes to the already implemented bridge. ## Evidence and limits of the evidence @@ -134,7 +135,7 @@ empty list elements, invalid dictionary members, or invalid singleton-alternativ combinations. Original within-slice order is preserved, including for ordered fallbacks, challenges and signature dictionaries. -### Proposed initial V2 combination table +### Accepted initial V2 combination table | Fields | Definition supporting combination | | --- | --- | @@ -184,7 +185,7 @@ budget. Recommend accepting positive values up to the existing 6,291,456-byte response ceiling, rejecting invalid configuration in New/NewStreaming. Omission uses the default; an explicit zero does not disable the bound. This follows the existing options' last-assignment-wins rule and fixed constructor configuration. -The numeric default and option are proposed API/library policy, not AWS quotas. +The numeric default and option are accepted API/library policy, not AWS quotas. The earlier proposal used 6 MiB unconditionally. A prototype with 157,286 distinct names near that weighted ceiling retained approximately 14.41 MiB for its @@ -207,7 +208,7 @@ allowing at most 7,943 entries; the 6 MiB override permits at most 190,650. Counting only name/value bytes would admit arbitrarily large slices of empty strings; the per-entry charge prevents this without a second count setting. -This is a proposed library policy. A response below an AWS envelope limit can +This is a library policy. A response below an AWS envelope limit can still exceed the weighted budget. A passing weighted budget does not promise that a response fits Lambda JSON, API Gateway, a browser, or a low-memory Lambda configuration. Map overhead, backing strings, encoding temporaries and the @@ -257,12 +258,44 @@ the already implemented bridge for the body. - Full semantic registry/parser: significantly larger scope than transport adaptation; unnecessary for ordinary single values and V1 preservation. -After approval, implement snapshot/preflight and the V1/V2 projections first. -Test mutation after commitment, case collisions, nil/empty distinctions, control -bytes/UTF-8, Connection nominations, cookies, quoted commas, list/dictionary -order, rejected repeated fields and weighted boundaries. Run allocation probes -before settling the resource implementation. Then add bounded stream-prefix -encoding and connect the shared snapshot to the separate HTTP writers. +The shared layer is implemented in response_headers.go. It owns canonical maps +and value slices, tracks the original preflight charge without refunds, preserves +automatic-header suppression and returns independent V1/V2 projections. New +validates WithResponseHeaderBudget after option application, so the final scalar +assignment wins. Only final status/body behavior and HTTP commitment timing +remain responsibilities of the separate writers. + +Connection parsing tolerates empty list elements under RFC 9110 section 5.6.1.2; +the input budget bounds that scanning. Nonempty malformed tokens remain faults. +TrailerPrefix entries, represented Trailer/Upgrade values and upgrade nominations +are unsupported; nil/empty Trailer/Upgrade slices emit nothing and are removed. +Strict Content-Length validation follows hop-field removal. A nominated length +cannot reappear through inference. These details follow the accepted transport +and suppression rules; they add no new public policy controls. +[HTTP list recipient requirements](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.6.1.2). + +stream_encoding.go encodes explicit status and multivalue metadata through direct +JSON v2, using the shared bounded destination with a 15,992-byte JSON limit. It +returns exactly sized storage containing JSON plus eight NUL bytes. It never +returns a partial prefix. A bridge test verifies that oversized metadata fails +before either headers or body become available. + +Tests cover input/projection ownership, casing, suppression, syntax/UTF-8, +Connection filtering, cookies, all 25 joinable names, ordered challenges and +signature dictionaries, rejected repeats, Content-Length syntax, configuration, +exact resource boundaries, generated-field accounting and exact encoded prefix +limits including escaping. Over-budget preflight rejection allocates no snapshot. +Production allocation samples are recorded in [the benchmark report](../benchmarks.md). + +Validation on Go 1.27.1: full race tests, vet, formatting/diff checks and Linux +arm64/amd64 builds pass. The default's measured retained snapshot matches the +probe at approximately 0.48 MiB for the high-cardinality fixture. Projection +allocations are reported separately; no new resource policy is needed. + +Next connect these primitives to the buffered and streaming HTTP writers, with +status/method rules, sniffing, length enforcement, commitment timing and late +unsupported-trailer detection. Constructor options and private primitives do +not make the unfinished adapter usable as a Lambda handler yet. Public invocation/identity work and the SDK Runtime API header compatibility question remain separate. No account APIs or live AWS deployments were used for diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 6485ce8..b7376c2 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -15,8 +15,8 @@ Keep related decisions small enough to review and implement independently. | [0004: Typed events](0004-typed-events.md) | Accepted | How can already typed events bypass edge's envelope JSON work? | | [0005: HTTP requests](0005-http-request-translation.md) | Accepted | How should raw and typed events become ordinary server requests? | | [0006: Buffered responses](0006-buffered-responses.md) | Accepted | How should HTTP responses, unsupported capabilities, and completion failures behave? | -| [0007: Response metadata](0007-response-metadata.md) | Qualified direction accepted; details open | What metadata budgets and V2 joinable-field set should apply? | +| [0007: Response metadata](0007-response-metadata.md) | Qualification resolved by 0010 | What metadata budgets and V2 joinable-field set should apply? | | [0008: Streaming architecture](0008-streaming-architecture.md) | Accepted | How should REST response streaming fit alongside the buffered adapter? | | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | -| [0010: Response header design](0010-response-header-design.md) | Proposed | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | +| [0010: Response header design](0010-response-header-design.md) | Accepted | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | | [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | diff --git a/docs/plan.md b/docs/plan.md index d81c95a..bea4732 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -107,17 +107,20 @@ invocation completion, and the documented multipart cleanup limit. The accepted [2026-09-14 plan review](reviews/2026-09-14-plan-review.md) requires response-contract refinements and completing a runnable adapter, including the minimum gateway identity foundation, before broader authentication work. These -enhancements are incorporated into decision 0006. Concrete metadata/header -policies, exported error categories, and identity APIs remain separate reviews. +enhancements are incorporated into decision 0006. Decision 0010 settles concrete +metadata/header policies; exported error categories and identity APIs remain +separate reviews. The user accepted the qualification to decision 0007: do not freeze arbitrary 64 KiB/1,024-entry metadata limits or the initial ten-field list without stronger resource and compatibility justification. [Decision 0010](decisions/0010-response-header-design.md) -now proposes shared snapshots, a 25-field V2 combination audit, weighted resource +now accepts shared snapshots, a 25-field V2 combination audit, weighted resource accounting, and a conservative exact streaming-prefix limit. The 2026-09-15 -allocation probe revises the proposed resource policy to a 256 KiB default with -an explicit positive override up to 6 MiB; this is not yet accepted. These details await -user review; no header-policy implementation has been added. Streaming is specified in +allocation probe revised the resource policy to a 256 KiB default with +an explicit positive override up to 6 MiB; the user approved it on 2026-09-15. +The shared snapshot, suppression/accounting logic, V1/V2 projections, option +validation and bounded streaming metadata prefix are implemented. Writer wiring +remains. Streaming is specified in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. The high-level architecture is accepted. [Decision 0009](decisions/0009-streaming-boundaries.md) @@ -141,8 +144,8 @@ sniffing, framing validation, and public entry points remain to be implemented. Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary errors, cleans up on failures/panics, and checks parent cancellation. The HTTP -writer and gateway header conversion await the concrete policy in -[decision 0007](decisions/0007-response-metadata.md). Public wiring still awaits +writer can now use the shared header implementation from +[decision 0010](decisions/0010-response-header-design.md). Public wiring still awaits the minimum identity contracts. [Initial codec benchmarks](benchmarks.md) record allocation baselines; they do not establish a peak-memory bound. @@ -176,6 +179,8 @@ presenting an entire frozen public API at once. - [x] Implement shared HTTP request conversion and request-lifetime cleanup. - [x] Implement the accepted action-header selector with ordinary HTTP middleware examples and native HTTP/raw/typed Gateway request contract tests. +- [x] Implement shared response-header snapshots, budget configuration, V1/V2 + projections and bounded REST streaming metadata prefix encoding. - [ ] Implement the buffered HTTP writer and gateway response conversion. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. - [x] Extract shared invocation ownership and implement the private streaming diff --git a/response_encoding.go b/response_encoding.go index fa63035..6b8d835 100644 --- a/response_encoding.go +++ b/response_encoding.go @@ -34,15 +34,21 @@ func marshalResponse(response any) ([]byte, error) { // allocate temporary codec storage; this is not a total-memory guarantee. type responseBuffer struct { data []byte + // A zero limit selects the buffered-response ceiling. + limit int } func (b *responseBuffer) Write(p []byte) (int, error) { - if len(p) > maxResponseBytes-len(b.data) { + limit := b.limit + if limit == 0 { + limit = maxResponseBytes + } + if len(p) > limit-len(b.data) { return 0, errResponseTooLarge } n := len(b.data) + len(p) if n > cap(b.data) { - capacity := min(maxResponseBytes, max(n, 2*cap(b.data), 512)) + capacity := min(limit, max(n, 2*cap(b.data), 512)) data := make([]byte, len(b.data), capacity) copy(data, b.data) b.data = data diff --git a/response_headers.go b/response_headers.go new file mode 100644 index 0000000..fd04041 --- /dev/null +++ b/response_headers.go @@ -0,0 +1,230 @@ +package edge + +import ( + "errors" + "fmt" + "net/http" + "slices" + "strconv" + "strings" + "unicode/utf8" +) + +const defaultResponseHeaderBudget = 256 * 1024 + +var ( + errResponseHeaderBudget = errors.New("edge: response header budget exceeded") + errResponseHeaders = errors.New("edge: invalid response headers") + errResponseTrailers = fmt.Errorf("edge: response trailers: %w", http.ErrNotSupported) + errResponseUpgrade = fmt.Errorf("edge: response upgrade: %w", http.ErrNotSupported) + errResponseHeaderRepeat = errors.New("edge: repeated response field cannot be represented in payload 2.0") +) + +// responseHeaders owns its map and value slices. Strings are immutable and may +// share storage with the input. One writer owns the snapshot until completion. +// remaining never refunds bytes trimmed, removed or suppressed at commitment. +type responseHeaders struct { + fields http.Header + remaining int + noContentType bool + noContentLength bool +} + +// chargeResponseHeaders bounds work and copying before sorting or allocation. +// Empty slices count as one entry so suppression markers also have a cost. +func chargeResponseHeaders(fields http.Header, remaining int) (int, error) { + for name, values := range fields { + for i := range max(1, len(values)) { + for _, size := range []int{len(name), 32} { + if size > remaining { + return 0, errResponseHeaderBudget + } + remaining -= size + } + if len(values) != 0 { + if len(values[i]) > remaining { + return 0, errResponseHeaderBudget + } + remaining -= len(values[i]) + } + } + } + return remaining, nil +} + +func snapshotResponseHeaders(fields http.Header, budget int) (*responseHeaders, error) { + if budget <= 0 || budget > maxResponseBytes { + return nil, errors.New("edge: invalid response header budget") + } + remaining, err := chargeResponseHeaders(fields, budget) + if err != nil { + return nil, err + } + keys := make([]string, 0, len(fields)) + for name, values := range fields { + if strings.HasPrefix(name, http.TrailerPrefix) { + return nil, errResponseTrailers + } + if !validToken(name) { + return nil, errResponseHeaders + } + for _, value := range values { + if !utf8.ValidString(value) || !validHeaderValue(value) { + return nil, errResponseHeaders + } + } + keys = append(keys, name) + } + slices.Sort(keys) + h := &responseHeaders{fields: make(http.Header, len(fields)), remaining: remaining} + for _, key := range keys { + name := http.CanonicalHeaderKey(key) + values := slices.Grow(h.fields[name], len(fields[key])) + for _, value := range fields[key] { + values = append(values, strings.Trim(value, " \t")) + } + h.fields[name] = values + } + // Check before Connection removal: a nomination must not hide an attempt + // to declare trailers or upgrade the response protocol. + if len(h.fields["Trailer"]) != 0 { + return nil, errResponseTrailers + } + if len(h.fields["Upgrade"]) != 0 { + return nil, errResponseUpgrade + } + for _, value := range h.fields["Connection"] { + for token := range strings.SplitSeq(value, ",") { + token = strings.Trim(token, " \t") + // RFC 9110 §5.6.1.2 requires tolerance of empty list elements. + // The original byte budget bounds even comma-only input. + if token == "" { + continue + } + if !validToken(token) { + return nil, errResponseHeaders + } + name := http.CanonicalHeaderKey(token) + if name == "Upgrade" { + return nil, errResponseUpgrade + } + h.remove(name) + } + } + for _, name := range []string{"Connection", "Proxy-Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization", "Te", "Trailer", "Transfer-Encoding", "Upgrade"} { + h.remove(name) + } + if _, _, err := responseContentLength(h.fields); err != nil { + return nil, err + } + return h, nil +} + +func (h *responseHeaders) remove(name string) { + delete(h.fields, name) + switch name { + case "Content-Type": + h.noContentType = true + case "Content-Length": + h.noContentLength = true + } +} + +// automatic adds a canonical field only when absent and not suppressed. It is +// used for generated Content-Type/Content-Length; removing a field never creates +// new budget. A failed addition leaves both fields and remaining unchanged. +func (h *responseHeaders) automatic(name, value string) error { + if _, present := h.fields[name]; present { + return nil + } + if name == "Content-Type" && h.noContentType || name == "Content-Length" && h.noContentLength { + return nil + } + remaining, err := chargeResponseHeaders(http.Header{name: {value}}, h.remaining) + if err != nil { + return err + } + h.fields[name] = []string{value} + h.remaining = remaining + return nil +} + +func responseContentLength(fields http.Header) (int64, bool, error) { + values := fields["Content-Length"] + if len(values) == 0 { + return 0, false, nil + } + if len(values) != 1 || values[0] == "" { + return 0, false, errResponseHeaders + } + for i := range len(values[0]) { + if values[0][i] < '0' || values[0][i] > '9' { + return 0, false, errResponseHeaders + } + } + n, err := strconv.ParseInt(values[0], 10, 64) + if err != nil { + return 0, false, errResponseHeaders + } + return n, true, nil +} + +// v1 returns an independent multivalue projection, omitting suppression markers. +// It is also the header representation used by REST streaming metadata. +func (h *responseHeaders) v1() map[string][]string { + count := 0 + for _, values := range h.fields { + if len(values) != 0 { + count++ + } + } + if count == 0 { + return nil + } + fields := make(map[string][]string, count) + for name, values := range h.fields { + if len(values) != 0 { + fields[name] = slices.Clone(values) + } + } + return fields +} + +func (h *responseHeaders) v2() (map[string]string, []string, error) { + // Validate multiplicity before allocating the projection or joining values. + count := 0 + for name, values := range h.fields { + if name == "Set-Cookie" || len(values) == 0 { + continue + } + if len(values) > 1 && !combinableResponseField(name) { + return nil, nil, errResponseHeaderRepeat + } + count++ + } + fields := make(map[string]string, count) + for name, values := range h.fields { + if name == "Set-Cookie" || len(values) == 0 { + continue + } + fields[name] = strings.Join(values, ", ") + } + return fields, slices.Clone(h.fields["Set-Cookie"]), nil +} + +// The field-specific list/dictionary audit and references are in decision 0010. +// This immutable switch permits representation conversion, not semantic repair. +func combinableResponseField(name string) bool { + switch name { + case "Accept-Ranges", "Allow", "Content-Encoding", "Content-Language", "Vary", "Www-Authenticate", "Cache-Control": + return true + case "Access-Control-Allow-Headers", "Access-Control-Allow-Methods", "Access-Control-Expose-Headers": + return true + case "Link", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Referrer-Policy", "Server-Timing", "Accept-Patch", "Accept-Ch": + return true + case "Cache-Status", "Proxy-Status", "Content-Digest", "Repr-Digest", "Want-Content-Digest", "Want-Repr-Digest", "Signature", "Signature-Input": + return true + default: + return false + } +} diff --git a/response_headers_bench_test.go b/response_headers_bench_test.go new file mode 100644 index 0000000..c6728a9 --- /dev/null +++ b/response_headers_bench_test.go @@ -0,0 +1,84 @@ +package edge + +import ( + "fmt" + "net/http" + "runtime" + "testing" +) + +func responseHeaderFixture(name string) http.Header { + if name == "ordinary" { + return http.Header{ + "Content-Type": {"application/json"}, + "Cache-Control": {"private", "no-store"}, + "Set-Cookie": {"a=1; Secure; HttpOnly; SameSite=Lax", "b=2; Secure; HttpOnly"}, + "Vary": {"Origin", "Accept-Encoding"}, + } + } + budget := defaultResponseHeaderBudget + if name == "distinct_names_6m" { + budget = maxResponseBytes + } + fields := make(http.Header) + for i := range budget / 40 { + fields[fmt.Sprintf("H-%06d", i)] = []string{""} + } + return fields +} + +func BenchmarkResponseHeaders(b *testing.B) { + for _, name := range []string{"ordinary", "distinct_names_default", "distinct_names_6m"} { + for _, operation := range []string{"snapshot", "v1", "v2"} { + b.Run(name+"/"+operation, func(b *testing.B) { + fields := responseHeaderFixture(name) + budget := defaultResponseHeaderBudget + if name == "distinct_names_6m" { + budget = maxResponseBytes + } + b.ReportAllocs() + for b.Loop() { + h, err := snapshotResponseHeaders(fields, budget) + if err != nil { + b.Fatal(err) + } + switch operation { + case "v1": + runtime.KeepAlive(h.v1()) + case "v2": + out, cookies, err := h.v2() + if err != nil { + b.Fatal(err) + } + runtime.KeepAlive(out) + runtime.KeepAlive(cookies) + } + runtime.KeepAlive(h) + } + }) + } + } +} + +func TestResponseHeaderRetainedMemory(t *testing.T) { + for _, name := range []string{"ordinary", "distinct_names_default", "distinct_names_6m"} { + t.Run(name, func(t *testing.T) { + fields := responseHeaderFixture(name) + budget := defaultResponseHeaderBudget + if name == "distinct_names_6m" { + budget = maxResponseBytes + } + runtime.GC() + runtime.GC() + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + h := mustSnapshot(t, fields, budget) + runtime.GC() + runtime.GC() + runtime.ReadMemStats(&after) + runtime.KeepAlive(fields) + runtime.KeepAlive(h) + t.Logf("additional retained heap = %d bytes (excludes input and projection; not peak memory)", int64(after.HeapAlloc)-int64(before.HeapAlloc)) + }) + } +} diff --git a/response_headers_test.go b/response_headers_test.go new file mode 100644 index 0000000..c1591b4 --- /dev/null +++ b/response_headers_test.go @@ -0,0 +1,284 @@ +package edge + +import ( + "errors" + "fmt" + "maps" + "net/http" + "slices" + "strings" + "testing" +) + +func mustSnapshot(t testing.TB, fields http.Header, budget int) *responseHeaders { + t.Helper() + h, err := snapshotResponseHeaders(fields, budget) + if err != nil { + t.Fatal(err) + } + return h +} + +func TestResponseHeaderSnapshotOwnership(t *testing.T) { + fields := http.Header{ + "X-Mixed": {" \tfirst\t ", "second"}, "x-mixed": {"third"}, + "X-Empty": {""}, "X-Suppressed": nil, + "Vary": nil, "vary": {"Origin"}, + "X-Unicode": {"café"}, "X-Interior": {"a\t b"}, + } + h := mustSnapshot(t, fields, defaultResponseHeaderBudget) + fields["X-Mixed"][0] = "changed" + clear(fields) + if !slices.Equal(h.fields.Values("X-Mixed"), []string{"first", "second", "third"}) { + t.Errorf("snapshot values = %v; want owned, trimmed values in lexical alias order", h.fields["X-Mixed"]) + } + if values, present := h.fields["X-Suppressed"]; !present || len(values) != 0 { + t.Errorf("suppression marker = %v, %t; want present and empty", values, present) + } + if h.fields.Get("Vary") != "Origin" || h.fields.Get("X-Unicode") != "café" || h.fields.Get("X-Interior") != "a\t b" { + t.Errorf("snapshot changed explicit content: %v", h.fields) + } + v1 := h.v1() + if _, present := v1["X-Suppressed"]; present { + t.Error("V1 emitted a suppression marker") + } + if !slices.Equal(v1["X-Empty"], []string{""}) { + t.Errorf("V1 empty field = %v; want one empty value", v1["X-Empty"]) + } + v1["X-Mixed"][0] = "changed again" + delete(v1, "X-Empty") + if h.fields.Get("X-Mixed") != "first" || len(h.fields["X-Empty"]) != 1 { + t.Error("V1 projection mutation changed snapshot") + } +} + +func TestResponseHeaderValidation(t *testing.T) { + for _, tt := range []struct { + name string + fields http.Header + want error + }{ + {name: "empty_name", fields: http.Header{"": {"v"}}, want: errResponseHeaders}, + {name: "space_name", fields: http.Header{"Bad Name": {"v"}}, want: errResponseHeaders}, + {name: "unicode_name", fields: http.Header{"Ünicode": nil}, want: errResponseHeaders}, + {name: "bad_utf8", fields: http.Header{"A": {"\xff"}}, want: errResponseHeaders}, + {name: "carriage_return", fields: http.Header{"A": {"v\r"}}, want: errResponseHeaders}, + {name: "newline", fields: http.Header{"A": {"v\n"}}, want: errResponseHeaders}, + {name: "nul", fields: http.Header{"A": {"\x00"}}, want: errResponseHeaders}, + {name: "del", fields: http.Header{"A": {"\x7f"}}, want: errResponseHeaders}, + {name: "control", fields: http.Header{"A": {"\x1f"}}, want: errResponseHeaders}, + {name: "trailer_prefix", fields: http.Header{http.TrailerPrefix + "Digest": nil}, want: http.ErrNotSupported}, + {name: "trailer", fields: http.Header{"trailer": {"Digest"}}, want: http.ErrNotSupported}, + {name: "upgrade", fields: http.Header{"upgrade": {"websocket"}}, want: http.ErrNotSupported}, + {name: "hidden_trailer", fields: http.Header{"Connection": {"Trailer"}, "Trailer": {"Digest"}}, want: http.ErrNotSupported}, + {name: "hidden_upgrade", fields: http.Header{"Connection": {"Upgrade"}, "Upgrade": {"websocket"}}, want: http.ErrNotSupported}, + {name: "upgrade_nomination", fields: http.Header{"Connection": {"upgrade"}}, want: http.ErrNotSupported}, + {name: "invalid_connection_token", fields: http.Header{"Connection": {"good, bad token"}}, want: errResponseHeaders}, + {name: "quoted_connection_token", fields: http.Header{"Connection": {`"Custom"`}}, want: errResponseHeaders}, + {name: "invalid_removed_field", fields: http.Header{"Connection": {"Custom"}, "Custom": {"\n"}}, want: errResponseHeaders}, + } { + t.Run(tt.name, func(t *testing.T) { + h, err := snapshotResponseHeaders(tt.fields, defaultResponseHeaderBudget) + if h != nil || !errors.Is(err, tt.want) { + t.Errorf("snapshot(%v) = %v, %v; want nil, %v", tt.fields, h, err, tt.want) + } + }) + } +} + +func TestResponseHeaderConnectionFiltering(t *testing.T) { + fields := http.Header{ + "Connection": {" custom , content-type, ,CONTENT-LENGTH, "}, + "connection": {"Other,Connection"}, + "Custom": {"secret"}, "Other": {"hidden"}, + "Content-Type": {"application/json"}, "Content-Length": {"bad", "length"}, + "Proxy-Connection": {"close"}, "Keep-Alive": {"timeout=5"}, + "Proxy-Authenticate": {"Basic"}, "Proxy-Authorization": {"credentials"}, + "Te": {"trailers"}, "Transfer-Encoding": {"chunked"}, + "Trailer": nil, "Upgrade": {}, "Remain": {"visible"}, + } + h := mustSnapshot(t, fields, defaultResponseHeaderBudget) + if !maps.EqualFunc(h.fields, http.Header{"Remain": {"visible"}}, slices.Equal[[]string]) { + t.Errorf("filtered headers = %v; want only Remain", h.fields) + } + for name, value := range map[string]string{"Content-Type": "text/plain", "Content-Length": "7"} { + if err := h.automatic(name, value); err != nil { + t.Fatal(err) + } + if _, present := h.fields[name]; present { + t.Errorf("automatic field %s resurrected Connection nomination", name) + } + } + if fields.Get("Custom") != "secret" || len(fields["Content-Length"]) != 2 { + t.Error("filtering mutated original headers") + } +} + +func TestResponseHeaderBudgetAndAutomaticFields(t *testing.T) { + for _, fields := range []http.Header{{"A": nil}, {"A": {}}, {"A": {""}}} { + if h, err := snapshotResponseHeaders(fields, 32); h != nil || !errors.Is(err, errResponseHeaderBudget) { + t.Errorf("snapshot(%v, 32) = %v, %v; want budget failure", fields, h, err) + } + h := mustSnapshot(t, fields, 33) + if h.remaining != 0 { + t.Errorf("snapshot(%v, 33) has %d uncharged bytes", fields, h.remaining) + } + } + for _, fields := range []http.Header{{"A": {"", ""}}, {"A": {" x "}}, {"Connection": {"close"}}, {"Set-Cookie": {"a=1"}}} { + charge := 0 + for name, values := range fields { + for _, value := range values { + charge += len(name) + len(value) + 32 + } + } + if _, err := snapshotResponseHeaders(fields, charge-1); !errors.Is(err, errResponseHeaderBudget) { + t.Errorf("snapshot(%v, %d) = %v; want rejection before trimming/removal", fields, charge-1, err) + } + h := mustSnapshot(t, fields, charge) + if h.remaining != 0 { + t.Errorf("snapshot refunded %d original bytes", h.remaining) + } + } + const typeCost = len("Content-Type") + len("text/plain") + 32 + h := mustSnapshot(t, nil, typeCost) + if err := h.automatic("Content-Type", "text/plain"); err != nil || h.remaining != 0 { + t.Fatalf("automatic type = %v, remaining %d; want exact fit", err, h.remaining) + } + if err := h.automatic("Content-Length", "0"); !errors.Is(err, errResponseHeaderBudget) { + t.Errorf("automatic length = %v; want budget failure", err) + } + if _, present := h.fields["Content-Length"]; present || h.remaining != 0 { + t.Error("failed automatic addition mutated snapshot") + } + for _, values := range [][]string{nil, {}, {""}, {"application/json"}} { + h := mustSnapshot(t, http.Header{"Content-Type": values}, defaultResponseHeaderBudget) + before := h.remaining + if err := h.automatic("Content-Type", "text/plain"); err != nil || h.remaining != before || !slices.Equal(h.fields["Content-Type"], values) { + t.Errorf("automatic type replaced or charged existing value %v", values) + } + } + tooMany := http.Header{"A": make([]string, defaultResponseHeaderBudget/33+1)} + if allocations := testing.AllocsPerRun(10, func() { + _, _ = snapshotResponseHeaders(tooMany, defaultResponseHeaderBudget) + }); allocations != 0 { + t.Errorf("over-budget preflight allocated %g objects; want none", allocations) + } +} + +func TestResponseContentLength(t *testing.T) { + for _, values := range [][]string{{""}, {"+1"}, {"-1"}, {"1,1"}, {"1", "1"}, {"1 2"}, {"9223372036854775808"}} { + if h, err := snapshotResponseHeaders(http.Header{"Content-Length": values}, defaultResponseHeaderBudget); h != nil || !errors.Is(err, errResponseHeaders) { + t.Errorf("Content-Length %v = %v, %v; want invalid response headers", values, h, err) + } + } + for _, tt := range []struct { + values []string + want int64 + set bool + }{ + {values: nil}, {values: []string{}}, + {values: []string{"0"}, set: true}, + {values: []string{" \t00012 "}, want: 12, set: true}, + {values: []string{"9223372036854775807"}, want: 9223372036854775807, set: true}, + } { + h := mustSnapshot(t, http.Header{"Content-Length": tt.values}, defaultResponseHeaderBudget) + n, set, err := responseContentLength(h.fields) + if err != nil || n != tt.want || set != tt.set { + t.Errorf("Content-Length %v = %d, %t, %v; want %d, %t", tt.values, n, set, err, tt.want, tt.set) + } + } +} + +func TestResponseHeaderV2Projection(t *testing.T) { + t.Run("suppressed_only", func(t *testing.T) { + h := mustSnapshot(t, http.Header{"Content-Type": nil, "Content-Length": {}}, defaultResponseHeaderBudget) + v2, cookies, err := h.v2() + if len(h.v1()) != 0 || len(v2) != 0 || len(cookies) != 0 || err != nil { + t.Errorf("suppressed fields produced V1/V2 output: %v, %v, %v", v2, cookies, err) + } + }) + // Each audited name has a fixture containing complete values. This tests + // conversion and ordering, not a second implementation of each field grammar. + for _, name := range []string{ + "Accept-Ranges", "Allow", "Content-Encoding", "Content-Language", "Vary", "WWW-Authenticate", "Cache-Control", + "Access-Control-Allow-Headers", "Access-Control-Allow-Methods", "Access-Control-Expose-Headers", + "Link", "Content-Security-Policy", "Content-Security-Policy-Report-Only", "Referrer-Policy", "Server-Timing", "Accept-Patch", "Accept-CH", + "Cache-Status", "Proxy-Status", "Content-Digest", "Repr-Digest", "Want-Content-Digest", "Want-Repr-Digest", "Signature", "Signature-Input", + } { + t.Run(name, func(t *testing.T) { + h := mustSnapshot(t, http.Header{name: {"first", "second"}}, defaultResponseHeaderBudget) + fields, _, err := h.v2() + if err != nil || fields[http.CanonicalHeaderKey(name)] != "first, second" { + t.Errorf("V2 %s = %v, %v; want ordered combination", name, fields, err) + } + }) + } + for _, name := range []string{"Content-Type", "Location", "ETag", "Date", "Access-Control-Allow-Origin", "Custom"} { + t.Run("reject_"+name, func(t *testing.T) { + h := mustSnapshot(t, http.Header{name: {"same", "same"}}, defaultResponseHeaderBudget) + if fields, cookies, err := h.v2(); fields != nil || cookies != nil || !errors.Is(err, errResponseHeaderRepeat) { + t.Errorf("V2 repeated %s = %v, %v, %v; want no partial response", name, fields, cookies, err) + } + if values := h.v1()[http.CanonicalHeaderKey(name)]; !slices.Equal(values, []string{"same", "same"}) { + t.Errorf("V1 lost repeated %s: %v", name, values) + } + }) + } + h := mustSnapshot(t, http.Header{ + "Set-Cookie": {"a=1; Expires=Wed, 21 Oct 2030 07:28:00 GMT", "b=2"}, + "WWW-Authenticate": {`Digest realm="a,b", nonce="x"`, `Basic realm="other"`}, + "Signature-Input": {`sig1=("@method" "@path");created=1`, `sig2=("content-digest");created=2`}, + "Custom": {"literal, comma"}, "Empty": {""}, "Suppressed": nil, + }, defaultResponseHeaderBudget) + fields, cookies, err := h.v2() + if err != nil { + t.Fatal(err) + } + if fields["Www-Authenticate"] != `Digest realm="a,b", nonce="x", Basic realm="other"` || fields["Signature-Input"] != `sig1=("@method" "@path");created=1, sig2=("content-digest");created=2` { + t.Errorf("V2 altered challenge/dictionary values: %v", fields) + } + if _, present := fields["Set-Cookie"]; present || !slices.Equal(cookies, h.fields["Set-Cookie"]) { + t.Errorf("V2 cookies = %v, headers %v; want separate values", cookies, fields) + } + if value, present := fields["Empty"]; !present || value != "" || fields["Custom"] != "literal, comma" { + t.Errorf("V2 changed single or empty values: %v", fields) + } + if _, present := fields["Suppressed"]; present { + t.Error("V2 emitted suppression marker") + } + cookies[0] = "changed" + if strings.HasPrefix(h.fields["Set-Cookie"][0], "changed") { + t.Error("V2 cookies alias snapshot slice") + } +} + +func TestResponseHeaderBudgetOption(t *testing.T) { + for _, tt := range []struct { + name string + opts []Option + want int + }{ + {name: "default", want: defaultResponseHeaderBudget}, + {name: "minimum", opts: []Option{WithResponseHeaderBudget(1)}, want: 1}, + {name: "maximum", opts: []Option{WithResponseHeaderBudget(maxResponseBytes)}, want: maxResponseBytes}, + {name: "last_assignment", opts: []Option{WithResponseHeaderBudget(0), WithResponseHeaderBudget(100)}, want: 100}, + } { + t.Run(tt.name, func(t *testing.T) { + a, err := New(http.NewServeMux(), tt.opts...) + if err != nil || a == nil { + t.Fatalf("New = %v, %v; want valid adapter", a, err) + } + if a.config.responseHeaderBudget != tt.want { + t.Errorf("configured budget = %d; want %d", a.config.responseHeaderBudget, tt.want) + } + }) + } + for _, budget := range []int{-1, 0, maxResponseBytes + 1, int(^uint(0) >> 1)} { + t.Run(fmt.Sprint(budget), func(t *testing.T) { + a, err := New(http.NewServeMux(), WithResponseHeaderBudget(100), WithResponseHeaderBudget(budget)) + if a != nil || err == nil { + t.Errorf("New with budget %d = %v, %v; want nil and configuration error", budget, a, err) + } + }) + } +} diff --git a/stream_encoding.go b/stream_encoding.go new file mode 100644 index 0000000..c6009f8 --- /dev/null +++ b/stream_encoding.go @@ -0,0 +1,33 @@ +package edge + +import ( + "encoding/json/v2" + "errors" +) + +// Edge's conservative interpretation includes the entire eight-NUL delimiter +// within AWS's documented first 16 KB. This is not a stream body limit. +const maxStreamPrefixBytes = 16000 + +var errStreamPrefixTooLarge = errors.New("edge: response metadata exceeds streaming prefix limit") + +func encodeStreamPrefix(status int, headers *responseHeaders) ([]byte, error) { + if status < 200 || status > 599 { + return nil, errors.New("edge: invalid final response status") + } + metadata := struct { + StatusCode int `json:"statusCode"` + MultiValueHeaders map[string][]string `json:"multiValueHeaders,omitzero"` + }{StatusCode: status, MultiValueHeaders: headers.v1()} + out := responseBuffer{limit: maxStreamPrefixBytes - 8} + if err := json.MarshalWrite(&out, metadata); err != nil { + if errors.Is(err, errResponseTooLarge) { + return nil, errStreamPrefixTooLarge + } + return nil, errors.New("edge: streaming metadata JSON encoding failed") + } + // Allocate exactly the framed length. The final zeroed bytes are the delimiter. + prefix := make([]byte, len(out.data)+8) + copy(prefix, out.data) + return prefix, nil +} diff --git a/stream_encoding_test.go b/stream_encoding_test.go new file mode 100644 index 0000000..8647d51 --- /dev/null +++ b/stream_encoding_test.go @@ -0,0 +1,121 @@ +package edge + +import ( + "bytes" + "context" + "encoding/json/v2" + "errors" + "io" + "net/http" + "slices" + "strings" + "testing" +) + +func TestStreamPrefixMetadata(t *testing.T) { + h := mustSnapshot(t, http.Header{ + "Custom": {"first", "second"}, "Set-Cookie": {"a=1", "b=2"}, + "Quoted": {"a\t\"b"}, "Suppressed": nil, + }, defaultResponseHeaderBudget) + prefix, err := encodeStreamPrefix(201, h) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(prefix[len(prefix)-8:], make([]byte, 8)) || bytes.ContainsRune(prefix[:len(prefix)-8], 0) { + t.Fatalf("stream prefix %q has invalid delimiter placement", prefix) + } + var metadata struct { + StatusCode int `json:"statusCode"` + MultiValueHeaders map[string][]string `json:"multiValueHeaders"` + Headers map[string]string `json:"headers"` + Cookies []string `json:"cookies"` + } + if err := json.Unmarshal(prefix[:len(prefix)-8], &metadata); err != nil { + t.Fatal(err) + } + if metadata.StatusCode != 201 || metadata.Headers != nil || metadata.Cookies != nil { + t.Errorf("metadata = %+v; want status 201 and only multivalue headers", metadata) + } + for _, name := range []string{"Custom", "Set-Cookie", "Quoted"} { + if !slices.Equal(metadata.MultiValueHeaders[name], h.fields[name]) { + t.Errorf("stream field %s = %v; want %v", name, metadata.MultiValueHeaders[name], h.fields[name]) + } + } + if _, present := metadata.MultiValueHeaders["Suppressed"]; present { + t.Error("stream metadata emitted suppression marker") + } + if cap(prefix) != len(prefix) { + t.Errorf("prefix retains capacity %d for length %d", cap(prefix), len(prefix)) + } +} + +func TestStreamPrefixExactLimit(t *testing.T) { + base, err := encodeStreamPrefix(200, mustSnapshot(t, http.Header{"A": {""}}, defaultResponseHeaderBudget)) + if err != nil { + t.Fatal(err) + } + for _, delta := range []int{-1, 0, 1} { + value := strings.Repeat("x", maxStreamPrefixBytes-len(base)+delta) + h := mustSnapshot(t, http.Header{"A": {value}}, defaultResponseHeaderBudget) + prefix, err := encodeStreamPrefix(200, h) + if delta > 0 { + if prefix != nil || !errors.Is(err, errStreamPrefixTooLarge) { + t.Errorf("over-limit prefix = %d bytes, %v; want nil and size error", len(prefix), err) + } + continue + } + if err != nil || len(prefix) != maxStreamPrefixBytes+delta { + t.Errorf("prefix with delta %d = %d bytes, %v; want %d bytes", delta, len(prefix), err, maxStreamPrefixBytes+delta) + } + } + for _, value := range []string{strings.Repeat(`"`, 8000), "a" + strings.Repeat("\t", 8000) + "b"} { + h := mustSnapshot(t, http.Header{"A": {value}}, defaultResponseHeaderBudget) + prefix, err := encodeStreamPrefix(200, h) + if prefix != nil || !errors.Is(err, errStreamPrefixTooLarge) { + t.Errorf("escaping prefix = %d bytes, %v; want encoded-size failure", len(prefix), err) + } + } + for _, status := range []int{0, 101, 199, 600, 999} { + prefix, err := encodeStreamPrefix(status, mustSnapshot(t, nil, defaultResponseHeaderBudget)) + if prefix != nil || err == nil { + t.Errorf("prefix status %d = %q, %v; want rejection", status, prefix, err) + } + } +} + +func TestStreamPrefixBridgePublication(t *testing.T) { + for _, large := range []bool{false, true} { + value := "value" + if large { + value = strings.Repeat("x", maxStreamPrefixBytes) + } + h := mustSnapshot(t, http.Header{"Custom": {value}}, defaultResponseHeaderBudget) + wroteBody := false + s, err := startStream(t.Context(), prepareEmptyStream, func(ctx context.Context, out *streamOutput) error { + prefix, err := encodeStreamPrefix(200, h) + if err != nil { + return err + } + if err := out.publish(prefix); err != nil { + return err + } + _, err = io.WriteString(out.body, "body") + wroteBody = true + return err + }, nil) + if large { + if s != nil || !errors.Is(err, errStreamPrefixTooLarge) || wroteBody { + t.Errorf("oversize handoff = %v, %v, body=%t; want no publication", s, err, wroteBody) + } + continue + } + if err != nil { + t.Fatal(err) + } + data, err := io.ReadAll(s) + closeErr := s.Close() + if err != nil || closeErr != nil || !wroteBody || !bytes.HasSuffix(data, append(make([]byte, 8), []byte("body")...)) { + t.Errorf("framed stream = %q, %v; Close=%v, wroteBody=%t", data, err, closeErr, wroteBody) + } + } +} From b7e86378914b7c1a93b95c1aa863c5561ae4b1cf Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 14:39:48 +0100 Subject: [PATCH 19/44] feat: implement buffered HTTP response writer and finalization --- README.md | 13 +- buffered.go | 182 ++++++++++ buffered_integration_test.go | 215 ++++++++++++ buffered_test.go | 312 ++++++++++++++++++ docs/decisions/0006-buffered-responses.md | 48 ++- docs/decisions/0010-response-header-design.md | 14 +- docs/plan.md | 14 +- 7 files changed, 780 insertions(+), 18 deletions(-) create mode 100644 buffered.go create mode 100644 buffered_integration_test.go create mode 100644 buffered_test.go diff --git a/README.md b/README.md index 0770bd0..cb90ccc 100644 --- a/README.md +++ b/README.md @@ -11,10 +11,11 @@ authorizer JSON separately. Request conversion covers URLs, headers, body bytes, and request lifetime. A shared invocation scope now covers cleanup and cancellation; private response encoding provides text/base64 selection and bounded direct JSON v2 output. Private response-header snapshots and V1/V2 projections now enforce -the accepted syntax, suppression, cookie and field-combination rules. The -buffered HTTP writer, public invocation methods, and identity extraction are not -implemented yet; the adapter cannot yet -be registered as a Lambda handler. +the accepted syntax, suppression, cookie and field-combination rules. The private +buffered HTTP writer now implements commitment, sniffing, bodyless/HEAD responses, +length enforcement and typed AWS response projection. Public invocation methods +and identity extraction are not implemented yet; the adapter cannot yet be +registered as a Lambda handler. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle @@ -51,8 +52,8 @@ value costs `len(name) + len(value) + 32`; nil/empty slices still cost a name pl 32, and generated headers consume remaining budget. This is separate from the complete buffered-envelope limit and the 16,000-byte streaming metadata prefix (including its delimiter). It is neither an AWS quota nor a total heap bound. -The option validates configuration now; the HTTP writers will apply it at -commitment when invocation is wired. +The option validates configuration now, and the private buffered writer applies +it at commitment. Public invocation and the streaming HTTP writer remain pending. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) diff --git a/buffered.go b/buffered.go new file mode 100644 index 0000000..6705d88 --- /dev/null +++ b/buffered.go @@ -0,0 +1,182 @@ +package edge + +import ( + "errors" + "fmt" + "math" + "net/http" + "strconv" + "strings" + + "github.com/aws/aws-lambda-go/events" +) + +var errInformationalResponse = fmt.Errorf("edge: informational response: %w", http.ErrNotSupported) + +// bufferedWriter belongs to one handler invocation. Only Header, WriteHeader +// and Write form its HTTP surface; it deliberately exposes no network or +// streaming capabilities. The owning call must finish it after ServeHTTP returns. +type bufferedWriter struct { + header http.Header + headers *responseHeaders + headerBudget int + method string + status int + body responseBuffer + written int64 + declaredLength int64 + hasLength bool + err error +} + +func newBufferedWriter(method string, headerBudget int) *bufferedWriter { + w := &bufferedWriter{header: make(http.Header), method: method, headerBudget: headerBudget} + if method == http.MethodHead { + w.body.limit = 512 + } + return w +} + +func (w *bufferedWriter) Header() http.Header { return w.header } + +func (w *bufferedWriter) WriteHeader(status int) { + // Like net/http, ignore every subsequent status after final commitment. + // A fault on the initial attempt cannot be repaired by a later status. + if w.status != 0 || w.err != nil { + return + } + if status >= 100 && status <= 199 { + w.err = errInformationalResponse + return + } + if status < 200 || status > 599 { + w.err = errors.New("edge: invalid final response status") + return + } + w.status = status + w.headers, w.err = snapshotResponseHeaders(w.header, w.headerBudget) + if w.err != nil { + return + } + if !bufferedBodyAllowed(status) { + w.headers.remove("Content-Length") + if status == http.StatusNotModified { + w.headers.remove("Content-Type") + } + } + w.declaredLength, w.hasLength, w.err = responseContentLength(w.headers.fields) +} + +func (w *bufferedWriter) Write(p []byte) (int, error) { + if w.status == 0 { + w.WriteHeader(http.StatusOK) + } + if w.err != nil { + return 0, w.err + } + if len(p) == 0 { + return 0, nil + } + if !bufferedBodyAllowed(w.status) { + return 0, http.ErrBodyNotAllowed + } + if w.hasLength && int64(len(p)) > w.declaredLength-w.written { + w.err = http.ErrContentLength + return 0, w.err + } + if int64(len(p)) > math.MaxInt64-w.written { + w.err = errResponseTooLarge + return 0, w.err + } + data := p + if w.method == http.MethodHead { + data = p[:min(len(p), 512-len(w.body.data))] + } + if _, err := w.body.Write(data); err != nil { + w.err = err + return 0, err + } + w.written += int64(len(p)) + return len(p), nil +} + +func bufferedBodyAllowed(status int) bool { + return status != http.StatusNoContent && status != http.StatusResetContent && status != http.StatusNotModified +} + +// finish freezes transport output without exposing the mutable application +// header map. Errors discard the response, even if the handler ignored Write's +// failure. Header-map changes after commitment are ignored except for trailers, +// which net/http would otherwise send after the committed response. +func (w *bufferedWriter) finish() (*bufferedResponse, error) { + if w.status == 0 { + w.WriteHeader(http.StatusOK) + } + if w.err != nil { + return nil, w.err + } + for name, values := range w.header { + declared := len(name) == len("Trailer") && strings.EqualFold(name, "Trailer") && len(values) != 0 + if strings.HasPrefix(name, http.TrailerPrefix) || declared { + return nil, errResponseTrailers + } + } + if w.method != http.MethodHead && w.hasLength && w.written != w.declaredLength { + return nil, fmt.Errorf("edge: response body length mismatch: %w", http.ErrContentLength) + } + if bufferedBodyAllowed(w.status) { + if len(w.body.data) > 0 && len(w.headers.fields["Content-Encoding"]) == 0 { + kind := http.DetectContentType(w.body.data[:min(len(w.body.data), 512)]) + if err := w.headers.automatic("Content-Type", kind); err != nil { + return nil, err + } + } + if w.method != http.MethodHead || w.written > 0 { + if err := w.headers.automatic("Content-Length", strconv.FormatInt(w.written, 10)); err != nil { + return nil, err + } + } + } + body := w.body.data + if w.method == http.MethodHead { + body = nil + } + return &bufferedResponse{status: w.status, headers: w.headers, body: body}, nil +} + +// bufferedResponse owns the finalized response. The writer is no longer usable. +// Projections retain the JSON-free typed boundary; marshalResponse is separate. +type bufferedResponse struct { + status int + headers *responseHeaders + body []byte +} + +func (r *bufferedResponse) v1() (events.APIGatewayProxyResponse, error) { + body, binary, err := encodeResponseBody(r.body, r.headers.fields) + if err != nil { + return events.APIGatewayProxyResponse{}, err + } + return events.APIGatewayProxyResponse{StatusCode: r.status, MultiValueHeaders: r.headers.v1(), Body: body, IsBase64Encoded: binary}, nil +} + +func (r *bufferedResponse) v2() (events.APIGatewayV2HTTPResponse, error) { + headers, cookies, err := r.headers.v2() + if err != nil { + return events.APIGatewayV2HTTPResponse{}, err + } + body, binary, err := encodeResponseBody(r.body, r.headers.fields) + if err != nil { + return events.APIGatewayV2HTTPResponse{}, err + } + return events.APIGatewayV2HTTPResponse{StatusCode: r.status, Headers: headers, Cookies: cookies, Body: body, IsBase64Encoded: binary}, nil +} + +// serveBuffered runs inside the caller's invocation scope. Capturing the method +// first prevents a handler's Request mutation from changing HEAD semantics. +// Panics propagate to that scope so cleanup precedes runtime panic handling. +func serveBuffered(handler http.Handler, r *http.Request, headerBudget int) (*bufferedResponse, error) { + w := newBufferedWriter(r.Method, headerBudget) + handler.ServeHTTP(w, r) + return w.finish() +} diff --git a/buffered_integration_test.go b/buffered_integration_test.go new file mode 100644 index 0000000..fc1f5d9 --- /dev/null +++ b/buffered_integration_test.go @@ -0,0 +1,215 @@ +package edge + +import ( + "bytes" + "compress/gzip" + "context" + "encoding/base64" + "errors" + "io" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "testing" + "time" +) + +func TestBufferedRequestIntegration(t *testing.T) { + mux := http.NewServeMux() + mux.HandleFunc("GET /content", func(w http.ResponseWriter, r *http.Request) { + http.ServeContent(w, r, "content.txt", time.Unix(1000, 0), strings.NewReader("0123456789")) + }) + mux.HandleFunc("GET /compressed", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain") + w.Header().Set("Content-Encoding", "gzip") + z := gzip.NewWriter(w) + io.WriteString(z, "compressed response") + z.Close() + }) + mux.HandleFunc("POST /echo/{id}", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Custom", r.Header.Get("Custom")) + w.Header().Set("Selected-Id", r.PathValue("id")) + io.Copy(w, r.Body) + }) + for _, format := range []string{"v1", "v2"} { + for _, path := range []string{"/content", "/compressed", "/echo/example"} { + t.Run(format+"_"+strings.ReplaceAll(path, "/", "_"), func(t *testing.T) { + t.Parallel() + var response *bufferedResponse + var scope context.Context + var body *trackedBody + err := withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + scope = ctx + v1, v2 := v1RequestEvent(), v2RequestEvent() + v1.Path, v2.RawPath = path, path + v1.Headers = map[string]string{"Custom": "request-specific", "Range": "bytes=2-5"} + v2.Headers = v1.Headers + if path == "/echo/example" { + v1.HTTPMethod = http.MethodPost + v2.RequestContext.HTTP.Method = http.MethodPost + v1.Body, v2.Body = "echo body", "echo body" + } + var r *http.Request + var err error + if format == "v1" { + r, err = requestV1(ctx, v1) + } else { + r, err = requestV2(ctx, v2) + } + if err != nil { + return err + } + body = &trackedBody{Reader: r.Body} + r.Body = body + inv.ownRequest(r) + response, err = serveBuffered(mux, r, defaultResponseHeaderBudget) + return err + }) + if err != nil { + t.Fatal(err) + } + if !body.closed || scope.Err() != context.Canceled { + t.Errorf("buffered invocation did not clean up: closed=%t context=%v", body.closed, scope.Err()) + } + var wireBody string + var binary bool + if format == "v1" { + out, err := response.v1() + if err != nil { + t.Fatal(err) + } + wireBody, binary = out.Body, out.IsBase64Encoded + } else { + out, err := response.v2() + if err != nil { + t.Fatal(err) + } + wireBody, binary = out.Body, out.IsBase64Encoded + } + decoded := []byte(wireBody) + if binary { + decoded, err = base64.StdEncoding.DecodeString(wireBody) + if err != nil { + t.Fatal(err) + } + } + switch path { + case "/content": + if response.status != 206 || string(decoded) != "2345" || response.headers.fields.Get("Content-Range") != "bytes 2-5/10" { + t.Errorf("range response = %d %q, headers %v", response.status, decoded, response.headers.fields) + } + case "/compressed": + if !binary || response.headers.fields.Get("Content-Encoding") != "gzip" { + t.Fatal("compressed response lost binary/encoding metadata") + } + z, err := gzip.NewReader(bytes.NewReader(decoded)) + if err != nil { + t.Fatal(err) + } + plain, err := io.ReadAll(z) + z.Close() + if err != nil || string(plain) != "compressed response" { + t.Errorf("decompressed response = %q, %v", plain, err) + } + case "/echo/example": + if string(decoded) != "echo body" || response.headers.fields.Get("Custom") != "request-specific" || response.headers.fields.Get("Selected-Id") != "example" { + t.Errorf("mux/echo response = %q, headers %v", decoded, response.headers.fields) + } + } + }) + } + } +} + +func TestBufferedInvocationFailureCleanup(t *testing.T) { + for _, scenario := range []string{"write_failure", "panic", "parent_cancel", "cleanup_failure"} { + t.Run(scenario, func(t *testing.T) { + parent, cancel := context.WithCancel(t.Context()) + defer cancel() + body := &trackedBody{Reader: http.NoBody} + failedBody := &failingCloseBody{Reader: http.NoBody} + var scope context.Context + var recovered any + var err error + func() { + defer func() { recovered = recover() }() + err = withInvocation(parent, func(ctx context.Context, inv *invocation) error { + scope = ctx + r := httptest.NewRequestWithContext(ctx, http.MethodGet, "/", nil) + r.Body = body + if scenario == "cleanup_failure" { + r.Body = failedBody + } + inv.ownRequest(r) + _, err := serveBuffered(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch scenario { + case "write_failure": + w.Header().Set("Content-Length", "1") + io.WriteString(w, "overflow") + case "panic": + panic("handler panic") + case "parent_cancel": + cancel() + default: + io.WriteString(w, "success before failed cleanup") + } + }), r, defaultResponseHeaderBudget) + return err + }) + }() + if !body.closed && !failedBody.closed || scope.Err() != context.Canceled { + t.Errorf("failure did not clean up body/context: %v, %v", body.closed || failedBody.closed, scope.Err()) + } + switch scenario { + case "panic": + if recovered != "handler panic" { + t.Errorf("recovered %v; want original handler panic", recovered) + } + case "write_failure": + if !errors.Is(err, http.ErrContentLength) { + t.Errorf("ignored Write failure returned %v", err) + } + case "parent_cancel": + if !errors.Is(err, context.Canceled) { + t.Errorf("canceled invocation returned %v", err) + } + case "cleanup_failure": + if err == nil || strings.Contains(err.Error(), "credential") { + t.Errorf("cleanup failure returned %v; want sanitized error", err) + } + } + }) + } +} + +func TestBufferedConfiguredBudgetAndSuppression(t *testing.T) { + for _, budget := range []int{1, defaultResponseHeaderBudget} { + t.Run(strconv.Itoa(budget), func(t *testing.T) { + a, err := New(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}), WithResponseHeaderBudget(budget)) + if err != nil { + t.Fatal(err) + } + result, err := serveBuffered(a.handler, httptest.NewRequest(http.MethodGet, "/", nil), a.config.responseHeaderBudget) + if budget == 1 { + if result != nil || !errors.Is(err, errResponseHeaderBudget) { + t.Errorf("tiny budget = %v, %v; want generated-length failure", result, err) + } + } else if err != nil || result == nil { + t.Errorf("default budget = %v, %v; want success", result, err) + } + }) + } + for _, field := range []string{"Content-Type", "Content-Length"} { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Set("Connection", field) + w.Write([]byte("body")) + result, err := w.finish() + if err != nil { + t.Fatal(err) + } + if _, present := result.headers.fields[field]; present { + t.Errorf("writer resurrected nominated field %s", field) + } + } +} diff --git a/buffered_test.go b/buffered_test.go new file mode 100644 index 0000000..29a6027 --- /dev/null +++ b/buffered_test.go @@ -0,0 +1,312 @@ +package edge + +import ( + "bytes" + "encoding/base64" + "errors" + "io" + "log" + "net/http" + "net/http/httptest" + "slices" + "strconv" + "strings" + "testing" + "time" +) + +func TestBufferedMatchesHTTP(t *testing.T) { + for _, tt := range []struct { + name string + method string + handler http.HandlerFunc + }{ + {name: "untouched", handler: func(http.ResponseWriter, *http.Request) {}}, + {name: "empty_write", handler: func(w http.ResponseWriter, _ *http.Request) { w.Write(nil) }}, + {name: "commitment", handler: func(w http.ResponseWriter, _ *http.Request) { + values := []string{"before"} + w.Header()["Custom"] = values + w.WriteHeader(http.StatusCreated) + values[0] = "after" + w.Header().Set("Content-Type", "changed/after-commit") + w.WriteHeader(http.StatusAccepted) + io.WriteString(w, "body") + }}, + {name: "split_sniff", handler: func(w http.ResponseWriter, _ *http.Request) { + io.WriteString(w, "<") + io.WriteString(w, "html>hello") + }}, + {name: "suppressed_type", handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + io.WriteString(w, "body") + }}, + {name: "suppressed_length", handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Length"] = nil + io.WriteString(w, "body") + }}, + {name: "declared_length", handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Length", "4") + io.WriteString(w, "body") + }}, + {name: "encoded_no_sniff", handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Encoding", "identity") + io.WriteString(w, "body") + }}, + {name: "application_error", handler: func(w http.ResponseWriter, r *http.Request) { http.Error(w, "denied", http.StatusForbidden) }}, + {name: "redirect", handler: func(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/next", http.StatusFound) }}, + {name: "head_empty", method: http.MethodHead, handler: func(http.ResponseWriter, *http.Request) {}}, + {name: "head_representation", method: http.MethodHead, handler: func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "representation") }}, + {name: "head_declared", method: http.MethodHead, handler: func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Length", "500") }}, + {name: "no_content", handler: func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Length", "99") + w.WriteHeader(http.StatusNoContent) + }}, + {name: "not_modified", handler: func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.Header().Set("Content-Length", "99") + w.WriteHeader(http.StatusNotModified) + }}, + } { + t.Run(tt.name, func(t *testing.T) { + method := tt.method + if method == "" { + method = http.MethodGet + } + r := httptest.NewRequestWithContext(t.Context(), method, "http://example.com/", nil) + got, err := serveBuffered(tt.handler, r, defaultResponseHeaderBudget) + if err != nil { + t.Fatal(err) + } + server := httptest.NewTestServer(t, tt.handler) + server.Config.ErrorLog = log.New(io.Discard, "", 0) + client := server.Client() + client.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse } + request, err := http.NewRequestWithContext(t.Context(), method, "http://example.com/", nil) + if err != nil { + t.Fatal(err) + } + request.Header.Set("Accept-Encoding", "identity") + response, err := client.Do(request) + if err != nil { + t.Fatal(err) + } + defer response.Body.Close() + body, err := io.ReadAll(response.Body) + if err != nil { + t.Fatal(err) + } + if got.status != response.StatusCode || !bytes.Equal(got.body, body) { + t.Errorf("buffered response = %d %q; HTTP = %d %q", got.status, got.body, response.StatusCode, body) + } + for _, name := range []string{"Content-Type", "Content-Length", "Content-Encoding", "Custom", "Location", "X-Content-Type-Options"} { + if !slices.Equal(got.headers.fields.Values(name), response.Header.Values(name)) { + t.Errorf("buffered %s = %v; HTTP = %v", name, got.headers.fields.Values(name), response.Header.Values(name)) + } + } + if _, present := got.headers.fields["Date"]; present { + t.Error("buffered writer generated Date") + } + }) + } +} + +func TestBufferedBodylessStatuses(t *testing.T) { + for _, status := range []int{204, 205, 304} { + t.Run(strconv.Itoa(status), func(t *testing.T) { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Set("Content-Length", "100") + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + if n, err := w.Write([]byte("forbidden")); n != 0 || !errors.Is(err, http.ErrBodyNotAllowed) { + t.Errorf("bodyless Write = %d, %v; want ErrBodyNotAllowed", n, err) + } + if n, err := w.Write(nil); n != 0 || err != nil { + t.Errorf("bodyless empty Write = %d, %v; want 0, nil", n, err) + } + result, err := w.finish() + if err != nil || result == nil { + t.Fatalf("finish = %v, %v; body rejection alone must not fail response", result, err) + } + if len(result.body) != 0 || result.headers.fields.Get("Content-Length") != "" { + t.Errorf("bodyless response contains body or length: %+v", result) + } + wantType := "application/json" + if status == 304 { + wantType = "" + } + if got := result.headers.fields.Get("Content-Type"); got != wantType { + t.Errorf("Content-Type = %q; want %q", got, wantType) + } + }) + } +} + +func TestBufferedStickyFaults(t *testing.T) { + for _, tt := range []struct { + name string + setup func(*bufferedWriter) + want error + }{ + {name: "length_overflow", want: http.ErrContentLength, setup: func(w *bufferedWriter) { + w.Header().Set("Content-Length", "2") + w.Write([]byte("too long")) + w.Header().Set("Content-Length", "100") + w.WriteHeader(200) + }}, + {name: "header_budget", want: errResponseHeaderBudget, setup: func(w *bufferedWriter) { w.Header().Set("Huge", strings.Repeat("x", defaultResponseHeaderBudget)) }}, + {name: "invalid_header", want: errResponseHeaders, setup: func(w *bufferedWriter) { w.Header().Set("Custom", "bad\nvalue") }}, + {name: "information", want: http.ErrNotSupported, setup: func(w *bufferedWriter) { w.WriteHeader(103); w.WriteHeader(200) }}, + {name: "upgrade", want: http.ErrNotSupported, setup: func(w *bufferedWriter) { w.WriteHeader(101) }}, + } { + t.Run(tt.name, func(t *testing.T) { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + tt.setup(w) + if n, err := w.Write([]byte("later")); n != 0 || !errors.Is(err, tt.want) { + t.Errorf("Write after fault = %d, %v; want 0, %v", n, err, tt.want) + } + if result, err := w.finish(); result != nil || !errors.Is(err, tt.want) { + t.Errorf("finish after ignored error = %v, %v; want nil, %v", result, err, tt.want) + } + }) + } + for _, status := range []int{-1, 0, 99, 600, 999} { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.WriteHeader(status) + w.WriteHeader(200) + if result, err := w.finish(); result != nil || err == nil { + t.Errorf("invalid initial status %d = %v, %v; want rejection", status, result, err) + } + } + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.WriteHeader(201) + for _, status := range []int{202, 103, 101, 0, 999} { + w.WriteHeader(status) + } + if result, err := w.finish(); err != nil || result.status != 201 { + t.Errorf("superfluous statuses changed commitment: %v, %v", result, err) + } +} + +func TestBufferedLengthMismatchAndHead(t *testing.T) { + for _, method := range []string{http.MethodGet, http.MethodHead} { + w := newBufferedWriter(method, defaultResponseHeaderBudget) + w.Header().Set("Content-Length", "100") + w.Write([]byte("small")) + result, err := w.finish() + if method == http.MethodGet { + if result != nil || !errors.Is(err, http.ErrContentLength) { + t.Errorf("GET short body = %v, %v; want mismatch error", result, err) + } + } else if err != nil || result == nil || len(result.body) != 0 || result.headers.fields.Get("Content-Length") != "100" { + t.Errorf("HEAD short representation = %v, %v; want declared length without body", result, err) + } + } + w := newBufferedWriter(http.MethodHead, defaultResponseHeaderBudget) + data := bytes.Repeat([]byte("x"), maxResponseBytes+1) + if n, err := w.Write(data); err != nil || n != len(data) { + t.Fatalf("HEAD large Write = %d, %v; want count without storing body", n, err) + } + if len(w.body.data) != 512 || cap(w.body.data) != 512 { + t.Errorf("HEAD retained length/capacity = %d/%d; want 512/512", len(w.body.data), cap(w.body.data)) + } + result, err := w.finish() + if err != nil || len(result.body) != 0 || result.headers.fields.Get("Content-Length") != strconv.Itoa(len(data)) { + t.Errorf("HEAD large finish = %v, %v; want counted length and no body", result, err) + } + r := httptest.NewRequest(http.MethodHead, "/", nil) + result, err = serveBuffered(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + r.Method = http.MethodGet + io.WriteString(w, "representation") + }), r, defaultResponseHeaderBudget) + if err != nil || len(result.body) != 0 { + t.Errorf("request mutation changed HEAD behavior: %v, %v", result, err) + } +} + +func TestBufferedLimitsAndEncoding(t *testing.T) { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Set("Content-Type", "text/plain") + data := bytes.Repeat([]byte("x"), maxResponseBytes) + if n, err := w.Write(data); n != len(data) || err != nil { + t.Fatalf("exact-limit Write = %d, %v", n, err) + } + result, err := w.finish() + if err != nil { + t.Fatal(err) + } + typed, err := result.v1() + if err != nil || len(typed.Body) != maxResponseBytes { + t.Fatalf("typed response = body length %d, %v; want exact body ceiling", len(typed.Body), err) + } + if raw, err := marshalResponse(typed); raw != nil || !errors.Is(err, errResponseTooLarge) { + t.Errorf("raw envelope at body ceiling = %d bytes, %v; want overhead rejection", len(raw), err) + } + w = newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + if _, err := w.Write(data); err != nil { + t.Fatal(err) + } + if n, err := w.Write([]byte("x")); n != 0 || !errors.Is(err, errResponseTooLarge) || len(w.body.data) != maxResponseBytes || cap(w.body.data) > maxResponseBytes { + t.Errorf("over-limit Write = %d, %v; retained length/cap %d/%d", n, err, len(w.body.data), cap(w.body.data)) + } + if result, err := w.finish(); result != nil || !errors.Is(err, errResponseTooLarge) { + t.Errorf("ignored body overflow = %v, %v; want no response", result, err) + } + for _, data := range [][]byte{[]byte("hello"), {0, 255, 1}} { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Add("Set-Cookie", "a=1") + w.Header().Add("Set-Cookie", "b=2") + w.Header().Add("Vary", "Origin") + w.Header().Add("Vary", "Accept-Encoding") + w.Write(data) + result, err := w.finish() + if err != nil { + t.Fatal(err) + } + v1, err := result.v1() + if err != nil { + t.Fatal(err) + } + v2, err := result.v2() + if err != nil { + t.Fatal(err) + } + if v1.Headers != nil || v2.MultiValueHeaders != nil || !slices.Equal(v1.MultiValueHeaders["Set-Cookie"], v2.Cookies) || v2.Headers["Vary"] != "Origin, Accept-Encoding" { + t.Errorf("gateway header projection differs: V1=%v V2=%v", v1, v2) + } + decoded := []byte(v2.Body) + if v2.IsBase64Encoded { + decoded, err = base64.StdEncoding.DecodeString(v2.Body) + } + if err != nil || !bytes.Equal(decoded, data) || v1.Body != v2.Body || v1.IsBase64Encoded != v2.IsBase64Encoded { + t.Errorf("gateway body changed: V1=%q V2=%q binary=%t error=%v", v1.Body, v2.Body, v2.IsBase64Encoded, err) + } + } +} + +func TestBufferedCapabilitiesAndLateTrailers(t *testing.T) { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + var handlerWriter http.ResponseWriter = w + if _, ok := handlerWriter.(http.Flusher); ok { + t.Error("buffered writer advertises flushing") + } + if _, ok := handlerWriter.(http.Hijacker); ok { + t.Error("buffered writer advertises hijacking") + } + if _, ok := handlerWriter.(http.Pusher); ok { + t.Error("buffered writer advertises pushing") + } + controller := http.NewResponseController(handlerWriter) + for _, err := range []error{controller.Flush(), controller.EnableFullDuplex(), controller.SetReadDeadline(time.Now()), controller.SetWriteDeadline(time.Now())} { + if !errors.Is(err, http.ErrNotSupported) { + t.Errorf("ResponseController capability returned %v; want ErrNotSupported", err) + } + } + for _, name := range []string{http.TrailerPrefix + "Digest", "Trailer", "trailer"} { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + io.WriteString(w, "committed") + w.Header()[name] = []string{"Digest"} + if result, err := w.finish(); result != nil || !errors.Is(err, http.ErrNotSupported) { + t.Errorf("late %s = %v, %v; want unsupported trailer fault", name, result, err) + } + } +} diff --git a/docs/decisions/0006-buffered-responses.md b/docs/decisions/0006-buffered-responses.md index b5af001..07a9516 100644 --- a/docs/decisions/0006-buffered-responses.md +++ b/docs/decisions/0006-buffered-responses.md @@ -1,6 +1,8 @@ # 0006: Buffered HTTP responses and invocation completion -Status: accepted with the 2026-09-14 review refinements; implementation underway. +Status: accepted with the 2026-09-14 review refinements. Private buffered writer, +header/body finalization and SDK response projections implemented on 2026-09-15. +Public invocation still awaits the minimum identity and exported error contracts. Reviewed against the implemented decoder/request layer on 2026-09-14. The [plan review](../reviews/2026-09-14-plan-review.md) was approved by the user and its @@ -196,3 +198,47 @@ exact envelope boundaries, metadata bounds, and sequential/concurrent isolation. Accepted by the user on 2026-09-14 with the plan-review enhancements. Implement settled behavior now; resolve the separately identified metadata, header-list, error API, and identity details before their dependent implementation. + +## Implementation and validation — 2026-09-15 + +buffered.go provides the private writer, synchronous serveBuffered call and typed +AWS response projections. It captures the original request method, snapshots +headers on the first final status, retains bounded body storage and reports +sticky write failures at completion. Automatic fields consume the remaining +metadata budget and respect explicit suppression and Connection nominations. +Typed projection does no JSON work; the separate raw encoder still accounts for +the complete envelope, including its overhead. + +HEAD counts successful representation writes while retaining at most 512 bytes. +It honors the supplied length without requiring that all representation bytes +be written, but rejects writes exceeding a supplied length, as Go's server does. +Its emitted body is empty. Statuses 204/205/304 reject nonempty writes without +turning that rejection alone into an invocation failure. Bodyless 205 remains +the accepted intentional difference from Go's behavior. + +Installed Go 1.27.1 server source was checked for response.WriteHeader, +response.write and chunkWriter.writeHeader. Once final status is committed, all +later WriteHeader calls are ignored, including invalid or informational codes. +This follows Go's superfluous-status behavior; it does not permit an initial +informational response. Late TrailerPrefix entries or represented Trailer +declarations still fail at completion. Ordinary late header-map changes cannot +alter the committed response. Header-map edits that are added and removed before +the writer observes them have no transport effect. + +Tests compare small-response status, body, Content-Type, Content-Length, +suppression, redirects and HEAD/204/304 behavior with Go 1.27's real HTTP server +over its in-memory test network. Additional cases cover 205, status validation, +ignored write errors, large HEAD representations, exact stored-body boundaries, +raw-envelope overhead, text/base64 conversion, cookies and repeated fields, +unsupported ResponseController capabilities and late trailers. Both Gateway +request formats exercise ServeMux path parameters, ServeContent ranges, gzip +output and request-body echoing. Invocation-scope tests cover cleanup on write +failure, panic, parent cancellation and cleanup failure. + +These tests assemble the private primitives inside the existing invocation scope. +They do not publish the pending Lambda invocation methods or establish gateway +identity. Streaming HTTP writer behavior is a separate implementation milestone. + +Validation: focused writer/integration tests, full race suite, vet, formatting +and diff checks, and Linux arm64/amd64 builds pass on Go 1.27.1. No live AWS +resources were invoked. diff --git a/docs/decisions/0010-response-header-design.md b/docs/decisions/0010-response-header-design.md index 45c9879..7b32c95 100644 --- a/docs/decisions/0010-response-header-design.md +++ b/docs/decisions/0010-response-header-design.md @@ -2,7 +2,8 @@ Status: accepted by the user on 2026-09-15, including the measured 256 KiB default and explicit override up to 6 MiB. Shared snapshots, projections and -streaming-prefix encoding are implemented; HTTP writer wiring follows. +streaming-prefix encoding are implemented and connected to the private buffered +HTTP writer. Streaming HTTP writer wiring follows. Terminology: V1/V2 in this record mean API Gateway payload formats 1.0/2.0, not versions of this Go module. The field-combination table concerns outgoing @@ -262,8 +263,8 @@ The shared layer is implemented in response_headers.go. It owns canonical maps and value slices, tracks the original preflight charge without refunds, preserves automatic-header suppression and returns independent V1/V2 projections. New validates WithResponseHeaderBudget after option application, so the final scalar -assignment wins. Only final status/body behavior and HTTP commitment timing -remain responsibilities of the separate writers. +assignment wins. Final status/body behavior and HTTP commitment timing belong +to the separate writers; the buffered writer now implements those rules. Connection parsing tolerates empty list elements under RFC 9110 section 5.6.1.2; the input budget bounds that scanning. Nonempty malformed tokens remain faults. @@ -292,9 +293,10 @@ arm64/amd64 builds pass. The default's measured retained snapshot matches the probe at approximately 0.48 MiB for the high-cardinality fixture. Projection allocations are reported separately; no new resource policy is needed. -Next connect these primitives to the buffered and streaming HTTP writers, with -status/method rules, sniffing, length enforcement, commitment timing and late -unsupported-trailer detection. Constructor options and private primitives do +The buffered writer now uses these primitives for status/method rules, sniffing, +length enforcement, commitment timing and late unsupported-trailer detection. +The streaming HTTP writer remains to be implemented. Constructor options and +private primitives do not make the unfinished adapter usable as a Lambda handler yet. Public invocation/identity work and the SDK Runtime API header compatibility diff --git a/docs/plan.md b/docs/plan.md index bea4732..27c683c 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -119,8 +119,9 @@ accounting, and a conservative exact streaming-prefix limit. The 2026-09-15 allocation probe revised the resource policy to a 256 KiB default with an explicit positive override up to 6 MiB; the user approved it on 2026-09-15. The shared snapshot, suppression/accounting logic, V1/V2 projections, option -validation and bounded streaming metadata prefix are implemented. Writer wiring -remains. Streaming is specified in +validation and bounded streaming metadata prefix are implemented. The buffered +writer now uses the shared header layer; streaming HTTP writer wiring remains. +Streaming is specified in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. The high-level architecture is accepted. [Decision 0009](decisions/0009-streaming-boundaries.md) @@ -144,8 +145,11 @@ sniffing, framing validation, and public entry points remain to be implemented. Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary errors, cleans up on failures/panics, and checks parent cancellation. The HTTP -writer can now use the shared header implementation from -[decision 0010](decisions/0010-response-header-design.md). Public wiring still awaits +writer now uses the shared header implementation from +[decision 0010](decisions/0010-response-header-design.md), enforces commitment and +body/length/status rules, and returns typed AWS response projections without +JSON work. Tests compare ordinary behavior with net/http and exercise request +conversion, mux/range/gzip handling and invocation cleanup. Public wiring still awaits the minimum identity contracts. [Initial codec benchmarks](benchmarks.md) record allocation baselines; they do not establish a peak-memory bound. @@ -181,7 +185,7 @@ presenting an entire frozen public API at once. examples and native HTTP/raw/typed Gateway request contract tests. - [x] Implement shared response-header snapshots, budget configuration, V1/V2 projections and bounded REST streaming metadata prefix encoding. -- [ ] Implement the buffered HTTP writer and gateway response conversion. +- [x] Implement the private buffered HTTP writer and gateway response conversion. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. - [x] Extract shared invocation ownership and implement the private streaming bridge, with handoff, backpressure, cancellation, cleanup, and error tests. From 83085f22c60ec3608bd4295e0344dc3c79770644 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 19:29:32 +0100 Subject: [PATCH 20/44] docs: propose shared identity and invocation error contracts --- docs/decisions/0012-shared-identity.md | 354 ++++++++++++++++++ .../0013-public-invocation-errors.md | 205 ++++++++++ docs/decisions/README.md | 2 + docs/plan.md | 7 + 4 files changed, 568 insertions(+) create mode 100644 docs/decisions/0012-shared-identity.md create mode 100644 docs/decisions/0013-public-invocation-errors.md diff --git a/docs/decisions/0012-shared-identity.md b/docs/decisions/0012-shared-identity.md new file mode 100644 index 0000000..7b50d59 --- /dev/null +++ b/docs/decisions/0012-shared-identity.md @@ -0,0 +1,354 @@ +# 0012: Shared identity, provenance, and producer composition + +Status: proposed for user review on 2026-09-15. Not implemented. + +This walks the dependencies needed for the first callable buffered adapter. +Decision 0013 covers the public errors used at those boundaries. Approval of the +implementation order does not approve these new API and trust decisions. + +## Evidence and constraints + +- `adapter.go` records WithGatewayIdentity, but does not yet produce identity. +- `decode.go` preserves raw authorizer JSON separately from SDK event types. + Typed entry points must not marshal events to get back onto that path. +- `invocation.go` already owns cancellation, the original body, multipart + cleanup, and panic propagation. Identity must fit inside that lifetime. +- Decision 0001 accepts anonymous by default, native gateway identity by opt-in, + and explicitly configured local verification. It leaves producer precedence + open. The saved Beakley review found both an invalid IAM/JWT union and an + anonymous context value that prevented subsequent bearer verification. +- Go's context documentation recommends private keys and request-scoped values. + Go rules favor concrete producer types and interfaces defined by consumers. + The Go 1.27 notes make direct JSON v2 available with strict defaults; generic + methods are available but do not, by themselves, justify a generic claims API. +- AWS documents different REST identity fields for IAM, Cognito, and custom + authorizers. HTTP API payload 1.0 and REST share a shape; payload version does + not identify the API product or its configured authorization type. +- The installed AWS SDK v1.55.0 has independent JWT, IAM, and Lambda authorizer + fields in V2, with JWT claims represented as map[string]string. V1 authorizer + data is map[string]any. These are data containers, not identity validators. +- AWS JWT authorizers validate aud preferentially to client_id when both exist; + that does not make those fields interchangeable in an application identity. + Scope checks also depend on route configuration. A gateway assertion does not + universally establish that an access-token-only policy was applied. + +Official sources checked through AWS public MCP and installed Go documentation: + +- [REST proxy identity fields](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) +- [HTTP API event formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) +- [JWT authorizer validation](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-jwt-authorizer.html) +- [Custom authorizer output](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-lambda-authorizer-output.html) +- [HTTP API custom authorizers](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html) +- [Cognito claim meanings](https://docs.aws.amazon.com/verifiedpermissions/latest/userguide/cognito-map-token-to-schema.html) +- [M2M authorization](https://aws.amazon.com/blogs/security/how-to-monitor-optimize-and-secure-amazon-cognito-machine-to-machine-authorization/) +- [STS session identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumedRoleUser.html) +- [SDK event types at v1.55.0](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.55.0/events) +- [Context values](https://pkg.go.dev/context#WithValue) +- [Go 1.27 release notes](https://go.dev/doc/go1.27) + +## Design graph + +```mermaid +flowchart TD + Entry[Raw Invoke or typed HandleV1 / HandleV2] --> Lifetime[Validate invocation; own request and context] + Lifetime --> Gateway{Gateway identity enabled?} + Gateway -->|No| Anonymous[No caller installed] + Gateway -->|Yes| Recognize[Recognize native assertion shape] + Recognize -->|Absent| Anonymous + Recognize -->|Malformed, conflicting, or unsupported| Error[Invocation error; no HTTP handler] + Recognize -->|Supported| Normalize[Preserve source representation; validate caller] + Normalize --> Install[Install one immutable caller] + Anonymous --> HTTP[Ordinary HTTP middleware] + Install --> HTTP + HTTP --> Local{Explicit local authentication?} + Local -->|No| Select[Parse action once] + Local -->|Yes| Verify[Verify token using configured issuer and client policy] + Verify -->|Invalid or unavailable dependency| AuthResponse[Middleware writes HTTP response] + Verify -->|Verified, no existing caller| Select + Verify -->|Existing caller| Conflict[Explicit composition policy required] + Select --> Resolve[Resolve application principal and grants] + Resolve --> Authorize[Authorize the selected action] + Authorize -->|Allow| Execute[Execute the same selected action] + Authorize -->|Deny or dependency failure| AuthResponse + Execute --> Response[Buffered completion or streaming handoff] + AuthResponse --> Response +``` + +The action-selector order relative to authentication is application policy. +The invariant is that authorization and execution use the same selection. +Identity is independent of the selected action and carries no dispatcher state. + +## I1. Concrete caller, with an enforced union + +**Question:** interface, exported struct with optional arms, or an opaque value? + +**Recommend:** a small concrete `identity.Caller` with private representation, +anonymous zero value, and mutually exclusive JWT/IAM views. Construction returns +an error for invalid inputs; users cannot set both arms. Read-only value methods +can share immutable backing data. There is no AWS dependency in identity. + +Proposed core surface (declarations illustrate the contract, not runnable code): + +```go +type Caller struct { /* private immutable representation */ } +func (c Caller) Kind() Kind +func (c Caller) Source() Source +func (c Caller) JWT() (JWT, bool) +func (c Caller) IAM() (IAM, bool) + +func FromContext(ctx context.Context) Caller +func WithCaller(ctx context.Context, caller Caller) (context.Context, error) +``` + +Kind distinguishes anonymous, JWT, and IAM. Source distinguishes no identity, +gateway assertion, locally verified token, and explicitly mapped custom +assertion. JWT/IAM views have private fields and read-only accessors. Avoid a +second Authenticated boolean that can disagree with the union. Do not make a +public equality/hash contract for the entire caller or infer equivalence from +matching subjects. Consumers may define small interfaces where actually needed. + +**Reasoning:** this prevents the reproduced dual-arm authorization bug at the +construction boundary. An implementable Caller interface or public pointer arms +would require every authorization rule to re-check the union. + +## I2. Construction is validation, not verification + +**Question:** what security guarantee does a constructed caller make? + +**Recommend:** constructors validate representation and own their inputs. The +producer is responsible for authentication and source attribution. Constructors +do not parse bearer headers, verify signatures, fetch keys, or decide permissions. +Source is useful provenance recorded by trusted application code, not a proof +that hostile code in the same process cannot forge. + +Identity is a snapshot established for a request, with no clock or network work +in its accessors. Expiry checks belong to the authenticating producer's admission +policy. Long-running streams do not silently refresh credentials or acquire new +permissions; any later reauthorization is explicit application policy. + +Recommended producer shape: + +```go +func NewJWT(claims Claims, source Source, opts ...JWTOption) (Caller, error) +func NewIAM(principalARN string, source Source, opts ...IAMOption) (Caller, error) +``` + +JWT options supply information that is separate from the claim object, notably +the gateway's dedicated scopes collection. They must not override issuer, +subject, or permissions already represented by conflicting source data. Reject +duplicate assignments for identity construction rather than using last-wins +to conceal contradictory assertions. No mutating caller builders or setters. + +JWT source may be gateway, verified token, or explicit custom mapping. Initially +IAM source may be gateway or explicit custom mapping; there is no local SigV4 +verifier. Reject unknown sources and impossible source/kind combinations. + +**Alternative:** an unforgeable-looking internal constructor would make external +verifiers and fixtures awkward while failing to secure against application code +that already controls the authorization pipeline. + +## I3. JWT principal and permission semantics + +**Recommend these distinct fields/accessors:** + +| Fact | Recommendation and reason | +| --- | --- | +| Issuer | Required nonempty exact string; do not lowercase or normalize. Verification decides the trusted issuer policy. | +| Subject | Optional nonempty value; when present, identify it together with issuer. Never use bare sub as a global identity key. | +| Client ID | Separate optional nonempty value. Do not substitute audience or silently treat a missing subject as a user. | +| Minimum identifying material | Require subject or client ID. A client-only caller is representable, but application policy must decide whether it is an acceptable machine principal. Missing sub alone does not prove a client-credentials flow. | +| Audience | Preserve string versus actual array at the claim layer; normalized audience collection only when its representation is understood. Do not copy it into client ID. | +| Token use | Preserve if supplied, with no invented default. Cognito access-token-only restrictions belong to authn's verifier policy. | +| Scopes | Separate from groups and grants. Prefer the explicit gateway scopes field when present; compare any independently interpretable claim scopes and reject disagreement rather than unioning them. | +| Cognito groups | Preserve the Cognito-specific name/meaning. Do not reinterpret arbitrary groups/roles claims as Cognito groups. | +| Application grants | Produced by application principal resolution in authz, not appended to the authenticated token's claims. | + +Use comma-ok accessors for optional facts and collections. A known empty +collection differs from unavailable data. Set-valued authorization checks are +case-sensitive and exact; normalization may deduplicate identical elements but +must not change their spelling or apply wildcard semantics. + +**Issue:** gateway string representations of aud, scp, or cognito:groups do not +provide a general reversible encoding for their original JSON types. Recommend +no bracket stripping, comma splitting, or reparsing a string as embedded JSON. +An actual JSON array can supply a list; a flattened ambiguous value cannot. +OAuth scope strings are a specific documented space-delimited representation, +not a precedent for splitting unrelated claims. + +If a recognized claim has an invalid actual type, reject the producer's input. +If the source has flattened away its type, retain that representation and mark +the normalized collection unavailable. A rule requiring unavailable information +must not authorize. A custom claim schema or local verification can resolve the +missing fidelity later; do not fill the gap with an unverified header JWT. + +## I4. Owned claims with explicit representation + +**Question:** can a map[string]any or map[string]string be the shared claims API? + +**Recommend:** an immutable `Claims`/`Claim` view, with separate construction from +raw JSON, gateway string maps, and already decoded SDK V1 values. The names and +complete accessor inventory are a follow-on API detail to review before that +part is implemented; the recommended semantics are concrete: + +- Raw claim JSON is validated with direct JSON v2. Preserve exact numeric text, + arrays, objects, null, and booleans. This is fidelity to the received claim + object, not a claim that API Gateway preserved the original JWT bytes. +- Gateway string values remain marked as gateway text. A value resembling JSON + is still text. Scalar claim interpretation must use its declared schema. +- Already decoded SDK values are copied directly without marshaling. Preserve + the available shape. An upstream float64 must not be presented as the original + exact JSON number, even when its current value happens to be integral. +- Validate UTF-8 on directly supplied strings as well as JSON input. Do not + coerce incorrectly typed standard claims or silently rewrite identifiers. +- Expose presence, representation, and checked scalar/list access, not unchecked + assertions. Wrong-type reads must be distinguishable from missing values. +- Copy input maps, slices, and JSON bytes; returned containers/bytes must not + provide mutation access. Immutable views may share private owned backing data. +- Retain unknown claims without promoting them to permissions. Do not store the + bearer token or complete Authorization header in identity. +- Do not expose a generic `Claim[T]` convenience in the first API. It would hide + conversion and fidelity policy before those conversions are well established. +- No implicit JSON export of an entire caller. Default diagnostic formatting + should describe kind/source without dumping claims or identifiers. Explicit + application logging remains application-owned. + +**Remaining resource issue:** cloning a typed V1 map can encounter huge input, +cycles, or arbitrary Go objects. Recommend a closed set of supported JSON-like +types, cycle/depth checks, and a measured claim resource budget before exposing +those constructors. Reject unsupported values without invoking custom marshal +methods. Numeric budget defaults need a separate measured proposal; no AWS +claim-size quota is inferred here. This can be investigated independently while +the caller/context/error core is implemented after approval. + +## I5. IAM facts remain exact facts + +**Recommend:** retain the exact principal ARN, parsed partition/account/service +and principal form, and optional gateway principal identifiers. Validate the +principal form rather than accepting any syntactically valid resource ARN. +Cross-check an independently supplied caller account ID against the ARN. + +Do not use requestContext.accountId as the caller account: it is API-owner +metadata. Do not use access-key ID, source IP, API key, mTLS metadata, or a Cognito +identity-pool ID alone as proof of IAM identity. Do not retain access-key/API-key +values in the shared caller. + +Keep an STS session ARN as an STS session ARN. Do not manufacture an IAM role ARN +or remove a session name to pretend to recover a stable role identity. Support +for the documented caller forms and their ARN grammar needs explicit fixtures +in the producer implementation; unsupported forms fail rather than falling back +to anonymous. ARN authorization matching and role/session resolution remain +authz policies, not IAM policy evaluation in identity. + +**Reasoning:** STS documents both the ARN and ID as including session identity; +losing that information changes what an application is authorizing. + +## I6. Context installation and invocation isolation + +**Recommend:** FromContext returns the anonymous zero value when absent. +WithCaller uses a private context key, returns a derived context, and never +mutates shared state. Supplying anonymous to an empty context is a no-op. It +cannot clear an existing nonanonymous caller. Installing another nonanonymous +caller, even one with apparently matching fields, returns identity.ErrConflict. + +Expose identity.ErrInvalidCaller for invalid construction and identity.ErrConflict +for producer collisions. Do not add a HasIdentity API whose presence test can +become an authentication shortcut. Nil context is invalid; installation returns +an error and FromContext requires a nonnil context like ordinary context use. + +**New invocation boundary recommendation:** reject a parent context already +carrying a nonanonymous identity, before running the HTTP handler, on raw and +typed entry points in either gateway mode. Preserve unrelated context values. +This makes accidental reuse of a previous request's caller visible. Native HTTP +middleware is unaffected because it does not pass through a Lambda entry point. + +**Alternatives:** inheriting the caller can authorize a different invocation; +silently masking it conceals a composition mistake and needs a clearing API. +Explicit support for a preauthenticated invocation could be added later if a +real consumer requires it. Do not infer that authorization from an arbitrary +context supplied to an entry point today. + +## I7. Gateway recognition: absent, invalid, and unsupported differ + +**Recommend this behavior when WithGatewayIdentity(true):** + +| Input | Outcome | +| --- | --- | +| Absent/null authorizer, null placeholder claims/scopes, no IAM assertion | Anonymous; handler runs | +| One supported native JWT or IAM assertion | Normalize, validate, install caller | +| Present recognized assertion missing required identity data or wrong-shaped fields | Invocation error; handler does not run | +| Simultaneous IAM/JWT or native/custom assertion candidates | Conflict error; never choose by field order | +| Nonempty custom authorizer context without a configured mapper | Unsupported identity error; never treat principalId alone as native IAM/JWT | +| No authenticated caller but sourceIp/userAgent/API-key metadata exists | Anonymous; those fields are not an authentication producer | + +With gateway identity disabled, authorizer data does not establish identity and +semantic identity validation is skipped. Raw JSON syntax is still validated by +the existing decoder. Identity opt-in is not a RequireAuthenticated option; +applications protect routes through authn/authz middleware. + +**Qualification:** V2 has explicit JWT/IAM/Lambda containers, but V1 has no +reliable universal authorizer-type discriminator. Follow documented native +claims/identity shapes conservatively; principalId/custom indicators take a V1 +event out of native inference. Do not claim this proves deployment settings. +An event producer can forge any event shape; trust in gateway assertions already +requires the intended restricted integration as accepted in decision 0001. + +Unknown extension fields alongside a recognized native assertion are not, by +themselves, another producer. Retain/ignore them according to the metadata +contract without promoting them to permissions. A nonempty authorizer object +with no recognized native producer remains unsupported. This permits documented +AWS field additions without making arbitrary custom contexts authoritative. + +The HTTP API 1.0 documentation supplies a null-authorizer example, not a complete +matrix of JWT/IAM claim encodings. Carry this as a fixture/documentation coverage +gap, especially for flattened collections. Do not invent a V1 array encoding or +silently restrict transport support to solve it. Add independently sourced +fixtures for each supported producer shape before claiming that combination. + +## I8. Local authentication and custom producers + +**Recommend:** the default local bearer middleware verifies only when no caller +has been established. If already authenticated input reaches it, fail as an +explicit composition conflict; do not skip based on context presence, replace +the caller, or merge permissions. Applications initially choose one producer per +route through ordinary middleware composition. + +This is deliberately conservative: a gateway-authenticated route normally still +contains its Authorization header. The adapter must not mistake that header for +a second authentication producer. A collision happens when another configured +producer runs, not merely because credentials are present in a header. + +Future gateway fallback or re-verification can be explicit policies with their +own comparison rules. A fallback policy would invoke the local verifier only +for anonymous input. Re-verification must validate its own issuer/client policy +and cannot equate identities by sub/client_id alone. Defer their option names +and credential-conflict behavior until the authn middleware design. + +Custom-authorizer mapping is also deferred. When added, it should return a +validated Caller through a context-aware function, with no silent fallback and +no automatic extra-claim merge. It may eventually need a distinct custom +principal kind; do not force an arbitrary principalId into IAM or JWT now. + +## Implementation graph and acceptance + +1. Approve/revise caller invariants, provenance, context, and error contracts. +2. Implement caller/context/error core and test construction/ownership/conflicts. +3. Review the precise claims accessors, budget measurements, and producer fixture + matrix. Implement shared claim normalization and minimum native producers. +4. Wire raw and typed buffered invocation using the same producers and errors. +5. Prove registration through the real SDK wrapper; add runnable examples. +6. Design local Cognito verification/JWKS behavior and authz against this core. +7. Reuse the same preparation boundary in the separate streaming entry points. + +Do not publish incomplete constructors or placeholder invocation methods to +represent these intermediate steps. The claim contract must be ready before +publishing constructors that depend on it. + +Acceptance includes anonymous-to-local-verifier composition, dual-source +rejection, raw/typed equivalence where information survives, explicit loss of +fidelity, copied input/output containers, preserved issuer/client distinctions, +no action/grant mutation of authenticated claims, and no cross-invocation caller +inheritance. No live AWS invocation is needed for this design review. + +## Resolution + +Awaiting user review. None of the proposed shared identity API is implemented. diff --git a/docs/decisions/0013-public-invocation-errors.md b/docs/decisions/0013-public-invocation-errors.md new file mode 100644 index 0000000..c6f2c82 --- /dev/null +++ b/docs/decisions/0013-public-invocation-errors.md @@ -0,0 +1,205 @@ +# 0013: Public invocation error contracts + +Status: proposed for user review on 2026-09-15. Not implemented. + +## Evidence + +Decisions 0002, 0004, 0006, and 0009 already settle response ownership, typed +zero responses on failure, sticky writer faults, cancellation precedence, and +the streaming handoff boundary. This proposal gives those outcomes a public Go +contract; it does not reopen them. + +Current code returns sanitized private errors from decode/request conversion, +response snapshots/encoding, cleanup, and streaming. It already preserves +http.ErrNotSupported and http.ErrContentLength where meaningful. Body-forbidden +writes return http.ErrBodyNotAllowed without poisoning an otherwise valid +response. `withInvocation` joins cleanup faults after the operation fault and +uses parent cancellation only when no operation/cleanup error exists. + +AWS documents that a Lambda function error or malformed proxy response produces +an API Gateway 502, while a Lambda API invocation rejection produces 500. A valid +proxy response controls the HTTP status; API Gateway does not retry Lambda +invocations. Go documents errors.Is, errors.AsType, and traversal of joined error +trees. The SDK's Handler interface has one result/error channel for a buffered +invocation, not a separate application HTTP error channel. + +Sources checked through AWS public MCP and installed Go documentation: + +- [API Gateway/Lambda errors](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway-errors.html) +- [SDK Handler at v1.55.0](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.55.0/lambda#Handler) +- [Go error inspection](https://pkg.go.dev/errors) +- [errors.Join](https://pkg.go.dev/errors#Join) +- [errors.AsType](https://pkg.go.dev/errors#AsType) +- [HTTP unsupported features](https://pkg.go.dev/net/http#ErrNotSupported) +- [Streaming boundary already approved](0009-streaming-boundaries.md) + +## E1. HTTP outcomes stay separate from invocation failure + +**Recommend retaining the accepted boundary:** + +| Outcome | Result | +| --- | --- | +| Malformed Lambda event, request conversion failure, invalid opted-in gateway assertion | Go invocation error; no HTTP handler | +| Local bearer rejected, permission denied, application validation failure | Middleware/handler writes HTTP response; nil invocation error if transport completes | +| Auth dependency unavailable | Middleware chooses its HTTP response, normally 503 under the later authn/authz contract; not automatically an adapter failure | +| Invalid response construction, unsupported committed behavior, resource limit, cleanup failure | Go invocation error; discard buffered result | +| Application explicitly writes 500 | Valid HTTP response with nil invocation error if transport completes | + +**Reasoning:** an invalid token presented to an explicitly configured verifier +is an application authentication outcome. An inconsistent gateway assertion is +a failure to establish the integration's promised identity. Turning both into +the same Go error would make routine authentication failures appear as gateway +502s. The transport must not manufacture 401/403/503 from arbitrary errors. + +Authn/authz HTTP status, bearer challenges, and public-route policies still need +their own review. These errors do not settle that middleware's complete API. + +## E2. Stable categories plus one diagnostic type + +**Question:** export each private error, only strings, or structured categories? + +**Recommend:** a small set of sentinels checked with errors.Is, plus an opaque +*edge.InvocationError carrying bounded diagnostic fields. Do not require parsing +Error() text and do not expose decoder or SDK implementation errors. + +| Proposed sentinel | Meaning | +| --- | --- | +| ErrInvalidInvocation | Invalid direct call, such as nil context or unusable Adapter | +| ErrInvalidEvent | Malformed supported event or failed request conversion | +| ErrUnsupportedEvent | Unsupported event family, version, or streaming product boundary | +| ErrIdentity | Failure in opted-in identity production or invocation identity isolation | +| ErrResponse | Invalid or unrepresentable HTTP response, including encoding failure | +| ErrLimitExceeded | Adapter-enforced response/header/stream-metadata resource limit | +| ErrCleanup | Request-resource cleanup failed | +| ErrStream | Stream lifecycle failure or recovered late producer/reporter panic | + +Use identity.ErrInvalidCaller and identity.ErrConflict as safe subordinate causes +of ErrIdentity. An unsupported custom producer can additionally match +errors.ErrUnsupported. Do not export one sentinel for every missing event field +or invalid header; those are diagnostics within a category. + +Proposed diagnostic shape: + +```go +type InvocationError struct { /* private sanitized fields and cause */ } +func (e *InvocationError) Error() string +func (e *InvocationError) Unwrap() error +func (e *InvocationError) Operation() string +func (e *InvocationError) Limit() (name string, maximum int64, ok bool) +``` + +Operation uses documented bounded labels: validate, decode, request, identity, +response, encode, cleanup, stream. Limit is present only for a limit failure and +reports its configured maximum and a bounded name such as response_headers, +buffered_body, buffered_envelope, or stream_metadata. Do not add arbitrary field +paths, credential values, request snapshots, or caller identifiers. + +The private cause can join a category and explicitly allowed subordinate errors; +Unwrap exposes only that safe tree. Avoid custom Is/As machinery where ordinary +wrapping suffices. Use errors.AsType[*edge.InvocationError] on Go 1.27 for details. +Do not add a generic Result[T], a second numeric error-code registry, or error +methods that prescribe HTTP status or automatic retries. + +Constructor/configuration errors remain ordinary startup errors, outside +InvocationError. Existing ActionHeader sentinels stay separate because header +selection is an application concern. Failure to call a constructor is an +ErrInvalidInvocation when someone directly invokes an unusable adapter. + +**Tradeoff:** eight broad categories commit less implementation detail than the +current private error list while distinguishing the operational decisions a +caller can reasonably make. Stable limit metadata avoids three more budget +sentinels and distinguishes an adjustable header budget from a payload ceiling. + +## E3. Preserve useful standard errors; sanitize the entire chain + +**Recommend:** preserve errors.Is for context.Canceled, context.DeadlineExceeded, +http.ErrNotSupported, and http.ErrContentLength where they are the actual cause. +Explicitly unsupported HTTP features also remain in ErrResponse when they are +invocation-fatal. A handler probing an absent ResponseController capability and +handling ErrNotSupported does not, by itself, poison the invocation. + +Keep http.ErrBodyNotAllowed on disallowed Write calls as accepted; it is not an +invocation failure if the response otherwise completes. Keep io.EOF as successful +stream termination. Private pipe teardown errors must not replace the real +cancellation or producer failure. + +Do not wrap raw JSON/base64/parser errors, custom mapper errors, OS cleanup paths, +or recovered panic payloads into public invocation errors. Formatting an unsafe +cause with %v still leaks its text; losing Unwrap alone is not sanitization. +Construct safe diagnostics at the originating boundary, not by parsing existing +private error strings at the public entry point. + +Do not expose arbitrary context.Cause(parent) automatically: it can contain +application secrets or private error types. Preserve the standard parent.Err() +classification instead. Callers still control their original context. + +The sanitized-error promise does not apply to an application panic deliberately +re-propagated to the SDK under the already accepted early-panic contract. Edge +does not promise to sanitize SDK/application logging or recover arbitrary +application panics into ordinary invocation errors. + +## E4. Primary failure, cleanup, and cancellation + +**Recommend preserving current precedence:** + +1. Return the operation's classified error first. +2. Join any separately classified cleanup failure after it. +3. Use parent.Err() if no operation/cleanup failure exists. +4. Normal internal child cancellation after successful work is not an error. + +Keep joined faults individually inspectable. errors.AsType sees the primary +InvocationError first; errors.Is can detect secondary ErrCleanup. Error text and +join formatting are diagnostic, not a stability promise. If there is a primary +failure plus concurrent parent cancellation, do not replace that failure with +the cancellation; callers can inspect their context separately. + +Any nonnil buffered invocation error produces nil raw bytes or a zero typed +response. No caller can use a partially encoded result as success. Cleanup +failure alone still fails the invocation, as already implemented and accepted. + +**Alternative:** logging cleanup faults and returning success hides resource +leaks in a reused Lambda process. Always joining cancellation would blur whether +it was the cause or merely concurrent shutdown. + +## E5. The same vocabulary across raw, typed, and streaming calls + +**Recommend:** classify where shared logic detects failure. Raw decoding and +final envelope encoding add their own operations; typed calls retain shared +semantic classifications without a JSON round trip. Never promise a typed call +detects duplicate keys, lost numbers, or the final size of an envelope encoded +later by the SDK. SDK-originated failures outside edge cannot be InvocationError. + +Before a streaming handoff, return nil reader and the classified error. After +handoff, expose the classified terminal error through the reader and the already +approved reporter; status/body cannot be replaced. Preserve the distinction +between an ordinary returned error and a late stream failure in documentation +and metrics. Do not claim AWS treats them as the same invocation-error metric. + +Recovered late producer panic matches ErrStream. A reporter panic adds another +sanitized ErrStream diagnostic without replacing the original error, exposing +the panic value, or recursively calling the reporter. Early producer panics +still re-panic on the entry goroutine after cleanup. A future streaming writer +must use this same classification at its response and limit fault sites. + +No new buffering, SDK patch, or custom runtime is needed for this error contract. +The documented Runtime API header discrepancy remains a separate streaming +release gate from decision 0009. + +## Implementation and acceptance + +- Introduce public categories/type and classify existing private failure sites. + Keep writer-local standard errors compatible with their accepted contracts. +- Use safe internal constructors for diagnostics; do not accept arbitrary + untrusted cause strings as diagnostic fields. +- Test errors.Is/AsType after outer wrapping and joined cleanup faults, sanitized + error trees, correct result discard, cancellation precedence, and limits. +- Test malformed event versus malformed application body, 401/403/500 responses, + invalid gateway assertions, default-off authorizer handling, and SDK registration. +- Include raw/typed parity only where the input retains the required information. +- Extend existing stream tests for classified early/late/reporter errors when + wiring streaming; no need to finish streaming before buffered invocation. + +## Resolution + +Awaiting user review. This is a documentation proposal; no runtime behavior has +changed. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index b7376c2..88675d0 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -20,3 +20,5 @@ Keep related decisions small enough to review and implement independently. | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | | [0010: Response header design](0010-response-header-design.md) | Accepted | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | | [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | +| [0012: Shared identity](0012-shared-identity.md) | Proposed | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | +| [0013: Invocation errors](0013-public-invocation-errors.md) | Proposed | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | diff --git a/docs/plan.md b/docs/plan.md index 27c683c..bfbb401 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -217,6 +217,13 @@ event cases; no auth or codec behavior depends on accidental SDK dispatch. ### 3. Identity and gateway producers +Design graph prepared on 2026-09-15 in [decision 0012](decisions/0012-shared-identity.md), +with [public invocation errors in decision 0013](decisions/0013-public-invocation-errors.md). +Both are proposals awaiting review, not implemented or approved contracts. +The recommended order is caller/context/error foundations, precise claim API and +native-producer fixtures, then raw/typed buffered invocation. Broader local +authentication and authorization follow that shared foundation. + - Review caller constructors, immutable/owned data, validation, anonymous state, and context-presence semantics. - Preserve issuer plus subject for JWT application identity; do not treat a From 79eb17051c449e1de8c255f99169ae05ca0ccbf3 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Tue, 15 Sep 2026 19:45:51 +0100 Subject: [PATCH 21/44] feat: classify invocation failures with sanitized public errors --- README.md | 8 + buffered.go | 16 +- decode.go | 28 +- docs/decisions/0012-shared-identity.md | 8 +- .../0013-public-invocation-errors.md | 24 +- docs/decisions/README.md | 4 +- docs/plan.md | 7 +- errors.go | 101 ++++++++ errors_test.go | 241 ++++++++++++++++++ invocation.go | 6 +- request.go | 33 ++- response_encoding.go | 8 +- response_headers.go | 18 +- response_headers_test.go | 4 +- stream.go | 8 +- stream_encoding.go | 6 +- stream_test.go | 8 + 17 files changed, 455 insertions(+), 73 deletions(-) create mode 100644 errors.go create mode 100644 errors_test.go diff --git a/README.md b/README.md index cb90ccc..e67557d 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,14 @@ length enforcement and typed AWS response projection. Public invocation methods and identity extraction are not implemented yet; the adapter cannot yet be registered as a Lambda handler. +The shared invocation error contract is implemented across those private stages: +public categories support `errors.Is`, and `errors.AsType[*edge.InvocationError]` +exposes sanitized operation and resource-limit diagnostics. HTTP error responses +remain successful transport outcomes; operation, cleanup, cancellation, and late +stream failures retain their accepted ownership and precedence. The shared +identity design is accepted, with precise claim accessors and resource limits +still awaiting their follow-up design review. + The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle is covered by race-enabled synthetic concurrency tests. Bounded JSON v2 metadata diff --git a/buffered.go b/buffered.go index 6705d88..5fc534c 100644 --- a/buffered.go +++ b/buffered.go @@ -1,8 +1,6 @@ package edge import ( - "errors" - "fmt" "math" "net/http" "strconv" @@ -11,7 +9,7 @@ import ( "github.com/aws/aws-lambda-go/events" ) -var errInformationalResponse = fmt.Errorf("edge: informational response: %w", http.ErrNotSupported) +var errInformationalResponse = invocationError("response", ErrResponse, "informational response is unsupported", http.ErrNotSupported) // bufferedWriter belongs to one handler invocation. Only Header, WriteHeader // and Write form its HTTP surface; it deliberately exposes no network or @@ -50,7 +48,7 @@ func (w *bufferedWriter) WriteHeader(status int) { return } if status < 200 || status > 599 { - w.err = errors.New("edge: invalid final response status") + w.err = invocationError("response", ErrResponse, "invalid final response status") return } w.status = status @@ -81,11 +79,11 @@ func (w *bufferedWriter) Write(p []byte) (int, error) { return 0, http.ErrBodyNotAllowed } if w.hasLength && int64(len(p)) > w.declaredLength-w.written { - w.err = http.ErrContentLength + w.err = invocationError("response", ErrResponse, "response exceeds declared content length", http.ErrContentLength) return 0, w.err } if int64(len(p)) > math.MaxInt64-w.written { - w.err = errResponseTooLarge + w.err = limitError("response", "buffered_body", math.MaxInt64, errResponseTooLarge) return 0, w.err } data := p @@ -93,8 +91,8 @@ func (w *bufferedWriter) Write(p []byte) (int, error) { data = p[:min(len(p), 512-len(w.body.data))] } if _, err := w.body.Write(data); err != nil { - w.err = err - return 0, err + w.err = limitError("response", "buffered_body", maxResponseBytes, errResponseTooLarge) + return 0, w.err } w.written += int64(len(p)) return len(p), nil @@ -122,7 +120,7 @@ func (w *bufferedWriter) finish() (*bufferedResponse, error) { } } if w.method != http.MethodHead && w.hasLength && w.written != w.declaredLength { - return nil, fmt.Errorf("edge: response body length mismatch: %w", http.ErrContentLength) + return nil, invocationError("response", ErrResponse, "response body length mismatch", http.ErrContentLength) } if bufferedBodyAllowed(w.status) { if len(w.body.data) > 0 && len(w.headers.fields["Content-Encoding"]) == 0 { diff --git a/decode.go b/decode.go index 681411a..4350019 100644 --- a/decode.go +++ b/decode.go @@ -28,22 +28,22 @@ func decodeEvent(payload []byte) (decodedEvent, error) { if err := json.Unmarshal(payload, &envelope); err != nil { // JSON errors can contain object names and input values. Do not expose // them, even through an unwrap chain, at this credential-bearing boundary. - return decodedEvent{}, errors.New("edge: invalid event JSON") + return decodedEvent{}, invocationError("decode", ErrInvalidEvent, "invalid event JSON") } if envelope == nil { - return decodedEvent{}, errors.New("edge: event must be a JSON object") + return decodedEvent{}, invocationError("decode", ErrInvalidEvent, "event must be a JSON object") } var version string if len(envelope.Version) != 0 { if err := json.Unmarshal(envelope.Version, &version); err != nil { - return decodedEvent{}, errors.New("edge: invalid payload version") + return decodedEvent{}, invocationError("decode", ErrInvalidEvent, "invalid payload version") } if version != "1.0" && version != "2.0" { - return decodedEvent{}, errors.New("edge: unsupported payload version") + return decodedEvent{}, invocationError("decode", ErrUnsupportedEvent, "unsupported payload version") } } if len(envelope.IsBase64Encoded) != 0 && envelope.IsBase64Encoded.Kind() != 't' && envelope.IsBase64Encoded.Kind() != 'f' { - return decodedEvent{}, errors.New("edge: isBase64Encoded must be a boolean") + return decodedEvent{}, invocationError("decode", ErrInvalidEvent, "isBase64Encoded must be a boolean") } if version == "2.0" { @@ -55,7 +55,7 @@ func decodeEvent(payload []byte) (decodedEvent, error) { } `json:"requestContext"` } if err := json.Unmarshal(payload, &wire, eventStringOptions); err != nil { - return decodedEvent{}, errors.New("edge: invalid v2 event fields") + return decodedEvent{}, invocationError("decode", ErrInvalidEvent, "invalid v2 event fields") } event := wire.APIGatewayV2HTTPRequest event.RequestContext = wire.RequestContext.APIGatewayV2HTTPRequestContext @@ -73,7 +73,7 @@ func decodeEvent(payload []byte) (decodedEvent, error) { } `json:"requestContext"` } if err := json.Unmarshal(payload, &wire, eventStringOptions); err != nil { - return decodedEvent{}, errors.New("edge: invalid v1 event fields") + return decodedEvent{}, invocationError("decode", ErrInvalidEvent, "invalid v1 event fields") } event := wire.APIGatewayProxyRequest event.RequestContext = wire.RequestContext.APIGatewayProxyRequestContext @@ -85,29 +85,29 @@ func decodeEvent(payload []byte) (decodedEvent, error) { func validateV1(event events.APIGatewayProxyRequest) error { if event.RequestContext.APIID == "" { - return errors.New("edge: missing requestContext.apiId") + return invocationError("validate", ErrInvalidEvent, "missing requestContext.apiId") } if event.HTTPMethod == "" { - return errors.New("edge: missing httpMethod") + return invocationError("validate", ErrInvalidEvent, "missing httpMethod") } if event.Path == "" { - return errors.New("edge: missing path") + return invocationError("validate", ErrInvalidEvent, "missing path") } return nil } func validateV2(event events.APIGatewayV2HTTPRequest) error { if event.Version != "" && event.Version != "2.0" { - return errors.New("edge: conflicting v2 payload version") + return invocationError("validate", ErrInvalidEvent, "conflicting v2 payload version") } if event.RequestContext.APIID == "" { - return errors.New("edge: missing requestContext.apiId") + return invocationError("validate", ErrInvalidEvent, "missing requestContext.apiId") } if event.RequestContext.HTTP.Method == "" { - return errors.New("edge: missing requestContext.http.method") + return invocationError("validate", ErrInvalidEvent, "missing requestContext.http.method") } if event.RawPath == "" { - return errors.New("edge: missing rawPath") + return invocationError("validate", ErrInvalidEvent, "missing rawPath") } return nil } diff --git a/docs/decisions/0012-shared-identity.md b/docs/decisions/0012-shared-identity.md index 7b50d59..3881621 100644 --- a/docs/decisions/0012-shared-identity.md +++ b/docs/decisions/0012-shared-identity.md @@ -1,6 +1,6 @@ # 0012: Shared identity, provenance, and producer composition -Status: proposed for user review on 2026-09-15. Not implemented. +Status: accepted as the starting contract on 2026-09-15; implementation pending. This walks the dependencies needed for the first callable buffered adapter. Decision 0013 covers the public errors used at those boundaries. Approval of the @@ -351,4 +351,8 @@ inheritance. No live AWS invocation is needed for this design review. ## Resolution -Awaiting user review. None of the proposed shared identity API is implemented. +The user agreed with all recommendations as a starting point on 2026-09-15. +This accepts I1-I8 and the implementation order. The explicitly deferred claim +accessor inventory, claim resource limits, producer fixture coverage, and later +authn/custom-mapper policies still require their recorded follow-up review. +None of the shared identity API is implemented yet. diff --git a/docs/decisions/0013-public-invocation-errors.md b/docs/decisions/0013-public-invocation-errors.md index c6f2c82..7fe28e6 100644 --- a/docs/decisions/0013-public-invocation-errors.md +++ b/docs/decisions/0013-public-invocation-errors.md @@ -1,6 +1,6 @@ # 0013: Public invocation error contracts -Status: proposed for user review on 2026-09-15. Not implemented. +Status: accepted on 2026-09-15; implemented for existing private transport stages. ## Evidence @@ -201,5 +201,23 @@ release gate from decision 0009. ## Resolution -Awaiting user review. This is a documentation proposal; no runtime behavior has -changed. +The user agreed with all recommendations as a starting point on 2026-09-15. +This accepts E1-E5, including the public categories and diagnostic methods. +Classification of the existing private transport stages is the first independent +implementation step. Public invocation still depends on the identity work. + +Implemented: eight public categories and InvocationError with Error, Unwrap, +Operation, and Limit. Existing decoder, semantic validation, request conversion, +response snapshots/projection, buffered writer, bounded encoding, cleanup, and +bridge-owned streaming failures now supply classified errors. Private low-level +charging/buffering errors remain allocation-free and receive diagnostics at the +stage that knows the original resource maximum. JSON encoder error trees are +discarded even when their safe limit cause is recognized. + +Validation covers wrapping and joined cleanup inspection, complete exposed error +trees, declared-length errors, original versus remaining header budgets, body +and envelope limits, stream metadata limits, private cancellation causes, late +producer/reporter panic categories, and successful HTTP 401/403/500/503 responses. +Go 1.27.1 tests with race detection, vet, formatting checks, and Linux arm64/amd64 +builds pass. Public identity errors and raw/typed invocation registration tests +remain dependent on the identity producers; they are not claimed as implemented. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 88675d0..6779155 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -20,5 +20,5 @@ Keep related decisions small enough to review and implement independently. | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | | [0010: Response header design](0010-response-header-design.md) | Accepted | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | | [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | -| [0012: Shared identity](0012-shared-identity.md) | Proposed | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | -| [0013: Invocation errors](0013-public-invocation-errors.md) | Proposed | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | +| [0012: Shared identity](0012-shared-identity.md) | Accepted; detailed claims follow-up pending | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | +| [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | diff --git a/docs/plan.md b/docs/plan.md index bfbb401..6a761c0 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -186,6 +186,8 @@ presenting an entire frozen public API at once. - [x] Implement shared response-header snapshots, budget configuration, V1/V2 projections and bounded REST streaming metadata prefix encoding. - [x] Implement the private buffered HTTP writer and gateway response conversion. +- [x] Implement accepted public error categories and sanitized diagnostics across + existing private stages, including cleanup and streaming bridge faults. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. - [x] Extract shared invocation ownership and implement the private streaming bridge, with handoff, backpressure, cancellation, cleanup, and error tests. @@ -219,8 +221,9 @@ event cases; no auth or codec behavior depends on accidental SDK dispatch. Design graph prepared on 2026-09-15 in [decision 0012](decisions/0012-shared-identity.md), with [public invocation errors in decision 0013](decisions/0013-public-invocation-errors.md). -Both are proposals awaiting review, not implemented or approved contracts. -The recommended order is caller/context/error foundations, precise claim API and +The user accepted both records as a starting point on 2026-09-15. The explicitly +deferred claim API, resource limits, and producer coverage remain follow-up work. +The accepted order is caller/context/error foundations, precise claim API and native-producer fixtures, then raw/typed buffered invocation. Broader local authentication and authorization follow that shared foundation. diff --git a/errors.go b/errors.go new file mode 100644 index 0000000..3ce586f --- /dev/null +++ b/errors.go @@ -0,0 +1,101 @@ +package edge + +import "errors" + +// Invocation failure categories support errors.Is through wrapping and joined +// errors. Application HTTP responses, including 4xx and 5xx, are not invocation +// failures when the transport completes successfully. +var ( + // ErrInvalidInvocation indicates an invalid direct invocation argument or adapter. + ErrInvalidInvocation = errors.New("edge: invalid invocation") + // ErrInvalidEvent indicates a malformed event or failed HTTP request conversion. + ErrInvalidEvent = errors.New("edge: invalid event") + // ErrUnsupportedEvent indicates an unsupported event family or payload version. + ErrUnsupportedEvent = errors.New("edge: unsupported event") + // ErrIdentity indicates failure to establish or isolate invocation identity. + ErrIdentity = errors.New("edge: identity failure") + // ErrResponse indicates an invalid or unrepresentable HTTP response. + ErrResponse = errors.New("edge: response failure") + // ErrLimitExceeded indicates an adapter-enforced resource limit. + ErrLimitExceeded = errors.New("edge: resource limit exceeded") + // ErrCleanup indicates failure to clean up invocation-owned request resources. + ErrCleanup = errors.New("edge: request cleanup failure") + // ErrStream indicates a stream lifecycle failure or a recovered late panic. + ErrStream = errors.New("edge: stream failure") +) + +// InvocationError describes a sanitized transport failure. Use errors.Is to +// inspect its category and errors.AsType[*InvocationError] for its diagnostics. +// Its text is diagnostic, not a stable identifier or an HTTP response body. +// It does not expose event values, application data, or unsafe underlying errors. +// A zero InvocationError describes an unclassified failure. +// +// Operation failures precede any joined cleanup failures. Parent cancellation +// is returned as the standard context error when there is no operation or cleanup +// failure. Application panics propagated to the runtime are not InvocationErrors. +type InvocationError struct { + operation string + detail string + cause error + limit *invocationLimit +} + +type invocationLimit struct { + name string + maximum int64 +} + +// Error returns a sanitized description of the failed operation. +func (e *InvocationError) Error() string { + if e == nil || e.operation == "" { + return "edge: invocation failed" + } + return "edge: " + e.operation + ": " + e.detail +} + +// Unwrap exposes the failure category and any explicitly preserved safe causes. +func (e *InvocationError) Unwrap() error { + if e == nil { + return nil + } + return e.cause +} + +// Operation identifies the failed operation: validate, decode, request, +// identity, response, encode, cleanup, or stream. A zero error returns "". +func (e *InvocationError) Operation() string { + if e == nil { + return "" + } + return e.operation +} + +// Limit reports the resource name and its effective maximum in bytes for a +// limit failure. Names are response_headers, buffered_body, buffered_envelope, +// and stream_metadata. A response header budget measures the documented charge, +// not wire bytes or heap use. buffered_body also covers base64 expansion and +// the representable HEAD byte count. Other failures return "", 0, false. +func (e *InvocationError) Limit() (name string, maximum int64, ok bool) { + if e == nil || e.limit == nil { + return "", 0, false + } + return e.limit.name, e.limit.maximum, true +} + +// invocationError accepts only package-owned diagnostic strings and causes. +// Never pass parser errors, application errors, or input values here. Error() +// sanitization alone would not make an unsafe Unwrap chain safe. +func invocationError(operation string, category error, detail string, causes ...error) *InvocationError { + cause := category + if len(causes) != 0 { + cause = errors.Join(append([]error{category}, causes...)...) + } + return &InvocationError{operation: operation, detail: detail, cause: cause} +} + +// limitError has the same safe-input requirements as invocationError. +func limitError(operation, name string, maximum int64, reason error) *InvocationError { + err := invocationError(operation, ErrLimitExceeded, "resource limit exceeded", reason) + err.limit = &invocationLimit{name: name, maximum: maximum} + return err +} diff --git a/errors_test.go b/errors_test.go new file mode 100644 index 0000000..21ec49b --- /dev/null +++ b/errors_test.go @@ -0,0 +1,241 @@ +package edge + +import ( + "context" + "errors" + "fmt" + "net/http" + "strings" + "testing" +) + +// Exercise real transport fault sites while the public invocation methods await +// identity producers. Assertions use the exported contract, not private causes. +func TestInvocationErrorCategories(t *testing.T) { + for _, tt := range []struct { + name string + fail func(*testing.T) error + category error + operation string + standard error + }{ + {name: "nil_context", category: ErrInvalidInvocation, operation: "validate", fail: func(t *testing.T) error { + return withInvocation(nil, func(context.Context, *invocation) error { + t.Error("invalid invocation ran application work") + return nil + }) + }}, + {name: "malformed_json", category: ErrInvalidEvent, operation: "decode", fail: func(t *testing.T) error { + _, err := decodeEvent([]byte(`{"credential-secret":`)) + return err + }}, + {name: "unsupported_version", category: ErrUnsupportedEvent, operation: "decode", fail: func(t *testing.T) error { + _, err := decodeEvent([]byte(`{"version":"credential-secret"}`)) + return err + }}, + {name: "invalid_envelope", category: ErrInvalidEvent, operation: "validate", fail: func(t *testing.T) error { + _, err := decodeEvent([]byte(`{}`)) + return err + }}, + {name: "typed_request", category: ErrInvalidEvent, operation: "request", fail: func(t *testing.T) error { + event := v2RequestEvent() + event.RawPath = "/credential-secret%bad%" + _, err := requestV2(t.Context(), event) + return err + }}, + {name: "response_header", category: ErrResponse, operation: "response", fail: func(t *testing.T) error { + _, err := snapshotResponseHeaders(http.Header{"Credential-secret": {"private\nvalue"}}, defaultResponseHeaderBudget) + return err + }}, + {name: "content_length", category: ErrResponse, operation: "response", standard: http.ErrContentLength, fail: func(t *testing.T) error { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Set("Content-Length", "0") + _, err := w.Write([]byte("credential-secret")) + return err + }}, + {name: "short_content_length", category: ErrResponse, operation: "response", standard: http.ErrContentLength, fail: func(t *testing.T) error { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Set("Content-Length", "1") + _, err := w.finish() + return err + }}, + {name: "trailers", category: ErrResponse, operation: "response", standard: http.ErrNotSupported, fail: func(t *testing.T) error { + _, err := snapshotResponseHeaders(http.Header{"Trailer": {"Credential-secret"}}, defaultResponseHeaderBudget) + return err + }}, + {name: "response_codec", category: ErrResponse, operation: "encode", fail: func(t *testing.T) error { + _, err := marshalResponse(map[string]string{"credential-secret": "private\xff"}) + return err + }}, + {name: "cleanup", category: ErrCleanup, operation: "cleanup", fail: func(t *testing.T) error { + return withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + inv.ownRequest(&http.Request{Body: &failingCloseBody{Reader: http.NoBody}}) + return nil + }) + }}, + {name: "stream_unpublished", category: ErrStream, operation: "stream", fail: func(t *testing.T) error { + s, err := startStream(t.Context(), func(context.Context, *invocation) error { return nil }, func(context.Context, *streamOutput) error { return nil }, nil) + if s != nil { + if closeErr := s.Close(); closeErr != nil { + t.Error(closeErr) + } + t.Error("unpublished stream returned a reader") + } + return err + }}, + } { + t.Run(tt.name, func(t *testing.T) { + err := tt.fail(t) + if err == nil { + t.Fatal("transport fault returned nil error") + } + wrapped := fmt.Errorf("caller: %w", err) + if !errors.Is(wrapped, tt.category) || tt.standard != nil && !errors.Is(wrapped, tt.standard) { + t.Errorf("wrapped error = %v, want category %v and standard cause %v", wrapped, tt.category, tt.standard) + } + diagnostic, ok := errors.AsType[*InvocationError](wrapped) + if !ok { + t.Fatalf("error %v has no InvocationError", wrapped) + } + if diagnostic.Operation() != tt.operation { + t.Errorf("Operation() = %q, want %q", diagnostic.Operation(), tt.operation) + } + if name, maximum, ok := diagnostic.Limit(); name != "" || maximum != 0 || ok { + t.Errorf("Limit() = %q, %d, %v for a non-limit failure", name, maximum, ok) + } + // Inspect every exposed cause, including joined siblings. An error's + // own sanitized text must not conceal a credential-bearing child. + pending := []error{wrapped} + for len(pending) != 0 { + node := pending[len(pending)-1] + pending = pending[:len(pending)-1] + if strings.Contains(strings.ToLower(node.Error()), "credential-secret") || strings.Contains(node.Error(), "private") { + t.Errorf("error tree exposes input in %T: %v", node, node) + } + switch node := node.(type) { + case interface{ Unwrap() []error }: + pending = append(pending, node.Unwrap()...) + case interface{ Unwrap() error }: + if inner := node.Unwrap(); inner != nil { + pending = append(pending, inner) + } + } + } + }) + } +} + +func TestInvocationErrorLimits(t *testing.T) { + for _, tt := range []struct { + name string + fail func(*testing.T) error + operation string + limit string + maximum int64 + }{ + {name: "original_headers", operation: "response", limit: "response_headers", maximum: 32, fail: func(t *testing.T) error { + _, err := snapshotResponseHeaders(http.Header{"A": nil}, 32) + return err + }}, + {name: "generated_headers", operation: "response", limit: "response_headers", maximum: 33, fail: func(t *testing.T) error { + h := mustSnapshot(t, http.Header{"A": nil}, 33) + return h.automatic("Content-Length", "0") + }}, + {name: "body_write", operation: "response", limit: "buffered_body", maximum: maxResponseBytes, fail: func(t *testing.T) error { + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + _, err := w.Write(make([]byte, maxResponseBytes+1)) + return err + }}, + {name: "binary_expansion", operation: "response", limit: "buffered_body", maximum: maxResponseBytes, fail: func(t *testing.T) error { + _, _, err := encodeResponseBody(make([]byte, maxResponseBytes/4*3+1), nil) + return err + }}, + {name: "envelope", operation: "encode", limit: "buffered_envelope", maximum: maxResponseBytes, fail: func(t *testing.T) error { + _, err := marshalResponse(map[string]string{"body": strings.Repeat("x", maxResponseBytes)}) + return err + }}, + {name: "stream_metadata", operation: "encode", limit: "stream_metadata", maximum: maxStreamPrefixBytes, fail: func(t *testing.T) error { + h := mustSnapshot(t, http.Header{"Large": {strings.Repeat("x", maxStreamPrefixBytes)}}, defaultResponseHeaderBudget) + _, err := encodeStreamPrefix(http.StatusOK, h) + return err + }}, + } { + t.Run(tt.name, func(t *testing.T) { + err := tt.fail(t) + if !errors.Is(err, ErrLimitExceeded) { + t.Fatalf("limit failure = %v, want ErrLimitExceeded", err) + } + diagnostic, ok := errors.AsType[*InvocationError](err) + if !ok { + t.Fatalf("error %v has no InvocationError", err) + } + name, maximum, ok := diagnostic.Limit() + if !ok || name != tt.limit || maximum != tt.maximum || diagnostic.Operation() != tt.operation { + t.Errorf("limit diagnostic = %q, %d, %v, %q; want %q, %d, true, %q", name, maximum, ok, diagnostic.Operation(), tt.limit, tt.maximum, tt.operation) + } + }) + } +} + +func TestInvocationErrorJoinedCleanup(t *testing.T) { + parent, cancel := context.WithCancel(t.Context()) + defer cancel() + err := withInvocation(parent, func(ctx context.Context, inv *invocation) error { + inv.ownRequest(&http.Request{Body: &failingCloseBody{Reader: http.NoBody}}) + cancel() + w := newBufferedWriter(http.MethodGet, defaultResponseHeaderBudget) + w.Header().Set("Content-Length", "0") + _, err := w.Write([]byte("x")) + return err + }) + wrapped := fmt.Errorf("caller: %w", err) + if !errors.Is(wrapped, ErrResponse) || !errors.Is(wrapped, http.ErrContentLength) || !errors.Is(wrapped, ErrCleanup) || errors.Is(wrapped, context.Canceled) { + t.Errorf("joined failure = %v, want response/length and cleanup without cancellation", wrapped) + } + if diagnostic, ok := errors.AsType[*InvocationError](wrapped); !ok || diagnostic.Operation() != "response" { + t.Errorf("primary diagnostic = %v, %v; want response before cleanup", diagnostic, ok) + } +} + +func TestInvocationCancellationDoesNotExposeCause(t *testing.T) { + parent, cancel := context.WithCancelCause(t.Context()) + private := errors.New("credential-secret") + cancel(private) + err := withInvocation(parent, func(context.Context, *invocation) error { + t.Error("canceled invocation ran application work") + return nil + }) + if !errors.Is(err, context.Canceled) || errors.Is(err, private) || strings.Contains(err.Error(), "credential-secret") { + t.Errorf("cancellation = %v, want only the standard cancellation cause", err) + } +} + +func TestHTTPFailureResponsesRemainSuccessfulInvocations(t *testing.T) { + for _, status := range []int{http.StatusUnauthorized, http.StatusForbidden, http.StatusInternalServerError, http.StatusServiceUnavailable} { + err := withInvocation(t.Context(), func(ctx context.Context, inv *invocation) error { + r, err := requestV2(ctx, v2RequestEvent()) + if err != nil { + return err + } + inv.ownRequest(r) + response, err := serveBuffered(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Error(w, "application failure", status) + }), r, defaultResponseHeaderBudget) + if err != nil { + return err + } + typed, err := response.v2() + if err != nil { + return err + } + if typed.StatusCode != status { + t.Errorf("HTTP status = %d, want %d", typed.StatusCode, status) + } + _, err = marshalResponse(typed) + return err + }) + if err != nil { + t.Errorf("HTTP %d response produced invocation error: %v", status, err) + } + } +} diff --git a/invocation.go b/invocation.go index 435a7f4..72504a9 100644 --- a/invocation.go +++ b/invocation.go @@ -26,12 +26,12 @@ func (inv *invocation) ownRequest(r *http.Request) { func (inv *invocation) cleanup(err error) error { if inv.body != nil { if closeErr := inv.body.Close(); closeErr != nil { - err = errors.Join(err, errors.New("edge: request body cleanup failed")) + err = errors.Join(err, invocationError("cleanup", ErrCleanup, "request body cleanup failed")) } } if inv.request != nil && inv.request.MultipartForm != nil { if removeErr := inv.request.MultipartForm.RemoveAll(); removeErr != nil { - err = errors.Join(err, errors.New("edge: multipart cleanup failed")) + err = errors.Join(err, invocationError("cleanup", ErrCleanup, "multipart cleanup failed")) } } return err @@ -39,7 +39,7 @@ func (inv *invocation) cleanup(err error) error { func invocationContext(parent context.Context) (context.Context, context.CancelFunc, error) { if parent == nil { - return nil, nil, errors.New("edge: nil invocation context") + return nil, nil, invocationError("validate", ErrInvalidInvocation, "nil invocation context") } if err := parent.Err(); err != nil { return nil, nil, err diff --git a/request.go b/request.go index f5a792a..253359e 100644 --- a/request.go +++ b/request.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "encoding/base64" - "errors" "io" "maps" "net/http" @@ -22,7 +21,7 @@ func requestV1(ctx context.Context, event events.APIGatewayProxyRequest) (*http. return nil, err } if !strings.HasPrefix(event.Path, "/") { - return nil, errors.New("edge: path must begin with a slash") + return nil, invocationError("request", ErrInvalidEvent, "path must begin with a slash") } query := make(url.Values) for key, values := range event.MultiValueQueryStringParameters { @@ -53,23 +52,23 @@ func requestV2(ctx context.Context, event events.APIGatewayV2HTTPRequest) (*http return nil, err } if !strings.HasPrefix(event.RawPath, "/") { - return nil, errors.New("edge: rawPath must begin with a slash") + return nil, invocationError("request", ErrInvalidEvent, "rawPath must begin with a slash") } path, err := url.PathUnescape(event.RawPath) if err != nil { - return nil, errors.New("edge: invalid rawPath encoding") + return nil, invocationError("request", ErrInvalidEvent, "invalid rawPath encoding") } u := &url.URL{Path: path, RawPath: event.RawPath, RawQuery: event.RawQueryString} // EscapedPath ignores an invalid RawPath hint. Reject rather than silently // replacing raw delimiters, whitespace, or an invalid encoded spelling. if u.EscapedPath() != event.RawPath { - return nil, errors.New("edge: invalid rawPath syntax") + return nil, invocationError("request", ErrInvalidEvent, "invalid rawPath syntax") } // Query decoding remains the handler's job, but a request target cannot // contain literal ASCII whitespace or control bytes on an HTTP request line. for i := 0; i < len(event.RawQueryString); i++ { if event.RawQueryString[i] <= ' ' || event.RawQueryString[i] == 0x7f { - return nil, errors.New("edge: invalid raw query syntax") + return nil, invocationError("request", ErrInvalidEvent, "invalid raw query syntax") } } header, err := requestHeaders(event.Headers, nil) @@ -79,7 +78,7 @@ func requestV2(ctx context.Context, event events.APIGatewayV2HTTPRequest) (*http if len(event.Cookies) != 0 { cookie := strings.Join(event.Cookies, "; ") if !validHeaderValue(cookie) { - return nil, errors.New("edge: invalid cookie header") + return nil, invocationError("request", ErrInvalidEvent, "invalid cookie header") } header.Set("Cookie", cookie) } @@ -97,12 +96,12 @@ func requestHeaders(single map[string]string, multi map[string][]string) (http.H header := make(http.Header) for _, key := range slices.Sorted(maps.Keys(multi)) { if !validToken(key) { - return nil, errors.New("edge: invalid header name") + return nil, invocationError("request", ErrInvalidEvent, "invalid header name") } name := http.CanonicalHeaderKey(key) for _, value := range multi[key] { if !validHeaderValue(value) { - return nil, errors.New("edge: invalid header value") + return nil, invocationError("request", ErrInvalidEvent, "invalid header value") } header[name] = append(header[name], value) } @@ -112,11 +111,11 @@ func requestHeaders(single map[string]string, multi map[string][]string) (http.H multiValues := header.Clone() for _, key := range slices.Sorted(maps.Keys(single)) { if !validToken(key) { - return nil, errors.New("edge: invalid header name") + return nil, invocationError("request", ErrInvalidEvent, "invalid header name") } value := single[key] if !validHeaderValue(value) { - return nil, errors.New("edge: invalid header value") + return nil, invocationError("request", ErrInvalidEvent, "invalid header value") } name := http.CanonicalHeaderKey(key) if !slices.Contains(multiValues[name], value) { @@ -128,21 +127,21 @@ func requestHeaders(single map[string]string, multi map[string][]string) (http.H func newRequest(ctx context.Context, r *http.Request, body string, binary bool) (*http.Request, error) { if ctx == nil { - return nil, errors.New("edge: nil invocation context") + return nil, invocationError("validate", ErrInvalidInvocation, "nil invocation context") } if !validToken(r.Method) { - return nil, errors.New("edge: invalid HTTP method") + return nil, invocationError("request", ErrInvalidEvent, "invalid HTTP method") } host := r.Host header := r.Header if hosts, present := header["Host"]; present { if len(hosts) != 1 || hosts[0] == "" { - return nil, errors.New("edge: invalid Host header") + return nil, invocationError("request", ErrInvalidEvent, "invalid Host header") } host = hosts[0] } if host != "" && !validAuthority(host) { - return nil, errors.New("edge: invalid request host") + return nil, invocationError("request", ErrInvalidEvent, "invalid request host") } delete(header, "Host") if r.Proto == "" { @@ -150,14 +149,14 @@ func newRequest(ctx context.Context, r *http.Request, body string, binary bool) } major, minor, ok := http.ParseHTTPVersion(r.Proto) if !ok { - return nil, errors.New("edge: invalid HTTP protocol") + return nil, invocationError("request", ErrInvalidEvent, "invalid HTTP protocol") } var data []byte if binary { var err error data, err = base64.StdEncoding.DecodeString(body) if err != nil { - return nil, errors.New("edge: invalid base64 request body") + return nil, invocationError("request", ErrInvalidEvent, "invalid base64 request body") } } else { data = []byte(body) diff --git a/response_encoding.go b/response_encoding.go index 6b8d835..21026ac 100644 --- a/response_encoding.go +++ b/response_encoding.go @@ -22,10 +22,10 @@ func marshalResponse(response any) ([]byte, error) { var out responseBuffer if err := json.MarshalWrite(&out, response); err != nil { if errors.Is(err, errResponseTooLarge) { - return nil, errResponseTooLarge + return nil, limitError("encode", "buffered_envelope", maxResponseBytes, errResponseTooLarge) } // Codec errors can include application values. Do not expose that chain. - return nil, errors.New("edge: response JSON encoding failed") + return nil, invocationError("encode", ErrResponse, "response JSON encoding failed") } return out.data, nil } @@ -61,7 +61,7 @@ func (b *responseBuffer) Write(p []byte) (int, error) { // changes only the gateway envelope representation, never the content bytes. func encodeResponseBody(body []byte, header http.Header) (string, bool, error) { if len(body) > maxResponseBytes { - return "", false, errResponseTooLarge + return "", false, limitError("response", "buffered_body", maxResponseBytes, errResponseTooLarge) } if len(body) == 0 { return "", false, nil @@ -70,7 +70,7 @@ func encodeResponseBody(body []byte, header http.Header) (string, bool, error) { return string(body), false, nil } if base64.StdEncoding.EncodedLen(len(body)) > maxResponseBytes { - return "", false, errResponseTooLarge + return "", false, limitError("response", "buffered_body", maxResponseBytes, errResponseTooLarge) } return base64.StdEncoding.EncodeToString(body), true, nil } diff --git a/response_headers.go b/response_headers.go index fd04041..3e07398 100644 --- a/response_headers.go +++ b/response_headers.go @@ -2,7 +2,6 @@ package edge import ( "errors" - "fmt" "net/http" "slices" "strconv" @@ -14,10 +13,10 @@ const defaultResponseHeaderBudget = 256 * 1024 var ( errResponseHeaderBudget = errors.New("edge: response header budget exceeded") - errResponseHeaders = errors.New("edge: invalid response headers") - errResponseTrailers = fmt.Errorf("edge: response trailers: %w", http.ErrNotSupported) - errResponseUpgrade = fmt.Errorf("edge: response upgrade: %w", http.ErrNotSupported) - errResponseHeaderRepeat = errors.New("edge: repeated response field cannot be represented in payload 2.0") + errResponseHeaders = invocationError("response", ErrResponse, "invalid response headers") + errResponseTrailers = invocationError("response", ErrResponse, "response trailers are unsupported", http.ErrNotSupported) + errResponseUpgrade = invocationError("response", ErrResponse, "response upgrade is unsupported", http.ErrNotSupported) + errResponseHeaderRepeat = invocationError("response", ErrResponse, "repeated response field cannot be represented in payload 2.0") ) // responseHeaders owns its map and value slices. Strings are immutable and may @@ -26,6 +25,7 @@ var ( type responseHeaders struct { fields http.Header remaining int + budget int noContentType bool noContentLength bool } @@ -54,11 +54,11 @@ func chargeResponseHeaders(fields http.Header, remaining int) (int, error) { func snapshotResponseHeaders(fields http.Header, budget int) (*responseHeaders, error) { if budget <= 0 || budget > maxResponseBytes { - return nil, errors.New("edge: invalid response header budget") + return nil, invocationError("validate", ErrInvalidInvocation, "invalid response header budget") } remaining, err := chargeResponseHeaders(fields, budget) if err != nil { - return nil, err + return nil, limitError("response", "response_headers", int64(budget), errResponseHeaderBudget) } keys := make([]string, 0, len(fields)) for name, values := range fields { @@ -76,7 +76,7 @@ func snapshotResponseHeaders(fields http.Header, budget int) (*responseHeaders, keys = append(keys, name) } slices.Sort(keys) - h := &responseHeaders{fields: make(http.Header, len(fields)), remaining: remaining} + h := &responseHeaders{fields: make(http.Header, len(fields)), remaining: remaining, budget: budget} for _, key := range keys { name := http.CanonicalHeaderKey(key) values := slices.Grow(h.fields[name], len(fields[key])) @@ -142,7 +142,7 @@ func (h *responseHeaders) automatic(name, value string) error { } remaining, err := chargeResponseHeaders(http.Header{name: {value}}, h.remaining) if err != nil { - return err + return limitError("response", "response_headers", int64(h.budget), errResponseHeaderBudget) } h.fields[name] = []string{value} h.remaining = remaining diff --git a/response_headers_test.go b/response_headers_test.go index c1591b4..35e9aec 100644 --- a/response_headers_test.go +++ b/response_headers_test.go @@ -158,8 +158,10 @@ func TestResponseHeaderBudgetAndAutomaticFields(t *testing.T) { } } tooMany := http.Header{"A": make([]string, defaultResponseHeaderBudget/33+1)} + // Charging must not copy rejected input. The caller now separately allocates + // a bounded InvocationError describing the configured limit. if allocations := testing.AllocsPerRun(10, func() { - _, _ = snapshotResponseHeaders(tooMany, defaultResponseHeaderBudget) + _, _ = chargeResponseHeaders(tooMany, defaultResponseHeaderBudget) }); allocations != 0 { t.Errorf("over-budget preflight allocated %g objects; want none", allocations) } diff --git a/stream.go b/stream.go index eefbc49..66687fd 100644 --- a/stream.go +++ b/stream.go @@ -9,9 +9,9 @@ import ( ) var ( - errStreamPanic = errors.New("edge: stream producer panicked") - errStreamReporter = errors.New("edge: stream error reporter panicked") - errStreamUnpublished = errors.New("edge: stream producer ended without publishing a response") + errStreamPanic = invocationError("stream", ErrStream, "stream producer panicked") + errStreamReporter = invocationError("stream", ErrStream, "stream error reporter panicked") + errStreamUnpublished = invocationError("stream", ErrStream, "stream producer ended without publishing a response") ) // streamOutput belongs to the producer. publish accepts an already validated, @@ -256,5 +256,5 @@ func (*responseStream) ContentType() string { // MarshalJSON deliberately declines serialization so the SDK selects its // io.Reader response path. Edge's framing codec uses JSON v2 separately. func (*responseStream) MarshalJSON() ([]byte, error) { - return nil, errors.New("edge: streaming response cannot be marshaled as JSON") + return nil, invocationError("encode", ErrStream, "streaming response cannot be marshaled as JSON") } diff --git a/stream_encoding.go b/stream_encoding.go index c6009f8..31cba3e 100644 --- a/stream_encoding.go +++ b/stream_encoding.go @@ -13,7 +13,7 @@ var errStreamPrefixTooLarge = errors.New("edge: response metadata exceeds stream func encodeStreamPrefix(status int, headers *responseHeaders) ([]byte, error) { if status < 200 || status > 599 { - return nil, errors.New("edge: invalid final response status") + return nil, invocationError("response", ErrResponse, "invalid final response status") } metadata := struct { StatusCode int `json:"statusCode"` @@ -22,9 +22,9 @@ func encodeStreamPrefix(status int, headers *responseHeaders) ([]byte, error) { out := responseBuffer{limit: maxStreamPrefixBytes - 8} if err := json.MarshalWrite(&out, metadata); err != nil { if errors.Is(err, errResponseTooLarge) { - return nil, errStreamPrefixTooLarge + return nil, limitError("encode", "stream_metadata", maxStreamPrefixBytes, errStreamPrefixTooLarge) } - return nil, errors.New("edge: streaming metadata JSON encoding failed") + return nil, invocationError("encode", ErrResponse, "streaming metadata JSON encoding failed") } // Allocate exactly the framed length. The final zeroed bytes are the delimiter. prefix := make([]byte, len(out.data)+8) diff --git a/stream_test.go b/stream_test.go index 6dc425a..84efeb3 100644 --- a/stream_test.go +++ b/stream_test.go @@ -408,6 +408,14 @@ func TestStreamLateFailurePreservesBytesAndReportsAfterCleanup(t *testing.T) { if mode == "panic" && !errors.Is(err, errStreamPanic) { t.Errorf("late panic = %v, want stream panic error", err) } + if mode == "panic" || mode == "reporter_panic" { + if diagnostic, ok := errors.AsType[*InvocationError](err); !errors.Is(err, ErrStream) || !ok || diagnostic.Operation() != "stream" { + t.Errorf("late panic classification = %v; want ErrStream with stream diagnostic", err) + } + } + if mode == "cleanup_error" && (!errors.Is(err, ErrCleanup) || !errors.Is(reported, ErrCleanup)) { + t.Errorf("late cleanup classification = %v; reported %v; want ErrCleanup on both", err, reported) + } if mode != "panic" && mode != "cleanup_error" && !errors.Is(err, http.ErrContentLength) { t.Errorf("terminal error lost primary fault: %v", err) } From 383753ddd220777a2139b744054e0f95df5ec30b Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 06:51:26 +0100 Subject: [PATCH 22/44] docs: propose claims API with measured resource limits --- docs/benchmarks.md | 55 +++++ docs/decisions/0014-claims-api.md | 231 +++++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 6 + internal/claimprobe/probe_test.go | 327 ++++++++++++++++++++++++++++++ 5 files changed, 620 insertions(+) create mode 100644 docs/decisions/0014-claims-api.md create mode 100644 internal/claimprobe/probe_test.go diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 120784c..e24e865 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -111,3 +111,58 @@ and projections; it is not peak memory or RSS. Median snapshot timings were approximately 1.36 µs, 1.27 ms and 52.17 ms respectively, with substantial timing variation in the largest case. These are exploratory samples, not CI thresholds or Lambda latency claims. They do not warrant changing the approved default. + +## Candidate claims storage (2026-09-16) + +The isolated `internal/claimprobe` test package measures a candidate owned tree, +not an implemented identity API. Go 1.27.1, darwin/arm64, Apple M2; three samples: + +```sh +go test ./internal/claimprobe -run '^TestSnapshotMemory$' -v -count=3 +go test ./internal/claimprobe -run '^$' -bench 'Benchmark(Snapshot|JSONScan)$' -benchmem -benchtime=100ms -count=3 +``` + +Median snapshot results, with retained heap measured separately: + +| Fixture | Weighted charge (bytes) | Time | Cumulative B/op | Allocs/op | Retained heap (bytes) | +| --- | ---: | ---: | ---: | ---: | ---: | +| Ordinary claims | 1,185 | 1.25 µs | 2,691 | 29 | 2,696 | +| Distinct names, 256 KiB allowance | 262,144 | 266 µs | 750,387 | 3,658 | 750,384 | +| Distinct names, 1 MiB allowance | 1,048,528 | 1.19 ms | 3,001,434 | 14,628 | 3,001,424 | +| Distinct names, 6 MiB allowance | 6,291,424 | 8.63 ms | 12,238,702 | 87,638 | 12,238,688 | +| Array of nulls | 262,085 | 56.1 µs | 262,902 | 4 | 262,912 | +| Array of singleton objects | 262,132 | 599 µs | 1,661,213 | 6,097 | 1,661,184 | +| Large string | 262,144 | 23.3 µs | 262,902 | 4 | 262,912 | +| 64 nested objects | 4,417 | 16.5 µs | 48,388 | 193 | 48,400 | + +Depth 65, a cyclic map, an exponentially expanded shared subtree and an oversized +array all reject before snapshot allocation: 0 B/op and 0 allocs/op in these +samples. This excludes any eventual public error construction. Shared subtrees +are charged per occurrence, not per distinct pointer. Rejection of that fixture +took about 20.6 µs; the container-length fast rejection took about 46 ns. + +The separate lexical JSON scan used direct jsontext, including strict duplicate +name validation, a wire-length check, an object root and a depth bound. Median +cumulative allocations were 1,200 bytes for ordinary claims, 692,717 for distinct +names, 524,696 for a large string and 12,904 for depth 64. These are lexical scan +costs only: no full raw-JSON-to-owned-tree implementation exists yet. Do not add +the timings to predict production latency or call lexical allocation a heap cap. + +The tree copies map keys, strings, maps and slices. Inputs remain live across +memory samples. Two garbage collections before and after each snapshot reduce +pool retention; small deltas remain noisy. Retained heap excludes input storage, +transient allocation, normalized JWT views, event decoding, and process RSS. The +singleton-object case shows why a 256 KiB accounting limit can retain 1.58 MiB. +No sample establishes the worst possible amplification across all shapes. + +The prototype covers a subset of the proposed typed inputs, using float64 for +decoded numbers. It does not implement exact numeric nodes, provenance, the +public getters, JWT validation, error translation, or dedicated scope accounting. +Those require production measurements and contract tests after approval. + +The SDK probe invokes a real local lambda.NewHandlerWithOptions around a typed +APIGatewayProxyRequest. For the literal 9007199254740993, default decoding yields +float64 9007199254740992; WithUseNumber(true) yields json.Number with the original +digits. This is evidence for data-type compatibility in +[decision 0014](decisions/0014-claims-api.md), not a proposal to use the v1 codec +inside edge. No live Lambda or API Gateway request was made. diff --git a/docs/decisions/0014-claims-api.md b/docs/decisions/0014-claims-api.md new file mode 100644 index 0000000..7c54645 --- /dev/null +++ b/docs/decisions/0014-claims-api.md @@ -0,0 +1,231 @@ +# 0014: Owned claims, numeric fidelity, and resource limits + +Status: proposed on 2026-09-16; awaiting review. Production identity APIs are +unchanged. This completes the claims follow-up explicitly left open by 0012. + +## Evidence and scope + +The raw decoder keeps authorizer JSON; SDK V1 events contain map[string]any and +SDK V2 JWT claims contain map[string]string. Typed entry points must preserve +their available data without an envelope JSON round trip. The identity package +must remain independent of AWS. Existing invocation errors report byte limits; +their Limit method cannot accurately report a container-depth limit. + +We reviewed the Go rules, Go 1.27 release notes, installed package documentation, +SDK v1.55.0 behavior, and AWS documentation through public AWS MCP. The isolated +[probe](../../internal/claimprobe/probe_test.go) and +[measurements](../benchmarks.md#candidate-claims-storage-2026-09-16) inform resource +policy. The probe is deliberately not a production claims implementation. + +Sources: + +- [Go 1.27](https://go.dev/doc/go1.27) and [JSON v2](https://pkg.go.dev/encoding/json/v2) +- [JSON token representation](https://pkg.go.dev/encoding/json/jsontext#Token.String) +- [SDK WithUseNumber](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.55.0/lambda#WithUseNumber) +- [Standard numeric data type](https://pkg.go.dev/encoding/json#Number) +- [API Gateway payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) +- [Gateway JWT validation](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-jwt-authorizer.html) +- [Cognito token verification](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html) +- [JWT claim definitions](https://www.rfc-editor.org/rfc/rfc7519.html) +- [OAuth scope syntax](https://www.rfc-editor.org/rfc/rfc6749#section-3.3) + +## C1. Small concrete API with checked reads + +Recommend these declarations; they illustrate the proposal, not existing APIs: + +```go +func ParseClaims(data jsontext.Value, opts ...ClaimsOption) (Claims, error) +func NewClaims(values map[string]any, opts ...ClaimsOption) (Claims, error) +func NewTextClaims(values map[string]string, opts ...ClaimsOption) (Claims, error) +func WithClaimsBudget(bytes int) ClaimsOption + +func (c Claims) Len() int +func (c Claims) Names() []string +func (c Claims) Lookup(name string) (Claim, bool) + +func (c Claim) Kind() ClaimKind +func (c Claim) Representation() Representation +func (c Claim) Text() (string, bool) +func (c Claim) Bool() (bool, bool) +func (c Claim) Array() ([]Claim, bool) +func (c Claim) Object() (Claims, bool) +func (c Claim) NumberText() (string, bool) +func (c Claim) Float64() (float64, bool) +``` + +Claims and Claim have private fields. Zero Claims is an empty object; zero Claim +is invalid, distinct from a present JSON null. ClaimKind distinguishes invalid, +null, boolean, string, number, array and object. Representation distinguishes +unknown, received JSON, supplied decoded values, and gateway text. It describes +the immediate representation, not the original token or verification strength; +Caller.Source continues to describe the identity producer. + +Lookup reports presence; the subsequent checked accessor reports whether that +read is possible. Thus missing, null, and wrong type remain distinguishable +without allocating errors for routine inspection. Names returns sorted names; +Array returns a new slice of immutable views. Object shares immutable backing +data. No getter returns a writable internal map or slice. + +NumberText succeeds for exact numeric representations, including native integer +inputs, and fails for supplied floating-point inputs even if integral. It never +parses strings. Float64 is an explicitly approximate conversion; it returns +false for nonnumbers or overflow/underflow reported by the standard conversion. +Ordinary representable rounding is permitted. Exact consumers use NumberText and +their chosen numeric parser. Text is deliberately not named String: String() has +an established diagnostic convention in Go. + +Reasoning: this supports custom claims without turning identity into a generic +JSON binding library. Reject generic getters, automatic coercion, arbitrary +Decode-into-target hooks, and a mutable map export. A convenience Strings method +can follow demonstrated consumer need; normalized JWT collections already have +their own comma-ok accessors under 0012. + +## C2. Accept SDK number preservation without using its codec + +The actual SDK handler probe sends the number 9007199254740993 through a typed +V1 event. Default SDK decoding supplies float64 9007199254740992; +lambda.WithUseNumber(true) supplies json.Number containing the original digits. +No downstream adapter can recover the lost digit in the default case. + +Recommend accepting encoding/json.Number as an explicitly recognized input +DATA TYPE. This requires a narrow encoding/json import, exclusively to identify +the type and convert its underlying string. Validate that string as exactly one +JSON number using jsontext; never call the v1 encoder, decoder, or compatibility +functions. This preserves the user's direct-v2 processing requirement while +supporting the AWS SDK's existing number-preserving option. + +NewClaims accepts only nil, bool, string, built-in signed/unsigned integer types +(excluding uintptr), finite float32/float64, json.Number, map[string]any, +map[string]string, []any and []string, recursively. Custom defined types, +pointers, structs, []byte and custom marshalers are rejected. Type aliases work +as their underlying identical Go type. No arbitrary String or Marshal method is +invoked. Nested string maps passed to NewClaims remain decoded values; +NewTextClaims specifically labels gateway text. + +Alternative: reject json.Number and make callers recursively translate SDK +values. That adds work at the exact typed boundary we promised to simplify. +Accepting arbitrary fmt.Stringer values would invoke user code and mistake text +for numbers. Neither is recommended. + +Raw JSON preserves numeric token text, semantic JSON types, and decoded string +contents. It does not preserve whitespace, object order, or escape spelling. +This representation must never be substituted for JWT signing bytes. ParseClaims +accepts exactly one object and rejects duplicates, invalid UTF-8, null roots and +trailing values. Typed nil root maps mean empty objects; nested typed nil maps +and slices retain their declared object/array shape as empty collections. A nil +interface is null. This is a declared typed-input contract, not reconstruction +of the upstream JSON null that might have produced a nil container. + +## C3. Own inputs and bound expansion before copying + +Recommend a 256 KiB default weighted budget, configurable to a positive value +no greater than 6 MiB. There is no unlimited setting. This is an engineering +policy, not an AWS JWT-size quota. The upper setting is an explicit application +opt-in and is not a claim that this much authentication data is ordinary. + +Charge 64 bytes for every value occurrence, including root, containers and null, +plus UTF-8 bytes of object names and string values, plus retained exact numeric +text. Native integers charge their decimal representation. Floating-point +storage is covered by the node charge. Count repeated subtrees each time they +will be copied. Unknown claims count equally. Do not refund duplicate values or +claim data omitted from normalized views. + +ParseClaims independently limits input byte length to the same configured +maximum before constructing a decoder, then applies the weighted tree budget. +Whitespace and escapes can therefore exhaust its wire allowance even when the +equivalent typed input fits. Typed input has no available original wire length; +we do not manufacture one by marshaling it. + +Limit nesting to 64 containers, counting the root as one. This also bounds cyclic +typed input without an allocation-heavy graph traversal. Use subtractive checked +accounting and container-length lower bounds before traversing or allocating. +Preflight typed data before copying, clone strings as well as containers so a +small substring cannot retain an unrelated large backing buffer, and never call +user methods. Inputs must not change concurrently during construction. After +construction, callers may mutate their originals and share claims concurrently. + +The candidate's ordinary fixture retained about 2.6 KiB. At a 256 KiB charge, +distinct names retained about 0.72 MiB and many singleton objects about 1.58 MiB. +Thus the charge is not a heap cap. JSON lexical validation, normalized JWT views, +source event storage and the rest of an invocation cost additional memory. +Decoder duplicate detection also allocates before the entire tree is built. +Keep the public accounting stable while permitting private storage improvements; +repeat allocation measurements against production code before public wiring. + +Dedicated gateway scopes must not bypass this budget. Propose +edge.WithIdentityClaimsBudget(bytes int) for gateway construction, and carry the +configured budget and consumed charge privately in Claims. NewJWT's explicitly +supplied scope collection consumes the remaining allowance as an array plus its +strings, even when it agrees with a scope claim. Derived normalized views reuse +immutable strings where possible; their allocation is bounded by the input but +is not part of the advertised heap measurement. Supplying duplicate JWT options +still fails under 0012. Claims resource options are configuration, with last +assignment winning like edge options; nil or invalid options fail construction. + +Alternative: a raw-byte-only limit misses large typed graphs; an entry-count-only +limit misses large values. A universal heap-byte promise is not supported by the +measurements. Arbitrarily deep or unlimited copying is not recommended. + +## C4. Preserve safe, correctly dimensioned errors + +Recommend identity.ErrInvalidClaims for malformed JSON, unsupported typed data, +invalid UTF-8, nonfinite numbers and excessive nesting. Recommend +identity.ErrClaimsLimit for wire/weighted byte exhaustion, with an opaque +*identity.ClaimsLimitError exposing Maximum() int64 and unwrapping only to that +sentinel. Construction errors contain no claim names, values, JSON fragments or +wrapped codec errors. Invalid configuration remains a construction error. + +At the invocation boundary, claim failure becomes operation identity with +edge.ErrIdentity. Byte exhaustion additionally matches edge.ErrLimitExceeded +and reports Limit() = ("identity_claims", configuredMaximum, true). Depth failure +has no byte-limit metadata: reporting “64 bytes” would be incorrect. Keep detailed +identity errors within identity and translate deliberately at the edge boundary; +do not introduce an AWS or edge dependency into identity. + +## C5. Preserve the gateway qualifications while normalizing JWT facts + +Received JSON fidelity does not establish original JWT fidelity. In particular, +a gateway-origin string aud or cognito:groups remains available through Text, +but a normalized collection remains unavailable when the source may have +flattened it. Actual arrays can be interpreted according to the claim schema. +This can cause a group/audience-dependent application rule to deny a legitimate +gateway-authenticated request. Recommend accepting that consequence rather than +inventing an undocumented delimiter or trusting an unverified bearer payload. +Local verification or an explicitly reviewed source schema is the remedy. + +Interpret canonical scope strings using OAuth's ASCII-space token grammar, +without Unicode-whitespace splitting or case changes. Preserve the original +claim and deduplicate normalized sets deterministically. A present dedicated +gateway scopes array is separately interpretable; an empty array means known +empty, while absent/null means unavailable. Typed SDK nil loses any distinction +between absent and null; a nonnil empty slice preserves known empty. + +AWS documents support for scope or scp in gateway authorization but does not give +a universal reversible flattened scp encoding. Recommend recognizing an actual +string array for scp; retain ambiguous gateway text without manufacturing a list. +Any broader issuer-specific scp string convention needs an explicit schema. +Wrong actual types fail under 0012; ambiguity caused by flattening is unavailable. + +Retain the accepted rule: prefer dedicated scopes, compare independently +interpretable scope sets, reject disagreement, never union. This is OUR strict +consistency policy, not an AWS-documented guarantee that all authorizer variants +produce equivalent collections. The checked docs establish route validation and +show a scopes array; they do not settle every projection/filtering detail. +Record this as a compatibility qualification in native-producer fixtures. Do not +broaden comparisons to opaque scp text or claim universal gateway coverage. + +## Review and implementation order + +Recommend approval of C1–C4, and confirmation of C5's explicit compatibility +tradeoffs. After approval: + +1. Implement the AWS-independent claims/caller/context foundation, with ownership, + numerical-fidelity, budget, cycle, type, and sanitized-error contract tests. +2. Specify the native-producer fixture matrix against documented event shapes; + expose unresolved IAM/V1 producer-recognition details before implementing them. +3. Implement reviewed gateway producers and wire the raw/typed buffered adapter. +4. Follow with local verification/authz and streaming HTTP writer work in the + accepted order. Do not pick a JWT library or issuer profile in this claims step. + +No new production APIs, claim normalization behavior, or resource options have +been implemented by this proposal. No live AWS resources were invoked. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 6779155..e942227 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -22,3 +22,4 @@ Keep related decisions small enough to review and implement independently. | [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | | [0012: Shared identity](0012-shared-identity.md) | Accepted; detailed claims follow-up pending | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | +| [0014: Claims API](0014-claims-api.md) | Proposed | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | diff --git a/docs/plan.md b/docs/plan.md index 6a761c0..b13abe9 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -227,6 +227,12 @@ The accepted order is caller/context/error foundations, precise claim API and native-producer fixtures, then raw/typed buffered invocation. Broader local authentication and authorization follow that shared foundation. +[Decision 0014](decisions/0014-claims-api.md), proposed on 2026-09-16, specifies the +claims constructor/accessor surface, SDK number-preserving input compatibility, +ownership and weighted resource limits, and gateway normalization qualifications. +An isolated storage/SDK probe is implemented and measured; production identity +APIs remain unchanged while these recommendations await review. + - Review caller constructors, immutable/owned data, validation, anonymous state, and context-presence semantics. - Preserve issuer plus subject for JWT application identity; do not treat a diff --git a/internal/claimprobe/probe_test.go b/internal/claimprobe/probe_test.go new file mode 100644 index 0000000..00b2bee --- /dev/null +++ b/internal/claimprobe/probe_test.go @@ -0,0 +1,327 @@ +// Package claimprobe measures a candidate owned claim tree. Nothing in edge +// imports it. This is a storage experiment, not an identity implementation: +// provenance, full input-type support, JWT projection and public APIs are absent. +package claimprobe + +import ( + "bytes" + "context" + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + "fmt" + "io" + "math" + "runtime" + "strings" + "testing" + "unicode/utf8" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-lambda-go/lambda" +) + +const ( + defaultBudget = 256 * 1024 + maxDepth = 64 + nodeCharge = 64 +) + +var errRejected = errors.New("probe input rejected") + +type node struct { + kind byte + origin byte + text string + number float64 + array []node + object map[string]node +} + +func charge(remaining *int, n int) bool { + if n > *remaining { + return false + } + *remaining -= n + return true +} + +// preflight counts every occurrence of a shared subtree. The depth bound also +// terminates cycles without allocating a visited map. It calls no user methods. +func preflight(v any, remaining *int, depth int) bool { + if !charge(remaining, nodeCharge) { + return false + } + switch v := v.(type) { + case nil, bool: + return true + case string: + return charge(remaining, len(v)) && utf8.ValidString(v) + case float64: + return !math.IsNaN(v) && !math.IsInf(v, 0) + case map[string]any: + if depth >= maxDepth || len(v) > *remaining/nodeCharge { + return false + } + for name, value := range v { + if !charge(remaining, len(name)) || !utf8.ValidString(name) || !preflight(value, remaining, depth+1) { + return false + } + } + return true + case []any: + if depth >= maxDepth || len(v) > *remaining/nodeCharge { + return false + } + for _, value := range v { + if !preflight(value, remaining, depth+1) { + return false + } + } + return true + default: + return false + } +} + +// copyNode is only called after preflight. Inputs must remain unchanged until +// copying completes, as with ordinary Go maps and slices passed to a function. +func copyNode(v any) node { + switch v := v.(type) { + case nil: + return node{kind: 'n'} + case bool: + if v { + return node{kind: 't'} + } + return node{kind: 'f'} + case string: + return node{kind: '"', text: strings.Clone(v)} + case float64: + return node{kind: '0', number: v} + case map[string]any: + out := node{kind: '{', object: make(map[string]node, len(v))} + for name, value := range v { + out.object[strings.Clone(name)] = copyNode(value) + } + return out + case []any: + out := node{kind: '[', array: make([]node, len(v))} + for i, value := range v { + out.array[i] = copyNode(value) + } + return out + default: + panic("probe copy without successful preflight") + } +} + +func snapshot(v map[string]any, budget int) (node, bool) { + remaining := budget + if !preflight(v, &remaining, 0) { + return node{}, false + } + return copyNode(v), true +} + +func fixture(name string) (map[string]any, int) { + budget := defaultBudget + switch name { + case "ordinary": + return map[string]any{ + "iss": "https://issuer.example", "sub": "subject", "client_id": "client", + "scope": "orders.read orders.write", "token_use": "access", + "aud": []any{"orders", "inventory"}, "cognito:groups": []any{"staff", "billing"}, + "exp": float64(1900000000), "tenant": map[string]any{"id": "tenant", "active": true}, + }, budget + case "distinct_names", "distinct_names_1m", "distinct_names_6m": + if name == "distinct_names_1m" { + budget = 1024 * 1024 + } else if name == "distinct_names_6m" { + budget = 6 * 1024 * 1024 + } + out := make(map[string]any) + for i := range (budget - nodeCharge) / (nodeCharge + 8) { + out[fmt.Sprintf("c%07d", i)] = "" + } + return out, budget + case "array", "over_budget": + n := (budget - 2*nodeCharge - len("items")) / nodeCharge + if name == "over_budget" { + n++ + } + return map[string]any{"items": make([]any, n)}, budget + case "singleton_objects": + n := (budget - 2*nodeCharge - len("items")) / (2*nodeCharge + 1) + values := make([]any, n) + for i := range values { + values[i] = map[string]any{"x": nil} + } + return map[string]any{"items": values}, budget + case "large_string": + return map[string]any{"text": strings.Repeat("a", budget-2*nodeCharge-len("text"))}, budget + case "depth_64", "depth_65": + depth := 64 + if name == "depth_65" { + depth++ + } + out := map[string]any{"leaf": "v"} + for range depth - 1 { + out = map[string]any{"next": out} + } + return out, budget + case "cycle": + out := make(map[string]any) + out["self"] = out + return out, budget + case "shared_subtree": + var value any = "leaf" + for range 18 { + value = []any{value, value} + } + return map[string]any{"tree": value}, budget + default: + panic("unknown probe fixture") + } +} + +var cases = []string{"ordinary", "distinct_names", "distinct_names_1m", "distinct_names_6m", "array", "singleton_objects", "large_string", "depth_64", "depth_65", "cycle", "shared_subtree", "over_budget"} + +func BenchmarkSnapshot(b *testing.B) { + for _, name := range cases { + b.Run(name, func(b *testing.B) { + input, budget := fixture(name) + b.ReportAllocs() + for b.Loop() { + out, ok := snapshot(input, budget) + runtime.KeepAlive(out) + runtime.KeepAlive(ok) + } + }) + } +} + +func TestSnapshotMemory(t *testing.T) { + t.Logf("toolchain=%s target=%s/%s", runtime.Version(), runtime.GOOS, runtime.GOARCH) + for _, name := range cases { + t.Run(name, func(t *testing.T) { + input, budget := fixture(name) + runtime.GC() + runtime.GC() + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + out, ok := snapshot(input, budget) + runtime.GC() + runtime.GC() + runtime.ReadMemStats(&after) + runtime.KeepAlive(input) + runtime.KeepAlive(out) + remaining := budget + accepted := preflight(input, &remaining, 0) + if accepted != ok { + t.Fatal("snapshot and preflight acceptance disagree") + } + t.Logf("budget=%d accepted=%t charge=%d retained_delta_bytes=%d", budget, ok, budget-remaining, int64(after.HeapAlloc)-int64(before.HeapAlloc)) + }) + } +} + +func TestCandidateBoundaries(t *testing.T) { + for _, name := range cases { + t.Run(name, func(t *testing.T) { + input, budget := fixture(name) + _, ok := snapshot(input, budget) + want := name != "depth_65" && name != "cycle" && name != "shared_subtree" && name != "over_budget" + if ok != want { + t.Errorf("snapshot(%s) accepted=%t, want %t", name, ok, want) + } + }) + } + input := map[string]any{"nested": map[string]any{"value": "before"}, "list": []any{"before"}} + out, ok := snapshot(input, defaultBudget) + if !ok { + t.Fatal("valid ownership fixture rejected") + } + input["nested"].(map[string]any)["value"] = "after" + input["list"].([]any)[0] = "after" + if out.object["nested"].object["value"].text != "before" || out.object["list"].array[0].text != "before" { + t.Error("snapshot aliases caller-owned containers") + } +} + +func TestSDKNumberFidelity(t *testing.T) { + const payload = `{"httpMethod":"GET","path":"/","requestContext":{"apiId":"example","authorizer":{"claims":{"counter":9007199254740993}}}}` + for _, useNumber := range []bool{false, true} { + var captured any + h := lambda.NewHandlerWithOptions(func(ctx context.Context, event events.APIGatewayProxyRequest) (bool, error) { + captured = event.RequestContext.Authorizer["claims"].(map[string]any)["counter"] + return true, nil + }, lambda.WithUseNumber(useNumber)) + if _, err := h.Invoke(t.Context(), []byte(payload)); err != nil { + t.Fatal(err) + } + t.Logf("UseNumber=%t type=%T value=%v", useNumber, captured, captured) + if !useNumber && captured != float64(9007199254740992) { + t.Errorf("default SDK number = %v, want rounded float64", captured) + } + if useNumber && fmt.Sprint(captured) != "9007199254740993" { + t.Errorf("UseNumber SDK number = %v, want exact literal", captured) + } + } +} + +// scanJSON measures bounded lexical validation separately from the tree. It +// uses jsontext directly; it does not decode through map[string]any and float64. +func scanJSON(input []byte, budget int) error { + if len(input) > budget { + return errRejected + } + d := jsontext.NewDecoder(bytes.NewReader(input)) + started, completed := false, false + for { + token, err := d.ReadToken() + if err == io.EOF && completed { + return nil + } + if err != nil { + return err + } + if completed || !started && token.Kind() != '{' { + return errRejected + } + started = true + if d.StackDepth() > maxDepth { + return errRejected + } + completed = d.StackDepth() == 0 + } +} + +func TestJSONScanBoundaries(t *testing.T) { + for _, input := range []string{``, `null`, `[]`, `{"a":`, `{"a":1,"a":2}`, `{} {}`, "{\"a\":\"\xff\"}"} { + if err := scanJSON([]byte(input), defaultBudget); err == nil { + t.Errorf("scanJSON(%q) accepted invalid claim object", input) + } + } + if err := scanJSON([]byte(`{"a":9007199254740993,"b":[null,true]}`), defaultBudget); err != nil { + t.Errorf("scanJSON(valid object) = %v", err) + } +} + +func BenchmarkJSONScan(b *testing.B) { + for _, name := range []string{"ordinary", "distinct_names", "large_string", "depth_64"} { + b.Run(name, func(b *testing.B) { + input, budget := fixture(name) + wire, err := json.Marshal(input) + if err != nil { + b.Fatal(err) + } + b.ReportAllocs() + for b.Loop() { + if err := scanJSON(wire, budget); err != nil { + b.Fatal(err) + } + } + }) + } +} From d308826217a6b5930fdfb28b8499c53150773b9f Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 07:21:47 +0100 Subject: [PATCH 23/44] feat: implement owned claims with bounded direct JSON processing --- README.md | 26 +- adapter.go | 18 +- adapter_test.go | 15 + docs/benchmarks.md | 38 +++ docs/decisions/0012-shared-identity.md | 8 +- docs/decisions/0014-claims-api.md | 36 +- docs/decisions/0015-iam-caller-forms.md | 56 ++++ docs/decisions/README.md | 5 +- docs/plan.md | 15 +- errors.go | 24 +- identity/claims.go | 204 ++++++++++++ identity/claims_bench_test.go | 107 ++++++ identity/claims_example_test.go | 34 ++ identity/claims_json.go | 130 ++++++++ identity/claims_test.go | 416 ++++++++++++++++++++++++ identity/claims_typed.go | 225 +++++++++++++ identity/errors.go | 45 +++ identity_errors_test.go | 64 ++++ identity_sdk_test.go | 37 +++ 19 files changed, 1485 insertions(+), 18 deletions(-) create mode 100644 docs/decisions/0015-iam-caller-forms.md create mode 100644 identity/claims.go create mode 100644 identity/claims_bench_test.go create mode 100644 identity/claims_example_test.go create mode 100644 identity/claims_json.go create mode 100644 identity/claims_test.go create mode 100644 identity/claims_typed.go create mode 100644 identity/errors.go create mode 100644 identity_errors_test.go create mode 100644 identity_sdk_test.go diff --git a/README.md b/README.md index e67557d..f9b7ff7 100644 --- a/README.md +++ b/README.md @@ -22,8 +22,9 @@ public categories support `errors.Is`, and `errors.AsType[*edge.InvocationError] exposes sanitized operation and resource-limit diagnostics. HTTP error responses remain successful transport outcomes; operation, cleanup, cancellation, and late stream failures retain their accepted ownership and precedence. The shared -identity design is accepted, with precise claim accessors and resource limits -still awaiting their follow-up design review. +claims API and resource limits are implemented in `identity`. Caller/context +construction, JWT normalization and gateway producers remain pending; the initial +IAM caller forms are recorded separately for review. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle @@ -63,6 +64,27 @@ complete buffered-envelope limit and the 16,000-byte streaming metadata prefix The option validates configuration now, and the private buffered writer applies it at commitment. Public invocation and the streaming HTTP writer remain pending. +## Owned claims + +`identity.ParseClaims`, `identity.NewClaims`, and `identity.NewTextClaims` accept +raw JSON, decoded Go values, and gateway string maps respectively. Claims are +immutable snapshots with checked accessors. Exact numeric text remains available +where the input preserves it; a decoded float64 is never advertised as an exact +original JSON number, and gateway text is never reparsed as an embedded array. + +Processing uses `encoding/json/jsontext` directly. The only production +`encoding/json` import recognizes its `Number` data type, supporting typed SDK +handlers configured with `lambda.WithUseNumber(true)`; no v1 codec is called. +See the [runnable examples](identity/claims_example_test.go). + +The default claim allowance is 256 KiB, configurable with +`identity.WithClaimsBudget(bytes)` up to 6 MiB. Each value costs 64 bytes plus +object-name, string and exact-number text bytes. Raw JSON also has a wire-length +limit of the same size. Nesting is limited to 64 containers. These are resource +policies, not heap caps; [measurements](docs/benchmarks.md) include allocation and +retained-memory costs. `edge.WithIdentityClaimsBudget(bytes)` records the matching +gateway configuration, whose producer wiring remains pending. + - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) - [Repository working agreement](AGENTS.md) diff --git a/adapter.go b/adapter.go index a64efd2..93fb866 100644 --- a/adapter.go +++ b/adapter.go @@ -19,6 +19,7 @@ type Adapter struct { type config struct { gatewayIdentity bool + identityClaimsBudget int responseHeaderBudget int } @@ -39,6 +40,18 @@ func WithGatewayIdentity(enabled bool) Option { } } +// WithIdentityClaimsBudget sets the gateway identity claims allowance in bytes. +// The default is 256 KiB; New rejects a final setting outside 1..6 MiB. +// Each claim value costs 64 bytes plus name, string and exact-number text bytes. +// Dedicated gateway scopes consume the same allowance. Raw claim JSON is also +// limited to this byte length. This is a resource policy, not a heap cap or AWS +// quota. It does not enable gateway identity. Extraction is not yet implemented. +func WithIdentityClaimsBudget(bytes int) Option { + return func(c *config) { + c.identityClaimsBudget = bytes + } +} + // WithResponseHeaderBudget sets the response header resource budget in bytes. // The default is 256 KiB. New rejects a final configured value outside 1..6 MiB; // zero does not disable the bound. Each original value costs its name length, @@ -72,7 +85,7 @@ func New(handler http.Handler, opts ...Option) (*Adapter, error) { } } - cfg := config{responseHeaderBudget: defaultResponseHeaderBudget} + cfg := config{identityClaimsBudget: 256 * 1024, responseHeaderBudget: defaultResponseHeaderBudget} for i, opt := range opts { if opt == nil { return nil, fmt.Errorf("edge: nil option at index %d", i) @@ -82,5 +95,8 @@ func New(handler http.Handler, opts ...Option) (*Adapter, error) { if cfg.responseHeaderBudget <= 0 || cfg.responseHeaderBudget > maxResponseBytes { return nil, errors.New("edge: response header budget must be between 1 and 6291456 bytes") } + if cfg.identityClaimsBudget <= 0 || cfg.identityClaimsBudget > maxResponseBytes { + return nil, errors.New("edge: identity claims budget must be between 1 and 6291456 bytes") + } return &Adapter{handler: handler, config: cfg}, nil } diff --git a/adapter_test.go b/adapter_test.go index 9d4412f..8c2ec5a 100644 --- a/adapter_test.go +++ b/adapter_test.go @@ -42,3 +42,18 @@ func TestNewDoesNotInvokeHandler(t *testing.T) { t.Errorf("New(handler, option) = (%v, %v), want (non-nil, nil)", adapter, err) } } + +func TestIdentityClaimsBudgetConfiguration(t *testing.T) { + for _, budget := range []int{-1, 0, 6*1024*1024 + 1} { + adapter, err := edge.New(http.NewServeMux(), edge.WithIdentityClaimsBudget(budget)) + if err == nil || adapter != nil { + t.Errorf("New(identity budget=%d) = %v, %v; want nil, error", budget, adapter, err) + } + } + for _, budget := range []int{1, 256 * 1024, 6 * 1024 * 1024} { + adapter, err := edge.New(http.NewServeMux(), edge.WithIdentityClaimsBudget(0), edge.WithIdentityClaimsBudget(budget)) + if err != nil || adapter == nil { + t.Errorf("New(identity budget=%d last) = %v, %v; want adapter, nil", budget, adapter, err) + } + } +} diff --git a/docs/benchmarks.md b/docs/benchmarks.md index e24e865..71606aa 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -166,3 +166,41 @@ float64 9007199254740992; WithUseNumber(true) yields json.Number with the origin digits. This is evidence for data-type compatibility in [decision 0014](decisions/0014-claims-api.md), not a proposal to use the v1 codec inside edge. No live Lambda or API Gateway request was made. + +## Implemented claims constructors (2026-09-16) + +The same Go 1.27.1 / darwin-arm64 Apple M2 environment, three samples per case: + +```sh +go test ./identity -run '^TestClaimsRetainedMemory$' -v -count=3 +go test ./identity -run '^$' -bench '^BenchmarkClaims$' -benchmem -benchtime=100ms -count=3 +``` + +Median bytes per operation and independently sampled retained heap in bytes: + +| Fixture | Typed B/op | Raw B/op | Typed retained | Raw retained | +| --- | ---: | ---: | ---: | ---: | +| Ordinary claims | 2,690 | 4,968 | 2,696 | 2,776 | +| Distinct names, 256 KiB budget | 750,396 | 2,691,610 | 750,400 | 689,360 | +| Distinct names, 6 MiB budget | 12,238,708 | 53,483,244 | 12,238,688 | 12,937,712 | +| Singleton objects, 256 KiB budget | 1,661,221 | 1,922,539 | 1,661,184 | 1,669,376 | +| Large string, 256 KiB budget | 262,912 | 1,311,952 | 262,912 | 262,912 | + +Ordinary typed construction took a median 1.24 µs and 30 allocations; ordinary +raw construction took 3.02 µs and 81 allocations. Raw construction includes strict +lexical validation and preflight, then owned-tree construction, while typed input +has no JSON round trip. Raw numbers retain their literal, while input float64 +values retain only the available floating-point value. Both paths clone strings. + +The largest opt-in raw case has substantial transient allocation (about 51 MiB +cumulative versus 12.3 MiB retained). Two parser passes, duplicate-name tracking, +string conversion/copying, and map growth contribute. These measurements do not +justify equating the 6 MiB weighted budget with memory use, or making it the +default. Future storage optimizations can retain the approved accounting policy. + +The raw and typed retained totals can differ because map construction capacity, +array growth and exact numeric storage differ. Inputs remain alive during memory +samples; two collections before and after reduce pool retention. Small deltas +are noisy (ordinary typed samples ranged from 2,696 to 8,016 bytes). No figure is +peak memory, RSS, total invocation allocation, or a Lambda latency prediction. +JWT normalized views and dedicated scopes are not implemented in these samples. diff --git a/docs/decisions/0012-shared-identity.md b/docs/decisions/0012-shared-identity.md index 3881621..34dcbb0 100644 --- a/docs/decisions/0012-shared-identity.md +++ b/docs/decisions/0012-shared-identity.md @@ -1,6 +1,7 @@ # 0012: Shared identity, provenance, and producer composition -Status: accepted as the starting contract on 2026-09-15; implementation pending. +Status: accepted as the starting contract on 2026-09-15; claims foundation +implemented under 0014, caller/context and gateway producers pending. This walks the dependencies needed for the first callable buffered adapter. Decision 0013 covers the public errors used at those boundaries. Approval of the @@ -355,4 +356,7 @@ The user agreed with all recommendations as a starting point on 2026-09-15. This accepts I1-I8 and the implementation order. The explicitly deferred claim accessor inventory, claim resource limits, producer fixture coverage, and later authn/custom-mapper policies still require their recorded follow-up review. -None of the shared identity API is implemented yet. +The claims constructors, immutable views and resource/error foundation are now +implemented under accepted decision 0014. Caller/context construction and native +gateway producers remain pending. Decision 0015 presents the supported IAM caller +forms left open by I5 for review before the complete caller union is published. diff --git a/docs/decisions/0014-claims-api.md b/docs/decisions/0014-claims-api.md index 7c54645..9be9ae9 100644 --- a/docs/decisions/0014-claims-api.md +++ b/docs/decisions/0014-claims-api.md @@ -1,7 +1,8 @@ # 0014: Owned claims, numeric fidelity, and resource limits -Status: proposed on 2026-09-16; awaiting review. Production identity APIs are -unchanged. This completes the claims follow-up explicitly left open by 0012. +Status: accepted on 2026-09-16; claims constructors/accessors and error foundation +implemented. JWT normalization and dedicated-scope accounting await caller +construction. This completes the claims design follow-up left open by 0012. ## Evidence and scope @@ -31,7 +32,7 @@ Sources: ## C1. Small concrete API with checked reads -Recommend these declarations; they illustrate the proposal, not existing APIs: +The accepted declarations below are now implemented: ```go func ParseClaims(data jsontext.Value, opts ...ClaimsOption) (Claims, error) @@ -227,5 +228,30 @@ tradeoffs. After approval: 4. Follow with local verification/authz and streaming HTTP writer work in the accepted order. Do not pick a JWT library or issuer profile in this claims step. -No new production APIs, claim normalization behavior, or resource options have -been implemented by this proposal. No live AWS resources were invoked. +At proposal time no production APIs or behavior had changed. Implementation +status is recorded below; no live AWS resources were invoked. + +## Resolution + +The user approved C1–C5 on 2026-09-16 and authorized implementation, including +the narrow json.Number data-type compatibility and gateway qualifications. +The native-producer fixture review remains a separate step before gateway wiring. + +## Implementation status + +Implemented in identity: all three claims constructors, checked immutable views, +exact numeric text and explicit approximate conversion, the closed input-type +set including json.Number, strict JSON processing through jsontext, typed/raw +preflight, byte/depth limits, sanitized errors and diagnostic formatting. +Inputs and returned containers have contract tests for ownership; raw numeric +and typed SDK number-preserving paths are tested through the actual SDK wrapper. + +The root adapter records and validates WithIdentityClaimsBudget. A private error +translation boundary classifies claim failures and reports identity_claims byte +limits without misreporting depth as bytes. Neither is wired to invocation yet. + +Production allocation measurements are recorded in docs/benchmarks.md. Race tests, +vet, JSON fuzzing and both Lambda-target builds validate the claims foundation. +Caller construction, dedicated scope accounting and JWT normalization are still +pending. Decision 0015 surfaces the explicitly deferred IAM caller-form review +before publishing the complete caller union; no placeholder constructor exists. diff --git a/docs/decisions/0015-iam-caller-forms.md b/docs/decisions/0015-iam-caller-forms.md new file mode 100644 index 0000000..b650fb3 --- /dev/null +++ b/docs/decisions/0015-iam-caller-forms.md @@ -0,0 +1,56 @@ +# 0015: Initial IAM caller forms + +Status: proposed on 2026-09-16; awaiting review. + +Decision 0012 requires validation of an actual caller principal form and explicitly +defers the supported-form fixtures. Implementing NewIAM now reaches that gate. +Claims implementation under accepted decision 0014 proceeds independently. + +## Evidence + +The [AWS Principal documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html), +read through public AWS MCP, distinguishes IAM users, account/root principals, +roles, assumed-role sessions, and STS federated-user sessions. It explains that +using assumed-role credentials makes the requester a role session principal. +Policy principal syntax also includes service and identity-provider principals; +these are not all interchangeable with an authenticated requester's ARN. + +The [AWS principal-key table](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html) +distinguishes root, user, federated user and assumed-role requesters and preserves +session identifiers. Existing edge code has no IAM producer to dictate a broader +compatibility requirement. Identity must remain independent of AWS libraries. + +## Recommendation + +Initially accept these exact forms in NewIAM: + +- IAM root: arn::iam:::root. +- IAM user: arn::iam:::user/. +- STS assumed-role session: arn::sts:::assumed-role//. +- STS federated-user session: arn::sts:::federated-user/. + +Preserve spelling, partition, account and full resource including session. Reject +bare account IDs, IAM role ARNs, wildcards and policy-only principal forms in this +initial request-caller constructor. Do not transform a role session into a role. +Recognizing root is a representation capability, not permission to perform an +application action. Root policy-principal delegation semantics are not applied. + +Use read-only IAM accessors PrincipalARN, Partition, AccountID, Service and +PrincipalType, with optional PrincipalID. WithIAMAccountID cross-checks an +independently supplied caller account; WithIAMPrincipalID preserves the optional +opaque identifier without parsing it into an identity. Duplicate assignments +fail under 0012. Never use API-owner account metadata as caller account. + +ARN lexical details and field limits must follow the relevant official API +definitions when implemented; this proposal does not invent normalization or +claim a universal API Gateway producer matrix. + +## Alternative and consequence + +Accepting every resource-policy Principal value would represent policy subjects +that are not the request caller the constructor describes. Accepting a bare role +ARN could also hide upstream loss of session identity. Recommend adding another +form only when a documented producer needs it, with an explicit fixture. + +This review gates the IAM constructor and complete caller union; it does not gate +the approved claims API, resource limits, or numerical fidelity implementation. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index e942227..a59f5a8 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -20,6 +20,7 @@ Keep related decisions small enough to review and implement independently. | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | | [0010: Response header design](0010-response-header-design.md) | Accepted | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | | [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | -| [0012: Shared identity](0012-shared-identity.md) | Accepted; detailed claims follow-up pending | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | +| [0012: Shared identity](0012-shared-identity.md) | Accepted; claims settled by 0014 | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | -| [0014: Claims API](0014-claims-api.md) | Proposed | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | +| [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | +| [0015: IAM caller forms](0015-iam-caller-forms.md) | Proposed | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | diff --git a/docs/plan.md b/docs/plan.md index b13abe9..7b3b273 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -227,11 +227,20 @@ The accepted order is caller/context/error foundations, precise claim API and native-producer fixtures, then raw/typed buffered invocation. Broader local authentication and authorization follow that shared foundation. -[Decision 0014](decisions/0014-claims-api.md), proposed on 2026-09-16, specifies the +[Decision 0014](decisions/0014-claims-api.md), accepted on 2026-09-16, specifies the claims constructor/accessor surface, SDK number-preserving input compatibility, ownership and weighted resource limits, and gateway normalization qualifications. -An isolated storage/SDK probe is implemented and measured; production identity -APIs remain unchanged while these recommendations await review. +The user approved its SDK numeric compatibility and gateway qualifications and +authorized implementation. Claims constructors, immutable accessors, strict raw +JSON and direct typed construction, byte/depth limits, and sanitized errors are +implemented. The adapter budget option is validated and a private invocation-error +translation boundary is tested; gateway extraction and invocation remain unwired. +Production allocation measurements supplement the original probe. Contract tests, +SDK integration, race detection, vet, JSON fuzzing and Lambda-target builds pass. +Caller/context construction and JWT normalization remain pending; proposed +[decision 0015](decisions/0015-iam-caller-forms.md) resolves the supported IAM forms +explicitly deferred by 0012 before publishing the complete caller union. +Native-producer fixture review remains separate from the claims approval. - Review caller constructors, immutable/owned data, validation, anonymous state, and context-presence semantics. diff --git a/errors.go b/errors.go index 3ce586f..7dd16ff 100644 --- a/errors.go +++ b/errors.go @@ -1,6 +1,10 @@ package edge -import "errors" +import ( + "errors" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) // Invocation failure categories support errors.Is through wrapping and joined // errors. Application HTTP responses, including 4xx and 5xx, are not invocation @@ -72,8 +76,9 @@ func (e *InvocationError) Operation() string { // Limit reports the resource name and its effective maximum in bytes for a // limit failure. Names are response_headers, buffered_body, buffered_envelope, -// and stream_metadata. A response header budget measures the documented charge, -// not wire bytes or heap use. buffered_body also covers base64 expansion and +// stream_metadata, and identity_claims. A response header or identity claims +// budget measures the documented charge; raw claims also have a wire-length +// limit. Neither budget measures heap use. buffered_body covers base64 expansion and // the representable HEAD byte count. Other failures return "", 0, false. func (e *InvocationError) Limit() (name string, maximum int64, ok bool) { if e == nil || e.limit == nil { @@ -99,3 +104,16 @@ func limitError(operation, name string, maximum int64, reason error) *Invocation err.limit = &invocationLimit{name: name, maximum: maximum} return err } + +// claimsError translates only package-owned claim failures. It deliberately +// rebuilds a safe error tree instead of wrapping the supplied error or its text. +// Gateway preparation will use this boundary when its producers are implemented. +func claimsError(err error) *InvocationError { + if limit, ok := errors.AsType[*identity.ClaimsLimitError](err); ok { + return limitError("identity", "identity_claims", limit.Maximum(), errors.Join(ErrIdentity, identity.ErrClaimsLimit)) + } + if errors.Is(err, identity.ErrInvalidClaims) { + return invocationError("identity", ErrIdentity, "invalid claims", identity.ErrInvalidClaims) + } + return invocationError("identity", ErrIdentity, "claims construction failed") +} diff --git a/identity/claims.go b/identity/claims.go new file mode 100644 index 0000000..92efbc4 --- /dev/null +++ b/identity/claims.go @@ -0,0 +1,204 @@ +// Package identity provides owned claim representations for authenticated callers. +// Constructing claims validates and copies data; it does not authenticate it. +package identity + +import ( + "errors" + "maps" + "slices" + "strconv" +) + +const ( + defaultClaimsBudget = 256 * 1024 + maxClaimsBudget = 6 * 1024 * 1024 + claimCharge = 64 + maxClaimDepth = 64 +) + +// ClaimKind describes a claim's available semantic type. +type ClaimKind uint8 + +// Claim kinds distinguish a missing/zero view from a present null. +const ( + ClaimInvalid ClaimKind = iota + ClaimNull + ClaimBoolean + ClaimString + ClaimNumber + ClaimArray + ClaimObject +) + +// Representation describes the immediate input representation, not its original +// token encoding or whether an authenticating producer verified it. +type Representation uint8 + +// Claim representations retain the distinction between JSON, decoded values, +// and gateway text that may have lost its original JSON type. +const ( + RepresentationUnknown Representation = iota + RepresentationJSON + RepresentationDecoded + RepresentationGatewayText +) + +// Claims is an immutable object of named claims. Its zero value is empty. +// Views may share owned backing storage and may be read concurrently. +// Use Lookup and its checked accessors to inspect values; diagnostic formatting +// does not reveal claim names or contents. There is no implicit JSON export. +type Claims struct { + members map[string]claimNode + budget int + charge int +} + +// Claim is an immutable view of a value. The zero value has kind ClaimInvalid. +// Its representation describes the available input, not authentication strength. +type Claim struct { + node claimNode + budget int +} + +// cost includes this node's entire subtree. It permits an object view to retain +// the original budget while accounting only for that object's contents. +type claimNode struct { + kind ClaimKind + representation Representation + cost uint32 + text string + number float64 + array []claimNode + object map[string]claimNode +} + +// ClaimsOption configures claims construction. A nil option is invalid. +// Scalar settings apply in order, with the last assignment winning. +type ClaimsOption func(*claimsConfig) + +type claimsConfig struct { + budget int +} + +// WithClaimsBudget sets the weighted resource allowance in bytes, defaulting to +// 256 KiB. Constructors reject a final setting outside 1..6 MiB. +// Each value costs 64 bytes plus object-name, string and exact-number text bytes. +// ParseClaims also limits the original JSON byte length to this maximum. +// The allowance is not an AWS quota or a heap cap. Nesting is limited separately +// to 64 containers, including the root, and cannot be disabled. +func WithClaimsBudget(bytes int) ClaimsOption { + return func(c *claimsConfig) { c.budget = bytes } +} + +func claimsBudget(opts []ClaimsOption) (int, error) { + c := claimsConfig{budget: defaultClaimsBudget} + for _, opt := range opts { + if opt == nil { + return 0, errors.New("identity: nil claims option") + } + opt(&c) + } + if c.budget <= 0 || c.budget > maxClaimsBudget { + return 0, errors.New("identity: claims budget must be between 1 and 6291456 bytes") + } + return c.budget, nil +} + +// Len returns the number of members in the object. +func (c Claims) Len() int { return len(c.members) } + +// Names returns an independently owned, sorted slice of member names. +func (c Claims) Names() []string { return slices.Sorted(maps.Keys(c.members)) } + +// Lookup reports whether name is present, including a present null value. +func (c Claims) Lookup(name string) (Claim, bool) { + n, ok := c.members[name] + if !ok { + return Claim{}, false + } + return Claim{node: n, budget: c.budget}, true +} + +// String returns a diagnostic description without claim names or values. +func (c Claims) String() string { return "identity.Claims" } + +// GoString returns the same sanitized description as String. +func (c Claims) GoString() string { return c.String() } + +// Kind returns the available semantic type. +func (c Claim) Kind() ClaimKind { return c.node.kind } + +// Representation returns the immediate input representation. +func (c Claim) Representation() Representation { return c.node.representation } + +// Text reports whether the claim is a string. It never converts other types. +func (c Claim) Text() (string, bool) { + if c.node.kind != ClaimString { + return "", false + } + return c.node.text, true +} + +// Bool reports whether the claim is a boolean. +func (c Claim) Bool() (bool, bool) { + if c.node.kind != ClaimBoolean { + return false, false + } + return c.node.number != 0, true +} + +// Array reports whether the claim is an array and returns a new slice of views. +// Replacing elements cannot mutate the original claims. +func (c Claim) Array() ([]Claim, bool) { + if c.node.kind != ClaimArray { + return nil, false + } + out := make([]Claim, len(c.node.array)) + for i, n := range c.node.array { + out[i] = Claim{node: n, budget: c.budget} + } + return out, true +} + +// Object reports whether the claim is an object and returns an immutable view. +func (c Claim) Object() (Claims, bool) { + if c.node.kind != ClaimObject { + return Claims{}, false + } + return Claims{members: c.node.object, budget: c.budget, charge: int(c.node.cost)}, true +} + +// NumberText reports whether an exact numeric representation is available. +// JSON and json.Number preserve their supplied literal; native integers use +// decimal text. Supplied float32/float64 and all strings return false, even for +// integral values. Exactness describes the supplied value, not lost upstream data. +func (c Claim) NumberText() (string, bool) { + if c.node.kind != ClaimNumber || c.node.text == "" { + return "", false + } + return c.node.text, true +} + +// Float64 explicitly converts a number to its nearest float64 representation. +// It reports false for nonnumbers or a conversion error; ordinary rounding is +// permitted. For supplied floating-point values it returns their stored value. +// Use NumberText and an exact parser when rounding is unacceptable. +func (c Claim) Float64() (float64, bool) { + if c.node.kind != ClaimNumber { + return 0, false + } + if c.node.text == "" { + return c.node.number, true + } + n, err := strconv.ParseFloat(c.node.text, 64) + if err != nil { + return 0, false + } + return n, true +} + +// String returns a diagnostic description without claim contents. +func (c Claim) String() string { return "identity.Claim" } + +// GoString returns the same sanitized description as String. +func (c Claim) GoString() string { return c.String() } diff --git a/identity/claims_bench_test.go b/identity/claims_bench_test.go new file mode 100644 index 0000000..c805c6d --- /dev/null +++ b/identity/claims_bench_test.go @@ -0,0 +1,107 @@ +package identity_test + +import ( + "encoding/json/jsontext" + "encoding/json/v2" + "fmt" + "runtime" + "strings" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func claimsFixture(name string) (map[string]any, int) { + budget := 256 * 1024 + switch name { + case "ordinary": + return map[string]any{ + "iss": "https://issuer.example", "sub": "subject", "client_id": "client", + "scope": "orders.read orders.write", "token_use": "access", + "aud": []any{"orders", "inventory"}, "cognito:groups": []any{"staff", "billing"}, + "exp": float64(1900000000), "tenant": map[string]any{"id": "tenant", "active": true}, + }, budget + case "distinct_names", "distinct_names_6m": + if name == "distinct_names_6m" { + budget = 6 * 1024 * 1024 + } + out := make(map[string]any) + for i := range (budget - 64) / 72 { + out[fmt.Sprintf("c%07d", i)] = "" + } + return out, budget + case "singleton_objects": + values := make([]any, (budget-128-5)/129) + for i := range values { + values[i] = map[string]any{"x": nil} + } + return map[string]any{"items": values}, budget + case "large_string": + return map[string]any{"text": strings.Repeat("a", budget-128-4)}, budget + default: + panic("unknown benchmark fixture") + } +} + +func BenchmarkClaims(b *testing.B) { + for _, name := range []string{"ordinary", "distinct_names", "distinct_names_6m", "singleton_objects", "large_string"} { + input, budget := claimsFixture(name) + wire, err := json.Marshal(input) + if err != nil { + b.Fatal(err) + } + for _, raw := range []bool{false, true} { + b.Run(fmt.Sprintf("%s/raw=%t", name, raw), func(b *testing.B) { + b.ReportAllocs() + for b.Loop() { + var c identity.Claims + var err error + if raw { + c, err = identity.ParseClaims(wire, identity.WithClaimsBudget(budget)) + } else { + c, err = identity.NewClaims(input, identity.WithClaimsBudget(budget)) + } + if err != nil { + b.Fatal(err) + } + runtime.KeepAlive(c) + } + }) + } + } +} + +func TestClaimsRetainedMemory(t *testing.T) { + for _, name := range []string{"ordinary", "distinct_names", "distinct_names_6m", "singleton_objects", "large_string"} { + input, budget := claimsFixture(name) + wire, err := json.Marshal(input) + if err != nil { + t.Fatal(err) + } + for _, raw := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/raw=%t", name, raw), func(t *testing.T) { + runtime.GC() + runtime.GC() + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + var c identity.Claims + var err error + if raw { + c, err = identity.ParseClaims(jsontext.Value(wire), identity.WithClaimsBudget(budget)) + } else { + c, err = identity.NewClaims(input, identity.WithClaimsBudget(budget)) + } + if err != nil { + t.Fatal(err) + } + runtime.GC() + runtime.GC() + runtime.ReadMemStats(&after) + runtime.KeepAlive(input) + runtime.KeepAlive(wire) + runtime.KeepAlive(c) + t.Logf("budget=%d retained_heap_delta=%d (excludes input; not peak memory)", budget, int64(after.HeapAlloc)-int64(before.HeapAlloc)) + }) + } + } +} diff --git a/identity/claims_example_test.go b/identity/claims_example_test.go new file mode 100644 index 0000000..810ff57 --- /dev/null +++ b/identity/claims_example_test.go @@ -0,0 +1,34 @@ +package identity_test + +import ( + "encoding/json/jsontext" + "fmt" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func ExampleParseClaims() { + claims, err := identity.ParseClaims(jsontext.Value(`{"tenant":"orders","sequence":9007199254740993}`)) + if err != nil { + fmt.Println(err) + return + } + if claim, present := claims.Lookup("sequence"); present { + if number, exact := claim.NumberText(); exact { + fmt.Println(number) + } + } + // Output: 9007199254740993 +} + +func ExampleNewTextClaims() { + claims, err := identity.NewTextClaims(map[string]string{"cognito:groups": `["staff"]`}) + if err != nil { + fmt.Println(err) + return + } + claim, present := claims.Lookup("cognito:groups") + _, array := claim.Array() + fmt.Println("present:", present, "array:", array) + // Output: present: true array: false +} diff --git a/identity/claims_json.go b/identity/claims_json.go new file mode 100644 index 0000000..b9f3b44 --- /dev/null +++ b/identity/claims_json.go @@ -0,0 +1,130 @@ +package identity + +import ( + "bytes" + "encoding/json/jsontext" + "io" + "strings" +) + +// ParseClaims validates exactly one JSON object and owns its semantic contents. +// It preserves numeric token text, but not whitespace, member order or string +// escape spelling. It rejects duplicate names, invalid UTF-8 and trailing values. +// Input bytes must not change during construction and are not retained afterward. +// Invalid input matches ErrInvalidClaims; byte exhaustion is a ClaimsLimitError. +// The representation is received JSON, not proof of original JWT fidelity. +func ParseClaims(data jsontext.Value, opts ...ClaimsOption) (Claims, error) { + maximum, err := claimsBudget(opts) + if err != nil { + return Claims{}, err + } + if len(data) > maximum { + return Claims{}, &ClaimsLimitError{maximum: int64(maximum)} + } + d := jsontext.NewDecoder(bytes.NewReader(data)) + if d.PeekKind() != '{' { + return Claims{}, ErrInvalidClaims + } + budget := claimBudget{maximum: maximum, remaining: maximum} + if _, err := readJSONClaim(d, &budget, 0, false); err != nil { + return Claims{}, err + } + if _, err := d.ReadToken(); err != io.EOF { + return Claims{}, ErrInvalidClaims + } + // Validation and accounting precede owned-tree allocation. The decoder may + // allocate its own bounded wire buffer and duplicate-name tracking on pass one. + d.Reset(bytes.NewReader(data)) + budget.remaining = maximum + n, err := readJSONClaim(d, &budget, 0, true) + if err != nil { + return Claims{}, err + } + return Claims{members: n.object, budget: maximum, charge: int(n.cost)}, nil +} + +func readJSONClaim(d *jsontext.Decoder, budget *claimBudget, depth int, build bool) (claimNode, error) { + start := budget.remaining + if err := budget.take(claimCharge); err != nil { + return claimNode{}, err + } + token, err := d.ReadToken() + if err != nil { + return claimNode{}, ErrInvalidClaims + } + n := claimNode{representation: RepresentationJSON} + switch token.Kind() { + case 'n': + n.kind = ClaimNull + case 't', 'f': + n.kind = ClaimBoolean + if token.Bool() { + n.number = 1 + } + case '"', '0': + n.kind = ClaimString + if token.Kind() == '0' { + n.kind = ClaimNumber + } + text := token.String() + if err := budget.take(len(text)); err != nil { + return claimNode{}, err + } + if build { + // Token data expires at the next decoder call. + n.text = strings.Clone(text) + } + case '{': + if depth >= maxClaimDepth { + return claimNode{}, ErrInvalidClaims + } + n.kind = ClaimObject + if build { + n.object = make(map[string]claimNode) + } + for d.PeekKind() != '}' { + key, err := d.ReadToken() + if err != nil || key.Kind() != '"' { + return claimNode{}, ErrInvalidClaims + } + name := key.String() + if err := budget.take(len(name)); err != nil { + return claimNode{}, err + } + if build { + name = strings.Clone(name) + } + child, err := readJSONClaim(d, budget, depth+1, build) + if err != nil { + return claimNode{}, err + } + if build { + n.object[name] = child + } + } + if _, err := d.ReadToken(); err != nil { + return claimNode{}, ErrInvalidClaims + } + case '[': + if depth >= maxClaimDepth { + return claimNode{}, ErrInvalidClaims + } + n.kind = ClaimArray + for d.PeekKind() != ']' { + child, err := readJSONClaim(d, budget, depth+1, build) + if err != nil { + return claimNode{}, err + } + if build { + n.array = append(n.array, child) + } + } + if _, err := d.ReadToken(); err != nil { + return claimNode{}, ErrInvalidClaims + } + default: + return claimNode{}, ErrInvalidClaims + } + n.cost = uint32(start - budget.remaining) + return n, nil +} diff --git a/identity/claims_test.go b/identity/claims_test.go new file mode 100644 index 0000000..dcd43d7 --- /dev/null +++ b/identity/claims_test.go @@ -0,0 +1,416 @@ +package identity_test + +import ( + "encoding/json" + "encoding/json/jsontext" + "errors" + "fmt" + "math" + "slices" + "strings" + "sync" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func parseClaims(t *testing.T, input string) identity.Claims { + t.Helper() + c, err := identity.ParseClaims(jsontext.Value(input)) + if err != nil { + t.Fatalf("ParseClaims(%q) = %v", input, err) + } + return c +} + +func lookup(t *testing.T, c identity.Claims, name string) identity.Claim { + t.Helper() + v, ok := c.Lookup(name) + if !ok { + t.Fatalf("Lookup(%q) missing from fixture", name) + } + return v +} + +func TestClaimsPresenceAndReads(t *testing.T) { + c := parseClaims(t, `{"null":null,"false":false,"true":true,"text":"","number":1,"array":[],"object":{}}`) + for name, kind := range map[string]identity.ClaimKind{ + "null": identity.ClaimNull, "false": identity.ClaimBoolean, "true": identity.ClaimBoolean, + "text": identity.ClaimString, "number": identity.ClaimNumber, "array": identity.ClaimArray, "object": identity.ClaimObject, + } { + v := lookup(t, c, name) + if v.Kind() != kind || v.Representation() != identity.RepresentationJSON { + t.Errorf("Lookup(%q): kind=%v representation=%v, want %v / JSON", name, v.Kind(), v.Representation(), kind) + } + if _, ok := v.Text(); ok != (kind == identity.ClaimString) { + t.Errorf("%s.Text() ok=%t, kind=%v", name, ok, kind) + } + if b, ok := v.Bool(); ok != (kind == identity.ClaimBoolean) || b != (name == "true") { + t.Errorf("%s.Bool() = %t, %t", name, b, ok) + } + if a, ok := v.Array(); ok != (kind == identity.ClaimArray) || len(a) != 0 { + t.Errorf("%s.Array() = %v, %t", name, a, ok) + } + if o, ok := v.Object(); ok != (kind == identity.ClaimObject) || o.Len() != 0 { + t.Errorf("%s.Object() length=%d, ok=%t", name, o.Len(), ok) + } + if _, ok := v.NumberText(); ok != (kind == identity.ClaimNumber) { + t.Errorf("%s.NumberText() ok=%t, kind=%v", name, ok, kind) + } + if _, ok := v.Float64(); ok != (kind == identity.ClaimNumber) { + t.Errorf("%s.Float64() ok=%t, kind=%v", name, ok, kind) + } + } + missing, ok := c.Lookup("missing") + if ok || missing.Kind() != identity.ClaimInvalid || missing.Representation() != identity.RepresentationUnknown { + t.Errorf("Lookup(missing) kind=%v representation=%v ok=%t", missing.Kind(), missing.Representation(), ok) + } + var empty identity.Claims + if empty.Len() != 0 || len(empty.Names()) != 0 { + t.Error("zero Claims is not empty") + } + if _, ok := empty.Lookup("anything"); ok { + t.Error("zero Claims contains a member") + } +} + +func TestClaimsNumberFidelity(t *testing.T) { + for _, literal := range []string{"9007199254740993", "18446744073709551615", "-0", "1.2300e+40", "1e9999"} { + t.Run(literal, func(t *testing.T) { + c := parseClaims(t, `{"n":`+literal+`}`) + n := lookup(t, c, "n") + if got, ok := n.NumberText(); !ok || got != literal { + t.Errorf("NumberText(%s) = %q, %t", literal, got, ok) + } + approx, ok := n.Float64() + if ok != (literal != "1e9999") { + t.Errorf("Float64(%s) = %v, %t", literal, approx, ok) + } + if literal == "-0" && !math.Signbit(approx) { + t.Error("Float64(-0) lost its sign") + } + }) + } + for _, tc := range []struct { + name string + input any + text string + }{ + {name: "int", input: int(-12), text: "-12"}, + {name: "int8", input: int8(-128), text: "-128"}, + {name: "int16", input: int16(-32768), text: "-32768"}, + {name: "int32", input: int32(-2147483648), text: "-2147483648"}, + {name: "int64", input: int64(math.MinInt64), text: "-9223372036854775808"}, + {name: "uint", input: uint(12), text: "12"}, + {name: "uint8", input: uint8(255), text: "255"}, + {name: "uint16", input: uint16(65535), text: "65535"}, + {name: "uint32", input: uint32(math.MaxUint32), text: "4294967295"}, + {name: "uint64", input: uint64(math.MaxUint64), text: "18446744073709551615"}, + {name: "json_number", input: json.Number("9007199254740993"), text: "9007199254740993"}, + {name: "float32", input: float32(42)}, + {name: "float64", input: float64(9007199254740992)}, + } { + t.Run(tc.name, func(t *testing.T) { + c, err := identity.NewClaims(map[string]any{"n": tc.input}) + if err != nil { + t.Fatal(err) + } + n := lookup(t, c, "n") + if text, ok := n.NumberText(); text != tc.text || ok != (tc.text != "") { + t.Errorf("NumberText(%T) = %q, %t; want %q", tc.input, text, ok, tc.text) + } + if _, ok := n.Float64(); !ok || n.Representation() != identity.RepresentationDecoded { + t.Errorf("numeric decoded input %T lost its value or representation", tc.input) + } + }) + } +} + +func TestGatewayTextIsNotReparsed(t *testing.T) { + input := map[string]string{"n": "12", "groups": `["admin"]`, "bool": "true", "null": "null"} + c, err := identity.NewTextClaims(input) + if err != nil { + t.Fatal(err) + } + for name, want := range input { + v := lookup(t, c, name) + if got, ok := v.Text(); !ok || got != want || v.Representation() != identity.RepresentationGatewayText { + t.Errorf("gateway %q = %q, %t, representation=%v", name, got, ok, v.Representation()) + } + } +} + +func TestClaimsOwnership(t *testing.T) { + array := []any{"first", map[string]string{"inner": "before"}} + nested := map[string]any{"list": array, "strings": []string{"before"}} + input := map[string]any{"nested": nested, "text": "before"} + c, err := identity.NewClaims(input) + if err != nil { + t.Fatal(err) + } + clear(input) + array[0] = "after" + array[1].(map[string]string)["inner"] = "after" + nested["strings"].([]string)[0] = "after" + clear(nested) + names := c.Names() + if !slices.Equal(names, []string{"nested", "text"}) { + t.Errorf("Names() = %v, want sorted names", names) + } + names[0] = "changed" + obj, ok := lookup(t, c, "nested").Object() + if !ok { + t.Fatal("owned nested object missing") + } + list, ok := lookup(t, obj, "list").Array() + if !ok || len(list) != 2 { + t.Fatalf("owned list length=%d, ok=%t", len(list), ok) + } + if text, ok := list[0].Text(); !ok || text != "first" { + t.Errorf("owned list[0] = %q, %t", text, ok) + } + inner, ok := list[1].Object() + if !ok { + t.Fatal("owned inner object missing") + } + if text, ok := lookup(t, inner, "inner").Text(); !ok || text != "before" { + t.Errorf("owned inner text = %q, %t", text, ok) + } + list[0] = identity.Claim{} + list, _ = lookup(t, obj, "list").Array() + if text, ok := list[0].Text(); !ok || text != "first" { + t.Errorf("Array result mutation changed original to %q, %t", text, ok) + } + texts, _ := lookup(t, obj, "strings").Array() + if text, ok := texts[0].Text(); !ok || text != "before" { + t.Errorf("owned []string element = %q, %t", text, ok) + } + wire := jsontext.Value(`{"key":"secret","n":9007199254740993}`) + parsed, err := identity.ParseClaims(wire) + if err != nil { + t.Fatal(err) + } + clear(wire) + if text, ok := lookup(t, parsed, "key").Text(); !ok || text != "secret" { + t.Errorf("input mutation changed parsed string to %q, %t", text, ok) + } + if text, ok := lookup(t, parsed, "n").NumberText(); !ok || text != "9007199254740993" { + t.Errorf("input mutation changed parsed number to %q, %t", text, ok) + } +} + +func TestNilContainerKinds(t *testing.T) { + c, err := identity.NewClaims(map[string]any{ + "map": map[string]any(nil), "strings_map": map[string]string(nil), + "array": []any(nil), "strings": []string(nil), "null": nil, + }) + if err != nil { + t.Fatal(err) + } + for name, want := range map[string]identity.ClaimKind{ + "map": identity.ClaimObject, "strings_map": identity.ClaimObject, + "array": identity.ClaimArray, "strings": identity.ClaimArray, "null": identity.ClaimNull, + } { + if got := lookup(t, c, name).Kind(); got != want { + t.Errorf("Kind(%s) = %v, want %v", name, got, want) + } + } +} + +type hostileClaim struct{} + +func (hostileClaim) String() string { panic("must not invoke arbitrary String methods") } +func (hostileClaim) MarshalJSON() ([]byte, error) { + panic("must not invoke arbitrary JSON methods") +} + +func TestInvalidTypedClaims(t *testing.T) { + type customString string + type customMap map[string]any + for name, input := range map[string]any{ + "nan": math.NaN(), "infinity": math.Inf(1), "float32_infinity": float32(math.Inf(-1)), + "custom": customString("text"), "custom_map": customMap{}, "pointer": new(1), + "uintptr": uintptr(1), "bytes": []byte("text"), "struct": hostileClaim{}, + "invalid_utf8": "\xff", "invalid_key": map[string]any{"\xff": nil}, + "invalid_number": json.Number("secret"), "spaced_number": json.Number(" 1"), + "trailing_space": json.Number("1 "), "leading_zero": json.Number("01"), + "multiple_numbers": json.Number("1 2"), "null_number": json.Number("null"), + "empty_number": json.Number(""), "hex_number": json.Number("0x1p2"), + } { + t.Run(name, func(t *testing.T) { + c, err := identity.NewClaims(map[string]any{"secret": input}) + if !errors.Is(err, identity.ErrInvalidClaims) || c.Len() != 0 { + t.Errorf("NewClaims(%s) length=%d error=%v; want invalid claims", name, c.Len(), err) + } + for cause := err; cause != nil; cause = errors.Unwrap(cause) { + if strings.Contains(cause.Error(), "secret") { + t.Errorf("error tree exposed claims: %v", cause) + } + } + }) + } +} + +func TestInvalidJSONClaims(t *testing.T) { + for _, input := range []string{ + ``, `null`, `[]`, `42`, `{"secret":`, `{"secret":1,"secret":2}`, + `{"x":1,"\u0078":2}`, `{} {}`, `{} secret`, "{\"secret\":\"\xff\"}", + `{"x":[1,]}`, `{"x":{"a":1,"a":2}}`, `{"x":"\ud800"}`, + } { + c, err := identity.ParseClaims(jsontext.Value(input)) + if !errors.Is(err, identity.ErrInvalidClaims) || c.Len() != 0 { + t.Errorf("ParseClaims(%q) length=%d error=%v; want invalid claims", input, c.Len(), err) + } + for cause := err; cause != nil; cause = errors.Unwrap(cause) { + if strings.Contains(cause.Error(), "secret") { + t.Errorf("error tree exposed JSON: %v", cause) + } + } + } +} + +func TestClaimsBudgets(t *testing.T) { + for _, tc := range []struct { + name string + input map[string]any + wire string + charge int + }{ + {name: "empty", wire: `{}`, charge: 64}, + {name: "string", input: map[string]any{"x": "a"}, wire: `{"x":"a"}`, charge: 130}, + {name: "unicode", input: map[string]any{"é": "é"}, wire: `{"é":"é"}`, charge: 132}, + {name: "number", input: map[string]any{"x": json.Number("1e2")}, wire: `{"x":1e2}`, charge: 132}, + {name: "nested", input: map[string]any{"x": []any{nil}}, wire: `{"x":[null]}`, charge: 193}, + } { + t.Run(tc.name, func(t *testing.T) { + for _, delta := range []int{-1, 0} { + maximum := tc.charge + delta + for name, construct := range map[string]func() (identity.Claims, error){ + "typed": func() (identity.Claims, error) { + return identity.NewClaims(tc.input, identity.WithClaimsBudget(maximum)) + }, + "raw": func() (identity.Claims, error) { + return identity.ParseClaims(jsontext.Value(tc.wire), identity.WithClaimsBudget(maximum)) + }, + } { + _, err := construct() + if delta == 0 { + if err != nil { + t.Errorf("%s exact budget %d rejected: %v", name, maximum, err) + } + continue + } + limit, ok := errors.AsType[*identity.ClaimsLimitError](err) + if !ok || limit.Maximum() != int64(maximum) || !errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("%s budget %d error=%v, want ClaimsLimitError with that maximum", name, maximum, err) + } + } + } + }) + } + _, err := identity.ParseClaims(jsontext.Value(`{}`+strings.Repeat(" ", 63)), identity.WithClaimsBudget(64)) + if !errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("wire length greater than 64 accepted: %v", err) + } + for _, opts := range [][]identity.ClaimsOption{ + {nil}, {identity.WithClaimsBudget(0)}, {identity.WithClaimsBudget(-1)}, {identity.WithClaimsBudget(6*1024*1024 + 1)}, + } { + if _, err := identity.NewClaims(nil, opts...); err == nil { + t.Error("invalid constructor configuration accepted") + } + } + if _, err := identity.NewClaims(nil, identity.WithClaimsBudget(0), identity.WithClaimsBudget(64)); err != nil { + t.Errorf("last budget assignment did not win: %v", err) + } + if _, err := identity.NewTextClaims(nil, identity.WithClaimsBudget(64)); err != nil { + t.Errorf("nil text map not an empty object: %v", err) + } + _, err = identity.NewTextClaims(map[string]string{"x": "a"}, identity.WithClaimsBudget(129)) + if !errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("text map did not enforce budget: %v", err) + } +} + +func TestClaimsBoundDepthAndExpansion(t *testing.T) { + for _, depth := range []int{64, 65} { + input := map[string]any{} + for range depth - 1 { + input = map[string]any{"x": input} + } + wire := strings.Repeat(`{"x":`, depth-1) + `{}` + strings.Repeat(`}`, depth-1) + _, typedErr := identity.NewClaims(input) + _, rawErr := identity.ParseClaims(jsontext.Value(wire)) + for name, err := range map[string]error{"typed": typedErr, "raw": rawErr} { + if depth == 64 && err != nil || depth == 65 && !errors.Is(err, identity.ErrInvalidClaims) { + t.Errorf("%s depth %d: %v", name, depth, err) + } + if errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("%s depth error misclassified as bytes: %v", name, err) + } + } + } + cycle := make(map[string]any) + cycle["self"] = cycle + if _, err := identity.NewClaims(cycle); !errors.Is(err, identity.ErrInvalidClaims) { + t.Errorf("cyclic map error=%v, want invalid claims", err) + } + var shared any = "leaf" + for range 18 { + shared = []any{shared, shared} + } + if _, err := identity.NewClaims(map[string]any{"shared": shared}); !errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("expanded shared subtree error=%v, want budget exceeded", err) + } +} + +func TestClaimsConcurrentReadsAndDiagnostics(t *testing.T) { + c := parseClaims(t, `{"secret_identifier":"secret_value","array":["secret_value"]}`) + v := lookup(t, c, "secret_identifier") + for _, value := range []any{c, v} { + for _, format := range []string{"%v", "%+v", "%#v"} { + if got := fmt.Sprintf(format, value); strings.Contains(got, "secret") { + t.Errorf("diagnostic %s exposed claims: %s", format, got) + } + } + } + var wg sync.WaitGroup + for range 8 { + wg.Go(func() { + for range 100 { + v, _ := c.Lookup("secret_identifier") + if text, ok := v.Text(); !ok || text != "secret_value" { + t.Errorf("concurrent Text() = %q, %t", text, ok) + } + names := c.Names() + names[0] = "changed" + } + }) + } + wg.Wait() +} + +func FuzzParseClaims(f *testing.F) { + for _, input := range []string{`{}`, `{"n":9007199254740993}`, `{"nested":[null,true,{"x":"y"}]}`, `{"x":1,"x":2}`, `null`} { + f.Add([]byte(input)) + } + f.Fuzz(func(t *testing.T, input []byte) { + c, err := identity.ParseClaims(input, identity.WithClaimsBudget(4096)) + if err != nil { + if !errors.Is(err, identity.ErrInvalidClaims) && !errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("ParseClaims returned unclassified input error: %v", err) + } + if c.Len() != 0 { + t.Error("ParseClaims returned a partial object on failure") + } + return + } + names := c.Names() + clear(input) + for _, name := range names { + v, ok := c.Lookup(name) + if !ok || v.Kind() == identity.ClaimInvalid || v.Representation() != identity.RepresentationJSON { + t.Error("accepted JSON lost its shape or ownership") + } + } + }) +} diff --git a/identity/claims_typed.go b/identity/claims_typed.go new file mode 100644 index 0000000..b3df6f9 --- /dev/null +++ b/identity/claims_typed.go @@ -0,0 +1,225 @@ +package identity + +import ( + "encoding/json" // Only the Number data type; all JSON processing uses jsontext. + "encoding/json/jsontext" + "io" + "math" + "strconv" + "strings" + "unicode/utf8" +) + +// NewClaims validates and owns already decoded values without a JSON round trip. +// Supported recursive values are nil, bool, string, built-in integers except +// uintptr, finite float32/float64, json.Number, map[string]any, map[string]string, +// []any and []string. Other defined types, pointers, structs and byte slices are +// rejected without calling user methods. A nil map is an empty object; a typed +// nil nested map/slice retains its object/array kind. A nil interface is null. +// Inputs must not change during construction; afterward they may be mutated. +// Invalid input matches ErrInvalidClaims; byte exhaustion is a ClaimsLimitError. +func NewClaims(values map[string]any, opts ...ClaimsOption) (Claims, error) { + return newTypedClaims(values, RepresentationDecoded, opts) +} + +// NewTextClaims owns gateway text without parsing embedded JSON or numbers. +// All values retain RepresentationGatewayText. Nil means an empty object. +// Ownership, options and errors follow NewClaims. +func NewTextClaims(values map[string]string, opts ...ClaimsOption) (Claims, error) { + return newTypedClaims(values, RepresentationGatewayText, opts) +} + +func newTypedClaims(values any, representation Representation, opts []ClaimsOption) (Claims, error) { + maximum, err := claimsBudget(opts) + if err != nil { + return Claims{}, err + } + budget := claimBudget{maximum: maximum, remaining: maximum} + if _, err := readTypedClaim(values, representation, &budget, 0, false); err != nil { + return Claims{}, err + } + budget.remaining = maximum + n, err := readTypedClaim(values, representation, &budget, 0, true) + if err != nil { + return Claims{}, err + } + return Claims{members: n.object, budget: maximum, charge: int(n.cost)}, nil +} + +// readTypedClaim preflights without copying when build is false. Container depth +// bounds cycles; each occurrence of a shared subtree consumes its own allowance. +func readTypedClaim(v any, representation Representation, budget *claimBudget, depth int, build bool) (claimNode, error) { + start := budget.remaining + if err := budget.take(claimCharge); err != nil { + return claimNode{}, err + } + n := claimNode{representation: representation} + switch v := v.(type) { + case nil: + n.kind = ClaimNull + case bool: + n.kind = ClaimBoolean + if v { + n.number = 1 + } + case string: + n.kind, n.text = ClaimString, v + case json.Number: + n.kind, n.text = ClaimNumber, string(v) + if err := budget.take(len(n.text)); err != nil { + return claimNode{}, err + } + if !validNumber(n.text) { + return claimNode{}, ErrInvalidClaims + } + if build { + n.text = strings.Clone(n.text) + } + n.cost = uint32(start - budget.remaining) + return n, nil + case int: + n.kind, n.text = ClaimNumber, strconv.FormatInt(int64(v), 10) + case int8: + n.kind, n.text = ClaimNumber, strconv.FormatInt(int64(v), 10) + case int16: + n.kind, n.text = ClaimNumber, strconv.FormatInt(int64(v), 10) + case int32: + n.kind, n.text = ClaimNumber, strconv.FormatInt(int64(v), 10) + case int64: + n.kind, n.text = ClaimNumber, strconv.FormatInt(v, 10) + case uint: + n.kind, n.text = ClaimNumber, strconv.FormatUint(uint64(v), 10) + case uint8: + n.kind, n.text = ClaimNumber, strconv.FormatUint(uint64(v), 10) + case uint16: + n.kind, n.text = ClaimNumber, strconv.FormatUint(uint64(v), 10) + case uint32: + n.kind, n.text = ClaimNumber, strconv.FormatUint(uint64(v), 10) + case uint64: + n.kind, n.text = ClaimNumber, strconv.FormatUint(v, 10) + case float32: + n.kind, n.number = ClaimNumber, float64(v) + case float64: + n.kind, n.number = ClaimNumber, v + case map[string]any: + if err := typedContainerBudget(budget, depth, len(v)); err != nil { + return claimNode{}, err + } + n.kind = ClaimObject + if build { + n.object = make(map[string]claimNode, len(v)) + } + for name, value := range v { + if err := claimNameBudget(budget, name); err != nil { + return claimNode{}, err + } + child, err := readTypedClaim(value, representation, budget, depth+1, build) + if err != nil { + return claimNode{}, err + } + if build { + n.object[strings.Clone(name)] = child + } + } + case map[string]string: + if err := typedContainerBudget(budget, depth, len(v)); err != nil { + return claimNode{}, err + } + n.kind = ClaimObject + if build { + n.object = make(map[string]claimNode, len(v)) + } + for name, value := range v { + if err := claimNameBudget(budget, name); err != nil { + return claimNode{}, err + } + child, err := readTypedClaim(value, representation, budget, depth+1, build) + if err != nil { + return claimNode{}, err + } + if build { + n.object[strings.Clone(name)] = child + } + } + case []any: + if err := typedContainerBudget(budget, depth, len(v)); err != nil { + return claimNode{}, err + } + n.kind = ClaimArray + if build { + n.array = make([]claimNode, len(v)) + } + for i, value := range v { + child, err := readTypedClaim(value, representation, budget, depth+1, build) + if err != nil { + return claimNode{}, err + } + if build { + n.array[i] = child + } + } + case []string: + if err := typedContainerBudget(budget, depth, len(v)); err != nil { + return claimNode{}, err + } + n.kind = ClaimArray + if build { + n.array = make([]claimNode, len(v)) + } + for i, value := range v { + child, err := readTypedClaim(value, representation, budget, depth+1, build) + if err != nil { + return claimNode{}, err + } + if build { + n.array[i] = child + } + } + default: + return claimNode{}, ErrInvalidClaims + } + if math.IsNaN(n.number) || math.IsInf(n.number, 0) { + return claimNode{}, ErrInvalidClaims + } + if err := budget.take(len(n.text)); err != nil { + return claimNode{}, err + } + if !utf8.ValidString(n.text) { + return claimNode{}, ErrInvalidClaims + } + if build { + n.text = strings.Clone(n.text) + } + n.cost = uint32(start - budget.remaining) + return n, nil +} + +func typedContainerBudget(budget *claimBudget, depth, length int) error { + if depth >= maxClaimDepth { + return ErrInvalidClaims + } + if length > budget.remaining/claimCharge { + return &ClaimsLimitError{maximum: int64(budget.maximum)} + } + return nil +} + +func claimNameBudget(budget *claimBudget, name string) error { + if err := budget.take(len(name)); err != nil { + return err + } + if !utf8.ValidString(name) { + return ErrInvalidClaims + } + return nil +} + +func validNumber(s string) bool { + d := jsontext.NewDecoder(strings.NewReader(s)) + token, err := d.ReadToken() + if err != nil || token.Kind() != '0' || token.String() != s { + return false + } + _, err = d.ReadToken() + return err == io.EOF +} diff --git a/identity/errors.go b/identity/errors.go new file mode 100644 index 0000000..158a21b --- /dev/null +++ b/identity/errors.go @@ -0,0 +1,45 @@ +package identity + +import "errors" + +var ( + // ErrInvalidClaims indicates malformed JSON, unsupported typed values, + // invalid UTF-8, nonfinite numbers, or excessive container nesting. + ErrInvalidClaims = errors.New("identity: invalid claims") + // ErrClaimsLimit indicates exhaustion of the wire or weighted byte allowance. + ErrClaimsLimit = errors.New("identity: claims budget exceeded") +) + +// ClaimsLimitError describes a claims byte allowance failure. Its diagnostics +// contain no claim names, values, or underlying parser errors. +// The zero value has an unspecified maximum of zero. +type ClaimsLimitError struct { + maximum int64 +} + +// Error returns a sanitized description of the failure. +func (e *ClaimsLimitError) Error() string { return "identity: claims budget exceeded" } + +// Unwrap returns ErrClaimsLimit. +func (e *ClaimsLimitError) Unwrap() error { return ErrClaimsLimit } + +// Maximum returns the configured allowance in bytes, or zero for a nil error. +func (e *ClaimsLimitError) Maximum() int64 { + if e == nil { + return 0 + } + return e.maximum +} + +type claimBudget struct { + maximum int + remaining int +} + +func (b *claimBudget) take(n int) error { + if n > b.remaining { + return &ClaimsLimitError{maximum: int64(b.maximum)} + } + b.remaining -= n + return nil +} diff --git a/identity_errors_test.go b/identity_errors_test.go new file mode 100644 index 0000000..6a04f2e --- /dev/null +++ b/identity_errors_test.go @@ -0,0 +1,64 @@ +package edge + +import ( + "encoding/json/jsontext" + "errors" + "strings" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func TestClaimsInvocationErrors(t *testing.T) { + _, malformed := identity.ParseClaims(jsontext.Value(`{"secret":`)) + _, exceeded := identity.NewClaims(map[string]any{"secret": "value"}, identity.WithClaimsBudget(64)) + deep := map[string]any{} + for range 64 { + deep = map[string]any{"secret": deep} + } + _, depth := identity.NewClaims(deep) + for _, tc := range []struct { + name string + cause error + limit bool + }{ + {name: "malformed", cause: malformed}, + {name: "bytes", cause: exceeded, limit: true}, + {name: "depth", cause: depth}, + {name: "unexpected", cause: errors.New("secret")}, + } { + t.Run(tc.name, func(t *testing.T) { + if tc.cause == nil { + t.Fatal("fixture unexpectedly accepted") + } + err := claimsError(tc.cause) + if !errors.Is(err, ErrIdentity) || errors.Is(err, ErrLimitExceeded) != tc.limit || err.Operation() != "identity" { + t.Errorf("claimsError(%s) category or operation incorrect: %v", tc.name, err) + } + name, maximum, ok := err.Limit() + if ok != tc.limit || tc.limit && (name != "identity_claims" || maximum != 64) { + t.Errorf("claimsError(%s).Limit() = %q, %d, %t", tc.name, name, maximum, ok) + } + if tc.name == "depth" && !errors.Is(err, identity.ErrInvalidClaims) { + t.Error("depth failure lost its structural classification") + } + var visit func(error) + visit = func(cause error) { + if cause == nil { + return + } + if strings.Contains(cause.Error(), "secret") { + t.Errorf("error tree exposed input: %v", cause) + } + if joined, ok := cause.(interface{ Unwrap() []error }); ok { + for _, child := range joined.Unwrap() { + visit(child) + } + return + } + visit(errors.Unwrap(cause)) + } + visit(err) + }) + } +} diff --git a/identity_sdk_test.go b/identity_sdk_test.go new file mode 100644 index 0000000..c461c2f --- /dev/null +++ b/identity_sdk_test.go @@ -0,0 +1,37 @@ +package edge_test + +import ( + "context" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/identity" + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-lambda-go/lambda" +) + +func TestSDKTypedClaimsFidelity(t *testing.T) { + const payload = `{"requestContext":{"authorizer":{"claims":{"counter":9007199254740993,"groups":["staff"]}}}}` + for _, useNumber := range []bool{false, true} { + var claims identity.Claims + handler := lambda.NewHandlerWithOptions(func(ctx context.Context, event events.APIGatewayProxyRequest) (bool, error) { + var err error + claims, err = identity.NewClaims(event.RequestContext.Authorizer["claims"].(map[string]any)) + return err == nil, err + }, lambda.WithUseNumber(useNumber)) + if _, err := handler.Invoke(t.Context(), []byte(payload)); err != nil { + t.Fatal(err) + } + number, present := claims.Lookup("counter") + text, exact := number.NumberText() + if !present || exact != useNumber || useNumber && text != "9007199254740993" { + t.Errorf("UseNumber(%t): present=%t exact=%t text=%q", useNumber, present, exact, text) + } + if approximate, ok := number.Float64(); !ok || approximate != float64(9007199254740992) { + t.Errorf("UseNumber(%t): Float64()=%v, %t", useNumber, approximate, ok) + } + groups, _ := claims.Lookup("groups") + if array, ok := groups.Array(); !ok || len(array) != 1 { + t.Errorf("UseNumber(%t): typed array lost: %v, %t", useNumber, array, ok) + } + } +} From 71e82ab522e25cbd1b27565efc851e9674a512b9 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 14:38:04 +0100 Subject: [PATCH 24/44] docs: accept IAM proof authentication and exported client helper --- README.md | 7 ++ docs/decisions/0012-shared-identity.md | 7 ++ docs/decisions/0015-iam-caller-forms.md | 46 ++++++++ .../0016-mixed-iam-oauth-authentication.md | 109 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 23 +++- 6 files changed, 191 insertions(+), 2 deletions(-) create mode 100644 docs/decisions/0016-mixed-iam-oauth-authentication.md diff --git a/README.md b/README.md index f9b7ff7..bceed06 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,13 @@ claims API and resource limits are implemented in `identity`. Caller/context construction, JWT normalization and gateway producers remain pending; the initial IAM caller forms are recorded separately for review. +The accepted authentication direction supports IAM credentials or an OAuth +bearer token on the same route without built-in `AWS_IAM` authorization. IAM +clients will use an exported Go helper to generate a signed STS GetCallerIdentity +proof; the server verifier submits that client-signed request to AWS and uses the +verified identity for application authorization. The helper, verifier and their +client/server examples are planned; the detailed protocol/API is not implemented. + The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle is covered by race-enabled synthetic concurrency tests. Bounded JSON v2 metadata diff --git a/docs/decisions/0012-shared-identity.md b/docs/decisions/0012-shared-identity.md index 34dcbb0..2d60d03 100644 --- a/docs/decisions/0012-shared-identity.md +++ b/docs/decisions/0012-shared-identity.md @@ -360,3 +360,10 @@ The claims constructors, immutable views and resource/error foundation are now implemented under accepted decision 0014. Caller/context construction and native gateway producers remain pending. Decision 0015 presents the supported IAM caller forms left open by I5 for review before the complete caller union is published. + +Accepted follow-up 0016 adds an online STS-verified IAM proof producer alongside +the gateway/custom producers described in I2. Its provenance representation will +be settled in the protocol/API review. For I8, one explicitly configured selector +may choose IAM proof or OAuth verification per request on the same route. It +still installs only one caller and never retries another credential mechanism +after authentication failure or merges identities. diff --git a/docs/decisions/0015-iam-caller-forms.md b/docs/decisions/0015-iam-caller-forms.md index b650fb3..3247dec 100644 --- a/docs/decisions/0015-iam-caller-forms.md +++ b/docs/decisions/0015-iam-caller-forms.md @@ -54,3 +54,49 @@ form only when a documented producer needs it, with an explicit fixture. This review gates the IAM constructor and complete caller union; it does not gate the approved claims API, resource limits, or numerical fidelity implementation. + +## Clarification: validation versus authentication + +The user asked what IAM validation can actually establish and whether authn can +authenticate an IAM token. Constructor validation means structural validity and +internal consistency only: ARN components, a supported principal resource form, +required names/session components, and agreement with an independently supplied +caller account. It cannot establish existence, possession of credentials, session +validity, permissions, or that the supplied event came from API Gateway. An ARN +and a self-declared source are identifiers/annotations, not authentication proofs. + +The [IAM identifiers reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html) +documents ARN components, blank regions for these IAM/STS forms, and exact +session representations. The initial supported-form restriction is our caller +model policy, not a test proving a principal is real. + +[Gateway IAM authorization](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control-iam.html) +requires signed requests and checks execute-api permission before invoking a +route. Recommend that path for IAM clients of edge's API Gateway integration: +AWS authenticates the request; edge validates and owns the resulting assertion. +This uses built-in AWS_IAM authorization, not a Lambda/JWT authorizer. Trust still +depends on the intended integration and permissions controlling Lambda invocation. + +An STS SessionToken alone is not a standalone bearer authentication protocol. +[Temporary credentials](https://aws.amazon.com/blogs/security/understanding-the-api-options-for-securely-delegating-access-to-your-aws-account/) +include an access key, secret key and session token; the keys sign requests and +the session token accompanies them. Calling +[GetCallerIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) +with the Lambda's credentials identifies the Lambda's role, not the HTTP caller. + +IAM-backed bearer authentication is possible with a specifically designed proof +protocol. [EKS documents](https://docs.aws.amazon.com/eks/latest/best-practices/identity-and-access-management.html) +a token containing a presigned STS request that its authenticator submits to AWS +for verification. A future edge authn component could use this general approach, +but would need its own explicit protocol, application binding, freshness/replay +policy, restricted STS endpoints and dependency-error handling. An arbitrary +execute-api signature is not interchangeable with an STS-signed proof. + +Recommend keeping that optional online authentication producer separate from +NewIAM and the gateway assertion adapter. It would also require a reviewed source +designation; the current source contract must not mislabel AWS verification as +locally verified JWT authentication. No such producer is approved or implemented. + +The user's subsequent same-route IAM/OAuth clarification is recorded in 0016. +For that goal, the proposed path is explicit mixed-credential middleware with an +online IAM proof verifier. Built-in AWS_IAM alone does not satisfy the goal. diff --git a/docs/decisions/0016-mixed-iam-oauth-authentication.md b/docs/decisions/0016-mixed-iam-oauth-authentication.md new file mode 100644 index 0000000..0eac94b --- /dev/null +++ b/docs/decisions/0016-mixed-iam-oauth-authentication.md @@ -0,0 +1,109 @@ +# 0016: IAM credentials or OAuth tokens on the same route + +Status: architecture and exported Go client helper accepted on 2026-09-16. +Detailed protocol/API design remains pending. No authentication producer or +client helper has been implemented. + +## Requirement and correction + +The original goal includes accepting IAM credentials or an OAuth bearer token +when calling the same API. Design for the same route and application handler, +without requiring built-in AWS_IAM authorization. The recommendation in 0015 to +prefer AWS_IAM for IAM clients alone does not meet that combined requirement. +Gateway IAM remains an optional deployment path, not a prerequisite or the sole +planned IAM authentication path. + +## Evidence + +- [API Gateway route authorization](https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-apigatewayv2-2018-11-29.html) + selects one authorization type, including NONE, JWT, AWS_IAM or CUSTOM. It is + not a built-in per-request choice between IAM and JWT authorization. +- [SigV4 signing](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html) + derives an HMAC key from the secret access key, date, region and service. Edge + does not possess a caller's IAM secret. An execute-api signature cannot be + transplanted onto an STS request for verification. +- [EKS authentication](https://docs.aws.amazon.com/eks/latest/best-practices/identity-and-access-management.html) + demonstrates tokens containing a presigned STS request submitted to AWS for + verification, followed by separate application authorization. +- [AWS IAM Authenticator](https://github.com/kubernetes-sigs/aws-iam-authenticator#how-does-it-work) + documents client presigning, server validation and submission to STS, and a + signed destination identifier preventing reuse across different clusters. + This establishes a protocol pattern, not a dependency or wire-format choice. +- [GetCallerIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) + identifies the credentials signing that request and requires no granted + permission for the operation. It does not establish application permissions. + +AWS definitions were checked through public AWS MCP; the authenticator's own +documentation supplies implementation-pattern evidence. Current code has no authn +middleware. The source rules in 0012 permit gateway/custom IAM and locally verified +JWT; online IAM verification requires a provenance refinement before implementation. + +## Accepted direction + +Configure a route without built-in gateway authentication and protect the handler +with explicit authn middleware. Select one configured verifier by an unambiguous +credential format, never by retrying another verifier after authentication fails. + +OAuth bearer input goes to its configured token verifier. The planned Cognito +path validates JWT access tokens; opaque OAuth tokens need a separately configured +issuer-specific mechanism. IAM clients use their existing AWS credential provider +to sign an STS GetCallerIdentity proof. An exported Go helper included in this +module generates the proof; this is a required deliverable, not an optional +consumer-owned implementation. The secret key stays with the client. + +The IAM verifier validates the proof contract and submits only an approved STS +request to AWS, using the client's signature rather than the Lambda's credentials. +Identity comes from the verified AWS response, not an ARN asserted by the client. +Either successful verifier constructs one immutable caller; common application +authorization decides whether it may execute the selected action. Successful IAM +authentication alone must not admit every AWS account to the application. + +This is one configured authentication selector choosing one producer per request, +compatible with the no-identity-merging direction in 0012. Conflicting credentials +must fail rather than select an identity by precedence. + +The proof requires signed application/environment binding, bounded freshness, an +explicit replay policy, constrained endpoint/action/method validation and bounded +network work. Dependency failures must remain distinct from invalid credentials. +Wire format, source enum, caching and detailed verification rules remain proposed +work for a concrete protocol review. + +## Tradeoffs and alternatives + +The IAM path adds an online STS dependency and a client protocol/helper. It is not +drop-in compatibility with generic execute-api SigV4 clients. With middleware +placement, unauthenticated traffic reaches the integration Lambda. + +A custom Lambda authorizer could host the same verifier selection before the +integration. It changes deployment/caching/assertion transport but does not remove +the proof requirement. Separate IAM/JWT routes could use native authorization and +a common backend, but would not meet the stricter same-route goal. + +## Resolution and deliverables + +The user confirmed that the client token is a signed GetCallerIdentity proof +which the server submits to STS to obtain verified identity. The user requires +documentation and a Go helper in the module's exported API. The client-helper +compatibility question is resolved; unchanged execute-api SigV4 clients are not +the contract of this authentication path. + +Deliver: + +- An exported Go client helper that generates the accepted proof using the + caller's AWS credentials, including temporary credentials where available. +- An authn verifier that validates the proof and forwards the client-signed STS + request without signing it again using server credentials. Construct identity + only from the successfully verified response. +- Explicit selection between IAM proof and configured OAuth bearer verification + on the same route, producing one caller for common application authorization. +- Consumer documentation and runnable client/server examples covering credential + acquisition, matching application binding, token transport and lifetime, + temporary credential handling, and authentication versus authorization. +- Contract tests for client/verifier interoperability, destination binding, + freshness, malformed/conflicting credentials, endpoint restrictions and STS + failure handling. Use local tests; no live AWS invocation is needed by default. + +The next design review will settle helper package placement and signatures, +credential-provider dependencies, versioned wire format and HTTP transport, +application binding, expiry/replay policy, endpoint/region policy, provenance and +error behavior. Acceptance of the architecture does not freeze these details. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index a59f5a8..50d8666 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -24,3 +24,4 @@ Keep related decisions small enough to review and implement independently. | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | | [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | | [0015: IAM caller forms](0015-iam-caller-forms.md) | Proposed | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | +| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture and exported Go helper accepted; protocol/API pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | diff --git a/docs/plan.md b/docs/plan.md index 7b3b273..40fd3d3 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -20,6 +20,13 @@ port, source copy, or compatibility-preserving update of Beakley. handling. Do not route our codec through `encoding/json`'s compatibility API. Check third-party parser behavior separately during dependency selection. - Keep ordinary `http.Handler` as the application boundary. +- Support the user's same-route goal: clients may authenticate using IAM + credentials or an OAuth bearer token without requiring built-in AWS_IAM + authorization. [Decision 0016](decisions/0016-mixed-iam-oauth-authentication.md) + accepts explicit credential selection and online STS-proof verification. + Include an exported Go helper that generates the signed GetCallerIdentity + proof, with consumer documentation and runnable client/server examples. + Detailed protocol and helper/verifier APIs remain a design-review step. - Provide first-class consumer request-header processing for header-selected action dispatchers. Preserve custom incoming headers independently of response combination rules. [Decision 0011](decisions/0011-request-header-processing.md) @@ -62,7 +69,7 @@ multiple identity producers remain separate decisions. | --- | --- | | `edge` (root) | Invocation adapter, payload decoding, HTTP translation, gateway metadata, optional gateway identity extraction | | `identity` | Validated callers, claim representation, context transport; standard library only | -| `authn` | Bearer authentication, explicitly named Cognito verification, JWKS caching | +| `authn` | Explicit IAM-proof/OAuth selection, online STS verification, Cognito verification and JWKS caching | | `authz` | Decisions, rules, combinators, middleware, application-principal resolution | | `edgetest` | Consumer identity fixtures and gateway event helpers | @@ -71,6 +78,10 @@ event types out of transport-independent packages. Reassess any additional exported package against a concrete consumer need; version-reporting machinery does not need to precede a working adapter. +The accepted IAM client helper belongs to the module's exported Go API. Its +package placement and credential-provider dependency are part of the detailed +0016 protocol review; do not assume a new package or root reexport prematurely. + ## Milestones and acceptance criteria ### 0. Persist the design and working agreement @@ -256,7 +267,15 @@ Native-producer fixture review remains separate from the claims approval. Acceptance: invalid/ambiguous identities cannot authorize; gateway and token claims cannot be silently mixed; unavailable claim fidelity remains explicit. -### 4. Cognito verification and JWKS resilience +### 4. Local authentication: Cognito/JWKS and IAM credentials + +- Implement the accepted client-helper/server-verifier direction in decision 0016 + after its detailed protocol review. Review a single authentication + selector, online STS proof verification and its provenance, application binding, + freshness/replay rules, endpoint constraints and dependency-error semantics. + Do not promise offline IAM verification or ordinary SigV4-client compatibility. +- Provide exported Go proof generation and consumer documentation with runnable + examples showing IAM and OAuth clients calling the same protected route. - Review a verifier requiring explicit issuer/client restrictions; accepting all pool clients must be a deliberate policy. From d46ba1d4081ace385feefc602a368143459131a6 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 15:49:48 +0100 Subject: [PATCH 25/44] docs: propose IAM proof protocol and verify SDK reconstruction --- .../0016-mixed-iam-oauth-authentication.md | 4 + docs/decisions/0017-iam-proof-protocol.md | 287 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 8 + docs/probes/iamproof/README.md | 44 +++ docs/probes/iamproof/go.mod | 17 ++ docs/probes/iamproof/go.sum | 16 + docs/probes/iamproof/proof_test.go | 129 ++++++++ 8 files changed, 506 insertions(+) create mode 100644 docs/decisions/0017-iam-proof-protocol.md create mode 100644 docs/probes/iamproof/README.md create mode 100644 docs/probes/iamproof/go.mod create mode 100644 docs/probes/iamproof/go.sum create mode 100644 docs/probes/iamproof/proof_test.go diff --git a/docs/decisions/0016-mixed-iam-oauth-authentication.md b/docs/decisions/0016-mixed-iam-oauth-authentication.md index 0eac94b..3acbf12 100644 --- a/docs/decisions/0016-mixed-iam-oauth-authentication.md +++ b/docs/decisions/0016-mixed-iam-oauth-authentication.md @@ -107,3 +107,7 @@ The next design review will settle helper package placement and signatures, credential-provider dependencies, versioned wire format and HTTP transport, application binding, expiry/replay policy, endpoint/region policy, provenance and error behavior. Acceptance of the architecture does not freeze these details. + +[Decision 0017](0017-iam-proof-protocol.md) now proposes those details, supported +by a standalone SDK reconstruction/size probe. It awaits user review; the probe +is not a production helper or authentication implementation. diff --git a/docs/decisions/0017-iam-proof-protocol.md b/docs/decisions/0017-iam-proof-protocol.md new file mode 100644 index 0000000..2592070 --- /dev/null +++ b/docs/decisions/0017-iam-proof-protocol.md @@ -0,0 +1,287 @@ +# 0017: IAM proof protocol and Go helper + +Status: proposed on 2026-09-16; awaiting review. Decision 0016 accepts the +architecture and exported helper, not the protocol/API details below. Only an +isolated SDK probe is implemented; there is no production IAM authentication yet. + +## Evidence + +AWS documentation was checked through public AWS MCP. SDK public API definitions +were checked with the Go documentation skill at the versions below. The existing +identity package implements claims, not Caller or an authentication producer. + +- [GetCallerIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) + returns Account, Arn and UserId for the signing credentials, without requiring + a granted permission. Its documented response is XML. It does not grant + permission to execute an application action. +- [Regional STS endpoints](https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html) + are AWS's recommendation. Use an explicit region, not implicit global routing. +- [SDK v4 presigning](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2@v1.47.0/aws/signer/v4#Signer.PresignHTTP) + returns the signed URL and required signed headers. It does not automatically + set expiration; its documentation warns that not every service uses + X-Amz-Expires. Do not infer a hard STS lifetime from that parameter. +- [AWS IAM Authenticator](https://github.com/kubernetes-sigs/aws-iam-authenticator/blob/master/pkg/token/token.go) + provides implementation evidence for a signed destination header and explicit + date checks. Its code notes that STS ignores X-Amz-Expires. That is reference + implementation evidence, not an AWS-documented service guarantee. +- [HTTP API quotas](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-quotas.html) + cap combined request-line and header-value size at 10,240 bytes. Proof transport + must share that space with the action header and all other request metadata. +- [STS credentials](https://docs.aws.amazon.com/STS/latest/APIReference/API_Credentials.html) + do not have a fixed documented session-token size. A library transport limit + must not be described as AWS's maximum credential size. +- [STS quotas](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html) + document a default shared 600 requests/second per account per Region for + credential-based STS requests, including GetCallerIdentity. +- [STS common errors](https://docs.aws.amazon.com/STS/latest/APIReference/CommonErrors.html) + include expired credentials/requests and unrecognized clients, but also a + throttling error with HTTP status 400. HTTP status alone cannot distinguish + invalid credentials from dependency failure. +- [Go 1.27](https://go.dev/doc/go1.27) and the existing module contract support + using encoding/json/v2 directly. XML responses use encoding/xml; no JSON v1 + shim is needed for the proposed protocol. + +Latest stable modules resolved on 2026-09-16: AWS SDK core v1.47.0, STS v1.51.0, +and config v1.33.5. Lambda Go remains v1.55.0. The [standalone probe](../probes/iamproof/README.md) +uses the first two; the root dependency list is unchanged. Recheck stable versions +when adding production dependencies. Explicit module versions provide repeatable +builds, not a permanent freeze on an old SDK. + +## P1. Public package and credential ownership + +Recommend one focused `iamproof` companion package for the exported client helper +and server verifier. Keep Caller in AWS-free `identity`; keep HTTP credential +selection in `authn`. This concrete client use justifies an additional package +under the plan's package-boundary rule. Avoid importing IAM SDK implementation +packages from shared selector code: accept a consumer-defined verifier function +there, so a JWT-only application does not link the IAM implementation. + +Proposed surface, with opaque concrete Generator, Verifier and Token types: + +```go +func NewGenerator(region, audience string, credentials aws.CredentialsProvider) (*Generator, error) +func (g *Generator) Generate(ctx context.Context) (Token, error) +func (t Token) Value() string +func (t Token) ExpiresAt() time.Time +func NewVerifier(region, audience string, opts ...VerifierOption) (*Verifier, error) +func WithTransport(transport http.RoundTripper) VerifierOption +func WithTimeout(timeout time.Duration) VerifierOption +func (v *Verifier) Verify(ctx context.Context, token string) (identity.Caller, error) +``` + +Value returns the versioned credential; the HTTP client sends `EdgeIAM ` followed +by that value. ExpiresAt is the earlier of proof expiry and credential expiry +when known, an upper bound rather than a guarantee of future AWS acceptance. +Token's ordinary string/debug formatting is redacted; extracting Value is +explicit. The zero Token is unusable and exposes empty value/zero expiry. + +Use the SDK's CredentialsProvider directly. Applications load/cache/refresh +credentials using their existing SDK configuration; the helper does not invent +another provider hierarchy or call AssumeRole on their behalf. Generate retrieves +credentials each time, so the provider can refresh. The provider may perform I/O; +local presigning itself makes no STS request. Require complete, unexpired +credentials; preserve temporary session tokens. Ignore provider AccountID as +authentication evidence. Configuration and objects are immutable after creation; +concurrent use requires a concurrency-safe provider/transport. + +Use SDK v4.Signer for signing and STS's public endpoint resolver for endpoint +metadata. STS PresignGetCallerIdentity is an alternative, but inserting the +application header and fixing signing time/expiry requires middleware hooks. +The small, fixed Query request makes the lower-level official signer clearer. +Never implement SigV4 cryptography ourselves. Verifier accepts no credentials +provider and never signs the forwarded request with the Lambda role. + +Alternative: put everything in authn or reexport it from edge. That couples +client generation to HTTP middleware or broadens the adapter's root API. + +## P2. Wire format and one credential selection + +Recommend `Authorization: EdgeIAM v1.`. Bearer continues +to select the configured OAuth verifier. Scheme matching is case insensitive; +the version and encoded token are strict. Repeated, combined, empty, unknown or +malformed credentials fail. Never try another verifier after a rejection. +Do not accept a second proof in a cookie, query parameter or alternate header. + +The JSON object has exactly these fields: + +```json +{"audience":"orders.production","credential":"ACCESSKEY/20260916/eu-west-2/sts/aws4_request","date":"20260916T120000Z","signature":"64-lowercase-hex-digits","sessionToken":"optional-temporary-credential-token"} +``` + +The signature placeholder above illustrates a field, not a valid test vector. +Require all fields except sessionToken, reject nulls/unknown/duplicate fields, +invalid UTF-8 and trailing data, and require canonical unpadded base64url. +Absent sessionToken means long-lived credentials; a present empty token fails. +Use JSON v2 directly. JSON member order does not affect the AWS signature. + +All remaining request components are fixed: HTTPS GET, regional STS host, `/`, +no body, Action=GetCallerIdentity, Version=2011-06-15, +X-Amz-Algorithm=AWS4-HMAC-SHA256, X-Amz-Expires=60, and +X-Amz-SignedHeaders=host;x-edge-iam-audience. The verifier constructs this request +from trusted configuration plus the validated variable fields. It accepts no +client-supplied URL, method, action, endpoint override or arbitrary headers. + +Credential scope must match the date, configured region, service sts and terminal +aws4_request. Validate syntax without claiming that an access key exists. Only +STS can authenticate it. Reject inputs before I/O where possible. + +Alternative: base64url of a complete presigned URL, as used by established +authenticator protocols. It is easier to inspect but admits more parser states +and expands escaped session tokens. The local probe reconstructs the SDK's signed +request across twelve cases and confirms the audience changes the signature. +A synthetic 4 KiB escape-heavy session token produces a 5,767-byte compact proof +versus 14,175 bytes for an encoded URL. This is a stress measurement, not a claim +about typical AWS tokens. The new wire format is our protocol, not EKS-compatible. + +## P3. Destination binding, lifetime and replay + +Require an explicit, matching application/environment audience at both ends. +Use 1-256 visible ASCII bytes without spaces, case sensitive and unnormalized; +deployments should use unique values such as orders.production. Compare the +envelope audience to server configuration before I/O; send the configured value +in X-Edge-IAM-Audience. Its inclusion in SignedHeaders makes changing it invalidate +the signature. A matching plaintext audience by itself proves nothing. + +Recommend a fixed 60-second proof lifetime in protocol v1, with at most 30 seconds +of future clock tolerance and no expired-token grace. Sign whole UTC seconds; +server requires issuedAt <= now+30s and now < issuedAt+60s. Recheck after STS returns +before creating a caller. Future tolerance means the maximum possible local +acceptance interval is 90 seconds, not a promised 60 seconds of wall time under +skew. STS may reject more strictly. Credential expiration may shorten validity. + +This is a bearer authentication proof: it can be reused for the same audience +while fresh. It does not bind the API method, action header or request body and +is not a single-use challenge. Require HTTPS to the application and STS. Client +code may reuse a generated proof until its expiry; every server verification +still calls STS. Do not log proof values, session tokens or signed URLs. + +Alternative: request-body binding or a nonce/challenge store. Either changes the +client flow and adds a separate protocol/state requirement. Recommend the short, +audience-bound bearer contract first, with this limitation explicit rather than +claiming general replay prevention. An STS success is not application authorization. + +## P4. Endpoints, limits and online verification + +Require one explicit regional endpoint per generator/verifier, resolved with +the stable STS SDK metadata. Validate configured region syntax; resolution is not +a promise that an account can use that region. Initial support uses standard +regional HTTPS endpoints; no global, FIPS, dual-stack or arbitrary URL switches. +Do not silently fall back to global STS or another region/partition. The probe +checks commercial, GovCloud and China resolution, not live AWS availability. + +Proposed engineering limits, not claimed AWS limits: + +| Boundary | Initial policy | Reason | +| --- | --- | --- | +| Encoded credential, including v1. prefix | 8 KiB, fixed | Bound decoding and leave some of the HTTP API header budget for other fields | +| STS response body | 64 KiB, fixed | Bound success/error XML parsing; the documented identity is small | +| Verification network deadline | 5 seconds by default; positive override | Bound STS work; caller's earlier context deadline still wins | +| Automatic retries / verification cache | None | Predictable per-request work and fresh AWS verification | + +Check the encoded limit before allocation and on generation. Return an explicit +size error from the helper; never truncate session credentials. The 8 KiB policy +does not guarantee that a request with other headers fits its gateway, nor that +every valid STS session fits this transport. The probe's 6 KiB synthetic session +already exceeds it. Endpoint policies/proxies may impose additional URL limits. + +Use an owned HTTP client with redirects disabled, no cookie jar, explicit context +deadline and bounded reads. Close every response body. WithTransport supplies +trusted deployment infrastructure, including custom TLS/proxy settings; it is +not an input taken from a request. A transport that ignores cancellation or +performs retries internally is outside guarantees edge can enforce. + +Read the documented XML response using encoding/xml. Require one expected +response/result, exactly one nonempty Account/Arn/UserId, expected namespaces, +no duplicate identity fields and no second document. Bounded nonidentity metadata +can be ignored for forward compatibility. Cross-check account and partition +against the returned ARN and configured endpoint partition. Reject contradictory +or unsupported identities. Do not assume that setting Accept to JSON is a +documented GetCallerIdentity response contract. + +No server-side cache initially means an STS round trip for every IAM request. +The shared 600 RPS default makes this unsuitable for unbounded throughput without +capacity planning. Application throttling/admission control remains necessary. +Caching would extend acceptance after credential invalidation and requires its +own TTL, key, eviction and overload policy; review it separately if needed. + +## P5. Caller provenance and errors + +Recommend accepting the four initial forms from decision 0015: IAM root/user and +STS assumed-role/federated-user sessions. Preserve the full ARN/session and STS +UserId, cross-check Account, and add SourceVerifiedIAMProof for this producer. +Do not convert a session ARN into a role ARN or infer grants from it. Provenance +records what trusted producer code did; manually constructing the same enum does +not authenticate anything. Root representation does not permit root access. + +Verify returns a caller only after successful STS verification, response checks, +and the final freshness check; every failure returns an anonymous zero caller. +It does not install context. The shared authn selector installs one caller and +honors the existing prohibition on replacing/merging an existing caller. + +Recommend inspectable package errors ErrInvalidProof, ErrUnavailable, +ErrCredentialsUnavailable and ErrProofTooLarge. Constructor configuration errors +are programmer/deployment errors; use a separate ErrInvalidConfiguration. +All error messages/chains are sanitized; do not wrap provider errors, raw XML, +HTTP url.Error values or SDK signing diagnostics containing credential material. +Preserve caller cancellation/deadline via errors.Is without exposing unsafe causes. + +| Condition | Classification and protected HTTP behavior | +| --- | --- | +| Missing/malformed/expired/oversized incoming proof, wrong binding/scope | Invalid credentials; 401 | +| Documented STS invalid-key/token/signature rejection | Invalid credentials; 401 | +| DNS/TLS/network failure, internal timeout, throttle, 5xx, redirect, unexpected status/response | Dependency unavailable; 503 | +| Successful STS response with missing/contradictory/unsupported identity | Dependency protocol failure; 503 | +| Authenticated caller denied the selected application action | Authorization denial; 403 | +| Client provider retrieval/signing failure or expired/incomplete credentials | Helper error; no proof returned | + +Do not classify every AWS 4xx as invalid credentials. Start the invalid-credential +allowlist with documented ExpiredTokenException, IncompleteSignature, +MissingAuthenticationToken, RequestExpired and UnrecognizedClientException; +include InvalidClientTokenId as documented for GetCallerIdentity in +[AWS's EKS troubleshooting reference](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html). +Verify any further service-specific aliases against official sources and fixtures +before adding them. Unrecognized errors fail closed as unavailable; throttling +remains unavailable even with HTTP 400. Caller cancellation returns the context error; +writing a response after disconnection follows the HTTP middleware lifecycle. +WWW-Authenticate challenges and shared selector API belong to the HTTP authn +review, not the proof verifier. These are HTTP failures, not Lambda invocation +errors under decision 0013. + +## Design graph and implementation sequence + +```mermaid +flowchart LR + Credentials[SDK credentials provider] --> Generate[Generate audience-bound STS proof] + Generate --> Header[Authorization EdgeIAM] + Header --> Select[Select exactly one verifier] + Bearer[Authorization Bearer] --> Select + Select --> Shape[Validate proof before I/O] + Shape --> STS[Send reconstructed client-signed request to regional STS] + STS --> IAM[Validate response and freshness; create IAM caller] + Select --> OAuth[Configured OAuth verifier] + IAM --> Caller[Install one shared caller] + OAuth --> Caller + Caller --> Authorize[Authorize selected action] +``` + +1. Review P1-P5, including the still-pending caller forms in 0015. +2. Complete the shared caller/context foundation and native gateway fixtures, + then the callable buffered adapter in the already accepted order. +3. Implement helper and verifier together with round-trip fixtures, current stable + SDK dependencies and an explicit versioned protocol specification. +4. Test malformed/duplicate/oversized inputs, tampering, wrong region/audience, + clock boundaries, credential expiry, endpoint/redirect constraints, XML + contradictions, cancellation, dependency classification and redacted errors. + Use synthetic credentials, fake transports and local servers, not live AWS. +5. Review/wire the common authn selector and complete IAM/OAuth client/server + examples plus authz examples. Keep action extraction separate from proof + parsing and require authorization for each selected action. + +The probe passes local SDK reconstruction and audience-change checks with race +detection. It does not prove real STS acceptance, deploy API Gateway, implement +the security validation above, or settle HTTP middleware APIs. A separately +authorized integration check can later validate deployed interoperability. + +## Resolution + +Pending user review. No production API or trust policy is changed by this record. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 50d8666..eccdeff 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -25,3 +25,4 @@ Keep related decisions small enough to review and implement independently. | [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | | [0015: IAM caller forms](0015-iam-caller-forms.md) | Proposed | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | | [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture and exported Go helper accepted; protocol/API pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | +| [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Proposed; isolated SDK probe passes | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | diff --git a/docs/plan.md b/docs/plan.md index 40fd3d3..a2aa2eb 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -81,6 +81,10 @@ does not need to precede a working adapter. The accepted IAM client helper belongs to the module's exported Go API. Its package placement and credential-provider dependency are part of the detailed 0016 protocol review; do not assume a new package or root reexport prematurely. +[Decision 0017](decisions/0017-iam-proof-protocol.md) proposes a focused `iamproof` +package, compact versioned proof, signed audience, short freshness policy and +bounded regional STS verification. Its isolated SDK probe passes; all production +API and trust-boundary choices in that record remain pending review. ## Milestones and acceptance criteria @@ -269,6 +273,10 @@ claims cannot be silently mixed; unavailable claim fidelity remains explicit. ### 4. Local authentication: Cognito/JWKS and IAM credentials +- Review the concrete protocol/helper/verifier proposal in decision 0017, + including replay, size and STS quota tradeoffs, before dependent implementation. + The standalone probe uses latest stable SDK core v1.47.0 and STS v1.51.0 as + resolved on 2026-09-16; root production dependencies have not changed. - Implement the accepted client-helper/server-verifier direction in decision 0016 after its detailed protocol review. Review a single authentication selector, online STS proof verification and its provenance, application binding, diff --git a/docs/probes/iamproof/README.md b/docs/probes/iamproof/README.md new file mode 100644 index 0000000..e108203 --- /dev/null +++ b/docs/probes/iamproof/README.md @@ -0,0 +1,44 @@ +# IAM proof design probe + +This standalone Go 1.27 module is research for proposed decision 0017, not a +production helper/verifier. It uses synthetic credentials and makes no AWS calls. +Its SDK requirements do not change the parent module's dependencies. Root +`go test ./...` and current CI do not traverse this nested module. + +Run from this directory: + +```sh +go test -v -race ./... +go vet ./... +``` + +The probe checks that AWS SDK v2 signing with an application header yields the +expected signed-header list, that a JSON v2 envelope can reconstruct all signed +URL parameters, and that changing the application header changes the signature. +Query ordering is normalized for comparison: SigV4 sorts parameters, whereas +the SDK appends the signature after signing. No signature is recreated by a +server credential in the reconstruction step. + +For standard commercial, GovCloud and China regional endpoints, all twelve +cases pass. These are SDK endpoint/signing checks, not deployed partition or +STS interoperability certification. No proof validation, freshness enforcement, +HTTP verification, response parsing or public API is implemented here. + +Measured token bytes for eu-west-2 and the audience `orders.production`: + +| Synthetic session token bytes | Compact JSON envelope | Base64url of full signed URL | +| --- | ---: | ---: | +| 0 | 282 | 493 | +| 1,024 | 1,671 | 3,935 | +| 4,096 | 5,767 | 14,175 | +| 6,144 | 8,498 | 21,002 | + +Both columns include a three-byte version prefix, but exclude the HTTP scheme +and field name. Session strings deliberately repeat `+/=a` to exercise URL +escaping. These are synthetic stress cases, not observed AWS token distributions +or a promised upper bound. Actual audiences, credential scopes, and session +tokens change the sizes. The last compact case exceeds the proposed 8 KiB limit. + +Latest stable versions were resolved on 2026-09-16 with `go list -m -json`: +AWS SDK core v1.47.0 and STS v1.51.0. Explicit versions and go.sum make this +experiment reproducible; they are not a policy to avoid future stable upgrades. diff --git a/docs/probes/iamproof/go.mod b/docs/probes/iamproof/go.mod new file mode 100644 index 0000000..6cb2042 --- /dev/null +++ b/docs/probes/iamproof/go.mod @@ -0,0 +1,17 @@ +module github.com/asteroid-computing/go-lambda-edge/docs/probes/iamproof + +go 1.27.0 + +require ( + github.com/aws/aws-sdk-go-v2 v1.47.0 + github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 +) + +require ( + github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3 // indirect + github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 // indirect + github.com/aws/smithy-go v1.28.1 // indirect +) diff --git a/docs/probes/iamproof/go.sum b/docs/probes/iamproof/go.sum new file mode 100644 index 0000000..1c2ebec --- /dev/null +++ b/docs/probes/iamproof/go.sum @@ -0,0 +1,16 @@ +github.com/aws/aws-sdk-go-v2 v1.47.0 h1:0jsHallhJCeaU0Ko48c/3FK1ctOQ7NpzggxriJOQ8MQ= +github.com/aws/aws-sdk-go-v2 v1.47.0/go.mod h1:bttEH6JqnUL8LepvDVfdrds/fZ5bCIxzpe3abyUrhDU= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3 h1:Hp/VgjP0BysR3OgLlR057Vz2LcbbVnoWeJ+3qWiS/fY= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3/go.mod h1:nwGV5qw7F1IZPgxCvA/ph8N2TAuz+BkRG/bXn808qMA= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3 h1:MUaM4f+kj1ZIBPZfUS8cxP1GKXXZtHJjAthy93AN7SM= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3/go.mod h1:6YmVmEVRI5ZZzRjCSsb9SryKH0hAlMRdgA7kG9aDvBU= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 h1:fuSCw4Z2qfRCztMPO3GXJNSiEp6Wee+WOLwrHHUMy9c= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3/go.mod h1:6SxcHheD1pPR5+kWm1wGvjlL/YqUsh267sAfEmN4K7A= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 h1:bAdDl/HkGCcGPoe25ToSHEw23VIxt6CT5fLcg111BKg= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19/go.mod h1:KaUzbLxv4CeSxh6ZCl9B4m7CuFenS8kUEaDs+f/DQr4= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 h1:bON1rJf67TSTDCKg816AAIE4xSTtoo9tl0XRkO72R+I= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3/go.mod h1:c5BBpjJcQXpfeq9iASyVKA3T6vX6B6LEXY4mL/gklDY= +github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 h1:Zpnqa6XtrNzXZnwbdCqHOXpXhMsa01ql/pcRQ1sb4hk= +github.com/aws/aws-sdk-go-v2/service/sts v1.51.0/go.mod h1:/8JRcdTt//hG0Q4BTmGbuOplT7ABe+5rdtqUHqXvYIM= +github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ= +github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= diff --git a/docs/probes/iamproof/proof_test.go b/docs/probes/iamproof/proof_test.go new file mode 100644 index 0000000..ad68894 --- /dev/null +++ b/docs/probes/iamproof/proof_test.go @@ -0,0 +1,129 @@ +// Package iamproofprobe checks proposed protocol mechanics using synthetic +// credentials. It is not an authentication implementation or an AWS acceptance test. +package iamproofprobe + +import ( + "encoding/base64" + "encoding/json/v2" + "net/http" + "net/url" + "strconv" + "strings" + "testing" + "time" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/aws/signer/v4" + "github.com/aws/aws-sdk-go-v2/service/sts" +) + +const emptyHash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + +type envelope struct { + Audience string `json:"audience"` + Credential string `json:"credential"` + Date string `json:"date"` + Signature string `json:"signature"` + SessionToken string `json:"sessionToken,omitempty"` +} + +func TestReconstruction(t *testing.T) { + for _, region := range []string{"eu-west-2", "us-gov-west-1", "cn-north-1"} { + t.Run(region, func(t *testing.T) { + for _, size := range []int{0, 1024, 4096, 6144} { + t.Run(strconv.Itoa(size), func(t *testing.T) { + ctx := t.Context() + endpoint, err := sts.NewDefaultEndpointResolverV2().ResolveEndpoint(ctx, sts.EndpointParameters{ + Region: new(region), UseGlobalEndpoint: new(false), + UseFIPS: new(false), UseDualStack: new(false), + }) + if err != nil { + t.Fatal(err) + } + creds := aws.Credentials{ + AccessKeyID: "AKIAIOSFODNN7EXAMPLE", SecretAccessKey: "synthetic-secret-never-valid-in-AWS", + SessionToken: strings.Repeat("+/=a", size/4), + } + when := time.Date(2026, 9, 16, 12, 0, 0, 0, time.UTC) + req := request(t, endpoint.URI.String(), "orders.production") + signer := v4.NewSigner(func(o *v4.SignerOptions) { o.DisableHeaderHoisting = true }) + signedURL, _, err := signer.PresignHTTP(ctx, creds, req, emptyHash, "sts", region, when) + if err != nil { + t.Fatal(err) + } + u, err := url.Parse(signedURL) + if err != nil { + t.Fatal(err) + } + q := u.Query() + if q.Get("X-Amz-SignedHeaders") != "host;x-edge-iam-audience" { + t.Fatal("unexpected signed headers") + } + wire, err := json.Marshal(envelope{ + Audience: "orders.production", Credential: q.Get("X-Amz-Credential"), + Date: q.Get("X-Amz-Date"), Signature: q.Get("X-Amz-Signature"), + SessionToken: q.Get("X-Amz-Security-Token"), + }) + if err != nil { + t.Fatal(err) + } + token := "v1." + base64.RawURLEncoding.EncodeToString(wire) + decoded, err := base64.RawURLEncoding.DecodeString(strings.TrimPrefix(token, "v1.")) + if err != nil { + t.Fatal(err) + } + var proof envelope + if err := json.Unmarshal(decoded, &proof, json.RejectUnknownMembers(true)); err != nil { + t.Fatal(err) + } + rebuilt := request(t, endpoint.URI.String(), "orders.production") + params := rebuilt.URL.Query() + params.Set("X-Amz-Algorithm", "AWS4-HMAC-SHA256") + params.Set("X-Amz-Credential", proof.Credential) + params.Set("X-Amz-Date", proof.Date) + params.Set("X-Amz-Signature", proof.Signature) + params.Set("X-Amz-SignedHeaders", "host;x-edge-iam-audience") + if proof.SessionToken != "" { + params.Set("X-Amz-Security-Token", proof.SessionToken) + } + rebuilt.URL.RawQuery = params.Encode() + // SigV4 canonicalizes query order; the SDK appends the signature + // after signing, whereas url.Values.Encode sorts all parameters. + u.RawQuery = q.Encode() + if rebuilt.URL.String() != u.String() { + t.Fatal("reconstruction changed the presigned URL") + } + tampered := request(t, endpoint.URI.String(), "orders.staging") + other, _, err := signer.PresignHTTP(ctx, creds, tampered, emptyHash, "sts", region, when) + if err != nil { + t.Fatal(err) + } + otherURL, err := url.Parse(other) + if err != nil { + t.Fatal(err) + } + if otherURL.Query().Get("X-Amz-Signature") == proof.Signature { + t.Fatal("audience did not affect signature") + } + t.Logf("session bytes=%d compact token bytes=%d URL token bytes=%d host=%s", + size, len(token), 3+base64.RawURLEncoding.EncodedLen(len(signedURL)), u.Host) + }) + } + }) + } +} + +func request(t *testing.T, endpoint, audience string) *http.Request { + t.Helper() + req, err := http.NewRequest(http.MethodGet, endpoint, nil) + if err != nil { + t.Fatal(err) + } + req.Header.Set("X-Edge-IAM-Audience", audience) + q := req.URL.Query() + q.Set("Action", "GetCallerIdentity") + q.Set("Version", "2011-06-15") + q.Set("X-Amz-Expires", "60") + req.URL.RawQuery = q.Encode() + return req +} From 1ef82e56bbcd5356da35557eca9b612ef486e3cc Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 16:16:24 +0100 Subject: [PATCH 26/44] feat: add immutable callers and context isolation --- README.md | 20 +- docs/decisions/0012-shared-identity.md | 20 +- docs/decisions/0014-claims-api.md | 13 +- docs/decisions/0015-iam-caller-forms.md | 27 +- docs/decisions/0017-iam-proof-protocol.md | 11 +- .../0018-gateway-identity-recognition.md | 147 ++++++++++ docs/decisions/README.md | 7 +- docs/plan.md | 36 +-- identity/caller.go | 75 +++++ identity/caller_example_test.go | 49 ++++ identity/caller_test.go | 269 ++++++++++++++++++ identity/claims.go | 5 +- identity/context.go | 29 ++ identity/errors.go | 4 + identity/iam.go | 211 ++++++++++++++ identity/jwt.go | 234 +++++++++++++++ invocation.go | 5 + invocation_test.go | 30 ++ 18 files changed, 1149 insertions(+), 43 deletions(-) create mode 100644 docs/decisions/0018-gateway-identity-recognition.md create mode 100644 identity/caller.go create mode 100644 identity/caller_example_test.go create mode 100644 identity/caller_test.go create mode 100644 identity/context.go create mode 100644 identity/iam.go create mode 100644 identity/jwt.go diff --git a/README.md b/README.md index bceed06..f9af370 100644 --- a/README.md +++ b/README.md @@ -22,16 +22,17 @@ public categories support `errors.Is`, and `errors.AsType[*edge.InvocationError] exposes sanitized operation and resource-limit diagnostics. HTTP error responses remain successful transport outcomes; operation, cleanup, cancellation, and late stream failures retain their accepted ownership and precedence. The shared -claims API and resource limits are implemented in `identity`. Caller/context -construction, JWT normalization and gateway producers remain pending; the initial -IAM caller forms are recorded separately for review. +claims API, caller/context construction, JWT normalization and resource limits +are implemented in `identity`. The private invocation boundary rejects inherited +callers. Native gateway extraction remains pending its fixture/recognition review. The accepted authentication direction supports IAM credentials or an OAuth bearer token on the same route without built-in `AWS_IAM` authorization. IAM clients will use an exported Go helper to generate a signed STS GetCallerIdentity proof; the server verifier submits that client-signed request to AWS and uses the verified identity for application authorization. The helper, verifier and their -client/server examples are planned; the detailed protocol/API is not implemented. +client/server examples are planned. The detailed protocol/API is accepted in +[decision 0017](docs/decisions/0017-iam-proof-protocol.md), but not implemented. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle @@ -73,6 +74,17 @@ it at commitment. Public invocation and the streaming HTTP writer remain pending ## Owned claims +`identity.NewJWT` and `identity.NewIAM` construct immutable, mutually exclusive +caller views. Zero Caller is anonymous. Constructors validate facts and source +attribution; they do not authenticate credentials or grant permissions. +JWT issuer, subject, client ID, audience, scopes and Cognito groups stay distinct. +Ambiguous gateway collections remain unavailable, while conflicting interpretable +scope sets fail. IAM callers retain exact ARN paths and session names. + +`identity.WithCaller` installs one caller in a derived context and rejects +replacement, even by an apparently identical caller. `identity.FromContext` +returns anonymous when absent. See the [caller examples](identity/caller_example_test.go). + `identity.ParseClaims`, `identity.NewClaims`, and `identity.NewTextClaims` accept raw JSON, decoded Go values, and gateway string maps respectively. Claims are immutable snapshots with checked accessors. Exact numeric text remains available diff --git a/docs/decisions/0012-shared-identity.md b/docs/decisions/0012-shared-identity.md index 2d60d03..05ca65c 100644 --- a/docs/decisions/0012-shared-identity.md +++ b/docs/decisions/0012-shared-identity.md @@ -1,7 +1,8 @@ # 0012: Shared identity, provenance, and producer composition Status: accepted as the starting contract on 2026-09-15; claims foundation -implemented under 0014, caller/context and gateway producers pending. +implemented under 0014; caller/context and JWT normalization implemented on +2026-09-16. Native gateway producers remain pending their fixture review. This walks the dependencies needed for the first callable buffered adapter. Decision 0013 covers the public errors used at those boundaries. Approval of the @@ -357,13 +358,20 @@ This accepts I1-I8 and the implementation order. The explicitly deferred claim accessor inventory, claim resource limits, producer fixture coverage, and later authn/custom-mapper policies still require their recorded follow-up review. The claims constructors, immutable views and resource/error foundation are now -implemented under accepted decision 0014. Caller/context construction and native -gateway producers remain pending. Decision 0015 presents the supported IAM caller -forms left open by I5 for review before the complete caller union is published. +implemented under accepted decision 0014. Caller/context construction, JWT +normalization and dedicated-scope accounting are now implemented. The user +approved IAM caller forms in 0015/0017 and the literal user-path refinement. +Native gateway producers remain pending the fixture/recognition review in 0018. Accepted follow-up 0016 adds an online STS-verified IAM proof producer alongside -the gateway/custom producers described in I2. Its provenance representation will -be settled in the protocol/API review. For I8, one explicitly configured selector +the gateway/custom producers described in I2. Accepted decision 0017 adds +SourceVerifiedIAMProof, restricted to IAM callers. For I8, one explicitly configured selector may choose IAM proof or OAuth verification per request on the same route. It still installs only one caller and never retries another credential mechanism after authentication failure or merges identities. + +The private shared invocation boundary now rejects inherited nonanonymous +callers with ErrIdentity and identity.ErrConflict before running request work. +Tests cover source/kind restrictions, exact IAM facts, scope consistency and +budgets, collection ownership, context collisions and sanitized diagnostics. +Full race tests, vet and Linux arm64/amd64 builds pass on Go 1.27.1. diff --git a/docs/decisions/0014-claims-api.md b/docs/decisions/0014-claims-api.md index 9be9ae9..8060190 100644 --- a/docs/decisions/0014-claims-api.md +++ b/docs/decisions/0014-claims-api.md @@ -1,8 +1,9 @@ # 0014: Owned claims, numeric fidelity, and resource limits Status: accepted on 2026-09-16; claims constructors/accessors and error foundation -implemented. JWT normalization and dedicated-scope accounting await caller -construction. This completes the claims design follow-up left open by 0012. +implemented. JWT normalization and dedicated-scope accounting were implemented +with caller construction on 2026-09-16. This completes the claims design follow-up +left open by 0012. ## Evidence and scope @@ -252,6 +253,8 @@ limits without misreporting depth as bytes. Neither is wired to invocation yet. Production allocation measurements are recorded in docs/benchmarks.md. Race tests, vet, JSON fuzzing and both Lambda-target builds validate the claims foundation. -Caller construction, dedicated scope accounting and JWT normalization are still -pending. Decision 0015 surfaces the explicitly deferred IAM caller-form review -before publishing the complete caller union; no placeholder constructor exists. +Caller construction, dedicated scope accounting and JWT normalization are now +implemented following approval of 0015/0017. WithGatewayScopes owns its supplied +collection, charges the remaining claims allowance and rejects disagreement. +Raw gateway string collections remain unavailable, even when the containing +envelope preserved JSON syntax. Race tests cover the completed foundation. diff --git a/docs/decisions/0015-iam-caller-forms.md b/docs/decisions/0015-iam-caller-forms.md index 3247dec..4c1c0d7 100644 --- a/docs/decisions/0015-iam-caller-forms.md +++ b/docs/decisions/0015-iam-caller-forms.md @@ -1,6 +1,7 @@ # 0015: Initial IAM caller forms -Status: proposed on 2026-09-16; awaiting review. +Status: accepted on 2026-09-16, including the four initial caller forms, with +approval of decision 0017. Constructor/view implementation is complete. Decision 0012 requires validation of an actual caller principal form and explicitly defers the supported-form fixtures. Implementing NewIAM now reaches that gate. @@ -30,7 +31,7 @@ Initially accept these exact forms in NewIAM: - STS federated-user session: arn::sts:::federated-user/. Preserve spelling, partition, account and full resource including session. Reject -bare account IDs, IAM role ARNs, wildcards and policy-only principal forms in this +bare account IDs, IAM role ARNs, wildcard expressions outside literal user paths and policy-only principal forms in this initial request-caller constructor. Do not transform a role session into a role. Recognizing root is a representation capability, not permission to perform an application action. Root policy-principal delegation semantics are not applied. @@ -100,3 +101,25 @@ locally verified JWT authentication. No such producer is approved or implemented The user's subsequent same-route IAM/OAuth clarification is recorded in 0016. For that goal, the proposed path is explicit mixed-credential middleware with an online IAM proof verifier. Built-in AWS_IAM alone does not satisfy the goal. + +## Accepted lexical refinement and implementation + +On 2026-09-16 the user approved preserving literal `*` and `?` in valid IAM user +paths. [CreateUser](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateUser.html) +permits these characters in its path grammar. Caller identifiers are exact facts, +never policy patterns. Wildcard account IDs, usernames, role names and session +names remain invalid. Follow the formal path pattern's U+0021..U+007E range; +the page's prose mentions DEL but its formal pattern excludes it. + +The constructor follows documented name/path lengths: IAM username 1..64 and +path 1..512; role name 1..64; session name 2..64 under +[AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html); +federated username 2..32 under +[GetFederationToken](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetFederationToken.html). +The account is exactly twelve ASCII digits and the ARN's region is empty. +Partition validation is lexical, not an assertion that an AWS partition exists; +the authenticating producer must enforce its configured partition. + +NewIAM and its read-only view are implemented with account consistency, +duplicate-option rejection, exact session preservation and sanitized diagnostics. +Online verification is not implemented by this constructor. diff --git a/docs/decisions/0017-iam-proof-protocol.md b/docs/decisions/0017-iam-proof-protocol.md index 2592070..715b8e6 100644 --- a/docs/decisions/0017-iam-proof-protocol.md +++ b/docs/decisions/0017-iam-proof-protocol.md @@ -1,8 +1,8 @@ # 0017: IAM proof protocol and Go helper -Status: proposed on 2026-09-16; awaiting review. Decision 0016 accepts the -architecture and exported helper, not the protocol/API details below. Only an -isolated SDK probe is implemented; there is no production IAM authentication yet. +Status: accepted on 2026-09-16. The user approved P1-P5 and the four caller forms +in decision 0015. The shared caller foundation and an isolated SDK probe are +implemented; there is no production IAM proof generation or authentication yet. ## Evidence @@ -284,4 +284,7 @@ authorized integration check can later validate deployed interoperability. ## Resolution -Pending user review. No production API or trust policy is changed by this record. +The user approved this proposal and decision 0015 on 2026-09-16 and authorized +implementation in the recorded order. Complete the shared caller/context and +buffered adapter foundation before the IAM helper/verifier and authn middleware. +The isolated probe remains the only implemented IAM-proof artifact at approval. diff --git a/docs/decisions/0018-gateway-identity-recognition.md b/docs/decisions/0018-gateway-identity-recognition.md new file mode 100644 index 0000000..8cc4c54 --- /dev/null +++ b/docs/decisions/0018-gateway-identity-recognition.md @@ -0,0 +1,147 @@ +# 0018: Native gateway identity recognition and fixture coverage + +Status: proposed on 2026-09-16. The caller/context foundation is implemented; +native gateway extraction and public invocation wiring await this review. +This is the producer-fixture follow-up explicitly reserved in 0012 and 0014. + +## Evidence and existing boundary + +Official AWS documentation was checked through public AWS MCP. Current decode.go +retains raw authorizer JSON separately from the SDK structs. V1 typed events have +an authorizer map; V2 typed events have JWT, IAM and Lambda authorizer members. +The root option still defaults gateway identity off. NewIAM/NewJWT now validate +owned facts, and the shared invocation guard rejects an inherited caller. + +- [REST proxy integration](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) + documents IAM identity fields, Cognito authorizer.claims, and custom-authorizer + principalId/context. Its example includes null claims/scopes placeholders. +- [HTTP proxy payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) + documents both payload versions and authorizer.jwt.claims/scopes in 2.0. The + 1.0 example has null placeholders, not a complete authenticated producer matrix. +- [Context variable definitions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html) + distinguish API-owner accountId from identity.accountId; identity.caller is the + signing principal, identity.user the principal authorized against resources, + and identity.userArn the effective authenticated user's ARN. +- [AWS Lambda Go V2 definitions](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/apigw.go) + and its [IAM fixture](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/testdata/apigw-v2-request-iam.json) + establish the iam member's userArn, accountId, userId and callerId field names. + That fixture uses a ten-digit account ID: it is serialization/schema evidence, + not a valid identity fixture or proof that two principal IDs are equivalent. + +Gateway assertions are trusted because the application opts into its intended +integration and controls Lambda invocation permissions. Event data cannot prove +its own provenance or reveal every deployment setting. This proposal does not +turn source tags, fields or an Authorization header into cryptographic evidence. + +## G1. Recognize shapes without guessing the API product + +Recommend these mappings; all paths are under requestContext: + +| Payload shape | IAM caller | JWT caller | +| --- | --- | --- | +| V1 / 1.0 | identity.userArn; optional identity.accountId and identity.user | authorizer.claims plus optional authorizer.scopes | +| V2 / 2.0 | authorizer.iam.userArn; optional iam.accountId and iam.userId | authorizer.jwt.claims plus optional jwt.scopes | + +Use the selected shape consistently for both raw and typed inputs. Do not infer +REST versus HTTP API from payload version, invent another HTTP 1.0 claim encoding, +or probe arbitrary nested claim maps until something looks like a caller. + +For V1 native claims, source documentation directly establishes the REST Cognito +path. For HTTP 1.0, the structure is documented but the authenticated matrix is +not fully established. Recommend implementing the same explicit shape contract +with synthetic fixtures and documenting that qualification. Do not advertise +deployed coverage of every HTTP 1.0 authorizer combination. An unexpected nonempty +authorizer fails as unsupported; it must not become anonymous. + +Alternative: restrict the whole V1 transport to REST, or infer identity from token +headers when the assertion differs. The former discards an accepted transport +surface and the latter changes the trust boundary. Neither is recommended. + +## G2. Effective user fields and partial IAM assertions + +Require the exact ARN for IAM construction. Cross-check a nonempty account field +from the identity/iam object; never pass the API-owner accountId as caller account. +Preserve V1 user or V2 userId as the optional opaque principal ID. Do not substitute +caller/callerId or require equality with it: the documented roles differ. +If user/userId is missing, leave PrincipalID unavailable rather than inventing it. + +For V1, nonempty userArn, user or caller signals an IAM candidate. A candidate with +missing/invalid userArn fails with ErrIdentity and identity.ErrInvalidCaller. +Account/access-key/Cognito-pool/network/API-key/mTLS metadata alone does not +establish IAM identity. It remains metadata; authorization sees anonymous. +For V2, a nonnull iam object is a candidate even when empty. Missing ARN fails. +Validate consumed scalar fields as strings or null; reject wrong types without +printing their contents. Raw empty optional strings match the SDK's empty values. + +This is a field-mapping policy, not a claim that a valid ARN authenticates itself. +The four accepted NewIAM forms and their literal path semantics apply unchanged. + +## G3. Absence, empty assertions and collisions + +Recommend the following precise presence policy: + +| Input with gateway identity enabled | Outcome | +| --- | --- | +| Missing/null authorizer, or empty untagged object, with no IAM candidate | Anonymous | +| V1 only claims/scopes missing or null | Anonymous; documented placeholder | +| V1 nonnull claims or scopes | JWT candidate; missing/empty required identity facts fail | +| V2 nonnull jwt object, including an empty object or null claims/scopes inside it | JWT candidate; missing identity facts fail | +| V2 nonnull iam object, including an empty object | IAM candidate; missing identity facts fail | +| Nonnull V1 principalId or V2 lambda member, including an empty custom context | Custom candidate; unsupported until a mapper is reviewed | +| Multiple native/custom candidates | ErrIdentity with identity.ErrConflict | +| Other nonempty unrecognized authorizer | ErrIdentity with errors.ErrUnsupported | + +The V2 rule clarifies 0012's null-placeholder language: a nonnull tagged producer +object expresses more intent than the documented V1 placeholder. Treat an empty +tagged assertion as broken integration data, rather than silently unauthenticated. +Null V2 members are absent. Unknown sibling metadata alongside a supported native +candidate is ignored for identity, as accepted in 0012; explicit native/custom +producer markers still collide. Inspect all markers before selecting a producer. + +Alternative: interpret every empty object as anonymous. That can conceal a +configured producer that failed to supply its required facts. Recommend failing +closed for explicit tagged producers while keeping the documented anonymous +events valid. + +With gateway identity disabled, none of these fields produces a caller and no +identity interpretation runs. Structural event JSON/type checks still apply as +already specified; opting out does not make malformed event JSON valid. + +## G4. Preserve fidelity and make test coverage honest + +Raw claims use ParseClaims on the retained claim-object bytes; V1 typed claims use +NewClaims directly; V2 typed claims use NewTextClaims. Do not serialize typed events +to recover unavailable presence or numeric information. Normalize all of them +with SourceGatewayAssertion and the accepted C5 collection qualifications. +Dedicated scopes use WithGatewayScopes and the configured remaining claims budget. +Claim budget exhaustion maps to ErrIdentity/ErrLimitExceeded and identity_claims. + +SDK typed decoding may already discard unknown authorizer members and collapse +null/absent scalar fields. Preserve raw/typed equivalence where the SDK retained +the distinction, and explicitly test/document the exceptions. Prefer the raw +entry point when full wire-shape inspection is required. Do not manufacture an +unknown custom producer from a zero SDK description after its fields were lost. + +Prepare independently authored fixtures labeled by evidence level: + +- Documentation-shaped anonymous V1/HTTP 1.0 placeholders and V2 JWT objects; + replace documentation placeholder claims with valid synthetic iss/sub/client_id. +- SDK-shaped V2 IAM with valid twelve-digit caller account, explicit cross-account + API owner, all four principal forms, and distinct userId/callerId values. +- REST Cognito/IAM cases using documented field meanings; label constructed + values synthetic, never as captured live traffic. +- HTTP 1.0 native cases labeled shape-contract tests, with deployed coverage + unverified; include an unexpected-shape rejection case. +- Adversarial collisions, partial/empty assertions, custom/unknown producers, + invalid types, claim/scope budget boundaries, flattened permissions and inherited + caller rejection, in both gateway modes and raw/typed paths where representable. + +The fixture suite must prove no handler runs after identity failure and no +identity is derived from a bearer header. Test actual SDK registration only after +the shared producers are reviewed and wired. No live deployment is required by +default; a future separately authorized capture can improve the coverage matrix. + +## Resolution + +Pending review of G1-G4. Existing identity constructors and invocation isolation +are complete and tested; this record does not implement gateway extraction. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index eccdeff..1726677 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -20,9 +20,10 @@ Keep related decisions small enough to review and implement independently. | [0009: Streaming boundaries](0009-streaming-boundaries.md) | Accepted | What API, lifecycle, handoff, and error rules should streaming use? | | [0010: Response header design](0010-response-header-design.md) | Accepted | How should shared snapshots, V2 combination, resource accounting, and streaming prefix limits work? | | [0011: Request header processing](0011-request-header-processing.md) | Accepted | How can consumers process custom headers and safely select actions for their own dispatchers? | -| [0012: Shared identity](0012-shared-identity.md) | Accepted; claims settled by 0014 | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | +| [0012: Shared identity](0012-shared-identity.md) | Accepted; caller/context foundation implemented | What caller, claim fidelity, provenance, and producer-composition contracts should be shared? | | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | | [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | -| [0015: IAM caller forms](0015-iam-caller-forms.md) | Proposed | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | +| [0015: IAM caller forms](0015-iam-caller-forms.md) | Accepted | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | | [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture and exported Go helper accepted; protocol/API pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | -| [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Proposed; isolated SDK probe passes | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | +| [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; isolated SDK probe passes | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | +| [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Proposed | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | diff --git a/docs/plan.md b/docs/plan.md index a2aa2eb..25b9c3c 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -69,7 +69,8 @@ multiple identity producers remain separate decisions. | --- | --- | | `edge` (root) | Invocation adapter, payload decoding, HTTP translation, gateway metadata, optional gateway identity extraction | | `identity` | Validated callers, claim representation, context transport; standard library only | -| `authn` | Explicit IAM-proof/OAuth selection, online STS verification, Cognito verification and JWKS caching | +| `authn` | Explicit IAM-proof/OAuth selection, Cognito verification and JWKS caching | +| `iamproof` | Exported client proof generator and bounded online STS verifier | | `authz` | Decisions, rules, combinators, middleware, application-principal resolution | | `edgetest` | Consumer identity fixtures and gateway event helpers | @@ -78,13 +79,13 @@ event types out of transport-independent packages. Reassess any additional exported package against a concrete consumer need; version-reporting machinery does not need to precede a working adapter. -The accepted IAM client helper belongs to the module's exported Go API. Its -package placement and credential-provider dependency are part of the detailed -0016 protocol review; do not assume a new package or root reexport prematurely. -[Decision 0017](decisions/0017-iam-proof-protocol.md) proposes a focused `iamproof` +The accepted IAM client helper belongs to the module's exported Go API. +[Decision 0017](decisions/0017-iam-proof-protocol.md), accepted on 2026-09-16, +specifies a focused `iamproof` package, compact versioned proof, signed audience, short freshness policy and -bounded regional STS verification. Its isolated SDK probe passes; all production -API and trust-boundary choices in that record remain pending review. +bounded regional STS verification. Its isolated SDK probe passes. The user also +approved the IAM caller forms in decision 0015. The shared caller/context +foundation is now implemented; gateway producers precede local authentication. ## Milestones and acceptance criteria @@ -252,10 +253,12 @@ implemented. The adapter budget option is validated and a private invocation-err translation boundary is tested; gateway extraction and invocation remain unwired. Production allocation measurements supplement the original probe. Contract tests, SDK integration, race detection, vet, JSON fuzzing and Lambda-target builds pass. -Caller/context construction and JWT normalization remain pending; proposed -[decision 0015](decisions/0015-iam-caller-forms.md) resolves the supported IAM forms -explicitly deferred by 0012 before publishing the complete caller union. -Native-producer fixture review remains separate from the claims approval. +Caller/context construction, JWT normalization and dedicated-scope accounting +are now implemented after approval of [decision 0015](decisions/0015-iam-caller-forms.md) +and 0017. IAM literal path punctuation follows the user's approved refinement. +The private invocation guard rejects inherited callers. Full race tests, vet, +formatting and Linux arm64/amd64 builds pass. Native-producer fixture review +remains separate from the foundation approval; see proposed decision 0018. - Review caller constructors, immutable/owned data, validation, anonymous state, and context-presence semantics. @@ -273,14 +276,13 @@ claims cannot be silently mixed; unavailable claim fidelity remains explicit. ### 4. Local authentication: Cognito/JWKS and IAM credentials -- Review the concrete protocol/helper/verifier proposal in decision 0017, - including replay, size and STS quota tradeoffs, before dependent implementation. +- Implement the accepted protocol/helper/verifier contract in decision 0017 + after the shared foundation and callable buffered adapter. The standalone probe uses latest stable SDK core v1.47.0 and STS v1.51.0 as resolved on 2026-09-16; root production dependencies have not changed. -- Implement the accepted client-helper/server-verifier direction in decision 0016 - after its detailed protocol review. Review a single authentication - selector, online STS proof verification and its provenance, application binding, - freshness/replay rules, endpoint constraints and dependency-error semantics. +- Implement the accepted client-helper/server-verifier direction in decisions + 0016/0017. Review the common HTTP authentication selector API and challenges; + preserve the accepted proof, provenance, binding, replay, endpoint and error rules. Do not promise offline IAM verification or ordinary SigV4-client compatibility. - Provide exported Go proof generation and consumer documentation with runnable examples showing IAM and OAuth clients calling the same protected route. diff --git a/identity/caller.go b/identity/caller.go new file mode 100644 index 0000000..2792263 --- /dev/null +++ b/identity/caller.go @@ -0,0 +1,75 @@ +package identity + +import "strconv" + +// Kind identifies the caller's mutually exclusive identity representation. +type Kind uint8 + +// Caller kinds include an anonymous zero value. +const ( + KindAnonymous Kind = iota + KindJWT + KindIAM +) + +// Source records the producer's assertion, not cryptographic evidence. Trusted +// application code must authenticate inputs before attributing a source. +type Source uint8 + +// Sources distinguish gateway assertions, verified tokens, custom mappings and +// IAM proofs verified online by STS. None is reserved for anonymous callers. +const ( + SourceNone Source = iota + SourceGatewayAssertion + SourceLocallyVerifiedToken + SourceCustomAssertion + SourceVerifiedIAMProof +) + +// Caller is an immutable identity snapshot with exactly one JWT or IAM view. +// Its zero value is anonymous. Constructors validate facts; they neither +// authenticate nor authorize. Copies may be shared concurrently. There is no +// identity equality contract or implicit JSON export. Diagnostics omit facts. +type Caller struct { + source Source + jwt *JWT + iam *IAM +} + +// Kind returns the identity representation, or KindAnonymous for the zero value. +func (c Caller) Kind() Kind { + if c.jwt != nil { + return KindJWT + } + if c.iam != nil { + return KindIAM + } + return KindAnonymous +} + +// Source returns the attributed producer, or SourceNone for an anonymous caller. +func (c Caller) Source() Source { return c.source } + +// JWT reports whether the caller has a JWT identity and returns an immutable view. +func (c Caller) JWT() (JWT, bool) { + if c.jwt == nil { + return JWT{}, false + } + return *c.jwt, true +} + +// IAM reports whether the caller has an IAM identity and returns an immutable view. +func (c Caller) IAM() (IAM, bool) { + if c.iam == nil { + return IAM{}, false + } + return *c.iam, true +} + +// String describes kind and source without exposing identifiers or claims. +func (c Caller) String() string { + return "identity.Caller{kind:" + strconv.Itoa(int(c.Kind())) + ",source:" + strconv.Itoa(int(c.source)) + "}" +} + +// GoString returns the same sanitized description as String. +func (c Caller) GoString() string { return c.String() } diff --git a/identity/caller_example_test.go b/identity/caller_example_test.go new file mode 100644 index 0000000..70df531 --- /dev/null +++ b/identity/caller_example_test.go @@ -0,0 +1,49 @@ +package identity_test + +import ( + "context" + "errors" + "fmt" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func ExampleNewJWT() { + // Synthetic facts for this example. In an application, an authenticating + // producer must establish these facts before constructing the caller. + claims, err := identity.NewClaims(map[string]any{ + "iss": "https://issuer.example", "client_id": "batch-worker", "scope": "orders.read", + }) + if err != nil { + panic(err) + } + caller, err := identity.NewJWT(claims, identity.SourceCustomAssertion) + if err != nil { + panic(err) + } + jwt, _ := caller.JWT() + client, _ := jwt.ClientID() + _, hasSubject := jwt.Subject() + scopes, known := jwt.Scopes() + fmt.Println(client, hasSubject, scopes, known) + // Output: batch-worker false [orders.read] true +} + +func ExampleWithCaller() { + // This validates a synthetic identifier; it does not authenticate it. + caller, err := identity.NewIAM("arn:aws:sts::123456789012:assumed-role/Worker/batch-1", identity.SourceCustomAssertion) + if err != nil { + panic(err) + } + ctx, err := identity.WithCaller(context.Background(), caller) + if err != nil { + panic(err) + } + iam, _ := identity.FromContext(ctx).IAM() + fmt.Println(iam.PrincipalARN()) + _, err = identity.WithCaller(ctx, caller) + fmt.Println(errors.Is(err, identity.ErrConflict)) + // Output: + // arn:aws:sts::123456789012:assumed-role/Worker/batch-1 + // true +} diff --git a/identity/caller_test.go b/identity/caller_test.go new file mode 100644 index 0000000..76d0294 --- /dev/null +++ b/identity/caller_test.go @@ -0,0 +1,269 @@ +package identity_test + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "slices" + "strings" + "sync" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func TestIAMCallerForms(t *testing.T) { + for _, tc := range []struct { + name, arn, partition, service string + form identity.IAMPrincipalType + }{ + {name: "root", arn: "arn:aws:iam::123456789012:root", partition: "aws", service: "iam", form: identity.IAMPrincipalRoot}, + {name: "user", arn: "arn:aws:iam::123456789012:user/engineering/Alice", partition: "aws", service: "iam", form: identity.IAMPrincipalUser}, + {name: "literal_path", arn: "arn:aws:iam::123456789012:user/team*?:x/%2F/Alice", partition: "aws", service: "iam", form: identity.IAMPrincipalUser}, + {name: "gov_session", arn: "arn:aws-us-gov:sts::123456789012:assumed-role/Operator/alice@example.com", partition: "aws-us-gov", service: "sts", form: identity.IAMPrincipalAssumedRole}, + {name: "china_federation", arn: "arn:aws-cn:sts::123456789012:federated-user/Bob", partition: "aws-cn", service: "sts", form: identity.IAMPrincipalFederatedUser}, + } { + t.Run(tc.name, func(t *testing.T) { + for _, source := range []identity.Source{identity.SourceGatewayAssertion, identity.SourceCustomAssertion, identity.SourceVerifiedIAMProof} { + c, err := identity.NewIAM(tc.arn, source, identity.WithIAMAccountID("123456789012"), identity.WithIAMPrincipalID("opaque:session")) + if err != nil { + t.Fatal(err) + } + i, ok := c.IAM() + if !ok || c.Kind() != identity.KindIAM || c.Source() != source { + t.Fatalf("IAM caller = %v, IAM present = %t", c, ok) + } + if _, ok := c.JWT(); ok { + t.Error("IAM caller also contains JWT") + } + if i.PrincipalARN() != tc.arn || i.Partition() != tc.partition || i.Service() != tc.service || i.AccountID() != "123456789012" || i.PrincipalType() != tc.form { + t.Error("IAM accessors did not preserve the supplied caller facts") + } + if id, ok := i.PrincipalID(); !ok || id != "opaque:session" { + t.Errorf("PrincipalID() = %q, %t", id, ok) + } + } + }) + } +} + +func TestIAMRejectsInvalidCallers(t *testing.T) { + for _, resource := range []string{ + "role/Operator", "user/*", "user/A?ice", "user/", "user//Alice", + "user/team name/Alice", "user/team\x7f/Alice", "user/Alice/", "user/" + strings.Repeat("a", 65), + "user/" + strings.Repeat("a", 511) + "/Alice", + "assumed-role/Operator/a", "assumed-role/Operator/session/extra", "assumed-role/*/session", + "assumed-role/Operator/" + strings.Repeat("x", 65), "federated-user/x", "federated-user/" + strings.Repeat("x", 33), + } { + for _, service := range []string{"iam", "sts"} { + c, err := identity.NewIAM("arn:aws:"+service+"::123456789012:"+resource, identity.SourceGatewayAssertion) + if !errors.Is(err, identity.ErrInvalidCaller) || c.Kind() != identity.KindAnonymous { + t.Errorf("NewIAM(%q, %q) = %v, %v; want anonymous/invalid", service, resource, c, err) + } + } + } + for _, arn := range []string{"", "123456789012", "arn:aws:iam:eu-west-2:123456789012:root", "arn:AWS:iam::123456789012:root", "arn:aws:iam::*:root", "arn:aws:iam::12345678901x:root", "arn:aws:iam::123456789012:root:extra"} { + if _, err := identity.NewIAM(arn, identity.SourceGatewayAssertion); !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("NewIAM(%q) = %v, want invalid", arn, err) + } + } + for _, opts := range [][]identity.IAMOption{ + {nil}, {identity.WithIAMAccountID("000000000000")}, {identity.WithIAMPrincipalID("")}, + {identity.WithIAMPrincipalID("\xff")}, {identity.WithIAMPrincipalID("id"), identity.WithIAMPrincipalID("id")}, + {identity.WithIAMAccountID("123456789012"), identity.WithIAMAccountID("123456789012")}, + } { + if _, err := identity.NewIAM("arn:aws:iam::123456789012:root", identity.SourceGatewayAssertion, opts...); !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("NewIAM with invalid/duplicate options = %v", err) + } + } +} + +func TestJWTFactsAndOwnership(t *testing.T) { + claims := parseClaims(t, `{"iss":"https://issuer/Exact","sub":"subject","client_id":"client","token_use":"access","aud":["b","a","b"],"scope":"write read read","scp":["read","write"],"cognito:groups":["Ops","Ops"],"custom":{"x":1}}`) + caller, err := identity.NewJWT(claims, identity.SourceLocallyVerifiedToken) + if err != nil { + t.Fatal(err) + } + j, ok := caller.JWT() + if !ok || caller.Kind() != identity.KindJWT || caller.Source() != identity.SourceLocallyVerifiedToken { + t.Fatalf("NewJWT = %v, JWT present=%t", caller, ok) + } + if _, ok := caller.IAM(); ok { + t.Error("JWT caller also contains IAM") + } + if j.Issuer() != "https://issuer/Exact" { + t.Errorf("Issuer() = %q", j.Issuer()) + } + for name, read := range map[string]func() (string, bool){"subject": j.Subject, "client": j.ClientID, "access": j.TokenUse} { + if value, ok := read(); !ok || value != name { + t.Errorf("scalar accessor = %q, %t; want %q, true", value, ok, name) + } + } + for _, tc := range []struct { + read func() ([]string, bool) + want []string + }{{j.Audience, []string{"a", "b"}}, {j.Scopes, []string{"read", "write"}}, {j.CognitoGroups, []string{"Ops"}}} { + values, known := tc.read() + if !known || !slices.Equal(values, tc.want) { + t.Errorf("collection = %v, %t; want %v, true", values, known, tc.want) + } + values[0] = "mutated" + again, _ := tc.read() + if !slices.Equal(again, tc.want) { + t.Error("returned slice mutated identity") + } + } + if j.Claims().Len() != claims.Len() { + t.Error("normalized view lost original claims") + } +} + +func TestJWTInvalidAndUnavailableFacts(t *testing.T) { + for _, input := range []string{ + `{}`, `{"iss":"issuer"}`, `{"iss":1,"sub":"x"}`, `{"iss":"issuer","sub":""}`, + `{"iss":"issuer","sub":"x","client_id":null}`, `{"iss":"issuer","sub":"x","token_use":false}`, + `{"iss":"issuer","sub":"x","aud":[1]}`, `{"iss":"issuer","sub":"x","cognito:groups":"[Ops]"}`, + `{"iss":"issuer","sub":"x","scope":"read write"}`, `{"iss":"issuer","sub":"x","scope":"read\twrite"}`, + `{"iss":"issuer","sub":"x","scope":"read write"}`, `{"iss":"issuer","sub":"x","scope":""}`, + `{"iss":"issuer","sub":"x","scope":"read","scp":["write"]}`, `{"iss":"issuer","sub":"x","scp":"read"}`, + } { + c, err := identity.NewJWT(parseClaims(t, input), identity.SourceLocallyVerifiedToken) + if !errors.Is(err, identity.ErrInvalidCaller) || c.Kind() != identity.KindAnonymous { + t.Errorf("NewJWT(%s) = %v, %v; want anonymous/invalid", input, c, err) + } + } + for _, source := range []identity.Source{identity.SourceNone, identity.SourceVerifiedIAMProof, 255} { + if _, err := identity.NewJWT(parseClaims(t, `{"iss":"issuer","sub":"x"}`), source); !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("NewJWT source %d = %v", source, err) + } + } + for _, source := range []identity.Source{identity.SourceNone, identity.SourceLocallyVerifiedToken, 255} { + if _, err := identity.NewIAM("arn:aws:iam::123456789012:root", source); !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("NewIAM source %d = %v", source, err) + } + } + // SourceGatewayAssertion matters even when the envelope itself is raw JSON. + for _, claims := range []identity.Claims{ + parseClaims(t, `{"iss":"issuer","client_id":"machine","aud":"client","cognito:groups":"[Ops]","scp":"[read]"}`), + func() identity.Claims { + c, err := identity.NewTextClaims(map[string]string{"iss": "issuer", "client_id": "machine", "aud": "client", "cognito:groups": "[Ops]", "scp": "[read]"}) + if err != nil { + t.Fatal(err) + } + return c + }(), + } { + c, err := identity.NewJWT(claims, identity.SourceGatewayAssertion) + if err != nil { + t.Fatal(err) + } + j, _ := c.JWT() + for _, read := range []func() ([]string, bool){j.Audience, j.Scopes, j.CognitoGroups} { + if _, known := read(); known { + t.Error("flattened gateway text became a normalized collection") + } + } + if _, present := j.Subject(); present { + t.Error("client-only identity acquired a subject") + } + } +} + +func TestDedicatedScopes(t *testing.T) { + claims := parseClaims(t, `{"iss":"issuer","sub":"subject","scope":"read write"}`) + supplied := []string{"write", "read", "read"} + option := identity.WithGatewayScopes(supplied) + c, err := identity.NewJWT(claims, identity.SourceGatewayAssertion, option) + if err != nil { + t.Fatal(err) + } + supplied[0] = "changed" + j, _ := c.JWT() + if got, known := j.Scopes(); !known || !slices.Equal(got, []string{"read", "write"}) { + t.Errorf("owned Scopes() = %v, %t", got, known) + } + for _, opts := range [][]identity.JWTOption{{nil}, {option, option}, {identity.WithGatewayScopes([]string{"read"})}, {identity.WithGatewayScopes([]string{})}} { + if _, err := identity.NewJWT(claims, identity.SourceGatewayAssertion, opts...); !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("invalid/conflicting scopes = %v", err) + } + } + if _, err := identity.NewJWT(claims, identity.SourceLocallyVerifiedToken, identity.WithGatewayScopes(nil)); !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("gateway scope option on local JWT = %v", err) + } + for _, scopes := range [][]string{nil, {}} { + c, err := identity.NewJWT(parseClaims(t, `{"iss":"issuer","sub":"x"}`), identity.SourceGatewayAssertion, identity.WithGatewayScopes(scopes)) + if err != nil { + t.Fatal(err) + } + j, _ := c.JWT() + if values, known := j.Scopes(); known != (scopes != nil) || len(values) != 0 { + t.Errorf("Scopes() = %v, %t, expected availability %t", values, known, scopes != nil) + } + } + // Root + two scalar values + names/text = 205; array + "read" = 132. + for _, maximum := range []int{336, 337} { + claims, err := identity.NewClaims(map[string]any{"iss": "issuer", "sub": "x"}, identity.WithClaimsBudget(maximum)) + if err != nil { + t.Fatal(err) + } + _, err = identity.NewJWT(claims, identity.SourceGatewayAssertion, identity.WithGatewayScopes([]string{"read"})) + if maximum == 336 { + limit, ok := errors.AsType[*identity.ClaimsLimitError](err) + if !ok || limit.Maximum() != int64(maximum) || !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("scope budget failure = %v, want limit %d and invalid caller", err, maximum) + } + } else if err != nil { + t.Errorf("exact scope budget rejected: %v", err) + } + } +} + +func TestCallerContextAndDiagnostics(t *testing.T) { + caller, err := identity.NewIAM("arn:aws:iam::123456789012:user/SecretName", identity.SourceVerifiedIAMProof) + if err != nil { + t.Fatal(err) + } + type unrelatedKey struct{} + parent := context.WithValue(t.Context(), unrelatedKey{}, "kept") + if got := identity.FromContext(parent); got.Kind() != identity.KindAnonymous || got.Source() != identity.SourceNone { + t.Errorf("empty context = %v", got) + } + unchanged, err := identity.WithCaller(parent, identity.Caller{}) + if err != nil || unchanged != parent { + t.Error("anonymous installation did not preserve empty context") + } + ctx, err := identity.WithCaller(parent, caller) + if err != nil || ctx.Value(unrelatedKey{}) != "kept" || identity.FromContext(ctx).Kind() != identity.KindIAM { + t.Fatalf("WithCaller lost context/caller: %v", err) + } + for _, next := range []identity.Caller{caller, {}} { + if ctx, err := identity.WithCaller(ctx, next); ctx != nil || !errors.Is(err, identity.ErrConflict) { + t.Errorf("reinstallation = %v, %v", ctx, err) + } + } + if ctx, err := identity.WithCaller(nil, caller); ctx != nil || !errors.Is(err, identity.ErrInvalidCaller) { + t.Errorf("nil context = %v, %v", ctx, err) + } + iam, _ := caller.IAM() + for _, value := range []any{caller, iam} { + for _, format := range []string{"%v", "%+v", "%#v"} { + if got := fmt.Sprintf(format, value); strings.Contains(got, "SecretName") || strings.Contains(got, "123456789012") { + t.Errorf("diagnostic leaked identity: %q", got) + } + } + wire, _ := json.Marshal(value) + if strings.Contains(string(wire), "SecretName") { + t.Error("JSON exported identity implicitly") + } + } + var wg sync.WaitGroup + for range 20 { + wg.Go(func() { + if identity.FromContext(ctx).Kind() != identity.KindIAM || identity.FromContext(parent).Kind() != identity.KindAnonymous { + t.Error("shared contexts did not preserve isolated immutable identity") + } + }) + } + wg.Wait() +} diff --git a/identity/claims.go b/identity/claims.go index 92efbc4..c69289e 100644 --- a/identity/claims.go +++ b/identity/claims.go @@ -1,5 +1,6 @@ -// Package identity provides owned claim representations for authenticated callers. -// Constructing claims validates and copies data; it does not authenticate it. +// Package identity provides immutable callers, owned claims and context transport. +// Construction validates representation; producers authenticate, and application +// policy authorizes. The package performs no network or signature verification. package identity import ( diff --git a/identity/context.go b/identity/context.go new file mode 100644 index 0000000..9205f45 --- /dev/null +++ b/identity/context.go @@ -0,0 +1,29 @@ +package identity + +import "context" + +type callerKey struct{} + +// FromContext returns the established caller or the anonymous zero value. +// Like ordinary context operations, it requires a nonnil context. +func FromContext(ctx context.Context) Caller { + caller, _ := ctx.Value(callerKey{}).(Caller) + return caller +} + +// WithCaller installs caller in a derived context. Anonymous on an empty context +// is a no-op. Any attempted installation over a nonanonymous caller, including +// an identical caller or anonymous, returns ErrConflict and a nil context. +// A nil context returns ErrInvalidCaller. Unrelated values are preserved. +func WithCaller(ctx context.Context, caller Caller) (context.Context, error) { + if ctx == nil { + return nil, ErrInvalidCaller + } + if FromContext(ctx).Kind() != KindAnonymous { + return nil, ErrConflict + } + if caller.Kind() == KindAnonymous { + return ctx, nil + } + return context.WithValue(ctx, callerKey{}, caller), nil +} diff --git a/identity/errors.go b/identity/errors.go index 158a21b..24eaa29 100644 --- a/identity/errors.go +++ b/identity/errors.go @@ -3,6 +3,10 @@ package identity import "errors" var ( + // ErrInvalidCaller indicates invalid caller facts, provenance or options. + ErrInvalidCaller = errors.New("identity: invalid caller") + // ErrConflict indicates an attempt to replace an established caller. + ErrConflict = errors.New("identity: caller conflict") // ErrInvalidClaims indicates malformed JSON, unsupported typed values, // invalid UTF-8, nonfinite numbers, or excessive container nesting. ErrInvalidClaims = errors.New("identity: invalid claims") diff --git a/identity/iam.go b/identity/iam.go new file mode 100644 index 0000000..34acdc3 --- /dev/null +++ b/identity/iam.go @@ -0,0 +1,211 @@ +package identity + +import ( + "strings" + "unicode/utf8" +) + +// IAMPrincipalType describes the supported request-caller form, not an IAM +// resource-policy Principal expression. +type IAMPrincipalType uint8 + +// Supported IAM principal forms preserve session identities. +const ( + IAMPrincipalUnknown IAMPrincipalType = iota + IAMPrincipalRoot + IAMPrincipalUser + IAMPrincipalAssumedRole + IAMPrincipalFederatedUser +) + +// IAM is an immutable view of an exact caller ARN and optional principal ID. +// Validation does not establish existence, credential possession or permission. +type IAM struct { + arn string + partition string + account string + service string + principalType IAMPrincipalType + principalID string +} + +// IAMOption supplies an additional caller fact. Nil and duplicate options fail. +type IAMOption func(*iamConfig) error + +type iamConfig struct { + account string + principalID string + accountSet bool + principalIDSet bool +} + +// WithIAMAccountID cross-checks an independently supplied caller account against +// the ARN. Do not pass API Gateway's API-owner requestContext.accountId here. +func WithIAMAccountID(account string) IAMOption { + return func(c *iamConfig) error { + if c.accountSet || !validAccount(account) { + return ErrInvalidCaller + } + c.accountSet, c.account = true, account + return nil + } +} + +// WithIAMPrincipalID preserves a nonempty UTF-8 principal identifier opaquely. +// It is not parsed into an identity or treated as evidence of authentication. +func WithIAMPrincipalID(id string) IAMOption { + return func(c *iamConfig) error { + if c.principalIDSet || id == "" || !utf8.ValidString(id) { + return ErrInvalidCaller + } + c.principalIDSet, c.principalID = true, id + return nil + } +} + +// NewIAM validates and owns an exact root, user, assumed-role session or +// federated-user ARN. Bare role ARNs and policy patterns are not caller forms. +// Literal punctuation in a valid IAM user path is preserved without wildcard +// interpretation. Source must be gateway, custom assertion or verified IAM proof. +// Errors match ErrInvalidCaller and contain no supplied identifiers. +func NewIAM(principalARN string, source Source, opts ...IAMOption) (Caller, error) { + if source != SourceGatewayAssertion && source != SourceCustomAssertion && source != SourceVerifiedIAMProof { + return Caller{}, ErrInvalidCaller + } + var cfg iamConfig + for _, opt := range opts { + if opt == nil { + return Caller{}, ErrInvalidCaller + } + if err := opt(&cfg); err != nil { + return Caller{}, err + } + } + if len(principalARN) > 2048 { + return Caller{}, ErrInvalidCaller + } + parts := strings.SplitN(principalARN, ":", 6) + if len(parts) != 6 || parts[0] != "arn" || !validPartition(parts[1]) || parts[3] != "" || !validAccount(parts[4]) { + return Caller{}, ErrInvalidCaller + } + if cfg.accountSet && cfg.account != parts[4] { + return Caller{}, ErrInvalidCaller + } + var form IAMPrincipalType + resource := parts[5] + switch parts[2] { + case "iam": + switch { + case resource == "root": + form = IAMPrincipalRoot + case strings.HasPrefix(resource, "user/") && validUserResource(resource[len("user/"):]): + form = IAMPrincipalUser + } + case "sts": + switch { + case strings.HasPrefix(resource, "assumed-role/"): + role, session, ok := strings.Cut(resource[len("assumed-role/"):], "/") + if ok && validIAMName(role, 1, 64) && validIAMName(session, 2, 64) { + form = IAMPrincipalAssumedRole + } + case strings.HasPrefix(resource, "federated-user/"): + if validIAMName(resource[len("federated-user/"):], 2, 32) { + form = IAMPrincipalFederatedUser + } + } + } + if form == IAMPrincipalUnknown { + return Caller{}, ErrInvalidCaller + } + return Caller{source: source, iam: &IAM{ + arn: strings.Clone(principalARN), partition: strings.Clone(parts[1]), + account: strings.Clone(parts[4]), service: strings.Clone(parts[2]), + principalType: form, principalID: strings.Clone(cfg.principalID), + }}, nil +} + +func validAccount(account string) bool { + if len(account) != 12 { + return false + } + for i := range len(account) { + if account[i] < '0' || account[i] > '9' { + return false + } + } + return true +} + +// Validate lexical partition syntax, not an allowlist of today's AWS partitions. +// A producer such as the STS verifier must establish its expected partition. +func validPartition(partition string) bool { + if partition == "" || partition[0] < 'a' || partition[0] > 'z' { + return false + } + for i := range len(partition) { + b := partition[i] + if !(b >= 'a' && b <= 'z' || b >= '0' && b <= '9' || b == '-') { + return false + } + } + return true +} + +func validIAMName(name string, minimum, maximum int) bool { + if len(name) < minimum || len(name) > maximum { + return false + } + for i := range len(name) { + b := name[i] + if !(b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' || b >= '0' && b <= '9' || strings.ContainsRune("_+=,.@-", rune(b))) { + return false + } + } + return true +} + +func validUserResource(resource string) bool { + last := strings.LastIndexByte(resource, '/') + if !validIAMName(resource[last+1:], 1, 64) { + return false + } + if last == -1 { + return true // The default IAM path is "/". + } + // The ARN's user/ separator contributes the path's leading slash. Follow + // CreateUser's formal path regex (U+0021..U+007E), not its prose DEL endpoint. + path := resource[:last+1] + if len(path)+1 > 512 || len(path) < 2 { + return false + } + for i := range len(path) { + if path[i] < 0x21 || path[i] > 0x7e { + return false + } + } + return true +} + +// PrincipalARN returns the complete ARN, including any path and session name. +func (i IAM) PrincipalARN() string { return i.arn } + +// Partition returns the exact partition component. +func (i IAM) Partition() string { return i.partition } + +// AccountID returns the caller's twelve-digit account ID. +func (i IAM) AccountID() string { return i.account } + +// Service returns iam or sts, or empty for a zero IAM view. +func (i IAM) Service() string { return i.service } + +// PrincipalType returns the validated caller form. +func (i IAM) PrincipalType() IAMPrincipalType { return i.principalType } + +// PrincipalID reports whether an opaque nonempty principal ID was supplied. +func (i IAM) PrincipalID() (string, bool) { return i.principalID, i.principalID != "" } + +// String returns a diagnostic description without identifiers. +func (i IAM) String() string { return "identity.IAM" } + +// GoString returns the same sanitized description as String. +func (i IAM) GoString() string { return i.String() } diff --git a/identity/jwt.go b/identity/jwt.go new file mode 100644 index 0000000..386d008 --- /dev/null +++ b/identity/jwt.go @@ -0,0 +1,234 @@ +package identity + +import ( + "errors" + "slices" + "strings" +) + +// JWT is an immutable view of normalized JWT facts and their owned claims. +// It makes no assertions about token validity, flow, expiry or permissions. +// Its zero value contains no identity facts. +type JWT struct { + claims Claims + issuer string + subject string + clientID string + tokenUse string + audience stringSet + scopes stringSet + groups stringSet +} + +type stringSet struct { + values []string + known bool +} + +// JWTOption supplies a separate assertion. Nil or duplicate options are invalid. +type JWTOption func(*jwtConfig) error + +type jwtConfig struct { + scopes []string + scopesSet bool +} + +// WithGatewayScopes supplies API Gateway's dedicated scopes collection. It is +// valid only with SourceGatewayAssertion. Nil means unavailable; a nonnil empty +// slice means known empty. Interpretable claim scopes must agree as exact sets. +// The supplied collection consumes the claims' remaining weighted budget before +// copying, even when its values duplicate a claim. Do not mutate it during NewJWT. +func WithGatewayScopes(scopes []string) JWTOption { + return func(c *jwtConfig) error { + if c.scopesSet { + return ErrInvalidCaller + } + c.scopesSet, c.scopes = true, scopes + return nil + } +} + +// NewJWT validates normalized facts and retains immutable claims. It requires +// a nonempty issuer and a subject or client_id, without conflating those fields. +// Allowed sources are gateway, locally verified token and custom assertion. +// It does not verify signatures or expiry. Invalid inputs match ErrInvalidCaller; +// a dedicated-scope budget failure additionally matches ErrClaimsLimit. +func NewJWT(claims Claims, source Source, opts ...JWTOption) (Caller, error) { + if source != SourceGatewayAssertion && source != SourceLocallyVerifiedToken && source != SourceCustomAssertion { + return Caller{}, ErrInvalidCaller + } + var cfg jwtConfig + for _, opt := range opts { + if opt == nil { + return Caller{}, ErrInvalidCaller + } + if err := opt(&cfg); err != nil { + return Caller{}, err + } + } + if cfg.scopesSet && source != SourceGatewayAssertion { + return Caller{}, ErrInvalidCaller + } + j := JWT{claims: claims} + for _, fact := range []struct { + name string + dest *string + }{{"iss", &j.issuer}, {"sub", &j.subject}, {"client_id", &j.clientID}, {"token_use", &j.tokenUse}} { + v, present := claims.Lookup(fact.name) + if !present { + continue + } + s, ok := v.Text() + if !ok || s == "" { + return Caller{}, ErrInvalidCaller + } + *fact.dest = s + } + if j.issuer == "" || j.subject == "" && j.clientID == "" { + return Caller{}, ErrInvalidCaller + } + var err error + j.audience, err = claimStrings(claims, "aud", source, true, false) + if err != nil { + return Caller{}, err + } + j.groups, err = claimStrings(claims, "cognito:groups", source, false, false) + if err != nil { + return Caller{}, err + } + j.scopes, err = claimStrings(claims, "scp", source, false, true) + if err != nil { + return Caller{}, err + } + if v, present := claims.Lookup("scope"); present { + s, ok := v.Text() + if !ok || s == "" { + return Caller{}, ErrInvalidCaller + } + // Validate before allocating; whitespace is specifically ASCII SP. + for part := range strings.SplitSeq(s, " ") { + if !validScope(part) { + return Caller{}, ErrInvalidCaller + } + } + parsed := normalizedSet(strings.Split(s, " ")) + j.scopes, err = consistentScopes(j.scopes, parsed) + if err != nil { + return Caller{}, err + } + } + if cfg.scopes != nil { + budget := claimBudget{maximum: claims.budget, remaining: claims.budget - claims.charge} + if err := budget.take(claimCharge); err != nil { + return Caller{}, errors.Join(ErrInvalidCaller, err) + } + if len(cfg.scopes) > budget.remaining/claimCharge { + return Caller{}, errors.Join(ErrInvalidCaller, &ClaimsLimitError{maximum: int64(budget.maximum)}) + } + for _, scope := range cfg.scopes { + if err := budget.take(claimCharge); err != nil { + return Caller{}, errors.Join(ErrInvalidCaller, err) + } + if err := budget.take(len(scope)); err != nil { + return Caller{}, errors.Join(ErrInvalidCaller, err) + } + if !validScope(scope) { + return Caller{}, ErrInvalidCaller + } + } + scopes := make([]string, len(cfg.scopes)) + for i, scope := range cfg.scopes { + scopes[i] = strings.Clone(scope) + } + j.scopes, err = consistentScopes(j.scopes, normalizedSet(scopes)) + if err != nil { + return Caller{}, err + } + } + return Caller{source: source, jwt: &j}, nil +} + +func claimStrings(claims Claims, name string, source Source, allowString, scope bool) (stringSet, error) { + v, present := claims.Lookup(name) + if !present { + return stringSet{}, nil + } + if s, ok := v.Text(); ok { + if source == SourceGatewayAssertion || v.Representation() == RepresentationGatewayText { + return stringSet{}, nil + } + if allowString && s != "" { + return stringSet{values: []string{s}, known: true}, nil + } + return stringSet{}, ErrInvalidCaller + } + if v.Kind() != ClaimArray { + return stringSet{}, ErrInvalidCaller + } + values := make([]string, len(v.node.array)) + for i, n := range v.node.array { + if n.kind != ClaimString || n.text == "" || scope && !validScope(n.text) { + return stringSet{}, ErrInvalidCaller + } + values[i] = n.text + } + return normalizedSet(values), nil +} + +func normalizedSet(values []string) stringSet { + slices.Sort(values) + return stringSet{values: slices.Compact(values), known: true} +} + +func consistentScopes(a, b stringSet) (stringSet, error) { + if a.known && !slices.Equal(a.values, b.values) { + return stringSet{}, ErrInvalidCaller + } + return b, nil +} + +// RFC 6749 scope-token = 1*(%x21 / %x23-5B / %x5D-7E). +func validScope(s string) bool { + if s == "" { + return false + } + for i := range len(s) { + b := s[i] + if b < 0x21 || b > 0x7e || b == '"' || b == '\\' { + return false + } + } + return true +} + +// Claims returns an immutable view of the original supplied claims. +func (j JWT) Claims() Claims { return j.claims } + +// Issuer returns the exact issuer, or empty for a zero JWT. +func (j JWT) Issuer() string { return j.issuer } + +// Subject reports whether a nonempty subject was supplied. +func (j JWT) Subject() (string, bool) { return j.subject, j.subject != "" } + +// ClientID reports whether a nonempty client_id was supplied. +func (j JWT) ClientID() (string, bool) { return j.clientID, j.clientID != "" } + +// TokenUse reports whether a nonempty token_use was supplied. +func (j JWT) TokenUse() (string, bool) { return j.tokenUse, j.tokenUse != "" } + +// Audience returns a copied sorted set and reports whether its meaning is known. +// Gateway string audiences remain available in Claims but are not split or parsed. +func (j JWT) Audience() ([]string, bool) { return slices.Clone(j.audience.values), j.audience.known } + +// Scopes returns a copied sorted set, distinguishing known empty from unavailable. +func (j JWT) Scopes() ([]string, bool) { return slices.Clone(j.scopes.values), j.scopes.known } + +// CognitoGroups returns a copied sorted set of cognito:groups when interpretable. +// It does not promote arbitrary groups or roles claims into Cognito groups. +func (j JWT) CognitoGroups() ([]string, bool) { return slices.Clone(j.groups.values), j.groups.known } + +// String returns a diagnostic description without identity facts. +func (j JWT) String() string { return "identity.JWT" } + +// GoString returns the same sanitized description as String. +func (j JWT) GoString() string { return j.String() } diff --git a/invocation.go b/invocation.go index 72504a9..b2a47a6 100644 --- a/invocation.go +++ b/invocation.go @@ -5,6 +5,8 @@ import ( "errors" "io" "net/http" + + "github.com/asteroid-computing/go-lambda-edge/identity" ) // invocation owns the transport-created request and original body. The context @@ -44,6 +46,9 @@ func invocationContext(parent context.Context) (context.Context, context.CancelF if err := parent.Err(); err != nil { return nil, nil, err } + if identity.FromContext(parent).Kind() != identity.KindAnonymous { + return nil, nil, invocationError("identity", ErrIdentity, "invocation context already has a caller", identity.ErrConflict) + } ctx, cancel := context.WithCancel(parent) return ctx, cancel, nil } diff --git a/invocation_test.go b/invocation_test.go index f36a540..11ad0fd 100644 --- a/invocation_test.go +++ b/invocation_test.go @@ -8,8 +8,38 @@ import ( "strings" "testing" "time" + + "github.com/asteroid-computing/go-lambda-edge/identity" ) +func TestInvocationRejectsInheritedCaller(t *testing.T) { + caller, err := identity.NewIAM("arn:aws:iam::123456789012:user/Alice", identity.SourceGatewayAssertion) + if err != nil { + t.Fatal(err) + } + parent, err := identity.WithCaller(t.Context(), caller) + if err != nil { + t.Fatal(err) + } + ran := false + err = withInvocation(parent, func(context.Context, *invocation) error { + ran = true + return nil + }) + if ran || !errors.Is(err, ErrIdentity) || !errors.Is(err, identity.ErrConflict) { + t.Fatalf("inherited caller: ran=%t, err=%v; want identity conflict before work", ran, err) + } + if detail, ok := errors.AsType[*InvocationError](err); !ok || detail.Operation() != "identity" { + t.Errorf("inherited caller error = %v, want identity operation", err) + } + if strings.Contains(err.Error(), "Alice") { + t.Error("isolation error leaked caller identity") + } + if err := withInvocation(t.Context(), func(context.Context, *invocation) error { return nil }); err != nil { + t.Errorf("independent next invocation rejected: %v", err) + } +} + func TestInvocationRejectsInvalidContextBeforeRunning(t *testing.T) { canceled, cancel := context.WithCancel(t.Context()) cancel() From 50b3e34bb097985fd3ac8b5ff009de957195df82 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 16:44:07 +0100 Subject: [PATCH 27/44] feat: wire buffered Lambda adapter and native gateway identity --- README.md | 51 ++- adapter.go | 15 +- adapter_bench_test.go | 67 ++++ adapter_example_test.go | 46 +++ adapter_invoke.go | 134 +++++++ adapter_invoke_test.go | 343 +++++++++++++++++ docs/benchmarks.md | 30 ++ docs/decisions/0002-adapter-boundary.md | 7 +- docs/decisions/0004-typed-events.md | 17 +- docs/decisions/0006-buffered-responses.md | 9 +- docs/decisions/0012-shared-identity.md | 8 +- docs/decisions/0014-claims-api.md | 5 +- .../0018-gateway-identity-recognition.md | 22 +- docs/decisions/README.md | 2 +- docs/gateway-fixtures.md | 44 +++ docs/plan.md | 37 +- errors.go | 2 +- gateway_identity.go | 318 +++++++++++++++ gateway_identity_test.go | 363 ++++++++++++++++++ 19 files changed, 1454 insertions(+), 66 deletions(-) create mode 100644 adapter_bench_test.go create mode 100644 adapter_example_test.go create mode 100644 adapter_invoke.go create mode 100644 adapter_invoke_test.go create mode 100644 docs/gateway-fixtures.md create mode 100644 gateway_identity.go create mode 100644 gateway_identity_test.go diff --git a/README.md b/README.md index f9af370..f846130 100644 --- a/README.md +++ b/README.md @@ -4,18 +4,23 @@ A Go 1.27 module being built to serve AWS API Gateway requests through ordinary `net/http` handlers in AWS Lambda, with companion identity, authentication, and authorization packages. -The root package is `edge`. This project is being designed and built from scratch. -The constructor, options, private event decoder, and request conversion are implemented. The decoder -uses AWS Lambda Go v1.55.0 event structs and encoding/json/v2 directly, retaining -authorizer JSON separately. Request conversion covers URLs, headers, body bytes, -and request lifetime. A shared invocation scope now covers cleanup and cancellation; -private response encoding provides text/base64 selection and bounded direct JSON -v2 output. Private response-header snapshots and V1/V2 projections now enforce -the accepted syntax, suppression, cookie and field-combination rules. The private -buffered HTTP writer now implements commitment, sniffing, bodyless/HEAD responses, -length enforcement and typed AWS response projection. Public invocation methods -and identity extraction are not implemented yet; the adapter cannot yet be -registered as a Lambda handler. +The root package is `edge`. The buffered adapter is implemented for REST proxy, +HTTP API payload 1.0 and HTTP API payload 2.0. It uses AWS Lambda Go v1.55.0 types +and direct encoding/json/v2 processing. Raw and already typed entry points share +HTTP conversion, response capture, identity policy, cancellation and cleanup. + +Create an adapter with `edge.New(handler, options...)`, handle the constructor +error, then register it with `lambda.Start(adapter)`. Pass the object, not +`adapter.Invoke`, to retain the raw JSON v2 boundary. For a fixed payload family, +use `lambda.Start(adapter.HandleV1)` or `lambda.Start(adapter.HandleV2)`; these +delegate envelope JSON to the SDK and do no envelope serialization inside edge. +See the [runnable registration examples](adapter_example_test.go). + +Gateway identity is disabled by default. `edge.WithGatewayIdentity(true)` enables +the reviewed native IAM and JWT/Cognito field mappings. Invalid, conflicting or +unsupported assertions fail before the handler; custom-authorizer mapping remains +unimplemented. Authorization headers are not independently authenticated. See the +[fixture coverage and fidelity qualifications](docs/gateway-fixtures.md). The shared invocation error contract is implemented across those private stages: public categories support `errors.Is`, and `errors.AsType[*edge.InvocationError]` @@ -23,8 +28,9 @@ exposes sanitized operation and resource-limit diagnostics. HTTP error responses remain successful transport outcomes; operation, cleanup, cancellation, and late stream failures retain their accepted ownership and precedence. The shared claims API, caller/context construction, JWT normalization and resource limits -are implemented in `identity`. The private invocation boundary rejects inherited -callers. Native gateway extraction remains pending its fixture/recognition review. +are implemented in `identity`. Every invocation rejects an inherited caller, +including when gateway identity is disabled. Constructors themselves do not +authenticate credentials or authorize application actions. The accepted authentication direction supports IAM credentials or an OAuth bearer token on the same route without built-in `AWS_IAM` authorization. IAM @@ -55,8 +61,7 @@ Use ordinary `http.Handler` middleware to process this and other custom headers. The consumer owns its action registry, metadata, authorization and HTTP error responses. Select once, authorize that selection, then execute it; do not select again from mutable headers. An action header does not authenticate a caller. -The selector is usable now with ordinary Go HTTP servers, independently of the -unfinished Lambda invocation methods. +The selector works with both ordinary Go HTTP servers and the Lambda adapter. See the [runnable middleware example](action_example_test.go) and [accepted header contract](docs/decisions/0011-request-header-processing.md). @@ -69,8 +74,9 @@ value costs `len(name) + len(value) + 32`; nil/empty slices still cost a name pl 32, and generated headers consume remaining budget. This is separate from the complete buffered-envelope limit and the 16,000-byte streaming metadata prefix (including its delimiter). It is neither an AWS quota nor a total heap bound. -The option validates configuration now, and the private buffered writer applies -it at commitment. Public invocation and the streaming HTTP writer remain pending. +The buffered writer applies the allowance at commitment. Raw invocation also +checks the complete serialized response against 6 MiB; typed callers own their +final envelope serialization and size. The streaming HTTP writer remains pending. ## Owned claims @@ -101,8 +107,13 @@ The default claim allowance is 256 KiB, configurable with object-name, string and exact-number text bytes. Raw JSON also has a wire-length limit of the same size. Nesting is limited to 64 containers. These are resource policies, not heap caps; [measurements](docs/benchmarks.md) include allocation and -retained-memory costs. `edge.WithIdentityClaimsBudget(bytes)` records the matching -gateway configuration, whose producer wiring remains pending. +retained-memory costs. `edge.WithIdentityClaimsBudget(bytes)` applies the matching +allowance to gateway claims and dedicated scopes. + +Handlers receive an owned request body and canceled context after invocation. +Multipart files parsed on the served request are removed on success, error or +panic. Middleware that parses multipart on a separate request copy owns cleanup +of files attached only to that copy. Handler panics propagate after cleanup. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) diff --git a/adapter.go b/adapter.go index 93fb866..ea3d73f 100644 --- a/adapter.go +++ b/adapter.go @@ -1,5 +1,6 @@ -// Package edge provides the foundation for adapting AWS API Gateway events to -// net/http handlers. Public invocation is not yet implemented. +// Package edge adapts AWS API Gateway Lambda proxy events to net/http handlers. +// It supports raw JSON v2 and already typed AWS events with optional native +// gateway identity. Buffered responses use one shared HTTP translation policy. package edge import ( @@ -11,7 +12,8 @@ import ( // Adapter holds an HTTP handler and its API Gateway adaptation configuration. // Construct an Adapter with [New]; its zero value is not usable. -// Configuration is fixed at construction. Invocation is not yet implemented. +// Configuration is fixed at construction. Calls own separate request state; +// a shared handler must be safe for concurrent use. type Adapter struct { handler http.Handler config config @@ -32,8 +34,7 @@ type Option func(*config) // The default is false. When enabled, the policy covers native IAM and // JWT/Cognito assertions; it does not configure authorization in AWS or enable // local token verification. Custom-authorizer contexts need an explicit mapper. -// This option currently records policy; invocation and extraction are not yet -// implemented. +// Unsupported or conflicting assertions fail before running the HTTP handler. func WithGatewayIdentity(enabled bool) Option { return func(c *config) { c.gatewayIdentity = enabled @@ -45,7 +46,7 @@ func WithGatewayIdentity(enabled bool) Option { // Each claim value costs 64 bytes plus name, string and exact-number text bytes. // Dedicated gateway scopes consume the same allowance. Raw claim JSON is also // limited to this byte length. This is a resource policy, not a heap cap or AWS -// quota. It does not enable gateway identity. Extraction is not yet implemented. +// quota. It does not enable gateway identity. func WithIdentityClaimsBudget(bytes int) Option { return func(c *config) { c.identityClaimsBudget = bytes @@ -58,7 +59,7 @@ func WithIdentityClaimsBudget(bytes int) Option { // value length and 32 bytes; nil/empty slices cost their name length and 32. // Generated headers also consume the budget. This is a library resource policy, // not an AWS quota or a bound on total heap usage. It does not change the separate -// buffered envelope or streaming metadata limits. Invocation is not yet wired. +// buffered envelope or streaming metadata limits. func WithResponseHeaderBudget(bytes int) Option { return func(c *config) { c.responseHeaderBudget = bytes diff --git a/adapter_bench_test.go b/adapter_bench_test.go new file mode 100644 index 0000000..8bd6781 --- /dev/null +++ b/adapter_bench_test.go @@ -0,0 +1,67 @@ +package edge_test + +import ( + "encoding/base64" + "encoding/json/v2" + "io" + "net/http" + "strings" + "testing" + + "github.com/aws/aws-lambda-go/events" + + edge "github.com/asteroid-computing/go-lambda-edge" +) + +func BenchmarkAdapterPublic(b *testing.B) { + for _, fixture := range []struct { + name string + size int + binary bool + }{{name: "small", size: 128}, {name: "binary64KiB", size: 64 * 1024, binary: true}, {name: "text5MiB", size: 5 * 1024 * 1024}} { + b.Run(fixture.name, func(b *testing.B) { + body := strings.Repeat("x", fixture.size) + mediaType := "text/plain" + if fixture.binary { + body = base64.StdEncoding.EncodeToString([]byte(body)) + mediaType = "application/octet-stream" + } + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", mediaType) + _, _ = io.Copy(w, r.Body) + })) + if err != nil { + b.Fatal(err) + } + v1 := events.APIGatewayProxyRequest{HTTPMethod: "POST", Path: "/", Body: body, IsBase64Encoded: fixture.binary, RequestContext: events.APIGatewayProxyRequestContext{APIID: "api"}} + v2 := events.APIGatewayV2HTTPRequest{Version: "2.0", RawPath: "/", Body: body, IsBase64Encoded: fixture.binary, RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "api", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "POST"}}} + raw1, err := json.Marshal(v1) + if err != nil { + b.Fatal(err) + } + raw2, err := json.Marshal(v2) + if err != nil { + b.Fatal(err) + } + for _, path := range []struct { + name string + run func() error + }{ + {"rawV1", func() error { _, err := a.Invoke(b.Context(), raw1); return err }}, + {"rawV2", func() error { _, err := a.Invoke(b.Context(), raw2); return err }}, + {"typedV1", func() error { _, err := a.HandleV1(b.Context(), v1); return err }}, + {"typedV2", func() error { _, err := a.HandleV2(b.Context(), v2); return err }}, + } { + b.Run(path.name, func(b *testing.B) { + b.ReportAllocs() + b.SetBytes(int64(fixture.size)) + for b.Loop() { + if err := path.run(); err != nil { + b.Fatal(err) + } + } + }) + } + }) + } +} diff --git a/adapter_example_test.go b/adapter_example_test.go new file mode 100644 index 0000000..b2b4b77 --- /dev/null +++ b/adapter_example_test.go @@ -0,0 +1,46 @@ +package edge_test + +import ( + "context" + "fmt" + "net/http" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-lambda-go/lambda" + + edge "github.com/asteroid-computing/go-lambda-edge" +) + +func ExampleAdapter_Invoke() { + adapter, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNoContent) + })) + if err != nil { + panic(err) + } + // Production main calls lambda.Start(adapter). Passing adapter.Invoke as a + // reflected function would give []byte ordinary JSON/base64 semantics. + handler := lambda.NewHandler(adapter) + response, err := handler.Invoke(context.Background(), []byte(`{"httpMethod":"GET","path":"/","requestContext":{"apiId":"example"}}`)) + fmt.Println(len(response) > 0, err) + // Output: true +} + +func ExampleAdapter_HandleV2() { + adapter, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusAccepted) + })) + if err != nil { + panic(err) + } + // Production main may call lambda.Start(adapter.HandleV2). Its runtime then + // owns envelope JSON. This direct typed call performs no envelope JSON work. + response, err := adapter.HandleV2(context.Background(), events.APIGatewayV2HTTPRequest{ + Version: "2.0", RawPath: "/", + RequestContext: events.APIGatewayV2HTTPRequestContext{ + APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "POST"}, + }, + }) + fmt.Println(response.StatusCode, err) + // Output: 202 +} diff --git a/adapter_invoke.go b/adapter_invoke.go new file mode 100644 index 0000000..f4e6540 --- /dev/null +++ b/adapter_invoke.go @@ -0,0 +1,134 @@ +package edge + +import ( + "context" + "net/http" + + "github.com/aws/aws-lambda-go/events" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Invoke adapts a REST proxy, HTTP API 1.0 or HTTP API 2.0 event using JSON v2 +// directly. Register the Adapter object with lambda.Start(adapter), not its +// Invoke method as a reflected function. Transport/identity failures return no +// usable payload; ordinary HTTP error responses remain successful invocations. +// Panics propagate after cleanup. The caller must not mutate payload during use. +func (a *Adapter) Invoke(ctx context.Context, payload []byte) ([]byte, error) { + if err := a.valid(); err != nil { + return nil, err + } + var result []byte + err := withInvocation(ctx, func(ctx context.Context, inv *invocation) error { + event, err := decodeEvent(payload) + if err != nil { + return err + } + response, err := a.buffered(ctx, inv, event, true) + if err != nil { + return err + } + var projected any + if event.v1 != nil { + projected, err = response.v1() + } else { + projected, err = response.v2() + } + if err != nil { + return err + } + result, err = marshalResponse(projected) + return err + }) + if err != nil { + return nil, err + } + return result, nil +} + +// HandleV1 adapts an already decoded REST/HTTP API 1.0 event without serializing +// envelopes inside edge. It can be registered as lambda.Start(adapter.HandleV1). +// The upstream decoder owns wire validation and numeric fidelity; the caller or +// runtime owns final response serialization and its complete envelope size. +// On failure it returns a zero response. Inputs must not change during the call. +func (a *Adapter) HandleV1(ctx context.Context, event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { + if err := a.valid(); err != nil { + return events.APIGatewayProxyResponse{}, err + } + var result events.APIGatewayProxyResponse + err := withInvocation(ctx, func(ctx context.Context, inv *invocation) error { + response, err := a.buffered(ctx, inv, decodedEvent{v1: &event}, false) + if err != nil { + return err + } + result, err = response.v1() + return err + }) + if err != nil { + return events.APIGatewayProxyResponse{}, err + } + return result, nil +} + +// HandleV2 adapts an already decoded HTTP API 2.0 event. Its ownership, error and +// serialization contract matches HandleV1. V2 SDK string claims cannot recover +// arrays or exact numeric values discarded by an upstream decoder. +func (a *Adapter) HandleV2(ctx context.Context, event events.APIGatewayV2HTTPRequest) (events.APIGatewayV2HTTPResponse, error) { + if err := a.valid(); err != nil { + return events.APIGatewayV2HTTPResponse{}, err + } + var result events.APIGatewayV2HTTPResponse + err := withInvocation(ctx, func(ctx context.Context, inv *invocation) error { + response, err := a.buffered(ctx, inv, decodedEvent{v2: &event}, false) + if err != nil { + return err + } + result, err = response.v2() + return err + }) + if err != nil { + return events.APIGatewayV2HTTPResponse{}, err + } + return result, nil +} + +func (a *Adapter) valid() error { + if a == nil || a.handler == nil { + return invocationError("validate", ErrInvalidInvocation, "uninitialized adapter") + } + return nil +} + +func (a *Adapter) buffered(ctx context.Context, inv *invocation, event decodedEvent, raw bool) (*bufferedResponse, error) { + var request *http.Request + var err error + if event.v1 != nil { + request, err = requestV1(ctx, *event.v1) + } else { + request, err = requestV2(ctx, *event.v2) + } + if err != nil { + return nil, err + } + inv.ownRequest(request) + if a.config.gatewayIdentity { + var caller identity.Caller + if raw { + caller, err = rawGatewayCaller(event, a.config.identityClaimsBudget) + } else { + caller, err = typedGatewayCaller(event, a.config.identityClaimsBudget) + } + if err != nil { + return nil, err + } + ctx, err = identity.WithCaller(ctx, caller) + if err != nil { + return nil, invocationError("identity", ErrIdentity, "caller installation failed", identity.ErrConflict) + } + request = request.WithContext(ctx) + // Track the actual served request for multipart cleanup, retaining the + // original transport body separately even if the handler replaces it. + inv.request = request + } + return serveBuffered(a.handler, request, a.config.responseHeaderBudget) +} diff --git a/adapter_invoke_test.go b/adapter_invoke_test.go new file mode 100644 index 0000000..8b23dbd --- /dev/null +++ b/adapter_invoke_test.go @@ -0,0 +1,343 @@ +package edge_test + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/json/v2" + "errors" + "io" + "mime/multipart" + "net/http" + "os" + "strings" + "sync" + "testing" + "time" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-lambda-go/lambda" + + edge "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +var _ lambda.Handler = (*edge.Adapter)(nil) + +func TestAdapterSDKRegistration(t *testing.T) { + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, err := io.ReadAll(r.Body) + if err != nil { + t.Error(err) + } + if !bytes.Equal(body, []byte{0, 255}) || r.URL.Path != "/orders" { + t.Errorf("request body=%v path=%q", body, r.URL.Path) + } + w.Header().Set("Content-Type", "application/octet-stream") + w.Header().Add("Vary", "Accept") + w.Header().Add("Vary", "Origin") + http.SetCookie(w, &http.Cookie{Name: "a", Value: "1"}) + http.SetCookie(w, &http.Cookie{Name: "b", Value: "2"}) + w.WriteHeader(400) + _, _ = w.Write(body) + })) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name string + handler any + v2 bool + legacy bool + }{ + {name: "raw_rest", handler: a, legacy: true}, + {name: "raw_http1", handler: a}, + {name: "raw_http2", handler: a, v2: true}, + {name: "typed_v1", handler: a.HandleV1}, + {name: "typed_v2", handler: a.HandleV2, v2: true}, + } { + t.Run(tc.name, func(t *testing.T) { + payload := gatewayFixture(tc.v2, "", "") + payload = append([]byte(`{"body":"AP8=","isBase64Encoded":true,`), payload[1:]...) + if tc.legacy { + payload = bytes.Replace(payload, []byte(`"version":"1.0",`), nil, 1) + } + out, err := lambda.NewHandler(tc.handler).Invoke(t.Context(), payload) + if err != nil { + t.Fatal(err) + } + var response struct { + StatusCode int `json:"statusCode"` + Body string `json:"body"` + Binary bool `json:"isBase64Encoded"` + Cookies []string `json:"cookies"` + MultiValueHeaders map[string][]string `json:"multiValueHeaders"` + } + if err := json.Unmarshal(out, &response); err != nil { + t.Fatalf("SDK did not return an object envelope: %v", err) + } + if response.StatusCode != 400 || response.Body != "AP8=" || !response.Binary { + t.Errorf("application error/binary response = %+v", response) + } + cookies := response.Cookies + if !tc.v2 { + cookies = response.MultiValueHeaders["Set-Cookie"] + } + if len(cookies) != 2 { + t.Errorf("cookie count = %d, want 2", len(cookies)) + } + }) + } +} + +func TestAdapterSDKPreservesTypedNumberOption(t *testing.T) { + var exact bool + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + jwt, _ := identity.FromContext(r.Context()).JWT() + claim, _ := jwt.Claims().Lookup("large") + text, ok := claim.NumberText() + exact = ok && text == "9007199254740993" + }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + payload := gatewayFixture(false, `{"claims":{"iss":"issuer","sub":"subject","large":9007199254740993}}`, "") + for _, useNumber := range []bool{false, true} { + _, err := lambda.NewHandlerWithOptions(a.HandleV1, lambda.WithUseNumber(useNumber)).Invoke(t.Context(), payload) + if err != nil || exact != useNumber { + t.Errorf("UseNumber=%t: exact=%t err=%v", useNumber, exact, err) + } + } +} + +func TestAdapterFailuresHaveNoResult(t *testing.T) { + var ran bool + a, err := edge.New(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { ran = true })) + if err != nil { + t.Fatal(err) + } + for _, payload := range [][]byte{[]byte(`{`), []byte(`{}`), []byte(`{"version":"3.0"}`)} { + out, err := a.Invoke(t.Context(), payload) + if err == nil || out != nil || ran { + t.Errorf("malformed invocation out=%v err=%v ran=%t", out, err, ran) + } + } + for _, invalid := range []*edge.Adapter{nil, {}} { + if _, err := invalid.Invoke(t.Context(), nil); !errors.Is(err, edge.ErrInvalidInvocation) { + t.Errorf("uninitialized Invoke = %v", err) + } + if out, err := invalid.HandleV1(t.Context(), events.APIGatewayProxyRequest{}); !errors.Is(err, edge.ErrInvalidInvocation) || out.StatusCode != 0 { + t.Errorf("uninitialized HandleV1 = %v", err) + } + if out, err := invalid.HandleV2(t.Context(), events.APIGatewayV2HTTPRequest{}); !errors.Is(err, edge.ErrInvalidInvocation) || out.StatusCode != 0 { + t.Errorf("uninitialized HandleV2 = %v", err) + } + } + caller, err := identity.NewIAM("arn:aws:iam::123456789012:root", identity.SourceCustomAssertion) + if err != nil { + t.Fatal(err) + } + parent, err := identity.WithCaller(t.Context(), caller) + if err != nil { + t.Fatal(err) + } + if out, err := a.Invoke(parent, gatewayFixture(false, "", "")); out != nil || !errors.Is(err, identity.ErrConflict) || ran { + t.Errorf("inherited caller: err=%v ran=%t", err, ran) + } +} + +func TestAdapterRequestIsolation(t *testing.T) { + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + jwt, ok := identity.FromContext(r.Context()).JWT() + if !ok { + _, _ = io.WriteString(w, "anonymous") + return + } + sub, _ := jwt.Subject() + r.Header.Set("Shared", sub) + _, _ = io.WriteString(w, sub) + }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + var wg sync.WaitGroup + for range 20 { + for _, subject := range []string{"Alice", "Bob", "anonymous"} { + wg.Go(func() { + auth := `{"claims":{"iss":"issuer","sub":"` + subject + `"}}` + if subject == "anonymous" { + auth = "null" + } + out, err := a.Invoke(t.Context(), gatewayFixture(false, auth, "")) + if err != nil { + t.Error(err) + return + } + var response events.APIGatewayProxyResponse + if err := json.Unmarshal(out, &response); err != nil || response.Body != subject { + t.Errorf("isolated subject %q got %q, err=%v", subject, response.Body, err) + } + }) + } + } + wg.Wait() +} + +func TestAdapterMultipartCleanupAndPanic(t *testing.T) { + var body bytes.Buffer + writer := multipart.NewWriter(&body) + part, err := writer.CreateFormFile("upload", "file.txt") + if err != nil { + t.Fatal(err) + } + _, _ = io.WriteString(part, "temporary file content") + if err := writer.Close(); err != nil { + t.Fatal(err) + } + for _, panicHandler := range []bool{false, true} { + var path string + var served context.Context + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + served = r.Context() + if err := r.ParseMultipartForm(0); err != nil { + t.Error(err) + return + } + file, err := r.MultipartForm.File["upload"][0].Open() + if err != nil { + t.Error(err) + return + } + if disk, ok := file.(*os.File); ok { + path = disk.Name() + } + _ = file.Close() + if panicHandler { + panic("expected synthetic panic") + } + }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + event := events.APIGatewayProxyRequest{Path: "/", HTTPMethod: "POST", Body: body.String(), Headers: map[string]string{"Content-Type": writer.FormDataContentType()}, RequestContext: events.APIGatewayProxyRequestContext{APIID: "api"}} + func() { + defer func() { + if caught := recover(); (caught != nil) != panicHandler { + t.Errorf("panic propagated=%t, want %t", caught != nil, panicHandler) + } + }() + if _, err := a.HandleV1(t.Context(), event); err != nil { + t.Error(err) + } + }() + if path == "" { + t.Fatal("multipart fixture did not spill to disk") + } + if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) { + t.Errorf("multipart temporary file remains: %v", err) + } + if !errors.Is(served.Err(), context.Canceled) { + t.Error("served context remained live after invocation") + } + } +} + +func TestAdapterCancellationAndStickyResponseFailure(t *testing.T) { + for _, cancelParent := range []bool{false, true} { + ctx, cancel := context.WithCancel(t.Context()) + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if cancelParent { + cancel() + _, _ = io.WriteString(w, "discarded") + return + } + w.WriteHeader(http.StatusEarlyHints) + _, _ = io.WriteString(w, "discarded") + })) + if err != nil { + t.Fatal(err) + } + out, err := a.Invoke(ctx, gatewayFixture(true, "", "")) + cancel() + var want error = http.ErrNotSupported + if cancelParent { + want = context.Canceled + } + if out != nil || !errors.Is(err, want) { + t.Errorf("discarded response=%s err=%v; want %v", out, err, want) + } + } +} + +func TestAdapterStandardHandlers(t *testing.T) { + mux := http.NewServeMux() + mux.HandleFunc("GET /files/{name}", func(w http.ResponseWriter, r *http.Request) { + http.ServeContent(w, r, r.PathValue("name"), time.Time{}, strings.NewReader("abcdef")) + }) + a, err := edge.New(mux) + if err != nil { + t.Fatal(err) + } + for _, v2 := range []bool{false, true} { + for _, redirect := range []bool{false, true} { + path := "/files/demo.txt" + if redirect { + path = "/files//demo.txt" + } + var status int + var body, location string + if v2 { + response, err := a.HandleV2(t.Context(), events.APIGatewayV2HTTPRequest{ + RawPath: path, Headers: map[string]string{"Range": "bytes=1-3"}, + RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "api", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}, + }) + if err != nil { + t.Fatal(err) + } + status, body, location = response.StatusCode, response.Body, response.Headers["Location"] + if response.IsBase64Encoded { + decoded, err := base64.StdEncoding.DecodeString(body) + if err != nil { + t.Fatal(err) + } + body = string(decoded) + } + } else { + response, err := a.HandleV1(t.Context(), events.APIGatewayProxyRequest{ + Path: path, HTTPMethod: "GET", Headers: map[string]string{"Range": "bytes=1-3"}, + RequestContext: events.APIGatewayProxyRequestContext{APIID: "api"}, + }) + if err != nil { + t.Fatal(err) + } + status, body = response.StatusCode, response.Body + location = http.Header(response.MultiValueHeaders).Get("Location") + } + if redirect { + if status != http.StatusTemporaryRedirect || location != "/files/demo.txt" { + t.Errorf("v2=%t redirect status=%d location=%q", v2, status, location) + } + } else if status != 206 || body != "bcd" { + t.Errorf("v2=%t ServeContent status=%d body=%q", v2, status, body) + } + } + } +} + +func TestAdapterRawEnvelopeLimitAndTypedOwnership(t *testing.T) { + body := strings.Repeat("x", 6*1024*1024) + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/plain") + _, _ = io.WriteString(w, body) + })) + if err != nil { + t.Fatal(err) + } + if out, err := a.Invoke(t.Context(), gatewayFixture(false, "", "")); out != nil || !errors.Is(err, edge.ErrLimitExceeded) { + t.Fatalf("raw envelope limit: result bytes=%d err=%v", len(out), err) + } + response, err := a.HandleV1(t.Context(), events.APIGatewayProxyRequest{Path: "/", HTTPMethod: "GET", RequestContext: events.APIGatewayProxyRequestContext{APIID: "api"}}) + if err != nil || len(response.Body) != len(body) { + t.Errorf("typed response should leave envelope serialization to caller: bytes=%d err=%v", len(response.Body), err) + } +} diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 71606aa..c6e2f10 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -204,3 +204,33 @@ samples; two collections before and after reduce pool retention. Small deltas are noisy (ordinary typed samples ranged from 2,696 to 8,016 bytes). No figure is peak memory, RSS, total invocation allocation, or a Lambda latency prediction. JWT normalized views and dedicated scopes are not implemented in these samples. + +## Public buffered invocation (2026-09-16) + +Go 1.27.1, darwin/arm64, Apple M2. These run the complete public adapter with +gateway identity disabled and a body-echo handler. Fixtures are created outside +the timed loop. Raw calls include event decoding and response JSON v2 encoding; +typed calls deliberately exclude upstream/downstream envelope codecs. They are +not an end-to-end SDK or Lambda performance comparison. + +```sh +go test -run '^$' -bench '^BenchmarkAdapterPublic$' -benchmem -benchtime=100ms -count=3 . +``` + +Median allocated bytes per invocation (three samples, not peak/live memory): + +| Body fixture | Raw V1 B/op | Raw V2 B/op | Typed V1 B/op | Typed V2 B/op | +| --- | ---: | ---: | ---: | ---: | +| 128-byte text | 6,230 | 5,357 | 3,792 | 3,536 | +| 64 KiB binary, base64 request/response | 777,999 | 776,900 | 322,514 | 322,258 | +| 5 MiB text, near buffered envelope limit | 37,776,984 | 37,777,048 | 15,731,675 | 15,731,427 | + +The small case used 65/48/35/33 allocations per invocation respectively. Large +raw envelopes incur substantial transient codec/copy costs despite bounded +output buffers. The 6 MiB wire limit is not a heap limit. These are initial local +baselines, without performance thresholds or Lambda latency claims. Short-run +timings vary with GC and host load; comparisons should use longer repeated runs. + +Initial three-iteration measurements included noticeable cold codec setup costs; +the table instead uses the repeated 100 ms runs above. Native identity costs are +additional and depend on claim shape and source fidelity. diff --git a/docs/decisions/0002-adapter-boundary.md b/docs/decisions/0002-adapter-boundary.md index d1af293..38ea8e7 100644 --- a/docs/decisions/0002-adapter-boundary.md +++ b/docs/decisions/0002-adapter-boundary.md @@ -131,5 +131,8 @@ external dependencies. Tests cover nil/typed-nil handlers, nil options, and construction without handler invocation. `go test -race ./...`, `go vet ./...`, and Linux arm64/amd64 package builds pass on Go 1.27.1; formatting is clean. -`Invoke`, identity behavior, duplicate-option behavior as observed during an -invocation, and integration tests follow the remaining payload/HTTP contracts. +Update 2026-09-16: Invoke, HandleV1 and HandleV2 are implemented using the shared +HTTP, native identity, response and cleanup boundaries. Lambda SDK object/function +registration tests pass alongside malformed-input, application-error, binary, +cookie, cancellation, panic and invocation-isolation cases. Registration examples +are runnable. Current public behavior is described in README.md and decision 0018. diff --git a/docs/decisions/0004-typed-events.md b/docs/decisions/0004-typed-events.md index d8c38da..e7f7405 100644 --- a/docs/decisions/0004-typed-events.md +++ b/docs/decisions/0004-typed-events.md @@ -1,6 +1,6 @@ # 0004: Typed event entry points -Status: accepted by the user; not implemented yet. +Status: accepted by the user; implemented and SDK-tested on 2026-09-16. ## Question @@ -33,13 +33,13 @@ func (a *Adapter) HandleV2(ctx context.Context, event events.APIGatewayV2HTTPReq ``` The user approved these names, exact signatures, and the boundary contract below. -They are not implemented yet; request conversion is complete, while response -encoding and gateway identity production remain under development. +Both methods are implemented, sharing the same request, response, ownership and +gateway identity primitives as raw invocation. The ordinary raw registration remains lambda.Start(adapter). A caller that chooses a fixed payload family can register lambda.Start(adapter.HandleV1) or lambda.Start(adapter.HandleV2), or pass an already decoded struct directly. -These examples describe the planned API, not working functionality today. +These registrations are exercised through the actual SDK handler wrapper. Use shared internal HTTP translation, response capture, and identity policy for both raw and typed entry points. Do not marshal a typed event and call Invoke; @@ -62,7 +62,7 @@ correct field presence, or preserved arbitrary numbers. In particular, V1 has no Version field to validate. Do not promise raw-wire validation equivalence or recover values lost by a caller's decoder. Identity extraction must respect the fidelity of the supplied source and must never supplement it with unverified -header JWT claims. Detailed authorizer representations remain under review. +header JWT claims. Authorizer mappings are settled in accepted decision 0018. ## Alternatives and tradeoffs @@ -92,7 +92,6 @@ upgrades must preserve the typed API and pass boundary compatibility tests. The user approved the proposed methods and boundary contract. This extends the earlier decision 0002 deferral of typed proxy methods in response to a concrete -consumer requirement; its raw invocation contract remains accepted. Record the -remaining decoding and HTTP translation decisions before implementing behavior -that depends on them. Do not add nonfunctional public method stubs merely to -claim the accepted API is implemented. +consumer requirement; its raw invocation contract remains accepted. The later +HTTP and identity decisions are now implemented behind both complete typed methods. +Tests verify SDK registration and the documented upstream-codec qualifications. diff --git a/docs/decisions/0006-buffered-responses.md b/docs/decisions/0006-buffered-responses.md index 07a9516..48fc01e 100644 --- a/docs/decisions/0006-buffered-responses.md +++ b/docs/decisions/0006-buffered-responses.md @@ -235,9 +235,12 @@ request formats exercise ServeMux path parameters, ServeContent ranges, gzip output and request-body echoing. Invocation-scope tests cover cleanup on write failure, panic, parent cancellation and cleanup failure. -These tests assemble the private primitives inside the existing invocation scope. -They do not publish the pending Lambda invocation methods or establish gateway -identity. Streaming HTTP writer behavior is a separate implementation milestone. +At that implementation stage these tests assembled private primitives. Update +2026-09-16: public Invoke/HandleV1/HandleV2 now use the same scope and establish +opted-in native gateway identity under decision 0018. Public-boundary tests cover +SDK registration, ServeMux path parameters/redirects, ServeContent ranges, cookies, +binary responses, complete raw-envelope limits, multipart cleanup and isolation. +Streaming HTTP writer behavior remains a separate implementation milestone. Validation: focused writer/integration tests, full race suite, vet, formatting and diff checks, and Linux arm64/amd64 builds pass on Go 1.27.1. No live AWS diff --git a/docs/decisions/0012-shared-identity.md b/docs/decisions/0012-shared-identity.md index 05ca65c..00204ab 100644 --- a/docs/decisions/0012-shared-identity.md +++ b/docs/decisions/0012-shared-identity.md @@ -2,13 +2,14 @@ Status: accepted as the starting contract on 2026-09-15; claims foundation implemented under 0014; caller/context and JWT normalization implemented on -2026-09-16. Native gateway producers remain pending their fixture review. +2026-09-16. Native gateway producers and buffered public invocation are now +implemented under accepted decision 0018. This walks the dependencies needed for the first callable buffered adapter. Decision 0013 covers the public errors used at those boundaries. Approval of the implementation order does not approve these new API and trust decisions. -## Evidence and constraints +## Evidence and constraints at proposal time - `adapter.go` records WithGatewayIdentity, but does not yet produce identity. - `decode.go` preserves raw authorizer JSON separately from SDK event types. @@ -361,7 +362,8 @@ The claims constructors, immutable views and resource/error foundation are now implemented under accepted decision 0014. Caller/context construction, JWT normalization and dedicated-scope accounting are now implemented. The user approved IAM caller forms in 0015/0017 and the literal user-path refinement. -Native gateway producers remain pending the fixture/recognition review in 0018. +Native gateway producers are implemented under the approved fixture/recognition +contract in 0018, with raw/typed fidelity qualifications documented separately. Accepted follow-up 0016 adds an online STS-verified IAM proof producer alongside the gateway/custom producers described in I2. Accepted decision 0017 adds diff --git a/docs/decisions/0014-claims-api.md b/docs/decisions/0014-claims-api.md index 8060190..bfc3fd3 100644 --- a/docs/decisions/0014-claims-api.md +++ b/docs/decisions/0014-claims-api.md @@ -236,7 +236,7 @@ status is recorded below; no live AWS resources were invoked. The user approved C1–C5 on 2026-09-16 and authorized implementation, including the narrow json.Number data-type compatibility and gateway qualifications. -The native-producer fixture review remains a separate step before gateway wiring. +The separate native-producer fixture review was later completed in decision 0018. ## Implementation status @@ -249,7 +249,8 @@ and typed SDK number-preserving paths are tested through the actual SDK wrapper. The root adapter records and validates WithIdentityClaimsBudget. A private error translation boundary classifies claim failures and reports identity_claims byte -limits without misreporting depth as bytes. Neither is wired to invocation yet. +limits without misreporting depth as bytes. Both are now wired to raw and typed +invocation through the native gateway producers under accepted decision 0018. Production allocation measurements are recorded in docs/benchmarks.md. Race tests, vet, JSON fuzzing and both Lambda-target builds validate the claims foundation. diff --git a/docs/decisions/0018-gateway-identity-recognition.md b/docs/decisions/0018-gateway-identity-recognition.md index 8cc4c54..72b9a5e 100644 --- a/docs/decisions/0018-gateway-identity-recognition.md +++ b/docs/decisions/0018-gateway-identity-recognition.md @@ -1,7 +1,7 @@ # 0018: Native gateway identity recognition and fixture coverage -Status: proposed on 2026-09-16. The caller/context foundation is implemented; -native gateway extraction and public invocation wiring await this review. +Status: accepted on 2026-09-16. The user approved G1-G4 and authorized native +gateway extraction and buffered invocation wiring. Both are implemented and tested. This is the producer-fixture follow-up explicitly reserved in 0012 and 0014. ## Evidence and existing boundary @@ -143,5 +143,19 @@ default; a future separately authorized capture can improve the coverage matrix. ## Resolution -Pending review of G1-G4. Existing identity constructors and invocation isolation -are complete and tested; this record does not implement gateway extraction. +The user approved G1-G4 on 2026-09-16 and authorized implementation. Existing +identity constructors, native extraction and buffered public invocation are now +implemented. Raw and typed paths share marker discovery and construction policy; +typed events are never serialized to inspect them. Scope preflight prevents large +temporary collections from bypassing the configured budget before NewJWT checks +their combined charge with claims. + +The [fixture inventory](../gateway-fixtures.md) labels documented shapes, SDK +schema evidence, synthetic/adversarial cases and unverified deployed coverage. +Tests exercise all four IAM forms, optional effective-user IDs, source conflicts, +raw numeric fidelity, flattened V2 text, owned inputs and discarded unknown SDK +fields. Actual Lambda SDK wrappers cover raw object and both typed registrations, +including V1 number preservation. Full race tests, vet and Lambda builds pass. +The raw identity-boundary fuzz target completed 482,700 synthetic executions in +an initial ten-second run without failures. Public raw/typed allocation baselines +are recorded in docs/benchmarks.md, including binary and near-limit payloads. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 1726677..00f8217 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -26,4 +26,4 @@ Keep related decisions small enough to review and implement independently. | [0015: IAM caller forms](0015-iam-caller-forms.md) | Accepted | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | | [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture and exported Go helper accepted; protocol/API pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | | [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; isolated SDK probe passes | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | -| [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Proposed | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | +| [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | diff --git a/docs/gateway-fixtures.md b/docs/gateway-fixtures.md new file mode 100644 index 0000000..8dab560 --- /dev/null +++ b/docs/gateway-fixtures.md @@ -0,0 +1,44 @@ +# Gateway identity coverage + +The fixtures in gateway_identity_test.go and adapter_invoke_test.go are synthetic +and independently authored. They are not captured AWS events or copied Beakley +tests. No live AWS service is invoked by the tests. + +| Cases | Evidence | Qualification | +| --- | --- | --- | +| REST anonymous and claims/scopes null placeholders | [REST proxy documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html) | API owner, network, pool and access-key metadata never become a caller | +| HTTP API 1.0 placeholders and envelope | [HTTP payload formats](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) | Authentication combinations use the reviewed shape contract; deployed combinations are unverified | +| V1 IAM and Cognito | REST proxy documentation and [context definitions](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html) | Exact userArn; caller account differs deliberately from API owner; user differs from signing caller | +| V2 JWT | HTTP payload formats | Valid synthetic issuer/subject replace documentation placeholders; raw arrays/numbers do not imply every gateway preserves them | +| V2 IAM | [SDK v1.55.0 schema](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/apigw.go) and [SDK IAM fixture](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/testdata/apigw-v2-request-iam.json) | The SDK fixture has a ten-digit account ID; our authentication fixtures use valid twelve-digit IDs | +| Malformed, ambiguous and unknown assertions | Accepted decisions 0012, 0014 and 0018 | These prove library failure policy, not that AWS generates these events | + +Raw and typed paths share native shape recognition, caller construction and +resource policy. Typed V1 claims are decoded values; typed V2 claims are gateway +text. Array/number fidelity lost by upstream decoding cannot be recovered. The +real Lambda SDK registration tests cover typed V1 WithUseNumber behavior. + +SDK decoding also discards unknown V2 authorizer keys. A raw unknown authorizer +fails as unsupported, while its already-decoded empty SDK description may be +indistinguishable from an empty authorizer. Typed calls cannot promise inspection +of discarded wire data. Use the raw Adapter when complete JSON shape validation +is required. Typed SDK V2 input cannot represent the raw array-claim fixture. + +In V1, IAM lives outside the authorizer object. A separate nonempty unknown +authorizer still fails as unsupported when IAM is present; it is not sibling +metadata within a recognized native authorizer. Unknown metadata within a +recognized native authorizer is ignored as specified in decision 0018. + +The tests exercise gateway identity disabled/enabled, required identity facts, +scope consistency and limits, explicit empty producer objects, native/custom +collisions and no fallback to a bearer header. Public invocation tests cover +SDK object/function dispatch, binary bodies, cookies, application error responses, +request isolation, inherited callers, panic/cancellation and multipart cleanup. + +NewIAM's own contract suite covers all four accepted principal forms and literal +user-path punctuation. Gateway fixtures additionally exercise those forms through +the invocation boundary, including preserved session IDs and optional principal IDs. + +The SDK's upstream decoder owns duplicate/unknown fields and number handling on +typed registration; the caller/runtime also owns the final typed response's +serialized envelope limit. Edge never serializes a typed event to inspect it. diff --git a/docs/plan.md b/docs/plan.md index 25b9c3c..9f74226 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -26,7 +26,7 @@ port, source copy, or compatibility-preserving update of Beakley. accepts explicit credential selection and online STS-proof verification. Include an exported Go helper that generates the signed GetCallerIdentity proof, with consumer documentation and runnable client/server examples. - Detailed protocol and helper/verifier APIs remain a design-review step. + Decision 0017 accepts the detailed protocol and helper/verifier APIs. - Provide first-class consumer request-header processing for header-selected action dispatchers. Preserve custom incoming headers independently of response combination rules. [Decision 0011](decisions/0011-request-header-processing.md) @@ -60,8 +60,9 @@ port, source copy, or compatibility-preserving update of Beakley. The user approved [decision 0001](decisions/0001-gateway-identity.md), superseding the review's initial gateway-default proposal: no gateway-derived identity by default; `WithGatewayIdentity(true)` opts in for both IAM and JWT/Cognito. Local -verification is explicitly composed. Constructor details and precedence between -multiple identity producers remain separate decisions. +verification is explicitly composed. Constructor details and producer conflicts +are settled in decisions 0012–0018; the common HTTP authentication-selector API +remains a later review. ## Planned package boundaries @@ -124,8 +125,8 @@ The accepted [2026-09-14 plan review](reviews/2026-09-14-plan-review.md) require response-contract refinements and completing a runnable adapter, including the minimum gateway identity foundation, before broader authentication work. These enhancements are incorporated into decision 0006. Decision 0010 settles concrete -metadata/header policies; exported error categories and identity APIs remain -separate reviews. +metadata/header policies, with errors and identity subsequently settled in +decisions 0012–0018. The user accepted the qualification to decision 0007: do not freeze arbitrary 64 KiB/1,024-entry metadata limits or the initial ten-field list without stronger @@ -207,10 +208,10 @@ presenting an entire frozen public API at once. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. - [x] Extract shared invocation ownership and implement the private streaming bridge, with handoff, backpressure, cancellation, cleanup, and error tests. -- [ ] Implement the minimum reviewed identity/context and native gateway producers +- [x] Implement the minimum reviewed identity/context and native gateway producers needed to honor WithGatewayIdentity(true) before publishing invocation. -- [ ] Implement raw invocation after the remaining transport contracts are reviewed. -- [ ] Implement the accepted typed entry points after shared HTTP translation +- [x] Implement raw invocation after the remaining transport contracts are reviewed. +- [x] Implement the accepted typed entry points after shared HTTP translation review, documenting upstream codec ownership. - Implement the adapter from scratch using JSON v2 directly. - Test raw invocation through the actual pinned AWS Lambda Go SDK boundary. @@ -227,8 +228,12 @@ presenting an entire frozen public API at once. CI configuration is present for Go 1.27.0/latest 1.27 patch tests, race detection, vet, and Lambda-target builds. It has not run on GitHub yet. Local Go 1.27.1 race -tests, vet, formatting checks, and both target builds pass for the initial -encoding/lifetime implementation. +tests, vet, formatting checks, and both target builds pass for the public buffered +adapter. SDK registration, gateway recognition/fidelity, owned claims, concurrent +identity isolation, multipart cleanup, panic/cancellation and raw envelope limits +are exercised. Runnable examples and initial public allocation benchmarks are +present. HTTP 1.0 authentication coverage is explicitly qualified in +[the fixture inventory](gateway-fixtures.md); no live deployment was exercised. Acceptance: the same application handler behaves as specified across all three event cases; no auth or codec behavior depends on accidental SDK dispatch. @@ -237,8 +242,9 @@ event cases; no auth or codec behavior depends on accidental SDK dispatch. Design graph prepared on 2026-09-15 in [decision 0012](decisions/0012-shared-identity.md), with [public invocation errors in decision 0013](decisions/0013-public-invocation-errors.md). -The user accepted both records as a starting point on 2026-09-15. The explicitly -deferred claim API, resource limits, and producer coverage remain follow-up work. +The user accepted both records as a starting point on 2026-09-15. The deferred +claim API, resource limits and native-producer coverage were subsequently resolved +in decisions 0014–0018. The accepted order is caller/context/error foundations, precise claim API and native-producer fixtures, then raw/typed buffered invocation. Broader local authentication and authorization follow that shared foundation. @@ -250,7 +256,8 @@ The user approved its SDK numeric compatibility and gateway qualifications and authorized implementation. Claims constructors, immutable accessors, strict raw JSON and direct typed construction, byte/depth limits, and sanitized errors are implemented. The adapter budget option is validated and a private invocation-error -translation boundary is tested; gateway extraction and invocation remain unwired. +translation boundary is tested and now wired through gateway extraction and +public invocation under decision 0018. Production allocation measurements supplement the original probe. Contract tests, SDK integration, race detection, vet, JSON fuzzing and Lambda-target builds pass. Caller/context construction, JWT normalization and dedicated-scope accounting @@ -258,7 +265,9 @@ are now implemented after approval of [decision 0015](decisions/0015-iam-caller- and 0017. IAM literal path punctuation follows the user's approved refinement. The private invocation guard rejects inherited callers. Full race tests, vet, formatting and Linux arm64/amd64 builds pass. Native-producer fixture review -remains separate from the foundation approval; see proposed decision 0018. +was approved in decision 0018 on 2026-09-16. Native extraction and buffered public +invocation wiring are now implemented and tested. Custom-authorizer mapping +remains a later explicit design; it is not silently treated as native identity. - Review caller constructors, immutable/owned data, validation, anonymous state, and context-presence semantics. diff --git a/errors.go b/errors.go index 7dd16ff..e5f0061 100644 --- a/errors.go +++ b/errors.go @@ -107,7 +107,7 @@ func limitError(operation, name string, maximum int64, reason error) *Invocation // claimsError translates only package-owned claim failures. It deliberately // rebuilds a safe error tree instead of wrapping the supplied error or its text. -// Gateway preparation will use this boundary when its producers are implemented. +// Gateway preparation uses this boundary for raw and typed claim construction. func claimsError(err error) *InvocationError { if limit, ok := errors.AsType[*identity.ClaimsLimitError](err); ok { return limitError("identity", "identity_claims", limit.Maximum(), errors.Join(ErrIdentity, identity.ErrClaimsLimit)) diff --git a/gateway_identity.go b/gateway_identity.go new file mode 100644 index 0000000..0e6a34c --- /dev/null +++ b/gateway_identity.go @@ -0,0 +1,318 @@ +package edge + +import ( + "bytes" + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + + "github.com/aws/aws-lambda-go/events" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Candidate discovery precedes construction, so malformed competing producers +// cannot disappear through validation or establish precedence by field order. +type gatewayCandidates struct { + iam func() (identity.Caller, error) + jwt func() (identity.Caller, error) + custom bool + unknown bool +} + +func (c gatewayCandidates) caller() (identity.Caller, error) { + count := 0 + for _, present := range []bool{c.iam != nil, c.jwt != nil, c.custom} { + if present { + count++ + } + } + if count > 1 { + return identity.Caller{}, invocationError("identity", ErrIdentity, "conflicting gateway assertions", identity.ErrConflict) + } + if c.custom || c.unknown { + return identity.Caller{}, invocationError("identity", ErrIdentity, "unsupported gateway assertion", errors.ErrUnsupported) + } + var caller identity.Caller + var err error + switch { + case c.iam != nil: + caller, err = c.iam() + case c.jwt != nil: + caller, err = c.jwt() + } + if err != nil { + if errors.Is(err, ErrLimitExceeded) { + return identity.Caller{}, err // A package-owned preflight limit. + } + if errors.Is(err, identity.ErrInvalidCaller) && !errors.Is(err, identity.ErrClaimsLimit) { + return identity.Caller{}, invocationError("identity", ErrIdentity, "invalid gateway caller", identity.ErrInvalidCaller) + } + return identity.Caller{}, claimsError(err) + } + return caller, nil +} + +func gatewayIAM(arn, account, user string) (identity.Caller, error) { + var opts []identity.IAMOption + if account != "" { + opts = append(opts, identity.WithIAMAccountID(account)) + } + if user != "" { + opts = append(opts, identity.WithIAMPrincipalID(user)) + } + return identity.NewIAM(arn, identity.SourceGatewayAssertion, opts...) +} + +func v1IAMCandidate(c *gatewayCandidates, v events.APIGatewayRequestIdentity) { + if v.UserArn != "" || v.User != "" || v.Caller != "" { + c.iam = func() (identity.Caller, error) { return gatewayIAM(v.UserArn, v.AccountID, v.User) } + } +} + +func rawObject(raw jsontext.Value) (map[string]jsontext.Value, error) { + if len(raw) == 0 || raw.Kind() == 'n' { + return nil, nil + } + if raw.Kind() != '{' { + return nil, identity.ErrInvalidCaller + } + var fields map[string]jsontext.Value + if err := json.Unmarshal(raw, &fields); err != nil { + return nil, identity.ErrInvalidCaller + } + return fields, nil +} + +func nonnull(raw jsontext.Value) bool { return len(raw) != 0 && raw.Kind() != 'n' } + +func rawGatewayCaller(event decodedEvent, budget int) (identity.Caller, error) { + fields, err := rawObject(event.authorizer) + if err != nil { + return identity.Caller{}, invocationError("identity", ErrIdentity, "invalid authorizer object", identity.ErrInvalidCaller) + } + var candidates gatewayCandidates + if event.v1 != nil { + v1IAMCandidate(&candidates, event.v1.RequestContext.Identity) + candidates.custom = nonnull(fields["principalId"]) + if nonnull(fields["claims"]) || nonnull(fields["scopes"]) { + candidates.jwt = func() (identity.Caller, error) { return rawGatewayJWT(fields, budget) } + } + for key := range fields { + if key != "claims" && key != "scopes" && key != "principalId" && candidates.jwt == nil { + candidates.unknown = true + } + } + } else { + candidates.custom = nonnull(fields["lambda"]) + if nonnull(fields["jwt"]) { + candidates.jwt = func() (identity.Caller, error) { + jwt, err := rawObject(fields["jwt"]) + if err != nil { + return identity.Caller{}, err + } + return rawGatewayJWT(jwt, budget) + } + } + if nonnull(fields["iam"]) { + candidates.iam = func() (identity.Caller, error) { + var facts struct { + ARN string `json:"userArn"` + Account string `json:"accountId"` + User string `json:"userId"` + } + if fields["iam"].Kind() != '{' { + return identity.Caller{}, identity.ErrInvalidCaller + } + if err := json.Unmarshal(fields["iam"], &facts); err != nil { + return identity.Caller{}, identity.ErrInvalidCaller + } + return gatewayIAM(facts.ARN, facts.Account, facts.User) + } + } + for key := range fields { + if key != "jwt" && key != "iam" && key != "lambda" && candidates.jwt == nil && candidates.iam == nil { + candidates.unknown = true + } + } + } + return candidates.caller() +} + +func rawGatewayJWT(fields map[string]jsontext.Value, budget int) (identity.Caller, error) { + if !nonnull(fields["claims"]) || fields["claims"].Kind() != '{' { + return identity.Caller{}, identity.ErrInvalidCaller + } + claims, err := identity.ParseClaims(fields["claims"], identity.WithClaimsBudget(budget)) + if err != nil { + return identity.Caller{}, err + } + scopes, err := rawGatewayScopes(fields["scopes"], budget) + if err != nil { + return identity.Caller{}, err + } + return identity.NewJWT(claims, identity.SourceGatewayAssertion, identity.WithGatewayScopes(scopes)) +} + +// Bound scope expansion before building the slice. NewJWT then enforces the +// combined claims/scopes charge and owns strings. Raw scopes use no extra wire +// allowance: whitespace does not change their weighted collection cost. +func rawGatewayScopes(raw jsontext.Value, maximum int) ([]string, error) { + if !nonnull(raw) { + return nil, nil + } + if raw.Kind() != '[' { + return nil, identity.ErrInvalidCaller + } + d := jsontext.NewDecoder(bytes.NewReader(raw)) + _, err := d.ReadToken() + if err != nil { + return nil, identity.ErrInvalidCaller + } + remaining, count := maximum-64, 0 + for d.PeekKind() != ']' { + token, err := d.ReadToken() + if err != nil || token.Kind() != '"' { + return nil, identity.ErrInvalidCaller + } + if remaining < 64 || len(token.String()) > remaining-64 { + return nil, gatewayScopeLimit(maximum) + } + remaining -= 64 + len(token.String()) + count++ + } + if remaining < 0 { + return nil, gatewayScopeLimit(maximum) + } + // The envelope decoder already checked syntax, duplicates and trailing data. + scopes := make([]string, 0, count) + if err := json.Unmarshal(raw, &scopes); err != nil { + return nil, identity.ErrInvalidCaller + } + return scopes, nil +} + +func gatewayScopeLimit(maximum int) error { + // Keep byte-limit metadata without constructing identity's opaque error type. + return limitError("identity", "identity_claims", int64(maximum), errors.Join(ErrIdentity, identity.ErrClaimsLimit)) +} + +func typedGatewayCaller(event decodedEvent, budget int) (identity.Caller, error) { + var candidates gatewayCandidates + if event.v1 != nil { + v1IAMCandidate(&candidates, event.v1.RequestContext.Identity) + fields := event.v1.RequestContext.Authorizer + candidates.custom = fields["principalId"] != nil + if fields["claims"] != nil || typedScopesPresent(fields["scopes"]) { + candidates.jwt = func() (identity.Caller, error) { + claims, err := typedGatewayClaims(fields["claims"], budget) + if err != nil { + return identity.Caller{}, err + } + scopes, err := typedGatewayScopes(fields["scopes"], budget) + if err != nil { + return identity.Caller{}, err + } + return identity.NewJWT(claims, identity.SourceGatewayAssertion, identity.WithGatewayScopes(scopes)) + } + } + for key := range fields { + if key != "claims" && key != "scopes" && key != "principalId" && candidates.jwt == nil { + candidates.unknown = true + } + } + } else if auth := event.v2.RequestContext.Authorizer; auth != nil { + candidates.custom = auth.Lambda != nil + if auth.IAM != nil { + candidates.iam = func() (identity.Caller, error) { + return gatewayIAM(auth.IAM.UserARN, auth.IAM.AccountID, auth.IAM.UserID) + } + } + if auth.JWT != nil { + candidates.jwt = func() (identity.Caller, error) { + claims, err := identity.NewTextClaims(auth.JWT.Claims, identity.WithClaimsBudget(budget)) + if err != nil { + return identity.Caller{}, err + } + return identity.NewJWT(claims, identity.SourceGatewayAssertion, identity.WithGatewayScopes(auth.JWT.Scopes)) + } + } + } + return candidates.caller() +} + +func typedGatewayClaims(input any, maximum int) (identity.Claims, error) { + switch input := input.(type) { + case nil: + return identity.Claims{}, identity.ErrInvalidCaller + case map[string]any: + return identity.NewClaims(input, identity.WithClaimsBudget(maximum)) + case map[string]string: + // V1 callers may supply a concrete string map. Bound the temporary + // conversion before allocating it, and retain decoded-value provenance. + remaining := maximum - 64 + if len(input) > max(remaining, 0)/64 { + return identity.Claims{}, gatewayScopeLimit(maximum) + } + for key, value := range input { + if remaining < 64 || len(key) > remaining-64 || len(value) > remaining-64-len(key) { + return identity.Claims{}, gatewayScopeLimit(maximum) + } + remaining -= 64 + len(key) + len(value) + } + values := make(map[string]any, len(input)) + for key, value := range input { + values[key] = value + } + return identity.NewClaims(values, identity.WithClaimsBudget(maximum)) + default: + return identity.Claims{}, identity.ErrInvalidCaller + } +} + +func typedGatewayScopes(input any, maximum int) ([]string, error) { + switch input := input.(type) { + case nil: + return nil, nil + case []string: + return input, nil // NewJWT preflights and owns this collection. + case []any: + if input == nil { + return nil, nil + } + remaining := maximum - 64 + if len(input) > max(remaining, 0)/64 { + return nil, gatewayScopeLimit(maximum) + } + for _, value := range input { + s, ok := value.(string) + if !ok { + return nil, identity.ErrInvalidCaller + } + if remaining < 64 || len(s) > remaining-64 { + return nil, gatewayScopeLimit(maximum) + } + remaining -= 64 + len(s) + } + scopes := make([]string, len(input)) + for i, value := range input { + scopes[i], _ = value.(string) + } + return scopes, nil + default: + return nil, identity.ErrInvalidCaller + } +} + +func typedScopesPresent(input any) bool { + switch scopes := input.(type) { + case nil: + return false + case []string: + return scopes != nil + case []any: + return scopes != nil + default: + return true // A nonnull wrong-shaped field is still a candidate. + } +} diff --git a/gateway_identity_test.go b/gateway_identity_test.go new file mode 100644 index 0000000..6a20511 --- /dev/null +++ b/gateway_identity_test.go @@ -0,0 +1,363 @@ +package edge_test + +import ( + "encoding/json/v2" + "errors" + "net/http" + "slices" + "strings" + "testing" + + "github.com/aws/aws-lambda-go/events" + + edge "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// These are independently authored synthetic shape fixtures, not live captures. +// docs/gateway-fixtures.md records sources and the limits of their evidence. +func gatewayFixture(v2 bool, authorizer, iam string) []byte { + if authorizer == "" { + authorizer = "null" + } + if iam == "" { + iam = `{}` + } + if v2 { + return []byte(`{"version":"2.0","rawPath":"/orders","headers":{"authorization":"Bearer unverified-secret"},"requestContext":{"apiId":"api","accountId":"999999999999","http":{"method":"POST"},"authorizer":` + authorizer + `}}`) + } + return []byte(`{"version":"1.0","path":"/orders","httpMethod":"POST","headers":{"authorization":"Bearer unverified-secret"},"requestContext":{"apiId":"api","accountId":"999999999999","identity":` + iam + `,"authorizer":` + authorizer + `}}`) +} + +func TestGatewayRecognitionMatrix(t *testing.T) { + jwt := `{"claims":{"iss":"issuer","sub":"subject","scope":"read write"},"scopes":["write","read"]}` + iam := `{"userArn":"arn:aws:iam::123456789012:user/Alice","accountId":"123456789012","user":"effective","caller":"signer"}` + iam2 := `{"iam":{"userArn":"arn:aws:sts::123456789012:assumed-role/Worker/session","accountId":"123456789012","userId":"effective","callerId":"signer"}}` + for _, tc := range []struct { + name string + v2 bool + auth, iam string + kind identity.Kind + failure error + rawOnly bool + }{ + {name: "v1_absent"}, + {name: "v1_placeholder", auth: `{"claims":null,"scopes":null}`}, + {name: "v1_metadata", iam: `{"accountId":"123456789012","accessKey":"not-a-proof","cognitoIdentityId":"pool-id","sourceIp":"192.0.2.1"}`}, + {name: "v1_iam", iam: iam, kind: identity.KindIAM}, + {name: "v1_jwt", auth: jwt, kind: identity.KindJWT}, + {name: "v1_iam_missing_arn", iam: `{"caller":"signer"}`, failure: identity.ErrInvalidCaller}, + {name: "v1_account_conflict", iam: `{"userArn":"arn:aws:iam::123456789012:root","accountId":"000000000000"}`, failure: identity.ErrInvalidCaller}, + {name: "v1_empty_claims", auth: `{"claims":{}}`, failure: identity.ErrInvalidCaller}, + {name: "v1_scopes_only", auth: `{"claims":null,"scopes":[]}`, failure: identity.ErrInvalidCaller}, + {name: "v1_scope_conflict", auth: `{"claims":{"iss":"issuer","sub":"s","scope":"read"},"scopes":["write"]}`, failure: identity.ErrInvalidCaller}, + {name: "v1_custom", auth: `{"principalId":"custom"}`, failure: errors.ErrUnsupported}, + {name: "v1_unknown", auth: `{"other":null}`, failure: errors.ErrUnsupported}, + {name: "v1_iam_unknown_authorizer", auth: `{"other":"custom"}`, iam: iam, failure: errors.ErrUnsupported}, + {name: "v1_dual_native", auth: jwt, iam: iam, failure: identity.ErrConflict}, + {name: "v1_native_custom", auth: `{"claims":{},"principalId":"custom"}`, failure: identity.ErrConflict}, + {name: "v1_unknown_metadata", auth: `{"claims":{"iss":"issuer","sub":"s"},"latency":10}`, kind: identity.KindJWT}, + {name: "v2_absent", v2: true}, + {name: "v2_null_members", v2: true, auth: `{"jwt":null,"iam":null,"lambda":null}`}, + {name: "v2_jwt", v2: true, auth: `{"jwt":` + jwt + `}`, kind: identity.KindJWT}, + {name: "v2_iam", v2: true, auth: iam2, kind: identity.KindIAM}, + {name: "v2_empty_jwt", v2: true, auth: `{"jwt":{}}`, failure: identity.ErrInvalidCaller}, + {name: "v2_jwt_null_facts", v2: true, auth: `{"jwt":{"claims":null,"scopes":null}}`, failure: identity.ErrInvalidCaller}, + {name: "v2_empty_iam", v2: true, auth: `{"iam":{}}`, failure: identity.ErrInvalidCaller}, + {name: "v2_empty_custom", v2: true, auth: `{"lambda":{}}`, failure: errors.ErrUnsupported}, + {name: "v2_conflict", v2: true, auth: `{"jwt":{},"iam":{}}`, failure: identity.ErrConflict}, + {name: "v2_custom_conflict", v2: true, auth: `{"iam":{},"lambda":{}}`, failure: identity.ErrConflict}, + {name: "v2_unknown", v2: true, auth: `{"other":"new"}`, failure: errors.ErrUnsupported, rawOnly: true}, + {name: "v2_bad_shape", v2: true, auth: `{"jwt":"invalid"}`, failure: identity.ErrInvalidCaller, rawOnly: true}, + {name: "v2_bad_shape_conflict", v2: true, auth: `{"jwt":"invalid","iam":{}}`, failure: identity.ErrConflict, rawOnly: true}, + } { + t.Run(tc.name, func(t *testing.T) { + for _, typed := range []bool{false, true} { + if typed && tc.rawOnly { + continue + } + for _, enabled := range []bool{false, true} { + var got identity.Caller + ran := false + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + ran = true + got = identity.FromContext(r.Context()) + w.WriteHeader(204) + }), edge.WithGatewayIdentity(enabled)) + if err != nil { + t.Fatal(err) + } + payload := gatewayFixture(tc.v2, tc.auth, tc.iam) + if !typed { + out, invokeErr := a.Invoke(t.Context(), payload) + err = invokeErr + if err != nil && out != nil { + t.Error("failed raw invocation returned a usable result") + } + } else if tc.v2 { + var event events.APIGatewayV2HTTPRequest + if err := json.Unmarshal(payload, &event); err != nil { + t.Fatal(err) + } + out, invokeErr := a.HandleV2(t.Context(), event) + err = invokeErr + if err != nil && out.StatusCode != 0 { + t.Error("failed typed invocation returned a usable result") + } + } else { + var event events.APIGatewayProxyRequest + if err := json.Unmarshal(payload, &event); err != nil { + t.Fatal(err) + } + _, err = a.HandleV1(t.Context(), event) + } + if enabled && tc.failure != nil { + if ran || !errors.Is(err, edge.ErrIdentity) || !errors.Is(err, tc.failure) { + t.Errorf("typed=%t enabled=%t ran=%t err=%v; want %v before handler", typed, enabled, ran, err, tc.failure) + } + continue + } + want := tc.kind + if !enabled { + want = identity.KindAnonymous + } + if err != nil || !ran || got.Kind() != want { + t.Fatalf("typed=%t enabled=%t ran=%t caller=%v err=%v; want kind %d", typed, enabled, ran, got, err, want) + } + if i, ok := got.IAM(); ok { + id, _ := i.PrincipalID() + if i.AccountID() != "123456789012" || id != "effective" { + t.Errorf("IAM used API-owner/signing principal: account=%q id=%q", i.AccountID(), id) + } + } + } + } + }) + } +} + +func TestGatewayFidelityAndOwnership(t *testing.T) { + var caller identity.Caller + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { caller = identity.FromContext(r.Context()) }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + payload := gatewayFixture(true, `{"jwt":{"claims":{"iss":"issuer","sub":"subject","aud":["client"],"cognito:groups":["Ops"],"custom":9007199254740993}}}`, "") + if _, err := a.Invoke(t.Context(), payload); err != nil { + t.Fatal(err) + } + clear(payload) + j, _ := caller.JWT() + number, _ := j.Claims().Lookup("custom") + if exact, ok := number.NumberText(); !ok || exact != "9007199254740993" { + t.Errorf("raw number = %q, %t", exact, ok) + } + if groups, known := j.CognitoGroups(); !known || !slices.Equal(groups, []string{"Ops"}) { + t.Errorf("raw groups = %v, %t", groups, known) + } + event := events.APIGatewayV2HTTPRequest{ + RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{ + APIID: "api", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}, + Authorizer: &events.APIGatewayV2HTTPRequestContextAuthorizerDescription{JWT: &events.APIGatewayV2HTTPRequestContextAuthorizerJWTDescription{ + Claims: map[string]string{"iss": "issuer", "sub": "subject", "aud": "[client]", "cognito:groups": "[Ops]", "custom": "9007199254740993"}, + }}, + }, + } + if _, err := a.HandleV2(t.Context(), event); err != nil { + t.Fatal(err) + } + event.RequestContext.Authorizer.JWT.Claims["sub"] = "changed" + j, _ = caller.JWT() + if sub, _ := j.Subject(); sub != "subject" { + t.Error("typed caller retained mutable SDK claims") + } + if _, known := j.CognitoGroups(); known { + t.Error("flattened groups became known") + } + number, _ = j.Claims().Lookup("custom") + if _, exact := number.NumberText(); exact || number.Representation() != identity.RepresentationGatewayText { + t.Error("typed V2 text became an exact number") + } +} + +func TestGatewayBudgetsAndInvalidScopeTypes(t *testing.T) { + for _, scopes := range []string{`[null]`, `42`, `["read",false]`, `[` + strings.Repeat(`"read",`, 500) + `"read"]`} { + ran := false + a, err := edge.New(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { ran = true }), edge.WithGatewayIdentity(true), edge.WithIdentityClaimsBudget(256)) + if err != nil { + t.Fatal(err) + } + _, err = a.Invoke(t.Context(), gatewayFixture(false, `{"claims":{"iss":"i","sub":"s"},"scopes":`+scopes+`}`, "")) + if ran || !errors.Is(err, edge.ErrIdentity) { + t.Errorf("scopes invoked=%t err=%v", ran, err) + } + if len(scopes) > 256 { + limit, ok := errors.AsType[*edge.InvocationError](err) + if !ok || !errors.Is(err, edge.ErrLimitExceeded) || !errors.Is(err, identity.ErrClaimsLimit) { + t.Fatalf("scope size error = %v", err) + } + name, maximum, present := limit.Limit() + if !present || name != "identity_claims" || maximum != 256 { + t.Errorf("scope limit = %q %d %t", name, maximum, present) + } + } + } +} + +func TestGatewayIAMFormsAndOptionalPrincipal(t *testing.T) { + for _, arn := range []string{ + "arn:aws:iam::123456789012:root", + "arn:aws:iam::123456789012:user/team*?/Alice", + "arn:aws-us-gov:sts::123456789012:assumed-role/Worker/session", + "arn:aws-cn:sts::123456789012:federated-user/Bob", + } { + for _, v2 := range []bool{false, true} { + var got identity.Caller + a, err := edge.New(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { got = identity.FromContext(r.Context()) }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + facts := `{"userArn":"` + arn + `","accountId":"123456789012","caller":"signer","callerId":"signer"}` + auth, iam := "", facts + if v2 { + auth, iam = `{"iam":`+facts+`}`, "" + } + payload := gatewayFixture(v2, auth, iam) + for _, typed := range []bool{false, true} { + if !typed { + _, err = a.Invoke(t.Context(), payload) + } else if v2 { + var event events.APIGatewayV2HTTPRequest + if err := json.Unmarshal(payload, &event); err != nil { + t.Fatal(err) + } + _, err = a.HandleV2(t.Context(), event) + } else { + var event events.APIGatewayProxyRequest + if err := json.Unmarshal(payload, &event); err != nil { + t.Fatal(err) + } + _, err = a.HandleV1(t.Context(), event) + } + if err != nil { + t.Fatal(err) + } + i, ok := got.IAM() + if !ok || i.PrincipalARN() != arn { + t.Errorf("IAM v2=%t typed=%t did not preserve %q", v2, typed, arn) + } + if _, present := i.PrincipalID(); present { + t.Error("signing caller was substituted for absent effective-user ID") + } + } + } + } +} + +type forbiddenEnvelopeMarshal struct{} + +func (forbiddenEnvelopeMarshal) MarshalJSON() ([]byte, error) { + panic("typed invocation must not serialize ignored event metadata") +} + +func TestTypedGatewayDoesNotMarshalAndChecksScopeBudget(t *testing.T) { + a, err := edge.New(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}), edge.WithGatewayIdentity(true), edge.WithIdentityClaimsBudget(350)) + if err != nil { + t.Fatal(err) + } + event := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{ + APIID: "api", Authorizer: map[string]any{ + "claims": map[string]string{"iss": "issuer", "sub": "subject"}, + "scopes": []any{"read"}, "ignored": forbiddenEnvelopeMarshal{}, + }, + }} + if _, err := a.HandleV1(t.Context(), event); err != nil { + t.Fatal(err) + } + event.RequestContext.Authorizer["scopes"] = []any{"read", "write"} + if _, err := a.HandleV1(t.Context(), event); !errors.Is(err, edge.ErrLimitExceeded) || !errors.Is(err, identity.ErrClaimsLimit) { + t.Errorf("combined typed scope budget error = %v", err) + } + event.RequestContext.Authorizer["scopes"] = make([]any, 1000) + if _, err := a.HandleV1(t.Context(), event); !errors.Is(err, edge.ErrLimitExceeded) { + t.Errorf("typed preflight budget error = %v", err) + } +} + +func TestTypedV1NilScopesAreUnavailable(t *testing.T) { + ran := false + a, err := edge.New(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + ran = true + if identity.FromContext(r.Context()).Kind() != identity.KindAnonymous { + t.Error("null placeholder produced identity") + } + }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + for _, scopes := range []any{nil, []any(nil), []string(nil)} { + ran = false + event := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{ + APIID: "api", Authorizer: map[string]any{"claims": nil, "scopes": scopes}, + }} + if _, err := a.HandleV1(t.Context(), event); err != nil || !ran { + t.Errorf("nil scope placeholder %T: ran=%t err=%v", scopes, ran, err) + } + } +} + +func TestTypedV2CannotDetectDiscardedUnknownProducer(t *testing.T) { + ran := false + a, err := edge.New(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + ran = true + if identity.FromContext(r.Context()).Kind() != identity.KindAnonymous { + t.Error("empty typed SDK description unexpectedly created identity") + } + }), edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + payload := gatewayFixture(true, `{"futureProducer":{"subject":"unknown"}}`, "") + if _, err := a.Invoke(t.Context(), payload); !errors.Is(err, errors.ErrUnsupported) || ran { + t.Errorf("raw unknown producer err=%v ran=%t", err, ran) + } + var typed events.APIGatewayV2HTTPRequest + if err := json.Unmarshal(payload, &typed); err != nil { + t.Fatal(err) + } + if _, err := a.HandleV2(t.Context(), typed); err != nil || !ran { + t.Errorf("typed information-loss exception err=%v ran=%t", err, ran) + } +} + +func FuzzGatewayIdentityBoundary(f *testing.F) { + for _, auth := range []string{ + `null`, `{}`, `{"jwt":{}}`, `{"iam":{},"jwt":{}}`, `{"lambda":{}}`, + `{"jwt":{"claims":{"iss":"issuer","sub":"s"},"scopes":[]}}`, + `{"claims":{"iss":"issuer","sub":"s"},"scopes":["read"]}`, + `{"iam":{"userArn":"arn:aws:iam::123456789012:root"}}`, + } { + f.Add(true, auth) + f.Add(false, auth) + } + f.Fuzz(func(t *testing.T, v2 bool, auth string) { + ran := false + a, err := edge.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + ran = true + c := identity.FromContext(r.Context()) + _, iam := c.IAM() + _, jwt := c.JWT() + if iam && jwt || c.Kind() != identity.KindAnonymous && c.Source() != identity.SourceGatewayAssertion { + t.Error("gateway produced an inconsistent caller") + } + w.WriteHeader(204) + }), edge.WithGatewayIdentity(true), edge.WithIdentityClaimsBudget(4096)) + if err != nil { + t.Fatal(err) + } + out, err := a.Invoke(t.Context(), gatewayFixture(v2, auth, "")) + if err != nil && (ran || out != nil) { + t.Fatal("invalid gateway input reached handler or returned usable output") + } + }) +} From 97b9cf7e7acf73e38b39442a175a7c7c0f0d817f Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 17:07:58 +0100 Subject: [PATCH 28/44] feat: add audience-bound IAM proof generation and verification --- README.md | 12 +- docs/decisions/0017-iam-proof-protocol.md | 58 ++- docs/decisions/README.md | 4 +- docs/iamproof.md | 144 ++++++ docs/plan.md | 14 +- go.mod | 15 +- go.sum | 16 + iamproof/doc.go | 17 + iamproof/errors.go | 12 + iamproof/example_test.go | 58 +++ iamproof/fuzz_test.go | 43 ++ iamproof/generate.go | 115 +++++ iamproof/iamproof_test.go | 592 ++++++++++++++++++++++ iamproof/network_test.go | 148 ++++++ iamproof/protocol.go | 187 +++++++ iamproof/verify.go | 164 ++++++ iamproof/xml.go | 93 ++++ 17 files changed, 1667 insertions(+), 25 deletions(-) create mode 100644 docs/iamproof.md create mode 100644 iamproof/doc.go create mode 100644 iamproof/errors.go create mode 100644 iamproof/example_test.go create mode 100644 iamproof/fuzz_test.go create mode 100644 iamproof/generate.go create mode 100644 iamproof/iamproof_test.go create mode 100644 iamproof/network_test.go create mode 100644 iamproof/protocol.go create mode 100644 iamproof/verify.go create mode 100644 iamproof/xml.go diff --git a/README.md b/README.md index f846130..cd44824 100644 --- a/README.md +++ b/README.md @@ -34,11 +34,13 @@ authenticate credentials or authorize application actions. The accepted authentication direction supports IAM credentials or an OAuth bearer token on the same route without built-in `AWS_IAM` authorization. IAM -clients will use an exported Go helper to generate a signed STS GetCallerIdentity -proof; the server verifier submits that client-signed request to AWS and uses the -verified identity for application authorization. The helper, verifier and their -client/server examples are planned. The detailed protocol/API is accepted in -[decision 0017](docs/decisions/0017-iam-proof-protocol.md), but not implemented. +clients use `iamproof.NewGenerator` to generate a signed STS GetCallerIdentity +proof; `iamproof.NewVerifier` submits that client-signed request to AWS and returns +a verified IAM caller. Both are implemented using the latest stable SDK core +v1.47.0 and STS v1.51.0, checked on 2026-09-16. See the +[IAM proof guide](docs/iamproof.md) and [runnable examples](iamproof/example_test.go). +The common HTTP credential selector, OAuth verification and application +authorization remain planned. No live STS interoperability test has run. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle diff --git a/docs/decisions/0017-iam-proof-protocol.md b/docs/decisions/0017-iam-proof-protocol.md index 715b8e6..65c1720 100644 --- a/docs/decisions/0017-iam-proof-protocol.md +++ b/docs/decisions/0017-iam-proof-protocol.md @@ -1,14 +1,16 @@ # 0017: IAM proof protocol and Go helper Status: accepted on 2026-09-16. The user approved P1-P5 and the four caller forms -in decision 0015. The shared caller foundation and an isolated SDK probe are -implemented; there is no production IAM proof generation or authentication yet. +in decision 0015. The production iamproof generator and verifier are implemented +after the shared caller and buffered-adapter foundations, with local validation. +Common HTTP authentication selection and live AWS interoperability remain pending. ## Evidence AWS documentation was checked through public AWS MCP. SDK public API definitions -were checked with the Go documentation skill at the versions below. The existing -identity package implements claims, not Caller or an authentication producer. +were checked with the Go documentation skill at the versions below. At initial +proposal time, identity implemented claims; caller construction, native gateway +production and the IAM proof producer have since been implemented. - [GetCallerIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) returns Account, Arn and UserId for the signing credentials, without requiring @@ -43,9 +45,9 @@ identity package implements claims, not Caller or an authentication producer. Latest stable modules resolved on 2026-09-16: AWS SDK core v1.47.0, STS v1.51.0, and config v1.33.5. Lambda Go remains v1.55.0. The [standalone probe](../probes/iamproof/README.md) -uses the first two; the root dependency list is unchanged. Recheck stable versions -when adding production dependencies. Explicit module versions provide repeatable -builds, not a permanent freeze on an old SDK. +uses the first two. They were rechecked as latest stable when added to production +go.mod on 2026-09-16. Explicit module versions provide repeatable builds, not a +permanent freeze on an old SDK. ## P1. Public package and credential ownership @@ -277,10 +279,11 @@ flowchart LR examples plus authz examples. Keep action extraction separate from proof parsing and require authorization for each selected action. -The probe passes local SDK reconstruction and audience-change checks with race -detection. It does not prove real STS acceptance, deploy API Gateway, implement -the security validation above, or settle HTTP middleware APIs. A separately -authorized integration check can later validate deployed interoperability. +The original probe passes local SDK reconstruction and audience-change checks +with race detection. The production package now implements the security +validation above. Neither establishes real STS acceptance, deploys API Gateway +or settles HTTP middleware APIs. A separately authorized integration check can +later validate deployed interoperability. ## Resolution @@ -288,3 +291,36 @@ The user approved this proposal and decision 0015 on 2026-09-16 and authorized implementation in the recorded order. Complete the shared caller/context and buffered adapter foundation before the IAM helper/verifier and authn middleware. The isolated probe remains the only implemented IAM-proof artifact at approval. + +## Implementation evidence, 2026-09-16 + +The production `iamproof` package implements P1-P5 with direct JSON v2, the SDK +signer, bounded XML verification, caller construction, sanitized errors and +runnable local client/configuration examples. The [consumer guide](../iamproof.md) +records operational constraints and the still-pending common HTTP selector. + +The SDK V2 endpoint result does not expose a partition ID. The implementation +uses the public legacy STS resolver for that metadata and requires agreement +with V2 on endpoint and signing region. This preserves the accepted SDK-metadata +boundary without private imports or a copied partition table. SDK disagreement +fails construction. Fixtures cover all eight partitions present in this release; +they do not assert service availability or account eligibility. + +Two additional rejection codes were checked through AWS MCP before inclusion: +[SignatureDoesNotMatch for GetCallerIdentity](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-troubleshoot.html) +and [ExpiredToken for expired temporary credentials](https://repost.aws/knowledge-center/iam-credentials-token). +Synthetic error fixtures exercise both. Other unrecognized errors remain +unavailable; no status-only invalid-credential classification was introduced. + +Local validation covers official-signer request reconstruction, all four caller +forms, opaque session tokens, wrong audience/scope, tampering, canonical encoding, +malformed/duplicate/null JSON, credential expiry, future/expiry boundaries, +expiry during verification, namespaces/duplicate or contradictory XML identity, +bounded reads and body closure, local TLS/redirect/cookie behavior, caller and +internal deadlines, sanitized failures, provider retrieval and concurrent use. +Initial ten-second fuzz runs completed 464,009 proof-parser and 172,753 XML-parser +executions without failures. These are synthetic/local results, not live STS +acceptance evidence. No automatic retry, result cache or HTTP selector was added. +Full module race tests, vet, formatting checks and Linux arm64/amd64 builds pass +on Go 1.27.1. The existing CI workflow includes this package automatically; no +GitHub CI run has been performed for these changes yet. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 00f8217..e808dfc 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -24,6 +24,6 @@ Keep related decisions small enough to review and implement independently. | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | | [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | | [0015: IAM caller forms](0015-iam-caller-forms.md) | Accepted | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | -| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture and exported Go helper accepted; protocol/API pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | -| [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; isolated SDK probe passes | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | +| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture accepted; IAM proof implemented under 0017; HTTP selector pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | +| [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; generator and verifier implemented with local validation | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | diff --git a/docs/iamproof.md b/docs/iamproof.md new file mode 100644 index 0000000..8b349cc --- /dev/null +++ b/docs/iamproof.md @@ -0,0 +1,144 @@ +# IAM proof generation and verification + +The `iamproof` package is implemented. It authenticates a client using its AWS +credentials without requiring an API Gateway `AWS_IAM` authorizer. The client +generates an audience-bound, presigned GetCallerIdentity proof; the server +reconstructs that signed request and asks regional STS to authenticate it. +The verifier never signs using the Lambda role or other server credentials. + +This is a library-defined proof protocol, not an ordinary SigV4 request to your +API, an EKS token, or an OAuth bearer token. The shared HTTP credential selector +and OAuth verification remain separate work. Merely registering the edge adapter +does not authenticate an Authorization header. + +## Client + +Use the credentials provider from your existing AWS SDK configuration: + +```go +generator, err := iamproof.NewGenerator("eu-west-2", "orders.production", cfg.Credentials) +if err != nil { + return err +} +token, err := generator.Generate(ctx) +if err != nil { + return err +} +req.Header.Set("Authorization", "EdgeIAM "+token.Value()) +req.Header.Set("Action", "ListOrders") +``` + +`cfg` is the application's AWS SDK configuration, and `req` is its outgoing +HTTPS request. The provider owns caching, refresh, AssumeRole and credential +retrieval I/O. Generate retrieves credentials each time and signs locally; it +does not contact STS. Temporary credentials and their session tokens are +supported. Provider AccountID is not used as identity evidence. + +Use a distinct, case-sensitive audience for each application/environment. The +audience is 1-256 visible ASCII bytes without spaces. Clients may reuse a token +before ExpiresAt, which is the earlier of the proof expiry and known credential +expiry. AWS may reject credentials earlier. The runnable +[client example](../iamproof/example_test.go) uses synthetic credentials and +makes no network requests. + +## Server + +Construct one verifier with the same explicit region and audience. Its +`Verify(ctx, tokenValue)` method takes the credential without `EdgeIAM ` and +returns `identity.Caller`. It never installs the caller in a context. Successful +callers have SourceVerifiedIAMProof, preserve the full ARN/session and STS UserId, +and have an account and partition checked against the response and endpoint. +Root, user, assumed-role session and federated-user identities are represented; +representation grants no application permission. + +The authentication layer must select exactly one credential, call exactly one +verifier, install the returned caller using identity.WithCaller, then authorize +the selected application action. Shared middleware and its HTTP challenges will +be reviewed separately. Never fall back to another verifier after a rejection. + +| Outcome | Inspect with errors.Is | Intended HTTP treatment | +| --- | --- | --- | +| Malformed, wrongly bound, expired or AWS-rejected proof | iamproof.ErrInvalidProof | Invalid credentials, 401 | +| Network failure, internal deadline, throttling, unexpected status/XML/identity | iamproof.ErrUnavailable | Dependency unavailable, 503 | +| Caller cancellation/deadline | context.Canceled / context.DeadlineExceeded | Follow request cancellation lifecycle | +| Invalid construction or unconfigured object | iamproof.ErrInvalidConfiguration | Configuration error | +| Client retrieval, expired/incomplete credentials or signing failure | iamproof.ErrCredentialsUnavailable | No proof generated | +| Client proof exceeds 8 KiB | iamproof.ErrProofTooLarge | No proof generated; never truncate | + +The package itself does not write these HTTP responses. Authorization denial +after successful authentication is separate and ordinarily produces 403. +All failures return an anonymous caller or zero token, and errors omit raw +provider/transport/XML diagnostics. Token string/debug formatting is redacted; +Value explicitly exposes the credential. Do not log that value, Authorization +headers, session tokens, signed URLs or transport dumps containing them. + +## Network, freshness and limits + +- Every verification contacts STS, with no application retry or result cache. + Size admission control against the shared STS quota, normally 600 requests per + second per account and Region for credential-based requests. +- The owned HTTP client refuses redirects and has no cookie jar. The default + deadline is five seconds and covers receiving the body. WithTimeout accepts a + positive override; an earlier caller deadline still wins. +- WithTransport supplies trusted deployment infrastructure. Its TLS/proxy + settings, concurrency safety, cancellation handling and internal retries are + the application's responsibility. The default is http.DefaultTransport as + configured when NewVerifier is called. Options apply in order, last value wins. +- Encoded credentials, including `v1.`, are limited to 8,192 bytes. This is not an + AWS session-token maximum, nor a guarantee the entire API request fits gateway + header limits. Incoming oversize proofs are invalid credentials. +- STS success/error bodies are limited to 65,536 bytes, parsed as XML and closed. + Unknown metadata is tolerated; duplicate/missing identity fields, unexpected + namespaces, contradictory accounts/partitions and second documents fail. +- Proofs expire at signing time plus 60 seconds, with no expired-proof grace. + Up to 30 seconds of future clock skew is allowed. Freshness is checked before + and after STS, independently of AWS enforcement of X-Amz-Expires. + +A proof can be replayed to the same audience while fresh; the possible local +acceptance interval under skew is up to 90 seconds. It does not bind the API +method, body or action header and is not a single-use challenge. HTTPS is +required to the application and STS. Neither freshness nor audience validation +alone authenticates a proof: STS must accept its signature. + +## Endpoint and dependency policy + +Production dependencies were checked against the Go module proxy on 2026-09-16: +AWS SDK core v1.47.0 and STS v1.51.0 were the latest stable releases. The root +adapter and identity package do not import the SDK; only iamproof does. + +Both constructors require an explicit standard regional endpoint. They expose +no global, FIPS, dual-stack or arbitrary endpoint switch. Endpoint resolution +uses the SDK V2 resolver. Because that public result omits the partition ID, +the package obtains it from the SDK's public legacy resolver and requires both +resolvers to agree on endpoint and signing region. Disagreement fails +construction. No private SDK APIs or copied partition table are used. + +Local fixtures cover all eight partitions in that SDK release, including +commercial, GovCloud, China, the four ISO partitions and European Sovereign +Cloud. This checks metadata and signing reconstruction, not regional account +eligibility or live service availability. + +## Protocol and verification evidence + +[Decision 0017](decisions/0017-iam-proof-protocol.md) specifies the compact JSON v2 +envelope, canonical unpadded base64url, fixed STS request and accepted error policy. +The library uses direct encoding/json/v2 for generation and parsing, and the +official SDK v4 signer for cryptography. It rejects unknown/duplicate JSON +members, nulls, invalid UTF-8, trailing data, noncanonical base64, wrong audience +or scope, and malformed dates/signatures before network work. + +Tests use synthetic credentials, fake transports, a local TLS server and virtual +time. They verify reconstruction against the official signer, tampering, +freshness boundaries, response validation, bounded reads, cleanup, cancellation, +error sanitization and concurrent use. There is no live STS or deployed Gateway +interoperability result yet; those checks require separate authorization. + +Official sources: + +- [GetCallerIdentity response contract](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html) +- [Regional STS endpoints](https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html) +- [STS credential fields](https://docs.aws.amazon.com/STS/latest/APIReference/API_Credentials.html) +- [STS common errors](https://docs.aws.amazon.com/STS/latest/APIReference/CommonErrors.html) +- [GetCallerIdentity SignatureDoesNotMatch](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-troubleshoot.html) +- [ExpiredToken with temporary credentials](https://repost.aws/knowledge-center/iam-credentials-token) +- [STS quotas](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html) diff --git a/docs/plan.md b/docs/plan.md index 9f74226..006519f 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -84,9 +84,10 @@ The accepted IAM client helper belongs to the module's exported Go API. [Decision 0017](decisions/0017-iam-proof-protocol.md), accepted on 2026-09-16, specifies a focused `iamproof` package, compact versioned proof, signed audience, short freshness policy and -bounded regional STS verification. Its isolated SDK probe passes. The user also +bounded regional STS verification. The production generator/verifier and local +contract tests are now implemented. The user also approved the IAM caller forms in decision 0015. The shared caller/context -foundation is now implemented; gateway producers precede local authentication. +foundation, gateway producers and callable buffered adapter are implemented. ## Milestones and acceptance criteria @@ -285,10 +286,11 @@ claims cannot be silently mixed; unavailable claim fidelity remains explicit. ### 4. Local authentication: Cognito/JWKS and IAM credentials -- Implement the accepted protocol/helper/verifier contract in decision 0017 - after the shared foundation and callable buffered adapter. - The standalone probe uses latest stable SDK core v1.47.0 and STS v1.51.0 as - resolved on 2026-09-16; root production dependencies have not changed. +- [x] Implement the accepted protocol/helper/verifier contract in decision 0017 + after the shared foundation and callable buffered adapter. Production iamproof + uses SDK core v1.47.0 and STS v1.51.0, rechecked as latest stable on 2026-09-16. + [The consumer guide](iamproof.md) covers credential ownership, endpoints, + replay/limits, error handling and the remaining integration qualifications. - Implement the accepted client-helper/server-verifier direction in decisions 0016/0017. Review the common HTTP authentication selector API and challenges; preserve the accepted proof, provenance, binding, replay, endpoint and error rules. diff --git a/go.mod b/go.mod index 3830d49..4a25dfc 100644 --- a/go.mod +++ b/go.mod @@ -2,4 +2,17 @@ module github.com/asteroid-computing/go-lambda-edge go 1.27.0 -require github.com/aws/aws-lambda-go v1.55.0 +require ( + github.com/aws/aws-lambda-go v1.55.0 + github.com/aws/aws-sdk-go-v2 v1.47.0 + github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 +) + +require ( + github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3 // indirect + github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 // indirect + github.com/aws/smithy-go v1.28.1 // indirect +) diff --git a/go.sum b/go.sum index 271d53f..3e12e6a 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,21 @@ github.com/aws/aws-lambda-go v1.55.0 h1:D+AUkk49hTldMnv+Ydt+/jdWVXWPsp8hlwY6Enob2FM= github.com/aws/aws-lambda-go v1.55.0/go.mod h1:V+NzkHNR6vBC8C1PDloqSLE+7jYWFiPvJJFiCiTm8nE= +github.com/aws/aws-sdk-go-v2 v1.47.0 h1:0jsHallhJCeaU0Ko48c/3FK1ctOQ7NpzggxriJOQ8MQ= +github.com/aws/aws-sdk-go-v2 v1.47.0/go.mod h1:bttEH6JqnUL8LepvDVfdrds/fZ5bCIxzpe3abyUrhDU= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3 h1:Hp/VgjP0BysR3OgLlR057Vz2LcbbVnoWeJ+3qWiS/fY= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.3/go.mod h1:nwGV5qw7F1IZPgxCvA/ph8N2TAuz+BkRG/bXn808qMA= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3 h1:MUaM4f+kj1ZIBPZfUS8cxP1GKXXZtHJjAthy93AN7SM= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.3/go.mod h1:6YmVmEVRI5ZZzRjCSsb9SryKH0hAlMRdgA7kG9aDvBU= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 h1:fuSCw4Z2qfRCztMPO3GXJNSiEp6Wee+WOLwrHHUMy9c= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3/go.mod h1:6SxcHheD1pPR5+kWm1wGvjlL/YqUsh267sAfEmN4K7A= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 h1:bAdDl/HkGCcGPoe25ToSHEw23VIxt6CT5fLcg111BKg= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19/go.mod h1:KaUzbLxv4CeSxh6ZCl9B4m7CuFenS8kUEaDs+f/DQr4= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 h1:bON1rJf67TSTDCKg816AAIE4xSTtoo9tl0XRkO72R+I= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3/go.mod h1:c5BBpjJcQXpfeq9iASyVKA3T6vX6B6LEXY4mL/gklDY= +github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 h1:Zpnqa6XtrNzXZnwbdCqHOXpXhMsa01ql/pcRQ1sb4hk= +github.com/aws/aws-sdk-go-v2/service/sts v1.51.0/go.mod h1:/8JRcdTt//hG0Q4BTmGbuOplT7ABe+5rdtqUHqXvYIM= +github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ= +github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= diff --git a/iamproof/doc.go b/iamproof/doc.go new file mode 100644 index 0000000..36497f3 --- /dev/null +++ b/iamproof/doc.go @@ -0,0 +1,17 @@ +// Package iamproof generates short-lived, audience-bound GetCallerIdentity +// proofs and verifies them online with regional AWS STS. It does not require an +// API Gateway authorizer. It neither authorizes application actions nor installs +// callers in a context; compose those responsibilities in the application. +// +// Send Token.Value as the credential in Authorization: EdgeIAM , over +// HTTPS. A proof is reusable within its freshness window and does not sign the +// application request body, method or action. Do not log credentials or signed +// STS requests. Generation uses the caller's AWS credentials; verification never +// signs with the server's credentials. Every verification calls STS, without an +// application retry or result cache. Account for STS quotas when sizing traffic. +// +// The version 1 protocol bounds encoded credentials to 8 KiB and STS responses +// to 64 KiB. Proofs expire 60 seconds after their signing time; verification +// permits up to 30 seconds of future clock skew, with no expired-proof grace. +// These are library policies, not AWS credential-size or expiration guarantees. +package iamproof diff --git a/iamproof/errors.go b/iamproof/errors.go new file mode 100644 index 0000000..a8f79e6 --- /dev/null +++ b/iamproof/errors.go @@ -0,0 +1,12 @@ +package iamproof + +import "errors" + +// Package errors contain no credentials, signed URLs or dependency diagnostics. +var ( + ErrInvalidConfiguration = errors.New("iamproof: invalid configuration") + ErrInvalidProof = errors.New("iamproof: invalid proof") + ErrUnavailable = errors.New("iamproof: verification unavailable") + ErrCredentialsUnavailable = errors.New("iamproof: credentials unavailable") + ErrProofTooLarge = errors.New("iamproof: proof too large") +) diff --git a/iamproof/example_test.go b/iamproof/example_test.go new file mode 100644 index 0000000..78b896f --- /dev/null +++ b/iamproof/example_test.go @@ -0,0 +1,58 @@ +package iamproof_test + +import ( + "context" + "fmt" + "net/http" + + "github.com/aws/aws-sdk-go-v2/aws" + + "github.com/asteroid-computing/go-lambda-edge/iamproof" +) + +func ExampleGenerator_Generate() { + ctx := context.Background() + // Synthetic credentials keep this example entirely local. In an application, + // pass cfg.Credentials from your existing AWS SDK configuration instead. + provider := aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { + return aws.Credentials{ + AccessKeyID: "AKIAIOSFODNN7EXAMPLE", SecretAccessKey: "synthetic-example-only", + }, nil + }) + generator, err := iamproof.NewGenerator("eu-west-2", "orders.production", provider) + if err != nil { + fmt.Println(err) + return + } + token, err := generator.Generate(ctx) + if err != nil { + fmt.Println(err) + return + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://orders.example.com/", nil) + if err != nil { + fmt.Println(err) + return + } + req.Header.Set("Authorization", "EdgeIAM "+token.Value()) + req.Header.Set("Action", "ListOrders") + // Sending req is the application's responsibility. Never log its credential. + fmt.Println(req.Header.Get("Action")) + fmt.Println(token) // Default diagnostics redact the proof. + // Output: + // ListOrders + // iamproof.Token(redacted) +} + +func ExampleNewVerifier() { + verifier, err := iamproof.NewVerifier("eu-west-2", "orders.production") + if err != nil { + fmt.Println(err) + return + } + // Pass verifier.Verify to your authentication layer. It accepts a context + // and the proof credential, returns an identity.Caller, and calls real STS. + // Construction alone makes no network request. + fmt.Println(verifier != nil) + // Output: true +} diff --git a/iamproof/fuzz_test.go b/iamproof/fuzz_test.go new file mode 100644 index 0000000..17fccfb --- /dev/null +++ b/iamproof/fuzz_test.go @@ -0,0 +1,43 @@ +package iamproof_test + +import ( + "errors" + "net/http" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func FuzzProofValidation(f *testing.F) { + f.Add("") + f.Add("v1.e30") + f.Add(generate(f).Value()) + v := verifier(f, roundTripFunc(func(*http.Request) (*http.Response, error) { + return response(403, errorXML("SignatureDoesNotMatch")), nil + })) + f.Fuzz(func(t *testing.T, token string) { + caller, err := v.Verify(t.Context(), token) + if caller.Kind() != identity.KindAnonymous || !errors.Is(err, iamproof.ErrInvalidProof) { + t.Fatalf("unverified input produced caller=%v err=%v", caller, err) + } + }) +} + +func FuzzSTSXML(f *testing.F) { + f.Add(successXML(testARN)) + f.Add(errorXML("InvalidClientTokenId")) + f.Add("") + f.Fuzz(func(t *testing.T, body string) { + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { return response(200, body), nil })) + caller, err := v.Verify(t.Context(), generate(t).Value()) + if err != nil { + assertFailure(t, caller, err, iamproof.ErrUnavailable) + return + } + iam, ok := caller.IAM() + if !ok || iam.Partition() != "aws" || caller.Source() != identity.SourceVerifiedIAMProof { + t.Fatal("XML success violated caller invariants") + } + }) +} diff --git a/iamproof/generate.go b/iamproof/generate.go new file mode 100644 index 0000000..1ade42e --- /dev/null +++ b/iamproof/generate.go @@ -0,0 +1,115 @@ +package iamproof + +import ( + "context" + "encoding/base64" + "encoding/json/v2" + "net/url" + "time" + "unicode/utf8" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/aws/signer/v4" +) + +// Token owns a versioned proof. Its zero value is unusable. Default string and +// debug formatting are redacted; Value explicitly exposes the credential. +type Token struct { + value string + expires time.Time +} + +// Value returns the versioned credential, without the EdgeIAM HTTP scheme. +func (t Token) Value() string { return t.value } + +// ExpiresAt returns the earlier of proof expiry and known credential expiry. +// This is an upper bound, not a guarantee of AWS acceptance until that time. +func (t Token) ExpiresAt() time.Time { return t.expires } + +// String returns a redacted diagnostic description. +func (t Token) String() string { return "iamproof.Token(redacted)" } + +// GoString returns the same redacted description as String. +func (t Token) GoString() string { return t.String() } + +// Generator presigns proofs using its credentials provider. It is safe for +// concurrent use if the provider is. A zero Generator is not configured. +type Generator struct { + cfg configuration + credentials aws.CredentialsProvider +} + +// NewGenerator requires an explicit standard STS region, a case-sensitive +// audience of 1-256 visible ASCII bytes without spaces, and a nonnil provider. +// The provider owns credential caching, refresh and any retrieval I/O. +func NewGenerator(region, audience string, credentials aws.CredentialsProvider) (*Generator, error) { + if isNil(credentials) { + return nil, ErrInvalidConfiguration + } + cfg, err := configure(region, audience) + if err != nil { + return nil, err + } + return &Generator{cfg: cfg, credentials: credentials}, nil +} + +// Generate retrieves fresh credentials and presigns locally, without calling +// STS. Failure returns a zero Token and a sanitized package or caller-context +// error. Oversized proofs fail with ErrProofTooLarge; credentials are not cut. +func (g *Generator) Generate(ctx context.Context) (Token, error) { + if g == nil || g.credentials == nil || ctx == nil { + return Token{}, ErrInvalidConfiguration + } + if err := ctx.Err(); err != nil { + return Token{}, err + } + creds, err := g.credentials.Retrieve(ctx) + if ctx.Err() != nil { + return Token{}, ctx.Err() + } + if err != nil { + return Token{}, ErrCredentialsUnavailable + } + now := time.Now() + if !validAccessKey(creds.AccessKeyID) || creds.SecretAccessKey == "" || !utf8.ValidString(creds.SessionToken) || creds.CanExpire && !now.Before(creds.Expires) { + return Token{}, ErrCredentialsUnavailable + } + // A session longer than the entire decoded budget cannot fit. Check before + // query escaping or JSON encoding can amplify a provider's oversized token. + if len(creds.SessionToken) > base64.RawURLEncoding.DecodedLen(maxProofBytes-3) { + return Token{}, ErrProofTooLarge + } + issued := now.UTC().Truncate(time.Second) + signer := v4.NewSigner(func(o *v4.SignerOptions) { o.DisableHeaderHoisting = true }) + signed, _, err := signer.PresignHTTP(ctx, creds, g.cfg.request(ctx), emptyHash, "sts", g.cfg.region, issued) + if ctx.Err() != nil { + return Token{}, ctx.Err() + } + if err != nil { + return Token{}, ErrCredentialsUnavailable + } + u, err := url.Parse(signed) + if err != nil { + return Token{}, ErrCredentialsUnavailable + } + q := u.Query() + if q.Get("X-Amz-SignedHeaders") != "host;x-edge-iam-audience" { + return Token{}, ErrCredentialsUnavailable + } + wire, err := json.Marshal(envelope{ + Audience: g.cfg.audience, Credential: q.Get("X-Amz-Credential"), + Date: q.Get("X-Amz-Date"), Signature: q.Get("X-Amz-Signature"), + SessionToken: q.Get("X-Amz-Security-Token"), + }) + if err != nil { + return Token{}, ErrCredentialsUnavailable + } + if 3+base64.RawURLEncoding.EncodedLen(len(wire)) > maxProofBytes { + return Token{}, ErrProofTooLarge + } + expires := issued.Add(proofLifetime) + if creds.CanExpire && creds.Expires.Before(expires) { + expires = creds.Expires + } + return Token{value: "v1." + base64.RawURLEncoding.EncodeToString(wire), expires: expires}, nil +} diff --git a/iamproof/iamproof_test.go b/iamproof/iamproof_test.go new file mode 100644 index 0000000..c30e476 --- /dev/null +++ b/iamproof/iamproof_test.go @@ -0,0 +1,592 @@ +package iamproof_test + +import ( + "context" + "encoding/base64" + "encoding/json/v2" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/aws/signer/v4" + + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +const ( + testAudience = "orders.production" + testARN = "arn:aws:sts::123456789012:assumed-role/Operator/alice@example.com" + namespace = "https://sts.amazonaws.com/doc/2011-06-15/" +) + +var testCredentials = aws.Credentials{ + AccessKeyID: "AKIAIOSFODNN7EXAMPLE", SecretAccessKey: "synthetic-secret-not-valid-in-AWS", + SessionToken: "synthetic+/=session-token", +} + +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +func generator(t testing.TB, region string, creds aws.Credentials) *iamproof.Generator { + t.Helper() + g, err := iamproof.NewGenerator(region, testAudience, aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { + return creds, nil + })) + if err != nil { + t.Fatal(err) + } + return g +} + +func generate(t testing.TB) iamproof.Token { + t.Helper() + token, err := generator(t, "eu-west-2", testCredentials).Generate(t.Context()) + if err != nil { + t.Fatal(err) + } + return token +} + +func verifier(t testing.TB, transport http.RoundTripper, opts ...iamproof.VerifierOption) *iamproof.Verifier { + t.Helper() + opts = append([]iamproof.VerifierOption{iamproof.WithTransport(transport)}, opts...) + v, err := iamproof.NewVerifier("eu-west-2", testAudience, opts...) + if err != nil { + t.Fatal(err) + } + return v +} + +func response(status int, body string) *http.Response { + return &http.Response{StatusCode: status, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(body))} +} + +func successXML(arn string) string { + return `` + arn + `opaque:session123456789012synthetic` +} + +func errorXML(code string) string { + return `Sender` + code + `sensitive upstream diagnosticsynthetic` +} + +func unpack(t testing.TB, token string) map[string]string { + t.Helper() + wire, err := base64.RawURLEncoding.DecodeString(strings.TrimPrefix(token, "v1.")) + if err != nil { + t.Fatal(err) + } + var fields map[string]string + if err := json.Unmarshal(wire, &fields); err != nil { + t.Fatal(err) + } + return fields +} + +func pack(t testing.TB, fields map[string]string) string { + t.Helper() + wire, err := json.Marshal(fields) + if err != nil { + t.Fatal(err) + } + return rawToken(string(wire)) +} + +func rawToken(wire string) string { return "v1." + base64.RawURLEncoding.EncodeToString([]byte(wire)) } + +func assertFailure(t testing.TB, caller identity.Caller, err, want error) { + t.Helper() + if !errors.Is(err, want) || caller.Kind() != identity.KindAnonymous { + t.Fatalf("Verify = %v, %v; want anonymous, %v", caller, err, want) + } +} + +// The transport recomputes a signature with synthetic credentials. This checks +// request reconstruction and tampering locally; it is not an AWS acceptance test. +func checkingTransport(t testing.TB, region, host, arn string, creds aws.Credentials) http.RoundTripper { + t.Helper() + return roundTripFunc(func(req *http.Request) (*http.Response, error) { + if req.Method != "GET" || req.URL.Scheme != "https" || req.URL.Host != host || req.URL.Path != "/" || req.Body != nil { + t.Error("verification changed the fixed STS destination, method or body") + } + q := req.URL.Query() + if strings.Contains(req.URL.RawQuery, "+") { + t.Error("reconstructed SigV4 query used form-encoded spaces") + } + if q.Get("Action") != "GetCallerIdentity" || q.Get("Version") != "2011-06-15" || q.Get("X-Amz-Expires") != "60" || q.Get("X-Amz-Algorithm") != "AWS4-HMAC-SHA256" || q.Get("X-Amz-SignedHeaders") != "host;x-edge-iam-audience" { + t.Error("verification changed fixed STS query fields") + } + if req.Header.Get("Authorization") != "" || req.Header.Get("Cookie") != "" || req.Header.Get("X-Edge-IAM-Audience") != testAudience { + t.Error("unexpected STS request headers") + } + when, err := time.Parse("20060102T150405Z", q.Get("X-Amz-Date")) + if err != nil { + t.Error(err) + return response(403, errorXML("SignatureDoesNotMatch")), nil + } + unsigned := req.Clone(req.Context()) + unsigned.URL.RawQuery = url.Values{"Action": {"GetCallerIdentity"}, "Version": {"2011-06-15"}, "X-Amz-Expires": {"60"}}.Encode() + signed, _, err := v4.NewSigner(func(o *v4.SignerOptions) { o.DisableHeaderHoisting = true }).PresignHTTP(req.Context(), creds, unsigned, + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "sts", region, when) + if err != nil { + t.Fatal(err) + } + expected, err := url.Parse(signed) + if err != nil { + t.Fatal(err) + } + if expected.Query().Encode() != q.Encode() { + return response(403, errorXML("SignatureDoesNotMatch")), nil + } + return response(200, successXML(arn)), nil + }) +} + +func TestRoundTripPartitionsAndCallerForms(t *testing.T) { + for _, tc := range []struct{ region, host, partition string }{ + {"eu-west-2", "sts.eu-west-2.amazonaws.com", "aws"}, + {"us-east-1", "sts.us-east-1.amazonaws.com", "aws"}, + {"us-gov-west-1", "sts.us-gov-west-1.amazonaws.com", "aws-us-gov"}, + {"cn-north-1", "sts.cn-north-1.amazonaws.com.cn", "aws-cn"}, + {"us-iso-east-1", "sts.us-iso-east-1.c2s.ic.gov", "aws-iso"}, + {"us-isob-east-1", "sts.us-isob-east-1.sc2s.sgov.gov", "aws-iso-b"}, + {"eu-isoe-west-1", "sts.eu-isoe-west-1.cloud.adc-e.uk", "aws-iso-e"}, + {"us-isof-south-1", "sts.us-isof-south-1.csp.hci.ic.gov", "aws-iso-f"}, + {"eusc-de-east-1", "sts.eusc-de-east-1.amazonaws.eu", "aws-eusc"}, + } { + t.Run(tc.region, func(t *testing.T) { + for _, resource := range []string{"iam::123456789012:root", "iam::123456789012:user/team*?/Alice", "sts::123456789012:assumed-role/Operator/alice@example.com", "sts::123456789012:federated-user/Bob"} { + for _, session := range []string{"", testCredentials.SessionToken, strings.Repeat("+/=a", 1024), "opaque space/&?%é"} { + creds := testCredentials + creds.SessionToken = session + arn := "arn:" + tc.partition + ":" + resource + g := generator(t, tc.region, creds) + token, err := g.Generate(t.Context()) + if err != nil { + t.Fatal(err) + } + v, err := iamproof.NewVerifier(tc.region, testAudience, iamproof.WithTransport(checkingTransport(t, tc.region, tc.host, arn, creds))) + if err != nil { + t.Fatal(err) + } + caller, err := v.Verify(t.Context(), token.Value()) + if err != nil { + t.Fatal(err) + } + iam, ok := caller.IAM() + id, idOK := iam.PrincipalID() + if !ok || iam.PrincipalARN() != arn || iam.AccountID() != "123456789012" || !idOK || id != "opaque:session" || caller.Source() != identity.SourceVerifiedIAMProof { + t.Fatalf("wrong verified caller: %v", caller) + } + if identity.FromContext(t.Context()).Kind() != identity.KindAnonymous { + t.Fatal("Verify installed a caller") + } + } + } + }) + } +} + +func TestMalformedProofsDoNotReachSTS(t *testing.T) { + token := generate(t).Value() + base, err := base64.RawURLEncoding.DecodeString(token[3:]) + if err != nil { + t.Fatal(err) + } + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { + t.Error("malformed proof reached STS") + return response(200, successXML(testARN)), nil + })) + cases := map[string]string{ + "empty": "", "scheme": "EdgeIAM " + token, "version": "v2." + token[3:], + "padding": token + "=", "newline": token[:10] + "\n" + token[10:], + "oversize": strings.Repeat("a", 8193), "null": rawToken("null"), "array": rawToken("[]"), + "trailing": rawToken(string(base) + "{}"), + "duplicate": rawToken(`{"audience":"other",` + string(base[1:])), + "utf8": rawToken(strings.Replace(string(base), testAudience, "\xff", 1)), + "nested": rawToken(`{"audience":{}}`), + } + for _, field := range []string{"audience", "credential", "date", "signature", "sessionToken"} { + fields := unpack(t, token) + fields[field] = "" + cases[field+"_empty"] = pack(t, fields) + wire, err := json.Marshal(unpack(t, token)) + if err != nil { + t.Fatal(err) + } + value, err := json.Marshal(unpack(t, token)[field]) + if err != nil { + t.Fatal(err) + } + cases[field+"_null"] = rawToken(strings.Replace(string(wire), `"`+field+`":`+string(value), `"`+field+`":null`, 1)) + if field != "sessionToken" { + delete(fields, field) + cases[field+"_missing"] = pack(t, fields) + } + } + for name, change := range map[string]func(map[string]string){ + "audience_case": func(p map[string]string) { p["audience"] = "Orders.production" }, + "unknown": func(p map[string]string) { p["endpoint"] = "https://attacker.invalid" }, + "region": func(p map[string]string) { + p["credential"] = strings.Replace(p["credential"], "eu-west-2", "us-east-1", 1) + }, + "scope_date": func(p map[string]string) { + p["credential"] = strings.Replace(p["credential"], p["date"][:8], "20000101", 1) + }, + "service": func(p map[string]string) { p["credential"] = strings.Replace(p["credential"], "/sts/", "/iam/", 1) }, + "terminal": func(p map[string]string) { p["credential"] += "/extra" }, + "key": func(p map[string]string) { + p["credential"] = strings.Replace(p["credential"], testCredentials.AccessKeyID, "bad-key", 1) + }, + "signature": func(p map[string]string) { p["signature"] = strings.Repeat("A", 64) }, + "short_sig": func(p map[string]string) { p["signature"] = "a" }, + "date_format": func(p map[string]string) { p["date"] = "2026-09-16T12:00:00Z" }, + } { + p := unpack(t, token) + change(p) + cases[name] = pack(t, p) + } + for name, token := range cases { + t.Run(name, func(t *testing.T) { + caller, err := v.Verify(t.Context(), token) + assertFailure(t, caller, err, iamproof.ErrInvalidProof) + }) + } +} + +func TestSignatureAndSessionTampering(t *testing.T) { + token := generate(t).Value() + v := verifier(t, checkingTransport(t, "eu-west-2", "sts.eu-west-2.amazonaws.com", testARN, testCredentials)) + for _, field := range []string{"signature", "sessionToken", "credential", "audience"} { + t.Run(field, func(t *testing.T) { + p := unpack(t, token) + switch field { + case "signature": + p[field] = strings.Repeat("0", 64) + case "credential": + p[field] = strings.Replace(p[field], testCredentials.AccessKeyID, "AKIAOTHERKEYEXAMPLE", 1) + default: + p[field] += "altered" + } + caller, err := v.Verify(t.Context(), pack(t, p)) + assertFailure(t, caller, err, iamproof.ErrInvalidProof) + }) + } +} + +func TestAudienceCannotBeRebound(t *testing.T) { + provider := aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { return testCredentials, nil }) + g, err := iamproof.NewGenerator("eu-west-2", "orders.staging", provider) + if err != nil { + t.Fatal(err) + } + token, err := g.Generate(t.Context()) + if err != nil { + t.Fatal(err) + } + p := unpack(t, token.Value()) + p["audience"] = testAudience // Plaintext binding now matches, signature does not. + v := verifier(t, checkingTransport(t, "eu-west-2", "sts.eu-west-2.amazonaws.com", testARN, testCredentials)) + caller, err := v.Verify(t.Context(), pack(t, p)) + assertFailure(t, caller, err, iamproof.ErrInvalidProof) +} + +func TestFreshnessBoundaries(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + base := unpack(t, generate(t).Value()) + var requests int + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { + requests++ + return response(200, successXML(testARN)), nil + })) + for _, tc := range []struct { + age time.Duration + valid bool + }{{-31 * time.Second, false}, {-30 * time.Second, true}, {0, true}, {59 * time.Second, true}, {60 * time.Second, false}} { + when := time.Now().Add(-tc.age).UTC() + base["date"] = when.Format("20060102T150405Z") + base["credential"] = testCredentials.AccessKeyID + "/" + when.Format("20060102") + "/eu-west-2/sts/aws4_request" + before := requests + caller, err := v.Verify(t.Context(), pack(t, base)) + if tc.valid { + if err != nil || caller.Kind() != identity.KindIAM || requests != before+1 { + t.Fatalf("age %v: caller=%v err=%v requests=%d", tc.age, caller, err, requests-before) + } + } else { + assertFailure(t, caller, err, iamproof.ErrInvalidProof) + if requests != before { + t.Error("stale/future proof reached STS") + } + } + } + // A proof that expires while STS is answering cannot establish a caller. + token := generate(t) + time.Sleep(59 * time.Second) + slow := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { + time.Sleep(time.Second) + return response(200, successXML(testARN)), nil + })) + caller, err := slow.Verify(t.Context(), token.Value()) + assertFailure(t, caller, err, iamproof.ErrInvalidProof) + }) +} + +func TestDependencyClassification(t *testing.T) { + for _, tc := range []struct { + name string + status int + body string + want error + }{ + {"expired_token", 403, errorXML("ExpiredToken"), iamproof.ErrInvalidProof}, + {"expired", 400, errorXML("ExpiredTokenException"), iamproof.ErrInvalidProof}, + {"incomplete", 400, errorXML("IncompleteSignature"), iamproof.ErrInvalidProof}, + {"missing", 403, errorXML("MissingAuthenticationToken"), iamproof.ErrInvalidProof}, + {"request_expired", 400, errorXML("RequestExpired"), iamproof.ErrInvalidProof}, + {"unrecognized", 403, errorXML("UnrecognizedClientException"), iamproof.ErrInvalidProof}, + {"invalid_key", 403, errorXML("InvalidClientTokenId"), iamproof.ErrInvalidProof}, + {"signature", 403, errorXML("SignatureDoesNotMatch"), iamproof.ErrInvalidProof}, + {"throttle_400", 400, errorXML("ThrottlingException"), iamproof.ErrUnavailable}, + {"throttle_429", 429, errorXML("Throttling"), iamproof.ErrUnavailable}, + {"unknown", 403, errorXML("AccessDenied"), iamproof.ErrUnavailable}, + {"server", 500, errorXML("InvalidClientTokenId"), iamproof.ErrUnavailable}, + {"unexpected_success", 201, successXML(testARN), iamproof.ErrUnavailable}, + {"redirect", 307, "", iamproof.ErrUnavailable}, + {"html", 403, "failed", iamproof.ErrUnavailable}, + {"empty", 200, "", iamproof.ErrUnavailable}, + {"error_as_success", 200, errorXML("InvalidClientTokenId"), iamproof.ErrUnavailable}, + {"duplicate_code", 403, strings.Replace(errorXML("InvalidClientTokenId"), "", "Throttling", 1), iamproof.ErrUnavailable}, + } { + t.Run(tc.name, func(t *testing.T) { + var count int + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { + count++ + r := response(tc.status, tc.body) + r.Header.Set("Location", "https://attacker.invalid/stolen") + return r, nil + })) + caller, err := v.Verify(t.Context(), generate(t).Value()) + assertFailure(t, caller, err, tc.want) + if count != 1 { + t.Errorf("transport called %d times; want one", count) + } + if strings.Contains(fmt.Sprintf("%+v %#v", err, err), "sensitive") { + t.Error("upstream response leaked in diagnostics") + } + }) + } +} + +func TestXMLIdentityValidation(t *testing.T) { + valid := successXML(testARN) + for name, body := range map[string]string{ + "wrong_namespace": strings.ReplaceAll(valid, namespace, "https://attacker.invalid"), + "no_namespace": strings.ReplaceAll(valid, ` xmlns="`+namespace+`"`, ""), + "foreign_arn": strings.Replace(valid, "", ``, 1), + "duplicate_arn": strings.Replace(valid, "", ""+testARN+"", 1), + "duplicate_result": strings.Replace(valid, "", + "", 1), + "missing_account": strings.Replace(valid, "123456789012", "", 1), + "empty_id": strings.Replace(valid, "opaque:session", "", 1), + "wrong_account": strings.Replace(valid, "123456789012", "000000000000", 1), + "wrong_partition": strings.Replace(valid, "arn:aws:", "arn:aws-cn:", 1), + "bare_role": successXML("arn:aws:iam::123456789012:role/Operator"), + "nested_arn": strings.Replace(valid, testARN, ""+testARN+"", 1), + "second_doc": valid + valid, + "trailing": valid + "garbage", + "dtd": `` + valid, + "broken": valid[:len(valid)-1], + "oversized": valid + strings.Repeat(" ", 65536), + } { + t.Run(name, func(t *testing.T) { + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { return response(200, body), nil })) + caller, err := v.Verify(t.Context(), generate(t).Value()) + assertFailure(t, caller, err, iamproof.ErrUnavailable) + }) + } + for _, body := range []string{ + strings.Replace(valid, "", "text", 1), + strings.Replace(valid, "", "", 1), + `` + testARN + `123456789012opaque:session`, + } { + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { return response(200, body), nil })) + if _, err := v.Verify(t.Context(), generate(t).Value()); err != nil { + t.Fatalf("bounded extra metadata rejected: %v", err) + } + } +} + +func TestCancellationAndTimeout(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + token := generate(t).Value() + v := verifier(t, roundTripFunc(func(r *http.Request) (*http.Response, error) { + <-r.Context().Done() + return nil, errors.New("sensitive signed URL in transport failure") + })) + start := time.Now() + caller, err := v.Verify(t.Context(), token) + assertFailure(t, caller, err, iamproof.ErrUnavailable) + if time.Since(start) != 5*time.Second || errors.Is(err, context.DeadlineExceeded) { + t.Fatal("internal timeout leaked or did not use the five-second default") + } + ctx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + caller, err = v.Verify(ctx, token) + assertFailure(t, caller, err, context.DeadlineExceeded) + ctx, stop := context.WithCancelCause(t.Context()) + stop(errors.New("sensitive cancellation cause")) + caller, err = v.Verify(ctx, token) + assertFailure(t, caller, err, context.Canceled) + if strings.Contains(err.Error(), "sensitive") { + t.Fatal("cancellation cause leaked") + } + }) +} + +func TestGeneratorCredentialFailuresAndRedaction(t *testing.T) { + for _, tc := range []struct { + name string + creds aws.Credentials + err error + want error + }{ + {"provider", testCredentials, errors.New("sensitive provider diagnostic"), iamproof.ErrCredentialsUnavailable}, + {"missing", aws.Credentials{}, nil, iamproof.ErrCredentialsUnavailable}, + {"key", aws.Credentials{AccessKeyID: "bad/key", SecretAccessKey: "secret"}, nil, iamproof.ErrCredentialsUnavailable}, + {"secret", aws.Credentials{AccessKeyID: testCredentials.AccessKeyID}, nil, iamproof.ErrCredentialsUnavailable}, + {"expired", aws.Credentials{AccessKeyID: testCredentials.AccessKeyID, SecretAccessKey: "secret", CanExpire: true, Expires: time.Now().Add(-time.Second)}, nil, iamproof.ErrCredentialsUnavailable}, + {"invalid_utf8", aws.Credentials{AccessKeyID: testCredentials.AccessKeyID, SecretAccessKey: "secret", SessionToken: "\xff"}, nil, iamproof.ErrCredentialsUnavailable}, + {"oversize", aws.Credentials{AccessKeyID: testCredentials.AccessKeyID, SecretAccessKey: "secret", SessionToken: strings.Repeat("+/=a", 1536)}, nil, iamproof.ErrProofTooLarge}, + {"huge", aws.Credentials{AccessKeyID: testCredentials.AccessKeyID, SecretAccessKey: "secret", SessionToken: strings.Repeat("x", 1<<20)}, nil, iamproof.ErrProofTooLarge}, + } { + t.Run(tc.name, func(t *testing.T) { + g, err := iamproof.NewGenerator("eu-west-2", testAudience, aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { return tc.creds, tc.err })) + if err != nil { + t.Fatal(err) + } + token, err := g.Generate(t.Context()) + if !errors.Is(err, tc.want) || token.Value() != "" || !token.ExpiresAt().IsZero() { + t.Fatalf("Generate = %v, %v; want zero token, %v", token, err, tc.want) + } + if strings.Contains(fmt.Sprintf("%+v %#v", err, err), "sensitive") { + t.Error("provider error leaked") + } + }) + } + token := generate(t) + for _, value := range []any{token, &token} { + for _, format := range []string{"%v", "%+v", "%#v", "%s", "%q"} { + diagnostic := fmt.Sprintf(format, value) + if strings.Contains(diagnostic, token.Value()) || strings.Contains(diagnostic, testCredentials.SessionToken) { + t.Fatalf("token leaked through %s", format) + } + } + } +} + +func TestGeneratorRetrievalAndExpiry(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var calls int + creds := testCredentials + creds.AccountID = "000000000000" // Provider metadata is not identity evidence. + creds.CanExpire, creds.Expires = true, time.Now().Add(20*time.Second) + g, err := iamproof.NewGenerator("eu-west-2", testAudience, aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { + calls++ + return creds, nil + })) + if err != nil { + t.Fatal(err) + } + for range 2 { + token, err := g.Generate(t.Context()) + if err != nil || !token.ExpiresAt().Equal(creds.Expires) { + t.Fatalf("credential expiry was not preserved: %v, %v", token.ExpiresAt(), err) + } + } + if calls != 2 { + t.Fatalf("Retrieve called %d times; want 2", calls) + } + time.Sleep(20 * time.Second) + if _, err := g.Generate(t.Context()); !errors.Is(err, iamproof.ErrCredentialsUnavailable) { + t.Fatalf("credentials accepted at their expiry: %v", err) + } + creds.CanExpire = false + token, err := g.Generate(t.Context()) + if err != nil || !token.ExpiresAt().Equal(time.Now().Add(time.Minute)) { + t.Fatalf("unexpected proof expiry: %v, %v", token.ExpiresAt(), err) + } + }) +} + +func TestInvalidConfigurationAndZeroValues(t *testing.T) { + provider := aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { return testCredentials, nil }) + for _, region := range []string{"", "aws-global", "fips-us-east-1", "us-east-1-fips", "EU-west-2", "eu-west-2.attacker.invalid", "eu-west-2/", "eu-west-2:443", strings.Repeat("a", 64) + "-west-1"} { + if g, err := iamproof.NewGenerator(region, testAudience, provider); g != nil || !errors.Is(err, iamproof.ErrInvalidConfiguration) { + t.Fatalf("invalid region %q accepted", region) + } + } + for _, audience := range []string{"", "has space", "a\n", "a\x7f", "café", strings.Repeat("a", 257)} { + if v, err := iamproof.NewVerifier("eu-west-2", audience); v != nil || !errors.Is(err, iamproof.ErrInvalidConfiguration) { + t.Fatalf("invalid audience %q accepted", audience) + } + } + var nilProvider aws.CredentialsProviderFunc + for _, p := range []aws.CredentialsProvider{nil, nilProvider} { + if _, err := iamproof.NewGenerator("eu-west-2", testAudience, p); !errors.Is(err, iamproof.ErrInvalidConfiguration) { + t.Fatal("nil provider accepted") + } + } + var nilTransport roundTripFunc + for _, opt := range []iamproof.VerifierOption{nil, iamproof.WithTransport(nil), iamproof.WithTransport(nilTransport), iamproof.WithTimeout(0), iamproof.WithTimeout(-1)} { + if _, err := iamproof.NewVerifier("eu-west-2", testAudience, opt); !errors.Is(err, iamproof.ErrInvalidConfiguration) { + t.Fatal("invalid option accepted") + } + } + for _, g := range []*iamproof.Generator{nil, new(iamproof.Generator)} { + if token, err := g.Generate(t.Context()); !errors.Is(err, iamproof.ErrInvalidConfiguration) || token.Value() != "" { + t.Fatal("unconfigured generator accepted") + } + } + for _, v := range []*iamproof.Verifier{nil, new(iamproof.Verifier)} { + caller, err := v.Verify(t.Context(), "") + assertFailure(t, caller, err, iamproof.ErrInvalidConfiguration) + } +} + +func TestConcurrentUseAndNoVerificationCache(t *testing.T) { + g := generator(t, "eu-west-2", testCredentials) + var calls atomic.Int64 + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { + calls.Add(1) + return response(200, successXML(testARN)), nil + })) + var group sync.WaitGroup + for range 32 { + group.Go(func() { + token, err := g.Generate(t.Context()) + if err != nil { + t.Error(err) + return + } + for range 2 { + if _, err := v.Verify(t.Context(), token.Value()); err != nil { + t.Error(err) + } + } + }) + } + group.Wait() + if calls.Load() != 64 { + t.Fatalf("STS calls = %d; want 64", calls.Load()) + } +} diff --git a/iamproof/network_test.go b/iamproof/network_test.go new file mode 100644 index 0000000..dde66c7 --- /dev/null +++ b/iamproof/network_test.go @@ -0,0 +1,148 @@ +package iamproof_test + +import ( + "context" + "errors" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/aws/aws-sdk-go-v2/aws" + + "github.com/asteroid-computing/go-lambda-edge/iamproof" +) + +type observedBody struct { + read func([]byte) (int, error) + bytes int + closed bool +} + +func (b *observedBody) Read(p []byte) (int, error) { + n, err := b.read(p) + b.bytes += n + return n, err +} + +func (b *observedBody) Close() error { + b.closed = true + return nil +} + +func TestResponseReadBoundAndCleanup(t *testing.T) { + for _, tc := range []struct { + name string + data string + want error + }{ + {"success", successXML(testARN), nil}, + {"exact_limit", successXML(testARN) + strings.Repeat(" ", 65536-len(successXML(testARN))), nil}, + {"oversized", strings.Repeat(" ", 1<<20), iamproof.ErrUnavailable}, + {"malformed", "broken", iamproof.ErrUnavailable}, + } { + t.Run(tc.name, func(t *testing.T) { + body := &observedBody{read: strings.NewReader(tc.data).Read} + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { + return &http.Response{StatusCode: 200, Body: body}, nil + })) + _, err := v.Verify(t.Context(), generate(t).Value()) + if !errors.Is(err, tc.want) || !body.closed || body.bytes > 65537 { + t.Fatalf("err=%v, closed=%v, bytes=%d; want %v, true, <=65537", err, body.closed, body.bytes, tc.want) + } + }) + } + body := &observedBody{read: func([]byte) (int, error) { return 0, errors.New("sensitive body error") }} + v := verifier(t, roundTripFunc(func(*http.Request) (*http.Response, error) { return &http.Response{StatusCode: 200, Body: body}, nil })) + caller, err := v.Verify(t.Context(), generate(t).Value()) + assertFailure(t, caller, err, iamproof.ErrUnavailable) + if !body.closed || strings.Contains(err.Error(), "sensitive") { + t.Fatal("body read failure was not sanitized and closed") + } +} + +func TestResponseBodyDeadline(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var body *observedBody + v := verifier(t, roundTripFunc(func(r *http.Request) (*http.Response, error) { + body = &observedBody{read: func([]byte) (int, error) { + <-r.Context().Done() + return 0, r.Context().Err() + }} + return &http.Response{StatusCode: 200, Body: body}, nil + }), iamproof.WithTimeout(time.Second)) + start := time.Now() + caller, err := v.Verify(t.Context(), generate(t).Value()) + assertFailure(t, caller, err, iamproof.ErrUnavailable) + if time.Since(start) != time.Second || !body.closed { + t.Fatal("network deadline did not cover the response body") + } + }) +} + +func TestGeneratorCallerCancellation(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + g, err := iamproof.NewGenerator("eu-west-2", testAudience, aws.CredentialsProviderFunc(func(ctx context.Context) (aws.Credentials, error) { + <-ctx.Done() + return aws.Credentials{}, errors.New("sensitive provider context error") + })) + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + token, err := g.Generate(ctx) + if !errors.Is(err, context.DeadlineExceeded) || token.Value() != "" { + t.Fatalf("canceled generation: %v, %v", token, err) + } + }) +} + +func TestHTTPTransportAndRedirect(t *testing.T) { + var calls atomic.Int64 + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if calls.Add(1) == 1 { + if r.Method != "GET" || r.URL.Path != "/" || r.URL.Query().Get("X-Amz-Signature") == "" || r.Header.Get("X-Edge-IAM-Audience") != testAudience { + t.Error("HTTP transport lost signed request fields") + } + w.Header().Set("Set-Cookie", "untrusted=must-not-return") + io.WriteString(w, successXML(testARN)) + return + } + if r.Header.Get("Cookie") != "" { + t.Error("verifier retained STS cookies") + } + w.Header().Set("Location", "https://attacker.invalid/stolen") + w.WriteHeader(http.StatusTemporaryRedirect) + })) + defer server.Close() + local, err := url.Parse(server.URL) + if err != nil { + t.Fatal(err) + } + transport := server.Client().Transport + v := verifier(t, roundTripFunc(func(req *http.Request) (*http.Response, error) { + if req.URL.Host != "sts.eu-west-2.amazonaws.com" { + t.Error("redirect reached the transport") + return nil, errors.New("unexpected destination") + } + // This trusted test transport routes STS to a local TLS fixture only. + localRequest := req.Clone(req.Context()) + localRequest.URL.Host = local.Host + return transport.RoundTrip(localRequest) + })) + token := generate(t) + if _, err := v.Verify(t.Context(), token.Value()); err != nil { + t.Fatal(err) + } + caller, err := v.Verify(t.Context(), token.Value()) + assertFailure(t, caller, err, iamproof.ErrUnavailable) + if calls.Load() != 2 { + t.Fatalf("HTTP calls=%d; want 2", calls.Load()) + } +} diff --git a/iamproof/protocol.go b/iamproof/protocol.go new file mode 100644 index 0000000..8edba31 --- /dev/null +++ b/iamproof/protocol.go @@ -0,0 +1,187 @@ +package iamproof + +import ( + "context" + "encoding/base64" + "encoding/json/jsontext" + "encoding/json/v2" + "net/http" + "net/url" + "reflect" + "regexp" + "strings" + "time" + + "github.com/aws/aws-sdk-go-v2/service/sts" +) + +const ( + maxProofBytes = 8 * 1024 + maxResponseBytes = 64 * 1024 + proofLifetime = time.Minute + futureTolerance = 30 * time.Second + dateLayout = "20060102T150405Z" + audienceHeader = "X-Edge-IAM-Audience" + emptyHash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" +) + +var regionPattern = regexp.MustCompile(`^[a-z]{2,8}(-[a-z]+)+-[0-9]+$`) + +type configuration struct { + region string + audience string + endpoint url.URL + partition string +} + +func configure(region, audience string) (configuration, error) { + if len(region) > 63 || !regionPattern.MatchString(region) || strings.HasPrefix(region, "fips-") || !validAudience(audience) { + return configuration{}, ErrInvalidConfiguration + } + endpoint, err := sts.NewDefaultEndpointResolverV2().ResolveEndpoint(context.Background(), sts.EndpointParameters{ + Region: new(region), UseGlobalEndpoint: new(false), UseFIPS: new(false), UseDualStack: new(false), + }) + if err != nil { + return configuration{}, ErrInvalidConfiguration + } + // The V2 public endpoint does not expose its partition. Obtain that metadata + // from the SDK's public legacy resolver, and require both resolvers to agree + // on the standard endpoint. No private SDK imports or copied partition table. + metadata, err := sts.NewDefaultEndpointResolver().ResolveEndpoint(region, sts.EndpointResolverOptions{}) + if err != nil || metadata.PartitionID == "" || metadata.SigningRegion != region || metadata.URL != endpoint.URI.String() { + return configuration{}, ErrInvalidConfiguration + } + u := endpoint.URI + if u.Scheme != "https" || !strings.HasPrefix(u.Host, "sts."+region+".") || u.User != nil || u.Port() != "" || u.RawQuery != "" || u.Fragment != "" || u.Path != "" && u.Path != "/" { + return configuration{}, ErrInvalidConfiguration + } + u.Path = "/" + return configuration{region: region, audience: audience, endpoint: u, partition: metadata.PartitionID}, nil +} + +func validAudience(audience string) bool { + if len(audience) == 0 || len(audience) > 256 { + return false + } + for i := range len(audience) { + if audience[i] < 0x21 || audience[i] > 0x7e { + return false + } + } + return true +} + +func isNil(value any) bool { + if value == nil { + return true + } + v := reflect.ValueOf(value) + switch v.Kind() { + case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice: + return v.IsNil() + } + return false +} + +type envelope struct { + Audience string `json:"audience"` + Credential string `json:"credential"` + Date string `json:"date"` + Signature string `json:"signature"` + SessionToken string `json:"sessionToken,omitempty"` +} + +func decodeProof(token string, cfg configuration) (envelope, time.Time, error) { + if len(token) > maxProofBytes || !strings.HasPrefix(token, "v1.") { + return envelope{}, time.Time{}, ErrInvalidProof + } + encoded := token[3:] + wire, err := base64.RawURLEncoding.Strict().DecodeString(encoded) + if err != nil || base64.RawURLEncoding.EncodeToString(wire) != encoded { + return envelope{}, time.Time{}, ErrInvalidProof + } + // Inspect raw members so explicit null and absent optional fields differ. + // JSON v2 rejects duplicate names, invalid UTF-8 and trailing data by default. + var members map[string]jsontext.Value + if err := json.Unmarshal(wire, &members); err != nil || len(members) < 4 || len(members) > 5 { + return envelope{}, time.Time{}, ErrInvalidProof + } + var p envelope + for name, raw := range members { + var target *string + switch name { + case "audience": + target = &p.Audience + case "credential": + target = &p.Credential + case "date": + target = &p.Date + case "signature": + target = &p.Signature + case "sessionToken": + target = &p.SessionToken + default: + return envelope{}, time.Time{}, ErrInvalidProof + } + if raw.Kind() != '"' || json.Unmarshal(raw, target) != nil || *target == "" { + return envelope{}, time.Time{}, ErrInvalidProof + } + } + issued, err := time.Parse(dateLayout, p.Date) + if err != nil || issued.Format(dateLayout) != p.Date || p.Audience != cfg.audience || len(p.Signature) != 64 { + return envelope{}, time.Time{}, ErrInvalidProof + } + for i := range len(p.Signature) { + b := p.Signature[i] + if !(b >= '0' && b <= '9' || b >= 'a' && b <= 'f') { + return envelope{}, time.Time{}, ErrInvalidProof + } + } + parts := strings.Split(p.Credential, "/") + if len(parts) != 5 || !validAccessKey(parts[0]) || parts[1] != p.Date[:8] || parts[2] != cfg.region || parts[3] != "sts" || parts[4] != "aws4_request" { + return envelope{}, time.Time{}, ErrInvalidProof + } + return p, issued, nil +} + +func validAccessKey(key string) bool { + if len(key) < 16 || len(key) > 128 { + return false + } + for i := range len(key) { + b := key[i] + if !(b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' || b >= '0' && b <= '9' || b == '_') { + return false + } + } + return true +} + +func fresh(issued, now time.Time) bool { + return !issued.After(now.Add(futureTolerance)) && now.Before(issued.Add(proofLifetime)) +} + +func (cfg configuration) request(ctx context.Context) *http.Request { + u := cfg.endpoint + u.RawQuery = url.Values{"Action": {"GetCallerIdentity"}, "Version": {"2011-06-15"}, "X-Amz-Expires": {"60"}}.Encode() + req := (&http.Request{Method: http.MethodGet, URL: &u, Header: make(http.Header)}).WithContext(ctx) + req.Header.Set(audienceHeader, cfg.audience) + return req +} + +func (p envelope) request(ctx context.Context, cfg configuration) *http.Request { + req := cfg.request(ctx) + query := req.URL.Query() + query.Set("X-Amz-Algorithm", "AWS4-HMAC-SHA256") + query.Set("X-Amz-SignedHeaders", "host;x-edge-iam-audience") + query.Set("X-Amz-Credential", p.Credential) + query.Set("X-Amz-Date", p.Date) + query.Set("X-Amz-Signature", p.Signature) + if p.SessionToken != "" { + query.Set("X-Amz-Security-Token", p.SessionToken) + } + // SigV4 uses RFC 3986 space escaping, not form encoding's '+'. Preserve + // the SDK's encoding even for an unusual opaque session-token value. + req.URL.RawQuery = strings.ReplaceAll(query.Encode(), "+", "%20") + return req +} diff --git a/iamproof/verify.go b/iamproof/verify.go new file mode 100644 index 0000000..72fb2a3 --- /dev/null +++ b/iamproof/verify.go @@ -0,0 +1,164 @@ +package iamproof + +import ( + "context" + "io" + "net/http" + "time" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Verifier authenticates each proof through its configured regional STS +// endpoint. It is safe for concurrent use if its transport is. Its zero value +// is not configured. It never obtains or uses server-side AWS credentials. +type Verifier struct { + cfg configuration + client *http.Client + timeout time.Duration +} + +// VerifierOption configures a verifier. Options apply in order; the last value +// for a setting wins. Nil options and invalid values fail construction. +type VerifierOption func(*verifierConfig) error + +type verifierConfig struct { + transport http.RoundTripper + timeout time.Duration +} + +// WithTransport sets trusted deployment infrastructure for STS requests. +// The transport must be nonnil, concurrency-safe and honor request cancellation. +// Internal transport retries and proxy/TLS policy belong to the transport owner. +func WithTransport(transport http.RoundTripper) VerifierOption { + return func(cfg *verifierConfig) error { + if isNil(transport) { + return ErrInvalidConfiguration + } + cfg.transport = transport + return nil + } +} + +// WithTimeout sets a positive verification network deadline, defaulting to five +// seconds. The caller's earlier context deadline still wins. +func WithTimeout(timeout time.Duration) VerifierOption { + return func(cfg *verifierConfig) error { + if timeout <= 0 { + return ErrInvalidConfiguration + } + cfg.timeout = timeout + return nil + } +} + +// NewVerifier requires the same region and audience as the client generator. +// Its owned HTTP client disables redirects and cookies. The default transport +// is http.DefaultTransport as configured when this constructor is called. +func NewVerifier(region, audience string, opts ...VerifierOption) (*Verifier, error) { + cfg, err := configure(region, audience) + if err != nil { + return nil, err + } + network := verifierConfig{transport: http.DefaultTransport, timeout: 5 * time.Second} + for _, opt := range opts { + if opt == nil { + return nil, ErrInvalidConfiguration + } + if err := opt(&network); err != nil { + return nil, err + } + } + if isNil(network.transport) { + return nil, ErrInvalidConfiguration + } + return &Verifier{ + cfg: cfg, timeout: network.timeout, + client: &http.Client{Transport: network.transport, CheckRedirect: func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + }}, + }, nil +} + +// Verify accepts Token.Value, without an HTTP authentication scheme. It returns +// an IAM caller with SourceVerifiedIAMProof only after STS confirms the proof, +// response invariants hold and the proof remains fresh. It does not install the +// caller in a context. Every failure returns an anonymous zero caller. +// +// Malformed or rejected credentials match ErrInvalidProof. Dependency failures, +// internal timeouts and unexpected responses match ErrUnavailable. Caller +// cancellation/deadline errors are preserved, without unsafe dependency causes. +func (v *Verifier) Verify(ctx context.Context, token string) (identity.Caller, error) { + if v == nil || v.client == nil || ctx == nil { + return identity.Caller{}, ErrInvalidConfiguration + } + if err := ctx.Err(); err != nil { + return identity.Caller{}, err + } + p, issued, err := decodeProof(token, v.cfg) + if err != nil || !fresh(issued, time.Now()) { + return identity.Caller{}, ErrInvalidProof + } + network, cancel := context.WithTimeout(ctx, v.timeout) + defer cancel() + resp, err := v.client.Do(p.request(network, v.cfg)) + if err != nil { + return identity.Caller{}, verificationFailure(ctx) + } + defer resp.Body.Close() + wire, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1)) + if ctx.Err() != nil { + return identity.Caller{}, ctx.Err() + } + if err != nil || network.Err() != nil || len(wire) > maxResponseBytes { + return identity.Caller{}, ErrUnavailable + } + if resp.StatusCode != http.StatusOK { + if resp.StatusCode >= 400 && resp.StatusCode < 500 { + fields, err := parseResponse(wire, false) + if err == nil && invalidCredentialCode(fields["Code"]) { + return identity.Caller{}, ErrInvalidProof + } + } + return identity.Caller{}, ErrUnavailable + } + fields, err := parseResponse(wire, true) + if err != nil { + return identity.Caller{}, ErrUnavailable + } + caller, err := identity.NewIAM(fields["Arn"], identity.SourceVerifiedIAMProof, + identity.WithIAMAccountID(fields["Account"]), identity.WithIAMPrincipalID(fields["UserId"])) + if err != nil { + return identity.Caller{}, ErrUnavailable + } + iam, ok := caller.IAM() + if !ok || iam.Partition() != v.cfg.partition { + return identity.Caller{}, ErrUnavailable + } + if ctx.Err() != nil { + return identity.Caller{}, ctx.Err() + } + if network.Err() != nil { + return identity.Caller{}, ErrUnavailable + } + if !fresh(issued, time.Now()) { + return identity.Caller{}, ErrInvalidProof + } + return caller, nil +} + +func verificationFailure(ctx context.Context) error { + if err := ctx.Err(); err != nil { + return err + } + return ErrUnavailable +} + +func invalidCredentialCode(code string) bool { + switch code { + case "ExpiredToken", "ExpiredTokenException", "IncompleteSignature", "MissingAuthenticationToken", "RequestExpired", + "UnrecognizedClientException", "InvalidClientTokenId", "SignatureDoesNotMatch": + return true + } + return false +} diff --git a/iamproof/xml.go b/iamproof/xml.go new file mode 100644 index 0000000..ee0b6d0 --- /dev/null +++ b/iamproof/xml.go @@ -0,0 +1,93 @@ +package iamproof + +import ( + "bytes" + "encoding/xml" + "io" + "strings" +) + +const stsNamespace = "https://sts.amazonaws.com/doc/2011-06-15/" + +// parseResponse consumes one complete, bounded document. Only direct children +// in the expected namespace can supply identity or error fields; duplicate +// results and fields fail instead of taking the last value as xml.Unmarshal does. +// Unknown metadata is ignored, but still parsed for well-formedness. +func parseResponse(wire []byte, success bool) (map[string]string, error) { + root, result := "GetCallerIdentityResponse", "GetCallerIdentityResult" + fields := map[string]string{"Account": "", "Arn": "", "UserId": ""} + if !success { + root, result = "ErrorResponse", "Error" + fields = map[string]string{"Code": ""} + } + decoder := xml.NewDecoder(bytes.NewReader(wire)) + var stack []xml.Name + seen := make(map[string]bool) + var rootSeen, resultSeen bool + var field string + var value strings.Builder + for { + token, err := decoder.Token() + if err == io.EOF { + break + } + if err != nil { + return nil, ErrUnavailable + } + switch token := token.(type) { + case xml.StartElement: + if field != "" { + return nil, ErrUnavailable // Identity fields contain text, not markup. + } + switch len(stack) { + case 0: + if rootSeen || token.Name != (xml.Name{Space: stsNamespace, Local: root}) { + return nil, ErrUnavailable + } + rootSeen = true + case 1: + if token.Name.Local == result { + if resultSeen || token.Name.Space != stsNamespace { + return nil, ErrUnavailable + } + resultSeen = true + } + case 2: + if stack[1] == (xml.Name{Space: stsNamespace, Local: result}) { + if _, known := fields[token.Name.Local]; known { + if seen[token.Name.Local] || token.Name.Space != stsNamespace { + return nil, ErrUnavailable + } + seen[token.Name.Local] = true + field = token.Name.Local + value.Reset() + } + } + } + stack = append(stack, token.Name) + case xml.EndElement: + if field != "" { + fields[field] = value.String() + field = "" + } + stack = stack[:len(stack)-1] // Decoder.Token checks matching/nesting. + case xml.CharData: + if field != "" { + value.Write(token) + } else if (len(stack) < 2 || len(stack) == 2 && stack[1].Local == result) && len(bytes.TrimSpace(token)) != 0 { + return nil, ErrUnavailable + } + case xml.Directive: + return nil, ErrUnavailable // No DTD or entity declarations in STS XML. + } + } + if !rootSeen || !resultSeen || len(stack) != 0 { + return nil, ErrUnavailable + } + for _, value := range fields { + if value == "" { + return nil, ErrUnavailable + } + } + return fields, nil +} From fcb2a210e303c9ac0480386d22cab91b8335ab81 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 17:45:16 +0100 Subject: [PATCH 29/44] docs: propose HTTP authentication selector and challenges --- .../0016-mixed-iam-oauth-authentication.md | 11 +- .../0019-http-authentication-selector.md | 316 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 6 + 4 files changed, 329 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/0019-http-authentication-selector.md diff --git a/docs/decisions/0016-mixed-iam-oauth-authentication.md b/docs/decisions/0016-mixed-iam-oauth-authentication.md index 3acbf12..ee72613 100644 --- a/docs/decisions/0016-mixed-iam-oauth-authentication.md +++ b/docs/decisions/0016-mixed-iam-oauth-authentication.md @@ -1,8 +1,8 @@ # 0016: IAM credentials or OAuth tokens on the same route Status: architecture and exported Go client helper accepted on 2026-09-16. -Detailed protocol/API design remains pending. No authentication producer or -client helper has been implemented. +The detailed IAM protocol/API was subsequently approved and implemented under +0017. The shared HTTP selector is proposed in 0019 and awaits review. ## Requirement and correction @@ -108,6 +108,7 @@ credential-provider dependencies, versioned wire format and HTTP transport, application binding, expiry/replay policy, endpoint/region policy, provenance and error behavior. Acceptance of the architecture does not freeze these details. -[Decision 0017](0017-iam-proof-protocol.md) now proposes those details, supported -by a standalone SDK reconstruction/size probe. It awaits user review; the probe -is not a production helper or authentication implementation. +[Decision 0017](0017-iam-proof-protocol.md) subsequently settled those details +and its production helper/verifier are implemented with local validation. +[Decision 0019](0019-http-authentication-selector.md) now proposes the remaining +shared HTTP selector API and response contract. No live STS check has run. diff --git a/docs/decisions/0019-http-authentication-selector.md b/docs/decisions/0019-http-authentication-selector.md new file mode 100644 index 0000000..b7a7e1b --- /dev/null +++ b/docs/decisions/0019-http-authentication-selector.md @@ -0,0 +1,316 @@ +# 0019: HTTP authentication selection and challenges + +Status: proposed on 2026-09-16; awaiting user review. The user authorized this +design walk after implementation of iamproof. No authn API or middleware is +implemented by this record. Recommendations A1-A6 are reviewable together. + +## Evidence and existing contracts + +- Decisions 0012/0016/0017 already require one explicit producer, no identity + merging, scheme-based selection, and no fallback after verification fails. + Gateway identity remains disabled by default. +- `identity/context.go` rejects every replacement of an established caller. + `iamproof.Verifier.Verify` returns a caller without installing it, preserves + caller cancellation and distinguishes invalid proof from unavailable STS. +- `action.go` already handles case aliases and duplicate/comma-bearing values. + `http.Header.Get`/`Values` alone cannot inspect noncanonical map aliases; the + installed Go docs explicitly require direct map access for those keys. +- [AWS payload-format documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html), + read through AWS public MCP, says HTTP API payload 2.0 combines duplicate + headers with commas. Payload 1.0 has multivalue headers. This is API Gateway's + event format, not a version of our module. +- [RFC 9110 sections 11.1, 11.3 and 11.6](https://www.rfc-editor.org/rfc/rfc9110.html#section-11) + define case-insensitive schemes, mandatory applicable challenges for 401, + and the WWW-Authenticate list grammar. Decision 0010 and existing response + tests already support separate V1 challenges and correct V2 combination. +- [RFC 6750 sections 2.1 and 3](https://www.rfc-editor.org/rfc/rfc6750.html#section-3) + define Bearer credential syntax and distinguish malformed requests (400), + invalid tokens (401), and inadequate scope (403). Missing credentials and + unsupported authentication methods receive challenges without error details. + A realm parameter provides a concrete challenge without disclosing policy. +- [AWS HTTP API CORS documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-cors.html), + read through AWS public MCP, describes automatic preflight responses when CORS + is configured and separate OPTIONS routing considerations. CORS is deployment + or outer HTTP middleware policy, not an authentication bypass to infer here. +- Go rules favor a small config struct when consumers set a few explicit + dependencies, concrete immutable implementations, and consumer-owned + interfaces. Go 1.27's request-header value-count limit does not replace these + checks for Lambda-created or manually constructed http.Header values. + +## Design graph + +```mermaid +flowchart TD + Request[HTTP request] --> Context[Check cancellation and existing caller] + Context -->|Caller already established| Conflict[Composition failure: 500] + Context -->|Anonymous| Header[Bound and parse one Authorization value] + Header -->|Missing or unsupported scheme| Challenge[401 with configured challenges] + Header -->|Ambiguous or malformed field| BadRequest[400] + Header -->|Too large| Large[431] + Header -->|Bearer| OAuth[Configured Bearer verifier] + Header -->|EdgeIAM| IAM[Configured IAM proof verifier] + OAuth --> Result[Classify error or validate returned caller] + IAM --> Result + Result -->|Invalid credentials| Invalid[401] + Result -->|Dependency failure| Unavailable[503] + Result -->|Broken verifier contract| Internal[500] + Result -->|Valid kind and source| Caller[Return one caller] + Caller --> Install[Middleware installs caller in derived context] + Install --> Action[Consumer selects action and processes extra headers] + Action --> Authz[Authorize that caller for that exact selection] + Authz -->|Allow| Dispatch[Execute selected action] + Authz -->|Deny| Forbidden[403 owned by authorization layer] +``` + +The action order is illustrative, not enforced by authn. A consumer may perform +cheap action parsing earlier, but must authorize and execute the same selection. + +## A1. Public boundary and configuration + +Recommend an opaque immutable `authn.Authenticator`, explicit verifier functions +and one small configuration value. Proposed signatures: + +```go +type VerifyFunc func(context.Context, string) (identity.Caller, error) + +type Config struct { + Bearer VerifyFunc + IAMProof VerifyFunc + Realm string + MaxAuthorizationBytes int +} + +func New(cfg Config) (*Authenticator, error) +func (a *Authenticator) Authenticate(ctx context.Context, headers http.Header) (identity.Caller, error) +func (a *Authenticator) Handler(next http.Handler) (http.Handler, error) + +type Error struct { /* private sanitized representation */ } +func (e *Error) Error() string +func (e *Error) Unwrap() error +func (e *Error) StatusCode() int +func (e *Error) Challenges() []string +``` + +At least one verifier is required; nil disables that scheme. New validates and +copies configuration, performs no I/O and does not call verifiers. Empty Realm +defaults to `edge`; explicit realms contain 1-256 printable ASCII bytes, with +quotes/backslashes correctly escaped in challenges. Realm is a display/protection +label, not the IAM proof audience or a JWT issuer restriction. + +Authenticate reads headers without modifying them, calls at most one verifier, +and returns a caller without installing context or writing a response. Every +failure returns an anonymous zero caller. Callers must not mutate the header map +concurrently. Handler validates its nonnil next handler, including typed nils, +and provides required-authentication middleware using Authenticate and +identity.WithCaller. Nil/zero Authenticators are configuration errors. + +Authenticator is safe for concurrent use when its configured functions are. +Consumers may define a narrow interface with Authenticate for their dispatcher; +there is no need for a provider-owned interface or another header container. +The lower-level method is the customization boundary for application error +envelopes; Challenges returns a copy. No global registry or callback that must +both write a response and correctly stop the handler chain. + +Alternative: variadic verifier-registration options or one generic scheme map. +The struct makes the two supported mechanisms visible, avoids repeated +registration semantics, and does not imply an extensible authentication plugin +framework. The root edge package keeps its existing options API. + +## A2. Strict header selection and a separate field-size limit + +Recommend accepting credentials only from Authorization. Inspect all ASCII +case-equivalent keys, including noncanonical Go map keys. No represented values +(absent/nil/empty slices) means missing; one empty represented value is malformed. +Repeated values fail even if identical. Any comma fails rather than selecting +one credential from Gateway's combined representation. Multiplicity has priority +over scheme/credential interpretation; size exhaustion may fail first. + +Trim outer SP/HTAB, recognize schemes with ASCII case folding, require one or +more literal spaces between scheme and credential, and preserve credential case +and bytes. Internal tabs/whitespace and control/non-ASCII header bytes fail. +Configured schemes require a nonempty token68-shaped credential (ASCII letters, +digits, `-._~+/`, with `=` permitted only at the end); invalid header grammar is +400. Do not assume Bearer is a JWT or decode claims to select a verifier. +EdgeIAM receives its versioned proof unchanged; its verifier owns stricter +JSON/base64/version checks, whose rejection is 401. Likewise, a grammar-valid +Bearer string containing invalid JWT contents reaches the selected verifier and +can yield 401. A structurally valid unsupported or disabled +scheme is a 401, with no verifier call. Malformed generic scheme framing is 400. + +Only header credentials participate in authentication. Do not read request +bodies, cookies or query strings to discover another credential, and never +fall back to them. Their presence does not independently establish identity; +applications must not compose a second producer that authenticates them. + +Recommend MaxAuthorizationBytes default 16 KiB, configurable from 1 byte to +1 MiB; zero selects the default, negatives or larger values fail construction. +Charge represented Authorization value bytes before trimming, allocation or +verification; do not concatenate values. This is a library work bound, not an +AWS limit or a limit on all request headers. It includes scheme and whitespace. +The independent iamproof 8 KiB credential limit remains fixed: a field under +this outer bound can still contain an oversized/invalid IAM proof and yield 401. + +Reasoning: the bound must fit the full approved IAM proof, scheme and framing, +while allowing larger Bearer deployments to opt in explicitly. AWS or the native +HTTP server can impose a smaller total header limit before authn is reached. +Duplicates discarded by an earlier proxy cannot be recovered or detected. + +## A3. Error interoperability without SDK coupling + +Recommend these inspectable authn sentinels: + +`ErrInvalidConfiguration`, `ErrMissingCredentials`, `ErrMalformedCredentials`, +`ErrUnsupportedScheme`, `ErrHeaderTooLarge`, `ErrInvalidCredentials`, +`ErrUnavailable`, and `ErrVerifierContract`. + +Use identity.ErrConflict for an existing caller; do not add a synonymous authn +sentinel. Runtime HTTP failures use the private-state Error wrapper above. +Its error chain contains only the sanitized category. Caller cancellation returns +ctx.Err directly, never context.Cause or a wrapped provider diagnostic. + +Verifier functions must classify a definite credential rejection with +authn.ErrInvalidCredentials. Explicit ErrUnavailable or any otherwise unknown +error becomes sanitized ErrUnavailable. If an error matches both categories, +unavailable wins. A verifier context error while the request context is still +live is also unavailable: it is an internal timeout, not caller cancellation. +Any nonnil error discards a simultaneously returned caller. + +Keep authn free of iamproof and AWS SDK imports. Recommend a documented explicit +adapter for the already implemented verifier: + +```go +verifyIAM := func(ctx context.Context, token string) (identity.Caller, error) { + caller, err := proofVerifier.Verify(ctx, token) + if errors.Is(err, iamproof.ErrInvalidProof) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + return caller, err +} +``` + +The authenticator checks request cancellation independently; other returned +errors are sanitized. Passing proofVerifier.Verify directly is deliberately +insufficient for classifying its package-specific invalid-proof sentinel and +would yield 503 for that error. Documentation and examples must make the adapter +visible rather than suggest direct wiring. The first-party Bearer verifier can +return authn's categories directly when implemented. + +Alternatives: importing iamproof into authn couples JWT-only binaries to STS; +making iamproof depend on HTTP authn reverses its transport-independent boundary; +a new exported errors package or reflective classification protocol adds machinery +for a mapping currently expressible in a few lines. This explicit adapter is the +main ergonomic tradeoff proposed for review. + +## A4. Success invariants and producer composition + +Require Bearer success to return KindJWT/SourceLocallyVerifiedToken and EdgeIAM +success to return KindIAM/SourceVerifiedIAMProof. Anonymous success, wrong kind, +gateway/custom provenance or another invalid result is ErrVerifierContract/500, +not a client credential rejection. Initial Bearer support therefore produces the +existing validated token identity; it does not add a new opaque-token principal +kind. The function must perform actual verification, not merely parse claims. + +Check an already established caller before parsing credentials or running any +verifier. Return identity.ErrConflict/500 even if credentials are absent or the +apparently same identity would result. This is a deployment/composition fault, +not an instruction for the client to obtain a different token. No skip, replace, +merge or fallback option initially. This implements the previously accepted +producer policy; the HTTP classification is new here. + +Recheck caller cancellation after verification and before installing/passing +control. Handler installs once in a derived context, leaves the original request +and headers intact, and preserves unrelated context values. It invokes next +exactly once only after success. Provider panics propagate under normal HTTP/ +edge lifecycle policy; do not disguise application bugs as invalid credentials. + +## A5. Default HTTP outcomes and challenges + +Recommend the following precise HTTP mapping, refining decision 0017's broad +malformed-credential wording at the HTTP framing boundary: + +| Condition | HTTP | WWW-Authenticate | +| --- | --- | --- | +| Missing header or unsupported/disabled scheme | 401 | All configured schemes, no error parameter | +| Repeated/comma-combined/empty field or malformed framing | 400 | Configured challenges; invalid_request only for unambiguously selected configured Bearer | +| Authorization field exceeds its configured bound | 431 | None | +| Selected Bearer credential rejected | 401 | Bearer with invalid_token | +| Selected IAM proof rejected, including invalid/oversized envelope | 401 | EdgeIAM challenge | +| Dependency or unknown verifier error | 503 | None | +| Existing caller or invalid success result | 500 | None | + +For missing credentials with both mechanisms configured, emit two values: + +```http +WWW-Authenticate: Bearer realm="edge" +WWW-Authenticate: EdgeIAM realm="edge" +``` + +Bearer comes first for client compatibility. Only advertise enabled mechanisms. +Use realm on every challenge. Do not invent OAuth error parameters for EdgeIAM, +echo token details, disclose issuer/account restrictions, or claim scopes are +insufficient at the authentication layer. No error_description, error_uri, +automatic Retry-After or WWW-Authenticate on dependency/internal failures. + +The built-in handler writes fixed generic text using net/http conventions, +sets Cache-Control: no-store, and preserves unrelated outer middleware headers +such as CORS. It owns/replaces WWW-Authenticate on its failure responses; stale +challenges from outer middleware must not survive a 500/503/431. It performs no +logging. Error.StatusCode/Challenges support consumers writing their own JSON +response using encoding/json/v2 without exposing credentials or raw causes. + +For caller cancellation/deadline, Authenticate preserves the context error. +Handler never invokes next and attempts a generic 503 without a challenge; an +already disconnected transport may reject the write. This avoids an unwritten +handler accidentally becoming an implicit 200 on an otherwise live native HTTP +connection. Edge's invocation cancellation rules still suppress a usable Lambda +response when its parent context is canceled. Do not promise wire delivery. + +This is an HTTP-level refinement awaiting approval, not a change to the existing +iamproof error contract. Malformed proof/token content still yields 401; malformed +Authorization framing yields 400. Authorization owns permission denial and 403. + +## A6. Routes, action dispatch and streaming + +Recommend required authentication only in the initial Handler. Public routes +omit this middleware; do not add an optional-authentication mode until its +missing-versus-invalid and existing-caller semantics have a concrete consumer. +No automatic OPTIONS bypass: configure CORS/preflight at the gateway or outside +the authentication middleware. Health checks are explicit public routes. + +Keep action extraction, extra headers, registry lookup and authorization in the +consumer's ordinary HTTP pipeline. Demonstrate both identity kinds reaching one +dispatcher, with each selected action independently authorized. IAM verification +does not grant all authenticated AWS accounts access. Do not bind verifier +selection to an action or pass the whole HTTP request to VerifyFunc. + +Authenticate and authorize before any response commitment or streaming flush. +The resulting caller is a snapshot; this middleware does not refresh credentials +or silently reauthorize during a long stream. Applications needing continuous +authorization own a separate explicit policy. + +HTTPS termination is a deployment requirement. Do not require r.TLS to be set: +API Gateway adaptation is not a live TLS server connection, and forwarded scheme +headers are not sufficient evidence to authenticate a caller. + +## Implementation after approval + +1. Implement pure configuration/selection/error handling with no SDK dependency. +2. Implement the low-level method and required-authentication HTTP wrapper. +3. Test header aliases, multiplicity, empty values, comma folding, whitespace, + grammar, byte limits, disabled schemes, exactly-one calls and no fallback. +4. Test error normalization, caller-plus-error, wrong success kind/source, + existing caller conflicts, cancellation, safe diagnostics and context isolation. +5. Test challenges/error responses on native HTTP plus raw/typed Gateway V1/V2; + retain the actual IAM generator/verifier path with local synthetic STS fixtures. + Inputs rejected earlier by transport validation cannot be claimed to reach + the HTTP middleware; retain that distinction in the fixture expectations. +6. Add runnable examples for the IAM error adapter, a consumer-provided Bearer + verifier, custom JSON failures and one action dispatcher. Label synthetic + token verifiers in tests/examples; never present them as production JWT checks. +7. Run race/vet/Lambda builds and targeted parser fuzzing. Separately review + Cognito/JWT/JWKS verification and authz; neither is approved implicitly here. + +## Resolution + +Pending user review of A1-A6, especially the public config/API, explicit IAM error +adapter, 400/401/431 distinction, and default cancellation response behavior. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index e808dfc..89a9555 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -27,3 +27,4 @@ Keep related decisions small enough to review and implement independently. | [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture accepted; IAM proof implemented under 0017; HTTP selector pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | | [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; generator and verifier implemented with local validation | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | +| [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Proposed; awaiting review | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | diff --git a/docs/plan.md b/docs/plan.md index 006519f..7d94c2a 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -286,6 +286,12 @@ claims cannot be silently mixed; unavailable claim fidelity remains explicit. ### 4. Local authentication: Cognito/JWKS and IAM credentials +The next design graph is recorded in proposed +[decision 0019](decisions/0019-http-authentication-selector.md): shared HTTP +selection, explicit verifier dependencies, header limits, context composition, +error interoperability and challenges. It awaits user review before authn +implementation; the IAM proof package remains independently usable. + - [x] Implement the accepted protocol/helper/verifier contract in decision 0017 after the shared foundation and callable buffered adapter. Production iamproof uses SDK core v1.47.0 and STS v1.51.0, rechecked as latest stable on 2026-09-16. From 40b31bac482ba3d8e8549fd941249ce81eabcdcd Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 18:04:30 +0100 Subject: [PATCH 30/44] feat: add explicit HTTP authentication selection and middleware --- README.md | 8 +- authn/authenticate.go | 119 ++++++ authn/authn_test.go | 403 ++++++++++++++++++ authn/doc.go | 16 + authn/errors.go | 85 ++++ authn/example_test.go | 153 +++++++ authn/fuzz_test.go | 38 ++ authn/gateway_test.go | 291 +++++++++++++ authn/handler.go | 72 ++++ authn/header.go | 97 +++++ docs/authn.md | 149 +++++++ .../0016-mixed-iam-oauth-authentication.md | 12 +- docs/decisions/0017-iam-proof-protocol.md | 7 +- .../0019-http-authentication-selector.md | 40 +- docs/decisions/README.md | 4 +- docs/iamproof.md | 9 +- docs/plan.md | 31 +- 17 files changed, 1503 insertions(+), 31 deletions(-) create mode 100644 authn/authenticate.go create mode 100644 authn/authn_test.go create mode 100644 authn/doc.go create mode 100644 authn/errors.go create mode 100644 authn/example_test.go create mode 100644 authn/fuzz_test.go create mode 100644 authn/gateway_test.go create mode 100644 authn/handler.go create mode 100644 authn/header.go create mode 100644 docs/authn.md diff --git a/README.md b/README.md index cd44824..7d0da5c 100644 --- a/README.md +++ b/README.md @@ -39,8 +39,12 @@ proof; `iamproof.NewVerifier` submits that client-signed request to AWS and retu a verified IAM caller. Both are implemented using the latest stable SDK core v1.47.0 and STS v1.51.0, checked on 2026-09-16. See the [IAM proof guide](docs/iamproof.md) and [runnable examples](iamproof/example_test.go). -The common HTTP credential selector, OAuth verification and application -authorization remain planned. No live STS interoperability test has run. +The shared `authn` selector and required-authentication middleware are implemented. +Configure Bearer and/or IAM-proof verifier functions; the +[authentication guide](docs/authn.md) shows the required IAM error mapping and +the custom dispatcher/HTTP response boundaries. The first-party Cognito/JWT/JWKS +verifier and authz package remain planned. Applications must supply actual Bearer +verification and authorize each action. No live STS interoperability test has run. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle diff --git a/authn/authenticate.go b/authn/authenticate.go new file mode 100644 index 0000000..f0782a5 --- /dev/null +++ b/authn/authenticate.go @@ -0,0 +1,119 @@ +package authn + +import ( + "context" + "errors" + "net/http" + "strings" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// VerifyFunc authenticates a credential without its HTTP scheme. It must return +// ErrInvalidCredentials for a definite rejection; unknown errors are treated as +// unavailable. Success requires JWT/SourceLocallyVerifiedToken for Bearer or +// IAM/SourceVerifiedIAMProof for EdgeIAM. Constructors alone do not verify tokens. +type VerifyFunc func(context.Context, string) (identity.Caller, error) + +// Config supplies explicit verifier dependencies and HTTP framing policy. +type Config struct { + // Bearer and IAMProof are independently optional; at least one is required. + // Each function must be safe for concurrent use when sharing an Authenticator. + Bearer VerifyFunc + IAMProof VerifyFunc + // Realm is a challenge label, not a proof audience or issuer restriction. + // Empty defaults to "edge"; explicit values allow 1-256 printable ASCII bytes. + Realm string + // MaxAuthorizationBytes bounds represented Authorization value bytes before + // trimming. Zero defaults to 16 KiB; explicit values must be in 1..1 MiB. + // This neither changes iamproof's 8 KiB proof limit nor bounds all headers. + MaxAuthorizationBytes int +} + +// Authenticator is an immutable selector configured by New. Its zero value is +// unconfigured. Shared use requires concurrency-safe verifier functions. +type Authenticator struct { + cfg Config + bearerChallenge string + iamChallenge string +} + +// New validates and copies cfg without I/O or invoking a verifier. It returns a +// nil Authenticator and ErrInvalidConfiguration for invalid configuration. +func New(cfg Config) (*Authenticator, error) { + if cfg.Bearer == nil && cfg.IAMProof == nil || cfg.MaxAuthorizationBytes < 0 || cfg.MaxAuthorizationBytes > 1<<20 { + return nil, ErrInvalidConfiguration + } + if cfg.MaxAuthorizationBytes == 0 { + cfg.MaxAuthorizationBytes = 16 * 1024 + } + if cfg.Realm == "" { + cfg.Realm = "edge" + } + if len(cfg.Realm) > 256 { + return nil, ErrInvalidConfiguration + } + for i := range len(cfg.Realm) { + if cfg.Realm[i] < 0x20 || cfg.Realm[i] > 0x7e { + return nil, ErrInvalidConfiguration + } + } + quoted := `"` + strings.ReplaceAll(strings.ReplaceAll(cfg.Realm, `\`, `\\`), `"`, `\"`) + `"` + a := &Authenticator{cfg: cfg} + if cfg.Bearer != nil { + a.bearerChallenge = "Bearer realm=" + quoted + } + if cfg.IAMProof != nil { + a.iamChallenge = "EdgeIAM realm=" + quoted + } + return a, nil +} + +// Authenticate selects at most one verifier and returns a verified caller. It +// neither modifies headers nor installs context. Callers must not mutate headers +// concurrently. Every failure returns an anonymous zero caller. +// +// An established caller fails with identity.ErrConflict before credential work. +// Runtime failures are *Error with sanitized categories; caller cancellation +// returns ctx.Err directly. Nil contexts and nil/zero Authenticators return +// ErrInvalidConfiguration. No provider diagnostic or context cause is exposed. +func (a *Authenticator) Authenticate(ctx context.Context, headers http.Header) (identity.Caller, error) { + if a == nil || a.cfg.MaxAuthorizationBytes == 0 || ctx == nil { + return identity.Caller{}, ErrInvalidConfiguration + } + if err := ctx.Err(); err != nil { + return identity.Caller{}, err + } + if identity.FromContext(ctx).Kind() != identity.KindAnonymous { + return identity.Caller{}, a.failure(identity.ErrConflict, http.StatusInternalServerError, noScheme) + } + selected, token, err := a.selectCredential(headers) + if err != nil { + return identity.Caller{}, err + } + verify := a.cfg.Bearer + if selected == iamProof { + verify = a.cfg.IAMProof + } + if err := ctx.Err(); err != nil { + return identity.Caller{}, err + } + caller, err := verify(ctx, token) + if ctx.Err() != nil { + return identity.Caller{}, ctx.Err() + } + if err != nil { + if errors.Is(err, ErrUnavailable) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) || !errors.Is(err, ErrInvalidCredentials) { + return identity.Caller{}, a.failure(ErrUnavailable, http.StatusServiceUnavailable, noScheme) + } + return identity.Caller{}, a.failure(ErrInvalidCredentials, http.StatusUnauthorized, selected) + } + valid := caller.Kind() == identity.KindJWT && caller.Source() == identity.SourceLocallyVerifiedToken + if selected == iamProof { + valid = caller.Kind() == identity.KindIAM && caller.Source() == identity.SourceVerifiedIAMProof + } + if !valid { + return identity.Caller{}, a.failure(ErrVerifierContract, http.StatusInternalServerError, noScheme) + } + return caller, nil +} diff --git a/authn/authn_test.go b/authn/authn_test.go new file mode 100644 index 0000000..22f63f7 --- /dev/null +++ b/authn/authn_test.go @@ -0,0 +1,403 @@ +package authn_test + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Synthetic callers stand in for already verified results. They do not prove +// that a token was authenticated; actual verification belongs to each producer. +func jwtCaller(t testing.TB, source identity.Source) identity.Caller { + t.Helper() + claims, err := identity.NewClaims(map[string]any{"iss": "https://issuer.example", "sub": "alice", "scope": "orders.read"}) + if err != nil { + t.Fatal(err) + } + caller, err := identity.NewJWT(claims, source) + if err != nil { + t.Fatal(err) + } + return caller +} + +func iamCaller(t testing.TB, source identity.Source) identity.Caller { + t.Helper() + caller, err := identity.NewIAM("arn:aws:iam::123456789012:user/Alice", source) + if err != nil { + t.Fatal(err) + } + return caller +} + +func authenticator(t testing.TB, cfg authn.Config) *authn.Authenticator { + t.Helper() + a, err := authn.New(cfg) + if err != nil { + t.Fatal(err) + } + return a +} + +func handler(t testing.TB, a *authn.Authenticator, next http.Handler) http.Handler { + t.Helper() + h, err := a.Handler(next) + if err != nil { + t.Fatal(err) + } + return h +} + +func failure(t testing.TB, caller identity.Caller, err, category error, status int, challenges ...string) { + t.Helper() + if caller.Kind() != identity.KindAnonymous || !errors.Is(err, category) { + t.Fatalf("Authenticate = %v, %v; want anonymous, %v", caller, err, category) + } + e, ok := errors.AsType[*authn.Error](err) + if !ok { + t.Fatalf("error type = %T; want *authn.Error", err) + } + if e.StatusCode() != status || !slices.Equal(e.Challenges(), challenges) { + t.Errorf("failure = %d %q; want %d %q", e.StatusCode(), e.Challenges(), status, challenges) + } +} + +func TestHeaderSelection(t *testing.T) { + jwt, iam := jwtCaller(t, identity.SourceLocallyVerifiedToken), iamCaller(t, identity.SourceVerifiedIAMProof) + const bearerChallenge = `Bearer realm="edge"` + const iamChallenge = `EdgeIAM realm="edge"` + for _, tc := range []struct { + name string + headers http.Header + kind identity.Kind + token string + category error + status int + challenges []string + }{ + {name: "bearer", headers: http.Header{"Authorization": {"Bearer Exact.Case+/_~=="}}, kind: identity.KindJWT, token: "Exact.Case+/_~=="}, + {name: "iam", headers: http.Header{"authorization": {"eDgEiAm v1.Abc_-"}}, kind: identity.KindIAM, token: "v1.Abc_-"}, + {name: "whitespace", headers: http.Header{"AUTHORIZATION": {" \tbEaReR Token\t "}}, kind: identity.KindJWT, token: "Token"}, + {name: "empty_alias", headers: http.Header{"Authorization": nil, "authorization": {"Bearer token"}}, kind: identity.KindJWT, token: "token"}, + {name: "missing", category: authn.ErrMissingCredentials, status: 401, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "no_values", headers: http.Header{"Authorization": {}, "authorization": nil}, category: authn.ErrMissingCredentials, status: 401, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "empty", headers: http.Header{"Authorization": {""}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "repeated", headers: http.Header{"Authorization": {"Bearer token", "Bearer token"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "aliases", headers: http.Header{"Authorization": {"Bearer token"}, "authorization": {"EdgeIAM token"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "comma", headers: http.Header{"Authorization": {"Bearer token,EdgeIAM token"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "unsupported", headers: http.Header{"Authorization": {"Basic abc="}}, category: authn.ErrUnsupportedScheme, status: 401, challenges: []string{bearerChallenge, iamChallenge}}, + {name: "missing_token", headers: http.Header{"Authorization": {"Bearer "}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "tab_separator", headers: http.Header{"Authorization": {"Bearer\ttoken"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "internal_space", headers: http.Header{"Authorization": {"Bearer one two"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "bad_padding", headers: http.Header{"Authorization": {"Bearer a=b"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "padding_only", headers: http.Header{"Authorization": {"Bearer ==="}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "punctuation", headers: http.Header{"Authorization": {"Bearer a!b"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "control", headers: http.Header{"Authorization": {"Bearer token\r\n"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "unicode", headers: http.Header{"Authorization": {"Bearer café"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge + `, error="invalid_request"`, iamChallenge}}, + {name: "scheme_punctuation", headers: http.Header{"Authorization": {"Bearer: token"}}, category: authn.ErrMalformedCredentials, status: 400, challenges: []string{bearerChallenge, iamChallenge}}, + } { + t.Run(tc.name, func(t *testing.T) { + var bearerCalls, iamCalls int + var received string + a := authenticator(t, authn.Config{ + Bearer: func(_ context.Context, token string) (identity.Caller, error) { + bearerCalls++ + received = token + return jwt, nil + }, + IAMProof: func(_ context.Context, token string) (identity.Caller, error) { + iamCalls++ + received = token + return iam, nil + }, + }) + caller, err := a.Authenticate(t.Context(), tc.headers) + if tc.category != nil { + failure(t, caller, err, tc.category, tc.status, tc.challenges...) + if bearerCalls+iamCalls != 0 { + t.Error("invalid header reached a verifier") + } + return + } + if err != nil || caller.Kind() != tc.kind || received != tc.token || bearerCalls+iamCalls != 1 || tc.kind == identity.KindJWT && bearerCalls != 1 || tc.kind == identity.KindIAM && iamCalls != 1 { + t.Errorf("Authenticate = %v, %v; token=%q calls=%d/%d", caller, err, received, bearerCalls, iamCalls) + } + if identity.FromContext(t.Context()).Kind() != identity.KindAnonymous { + t.Fatal("Authenticate installed a caller") + } + }) + } +} + +func TestLimitsAndDisabledSchemes(t *testing.T) { + verify := func(context.Context, string) (identity.Caller, error) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + a := authenticator(t, authn.Config{Bearer: verify, MaxAuthorizationBytes: 8}) + caller, err := a.Authenticate(t.Context(), http.Header{"Authorization": {"Bearer x"}}) + failure(t, caller, err, authn.ErrInvalidCredentials, 401, `Bearer realm="edge", error="invalid_token"`) + for _, headers := range []http.Header{{"Authorization": {"Bearer xx"}}, {"Authorization": {" Bearer x "}}, {"Authorization": {"Bearer x"}, "authorization": {"x"}}} { + caller, err := a.Authenticate(t.Context(), headers) + failure(t, caller, err, authn.ErrHeaderTooLarge, 431) + } + for _, tc := range []struct { + cfg authn.Config + value string + challenge string + }{ + {cfg: authn.Config{Bearer: verify}, value: "EdgeIAM proof", challenge: `Bearer realm="edge"`}, + {cfg: authn.Config{IAMProof: verify}, value: "Bearer token", challenge: `EdgeIAM realm="edge"`}, + } { + caller, err := authenticator(t, tc.cfg).Authenticate(t.Context(), http.Header{"Authorization": {tc.value}}) + failure(t, caller, err, authn.ErrUnsupportedScheme, 401, tc.challenge) + } + for _, size := range []int{16384, 16385} { + a := authenticator(t, authn.Config{Bearer: verify}) + caller, err := a.Authenticate(t.Context(), http.Header{"Authorization": {"Bearer " + strings.Repeat("x", size-7)}}) + if size == 16384 { + failure(t, caller, err, authn.ErrInvalidCredentials, 401, `Bearer realm="edge", error="invalid_token"`) + } else { + failure(t, caller, err, authn.ErrHeaderTooLarge, 431) + } + } +} + +func TestVerifierErrorsAndResults(t *testing.T) { + jwt := jwtCaller(t, identity.SourceLocallyVerifiedToken) + iam := iamCaller(t, identity.SourceVerifiedIAMProof) + secret := errors.New("secret token and provider URL") + for _, tc := range []struct { + name string + caller identity.Caller + err error + category error + status int + }{ + {name: "invalid", err: fmt.Errorf("secret: %w", authn.ErrInvalidCredentials), category: authn.ErrInvalidCredentials, status: 401}, + {name: "unavailable", err: authn.ErrUnavailable, category: authn.ErrUnavailable, status: 503}, + {name: "both", err: errors.Join(authn.ErrInvalidCredentials, authn.ErrUnavailable), category: authn.ErrUnavailable, status: 503}, + {name: "unknown", err: secret, category: authn.ErrUnavailable, status: 503}, + {name: "internal_cancel", err: context.Canceled, category: authn.ErrUnavailable, status: 503}, + {name: "internal_deadline", err: context.DeadlineExceeded, category: authn.ErrUnavailable, status: 503}, + {name: "error_with_caller", caller: jwt, err: secret, category: authn.ErrUnavailable, status: 503}, + {name: "anonymous_success", category: authn.ErrVerifierContract, status: 500}, + {name: "wrong_kind", caller: iam, category: authn.ErrVerifierContract, status: 500}, + {name: "gateway_jwt", caller: jwtCaller(t, identity.SourceGatewayAssertion), category: authn.ErrVerifierContract, status: 500}, + {name: "custom_jwt", caller: jwtCaller(t, identity.SourceCustomAssertion), category: authn.ErrVerifierContract, status: 500}, + } { + t.Run(tc.name, func(t *testing.T) { + var count int + a := authenticator(t, authn.Config{ + Bearer: func(context.Context, string) (identity.Caller, error) { count++; return tc.caller, tc.err }, + IAMProof: func(context.Context, string) (identity.Caller, error) { + t.Error("fallback verifier called") + return iam, nil + }, + }) + caller, err := a.Authenticate(t.Context(), http.Header{"Authorization": {"Bearer secret"}}) + var challenges []string + if tc.status == 401 { + challenges = []string{`Bearer realm="edge", error="invalid_token"`} + } + failure(t, caller, err, tc.category, tc.status, challenges...) + if count != 1 || errors.Is(err, secret) || strings.Contains(fmt.Sprintf("%v %+v %#v", err, err, err), "secret") { + t.Error("wrong verifier count or unsafe diagnostic/error chain") + } + }) + } + for _, caller := range []identity.Caller{jwt, iamCaller(t, identity.SourceGatewayAssertion), iamCaller(t, identity.SourceCustomAssertion)} { + a := authenticator(t, authn.Config{IAMProof: func(context.Context, string) (identity.Caller, error) { return caller, nil }}) + got, err := a.Authenticate(t.Context(), http.Header{"Authorization": {"EdgeIAM proof"}}) + failure(t, got, err, authn.ErrVerifierContract, 500) + } +} + +func TestContextIsolationAndConflict(t *testing.T) { + jwt := jwtCaller(t, identity.SourceLocallyVerifiedToken) + var calls int + a := authenticator(t, authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { calls++; return jwt, nil }}) + ctx, err := identity.WithCaller(t.Context(), jwt) + if err != nil { + t.Fatal(err) + } + caller, err := a.Authenticate(ctx, nil) + failure(t, caller, err, identity.ErrConflict, 500) + if calls != 0 { + t.Fatal("existing caller reached verifier") + } + type key struct{} + r := httptest.NewRequestWithContext(context.WithValue(t.Context(), key{}, "keep"), "GET", "/", nil) + r.Header.Set("Authorization", "Bearer token") + r.Header.Set("Action", "orders.read") + var nextCalls int + h := handler(t, a, http.HandlerFunc(func(w http.ResponseWriter, next *http.Request) { + nextCalls++ + if identity.FromContext(next.Context()).Kind() != identity.KindJWT || next.Context().Value(key{}) != "keep" || next.Header.Get("Action") != "orders.read" { + t.Error("middleware lost caller, unrelated context or action") + } + w.WriteHeader(204) + })) + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != 204 || nextCalls != 1 || identity.FromContext(r.Context()).Kind() != identity.KindAnonymous || r.Header.Get("Authorization") != "Bearer token" { + t.Error("middleware modified original request or failed to call next once") + } +} + +func TestCancellation(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + jwt := jwtCaller(t, identity.SourceLocallyVerifiedToken) + for _, before := range []bool{false, true} { + ctx, cancel := context.WithCancelCause(t.Context()) + if before { + cancel(errors.New("secret context cause")) + } + var calls int + a := authenticator(t, authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { + calls++ + cancel(errors.New("secret context cause")) + return jwt, nil + }}) + caller, err := a.Authenticate(ctx, http.Header{"Authorization": {"Bearer token"}}) + if err != context.Canceled || caller.Kind() != identity.KindAnonymous || before && calls != 0 { + t.Fatalf("canceled Authenticate = %v, %v; calls=%d", caller, err, calls) + } + w := httptest.NewRecorder() + handler(t, a, http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Error("canceled request ran next") })).ServeHTTP(w, httptest.NewRequestWithContext(ctx, "GET", "/", nil)) + if w.Code != 503 || w.Header().Get("WWW-Authenticate") != "" { + t.Error("canceled middleware did not produce generic 503") + } + } + ctx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + a := authenticator(t, authn.Config{Bearer: func(ctx context.Context, _ string) (identity.Caller, error) { <-ctx.Done(); return jwt, nil }}) + caller, err := a.Authenticate(ctx, http.Header{"Authorization": {"Bearer token"}}) + if err != context.DeadlineExceeded || caller.Kind() != identity.KindAnonymous { + t.Fatalf("deadline Authenticate = %v, %v", caller, err) + } + }) +} + +func TestFailureResponseOwnership(t *testing.T) { + for _, providerErr := range []error{authn.ErrInvalidCredentials, authn.ErrUnavailable, nil} { + a := authenticator(t, authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { return identity.Caller{}, providerErr }}) + h := handler(t, a, http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Error("failure ran next") })) + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("Authorization", "Bearer secret") + w := httptest.NewRecorder() + w.Header()["www-authenticate"] = []string{"stale secret"} + w.Header()["WWW-Authenticate"] = []string{"another stale challenge"} + w.Header()["cache-control"] = []string{"public"} + w.Header().Set("Access-Control-Allow-Origin", "https://app.example") + h.ServeHTTP(w, r) + if strings.Contains(w.Body.String(), "secret") || w.Header().Get("Cache-Control") != "no-store" || w.Header().Get("Access-Control-Allow-Origin") != "https://app.example" { + t.Error("failure exposed credentials or lost response header ownership") + } + for name, values := range w.Header() { + if strings.EqualFold(name, "WWW-Authenticate") && (providerErr != authn.ErrInvalidCredentials || !slices.Equal(values, []string{`Bearer realm="edge", error="invalid_token"`})) { + t.Errorf("unexpected challenges: %q", values) + } + } + } +} + +func TestConfigurationAndChallengeOwnership(t *testing.T) { + verify := func(context.Context, string) (identity.Caller, error) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + for _, cfg := range []authn.Config{{}, {Bearer: verify, Realm: "bad\r\n"}, {Bearer: verify, Realm: "café"}, {Bearer: verify, Realm: strings.Repeat("a", 257)}, {Bearer: verify, MaxAuthorizationBytes: -1}, {Bearer: verify, MaxAuthorizationBytes: 1<<20 + 1}} { + if a, err := authn.New(cfg); a != nil || !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Fatalf("invalid configuration accepted: %v", err) + } + } + cfg := authn.Config{Bearer: verify, Realm: `orders "prod"\west`} + a := authenticator(t, cfg) + cfg.Bearer, cfg.Realm = nil, "changed" + _, err := a.Authenticate(t.Context(), nil) + e, ok := errors.AsType[*authn.Error](err) + if !ok || !slices.Equal(e.Challenges(), []string{`Bearer realm="orders \"prod\"\\west"`}) { + t.Fatalf("realm not correctly escaped/snapshotted: %v", err) + } + challenges := e.Challenges() + challenges[0] = "changed" + if e.Challenges()[0] == "changed" { + t.Fatal("Challenges exposed mutable backing data") + } + for _, a := range []*authn.Authenticator{nil, new(authn.Authenticator)} { + if _, err := a.Authenticate(t.Context(), nil); !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Fatal("unconfigured authenticator accepted") + } + if _, err := a.Handler(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})); !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Fatal("unconfigured Handler accepted") + } + } + if _, err := a.Authenticate(nil, nil); !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Fatal("nil context accepted") + } + var nilFunc http.HandlerFunc + var nilPointer *http.ServeMux + for _, next := range []http.Handler{nil, nilFunc, nilPointer} { + if _, err := a.Handler(next); !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Fatal("nil next handler accepted") + } + } +} + +func TestConcurrentRequests(t *testing.T) { + jwt, iam := jwtCaller(t, identity.SourceLocallyVerifiedToken), iamCaller(t, identity.SourceVerifiedIAMProof) + a := authenticator(t, authn.Config{ + Bearer: func(context.Context, string) (identity.Caller, error) { return jwt, nil }, + IAMProof: func(context.Context, string) (identity.Caller, error) { return iam, nil }, + }) + var count atomic.Int64 + h := handler(t, a, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + count.Add(1) + kind := identity.KindJWT + if strings.HasPrefix(r.Header.Get("Authorization"), "EdgeIAM") { + kind = identity.KindIAM + } + if identity.FromContext(r.Context()).Kind() != kind { + t.Error("caller crossed request boundaries") + } + w.WriteHeader(204) + })) + var group sync.WaitGroup + for i := range 64 { + group.Go(func() { + r := httptest.NewRequestWithContext(t.Context(), "GET", "/", nil) + scheme := "Bearer" + if i%2 == 0 { + scheme = "EdgeIAM" + } + r.Header.Set("Authorization", scheme+" token") + h.ServeHTTP(httptest.NewRecorder(), r) + }) + } + group.Wait() + if count.Load() != 64 { + t.Fatalf("handler ran %d times; want 64", count.Load()) + } +} + +func TestPanicPropagation(t *testing.T) { + marker := new(int) + a := authenticator(t, authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { panic(marker) }}) + defer func() { + if got := recover(); got != marker { + t.Errorf("panic=%v; want original marker", got) + } + }() + a.Authenticate(t.Context(), http.Header{"Authorization": {"Bearer token"}}) +} diff --git a/authn/doc.go b/authn/doc.go new file mode 100644 index 0000000..e5b0768 --- /dev/null +++ b/authn/doc.go @@ -0,0 +1,16 @@ +// Package authn selects one configured Bearer or EdgeIAM verifier from an HTTP +// Authorization header and transports the verified caller to protected handlers. +// It does not implement token verification, authorize actions, or infer trust +// from a gateway, a TLS field, an action header or forwarded headers. +// +// Use New to configure explicit verifier functions. Authenticate supports custom +// HTTP pipelines without writing responses or installing context. Handler adds +// required-authentication middleware. Public routes omit that middleware; CORS +// and preflight policy belong to the gateway or outer middleware. +// +// Verifiers must authenticate credentials before returning a caller and classify +// rejection with ErrInvalidCredentials. Adapt iamproof.ErrInvalidProof explicitly +// when wiring an IAM proof verifier; this package does not import AWS packages. +// Never retry another verifier after rejection. Authenticate and authorize the +// selected action before committing a response or starting a stream. +package authn diff --git a/authn/errors.go b/authn/errors.go new file mode 100644 index 0000000..9590c95 --- /dev/null +++ b/authn/errors.go @@ -0,0 +1,85 @@ +package authn + +import ( + "errors" + "net/http" + "slices" +) + +// Inspectable failure categories never contain credentials or provider errors. +var ( + ErrInvalidConfiguration = errors.New("authn: invalid configuration") + ErrMissingCredentials = errors.New("authn: missing credentials") + ErrMalformedCredentials = errors.New("authn: malformed credentials") + ErrUnsupportedScheme = errors.New("authn: unsupported scheme") + ErrHeaderTooLarge = errors.New("authn: authorization header too large") + ErrInvalidCredentials = errors.New("authn: invalid credentials") + ErrUnavailable = errors.New("authn: verification unavailable") + ErrVerifierContract = errors.New("authn: invalid verifier result") +) + +// Error describes a sanitized HTTP authentication failure. Inspect its category +// with errors.Is and its HTTP treatment with errors.AsType[*Error]. Its private +// state never retains a supplied credential or an underlying provider error. +// The zero value describes an internal failure, without a challenge. +type Error struct { + category error + status int + challenges []string +} + +// Error returns a diagnostic containing only the sanitized category. +func (e *Error) Error() string { return e.Unwrap().Error() } + +// Unwrap exposes only a package category or identity.ErrConflict. +func (e *Error) Unwrap() error { + if e == nil || e.category == nil { + return ErrVerifierContract + } + return e.category +} + +// StatusCode returns the recommended HTTP status, or 500 for a zero Error. +func (e *Error) StatusCode() int { + if e == nil || e.status == 0 { + return http.StatusInternalServerError + } + return e.status +} + +// Challenges returns an owned copy of complete WWW-Authenticate values. +func (e *Error) Challenges() []string { + if e == nil { + return nil + } + return slices.Clone(e.challenges) +} + +// GoString returns the same sanitized diagnostic used by Error. +func (e *Error) GoString() string { return e.Error() } + +func (a *Authenticator) failure(category error, status int, selected scheme) *Error { + e := &Error{category: category, status: status} + if status != http.StatusBadRequest && status != http.StatusUnauthorized { + return e + } + if category == ErrInvalidCredentials { + if selected == bearer { + e.challenges = []string{a.bearerChallenge + `, error="invalid_token"`} + } else { + e.challenges = []string{a.iamChallenge} + } + return e + } + if a.bearerChallenge != "" { + challenge := a.bearerChallenge + if category == ErrMalformedCredentials && selected == bearer { + challenge += `, error="invalid_request"` + } + e.challenges = append(e.challenges, challenge) + } + if a.iamChallenge != "" { + e.challenges = append(e.challenges, a.iamChallenge) + } + return e +} diff --git a/authn/example_test.go b/authn/example_test.go new file mode 100644 index 0000000..cd646e6 --- /dev/null +++ b/authn/example_test.go @@ -0,0 +1,153 @@ +package authn_test + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "net/http" + "net/http/httptest" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func ExampleAuthenticator_Handler() { + proofVerifier, err := iamproof.NewVerifier("eu-west-2", "orders.production") + if err != nil { + fmt.Println(err) + return + } + verifyIAM := func(ctx context.Context, token string) (identity.Caller, error) { + caller, err := proofVerifier.Verify(ctx, token) + if errors.Is(err, iamproof.ErrInvalidProof) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + return caller, err + } + // Synthetic fixture only. Replace this function with your actual issuer/key/ + // signature/expiry/client-policy verifier. Parsing claims is NOT verification. + verifyBearer := func(_ context.Context, token string) (identity.Caller, error) { + if token != "synthetic-example-token" { + return identity.Caller{}, authn.ErrInvalidCredentials + } + claims, err := identity.NewClaims(map[string]any{"iss": "https://issuer.example", "sub": "alice"}) + if err != nil { + return identity.Caller{}, err + } + return identity.NewJWT(claims, identity.SourceLocallyVerifiedToken) + } + a, err := authn.New(authn.Config{Bearer: verifyBearer, IAMProof: verifyIAM, Realm: "orders"}) + if err != nil { + fmt.Println(err) + return + } + selector, err := edge.NewActionHeader("Action") + if err != nil { + fmt.Println(err) + return + } + dispatcher := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, err := selector.Parse(r.Header) + if err != nil { + http.Error(w, "invalid action", http.StatusBadRequest) + return + } + // Consumer-owned authorization: deny by default. Neither successful + // token verification nor AWS account membership grants this action alone. + caller := identity.FromContext(r.Context()) + allowed := false + if jwt, ok := caller.JWT(); ok { + sub, present := jwt.Subject() + allowed = present && jwt.Issuer() == "https://issuer.example" && sub == "alice" + } + if iam, ok := caller.IAM(); ok { + allowed = iam.PrincipalARN() == "arn:aws:iam::123456789012:user/Alice" + } + if !allowed || action != "orders.read" { + http.Error(w, "forbidden", http.StatusForbidden) + return + } + // Execute the exact selection just authorized. Extra consumer headers + // remain available through r.Header; authn does not interpret them. + w.WriteHeader(http.StatusNoContent) + }) + protected, err := a.Handler(dispatcher) + if err != nil { + fmt.Println(err) + return + } + // protected also goes directly to edge.New for Lambda registration. + // This local example exercises only the synthetic Bearer path; no AWS calls. + r := httptest.NewRequest(http.MethodPost, "/", nil) + r.Header.Set("Authorization", "Bearer synthetic-example-token") + r.Header.Set("Action", "orders.read") + w := httptest.NewRecorder() + protected.ServeHTTP(w, r) + fmt.Println(w.Code) + // The malformed proof is rejected before any STS call, using the mapping. + r.Header.Set("Authorization", "EdgeIAM v2.invalid") + w = httptest.NewRecorder() + protected.ServeHTTP(w, r) + fmt.Println(w.Code) + fmt.Println(w.Header().Get("WWW-Authenticate")) + // Output: + // 204 + // 401 + // EdgeIAM realm="orders" +} + +func ExampleAuthenticator_Authenticate() { + // This example demonstrates a custom JSON error envelope. Its verifier + // rejects all credentials and is never called for the missing-header case. + a, err := authn.New(authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { + return identity.Caller{}, authn.ErrInvalidCredentials + }}) + if err != nil { + fmt.Println(err) + return + } + h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + caller, err := a.Authenticate(r.Context(), r.Header) + if err != nil { + status := http.StatusInternalServerError + if failure, ok := errors.AsType[*authn.Error](err); ok { + status = failure.StatusCode() + for _, challenge := range failure.Challenges() { + w.Header().Add("WWW-Authenticate", challenge) + } + } else if r.Context().Err() != nil { + status = http.StatusServiceUnavailable + } + body, encodeErr := json.Marshal(struct { + Error string `json:"error"` + }{Error: http.StatusText(status)}) + if encodeErr != nil { + http.Error(w, "internal error", http.StatusInternalServerError) + return + } + w.Header().Set("Cache-Control", "no-store") + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + if _, err := w.Write(body); err != nil { + return // The transport can no longer deliver this response. + } + return + } + // A real custom pipeline would install caller, authorize its selected + // action, then invoke the application. No such success is possible here. + _ = caller + http.Error(w, "no action authorized", http.StatusForbidden) + }) + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/", nil)) + fmt.Println(w.Code) + fmt.Println(w.Header().Get("WWW-Authenticate")) + fmt.Println(w.Body.String()) + // Output: + // 401 + // Bearer realm="edge" + // {"error":"Unauthorized"} +} diff --git a/authn/fuzz_test.go b/authn/fuzz_test.go new file mode 100644 index 0000000..9f17fff --- /dev/null +++ b/authn/fuzz_test.go @@ -0,0 +1,38 @@ +package authn_test + +import ( + "context" + "errors" + "net/http" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func FuzzAuthorization(f *testing.F) { + f.Add("Authorization", "Bearer abc", "", false) + f.Add("authorization", "EdgeIAM v1.abc", "Bearer def", true) + f.Add("AUTHORIZATION", "Bearer\tfoo", "", false) + f.Add("Authorization", "Bearer abc,def", "", false) + f.Fuzz(func(t *testing.T, key, value, extra string, repeated bool) { + var calls int + verify := func(context.Context, string) (identity.Caller, error) { + calls++ + return identity.Caller{}, authn.ErrInvalidCredentials + } + a := authenticator(t, authn.Config{Bearer: verify, IAMProof: verify}) + headers := http.Header{key: {value}} + if repeated { + headers[key] = append(headers[key], extra) + } + caller, err := a.Authenticate(t.Context(), headers) + failure, ok := errors.AsType[*authn.Error](err) + if caller.Kind() != identity.KindAnonymous || !ok || calls > 1 || repeated && calls != 0 { + t.Fatalf("untrusted input returned caller=%v err=%v calls=%d", caller, err, calls) + } + if failure.StatusCode() == 401 && len(failure.Challenges()) == 0 { + t.Fatal("401 without a challenge") + } + }) +} diff --git a/authn/gateway_test.go b/authn/gateway_test.go new file mode 100644 index 0000000..15cc094 --- /dev/null +++ b/authn/gateway_test.go @@ -0,0 +1,291 @@ +package authn_test + +import ( + "context" + "encoding/json/v2" + "errors" + "io" + "net/http" + "net/http/httptest" + "slices" + "strings" + "sync/atomic" + "testing" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-sdk-go-v2/aws" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +type transportFunc func(*http.Request) (*http.Response, error) + +func (f transportFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +func localProofVerifier(t testing.TB) (*iamproof.Verifier, iamproof.Token, *atomic.Int64) { + t.Helper() + g, err := iamproof.NewGenerator("eu-west-2", "orders.production", aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { + return aws.Credentials{AccessKeyID: "AKIAIOSFODNN7EXAMPLE", SecretAccessKey: "synthetic-only", SessionToken: "synthetic+/="}, nil + })) + if err != nil { + t.Fatal(err) + } + token, err := g.Generate(t.Context()) + if err != nil { + t.Fatal(err) + } + count := new(atomic.Int64) + v, err := iamproof.NewVerifier("eu-west-2", "orders.production", iamproof.WithTransport(transportFunc(func(r *http.Request) (*http.Response, error) { + count.Add(1) + if r.URL.Host != "sts.eu-west-2.amazonaws.com" || r.Header.Get("X-Edge-IAM-Audience") != "orders.production" || r.URL.Query().Get("Action") != "GetCallerIdentity" || r.URL.Query().Get("X-Amz-Signature") == "" { + t.Error("incorrect reconstructed STS request") + } + // A synthetic STS response exercises production composition, not AWS + // acceptance. Signature reconstruction is separately tested by iamproof. + return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader(`arn:aws:iam::123456789012:user/Alice123456789012AIDAEXAMPLE`))}, nil + }))) + if err != nil { + t.Fatal(err) + } + return v, token, count +} + +func mapProofErrors(v *iamproof.Verifier) authn.VerifyFunc { + return func(ctx context.Context, token string) (identity.Caller, error) { + caller, err := v.Verify(ctx, token) + if errors.Is(err, iamproof.ErrInvalidProof) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + return caller, err + } +} + +func TestAuthenticationAcrossTransports(t *testing.T) { + proofVerifier, proof, requests := localProofVerifier(t) + jwt := jwtCaller(t, identity.SourceLocallyVerifiedToken) + a := authenticator(t, authn.Config{ + Bearer: func(_ context.Context, token string) (identity.Caller, error) { + switch token { + case "synthetic-jwt": + return jwt, nil + case "unavailable": + return identity.Caller{}, authn.ErrUnavailable + default: + return identity.Caller{}, authn.ErrInvalidCredentials + } + }, + IAMProof: mapProofErrors(proofVerifier), + }) + selector, err := edge.NewActionHeader("Action") + if err != nil { + t.Fatal(err) + } + h := handler(t, a, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, err := selector.Parse(r.Header) + if err != nil { + http.Error(w, "invalid action", 400) + return + } + caller := identity.FromContext(r.Context()) + allowed := false + if iam, ok := caller.IAM(); ok { + allowed = iam.PrincipalARN() == "arn:aws:iam::123456789012:user/Alice" + } + if jwt, ok := caller.JWT(); ok { + sub, _ := jwt.Subject() + allowed = jwt.Issuer() == "https://issuer.example" && sub == "alice" + } + // Deny by default and authorize this exact action before executing it. + if !allowed || action != "orders.read" { + http.Error(w, "forbidden", 403) + return + } + if r.Header.Get("Custom-Trace") != "keep" { + t.Error("middleware lost extra consumer header") + } + w.WriteHeader(204) + })) + for _, tc := range []struct { + name string + values []string + action string + status int + challenges []string + }{ + {name: "missing", status: 401, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, + {name: "bearer_success", values: []string{"Bearer synthetic-jwt"}, status: 204}, + {name: "iam_success", values: []string{"EdgeIAM " + proof.Value()}, status: 204}, + {name: "bearer_rejected", values: []string{"Bearer invalid"}, status: 401, challenges: []string{`Bearer realm="edge", error="invalid_token"`}}, + {name: "iam_rejected", values: []string{"EdgeIAM v2.invalid"}, status: 401, challenges: []string{`EdgeIAM realm="edge"`}}, + {name: "iam_proof_size", values: []string{"EdgeIAM v1." + strings.Repeat("A", 8192)}, status: 401, challenges: []string{`EdgeIAM realm="edge"`}}, + {name: "duplicates", values: []string{"Bearer synthetic-jwt", "Bearer synthetic-jwt"}, status: 400, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, + {name: "combined", values: []string{"Bearer synthetic-jwt,EdgeIAM invalid"}, status: 400, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, + {name: "malformed", values: []string{"Bearer a=b"}, status: 400, challenges: []string{`Bearer realm="edge", error="invalid_request"`, `EdgeIAM realm="edge"`}}, + {name: "unsupported", values: []string{"Basic abc"}, status: 401, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, + {name: "dependency", values: []string{"Bearer unavailable"}, status: 503}, + {name: "authz_denial", values: []string{"Bearer synthetic-jwt"}, action: "orders.delete", status: 403}, + {name: "iam_authz_denial", values: []string{"EdgeIAM " + proof.Value()}, action: "orders.delete", status: 403}, + {name: "field_size", values: []string{"Bearer " + strings.Repeat("A", 16384)}, status: 431}, + } { + t.Run(tc.name, func(t *testing.T) { + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { + t.Run(format, func(t *testing.T) { + action := tc.action + if action == "" { + action = "orders.read" + } + status, headers := serveFormat(t, h, format, tc.values, action) + wantChallenges := tc.challenges + if strings.HasSuffix(format, "v2") && len(wantChallenges) > 0 { + wantChallenges = []string{strings.Join(wantChallenges, ", ")} + } + if status != tc.status || !slices.Equal(headers.Values("WWW-Authenticate"), wantChallenges) { + t.Errorf("response = %d %q; want %d %q", status, headers.Values("WWW-Authenticate"), tc.status, wantChallenges) + } + }) + } + }) + } + if requests.Load() != 12 { + t.Fatalf("STS requests=%d; want 12 for the two valid-proof cases across six transports", requests.Load()) + } +} + +func serveFormat(t *testing.T, h http.Handler, format string, values []string, action string) (int, http.Header) { + t.Helper() + if format == "native" { + server := httptest.NewTestServer(t, h) + req, err := http.NewRequestWithContext(t.Context(), "GET", "http://example.com/", nil) + if err != nil { + t.Fatal(err) + } + req.Header = http.Header{"Authorization": values, "Action": {action}, "Custom-Trace": {"keep"}} + resp, err := server.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if _, err := io.Copy(io.Discard, resp.Body); err != nil { + t.Fatal(err) + } + return resp.StatusCode, resp.Header + } + adapter, err := edge.New(h) + if err != nil { + t.Fatal(err) + } + v1 := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}, + Headers: map[string]string{"action": action, "custom-trace": "keep"}} + v2 := events.APIGatewayV2HTTPRequest{Version: "2.0", RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}, + Headers: map[string]string{"action": action, "custom-trace": "keep"}} + if len(values) != 0 { + v1.Headers["authorization"] = values[0] + v1.MultiValueHeaders = map[string][]string{"Authorization": values} + v2.Headers["authorization"] = strings.Join(values, ",") + } + var resp1 events.APIGatewayProxyResponse + var resp2 events.APIGatewayV2HTTPResponse + switch format { + case "typed_v1": + resp1, err = adapter.HandleV1(t.Context(), v1) + case "typed_v2": + resp2, err = adapter.HandleV2(t.Context(), v2) + default: + var event any = v1 + if format == "raw_http_v2" { + event = v2 + } + wire, marshalErr := json.Marshal(event) + if marshalErr != nil { + t.Fatal(marshalErr) + } + if format == "raw_http_v1" { + wire = append([]byte(`{"version":"1.0",`), wire[1:]...) + } + out, invokeErr := adapter.Invoke(t.Context(), wire) + if invokeErr != nil { + t.Fatal(invokeErr) + } + if format == "raw_http_v2" { + err = json.Unmarshal(out, &resp2) + } else { + err = json.Unmarshal(out, &resp1) + } + } + if err != nil { + t.Fatal(err) + } + if strings.HasSuffix(format, "v2") { + headers := make(http.Header) + for name, value := range resp2.Headers { + headers.Set(name, value) + } + return resp2.StatusCode, headers + } + return resp1.StatusCode, http.Header(resp1.MultiValueHeaders) +} + +func TestIAMErrorMappingIsExplicit(t *testing.T) { + v, _, _ := localProofVerifier(t) + for _, mapped := range []bool{false, true} { + verify := authn.VerifyFunc(v.Verify) + if mapped { + verify = mapProofErrors(v) + } + a := authenticator(t, authn.Config{IAMProof: verify}) + caller, err := a.Authenticate(t.Context(), http.Header{"Authorization": {"EdgeIAM v2.invalid"}}) + if mapped { + failure(t, caller, err, authn.ErrInvalidCredentials, 401, `EdgeIAM realm="edge"`) + } else { + failure(t, caller, err, authn.ErrUnavailable, 503) + } + } +} + +func TestGatewayProducerConflictAndInvocationCancellation(t *testing.T) { + a := authenticator(t, authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { + t.Error("conflicting/canceled request reached verifier") + return identity.Caller{}, authn.ErrInvalidCredentials + }}) + h := handler(t, a, http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Error("failure reached handler") })) + adapter, err := edge.New(h, edge.WithGatewayIdentity(true)) + if err != nil { + t.Fatal(err) + } + event := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{ + APIID: "example", Identity: events.APIGatewayRequestIdentity{UserArn: "arn:aws:iam::123456789012:user/Alice", AccountID: "123456789012"}, + }} + resp, err := adapter.HandleV1(t.Context(), event) + if err != nil || resp.StatusCode != 500 || len(resp.MultiValueHeaders["Www-Authenticate"]) != 0 { + t.Fatalf("gateway/local conflict: response=%d err=%v; want HTTP 500", resp.StatusCode, err) + } + ctx, cancel := context.WithCancel(t.Context()) + cancel() + resp, err = adapter.HandleV1(ctx, event) + if !errors.Is(err, context.Canceled) || resp.StatusCode != 0 { + t.Fatalf("canceled invocation returned usable response=%d err=%v", resp.StatusCode, err) + } +} + +func TestNoAlternateCredentialsOrOptionsBypass(t *testing.T) { + a := authenticator(t, authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { + t.Error("alternate credential source reached verifier") + return identity.Caller{}, authn.ErrInvalidCredentials + }}) + h := handler(t, a, http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Error("anonymous request reached next") })) + for _, method := range []string{http.MethodPost, http.MethodOptions} { + r := httptest.NewRequest(method, "/?access_token=secret", strings.NewReader("access_token=secret")) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + r.AddCookie(&http.Cookie{Name: "Authorization", Value: "secret"}) + r.Header.Set("X-Authorization", "Bearer secret") + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + body, err := io.ReadAll(r.Body) + if err != nil || string(body) != "access_token=secret" || w.Code != 401 { + t.Fatalf("method %s: status=%d body=%q err=%v", method, w.Code, body, err) + } + } +} diff --git a/authn/handler.go b/authn/handler.go new file mode 100644 index 0000000..19c034f --- /dev/null +++ b/authn/handler.go @@ -0,0 +1,72 @@ +package authn + +import ( + "context" + "errors" + "net/http" + "reflect" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Handler constructs required-authentication middleware. It validates next, +// including typed nils. Success installs one caller in a derived request context +// and invokes next exactly once. It leaves the original request and headers intact. +// +// Failure writes a generic text response, no-store and the applicable challenges; +// next is not invoked. Caller cancellation attempts 503 without a challenge, but +// delivery is not guaranteed. Provider/handler panics propagate. Public routes +// and CORS/preflight must be configured separately; there is no OPTIONS bypass. +func (a *Authenticator) Handler(next http.Handler) (http.Handler, error) { + if a == nil || a.cfg.MaxAuthorizationBytes == 0 || next == nil { + return nil, ErrInvalidConfiguration + } + v := reflect.ValueOf(next) + switch v.Kind() { + case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice: + if v.IsNil() { + return nil, ErrInvalidConfiguration + } + } + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + caller, err := a.Authenticate(r.Context(), r.Header) + if err != nil { + writeFailure(w, err) + return + } + ctx, err := identity.WithCaller(r.Context(), caller) + if err != nil { + writeFailure(w, a.failure(identity.ErrConflict, http.StatusInternalServerError, noScheme)) + return + } + if err := ctx.Err(); err != nil { + writeFailure(w, err) + return + } + next.ServeHTTP(w, r.WithContext(ctx)) + }), nil +} + +func writeFailure(w http.ResponseWriter, err error) { + status := http.StatusInternalServerError + var challenges []string + if failure, ok := errors.AsType[*Error](err); ok { + status, challenges = failure.StatusCode(), failure.Challenges() + } else if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { + status = http.StatusServiceUnavailable + } + headers := w.Header() + // Direct map users can leave noncanonical aliases. Own these failure fields + // without dropping unrelated outer middleware headers such as CORS. + for name := range headers { + switch http.CanonicalHeaderKey(name) { + case "Www-Authenticate", "Cache-Control": + delete(headers, name) + } + } + for _, challenge := range challenges { + headers.Add("WWW-Authenticate", challenge) + } + headers.Set("Cache-Control", "no-store") + http.Error(w, http.StatusText(status), status) +} diff --git a/authn/header.go b/authn/header.go new file mode 100644 index 0000000..f2a1d51 --- /dev/null +++ b/authn/header.go @@ -0,0 +1,97 @@ +package authn + +import ( + "net/http" + "strings" +) + +type scheme uint8 + +const ( + noScheme scheme = iota + bearer + iamProof +) + +func (a *Authenticator) selectCredential(headers http.Header) (scheme, string, error) { + var value string + var found, multiple bool + remaining := a.cfg.MaxAuthorizationBytes + for name, values := range headers { + if len(name) != len("Authorization") || http.CanonicalHeaderKey(name) != "Authorization" { + continue + } + for _, v := range values { + if len(v) > remaining { + return noScheme, "", a.failure(ErrHeaderTooLarge, http.StatusRequestHeaderFieldsTooLarge, noScheme) + } + remaining -= len(v) + multiple = multiple || found + value, found = v, true + } + } + if !found { + return noScheme, "", a.failure(ErrMissingCredentials, http.StatusUnauthorized, noScheme) + } + if multiple || strings.Contains(value, ",") { + return noScheme, "", a.failure(ErrMalformedCredentials, http.StatusBadRequest, noScheme) + } + value = strings.Trim(value, " \t") + end := strings.IndexAny(value, " \t") + if end < 0 { + end = len(value) + } + name := value[:end] + selected := noScheme + switch { + case strings.EqualFold(name, "Bearer") && a.cfg.Bearer != nil: + selected = bearer + case strings.EqualFold(name, "EdgeIAM") && a.cfg.IAMProof != nil: + selected = iamProof + } + malformed := func() (scheme, string, error) { + return noScheme, "", a.failure(ErrMalformedCredentials, http.StatusBadRequest, selected) + } + if name == "" || end == len(value) || value[end] != ' ' { + return malformed() + } + for i := range len(name) { + if !tokenChar(name[i]) { + return malformed() + } + } + credential := strings.TrimLeft(value[end:], " ") + if credential == "" { + return malformed() + } + for i := range len(credential) { + if credential[i] <= 0x20 || credential[i] >= 0x7f { + return malformed() + } + } + if selected == noScheme { + return noScheme, "", a.failure(ErrUnsupportedScheme, http.StatusUnauthorized, noScheme) + } + if !token68(credential) { + return malformed() + } + return selected, credential, nil +} + +func tokenChar(b byte) bool { + return b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' || b >= '0' && b <= '9' || strings.ContainsRune("!#$%&'*+-.^_`|~", rune(b)) +} + +func token68(value string) bool { + content := strings.TrimRight(value, "=") + if content == "" { + return false + } + for i := range len(content) { + b := content[i] + if !(b >= 'a' && b <= 'z' || b >= 'A' && b <= 'Z' || b >= '0' && b <= '9' || strings.ContainsRune("-._~+/", rune(b))) { + return false + } + } + return true +} diff --git a/docs/authn.md b/docs/authn.md new file mode 100644 index 0000000..5764bdc --- /dev/null +++ b/docs/authn.md @@ -0,0 +1,149 @@ +# HTTP authentication + +`authn` is an implemented, SDK-independent HTTP authentication selector. It +chooses exactly one configured Bearer or EdgeIAM verifier, checks the returned +caller and, when used as middleware, installs it in the request context. It works +with ordinary Go HTTP servers and the root edge adapter's raw/typed entry points. + +It does not implement JWT verification or grant application permissions. The +first-party Cognito/JWT/JWKS verifier and the authz package remain future work. +Supply your own fully verifying Bearer function today, use IAM proofs alone, +or configure both. Default gateway identity remains disabled. + +## Configure both mechanisms + +```go +proofVerifier, err := iamproof.NewVerifier("eu-west-2", "orders.production") +if err != nil { + return err +} +verifyIAM := func(ctx context.Context, token string) (identity.Caller, error) { + caller, err := proofVerifier.Verify(ctx, token) + if errors.Is(err, iamproof.ErrInvalidProof) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + return caller, err +} +authenticator, err := authn.New(authn.Config{ + Bearer: verifyBearer, // Your actual verification function. + IAMProof: verifyIAM, + Realm: "orders", +}) +if err != nil { + return err +} +protected, err := authenticator.Handler(dispatcher) +if err != nil { + return err +} +adapter, err := edge.New(protected) +if err != nil { + return err +} +lambda.Start(adapter) +``` + +The IAM error adapter above is required: passing `proofVerifier.Verify` directly +leaves its package-specific rejection unclassified and results in 503 instead +of 401. This explicit mapping lets JWT-only applications use authn without +importing the AWS SDK. No dependency versions changed for this package. + +`verifyBearer` has signature +`func(context.Context, string) (identity.Caller, error)`. It receives the exact +credential without the scheme. It must validate the intended issuer, signature, +time and client/audience policy before constructing a JWT caller with +SourceLocallyVerifiedToken. Parsing a JWT or calling identity.NewJWT is not +verification. Return authn.ErrInvalidCredentials only for a definite credential +rejection; authn.ErrUnavailable or unclassified errors become 503. No raw provider +diagnostics are retained in the selector's returned error. + +The IAM function must return KindIAM/SourceVerifiedIAMProof. Bearer must return +KindJWT/SourceLocallyVerifiedToken. Anonymous, gateway/custom, or wrong-kind +success is a verifier contract failure, not client rejection. All functions must +be concurrency-safe when their Authenticator is shared. + +Omitting either function disables that scheme. Both omitted is a configuration +error. New copies configuration and performs no I/O. Realm defaults to `edge` +and is only a challenge label; it does not configure proof audience or JWT trust. +Explicit realms allow up to 256 printable ASCII bytes and are escaped safely. + +## Request selection + +Only Authorization establishes credentials. Cookies, query parameters, bodies +and alternate headers are not examined as authentication sources. The parser +checks all case-equivalent Authorization map keys, rejects repeated values even +if identical, and rejects comma-combined credentials. It does not try a second +verifier after any failure. + +Schemes are case-insensitive. Outer spaces/tabs are trimmed; the separator must +be one or more spaces. Token bytes and case are preserved. Configured schemes +require token68-shaped header credentials. Bearer contents are not decoded to +select a verifier. The IAM verifier applies its stricter proof format afterward. + +MaxAuthorizationBytes defaults to 16 KiB, counting represented field-value bytes +before trimming. An explicit value must be 1 byte through 1 MiB; zero requests the +default. This is separate from the IAM proof's fixed 8 KiB credential limit and +from gateway/server limits on the whole request. Earlier proxies can discard +duplicate information; no middleware can reconstruct it afterward. + +An already established caller produces identity.ErrConflict before verifier +work. Do not place this local authentication middleware behind an enabled gateway +producer for the same route. There is no implicit skip, merge or replacement. + +## Errors and HTTP customization + +Authenticate(ctx, headers) returns one caller without installing it or writing a +response. Failures return an anonymous zero caller. Inspect categories with +errors.Is and HTTP details with errors.AsType[*authn.Error]. Challenges returns +an owned copy. Caller cancellation/deadline returns ctx.Err directly, without +exposing context.Cause. + +| Category | Default HTTP response | +| --- | --- | +| ErrMissingCredentials / ErrUnsupportedScheme | 401; configured challenges | +| ErrMalformedCredentials | 400; configured challenges, Bearer invalid_request when unambiguously selected | +| ErrHeaderTooLarge | 431; no challenge | +| ErrInvalidCredentials | 401; selected scheme, Bearer invalid_token where applicable | +| ErrUnavailable or unknown verifier error | 503; no challenge | +| ErrVerifierContract / identity.ErrConflict | 500; no challenge | +| Caller cancellation/deadline | Best-effort 503; no challenge | + +An Error chain contains only its sanitized category. A nonnil verifier error +discards any returned caller. Unavailable takes precedence over invalid +credentials when an error matches both. An internal timeout with a live request +context is unavailable, not caller cancellation. + +Handler returns fixed generic text and Cache-Control: no-store on failure, +replaces WWW-Authenticate and preserves unrelated response headers such as CORS. +It does not log, add Retry-After or include detailed error descriptions. Request +cancellation prevents the next handler from running; delivery of the 503 is not +guaranteed. Edge still rejects a usable response for a canceled invocation. + +For custom JSON responses, use Authenticate and the Error accessors in ordinary +HTTP middleware. Serialize application envelopes directly with encoding/json/v2. +On success, install the returned caller with identity.WithCaller before dispatch. +See the [runnable examples](../authn/example_test.go) for error mapping, a custom +JSON failure response and a shared action dispatcher. Their synthetic Bearer +fixture is explicitly not a production token verifier. + +## Action dispatch and deployment + +Select an action once using edge.ActionHeader or your own header processor. +Authorize that caller for that exact selection, then execute it. Both IAM and +JWT callers can reach the same dispatcher. Successful IAM authentication does +not permit every AWS identity to execute your actions; deny by default. + +Complete authentication and authorization before writing or flushing a response. +The caller is a snapshot and is not automatically refreshed during a stream. +Application-specific ongoing authorization remains explicit. + +Protect only intended routes. Public routes omit Handler. There is no automatic +OPTIONS bypass: configure CORS/preflight at API Gateway or in outer middleware. +Deploy over HTTPS; r.TLS need not be populated behind API Gateway, and forwarded +scheme headers do not themselves establish trust. + +Local tests cover native in-memory HTTP, raw REST/HTTP API payloads and typed +V1/V2 methods, including challenge combination and actual IAM helper/verifier +composition against synthetic STS responses. No live AWS service or deployment +was used. [Decision 0019](decisions/0019-http-authentication-selector.md) records +the accepted contract and its official documentation sources. diff --git a/docs/decisions/0016-mixed-iam-oauth-authentication.md b/docs/decisions/0016-mixed-iam-oauth-authentication.md index ee72613..88f4856 100644 --- a/docs/decisions/0016-mixed-iam-oauth-authentication.md +++ b/docs/decisions/0016-mixed-iam-oauth-authentication.md @@ -2,7 +2,8 @@ Status: architecture and exported Go client helper accepted on 2026-09-16. The detailed IAM protocol/API was subsequently approved and implemented under -0017. The shared HTTP selector is proposed in 0019 and awaits review. +0017. The shared HTTP selector was accepted and implemented under 0019. +First-party JWT verification and authorization remain separate work. ## Requirement and correction @@ -34,8 +35,8 @@ planned IAM authentication path. permission for the operation. It does not establish application permissions. AWS definitions were checked through public AWS MCP; the authenticator's own -documentation supplies implementation-pattern evidence. Current code has no authn -middleware. The source rules in 0012 permit gateway/custom IAM and locally verified +documentation supplies implementation-pattern evidence. At proposal time there +was no authn middleware. The source rules in 0012 permit gateway/custom IAM and locally verified JWT; online IAM verification requires a provenance refinement before implementation. ## Accepted direction @@ -110,5 +111,6 @@ error behavior. Acceptance of the architecture does not freeze these details. [Decision 0017](0017-iam-proof-protocol.md) subsequently settled those details and its production helper/verifier are implemented with local validation. -[Decision 0019](0019-http-authentication-selector.md) now proposes the remaining -shared HTTP selector API and response contract. No live STS check has run. +[Decision 0019](0019-http-authentication-selector.md) settled the remaining +shared HTTP selector API and response contract, now implemented with local +validation and synthetic Bearer fixtures. No live STS check has run. diff --git a/docs/decisions/0017-iam-proof-protocol.md b/docs/decisions/0017-iam-proof-protocol.md index 65c1720..10579a2 100644 --- a/docs/decisions/0017-iam-proof-protocol.md +++ b/docs/decisions/0017-iam-proof-protocol.md @@ -3,7 +3,8 @@ Status: accepted on 2026-09-16. The user approved P1-P5 and the four caller forms in decision 0015. The production iamproof generator and verifier are implemented after the shared caller and buffered-adapter foundations, with local validation. -Common HTTP authentication selection and live AWS interoperability remain pending. +Common HTTP authentication selection is implemented under decision 0019; +live AWS interoperability remains pending. ## Evidence @@ -245,8 +246,8 @@ Verify any further service-specific aliases against official sources and fixture before adding them. Unrecognized errors fail closed as unavailable; throttling remains unavailable even with HTTP 400. Caller cancellation returns the context error; writing a response after disconnection follows the HTTP middleware lifecycle. -WWW-Authenticate challenges and shared selector API belong to the HTTP authn -review, not the proof verifier. These are HTTP failures, not Lambda invocation +WWW-Authenticate challenges and the shared selector API are specified by +decision 0019, separately from the proof verifier. These are HTTP failures, not Lambda invocation errors under decision 0013. ## Design graph and implementation sequence diff --git a/docs/decisions/0019-http-authentication-selector.md b/docs/decisions/0019-http-authentication-selector.md index b7a7e1b..49c3954 100644 --- a/docs/decisions/0019-http-authentication-selector.md +++ b/docs/decisions/0019-http-authentication-selector.md @@ -1,8 +1,8 @@ # 0019: HTTP authentication selection and challenges -Status: proposed on 2026-09-16; awaiting user review. The user authorized this -design walk after implementation of iamproof. No authn API or middleware is -implemented by this record. Recommendations A1-A6 are reviewable together. +Status: accepted on 2026-09-16. The user approved A1-A6 and authorized +implementation after reviewing this design. Configuration, header selection, +verification/result checks, errors, middleware and examples are implemented. ## Evidence and existing contracts @@ -265,7 +265,7 @@ handler accidentally becoming an implicit 200 on an otherwise live native HTTP connection. Edge's invocation cancellation rules still suppress a usable Lambda response when its parent context is canceled. Do not promise wire delivery. -This is an HTTP-level refinement awaiting approval, not a change to the existing +This is an accepted HTTP-level refinement, not a change to the existing iamproof error contract. Malformed proof/token content still yields 401; malformed Authorization framing yields 400. Authorization owns permission denial and 403. @@ -312,5 +312,33 @@ headers are not sufficient evidence to authenticate a caller. ## Resolution -Pending user review of A1-A6, especially the public config/API, explicit IAM error -adapter, 400/401/431 distinction, and default cancellation response behavior. +The user approved A1-A6 on 2026-09-16, including the public config/API, explicit +IAM error adapter, 400/401/431 distinction and default cancellation response. +Implement within these boundaries; Cognito/JWKS and authz remain separate reviews. + +## Implementation evidence + +The production authn package implements A1-A6. Dependency inspection confirms +only standard-library and identity imports; no iamproof or AWS SDK dependency +is introduced. The explicit IAM error adapter is documented and tested, including +the qualified 503 behavior when consumers omit it. + +Local tests cover case aliases, empty/repeated/comma-combined fields, grammar and +size limits, scheme selection, no fallback, error precedence and sanitization, +wrong caller kinds/sources, gateway/local conflicts, cancellation, context and +challenge ownership, concurrency, and propagation of verifier panics. Actual +IAM generation/verification uses synthetic STS transport fixtures through native +HTTP and all raw/typed Gateway entry points. Both caller kinds reach one consumer +dispatcher with independent action permission checks. Challenge combination +works in payload 2.0 without changing the existing header contract. Inputs that +transport validation rejects earlier remain outside the HTTP middleware contract. + +Runnable examples cover the explicit error adapter, synthetic consumer Bearer +verification, a shared action dispatcher and direct JSON v2 failure responses. +They clearly distinguish synthetic token fixtures from production verification. +The [consumer guide](../authn.md) records deployment and composition constraints. + +Full module race tests, vet, formatting checks and Linux arm64/amd64 builds pass +on Go 1.27.1. A ten-second Authorization fuzz run completed 726,457 executions +without failures. No live AWS verification, resource deployment or GitHub CI run +was performed. Existing CI automatically includes the new package. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 89a9555..9da0a92 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -24,7 +24,7 @@ Keep related decisions small enough to review and implement independently. | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | | [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | | [0015: IAM caller forms](0015-iam-caller-forms.md) | Accepted | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | -| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture accepted; IAM proof implemented under 0017; HTTP selector pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | +| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture accepted; IAM proof and HTTP selector implemented; first-party JWT verification pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | | [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; generator and verifier implemented with local validation | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | -| [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Proposed; awaiting review | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | +| [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | diff --git a/docs/iamproof.md b/docs/iamproof.md index 8b349cc..aae695c 100644 --- a/docs/iamproof.md +++ b/docs/iamproof.md @@ -8,8 +8,8 @@ The verifier never signs using the Lambda role or other server credentials. This is a library-defined proof protocol, not an ordinary SigV4 request to your API, an EKS token, or an OAuth bearer token. The shared HTTP credential selector -and OAuth verification remain separate work. Merely registering the edge adapter -does not authenticate an Authorization header. +is implemented in `authn`; first-party OAuth verification remains separate work. +Merely registering the edge adapter does not authenticate an Authorization header. ## Client @@ -53,8 +53,9 @@ representation grants no application permission. The authentication layer must select exactly one credential, call exactly one verifier, install the returned caller using identity.WithCaller, then authorize -the selected application action. Shared middleware and its HTTP challenges will -be reviewed separately. Never fall back to another verifier after a rejection. +the selected application action. The [authn guide](authn.md) shows this middleware +and the explicit adapter mapping iamproof.ErrInvalidProof to +authn.ErrInvalidCredentials. Never fall back to another verifier after rejection. | Outcome | Inspect with errors.Is | Intended HTTP treatment | | --- | --- | --- | diff --git a/docs/plan.md b/docs/plan.md index 7d94c2a..c96263e 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -61,8 +61,9 @@ The user approved [decision 0001](decisions/0001-gateway-identity.md), supersedi the review's initial gateway-default proposal: no gateway-derived identity by default; `WithGatewayIdentity(true)` opts in for both IAM and JWT/Cognito. Local verification is explicitly composed. Constructor details and producer conflicts -are settled in decisions 0012–0018; the common HTTP authentication-selector API -remains a later review. +are settled in decisions 0012–0018; decision 0019 accepts the now-implemented +common HTTP authentication selector. First-party JWT verification and authz +remain separate reviews. ## Planned package boundaries @@ -286,23 +287,35 @@ claims cannot be silently mixed; unavailable claim fidelity remains explicit. ### 4. Local authentication: Cognito/JWKS and IAM credentials -The next design graph is recorded in proposed +The implemented HTTP authentication design is recorded in accepted [decision 0019](decisions/0019-http-authentication-selector.md): shared HTTP selection, explicit verifier dependencies, header limits, context composition, -error interoperability and challenges. It awaits user review before authn -implementation; the IAM proof package remains independently usable. +error interoperability and challenges. The user approved A1-A6 on 2026-09-16; +authn configuration, selection, errors, middleware and examples are implemented. +The IAM proof package remains independently usable. - [x] Implement the accepted protocol/helper/verifier contract in decision 0017 after the shared foundation and callable buffered adapter. Production iamproof uses SDK core v1.47.0 and STS v1.51.0, rechecked as latest stable on 2026-09-16. [The consumer guide](iamproof.md) covers credential ownership, endpoints, replay/limits, error handling and the remaining integration qualifications. -- Implement the accepted client-helper/server-verifier direction in decisions - 0016/0017. Review the common HTTP authentication selector API and challenges; - preserve the accepted proof, provenance, binding, replay, endpoint and error rules. - Do not promise offline IAM verification or ordinary SigV4-client compatibility. +- [x] Implement the shared HTTP selector and challenges under decision 0019, + preserving the proof, provenance, binding, replay, endpoint and error contracts. + Configuration and verifier functions are explicit, with an SDK-independent + package and consumer-owned action dispatch and authorization. + [The authn guide](authn.md) documents IAM error mapping, header bounds, + cancellation and custom JSON response handling. - Provide exported Go proof generation and consumer documentation with runnable examples showing IAM and OAuth clients calling the same protected route. + IAM generation, selector/dispatcher examples and local mixed-identity contract + tests now exist; the example Bearer verifier is explicitly synthetic. Real + first-party Cognito/OAuth verification and deployed interoperability remain. + +The authn milestone passes full module race tests, vet, formatting checks and +Linux arm64/amd64 builds on Go 1.27.1. Authorization-parser fuzzing completed +726,457 executions without failures. Native in-memory HTTP and raw/typed +Gateway fixtures verify challenges and IAM-helper composition with local STS +fixtures. No live AWS or GitHub CI run was performed for these changes. - Review a verifier requiring explicit issuer/client restrictions; accepting all pool clients must be a deliberate policy. From 7a81fcbc9310b6a6bf221e33686417730f493ce6 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 20:02:58 +0100 Subject: [PATCH 31/44] docs: propose Cognito verification and JWKS cache design --- docs/decisions/0020-cognito-verification.md | 381 ++++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 6 + 3 files changed, 388 insertions(+) create mode 100644 docs/decisions/0020-cognito-verification.md diff --git a/docs/decisions/0020-cognito-verification.md b/docs/decisions/0020-cognito-verification.md new file mode 100644 index 0000000..ef4c346 --- /dev/null +++ b/docs/decisions/0020-cognito-verification.md @@ -0,0 +1,381 @@ +# 0020: Cognito access-token verification and JWKS ownership + +Status: proposed; design review only, no dependent implementation approved. +Reviewed against local code and official sources on 2026-09-16, with Go 1.27.1. + +## Evidence and scope + +The implemented `authn.VerifyFunc` already accepts a credential and returns one +`identity.Caller`. `authn.Authenticator` owns scheme selection, challenges and +context installation. `identity.ParseClaims` retains exact JSON number text; +`identity.NewJWT` validates identity shape but explicitly does not authenticate. +We can add a verifier without changing these boundaries or importing AWS SDKs +into authn. Decisions 0012, 0014, 0016 and 0019 remain in force. + +AWS sources were consulted through the public AWS MCP documentation tools: + +- [Verifying Cognito JWTs](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html): + Cognito uses RS256, separate access/ID signing keys, `kid` selection and JWKS. + Validate issuer, expiry, token use, and the appropriate client claim. AWS + recommends caching keys, refreshing periodically and fetching on an unknown key. +- [Access-token claims](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-the-access-token.html): + `client_id` identifies the app client; `aud` is the API resource when bound. + Scopes and groups are distinct. Do not impose a UUID grammar on `sub`. +- [Resource servers and resource binding](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-define-resource-servers.html): + resource binding is available in supported user authorization flows, not + client-credentials M2M grants or SDK authentication models. Requiring `aud` + universally would exclude legitimate intended consumers. +- [Issuer configuration](https://docs.aws.amazon.com/help-panel/cognito/latest/console/hp-issuer.html) + and [AWS's multi-Region architecture guidance](https://aws.amazon.com/blogs/security/architecting-resilient-authentication-with-amazon-cognito-multi-region-replication/): + both original `cognito-idp` and updated `issuer-cognito-idp` issuer URLs exist. + Updated issuers support replicated key availability; issuer equality remains + exact. Do not bake the original hostname convention into the verifier API. +- [Token revocation](https://docs.aws.amazon.com/cognito/latest/developerguide/token-revocation.html): + revoked tokens can still pass offline signature/expiration verification. + +[JWT BCP](https://www.rfc-editor.org/rfc/rfc8725.html) supports explicit algorithm, +issuer, audience and token-profile restrictions. [JWS](https://www.rfc-editor.org/rfc/rfc7515.html) +defines the compact signing input and critical-header handling. +[JWA](https://datatracker.ietf.org/doc/html/rfc7518#section-3.3) defines RS256, +requires RSA keys of at least 2048 bits, and defines unsigned `n`/`e` encoding. +[JWT](https://www.rfc-editor.org/rfc/rfc7519.html) allows fractional NumericDate +values: the integer-only proposal below is our narrower Cognito policy, not a +universal JWT rule. [JWK](https://www.rfc-editor.org/rfc/rfc7517.html) supplies the +key metadata model; our ambiguity and resource limits are additional policy. + +AWS does not prescribe our cache durations, limits, failure classification or +public Go API. Those are proposed engineering decisions below. + +## Design graph + +```mermaid +flowchart TD + Bearer[Selected Bearer credential] --> Bound[Bound and parse compact JWT with JSON v2] + Bound --> Policy[Reject wrong issuer, client, token use, audience or time] + Policy --> Cache[Resolve kid in this verifier's trusted JWKS snapshot] + Cache -->|Known and fresh| Verify[Go RSA SHA-256 signature verification] + Cache -->|Cold, expired or unknown| Gate[Refresh gate and shared bounded fetch] + Gate -->|Usable successful refresh| Cache + Gate -->|Unavailable or backoff| Unavailable[ErrUnavailable: HTTP 503] + Cache -->|Missing after successful refresh or during success cooldown| Invalid[ErrInvalidCredentials: HTTP 401] + Verify -->|Invalid signature| Invalid + Verify -->|Valid| Recheck[Recheck time and caller cancellation] + Recheck --> Caller[Construct locally verified JWT caller] + Caller --> Middleware[Existing authn installs identity] + Middleware --> Authorization[Consumer action selection and authorization] +``` + +The graph's pre-verification claim checks can only reject. No unverified claim +becomes an identity, permission, network destination or additional trust anchor. +All paths that construct a caller require a verified signature. + +## C1. A concrete Cognito verifier, with explicit trust configuration + +Recommend adding the following to `authn`, retaining stdlib plus identity imports: + +```go +type CognitoClient struct { + ClientID string + Audience string + AllowUnbound bool +} + +type CognitoConfig struct { + Issuer string + Clients []CognitoClient + ClockSkew time.Duration + CacheTTL time.Duration + Timeout time.Duration + Transport http.RoundTripper + MaxTokenBytes int + ClaimsBudget int +} + +func NewCognitoVerifier(cfg CognitoConfig) (*CognitoVerifier, error) +func (v *CognitoVerifier) Verify(ctx context.Context, token string) (identity.Caller, error) +func (v *CognitoVerifier) Warm(ctx context.Context) error +``` + +Construction validates/copies config without I/O. The opaque verifier is safe for +concurrent use; do not copy it after construction. Reuse one instance per issuer +and policy. `Warm` optionally obtains a fresh snapshot before accepting traffic; +it obeys refresh/backoff gates and does not force a refresh of a fresh snapshot. +It establishes key availability, not that any particular token will be accepted. +No public cache interface, global cache, background refresh loop or Close method. + +Require one exact nonempty HTTPS issuer and at least one distinct, nonempty +ClientID. Reject duplicate client policies, even identical ones. Issuer URLs must +have a host and nonempty pool path, with no credentials, query, fragment, escaped +path, dot segments or trailing slash. Derive exactly +`Issuer + "/.well-known/jwks.json"`; do not run discovery or read URLs from tokens. +Do not rewrite or case-fold issuer/audience identifiers. Invalid configuration, +nil contexts and nil/zero verifiers return `ErrInvalidConfiguration`. + +Nil Transport selects the standard default transport; a typed-nil transport is +invalid configuration. Configuration slices are owned copies; a supplied +transport remains a shared caller-owned dependency. + +The issuer is a trusted operator-supplied endpoint, not proof that a URL belongs +to AWS. Avoid an embedded AWS hostname/partition table. This supports either +Cognito issuer format without bringing endpoint SDKs into authn. The default +transport verifies TLS normally; a custom transport is a trusted dependency. +Do not accept an untrusted request value as configuration. + +Initially one exact issuer per verifier and access tokens only. Migration between +issuer URLs requires explicit configuration changes or consumer-owned routing +over a finite configured issuer set; never retry arbitrary issuers after failure. +Generic OAuth providers, ID tokens, multi-issuer routing and discovery need their +own reviewed profiles. The existing VerifyFunc still permits other providers. + +Alternative: pool ID plus region hides issuer evolution; a generic JWT engine +expands the algorithm/claim surface before we have another concrete provider. +Keeping this boundary in authn also lets consumers wire `Bearer: verifier.Verify` +directly, without the error adapter required by iamproof. + +## C2. Client and resource audience are independent restrictions + +Require `token_use == "access"` and an exact allowed `client_id` for every token. +Do not use `aud` as a fallback client ID. No implicit allow-all-clients switch. + +| Client policy | Absent aud | Exact matching aud | Other aud | +| --- | --- | --- | --- | +| Audience set; AllowUnbound false | Reject | Accept | Reject | +| Audience set; AllowUnbound true | Accept | Accept | Reject | +| Audience empty; AllowUnbound true | Accept | Reject | Reject | +| Audience empty; AllowUnbound false | Invalid configuration | Invalid configuration | Invalid configuration | + +Recommend a nonempty string `aud` when present, matching Cognito's single-resource +binding profile. Arrays, null and empty strings fail. General JWT permits arrays; +supporting that wider representation is not required for this initial profile. +Audience identifiers are exact strings; do not equate the configured resource +with the HTTP Host, request URL, client ID or IAM proof audience automatically. + +Use separate client entries for a resource-bound user client and an unbound M2M +client. Enabling AllowUnbound with an Audience deliberately permits both bound +and unbound flows for that client; prefer separate clients where practical. +AllowUnbound never means accept any audience supplied in the token. + +Reasoning: this preserves M2M and SDK user authentication while making the loss +of resource binding visible in configuration. Unbound tokens can otherwise be +replayed across APIs trusting the same issuer/client. Applications should use +dedicated clients and resource-specific scopes and authorize each action. +Successful authentication alone grants no action access. + +Require client_id even when sub exists; permit absent sub, but a present sub must +be nonempty text. Do not infer human/M2M identity from the presence or format of +sub. Preserve scopes/groups/custom claims through the existing identity model, +including its rejection of malformed/conflicting normalized facts. Missing +scopes grant nothing; do not add implicit scope requirements to authentication. + +## C3. Direct JSON v2 plus a narrow, fixed RS256 verification path + +Latest stable module versions were queried and their downloaded source inspected +on 2026-09-16. No dependency was added to this repository: + +| Candidate | Inspected ordinary parsing path | +| --- | --- | +| golang-jwt/jwt/v5 v5.3.1 | `parser.go` imports encoding/json | +| lestrrat-go/jwx/v3 v3.3.0 | `internal/json/stdlib.go` wraps encoding/json; alternative codec is not stdlib JSON v2 | +| go-jose/go-jose/v4 v4.1.5 | `jws.go` uses its own JSON fork | + +Sources: [jwt parser](https://github.com/golang-jwt/jwt/blob/v5.3.1/parser.go), +[jwx codec](https://github.com/lestrrat-go/jwx/blob/v3.3.0/internal/json/stdlib.go), +[go-jose parser](https://github.com/go-jose/go-jose/blob/v4.1.5/jws.go). +These are codec compatibility findings, not claims that the libraries are unsafe. +Using only jwt's RSA Verify method would still leave us owning JSON/JOSE/claims +and add a dependency around the same standard-library RSA operation. + +Recommend direct `encoding/json/v2`/`jsontext` for all three JSON objects: header, +claims and JWKS. Use `crypto/sha256` and +`rsa.VerifyPKCS1v15(key, crypto.SHA256, digest, signature)` for the cryptographic +operation, verified against installed Go documentation. We own the narrow JOSE +validation, not RSA arithmetic. Require independent known-answer tests in +addition to locally generated round trips. This is real security maintenance; +revisit dependency selection before adding algorithms or generic JOSE features. + +Require exactly three nonempty compact segments; canonical unpadded base64url +with no whitespace or ignored CR/LF. Reject duplicate JSON members, invalid UTF-8, +trailing JSON and nonobject header/claims. Verify the original encoded +`header.payload` bytes; never reserialize the signing input. + +Require exact `alg: "RS256"`, a nonempty bounded opaque kid, and absent typ or +`typ: "JWT"`. Reject crit, b64, embedded keys, key/certificate URLs, certificate +chains, compression and nested-content declarations in the token header. This +profile implements none of those optional features. Ignore other noncritical +extensions within the header bound. Never treat kid as a URL or filesystem path. +Do not retry a known key after signature failure or try every key in the set. + +JWKS must be one object with a bounded keys array. Reject duplicate nonempty kids +across the set, including otherwise unsupported key types, to avoid ambiguity. +Ignore explicitly unsupported key types/algorithms/usages; validate every key +eligible for RS256 verification before publishing the snapshot. Require at least +one usable key. For eligible RSA keys require nonempty kid and canonical unsigned +n/e, odd modulus of 2048..4096 bits, odd exponent in 3..2^31-1, no private RSA +parameters, optional alg equal to RS256, optional use equal to sig, and optional +key_ops exactly ["verify"]. Malformed eligible keys invalidate the fetched set; +never partially publish it. Unknown metadata stays bounded and cannot add trust. + +The 4096-bit ceiling, strict typ/aud/header profile and duplicate-kid rejection +are our resource/ambiguity policies, not claimed AWS service limits. A future +Cognito change outside this profile must produce a clear reviewed compatibility +change, not silently broaden algorithms or trust. + +## C4. Time, bounds and error contract + +Require exp and iat; nbf is optional. Recommend plain, nonnegative integer JSON +seconds in 0..253402300799 (through year 9999), parsed from exact number text +without float64. Reject strings, fractions, exponent notation, null and overflow. +AWS describes Unix timestamps and publishes integer examples; it does not +guarantee this lexical restriction. It is an explicit narrow-profile choice to +avoid rounding and date arithmetic surprises, with a compatibility cost. + +Require exp > iat and, when present, nbf < exp. With leeway L, accept only while +now < exp + L, iat <= now + L, and nbf <= now + L. Zero ClockSkew means zero +leeway; allow an explicit 0..5 minutes. Do not default to extending token life or +infer the issuer's configured maximum lifetime. Recheck times after network and +signature work. Use wall time for token dates and elapsed time for local cache +intervals; Lambda thaw must not revive expired tokens or cache entries. + +Proposed bounds/defaults, owned by this module: + +| Setting | Default / accepted range | +| --- | --- | +| MaxTokenBytes | 16 KiB; explicit 1 byte..1 MiB | +| ClaimsBudget | Existing identity 256 KiB; explicit 1 byte..6 MiB | +| Decoded protected header | Fixed 4 KiB | +| kid | Fixed maximum 256 UTF-8 bytes | +| JWKS response | Fixed 64 KiB after decoding; at most 32 keys | +| CacheTTL | 15 minutes; explicit 1 minute..24 hours | +| Timeout | 5 seconds; explicit positive duration up to 1 minute | + +Zero selects defaults except ClockSkew. Negative/out-of-range config fails. +Token size and weighted claims budget are independent. The outer Authorization +limit includes the scheme and can reject a token before this verifier; document +coordinating those limits. Preserve the identity nesting limit (64) and impose +the same maximum JSON nesting depth on header/JWKS, including ignored fields. + +Bad token syntax, limits, profile, claims, signature or definitively missing key +return `ErrInvalidCredentials`; dependency/network/JWKS failures return +`ErrUnavailable`. These are sanitized categories, with no token, claims, kid, +provider body or raw transport error in their error chain. Nil caller on every +failure means the zero identity.Caller value. Preserve the caller's ctx.Err, +never context.Cause. Return a caller with SourceLocallyVerifiedToken only after +all checks. Verify neither installs context nor writes HTTP; the existing +selector handles 401/503 and composition conflicts. + +## C5. Cache freshness, rotation and outage behavior + +Keep an immutable key snapshot per verifier, replaced atomically after a fully +validated successful response. Cache keys only: no token/caller/result cache and +no unbounded per-kid negative-cache map. No union with keys omitted by a new set. + +| State | Recommended behavior | +| --- | --- | +| Known key in fresh snapshot | Verify locally, including during an unrelated refresh failure | +| Cold or expired snapshot | Fetch or join one fetch; unavailable/backoff returns 503 | +| Unknown key in fresh snapshot | Refresh if eligible; otherwise use the outcome rules below | +| Successful refresh still lacks key | 401 | +| Unknown key during cooldown after successful fetch | 401; do not refetch for every random kid | +| Unknown key after failed required refresh | 503 until a later successful refresh resolves availability | +| Known key but bad signature | 401; no refresh triggered by signature failure | +| Refresh fails after snapshot expires | 503; do not use expired keys | + +TTL is measured from completion of a successful validated fetch. Default 15 +minutes bounds how long removed keys can remain trusted without another fetch. +On-demand expiry refresh satisfies periodic freshness without a Lambda-hostile +background timer. Do not serve expired keys during outages initially. This trades +availability for a finite key-trust interval; accepting expired keys would need +a separate explicit policy. Offline verification still is not token revocation. + +Separate three clocks: + +1. Snapshot TTL: whether an existing key is usable. +2. Unknown-key cooldown: no such refresh within 30 seconds of a successful fetch + or another unknown-key refresh attempt. It is per verifier, not per kid. +3. Failure backoff: after failed fetches, wait 5, 10, 20, 40, then 60 seconds, + capped at 60, measured from completion; a success resets the sequence. Apply + this to cold/expired caches too, fixing the old Beakley cold-cache fetch storm. + +When both gates apply, honor the later time. Waiters join an existing fetch +before considering gates; do not launch a queued retry inside one Verify call. +Other requests using still-fresh known keys do not wait on that fetch. Refresh +publication precedes waking waiters, and waiters check the new snapshot and their +own token times. Retain the latest failed-fetch state for missing-key error +classification until a successful fetch replaces it. + +A genuinely new key can be rejected during the 30-second successful-fetch +cooldown. A same-kid replacement can wait until TTL expiry because invalid +signatures do not trigger refresh. These are deliberate bounded rotation +tradeoffs; AWS's documented unknown-kid rotation path is supported. These limits +apply per verifier/process, not across a Lambda fleet; do not claim fleet-wide +rate limiting or an AWS key-retention guarantee. + +## C6. Network and cancellation ownership + +Use an owned http.Client around the supplied concurrent-safe RoundTripper, with +normal default TLS behavior, no cookie jar, redirects or application retries. +Accept HTTP 200 only; bound and close bodies on every path. One GET targets the +configured endpoint; never send credentials/tokens or forward request headers. +Keep cache lifetime an explicit key-trust policy, not a general HTTP cache: +do not extend it from Cache-Control, stale responses, 304 or failed requests. +No initial ETag/conditional-request implementation. + +At most one fetch is in flight per verifier. It has its own background context +and configured timeout, with no caller context values. Each waiter can cancel +independently and immediately return ctx.Err. A canceled waiter must not abort +other waiters' fetch. Recommend letting the shared fetch finish within its bound +even if all waiters leave; it can populate the cache for subsequent requests. +This avoids one initiating request owning everyone's availability and bounds +remaining work without a permanent goroutine or reference-counted cancellation +protocol. Lambda can freeze that work; it is not a durable background job. + +Custom transports must honor context cancellation; arbitrary user code cannot +be forcibly stopped by this library. Do not spawn replacement fetches around a +transport still running after its deadline. Reject late successful publication +after the fetch deadline and return caller cancellation ahead of other outcomes. +The verifier owns neither shutdown nor mutation of a supplied transport. + +## C7. Offline identity and downstream policy + +Document that this verifies signatures and configured token claims, not current +user enablement, sign-out, Cognito revocation or application permissions. Do not +add GetUser/introspection calls implicitly: that would change latency, scope and +M2M compatibility. Future revocation integration is a separate explicit boundary. + +Authentication occurs before action authorization and before streaming commits. +The caller remains an invocation snapshot; no automatic token renewal or +mid-stream reauthentication. Preserve the approved separation of scopes, groups +and application-resolved grants. Never reconcile claims from an unverified token +with an independently asserted gateway identity. + +## Implementation and acceptance after approval + +1. Pure config, compact parsing, exact time/profile validation and JSON v2 JWKS + validation. Add independent RFC RS256 known-answer coverage for the signature + primitive, plus signed Cognito-shaped synthetic fixtures for the full profile. +2. Cache state machine and bounded network ownership. Deterministic tests must + cover cold concurrent success/failure, expiry, rotation/removal, unknown-kid + churn, cooldown/backoff intersections, canceled initiator/other waiters/all + waiters, late transport success and absence of indefinite stale use. +3. Connect Verify/Warm and sanitized categories. Test algorithm confusion, + duplicate/invalid JSON, base64 variants, token-controlled URLs, time bounds, + ambiguous/malformed JWKS, bad signatures without refetch, client/audience + combinations, ID-token rejection and M2M-shaped identities. +4. Replace the illustrative synthetic Bearer verification in integration tests + with actual cryptographic verification against local JWKS fixtures. Retain + real IAM helper/local STS composition and native/raw/typed Gateway coverage. +5. Add runnable configuration examples for resource-bound users and explicitly + unbound M2M clients. Explain revocation, transport ownership, rotation delay, + cache outage behavior, issuer migration and required action authorization. +6. Run race/vet/Linux builds and focused token/JWKS fuzzing. Use Go 1.27 in-memory + HTTP and synctest where appropriate. No live AWS resources or tokens required. + +Claims about deployed interoperability remain qualified until separately tested. +No source files, module dependencies or runtime behavior changed in this review. + +## Resolution + +Awaiting review of C1-C7. In particular: approve the narrow stdlib/JSON v2 RS256 +implementation, explicit per-client unbound allowance, strict profile limits, +15-minute cache with no expired-key fallback, and shared fetch cancellation +ownership before dependent implementation. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 9da0a92..3de231f 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -28,3 +28,4 @@ Keep related decisions small enough to review and implement independently. | [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; generator and verifier implemented with local validation | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | | [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | +| [0020: Cognito verification](0020-cognito-verification.md) | Proposed; design review only | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | diff --git a/docs/plan.md b/docs/plan.md index c96263e..5d5b787 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -317,6 +317,12 @@ Linux arm64/amd64 builds on Go 1.27.1. Authorization-parser fuzzing completed Gateway fixtures verify challenges and IAM-helper composition with local STS fixtures. No live AWS or GitHub CI run was performed for these changes. +- [Decision 0020](decisions/0020-cognito-verification.md) records the proposed + Cognito access-token verifier/JWKS design, reviewed against AWS documentation, + current identity/authn code and current stable Go JWT libraries on 2026-09-16. + It proposes direct JSON v2 plus stdlib RS256, per-client resource binding and + explicit unbound allowances for M2M/SDK flows, and bounded on-demand key caching. + This is a pending design review, not approved or implemented behavior. - Review a verifier requiring explicit issuer/client restrictions; accepting all pool clients must be a deliberate policy. - Keep access-token `client_id`, ID-token audience, and access-token resource From 9d9b69019e70ade3f00d41f5680b094ead2f3199 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 20:53:09 +0100 Subject: [PATCH 32/44] feat: verify Cognito access tokens with bounded JWKS caching --- README.md | 9 +- authn/cognito.go | 167 +++++++++ authn/cognito_cache.go | 133 +++++++ authn/cognito_cache_test.go | 353 +++++++++++++++++++ authn/cognito_example_test.go | 37 ++ authn/cognito_jwks.go | 129 +++++++ authn/cognito_jwks_test.go | 160 +++++++++ authn/cognito_test.go | 363 ++++++++++++++++++++ authn/cognito_token.go | 171 +++++++++ authn/cognito_vector_test.go | 46 +++ authn/doc.go | 6 +- authn/gateway_test.go | 37 +- authn/testdata/cognito-fuzz.json | 1 + docs/authn.md | 15 +- docs/cognito.md | 159 +++++++++ docs/decisions/0020-cognito-verification.md | 42 ++- docs/decisions/README.md | 4 +- docs/plan.md | 40 ++- 18 files changed, 1819 insertions(+), 53 deletions(-) create mode 100644 authn/cognito.go create mode 100644 authn/cognito_cache.go create mode 100644 authn/cognito_cache_test.go create mode 100644 authn/cognito_example_test.go create mode 100644 authn/cognito_jwks.go create mode 100644 authn/cognito_jwks_test.go create mode 100644 authn/cognito_test.go create mode 100644 authn/cognito_token.go create mode 100644 authn/cognito_vector_test.go create mode 100644 authn/testdata/cognito-fuzz.json create mode 100644 docs/cognito.md diff --git a/README.md b/README.md index 7d0da5c..b25fa47 100644 --- a/README.md +++ b/README.md @@ -42,9 +42,12 @@ v1.47.0 and STS v1.51.0, checked on 2026-09-16. See the The shared `authn` selector and required-authentication middleware are implemented. Configure Bearer and/or IAM-proof verifier functions; the [authentication guide](docs/authn.md) shows the required IAM error mapping and -the custom dispatcher/HTTP response boundaries. The first-party Cognito/JWT/JWKS -verifier and authz package remain planned. Applications must supply actual Bearer -verification and authorize each action. No live STS interoperability test has run. +the custom dispatcher/HTTP response boundaries. `authn.NewCognitoVerifier` now +provides RS256 access-token verification with direct JSON v2, explicit issuer/ +client/resource restrictions and bounded JWKS caching; see the +[Cognito guide](docs/cognito.md). Wire its Verify method as the Bearer verifier. +The authz package remains planned; applications must authorize each action. +No live Cognito/STS interoperability test has run. The accepted streaming design now has a private bridge with incremental delivery, backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle diff --git a/authn/cognito.go b/authn/cognito.go new file mode 100644 index 0000000..da4fbd5 --- /dev/null +++ b/authn/cognito.go @@ -0,0 +1,167 @@ +package authn + +import ( + "context" + "net/http" + "net/url" + "path" + "reflect" + "strings" + "sync" + "time" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// CognitoClient restricts one app client independently of its resource audience. +// Audience requires an exact string aud. AllowUnbound additionally permits an +// absent aud; with an empty Audience it permits only absent aud. At least one +// of Audience or AllowUnbound is required. No policy accepts an arbitrary aud. +type CognitoClient struct { + ClientID string + Audience string + AllowUnbound bool +} + +// CognitoConfig supplies explicit trust and resource policy for access tokens. +type CognitoConfig struct { + // Issuer is an exact HTTPS pool URL, without trailing slash, escaped path, + // dot segments, credentials, query or fragment. JWKS is fetched from this + // trusted endpoint plus /.well-known/jwks.json, never from token headers. + Issuer string + // Clients must contain at least one policy with a distinct nonempty ClientID. + Clients []CognitoClient + // ClockSkew permits 0..5 minutes of leeway. Zero does not extend token life. + ClockSkew time.Duration + // CacheTTL defaults to 15 minutes; explicit values allow 1 minute..24 hours. + // Expired keys are not used during outages. No token results are cached. + CacheTTL time.Duration + // Timeout defaults to 5 seconds; explicit values must be positive and <=1 minute. + Timeout time.Duration + // Transport defaults to http.DefaultTransport. A supplied transport remains + // caller-owned, must support concurrency and must honor context cancellation. + Transport http.RoundTripper + // MaxTokenBytes defaults to 16 KiB; explicit values allow 1 byte..1 MiB. + // The separate Authorization field bound also includes the scheme/framing. + MaxTokenBytes int + // ClaimsBudget defaults to identity's 256 KiB weighted allowance. Explicit + // values allow 1 byte..6 MiB, independently of the compact token byte limit. + ClaimsBudget int +} + +// CognitoVerifier verifies a narrow RS256 Cognito access-token profile using +// JSON v2 and standard-library cryptography. It does not check revocation or +// grant permissions. It is safe for concurrent use, but must not be copied. +// Its zero value is unconfigured. Reuse one instance to share its bounded cache. +type CognitoVerifier struct { + cfg CognitoConfig + clients map[string]CognitoClient + http *http.Client + + mu sync.Mutex + snapshot jwksSnapshot + flight *jwksFlight + unknownAfter time.Time + retryAfter time.Time + backoff time.Duration + failed bool +} + +// NewCognitoVerifier validates and copies cfg without I/O. Invalid configuration +// returns a nil verifier and ErrInvalidConfiguration. It trusts the configured +// issuer endpoint; it does not establish that the operator supplied an AWS URL. +func NewCognitoVerifier(cfg CognitoConfig) (*CognitoVerifier, error) { + u, err := url.Parse(cfg.Issuer) + if err != nil || u.Scheme != "https" || u.Hostname() == "" || u.User != nil || u.Opaque != "" { + return nil, ErrInvalidConfiguration + } + if u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || strings.ContainsAny(cfg.Issuer, "?#%") { + return nil, ErrInvalidConfiguration + } + if u.Path == "" || u.Path == "/" || u.RawPath != "" || strings.HasSuffix(u.Path, "/") || path.Clean(u.Path) != u.Path { + return nil, ErrInvalidConfiguration + } + if len(cfg.Clients) == 0 || cfg.ClockSkew < 0 || cfg.ClockSkew > 5*time.Minute || cfg.CacheTTL < 0 || cfg.Timeout < 0 || cfg.MaxTokenBytes < 0 || cfg.MaxTokenBytes > 1<<20 || cfg.ClaimsBudget < 0 || cfg.ClaimsBudget > 6<<20 { + return nil, ErrInvalidConfiguration + } + if cfg.CacheTTL == 0 { + cfg.CacheTTL = 15 * time.Minute + } + if cfg.Timeout == 0 { + cfg.Timeout = 5 * time.Second + } + if cfg.CacheTTL < time.Minute || cfg.CacheTTL > 24*time.Hour || cfg.Timeout > time.Minute { + return nil, ErrInvalidConfiguration + } + if cfg.MaxTokenBytes == 0 { + cfg.MaxTokenBytes = 16 * 1024 + } + if cfg.ClaimsBudget == 0 { + cfg.ClaimsBudget = 256 * 1024 + } + if cfg.Transport == nil { + cfg.Transport = http.DefaultTransport + } + value := reflect.ValueOf(cfg.Transport) + switch value.Kind() { + case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice: + if value.IsNil() { + return nil, ErrInvalidConfiguration + } + } + clients := make(map[string]CognitoClient, len(cfg.Clients)) + for _, client := range cfg.Clients { + if client.ClientID == "" || client.Audience == "" && !client.AllowUnbound { + return nil, ErrInvalidConfiguration + } + if _, duplicate := clients[client.ClientID]; duplicate { + return nil, ErrInvalidConfiguration + } + clients[client.ClientID] = client + } + // Retain only the owned map, never the caller's mutable slice. + cfg.Clients = nil + return &CognitoVerifier{cfg: cfg, clients: clients, http: &http.Client{ + Transport: cfg.Transport, + CheckRedirect: func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + }, + }}, nil +} + +// Verify returns one locally verified caller without installing it in context. +// It accepts access tokens only, with integer exp/iat and optional nbf, and a +// string aud when present. Rejections match ErrInvalidCredentials; unavailable +// JWKS matches ErrUnavailable. Caller cancellation returns ctx.Err, never its +// cause. Every failure returns a zero caller and sanitized diagnostics. +func (v *CognitoVerifier) Verify(ctx context.Context, token string) (identity.Caller, error) { + if v == nil || v.http == nil || ctx == nil { + return identity.Caller{}, ErrInvalidConfiguration + } + if err := ctx.Err(); err != nil { + return identity.Caller{}, err + } + caller, err := v.verify(ctx, token) + if ctx.Err() != nil { + return identity.Caller{}, ctx.Err() + } + return caller, err +} + +// Warm obtains a fresh usable JWKS snapshot, sharing any required fetch. It +// obeys refresh/backoff gates; it does not force refresh, validate a token or +// start a background refresh loop. Failures are ErrUnavailable, caller ctx.Err +// or ErrInvalidConfiguration for a nil context or unconfigured verifier. +func (v *CognitoVerifier) Warm(ctx context.Context) error { + if v == nil || v.http == nil || ctx == nil { + return ErrInvalidConfiguration + } + if err := ctx.Err(); err != nil { + return err + } + _, err := v.key(ctx, "") + if ctx.Err() != nil { + return ctx.Err() + } + return err +} diff --git a/authn/cognito_cache.go b/authn/cognito_cache.go new file mode 100644 index 0000000..61807a4 --- /dev/null +++ b/authn/cognito_cache.go @@ -0,0 +1,133 @@ +package authn + +import ( + "context" + "crypto/rsa" + "io" + "net/http" + "time" +) + +type jwksSnapshot struct { + keys map[string]*rsa.PublicKey + expires time.Time +} + +type jwksFlight struct { + done chan struct{} + ctx context.Context + err error // Published by closing done; never contains provider diagnostics. +} + +type jwksKey struct { + public *rsa.PublicKey + expires time.Time +} + +// An empty kid means Warm. Every path performs at most one fetch/join. A +// successful fetch missing the requested kid must not start a refresh loop. +func (v *CognitoVerifier) key(ctx context.Context, kid string) (jwksKey, error) { + v.mu.Lock() + now := time.Now() + fresh := now.Before(v.snapshot.expires) + if fresh && (kid == "" || v.snapshot.keys[kid] != nil) { + key := jwksKey{public: v.snapshot.keys[kid], expires: v.snapshot.expires} + v.mu.Unlock() + return key, nil + } + flight := v.flight + if flight == nil { + if now.Before(v.retryAfter) || fresh && now.Before(v.unknownAfter) { + err := ErrUnavailable + if fresh && !v.failed { + err = ErrInvalidCredentials + } + v.mu.Unlock() + return jwksKey{}, err + } + fetchCtx, cancel := context.WithTimeout(context.Background(), v.cfg.Timeout) + flight = &jwksFlight{done: make(chan struct{}), ctx: fetchCtx} + v.flight = flight + if fresh { + v.unknownAfter = now.Add(30 * time.Second) + } + go v.refresh(flight, cancel) + } + v.mu.Unlock() + select { + case <-ctx.Done(): + return jwksKey{}, ctx.Err() + case <-flight.ctx.Done(): + // A misbehaving transport may still be running. Retain its flight so + // subsequent callers cannot accumulate replacement fetch goroutines. + // Completion also cancels the fetch context; prefer its published + // result when both signals are ready. + select { + case <-flight.done: + default: + return jwksKey{}, ErrUnavailable + } + case <-flight.done: + } + if flight.err != nil { + return jwksKey{}, ErrUnavailable + } + v.mu.Lock() + defer v.mu.Unlock() + if !time.Now().Before(v.snapshot.expires) { + return jwksKey{}, ErrUnavailable + } + key := v.snapshot.keys[kid] + if kid != "" && key == nil { + return jwksKey{}, ErrInvalidCredentials + } + return jwksKey{public: key, expires: v.snapshot.expires}, nil +} + +func (v *CognitoVerifier) refresh(flight *jwksFlight, cancel context.CancelFunc) { + defer cancel() + keys, err := v.fetchKeys(flight.ctx) + v.mu.Lock() + defer v.mu.Unlock() + now := time.Now() + deadline, _ := flight.ctx.Deadline() + if flight.ctx.Err() != nil || !now.Before(deadline) { + err = ErrUnavailable + } + if err != nil { + v.failed = true + v.backoff = min(max(v.backoff*2, 5*time.Second), time.Minute) + v.retryAfter = now.Add(v.backoff) + flight.err = ErrUnavailable + } else { + // Publish only a complete validated set. Removed keys do not survive + // in a union, and failure never extends the old snapshot's lifetime. + v.snapshot = jwksSnapshot{keys: keys, expires: now.Add(v.cfg.CacheTTL)} + v.unknownAfter = now.Add(30 * time.Second) + v.failed = false + v.backoff = 0 + v.retryAfter = time.Time{} + } + v.flight = nil + close(flight.done) +} + +func (v *CognitoVerifier) fetchKeys(ctx context.Context) (map[string]*rsa.PublicKey, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, v.cfg.Issuer+"/.well-known/jwks.json", nil) + if err != nil { + return nil, ErrUnavailable + } + resp, err := v.http.Do(req) + if err != nil { + return nil, ErrUnavailable + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, ErrUnavailable + } + body, err := io.ReadAll(io.LimitReader(resp.Body, 64*1024+1)) + if err != nil || len(body) > 64*1024 { + return nil, ErrUnavailable + } + return parseJWKS(body) +} diff --git a/authn/cognito_cache_test.go b/authn/cognito_cache_test.go new file mode 100644 index 0000000..d013517 --- /dev/null +++ b/authn/cognito_cache_test.go @@ -0,0 +1,353 @@ +package authn_test + +import ( + "context" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/asteroid-computing/go-lambda-edge/authn" +) + +func TestCognitoColdFetchSharingAndBackoff(t *testing.T) { + key := signingKey(t) + body := keySet(t, publicJWK(key, "key")) + for _, fail := range []bool{false, true} { + synctest.Test(t, func(t *testing.T) { + var calls atomic.Int64 + release := make(chan struct{}) + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + calls.Add(1) + <-release + if fail { + return nil, errors.New("private network diagnostic") + } + return jwksResponse(body), nil + }))) + var wg sync.WaitGroup + results := make(chan error, 64) + for range 64 { + wg.Go(func() { results <- v.Warm(t.Context()) }) + } + synctest.Wait() + if calls.Load() != 1 { + t.Fatalf("concurrent cold requests fetched %d times", calls.Load()) + } + close(release) + wg.Wait() + close(results) + want := error(nil) + if fail { + want = authn.ErrUnavailable + } + for err := range results { + if !errors.Is(err, want) || err != nil && strings.Contains(err.Error(), "private") { + t.Fatalf("shared fetch result %v; want %v", err, want) + } + } + for range 20 { + if err := v.Warm(t.Context()); !errors.Is(err, want) { + t.Fatalf("Warm during cache/backoff: %v", err) + } + } + if calls.Load() != 1 { + t.Fatal("cold failure bypassed backoff or fresh Warm fetched again") + } + if !fail { + return + } + for i, delay := range []time.Duration{5, 10, 20, 40, 60, 60} { + synctest.Sleep(delay*time.Second - time.Nanosecond) + v.Warm(t.Context()) + if calls.Load() != int64(i+1) { + t.Fatal("retry occurred before backoff elapsed") + } + synctest.Sleep(time.Nanosecond) + v.Warm(t.Context()) + if calls.Load() != int64(i+2) { + t.Fatal("retry did not occur at backoff boundary") + } + } + }) + } +} + +func TestCognitoRotationRemovalAndOutage(t *testing.T) { + key := signingKey(t) + synctest.Test(t, func(t *testing.T) { + var calls atomic.Int64 + body := keySet(t, publicJWK(key, "old")) + fail := false + cfg := cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + calls.Add(1) + if fail { + return nil, errors.New("offline") + } + return jwksResponse(body), nil + })) + cfg.CacheTTL = time.Minute + v := cognitoVerifier(t, cfg) + old := signedToken(t, key, "old", accessClaims()) + rotated := signedToken(t, key, "new", accessClaims()) + verifyResult(t, v, t.Context(), old, nil) + body = keySet(t, publicJWK(key, "new")) + for range 10 { + verifyResult(t, v, t.Context(), rotated, authn.ErrInvalidCredentials) + } + if calls.Load() != 1 { + t.Fatal("unknown kid bypassed successful-fetch cooldown") + } + synctest.Sleep(30 * time.Second) + verifyResult(t, v, t.Context(), rotated, nil) + verifyResult(t, v, t.Context(), old, authn.ErrInvalidCredentials) + if calls.Load() != 2 { + t.Fatal("rotation failed to replace snapshot atomically") + } + synctest.Sleep(30 * time.Second) + fail = true + verifyResult(t, v, t.Context(), old, authn.ErrUnavailable) + verifyResult(t, v, t.Context(), rotated, nil) + synctest.Sleep(5 * time.Second) + verifyResult(t, v, t.Context(), old, authn.ErrUnavailable) + if calls.Load() != 3 { + t.Fatal("failure backoff bypassed longer unknown-key cooldown") + } + synctest.Sleep(25 * time.Second) + verifyResult(t, v, t.Context(), rotated, authn.ErrUnavailable) + if calls.Load() != 4 { + t.Fatal("expired keys were used or expiry did not refresh") + } + fail = false + body = keySet(t, publicJWK(key, "new")) + synctest.Sleep(10 * time.Second) + verifyResult(t, v, t.Context(), rotated, nil) + if calls.Load() != 5 { + t.Fatal("recovery did not replace failed state") + } + verifyResult(t, v, t.Context(), old, authn.ErrInvalidCredentials) + }) +} + +func TestCognitoWaiterCancellation(t *testing.T) { + key := signingKey(t) + body := keySet(t, publicJWK(key, "key")) + for _, allCanceled := range []bool{false, true} { + synctest.Test(t, func(t *testing.T) { + var calls atomic.Int64 + release := make(chan struct{}) + type requestKey struct{} + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(r *http.Request) (*http.Response, error) { + calls.Add(1) + if r.Context().Value(requestKey{}) != nil { + t.Error("shared fetch retained a caller context value") + } + select { + case <-r.Context().Done(): + return nil, r.Context().Err() + case <-release: + return jwksResponse(body), nil + } + }))) + ctx, cancel := context.WithCancelCause(context.WithValue(t.Context(), requestKey{}, "private")) + first := make(chan error, 1) + go func() { first <- v.Warm(ctx) }() + synctest.Wait() + otherCtx, cancelOther := context.WithCancel(t.Context()) + defer cancelOther() + second := make(chan error, 1) + go func() { second <- v.Warm(otherCtx) }() + synctest.Wait() + cancel(errors.New("sensitive cancellation cause")) + if err := <-first; err != context.Canceled { + t.Fatalf("canceled waiter returned %v", err) + } + if allCanceled { + cancelOther() + if err := <-second; err != context.Canceled { + t.Fatalf("second canceled waiter returned %v", err) + } + } + close(release) + synctest.Wait() + if !allCanceled { + if err := <-second; err != nil { + t.Fatalf("initiator cancellation disrupted other waiter: %v", err) + } + } + verifyResult(t, v, t.Context(), signedToken(t, key, "key", accessClaims()), nil) + if calls.Load() != 1 { + t.Fatal("completed shared fetch was not retained") + } + verifyResult(t, v, ctx, "malformed", context.Canceled) + }) + } +} + +func TestCognitoFetchDeadlineAndLateTransport(t *testing.T) { + key := signingKey(t) + body := keySet(t, publicJWK(key, "key")) + synctest.Test(t, func(t *testing.T) { + var calls atomic.Int64 + release := make(chan struct{}) + cfg := cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + calls.Add(1) + <-release // Deliberately violates the transport cancellation contract. + return jwksResponse(body), nil + })) + cfg.Timeout = time.Second + v := cognitoVerifier(t, cfg) + start := time.Now() + if err := v.Warm(t.Context()); !errors.Is(err, authn.ErrUnavailable) || time.Since(start) != time.Second { + t.Fatalf("bounded waiter returned %v after %v", err, time.Since(start)) + } + for range 10 { + if err := v.Warm(t.Context()); !errors.Is(err, authn.ErrUnavailable) { + t.Fatal(err) + } + } + if calls.Load() != 1 { + t.Fatal("slow transport accumulated replacement fetches") + } + close(release) + synctest.Wait() + if err := v.Warm(t.Context()); !errors.Is(err, authn.ErrUnavailable) { + t.Fatal("late success was published") + } + synctest.Sleep(5 * time.Second) + if err := v.Warm(t.Context()); err != nil || calls.Load() != 2 { + t.Fatalf("late transport cleanup prevented recovery: %v", err) + } + }) +} + +func TestCognitoFreshKeysDoNotWaitForRefresh(t *testing.T) { + key := signingKey(t) + body := keySet(t, publicJWK(key, "old")) + synctest.Test(t, func(t *testing.T) { + var calls atomic.Int64 + release := make(chan struct{}) + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + if calls.Add(1) > 1 { + <-release + return nil, errors.New("offline") + } + return jwksResponse(body), nil + }))) + old := signedToken(t, key, "old", accessClaims()) + verifyResult(t, v, t.Context(), old, nil) + synctest.Sleep(30 * time.Second) + unknown := signedToken(t, key, "unknown", accessClaims()) + result := make(chan error, 1) + go func() { _, err := v.Verify(t.Context(), unknown); result <- err }() + synctest.Wait() + start := time.Now() + verifyResult(t, v, t.Context(), old, nil) + if time.Now() != start { + t.Fatal("fresh known key waited for unrelated refresh") + } + close(release) + if err := <-result; !errors.Is(err, authn.ErrUnavailable) { + t.Fatalf("failed unknown-key refresh: %v", err) + } + }) +} + +type deadlineBody struct { + ctx context.Context + closed atomic.Bool +} + +func (b *deadlineBody) Read([]byte) (int, error) { + <-b.ctx.Done() + return 0, b.ctx.Err() +} + +func (b *deadlineBody) Close() error { b.closed.Store(true); return nil } + +func TestCognitoBodyDeadline(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var body *deadlineBody + cfg := cognitoConfig(transportFunc(func(r *http.Request) (*http.Response, error) { + body = &deadlineBody{ctx: r.Context()} + return &http.Response{StatusCode: 200, Body: body}, nil + })) + cfg.Timeout = time.Second + v := cognitoVerifier(t, cfg) + start := time.Now() + err := v.Warm(t.Context()) + synctest.Wait() + if !errors.Is(err, authn.ErrUnavailable) || time.Since(start) != time.Second || !body.closed.Load() { + t.Fatalf("body deadline: err=%v elapsed=%v closed=%v", err, time.Since(start), body.closed.Load()) + } + }) +} + +type jwksBody struct { + reader io.Reader + read int + closed bool +} + +func (b *jwksBody) Read(p []byte) (int, error) { + n, err := b.reader.Read(p) + b.read += n + return n, err +} + +func (b *jwksBody) Close() error { b.closed = true; return nil } + +func TestCognitoHTTPBoundAndOwnership(t *testing.T) { + key := signingKey(t) + good := keySet(t, publicJWK(key, "key")) + for _, tc := range []struct { + status int + body string + want error + }{ + {200, good, nil}, {200, good + strings.Repeat(" ", 65536-len(good)), nil}, + {200, strings.Repeat(" ", 1<<20), authn.ErrUnavailable}, + {200, "not JSON", authn.ErrUnavailable}, {302, good, authn.ErrUnavailable}, + {304, good, authn.ErrUnavailable}, {429, good, authn.ErrUnavailable}, {500, good, authn.ErrUnavailable}, + } { + body := &jwksBody{reader: strings.NewReader(tc.body)} + var calls atomic.Int64 + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(r *http.Request) (*http.Response, error) { + calls.Add(1) + if r.URL.String() != cognitoIssuer+"/.well-known/jwks.json" || r.Method != "GET" || len(r.Header) != 0 || r.Body != nil { + t.Error("fetch destination or request contents changed") + } + return &http.Response{StatusCode: tc.status, Header: http.Header{"Location": {"https://attacker.invalid/keys"}}, Body: body}, nil + }))) + err := v.Warm(t.Context()) + if !errors.Is(err, tc.want) || calls.Load() != 1 || !body.closed || body.read > 65537 { + t.Fatalf("status=%d result=%v calls=%d closed=%v bytes=%d", tc.status, err, calls.Load(), body.closed, body.read) + } + } + // The real HTTP transport exercises TLS and response handling locally. + var calls atomic.Int64 + server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + if r.URL.Path != "/pool/.well-known/jwks.json" || r.Header.Get("Authorization") != "" || r.Header.Get("Cookie") != "" { + t.Error("unexpected network request") + } + w.Header().Set("Set-Cookie", "untrusted=never-retained") + io.WriteString(w, good) + })) + defer server.Close() + cfg := cognitoConfig(server.Client().Transport) + cfg.Issuer = server.URL + "/pool" + v := cognitoVerifier(t, cfg) + claims := accessClaims() + claims["iss"] = cfg.Issuer + verifyResult(t, v, t.Context(), signedToken(t, key, "key", claims), nil) + if calls.Load() != 1 { + t.Fatal("real HTTP verification did not fetch keys") + } +} diff --git a/authn/cognito_example_test.go b/authn/cognito_example_test.go new file mode 100644 index 0000000..289b94a --- /dev/null +++ b/authn/cognito_example_test.go @@ -0,0 +1,37 @@ +package authn_test + +import ( + "context" + "fmt" + "net/http" + + "github.com/asteroid-computing/go-lambda-edge/authn" +) + +func ExampleNewCognitoVerifier() { + verifier, err := authn.NewCognitoVerifier(authn.CognitoConfig{ + Issuer: "https://issuer-cognito-idp.eu-west-2.amazonaws.com/eu-west-2_example", + Clients: []authn.CognitoClient{ + {ClientID: "user-app-client", Audience: "https://orders.example"}, + // Cognito M2M grants do not provide resource binding. This permits + // absent aud only; any audience on this client's token is rejected. + {ClientID: "machine-app-client", AllowUnbound: true}, + }, + }) + if err != nil { + fmt.Println(err) + return + } + a, err := authn.New(authn.Config{Bearer: verifier.Verify}) + if err != nil { + fmt.Println(err) + return + } + // Constructors perform no I/O. Verification is lazy; optional Warm(ctx) + // can fetch keys during application initialization. Reuse the verifier. + // Missing credentials are rejected without contacting the issuer. + _, err = a.Authenticate(context.Background(), make(http.Header)) + fmt.Println(err) + // Output: + // authn: missing credentials +} diff --git a/authn/cognito_jwks.go b/authn/cognito_jwks.go new file mode 100644 index 0000000..ae517ad --- /dev/null +++ b/authn/cognito_jwks.go @@ -0,0 +1,129 @@ +package authn + +import ( + "bytes" + "crypto/rsa" + "encoding/json/jsontext" + "encoding/json/v2" + "io" + "math/big" + "unicode/utf8" +) + +// jwtObject validates all JSON, including ignored extension values, before +// allocating the member map. JSON v2 rejects duplicate names and invalid UTF-8. +func jwtObject(data []byte) (map[string]jsontext.Value, error) { + d := jsontext.NewDecoder(bytes.NewReader(data)) + if d.PeekKind() != '{' { + return nil, ErrInvalidCredentials + } + for { + if _, err := d.ReadToken(); err != nil || d.StackDepth() > 64 { + return nil, ErrInvalidCredentials + } + if d.StackDepth() == 0 { + break + } + } + if _, err := d.ReadToken(); err != io.EOF { + return nil, ErrInvalidCredentials + } + var object map[string]jsontext.Value + if err := json.Unmarshal(data, &object); err != nil { + return nil, ErrInvalidCredentials + } + return object, nil +} + +func jwtText(object map[string]jsontext.Value, name string) string { + var s string + if err := json.Unmarshal(object[name], &s); err != nil { + return "" + } + return s +} + +func validKID(kid string) bool { return kid != "" && len(kid) <= 256 && utf8.ValidString(kid) } + +func parseJWKS(data []byte) (map[string]*rsa.PublicKey, error) { + if len(data) > 64*1024 { + return nil, ErrUnavailable + } + object, err := jwtObject(data) + if err != nil { + return nil, ErrUnavailable + } + var entries []map[string]jsontext.Value + if err := json.Unmarshal(object["keys"], &entries); err != nil || len(entries) == 0 || len(entries) > 32 { + return nil, ErrUnavailable + } + keys := make(map[string]*rsa.PublicKey) + seen := make(map[string]bool) + for _, entry := range entries { + kid := jwtText(entry, "kid") + if kid != "" { + if seen[kid] { + return nil, ErrUnavailable + } + seen[kid] = true + } + kty := jwtText(entry, "kty") + if kty == "" { + return nil, ErrUnavailable + } + if kty != "RSA" { + continue + } + unsupported := false + for _, field := range []struct{ name, want string }{{"alg", "RS256"}, {"use", "sig"}} { + if _, present := entry[field.name]; present { + text := jwtText(entry, field.name) + if text == "" { + return nil, ErrUnavailable + } + unsupported = unsupported || text != field.want + } + } + if unsupported { + continue + } + if !validKID(kid) { + return nil, ErrUnavailable + } + if raw, present := entry["key_ops"]; present { + var operations []string + if err := json.Unmarshal(raw, &operations); err != nil || len(operations) != 1 || operations[0] != "verify" { + return nil, ErrUnavailable + } + } + for _, private := range []string{"d", "p", "q", "dp", "dq", "qi", "oth"} { + if _, present := entry[private]; present { + return nil, ErrUnavailable + } + } + n, err := decodeJWTPart(jwtText(entry, "n"), 512) + if err != nil || len(n) < 256 || n[0] == 0 || n[len(n)-1]&1 == 0 { + return nil, ErrUnavailable + } + modulus := new(big.Int).SetBytes(n) + if modulus.BitLen() < 2048 || modulus.BitLen() > 4096 { + return nil, ErrUnavailable + } + e, err := decodeJWTPart(jwtText(entry, "e"), 4) + if err != nil || len(e) == 0 || e[0] == 0 { + return nil, ErrUnavailable + } + var exponent int64 + for _, b := range e { + exponent = exponent<<8 | int64(b) + } + if exponent < 3 || exponent > 1<<31-1 || exponent&1 == 0 { + return nil, ErrUnavailable + } + keys[kid] = &rsa.PublicKey{N: modulus, E: int(exponent)} + } + if len(keys) == 0 { + return nil, ErrUnavailable + } + return keys, nil +} diff --git a/authn/cognito_jwks_test.go b/authn/cognito_jwks_test.go new file mode 100644 index 0000000..cda2462 --- /dev/null +++ b/authn/cognito_jwks_test.go @@ -0,0 +1,160 @@ +package authn_test + +import ( + "encoding/base64" + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + "fmt" + "net/http" + "os" + "strings" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/authn" +) + +func TestCognitoJWKSValidation(t *testing.T) { + key := signingKey(t) + for _, tc := range []struct { + name string + mutate func(map[string]any) + }{ + {"missing_kid", func(k map[string]any) { delete(k, "kid") }}, + {"empty_kid", func(k map[string]any) { k["kid"] = "" }}, + {"large_kid", func(k map[string]any) { k["kid"] = strings.Repeat("x", 257) }}, + {"null_alg", func(k map[string]any) { k["alg"] = nil }}, + {"null_use", func(k map[string]any) { k["use"] = nil }}, + {"missing_n", func(k map[string]any) { delete(k, "n") }}, + {"padded_n", func(k map[string]any) { k["n"] = k["n"].(string) + "=" }}, + {"newline_n", func(k map[string]any) { k["n"] = k["n"].(string) + "\n" }}, + {"leading_zero_n", func(k map[string]any) { + k["n"] = base64.RawURLEncoding.EncodeToString(append([]byte{0}, key.N.Bytes()...)) + }}, + {"small_n", func(k map[string]any) { k["n"] = "AQ" }}, + {"even_n", func(k map[string]any) { + b := key.N.Bytes() + b[len(b)-1] &^= 1 + k["n"] = base64.RawURLEncoding.EncodeToString(b) + }}, + {"oversize_n", func(k map[string]any) { + k["n"] = base64.RawURLEncoding.EncodeToString([]byte(strings.Repeat("x", 513))) + }}, + {"small_e", func(k map[string]any) { k["e"] = "AQ" }}, + {"even_e", func(k map[string]any) { k["e"] = "BA" }}, + {"leading_zero_e", func(k map[string]any) { k["e"] = "AAEAAQ" }}, + {"large_e", func(k map[string]any) { k["e"] = "_____w" }}, + {"noncanonical_e", func(k map[string]any) { k["e"] = "Ax" }}, + {"private", func(k map[string]any) { k["d"] = nil }}, + {"null_operations", func(k map[string]any) { k["key_ops"] = nil }}, + {"signing_operations", func(k map[string]any) { k["key_ops"] = []string{"sign", "verify"} }}, + } { + t.Run(tc.name, func(t *testing.T) { + jwk := publicJWK(key, "key") + tc.mutate(jwk) + // A good key must not hide a malformed eligible key in the same set. + body := keySet(t, publicJWK(key, "good"), jwk) + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil }))) + if err := v.Warm(t.Context()); !errors.Is(err, authn.ErrUnavailable) { + t.Fatalf("malformed JWKS returned %v", err) + } + }) + } + good := publicJWK(key, "key") + goodWire := keySet(t, good) + for _, body := range []string{ + `null`, `[]`, `{}`, `{"keys":null}`, `{"keys":[]}`, `{"keys":[null]}`, `{"keys":[{}]}`, + keySet(t, good, good), keySet(t, good, map[string]any{"kty": "EC", "kid": "key"}), + keySet(t, good) + `{}`, `{"keys":[],"keys":[]}`, `{"keys":[],"ignored":{"x":1,"x":2}}`, + `{"keys":[],"ignored":` + strings.Repeat("[", 64) + "0" + strings.Repeat("]", 64) + `}`, + goodWire[:len(goodWire)-1] + `,"ignored":{"x":1,"x":2}}`, + goodWire[:len(goodWire)-1] + `,"ignored":` + strings.Repeat("[", 64) + "0" + strings.Repeat("]", 64) + `}`, + } { + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil }))) + if err := v.Warm(t.Context()); !errors.Is(err, authn.ErrUnavailable) { + t.Fatalf("invalid key set accepted: %v", err) + } + } + // Unsupported keys do not participate in signature verification. Optional + // metadata may be absent; an opaque kid can contain '=' or URL-like text. + minimal := publicJWK(key, "https://untrusted.invalid/key=") + delete(minimal, "use") + delete(minimal, "alg") + minimal["key_ops"] = []string{"verify"} + body := keySet(t, minimal, map[string]any{"kty": "EC", "kid": "other"}, map[string]any{"kty": "RSA", "kid": "encryption", "use": "enc"}) + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(r *http.Request) (*http.Response, error) { + if r.URL.String() != cognitoIssuer+"/.well-known/jwks.json" { + t.Error("kid became a network destination") + } + return jwksResponse(body), nil + }))) + verifyResult(t, v, t.Context(), signedToken(t, key, minimal["kid"].(string), accessClaims()), nil) + for _, count := range []int{32, 33} { + var entries []map[string]any + for i := range count { + entries = append(entries, publicJWK(key, fmt.Sprint(i))) + } + body := keySet(t, entries...) + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil }))) + want := error(nil) + if count > 32 { + want = authn.ErrUnavailable + } + if err := v.Warm(t.Context()); !errors.Is(err, want) { + t.Fatalf("%d keys: error=%v; want %v", count, err, want) + } + } +} + +func FuzzCognitoToken(f *testing.F) { + valid, body := cognitoFuzzFixture(f) + v := cognitoVerifier(f, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil }))) + if err := v.Warm(f.Context()); err != nil { + f.Fatal(err) + } + verifyResult(f, v, f.Context(), valid, nil) + f.Add(valid) + f.Add("a.b.c") + f.Add("..") + f.Fuzz(func(t *testing.T, token string) { + caller, err := v.Verify(t.Context(), token) + if err == nil && token != valid { + t.Fatal("altered input authenticated without a matching signature") + } + if err != nil && !errors.Is(err, authn.ErrInvalidCredentials) && !errors.Is(err, authn.ErrUnavailable) { + t.Fatalf("unexpected error category: %v", err) + } + if err != nil && caller.Kind() != 0 { + t.Fatal("invalid token returned an identity") + } + }) +} + +func FuzzCognitoJWKS(f *testing.F) { + _, body := cognitoFuzzFixture(f) + f.Add(body) + f.Add(`{"keys":[{"kid":"k","kty":"RSA","n":"AA","e":"AQAB"}]}`) + f.Add(`{"keys":[]}`) + f.Fuzz(func(t *testing.T, body string) { + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil }))) + if err := v.Warm(t.Context()); err != nil && !errors.Is(err, authn.ErrUnavailable) { + t.Fatalf("unexpected key-set error: %v", err) + } + }) +} + +func cognitoFuzzFixture(t testing.TB) (string, string) { + t.Helper() + data, err := os.ReadFile("testdata/cognito-fuzz.json") + if err != nil { + t.Fatal(err) + } + var fixture struct { + Token string `json:"token"` + JWKS jsontext.Value `json:"jwks"` + } + if err := json.Unmarshal(data, &fixture); err != nil { + t.Fatal(err) + } + return fixture.Token, string(fixture.JWKS) +} diff --git a/authn/cognito_test.go b/authn/cognito_test.go new file mode 100644 index 0000000..a55ed9e --- /dev/null +++ b/authn/cognito_test.go @@ -0,0 +1,363 @@ +package authn_test + +import ( + "context" + "crypto" + "crypto/rand" + "crypto/rsa" + "crypto/sha256" + "encoding/base64" + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + "io" + "math/big" + "net/http" + "strings" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +const cognitoIssuer = "https://issuer.example/pool" + +func signingKey(t testing.TB) *rsa.PrivateKey { + t.Helper() + key, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + t.Fatal(err) + } + return key +} + +func encodeJSON(t testing.TB, value any) []byte { + t.Helper() + b, err := json.Marshal(value) + if err != nil { + t.Fatal(err) + } + return b +} + +func publicJWK(key *rsa.PrivateKey, kid string) map[string]any { + return map[string]any{"kty": "RSA", "alg": "RS256", "use": "sig", "kid": kid, + "n": base64.RawURLEncoding.EncodeToString(key.N.Bytes()), + "e": base64.RawURLEncoding.EncodeToString(big.NewInt(int64(key.E)).Bytes())} +} + +func keySet(t testing.TB, entries ...map[string]any) string { + t.Helper() + return string(encodeJSON(t, map[string]any{"keys": entries})) +} + +func accessClaims() map[string]any { + now := time.Now().Unix() + return map[string]any{"iss": cognitoIssuer, "client_id": "app", "sub": "alice", "token_use": "access", + "aud": "https://orders.example", "iat": now, "exp": now + 3600, "scope": "orders.read", "cognito:groups": []string{"operators"}, + "precise": jsontext.Value("9007199254740993")} +} + +func signRawToken(t testing.TB, key *rsa.PrivateKey, header, payload []byte) string { + t.Helper() + input := base64.RawURLEncoding.EncodeToString(header) + "." + base64.RawURLEncoding.EncodeToString(payload) + digest := sha256.Sum256([]byte(input)) + signature, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:]) + if err != nil { + t.Fatal(err) + } + return input + "." + base64.RawURLEncoding.EncodeToString(signature) +} + +func signedToken(t testing.TB, key *rsa.PrivateKey, kid string, claims map[string]any) string { + t.Helper() + return signRawToken(t, key, encodeJSON(t, map[string]any{"alg": "RS256", "kid": kid}), encodeJSON(t, claims)) +} + +func cognitoConfig(transport http.RoundTripper) authn.CognitoConfig { + return authn.CognitoConfig{Issuer: cognitoIssuer, Clients: []authn.CognitoClient{{ClientID: "app", Audience: "https://orders.example"}}, Transport: transport} +} + +func cognitoVerifier(t testing.TB, cfg authn.CognitoConfig) *authn.CognitoVerifier { + t.Helper() + v, err := authn.NewCognitoVerifier(cfg) + if err != nil { + t.Fatal(err) + } + return v +} + +func jwksResponse(body string) *http.Response { + return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(body))} +} + +func verifyResult(t testing.TB, v *authn.CognitoVerifier, ctx context.Context, token string, want error) identity.Caller { + t.Helper() + caller, err := v.Verify(ctx, token) + if !errors.Is(err, want) { + t.Fatalf("Verify error = %v; want %v", err, want) + } + if err != nil { + if caller.Kind() != identity.KindAnonymous { + t.Fatal("verification failure returned an identity") + } + return caller + } + if caller.Kind() != identity.KindJWT || caller.Source() != identity.SourceLocallyVerifiedToken { + t.Fatalf("unexpected verified caller: %v", caller) + } + if identity.FromContext(ctx).Kind() != identity.KindAnonymous { + t.Fatal("Verify installed a caller") + } + return caller +} + +func TestCognitoConfiguration(t *testing.T) { + never := transportFunc(func(*http.Request) (*http.Response, error) { + t.Error("constructor performed I/O") + return nil, errors.New("unexpected I/O") + }) + for _, issuer := range []string{"", "http://issuer.example/pool", "https://issuer.example", "https://issuer.example/", "https://user@issuer.example/pool", "https://issuer.example/pool/", "https://issuer.example/a/../pool", "https://issuer.example/%70ool", "https://issuer.example/pool?", "https://issuer.example/pool#", "https://issuer.example/pool?q=x", "https://issuer.example/pool#x"} { + cfg := cognitoConfig(never) + cfg.Issuer = issuer + if v, err := authn.NewCognitoVerifier(cfg); v != nil || !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Errorf("issuer %q: got %v, %v", issuer, v, err) + } + } + for _, mutate := range []func(*authn.CognitoConfig){ + func(c *authn.CognitoConfig) { c.Clients = nil }, + func(c *authn.CognitoConfig) { c.Clients[0].ClientID = "" }, + func(c *authn.CognitoConfig) { c.Clients[0].Audience = "" }, + func(c *authn.CognitoConfig) { c.Clients = append(c.Clients, c.Clients[0]) }, + func(c *authn.CognitoConfig) { c.ClockSkew = -1 }, + func(c *authn.CognitoConfig) { c.ClockSkew = 5*time.Minute + 1 }, + func(c *authn.CognitoConfig) { c.CacheTTL = time.Second }, + func(c *authn.CognitoConfig) { c.CacheTTL = 24*time.Hour + 1 }, + func(c *authn.CognitoConfig) { c.CacheTTL = -1 }, + func(c *authn.CognitoConfig) { c.Timeout = -1 }, + func(c *authn.CognitoConfig) { c.Timeout = time.Minute + 1 }, + func(c *authn.CognitoConfig) { c.MaxTokenBytes = -1 }, + func(c *authn.CognitoConfig) { c.MaxTokenBytes = 1<<20 + 1 }, + func(c *authn.CognitoConfig) { c.ClaimsBudget = -1 }, + func(c *authn.CognitoConfig) { c.ClaimsBudget = 6<<20 + 1 }, + func(c *authn.CognitoConfig) { c.Transport = (*http.Transport)(nil) }, + func(c *authn.CognitoConfig) { c.Transport = transportFunc(nil) }, + } { + cfg := cognitoConfig(never) + mutate(&cfg) + if v, err := authn.NewCognitoVerifier(cfg); v != nil || !errors.Is(err, authn.ErrInvalidConfiguration) { + t.Errorf("invalid config produced %v, %v", v, err) + } + } + for _, issuer := range []string{"https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_example", "https://issuer-cognito-idp.eu-west-2.amazonaws.com/eu-west-2_example"} { + cfg := cognitoConfig(never) + cfg.Issuer = issuer + cognitoVerifier(t, cfg) + } + for _, v := range []*authn.CognitoVerifier{nil, new(authn.CognitoVerifier)} { + _, err := v.Verify(t.Context(), "x") + if !errors.Is(err, authn.ErrInvalidConfiguration) || !errors.Is(v.Warm(t.Context()), authn.ErrInvalidConfiguration) { + t.Fatal("unconfigured verifier did not fail configuration validation") + } + } + v := cognitoVerifier(t, cognitoConfig(never)) + _, err := v.Verify(nil, "x") + if !errors.Is(err, authn.ErrInvalidConfiguration) || !errors.Is(v.Warm(nil), authn.ErrInvalidConfiguration) { + t.Fatal("nil context did not fail configuration validation") + } +} + +func TestCognitoClientAudienceAndClaims(t *testing.T) { + key := signingKey(t) + body := keySet(t, publicJWK(key, "access-key=")) + for _, tc := range []struct { + name string + policy authn.CognitoClient + aud any + present bool + want error + }{ + {"bound", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example"}, "https://orders.example", true, nil}, + {"bound_missing", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example"}, nil, false, authn.ErrInvalidCredentials}, + {"bound_wrong", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example"}, "elsewhere", true, authn.ErrInvalidCredentials}, + {"mixed_bound", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example", AllowUnbound: true}, "https://orders.example", true, nil}, + {"mixed_unbound", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example", AllowUnbound: true}, nil, false, nil}, + {"mixed_wrong", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example", AllowUnbound: true}, "elsewhere", true, authn.ErrInvalidCredentials}, + {"m2m", authn.CognitoClient{ClientID: "app", AllowUnbound: true}, nil, false, nil}, + {"m2m_wrong", authn.CognitoClient{ClientID: "app", AllowUnbound: true}, "https://orders.example", true, authn.ErrInvalidCredentials}, + {"null", authn.CognitoClient{ClientID: "app", AllowUnbound: true}, nil, true, authn.ErrInvalidCredentials}, + {"empty", authn.CognitoClient{ClientID: "app", AllowUnbound: true}, "", true, authn.ErrInvalidCredentials}, + {"array", authn.CognitoClient{ClientID: "app", Audience: "https://orders.example"}, []string{"https://orders.example"}, true, authn.ErrInvalidCredentials}, + } { + t.Run(tc.name, func(t *testing.T) { + var calls atomic.Int64 + cfg := cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + calls.Add(1) + return jwksResponse(body), nil + })) + cfg.Clients = []authn.CognitoClient{tc.policy} + v := cognitoVerifier(t, cfg) + cfg.Clients[0].ClientID = "mutated-after-construction" + claims := accessClaims() + delete(claims, "sub") // Client-only identities are intentional. + if tc.present { + claims["aud"] = tc.aud + } else { + delete(claims, "aud") + } + caller := verifyResult(t, v, t.Context(), signedToken(t, key, "access-key=", claims), tc.want) + if tc.want != nil && calls.Load() != 0 { + t.Fatal("disallowed audience caused I/O") + } + if tc.want == nil { + jwt, _ := caller.JWT() + value, _ := jwt.Claims().Lookup("precise") + number, _ := value.NumberText() + if number != "9007199254740993" { + t.Fatalf("number fidelity lost: %q", number) + } + } + }) + } + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil }))) + for _, tc := range []struct { + name string + value any + }{ + {"iss", "https://elsewhere.example/pool"}, {"iss", nil}, {"client_id", "other"}, {"client_id", nil}, + {"token_use", "id"}, {"token_use", nil}, {"sub", ""}, {"sub", 42}, {"scope", "read\twrite"}, + {"cognito:groups", "operators"}, {"scp", []string{"inconsistent"}}, + } { + claims := accessClaims() + claims[tc.name] = tc.value + verifyResult(t, v, t.Context(), signedToken(t, key, "access-key=", claims), authn.ErrInvalidCredentials) + } +} + +func TestCognitoTokenSyntaxAndSignature(t *testing.T) { + key := signingKey(t) + var calls atomic.Int64 + v := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + calls.Add(1) + return jwksResponse(keySet(t, publicJWK(key, "key"))), nil + }))) + payload := encodeJSON(t, accessClaims()) + for _, header := range []string{ + `null`, `[]`, `{"alg":"RS256","kid":"key","alg":"RS256"}`, `{"alg":"none","kid":"key"}`, + `{"alg":"HS256","kid":"key"}`, `{"alg":"RS384","kid":"key"}`, `{"alg":"RS256","kid":""}`, + `{"alg":"RS256","kid":"key","typ":null}`, `{"alg":"RS256","kid":"key","typ":"at+jwt"}`, + `{"alg":"RS256","kid":"key","extension":{"x":1,"x":2}}`, + `{"alg":"RS256","kid":"key"} {}`, `{"alg":"RS256","kid":"key","x":"` + string([]byte{0xff}) + `"}`, + } { + verifyResult(t, v, t.Context(), signRawToken(t, key, []byte(header), payload), authn.ErrInvalidCredentials) + } + for _, extension := range []string{"crit", "b64", "jku", "jwk", "x5u", "x5c", "zip", "cty"} { + header := map[string]any{"alg": "RS256", "kid": "key", extension: nil} + verifyResult(t, v, t.Context(), signRawToken(t, key, encodeJSON(t, header), payload), authn.ErrInvalidCredentials) + } + valid := signedToken(t, key, "key", accessClaims()) + parts := strings.Split(valid, ".") + for _, token := range []string{"", ".", "..", valid + ".x", " " + valid, parts[0] + "=." + parts[1] + "." + parts[2], parts[0] + "." + parts[1] + "." + parts[2] + "\n", strings.Join(parts[:2], ".") + ".AA"} { + verifyResult(t, v, t.Context(), token, authn.ErrInvalidCredentials) + } + for _, badPayload := range []string{`null`, `[]`, string(payload) + `{}`, `{"iss":"x","iss":"x"}`, `{"x":"\ud800"}`} { + verifyResult(t, v, t.Context(), signRawToken(t, key, []byte(`{"alg":"RS256","kid":"key"}`), []byte(badPayload)), authn.ErrInvalidCredentials) + } + if calls.Load() != 0 { + t.Fatal("malformed or disallowed token caused I/O") + } + verifyResult(t, v, t.Context(), valid, nil) + // Alter a signed permission claim while retaining the original signature. + changed := strings.Replace(string(payload), "orders.read", "orders.root", 1) + tampered := parts[0] + "." + base64.RawURLEncoding.EncodeToString([]byte(changed)) + "." + parts[2] + verifyResult(t, v, t.Context(), tampered, authn.ErrInvalidCredentials) + verifyResult(t, v, t.Context(), signedToken(t, signingKey(t), "key", accessClaims()), authn.ErrInvalidCredentials) + if calls.Load() != 1 { + t.Fatal("bad signature triggered a key refresh") + } + // Whitespace and member ordering are signed bytes, not a canonical JSON object. + spaced := signRawToken(t, key, []byte("{ \"kid\" : \"key\", \"alg\": \"RS256\", \"typ\": \"JWT\" }"), append([]byte(" \n"), payload...)) + verifyResult(t, v, t.Context(), spaced, nil) + for _, size := range []int{4096, 4097} { + header := `{"alg":"RS256","kid":"key","padding":""}` + header = strings.Replace(header, `"padding":""`, `"padding":"`+strings.Repeat("x", size-len(header))+`"`, 1) + want := error(nil) + if size > 4096 { + want = authn.ErrInvalidCredentials + } + verifyResult(t, v, t.Context(), signRawToken(t, key, []byte(header), payload), want) + } + for _, levels := range []int{63, 64} { + header := `{"alg":"RS256","kid":"key","extension":` + strings.Repeat("[", levels) + "0" + strings.Repeat("]", levels) + `}` + want := error(nil) + if levels == 64 { // The enclosing header object is also a container. + want = authn.ErrInvalidCredentials + } + verifyResult(t, v, t.Context(), signRawToken(t, key, []byte(header), payload), want) + } +} + +func TestCognitoDatesAndBounds(t *testing.T) { + key := signingKey(t) + synctest.Test(t, func(t *testing.T) { + body := keySet(t, publicJWK(key, "key")) + cfg := cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { return jwksResponse(body), nil })) + v := cognitoVerifier(t, cfg) + now := time.Now().Unix() + for _, tc := range []struct { + name string + value any + }{ + {"exp", now}, {"exp", nil}, {"exp", "9999999999"}, {"exp", jsontext.Value("1e10")}, + {"exp", jsontext.Value("9999999999.0")}, {"exp", jsontext.Value("253402300800")}, + {"exp", jsontext.Value("999999999999999999999999")}, {"iat", -1}, {"iat", now + 1}, + {"iat", nil}, {"nbf", now + 1}, {"nbf", nil}, {"nbf", now + 3600}, + } { + claims := accessClaims() + claims[tc.name] = tc.value + verifyResult(t, v, t.Context(), signedToken(t, key, "key", claims), authn.ErrInvalidCredentials) + } + for _, name := range []string{"exp", "iat"} { + claims := accessClaims() + delete(claims, name) + verifyResult(t, v, t.Context(), signedToken(t, key, "key", claims), authn.ErrInvalidCredentials) + } + claims := accessClaims() + claims["nbf"] = now + valid := signedToken(t, key, "key", claims) + verifyResult(t, v, t.Context(), valid, nil) + cfg.ClockSkew = time.Second + leeway := cognitoVerifier(t, cfg) + claims["iat"] = now + 1 + claims["nbf"] = now + 1 + verifyResult(t, leeway, t.Context(), signedToken(t, key, "key", claims), nil) + claims["iat"] = now - 2 + claims["nbf"] = now - 2 + claims["exp"] = now - 1 + verifyResult(t, leeway, t.Context(), signedToken(t, key, "key", claims), authn.ErrInvalidCredentials) + claims["exp"] = now + verifyResult(t, leeway, t.Context(), signedToken(t, key, "key", claims), nil) + for _, limit := range []int{len(valid) - 1, len(valid)} { + cfg.MaxTokenBytes = limit + want := error(nil) + if limit < len(valid) { + want = authn.ErrInvalidCredentials + } + verifyResult(t, cognitoVerifier(t, cfg), t.Context(), valid, want) + } + cfg.ClaimsBudget = 1 + verifyResult(t, cognitoVerifier(t, cfg), t.Context(), valid, authn.ErrInvalidCredentials) + cfg.ClaimsBudget = 0 + cfg.MaxTokenBytes = 0 + cfg.Transport = transportFunc(func(*http.Request) (*http.Response, error) { + time.Sleep(2 * time.Second) + return jwksResponse(body), nil + }) + cfg.ClockSkew = 0 + claims["exp"] = now + 1 + verifyResult(t, cognitoVerifier(t, cfg), t.Context(), signedToken(t, key, "key", claims), authn.ErrInvalidCredentials) + }) +} diff --git a/authn/cognito_token.go b/authn/cognito_token.go new file mode 100644 index 0000000..ebc3690 --- /dev/null +++ b/authn/cognito_token.go @@ -0,0 +1,171 @@ +package authn + +import ( + "context" + "crypto" + "crypto/rsa" + "crypto/sha256" + "encoding/base64" + "encoding/json/jsontext" + "strconv" + "strings" + "time" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func (v *CognitoVerifier) verify(ctx context.Context, token string) (identity.Caller, error) { + if len(token) > v.cfg.MaxTokenBytes { + return identity.Caller{}, ErrInvalidCredentials + } + headerText, rest, ok := strings.Cut(token, ".") + if !ok { + return identity.Caller{}, ErrInvalidCredentials + } + payloadText, signatureText, ok := strings.Cut(rest, ".") + if !ok { + return identity.Caller{}, ErrInvalidCredentials + } + headerBytes, err := decodeJWTPart(headerText, 4096) + if err != nil { + return identity.Caller{}, err + } + header, err := jwtObject(headerBytes) + if err != nil || jwtText(header, "alg") != "RS256" || !validKID(jwtText(header, "kid")) { + return identity.Caller{}, ErrInvalidCredentials + } + if _, present := header["typ"]; present && jwtText(header, "typ") != "JWT" { + return identity.Caller{}, ErrInvalidCredentials + } + for _, name := range []string{"crit", "b64", "jku", "jwk", "x5u", "x5c", "zip", "cty"} { + if _, present := header[name]; present { + return identity.Caller{}, ErrInvalidCredentials + } + } + payload, err := decodeJWTPart(payloadText, v.cfg.MaxTokenBytes) + if err != nil { + return identity.Caller{}, err + } + signature, err := decodeJWTPart(signatureText, 512) + if err != nil || len(signature) < 256 { + return identity.Caller{}, ErrInvalidCredentials + } + claims, err := identity.ParseClaims(jsontext.Value(payload), identity.WithClaimsBudget(v.cfg.ClaimsBudget)) + if err != nil || claimText(claims, "iss") != v.cfg.Issuer || claimText(claims, "token_use") != "access" { + return identity.Caller{}, ErrInvalidCredentials + } + client, allowed := v.clients[claimText(claims, "client_id")] + if !allowed { + return identity.Caller{}, ErrInvalidCredentials + } + if aud, present := claims.Lookup("aud"); present { + text, ok := aud.Text() + if !ok || text == "" || text != client.Audience { + return identity.Caller{}, ErrInvalidCredentials + } + } else if !client.AllowUnbound { + return identity.Caller{}, ErrInvalidCredentials + } + dates, err := readTokenDates(claims) + if err != nil || !dates.valid(time.Now(), v.cfg.ClockSkew) { + return identity.Caller{}, ErrInvalidCredentials + } + if err := ctx.Err(); err != nil { + return identity.Caller{}, err + } + key, err := v.key(ctx, jwtText(header, "kid")) + if err != nil { + return identity.Caller{}, err + } + if !verifyRS256(key.public, token[:len(headerText)+1+len(payloadText)], signature) || !dates.valid(time.Now(), v.cfg.ClockSkew) { + return identity.Caller{}, ErrInvalidCredentials + } + caller, err := identity.NewJWT(claims, identity.SourceLocallyVerifiedToken) + if err != nil { + return identity.Caller{}, ErrInvalidCredentials + } + // A process pause after key lookup must not allow a now-expired key or + // token to establish an identity on resumption. + now := time.Now() + if !dates.valid(now, v.cfg.ClockSkew) { + return identity.Caller{}, ErrInvalidCredentials + } + if !now.Before(key.expires) { + return identity.Caller{}, ErrUnavailable + } + return caller, nil +} + +func verifyRS256(key *rsa.PublicKey, input string, signature []byte) bool { + if len(signature) != key.Size() { + return false + } + digest := sha256.Sum256([]byte(input)) + return rsa.VerifyPKCS1v15(key, crypto.SHA256, digest[:], signature) == nil +} + +func decodeJWTPart(text string, limit int) ([]byte, error) { + if text == "" || len(text) > base64.RawURLEncoding.EncodedLen(limit) { + return nil, ErrInvalidCredentials + } + // Strict decoding checks unused bits but still ignores CR/LF. Check the + // alphabet first so whitespace, padding and extra segments cannot disappear. + for i := range len(text) { + c := text[i] + if !(c >= 'A' && c <= 'Z' || c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '-' || c == '_') { + return nil, ErrInvalidCredentials + } + } + b, err := base64.RawURLEncoding.Strict().DecodeString(text) + if err != nil || len(b) > limit { + return nil, ErrInvalidCredentials + } + return b, nil +} + +func claimText(claims identity.Claims, name string) string { + v, _ := claims.Lookup(name) + s, _ := v.Text() + return s +} + +type tokenDates struct { + exp time.Time + iat time.Time + nbf time.Time +} + +func readTokenDates(claims identity.Claims) (tokenDates, error) { + var dates tokenDates + for _, field := range []struct { + name string + dest *time.Time + }{{"exp", &dates.exp}, {"iat", &dates.iat}, {"nbf", &dates.nbf}} { + value, present := claims.Lookup(field.name) + if !present && field.name == "nbf" { + continue + } + text, ok := value.NumberText() + if !ok || len(text) == 0 || len(text) > 12 { + return tokenDates{}, ErrInvalidCredentials + } + for i := range len(text) { + if text[i] < '0' || text[i] > '9' { + return tokenDates{}, ErrInvalidCredentials + } + } + seconds, err := strconv.ParseInt(text, 10, 64) + if err != nil || seconds > 253402300799 { + return tokenDates{}, ErrInvalidCredentials + } + *field.dest = time.Unix(seconds, 0) + } + if !dates.iat.Before(dates.exp) || !dates.nbf.IsZero() && !dates.nbf.Before(dates.exp) { + return tokenDates{}, ErrInvalidCredentials + } + return dates, nil +} + +func (d tokenDates) valid(now time.Time, skew time.Duration) bool { + return now.Before(d.exp.Add(skew)) && !d.iat.After(now.Add(skew)) && (d.nbf.IsZero() || !d.nbf.After(now.Add(skew))) +} diff --git a/authn/cognito_vector_test.go b/authn/cognito_vector_test.go new file mode 100644 index 0000000..0ea57ce --- /dev/null +++ b/authn/cognito_vector_test.go @@ -0,0 +1,46 @@ +package authn + +import ( + "encoding/base64" + "testing" +) + +// Public key, encoded signing input and signature from RFC 7515 Appendix A.2: +// https://www.rfc-editor.org/rfc/rfc7515.html#appendix-A.2 +// This independently published vector tests the primitive, not our Cognito +// profile: the original RFC token has neither a Cognito issuer nor access claims. +func TestRS256RFC7515Vector(t *testing.T) { + const n = "ofgWCuLjybRlzo0tZWJjNiuSfb4p4fAkd_wWJcyQoTbji9k0l8W26mPddx" + + "HmfHQp-Vaw-4qPCJrcS2mJPMEzP1Pt0Bm4d4QlL-yRT-SFd2lZS-pCgNMs" + + "D1W_YpRPEwOWvG6b32690r2jZ47soMZo9wGzjb_7OMg0LOL-bSf63kpaSH" + + "SXndS5z5rexMdbBYUsLA9e-KXBdQOS-UTo7WTBEMa2R2CapHg665xsmtdV" + + "MTBQY4uDZlxvb3qCo5ZwKh9kG4LT6_I5IhlJH7aGhyxXFvUK-DWNmoudF8" + + "NAco9_h9iaGNj8q2ethFkMLs91kzk2PAcDTW9gb54h4FRWyuXpoQ" + const input = "eyJhbGciOiJSUzI1NiJ9." + + "eyJpc3MiOiJqb2UiLA0KICJleHAiOjEzMDA4MTkzODAsDQogImh0dHA6Ly9leGFt" + + "cGxlLmNvbS9pc19yb290Ijp0cnVlfQ" + const signature = "cC4hiUPoj9Eetdgtv3hF80EGrhuB__dzERat0XF9g2VtQgr9PJbu3XOiZj5RZmh7" + + "AAuHIm4Bh-0Qc_lF5YKt_O8W2Fp5jujGbds9uJdbF9CUAr7t1dnZcAcQjbKBYNX4" + + "BAynRFdiuB--f_nZLgrnbyTyWzO75vRK5h6xBArLIARNPvkSjtQBMHlb1L07Qe7K" + + "0GarZRmB_eSN9383LcOLn6_dO--xi12jzDwusC-eOkHWEsqtFZESc6BfI7noOPqv" + + "hJ1phCnvWh6IeYI2w9QOYEUipUTI8np6LbgGY9Fs98rqVt5AXLIhWkWywlVmtVrB" + + "p0igcN_IoypGlUPQGe77Rw" + keys, err := parseJWKS([]byte(`{"keys":[{"kty":"RSA","kid":"rfc","n":"` + n + `","e":"AQAB"}]}`)) + if err != nil { + t.Fatal(err) + } + sig, err := base64.RawURLEncoding.DecodeString(signature) + if err != nil { + t.Fatal(err) + } + if !verifyRS256(keys["rfc"], input, sig) { + t.Fatal("independent RS256 known-answer vector rejected") + } + if verifyRS256(keys["rfc"], input+"A", sig) || verifyRS256(keys["rfc"], input, sig[1:]) { + t.Fatal("altered RFC vector accepted") + } + sig[10] ^= 1 + if verifyRS256(keys["rfc"], input, sig) { + t.Fatal("altered signature accepted") + } +} diff --git a/authn/doc.go b/authn/doc.go index e5b0768..788e461 100644 --- a/authn/doc.go +++ b/authn/doc.go @@ -1,7 +1,9 @@ // Package authn selects one configured Bearer or EdgeIAM verifier from an HTTP // Authorization header and transports the verified caller to protected handlers. -// It does not implement token verification, authorize actions, or infer trust -// from a gateway, a TLS field, an action header or forwarded headers. +// NewCognitoVerifier provides local RS256 Cognito access-token verification with +// direct JSON v2 and bounded JWKS caching. It does not check token revocation, +// authorize actions, or infer trust from a gateway, a TLS field, an action header +// or forwarded headers. // // Use New to configure explicit verifier functions. Authenticate supports custom // HTTP pipelines without writing responses or installing context. Handler adds diff --git a/authn/gateway_test.go b/authn/gateway_test.go index 15cc094..fc96ee7 100644 --- a/authn/gateway_test.go +++ b/authn/gateway_test.go @@ -65,25 +65,14 @@ func mapProofErrors(v *iamproof.Verifier) authn.VerifyFunc { func TestAuthenticationAcrossTransports(t *testing.T) { proofVerifier, proof, requests := localProofVerifier(t) - jwt := jwtCaller(t, identity.SourceLocallyVerifiedToken) - a := authenticator(t, authn.Config{ - Bearer: func(_ context.Context, token string) (identity.Caller, error) { - switch token { - case "synthetic-jwt": - return jwt, nil - case "unavailable": - return identity.Caller{}, authn.ErrUnavailable - default: - return identity.Caller{}, authn.ErrInvalidCredentials - } - }, - IAMProof: mapProofErrors(proofVerifier), - }) + key := signingKey(t) + jwt := signedToken(t, key, "integration", accessClaims()) + keys := keySet(t, publicJWK(key, "integration")) selector, err := edge.NewActionHeader("Action") if err != nil { t.Fatal(err) } - h := handler(t, a, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + dispatcher := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { action, err := selector.Parse(r.Header) if err != nil { http.Error(w, "invalid action", 400) @@ -96,7 +85,7 @@ func TestAuthenticationAcrossTransports(t *testing.T) { } if jwt, ok := caller.JWT(); ok { sub, _ := jwt.Subject() - allowed = jwt.Issuer() == "https://issuer.example" && sub == "alice" + allowed = jwt.Issuer() == cognitoIssuer && sub == "alice" } // Deny by default and authorize this exact action before executing it. if !allowed || action != "orders.read" { @@ -107,7 +96,7 @@ func TestAuthenticationAcrossTransports(t *testing.T) { t.Error("middleware lost extra consumer header") } w.WriteHeader(204) - })) + }) for _, tc := range []struct { name string values []string @@ -116,7 +105,7 @@ func TestAuthenticationAcrossTransports(t *testing.T) { challenges []string }{ {name: "missing", status: 401, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, - {name: "bearer_success", values: []string{"Bearer synthetic-jwt"}, status: 204}, + {name: "bearer_success", values: []string{"Bearer " + jwt}, status: 204}, {name: "iam_success", values: []string{"EdgeIAM " + proof.Value()}, status: 204}, {name: "bearer_rejected", values: []string{"Bearer invalid"}, status: 401, challenges: []string{`Bearer realm="edge", error="invalid_token"`}}, {name: "iam_rejected", values: []string{"EdgeIAM v2.invalid"}, status: 401, challenges: []string{`EdgeIAM realm="edge"`}}, @@ -125,12 +114,20 @@ func TestAuthenticationAcrossTransports(t *testing.T) { {name: "combined", values: []string{"Bearer synthetic-jwt,EdgeIAM invalid"}, status: 400, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, {name: "malformed", values: []string{"Bearer a=b"}, status: 400, challenges: []string{`Bearer realm="edge", error="invalid_request"`, `EdgeIAM realm="edge"`}}, {name: "unsupported", values: []string{"Basic abc"}, status: 401, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, - {name: "dependency", values: []string{"Bearer unavailable"}, status: 503}, - {name: "authz_denial", values: []string{"Bearer synthetic-jwt"}, action: "orders.delete", status: 403}, + {name: "dependency", values: []string{"Bearer " + jwt}, status: 503}, + {name: "authz_denial", values: []string{"Bearer " + jwt}, action: "orders.delete", status: 403}, {name: "iam_authz_denial", values: []string{"EdgeIAM " + proof.Value()}, action: "orders.delete", status: 403}, {name: "field_size", values: []string{"Bearer " + strings.Repeat("A", 16384)}, status: 431}, } { t.Run(tc.name, func(t *testing.T) { + verifier := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + if tc.name == "dependency" { + return nil, errors.New("synthetic JWKS outage") + } + return jwksResponse(keys), nil + }))) + a := authenticator(t, authn.Config{Bearer: verifier.Verify, IAMProof: mapProofErrors(proofVerifier)}) + h := handler(t, a, dispatcher) for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { t.Run(format, func(t *testing.T) { action := tc.action diff --git a/authn/testdata/cognito-fuzz.json b/authn/testdata/cognito-fuzz.json new file mode 100644 index 0000000..252eb37 --- /dev/null +++ b/authn/testdata/cognito-fuzz.json @@ -0,0 +1 @@ +{"note":"Synthetic fuzz fixture only. Fixed dates intentionally exercise lifetime boundaries; never use these as production token lifetime guidance. No private key is retained.","token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImZ1enotb25seSJ9.eyJpc3MiOiJodHRwczovL2lzc3Vlci5leGFtcGxlL3Bvb2wiLCJjbGllbnRfaWQiOiJhcHAiLCJ0b2tlbl91c2UiOiJhY2Nlc3MiLCJzdWIiOiJhbGljZSIsImF1ZCI6Imh0dHBzOi8vb3JkZXJzLmV4YW1wbGUiLCJpYXQiOjAsImV4cCI6MjUzNDAyMzAwNzk5fQ.PxlSme6T71zdwMEY3626Dq0ajFW0DlmQwSZgZfj_J9Qk7zGhO1sKGXvW1BuPHFy_gtlUXdFE8pIuFf1m2gyJmo5aC68ro_dpgi4stvE4cuMoljuNMztFHh0-pSRK8J3145z-c6W5lWb1PGCwe_DV1yFubx_4UeTRmImPhh9ypImUEqEWkdZEMat-Kij44qTmZYBXbngN_ukmbx8xwg60gybI6FIucAQitFrrSmV3b5REwMpaycAAmK6IPgV5eFqyji7U_pFSJuPKPzCA0TCPiCgwpVO7tlmeb9yqblhfsm03v6IGZpKRz5vNZExR_5LgKypXXS3U_x6bhe1qcmzMWQ","jwks":{"keys":[{"use":"sig","n":"vNzBE2TXj4CHdsABUYWvzUZODXbbwTd1ZvnTaA-b7dE5giKXqSxCgvDMnePqYozOXJgFRSaJu4txRbETgblhIjzdqRiBMxMUfjR9Jco0YZuUUQH-FiYFvuVhtJ-olWp9xh_15Ig1VeWjuGCgGNw_w9J2jpsRtHwR9PxjqVO5TceNo1GaFYVG5dZt3zSrx8tuHbzBTAzjkl2gfxcGrHznEk-7l053-KuPIjMiUwToAjYy9jZZ6s6HPCZK1SA85ATOC2PS1fSInmVVL_cFUb0W22b6sz6wgLJWT2BSkpXtpd-C2vB4GbVQsA_Id1Ii9cGgP7I3UD07Z-gecUIg-tXmyQ","e":"AQAB","kid":"fuzz-only","kty":"RSA","alg":"RS256"}]}} diff --git a/docs/authn.md b/docs/authn.md index 5764bdc..5964656 100644 --- a/docs/authn.md +++ b/docs/authn.md @@ -5,10 +5,12 @@ chooses exactly one configured Bearer or EdgeIAM verifier, checks the returned caller and, when used as middleware, installs it in the request context. It works with ordinary Go HTTP servers and the root edge adapter's raw/typed entry points. -It does not implement JWT verification or grant application permissions. The -first-party Cognito/JWT/JWKS verifier and the authz package remain future work. -Supply your own fully verifying Bearer function today, use IAM proofs alone, -or configure both. Default gateway identity remains disabled. +`authn.NewCognitoVerifier` implements local RS256 Cognito access-token verification +with direct JSON v2 and bounded JWKS caching. Supply its Verify method, another +fully verifying Bearer function, IAM proofs alone, or both mechanisms. See the +[Cognito guide](cognito.md) for issuer/client/audience restrictions and cache +ownership. The authz package remains future work; applications must authorize +actions. Default gateway identity remains disabled. ## Configure both mechanisms @@ -25,7 +27,7 @@ verifyIAM := func(ctx context.Context, token string) (identity.Caller, error) { return caller, err } authenticator, err := authn.New(authn.Config{ - Bearer: verifyBearer, // Your actual verification function. + Bearer: verifyBearer, // For Cognito: cognitoVerifier.Verify. IAMProof: verifyIAM, Realm: "orders", }) @@ -144,6 +146,7 @@ scheme headers do not themselves establish trust. Local tests cover native in-memory HTTP, raw REST/HTTP API payloads and typed V1/V2 methods, including challenge combination and actual IAM helper/verifier -composition against synthetic STS responses. No live AWS service or deployment +composition against synthetic STS responses, plus actual RS256 verification +against local JWKS fixtures. No live AWS service or deployment was used. [Decision 0019](decisions/0019-http-authentication-selector.md) records the accepted contract and its official documentation sources. diff --git a/docs/cognito.md b/docs/cognito.md new file mode 100644 index 0000000..dcaa2d6 --- /dev/null +++ b/docs/cognito.md @@ -0,0 +1,159 @@ +# Cognito access-token verification + +`authn.NewCognitoVerifier` provides local RS256 verification using Go 1.27's +`encoding/json/v2`/`jsontext` and standard-library RSA. It returns an immutable +JWT caller with `SourceLocallyVerifiedToken`. It performs no AWS account API +calls and adds no SDK or JWT-library dependency. + +## Configure explicit issuer, clients and resource binding + +```go +verifier, err := authn.NewCognitoVerifier(authn.CognitoConfig{ + Issuer: "https://issuer-cognito-idp.eu-west-2.amazonaws.com/eu-west-2_example", + Clients: []authn.CognitoClient{ + {ClientID: "user-app-client", Audience: "https://orders.example"}, + {ClientID: "machine-app-client", AllowUnbound: true}, + }, +}) +if err != nil { + return err +} +authenticator, err := authn.New(authn.Config{Bearer: verifier.Verify}) +if err != nil { + return err +} +protected, err := authenticator.Handler(dispatcher) +if err != nil { + return err +} +``` + +Use `protected` with an ordinary HTTP server or `edge.New`. Include IAMProof +with the explicit error adapter in [the HTTP authentication guide](authn.md) to +accept IAM and Cognito credentials on the same route. Authentication must precede +action authorization and any response/stream commitment. Gateway identity stays +disabled for these routes; there is no merge or fallback between producers. + +Issuer is the exact configured Cognito pool issuer URL. Both original +`cognito-idp` and updated `issuer-cognito-idp` formats work. Supply an HTTPS URL +with a pool path and no trailing slash, escaped path, dot segments, credentials, +query or fragment. The library appends `/.well-known/jwks.json` to this trusted +configuration. It never uses a token's issuer or JOSE URLs to discover a server. +The constructor does not prove that your configured URL belongs to AWS. + +Every token must be an access token from an explicitly listed client. ID tokens +are rejected. `client_id` identifies the app client; `aud` identifies the bound +resource. They are never interchangeable: + +| Policy | Absent aud | Matching string aud | Other aud | +| --- | --- | --- | --- | +| Audience set | Reject | Accept | Reject | +| Audience set, AllowUnbound true | Accept | Accept | Reject | +| Audience empty, AllowUnbound true | Accept | Reject | Reject | + +Audience empty with AllowUnbound false is invalid configuration. Null, empty or +array audiences fail verification. Missing sub is allowed when client_id is +valid; sub is not assumed to be a UUID or evidence of a particular grant flow. + +Cognito resource binding is unavailable for client-credentials M2M grants and SDK +authentication models. AllowUnbound makes that exception explicit per client. +Prefer dedicated app clients and resource-specific scopes for APIs accepting +unbound tokens: other APIs trusting that same issuer/client could accept them. +Scopes and Cognito groups remain separate identity facts; your dispatcher must +authorize the exact selected action. A valid token alone grants nothing. + +## Lifecycle and key cache + +Construct once and reuse the verifier across concurrent requests/invocations. +Construction performs no I/O. `Warm(ctx)` optionally fetches keys before serving; +otherwise Verify fetches lazily. Warm is not a forced refresh or a token check. +It returns immediately when a usable snapshot is fresh. + +The cache stores only keys, never tokens or verification results. A successful +fetch atomically replaces the complete snapshot, including removal of omitted +keys. Known fresh keys verify locally even while another refresh is failing. +Expired keys are never used during an outage. + +- CacheTTL defaults to 15 minutes, configurable from 1 minute through 24 hours. + It starts at successful fetch completion. No autonomous refresh loop runs. +- An unknown kid can trigger a fetch, with a 30-second cooldown from the last + success or unknown-key attempt. Random kid values do not grow a negative cache. +- Failed fetches back off for 5, 10, 20, 40, then 60 seconds, capped at 60. + This includes cold-cache failures. Where both gates apply, the later wins. +- Concurrent callers share at most one fetch. Each call fetches/joins at most + once. A known-key signature failure never triggers an additional fetch. + +A newly rotated key can be rejected during a successful-fetch cooldown. A +same-kid replacement may wait until TTL expiry. These limits are per verifier, +not fleet-wide AWS rate controls. Shorter TTL reduces the window in which a +removed key remains locally trusted and increases fetch frequency. + +Cancellation stops an individual caller's wait without canceling other waiters. +The shared fetch uses its own bounded context and no request context values. +It may finish and populate the cache after all waiters leave; Lambda can freeze +that work, so it is not a durable background job. There is no Close method. + +Timeout defaults to 5 seconds and accepts a positive duration up to 1 minute. +It covers headers, body and key validation. Redirects, cookie storage and +application retries are disabled; only HTTP 200 is accepted. Cache lifetime is +explicit key-trust policy, not an HTTP Cache-Control/ETag cache. + +Transport defaults to http.DefaultTransport with normal TLS verification. A +custom RoundTripper is a trusted, caller-owned dependency: it must support +concurrency and honor context cancellation, including body reads. The library +cannot kill arbitrary blocked Go code. It bounds waiter duration and retains a +still-running fetch to prevent accumulating replacement goroutines; late success +after its deadline cannot populate the cache. It does not mutate or close the +supplied transport. + +## Token profile, limits and failures + +Only compact RS256 tokens are supported. JSON is strict about duplicate members, +UTF-8, trailing data and a maximum depth of 64 containers. Base64url must be +canonical and unpadded. Optional typ must be JWT; critical headers, unencoded +payloads, token-supplied keys/URLs/certificate chains, compression and nested +content declarations are rejected. kid is an opaque identifier, never a URL. + +exp and iat are required plain nonnegative integer JSON seconds; nbf is optional. +Values must fit through year 9999. Fractions, exponents and string dates are +rejected. Require exp > iat and nbf < exp when present, and check expiration, +future iat and nbf both before and after expensive work. ClockSkew defaults to +zero and permits explicit leeway up to five minutes. These profile restrictions +are narrower than general JWT, not additional claims about AWS service limits. + +MaxTokenBytes defaults to 16 KiB and accepts 1 byte..1 MiB. ClaimsBudget defaults +to the identity package's weighted 256 KiB allowance, configurable up to 6 MiB. +The separate authn MaxAuthorizationBytes includes the scheme and spaces: raise +both where appropriate. Gateway/server total header limits still apply. + +Fixed bounds are a 4 KiB decoded protected header, a 256-byte kid, a 64 KiB JWKS +response after transport decoding, at most 32 keys, and 2048..4096-bit RSA keys. +Duplicate nonempty kids are rejected even across unsupported key types. +Malformed eligible signing keys invalidate the entire fetched set; previously +cached keys remain usable only until their original expiry. No indefinite union +of removed keys or partial publication of malformed sets occurs. + +| Verify outcome | Category | Existing middleware response | +| --- | --- | --- | +| Invalid syntax, profile, claims, signature or definitively absent key | ErrInvalidCredentials | 401, Bearer invalid_token | +| Missing key during successful-fetch cooldown | ErrInvalidCredentials | 401; rotation can be briefly delayed | +| Network/JWKS failure, expired cache outage, or failed required refresh/backoff | ErrUnavailable | 503, no challenge | +| Caller canceled or deadline exceeded | ctx.Err() | Best-effort 503; no application handler | + +Each failure returns a zero caller. Diagnostics contain no raw token, claims, +kid, response body, transport cause or cancellation cause. Direct Verify does not +install context or write a response. Constructors and nil/zero-verifier misuse +return ErrInvalidConfiguration. Warm reports key unavailability or cancellation, +not invalid credentials. + +Offline signature/claim verification cannot establish current Cognito revocation, +sign-out or user-enabled state. No GetUser/introspection call is implied. The +resulting caller is a snapshot, including during a long stream; ongoing policy +checks belong to the application. + +Local validation includes an independent RFC RS256 vector, signed synthetic +tokens, JWKS rotation/outage/concurrency tests, and actual Bearer/IAM composition +over native HTTP and raw/typed API Gateway entry points. No live Cognito or AWS +deployment was used. [Decision 0020](decisions/0020-cognito-verification.md) records +the approved tradeoffs and official sources. The [runnable configuration example](../authn/cognito_example_test.go) +shows resource-bound users and explicitly unbound M2M clients without network I/O. diff --git a/docs/decisions/0020-cognito-verification.md b/docs/decisions/0020-cognito-verification.md index ef4c346..9d78632 100644 --- a/docs/decisions/0020-cognito-verification.md +++ b/docs/decisions/0020-cognito-verification.md @@ -1,6 +1,6 @@ # 0020: Cognito access-token verification and JWKS ownership -Status: proposed; design review only, no dependent implementation approved. +Status: accepted; verifier, key cache, examples and local validation implemented. Reviewed against local code and official sources on 2026-09-16, with Go 1.27.1. ## Evidence and scope @@ -371,11 +371,45 @@ with an independently asserted gateway identity. HTTP and synctest where appropriate. No live AWS resources or tokens required. Claims about deployed interoperability remain qualified until separately tested. -No source files, module dependencies or runtime behavior changed in this review. +The initial design review changed no runtime behavior. Implementation evidence +following approval is recorded below; module dependencies remain unchanged. ## Resolution -Awaiting review of C1-C7. In particular: approve the narrow stdlib/JSON v2 RS256 +The user approved C1-C7 on 2026-09-16, including the narrow stdlib/JSON v2 RS256 implementation, explicit per-client unbound allowance, strict profile limits, 15-minute cache with no expired-key fallback, and shared fetch cancellation -ownership before dependent implementation. +ownership. Implement and validate within these boundaries. + +## Implementation evidence + +authn now implements C1-C7 through NewCognitoVerifier, Verify and Warm, using +stdlib cryptography and direct JSON v2 with no new module dependency. Config is +validated without I/O and client policies are copied. Verification preserves +exact claim numbers, requires the reviewed token profile and returns only a +locally verified caller. Token time and the selected snapshot's key expiry are +checked before returning identity, including after signature/identity work. + +The cache owns one bounded shared fetch, atomic complete snapshots, successful +refresh cooldown and cold/expired failure backoff. Cancellation remains private +to each waiter. A transport that ignores cancellation cannot cause replacement +fetch accumulation or publish late success. Completion versus context-cleanup +signals prefer the already published result, avoiding spurious unavailability. + +Tests include the independent RFC 7515 A.2 RSA vector, signed synthetic claims, +client/audience combinations, malformed/ambiguous JSON and keys, exact header, +nesting, response and key-count bounds, integer dates, signature tampering, +removal/rotation/outages, 64 simultaneous cold waiters, exponential backoff, +known-key availability during refresh, independent/all-waiter cancellation, +body deadlines and a deliberately noncompliant late transport. Actual local +TLS transport is covered. Native in-memory HTTP plus all raw/typed Gateway +integration fixtures now use genuine RS256 verification against local JWKS, +alongside the existing IAM helper/local STS path. + +Full module race tests, vet, formatting checks and Linux arm64/amd64 builds pass +on Go 1.27.1. Final fixed-fixture fuzz runs completed 136,129 compact-token cases +and 280,903 JWKS cases without failures. The synthetic public-key/token fixture +contains no private key and gives every fuzz worker identical input. The +[consumer guide](../cognito.md) and runnable configuration example document +binding, limits, cache/cancellation ownership, revocation and deployment scope. +No live Cognito, STS, AWS deployment or GitHub CI run was performed. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 3de231f..c6920f2 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -24,8 +24,8 @@ Keep related decisions small enough to review and implement independently. | [0013: Invocation errors](0013-public-invocation-errors.md) | Accepted | How should callers inspect sanitized failures across raw, typed, and streaming boundaries? | | [0014: Claims API](0014-claims-api.md) | Accepted; claims foundation implemented | Which checked claim reads, SDK numeric inputs, ownership rules, and measured resource limits should identity expose? | | [0015: IAM caller forms](0015-iam-caller-forms.md) | Accepted | Which ARN forms represent an authenticated request caller in the initial IAM constructor? | -| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture accepted; IAM proof and HTTP selector implemented; first-party JWT verification pending | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | +| [0016: Mixed IAM/OAuth authentication](0016-mixed-iam-oauth-authentication.md) | Architecture accepted; IAM proof, HTTP selector and Cognito verification implemented | How can one route accept IAM credentials or OAuth tokens without built-in AWS_IAM authorization? | | [0017: IAM proof protocol](0017-iam-proof-protocol.md) | Accepted; generator and verifier implemented with local validation | What helper/verifier API, wire format, destination binding, replay policy and STS boundary should the IAM path use? | | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | | [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | -| [0020: Cognito verification](0020-cognito-verification.md) | Proposed; design review only | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | +| [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | diff --git a/docs/plan.md b/docs/plan.md index 5d5b787..0fe085c 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -62,8 +62,8 @@ the review's initial gateway-default proposal: no gateway-derived identity by default; `WithGatewayIdentity(true)` opts in for both IAM and JWT/Cognito. Local verification is explicitly composed. Constructor details and producer conflicts are settled in decisions 0012–0018; decision 0019 accepts the now-implemented -common HTTP authentication selector. First-party JWT verification and authz -remain separate reviews. +common HTTP authentication selector. Decision 0020 accepts the now-implemented +Cognito access-token verifier and JWKS cache. Authz remains a separate review. ## Planned package boundaries @@ -308,8 +308,9 @@ The IAM proof package remains independently usable. - Provide exported Go proof generation and consumer documentation with runnable examples showing IAM and OAuth clients calling the same protected route. IAM generation, selector/dispatcher examples and local mixed-identity contract - tests now exist; the example Bearer verifier is explicitly synthetic. Real - first-party Cognito/OAuth verification and deployed interoperability remain. + tests now exist. The original selector example still labels its synthetic + Bearer function; Cognito configuration examples and actual RS256 verification + now accompany it. Deployed interoperability remains unverified. The authn milestone passes full module race tests, vet, formatting checks and Linux arm64/amd64 builds on Go 1.27.1. Authorization-parser fuzzing completed @@ -317,24 +318,31 @@ Linux arm64/amd64 builds on Go 1.27.1. Authorization-parser fuzzing completed Gateway fixtures verify challenges and IAM-helper composition with local STS fixtures. No live AWS or GitHub CI run was performed for these changes. -- [Decision 0020](decisions/0020-cognito-verification.md) records the proposed +- [x] [Decision 0020](decisions/0020-cognito-verification.md) records the accepted Cognito access-token verifier/JWKS design, reviewed against AWS documentation, current identity/authn code and current stable Go JWT libraries on 2026-09-16. - It proposes direct JSON v2 plus stdlib RS256, per-client resource binding and + It specifies direct JSON v2 plus stdlib RS256, per-client resource binding and explicit unbound allowances for M2M/SDK flows, and bounded on-demand key caching. - This is a pending design review, not approved or implemented behavior. -- Review a verifier requiring explicit issuer/client restrictions; accepting all - pool clients must be a deliberate policy. -- Keep access-token `client_id`, ID-token audience, and access-token resource + The user approved C1-C7 on 2026-09-16; the verifier/cache are implemented. +- [x] Require explicit issuer/client restrictions; there is no implicit all-client + policy. Constructors perform no I/O and own their client configuration. +- [x] Keep access-token `client_id`, ID-token audience, and access-token resource audience validation separate. Preserve access-token-only defaults unless deliberately widened. -- Review algorithm/key selection, expiry/not-before/issued-at behavior, leeway, - malformed claims, and the chosen JWT dependency's JSON behavior. -- Separate cache freshness, unknown-key refresh, and failure backoff. -- Bound network duration and JWKS response size. Define stale-key behavior, +- [x] Implement reviewed algorithm/key selection, expiry/not-before/issued-at + behavior, leeway and malformed claims. Own JSON v2 parsing and use stdlib RSA. +- [x] Separate cache freshness, unknown-key refresh, and failure backoff. +- [x] Bound network duration and JWKS response size. Define stale-key behavior, rotation, concurrency, waiter cancellation, and HTTP client ownership. -- Use deterministic cache/rotation/outage tests; consider Go 1.27 test networking - and `testing/synctest` where useful. +- [x] Use deterministic cache/rotation/outage tests with testing/synctest and + native in-memory HTTP integration, plus an independent RFC RS256 vector. + +The Cognito milestone passes full race tests, vet, formatting and Linux +arm64/amd64 builds on Go 1.27.1. Final fixed-fixture fuzz runs completed 136,129 +token cases and 280,903 JWKS cases without failures. Production authn remains +stdlib/identity-only. [The Cognito guide](cognito.md) documents configuration, +resource binding, cache ownership and offline-verification limitations. There +was no live AWS or GitHub CI run. Acceptance: malformed credentials fail closed, failures do not create fetch storms, and dependency outages remain distinguishable from invalid credentials. From eb0661bec04a534a2aaebe1dd1b5dcdfb84ac8e5 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Wed, 16 Sep 2026 23:20:57 +0100 Subject: [PATCH 33/44] docs: propose application authorization rules and dispatch boundaries --- docs/decisions/0021-authorization-rules.md | 340 +++++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 7 + 3 files changed, 348 insertions(+) create mode 100644 docs/decisions/0021-authorization-rules.md diff --git a/docs/decisions/0021-authorization-rules.md b/docs/decisions/0021-authorization-rules.md new file mode 100644 index 0000000..87427be --- /dev/null +++ b/docs/decisions/0021-authorization-rules.md @@ -0,0 +1,340 @@ +# 0021: Application authorization rules and dispatcher integration + +Status: proposed; no dependent implementation approved. +Reviewed on 2026-09-16 against the implemented identity/authn/edge boundaries. + +## Evidence and scope + +Decisions 0012/0014 preserve immutable caller facts and distinguish missing or +uninterpretable collections. Decisions 0019/0020 establish one authenticated +caller before application dispatch. `action.go` already selects a header value +without owning a dispatcher. `authn/gateway_test.go` currently performs explicit +consumer-owned authorization for both IAM and JWT callers. This review replaces +that ad hoc rule logic with reusable predicates, not an application framework. + +Official AWS sources checked through the public AWS MCP documentation tools: + +- [GetCallerIdentity](https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-sts-2011-06-15.html): + no permission to perform that operation is required; even an explicit deny + does not prevent obtaining the identity information. Our STS proof cannot + establish permission to invoke an application action or an execute-api route. +- [Cognito resource servers/scopes](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-define-resource-servers.html): + the resource server checks scopes for the protected operation. Scope strings + identify permissions by application policy, not by a built-in edge action map. +- [Cognito groups](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-user-groups.html): + group membership and IAM-role assignment/selection are distinct mechanisms. + A cognito:groups value is not itself an IAM permission or an application grant. +- [IAM role principals](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html): + IAM can bind a role ARN in a policy to its unique principal ID, so deleting and + recreating the role changes that trust relationship. Plain ARN/name comparisons + in our library do not provide that protection or IAM policy evaluation. +- [Role paths](https://aws.amazon.com/blogs/security/how-to-enforce-creation-of-roles-in-a-specific-path-use-iam-role-naming-in-hierarchy-models/): + role names are unique within an account even across paths. Paths remain part + of IAM role ARNs; do not invent one from an STS session ARN. +- [AssumedRoleUser](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumedRoleUser.html) + and [AssumeRole](https://docs.aws.amazon.com/cli/latest/reference/sts/assume-role.html): + session identity includes the session name; it is not proof of the human or + workload that originally obtained the role session. Existing identity validation + already distinguishes the supported IAM principal forms. + +[RFC 6750](https://www.rfc-editor.org/rfc/rfc6750.html#section-3) distinguishes +authentication failure from insufficient privilege and specifies Bearer +challenges. Error/HTTP ownership must retain that distinction. Official installed +[Go context documentation](https://pkg.go.dev/context#Context), the previously +loaded Go rules and Go 1.27 notes support explicit context-aware callbacks, +consumer-owned interfaces and direct JSON v2 in response examples. + +The old Beakley volume is not mounted at its recorded path in this session. +This review uses its persisted findings (especially dual-arm identity and glob +matching) plus our current code; no old source or tests were copied. + +## Design graph + +```mermaid +flowchart TD + Authenticate[One configured identity producer] --> Select[Parse action once] + Select --> Registry[Consumer selects immutable policy and handler entry] + Registry --> Target[Validate request and identify exact target resource] + Target --> Authorize[Rule.Authorize with caller, action and resource] + Authorize --> Guard[Check configuration, cancellation and authenticated caller] + Guard --> Facts[Ordered checks over immutable caller facts] + Facts -->|Custom application check| Resolve[Resolve principal and grants with caller context] + Resolve --> FactsResult[Allow, no match or dependency failure] + Facts --> FactsResult + FactsResult -->|Deny or failure| Respond[Consumer renders safe HTTP failure] + FactsResult -->|Allow| Execute[Execute same selected entry and resource] + Execute --> Commit[Response or stream commitment] +``` + +No rule reads an action header, changes identity, dispatches a handler or retries +authentication. Validation/selection order can vary, but the selected action, +target and authorization result must govern the operation actually executed. + +## Z1. Small concrete rules with explicit request facts + +Recommend a stdlib/identity-only `authz` package with this initial core: + +```go +type Request struct { + Caller identity.Caller + Action string + Resource string +} + +type CheckFunc func(context.Context, Request) (bool, error) +type Rule struct { /* private immutable configuration */ } + +func Check(check CheckFunc) (Rule, error) +func All(rules ...Rule) (Rule, error) +func Any(rules ...Rule) (Rule, error) +func (r Rule) Authorize(ctx context.Context, request Request) error +``` + +Authorize returns nil only for an explicit allow. Request is passed by value; +Caller is already immutable, and Action/Resource are exact strings. Action must +be nonempty UTF-8, without whitespace trimming or case normalization by authz. +Resource is optional UTF-8: an empty value means the policy does not require a +resource identifier. A resource-aware custom check must reject an absent target. +Resource is an application identifier, not automatically an ARN, path or URL. + +Consumers populate these values after their own parsing/canonicalization. The +core does not read identity from context as a competing source: pass the already +established caller explicitly. It neither replaces caller context nor caches a +permission result. Construct rules once; copies share private immutable nodes. +Custom callbacks must be concurrency-safe when rules are shared. + +Reasoning: action dispatch is a primary consumer use case; explicit action and +target avoid hiding authorization parameters in context values. A generic facts +map, request body, http.Request dependency or exported mutable policy tree would +add ownership and interpretation problems. Consumers can define an Authorize +interface where needed; the producer need not export a redundant interface. + +## Z2. Positive predicates with exact, issuer-qualified comparisons + +Recommend these constructors, each returning `(Rule, error)`: + +```go +func Sources(allowed ...identity.Source) (Rule, error) +func JWTSubject(issuer, subject string) (Rule, error) +func JWTClient(issuer, clientID string) (Rule, error) +func JWTScopes(issuer string, required ...string) (Rule, error) +func CognitoGroups(issuer string, required ...string) (Rule, error) +func IAMPrincipal(principalARN string) (Rule, error) +func IAMRoleSessions(partition, accountID, roleName string) (Rule, error) +``` + +JWTScopes and CognitoGroups require **all** supplied values. Use Any of separate +rules when one alternative suffices. Constructors require nonempty identifiers, +valid UTF-8 and at least one required value; scopes use the existing OAuth +scope-token grammar. Duplicate set values may be deduplicated without changing +spelling. Empty or unknown source lists and SourceNone are invalid configuration. +Sources permits any one explicitly supplied nonanonymous source. + +Wrong caller kind, missing facts and unavailable/flattened collections do not +match. A missing collection is not a network outage and does not become an empty +set that can satisfy a negated rule. Never split flattened Cognito groups or +parse them as embedded JSON. A literal scope such as `orders/*` is an exact scope +name; it does not authorize other scope strings. Matching is case-sensitive. + +Always qualify JWT subject/client/scope/group rules with exact issuer. Client +matching does not assert a machine grant; subject matching does not establish a +human identity. Scopes, groups and resolved application grants never substitute +for each other. These rules use established facts; they do not independently +enforce token_use, signature, lifetime, revocation or audience. Those remain +the configured authentication producer's responsibility, including on gateway +routes. Combine rules when the application requires additional claim restrictions. + +By default the predicates accept any supported producer of the matching kind. +Applications may put Sources in an outer All to restrict provenance. Source is +trusted-code attribution, not an unforgeable credential. Do not silently exclude +gateway/custom producers that the existing identity contract supports. +Sources alone is a broad allow for callers from those producers; examples must +combine it with the actual permission rule rather than present it as sufficient +application authorization. + +Alternative: bare scope/group/subject matching is shorter but permits accidental +collisions across issuers. Separate AnyScopes/AllScopes variants multiply the API; +the two general combinators provide the intended distinction. + +## Z3. IAM exact principals and explicit role-session families + +IAMPrincipal accepts the same supported caller ARN forms as identity.NewIAM and +compares the entire ARN literally. In particular: + +- A root ARN matches that root caller only, not every principal in the account. +- A user path's literal `*` or `?` remains literal, following decision 0015. +- An assumed-role ARN matches the full session name, not all sessions of a role. +- Bare IAM role ARNs are not request-caller ARNs and are invalid here. + +IAMRoleSessions is the explicit broader operation. Require exact partition, +twelve-digit account and a valid role name. Match only an already validated +assumed-role caller with those components and any valid session name. Reject +wildcard configuration and role paths; do not replace `sts` with `iam` or return +a reconstructed role ARN. The authenticated caller retains its complete ARN. + +This is a **role-name policy**. Recreating a role with the same name/account can +match again. It does not bind a role's unique ID, identify the original human, +evaluate a role's permission boundary/session policy, or prove execute-api +permission. Applications needing lifecycle-sensitive identity must resolve and +compare their explicitly enrolled stable identifiers. PrincipalID remains an +optional opaque fact; do not introduce automatic parsing into identity here. + +No path.Match, regex, ARN glob engine, IAM policy parser, account-wide allow +helper or inferred administrator/root override initially. A custom check can +express additional application policy explicitly. This avoids reproducing the +old separator/wildcard ambiguity while supporting practical varying sessions. + +## Z4. Ordered combinators, with errors stopping evaluation + +Recommend ordinary left-to-right short circuit evaluation: + +| Operator | Child returns false, nil | Child returns true, nil | Child returns any error | +| --- | --- | --- | --- | +| All | Stop: no match | Continue; match after last child | Stop: failure | +| Any | Continue; no match after last child | Stop: match | Stop: failure | + +An observed dependency failure never becomes a nonmatch followed by trying +another permission path. Discard a true result accompanied by an error. Require +nonempty combinators and valid child rules at construction; never allow the +vacuous truth of an empty All. A zero Rule is unconfigured and cannot allow. + +This is ordered evaluation: reordering may change work and error outcomes. +For example, Any(allow, unavailable) allows without calling its second child; +Any(unavailable, allow) fails. This is intentional short circuit behavior, not +a claim that all possible branch errors have been evaluated. Keep mandatory +tenant, suspension or revocation checks outside alternative allow branches: + +`All(mandatoryApplicationGuard, Any(jwtPermissionPath, iamPermissionPath))` + +Each alternative should start with cheap applicable identity restrictions before +calling a provider. No parallel evaluation, retries, hidden caching, or promises +that callbacks execute for audit purposes. Reusing a check in multiple evaluated +positions can call it more than once; callbacks must tolerate this. + +Alternative: continue after an error to seek an independent allow improves some +outage cases but can authorize after a failed application-policy dependency. +Evaluating every branch to give errors absolute precedence loses short-circuit +isolation and can turn unrelated services into mandatory dependencies. Prefer +the ordered contract and explicit policy composition initially. + +Do not add Not/None initially. Without a richer missing-fact model, negating a +failed match could authorize callers whose scopes/groups are unavailable. +Public/anonymous routes intentionally bypass authorization; no AllowAll escape +hatch is needed in the initial protected-route API. + +## Z5. Sanitized outcomes and bounded evaluation + +Recommend sentinels ErrInvalidConfiguration, ErrInvalidRequest, +ErrUnauthenticated, ErrDenied and ErrUnavailable, inspected with errors.Is. +No public mutable Decision struct with an allow bit that can disagree with an +error, and no raw callback error chain or claims in diagnostics. + +Authorize checks configuration/non-nil context, then caller cancellation, +request string validity, and a nonanonymous caller before invoking any check. +Anonymous requests always return ErrUnauthenticated; even a permissive custom +callback cannot authorize them. A zero rule or nil check fails configuration +validation. A valid rule with a final nonmatch returns ErrDenied. + +Custom checks use false,nil for ordinary denial. **Any nonnil callback error is +sanitized to ErrUnavailable**, including an internal context error while the +request context remains live. Callback-returned package sentinels do not bypass +that contract. Recheck caller cancellation between checks and before returning; +ctx.Err wins over results and provider diagnostics, never context.Cause. Panics +from trusted callbacks propagate; do not disguise programmer bugs as denial. + +Checks run synchronously with the caller's context. Consumers own provider +timeouts and must honor cancellation; the library cannot preempt arbitrary Go +code. No goroutine per check and no callback invocation at construction. + +Recommend maximum policy depth 32 and maximum 1,024 expanded node visits per +constructed rule. Count reused subtrees at every occurrence using checked +arithmetic. These are policy-configuration work bounds, not AWS limits or limits +on callback internals. Constructors copy supplied slices, reject oversized trees +and permit immutable sharing. No global policy registry or automatic logging. + +## Z6. Application principal/grant resolution as a consumer-owned check + +Recommend supporting resolution immediately through CheckFunc, while deferring +a universal exported Principal/Grant/Resolver model until a concrete application +schema is reviewed. This is an explicit narrowing of the first authz milestone, +not an implicit decision that token groups are sufficient application grants. + +The consumer's check receives caller, action and resource. It may resolve an +application principal, tenant membership and resource-scoped grants in one +context-aware service call. It returns false,nil for a known unregistered, +suspended or unauthorized principal, and an error when the decision cannot be +made. The server chooses which grant service is authoritative. Never source +these grants from caller-supplied headers or overwrite authenticated claims. + +Use issuer plus subject for subject identity, issuer plus client_id for an +explicitly supported client identity, and an intentionally chosen IAM enrollment +key. Do not fall back from a missing subject to treating client_id as a user; +do not infer human/M2M status from subject presence. Account/session names and +email addresses are not automatic global application-principal identifiers. + +Keep one logical grant lookup inside one check. The core does not guarantee +deduplication across checks or install a second context principal. If execution +needs the resolved object, the application can resolve it once before invoking +its policy and retain that immutable object in its own request execution plan. +An exported resolver/cache abstraction, grant wildcards and logging hooks need +a later reviewed consumer requirement, rather than speculative fields now. + +## Z7. HTTP ownership and action/resource consistency + +Recommend no authz.Handler factory in the first cut. Provide a runnable ordinary +HTTP dispatcher example using existing authn middleware and ActionHeader. The +same registry entry owns its Rule and handler; select it once, authorize its +action/target, then execute that entry. Unknown actions never fall through to an +unprotected default. The consumer chooses its 400/404 response policy. + +This makes custom headers first-class through net/http without an additional +header container, mutable context action or second action-parser callback. +Do not bind checks to rereadable raw headers. Object-level authorization must +use the same target object/version as execution; transactional consistency for +mutable application data remains the application's responsibility. + +HTTP examples should distinguish: + +| Outcome | HTTP treatment owned by the consumer | +| --- | --- | +| Unauthenticated | Ordinarily handled by authn's 401/challenges; gateway-only routes configure their own challenges | +| Authenticated denial | 403 | +| Authorization dependency unavailable | 503, no authentication challenge | +| Configuration/invalid authorization request | 500; parse bad client input earlier into an appropriate 4xx | +| Caller cancellation | Do not execute; best-effort 503, subject to transport cancellation | + +For Bearer-protected resources, include the applicable Bearer challenge on +denial as required by RFC 6750. Use a generic realm challenge when the denial +reason is not exposed; insufficient_scope and a scope hint are only appropriate +when the application can accurately describe that requirement. Do not label +every IAM/group/tenant denial as insufficient_scope or invent EdgeIAM OAuth +parameters. The core does not guess an authentication scheme from Caller.Kind. +Examples know their configured route authentication and own header formatting. + +Preserve CORS headers, avoid credentials/identifiers in responses, and use direct +JSON v2 for JSON examples. An error response does not execute an action. Finish +authorization before any response commitment or streaming flush. This proposal +does not add mid-stream reauthorization or a framework-owned audit stream. + +## Implementation after approval + +1. Implement core immutable rules, constructor validation and sanitized outcomes. +2. Add exact JWT/IAM/source predicates and ordered All/Any with bounded trees. +3. Test empty/zero/nil rules, anonymous/custom-check bypass attempts, source/kind + gates, both callback result/error combinations, cancellation and error order. +4. Test unavailable gateway collections, cross-issuer collisions, literal scope + punctuation, exact IAM paths/session/root, role/account/partition boundaries + and explicit role-name recreation semantics. Fuzz policy input validation. +5. Add consumer-owned grant lookup and resource authorization fixtures, including + 403 versus 503, zero provider calls for inapplicable branches, and no grants + being injected into token identity. Test the exact selected action/target. +6. Exercise IAM and cryptographically verified Cognito callers with native HTTP + plus raw/typed Gateway fixtures. Cover safe challenges and error envelopes. +7. Add the consumer guide/examples; run race/vet/Lambda builds. No live AWS calls. + +## Resolution + +Awaiting review of Z1-Z7. The main choices are explicit action/resource inputs, +exact issuer-qualified predicates, a deliberately named role-session-family +check, ordered error-stopping combinators, and consumer-owned grant resolution +and HTTP dispatch for the initial release. No production behavior changes here. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index c6920f2..0911753 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -29,3 +29,4 @@ Keep related decisions small enough to review and implement independently. | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | | [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | | [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | +| [0021: Authorization rules](0021-authorization-rules.md) | Proposed; design review only | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | diff --git a/docs/plan.md b/docs/plan.md index 0fe085c..bb0e06b 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -349,6 +349,13 @@ storms, and dependency outages remain distinguishable from invalid credentials. ### 5. Authorization and consumer ergonomics +[Decision 0021](decisions/0021-authorization-rules.md) records the proposed initial +authz API and design graph. It recommends immutable rules over explicit caller, +action and resource facts, exact predicates and ordered combinators. Application +principal/grant resolution and HTTP dispatch initially remain consumer-owned +through context-aware custom checks and runnable examples. That narrowing and +the public/error contracts await user review; no authz code is implemented yet. + - Implement reviewed rules/combinators, scope/group/grant separation, principal resolution, and explicit authentication precedence. - Review missing/invalid credential responses (ordinarily 401), authenticated From 798d24fac4515bad5fb035b3471af38f95dc9c0e Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Thu, 17 Sep 2026 03:23:19 +0100 Subject: [PATCH 34/44] feat(authz): add exact authorization rules and dispatcher examples --- README.md | 5 +- authn/authorization_test.go | 210 ++++++++++++++ authn/gateway_test.go | 57 ++-- authz/doc.go | 16 ++ authz/example_test.go | 217 ++++++++++++++ authz/predicates.go | 159 +++++++++++ authz/predicates_test.go | 312 +++++++++++++++++++++ authz/rule.go | 156 +++++++++++ authz/rule_test.go | 267 ++++++++++++++++++ authz/transport_test.go | 89 ++++++ docs/authn.md | 9 +- docs/authz.md | 191 +++++++++++++ docs/decisions/0021-authorization-rules.md | 35 ++- docs/decisions/README.md | 2 +- docs/plan.md | 44 +-- 15 files changed, 1728 insertions(+), 41 deletions(-) create mode 100644 authn/authorization_test.go create mode 100644 authz/doc.go create mode 100644 authz/example_test.go create mode 100644 authz/predicates.go create mode 100644 authz/predicates_test.go create mode 100644 authz/rule.go create mode 100644 authz/rule_test.go create mode 100644 authz/transport_test.go create mode 100644 docs/authz.md diff --git a/README.md b/README.md index b25fa47..3a640aa 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,10 @@ the custom dispatcher/HTTP response boundaries. `authn.NewCognitoVerifier` now provides RS256 access-token verification with direct JSON v2, explicit issuer/ client/resource restrictions and bounded JWKS caching; see the [Cognito guide](docs/cognito.md). Wire its Verify method as the Bearer verifier. -The authz package remains planned; applications must authorize each action. +The `authz` package now provides exact IAM/JWT predicates, ordered All/Any rules +and context-aware application grant checks. Consumers own their action registry, +resource selection and HTTP responses; see the [authorization guide](docs/authz.md) +and [runnable dispatcher examples](authz/example_test.go). No live Cognito/STS interoperability test has run. The accepted streaming design now has a private bridge with incremental delivery, diff --git a/authn/authorization_test.go b/authn/authorization_test.go new file mode 100644 index 0000000..3435033 --- /dev/null +++ b/authn/authorization_test.go @@ -0,0 +1,210 @@ +package authn_test + +import ( + "context" + "encoding/json/v2" + "errors" + "net/http" + "strings" + "sync/atomic" + "testing" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/authz" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Exercise actual token verification and STS-proof parsing before application +// policy. Only JWKS/STS network responses and the grant service are synthetic. +func TestAuthorizationAcrossTransports(t *testing.T) { + key := signingKey(t) + keys := keySet(t, publicJWK(key, "authorization")) + proofVerifier, proof, _ := localProofVerifier(t) + verifier := cognitoVerifier(t, cognitoConfig(transportFunc(func(*http.Request) (*http.Response, error) { + return jwksResponse(keys), nil + }))) + a := authenticator(t, authn.Config{Bearer: verifier.Verify, IAMProof: mapProofErrors(proofVerifier)}) + selector, err := edge.NewActionHeader("Action") + if err != nil { + t.Fatal(err) + } + scopes, err := authz.JWTScopes(cognitoIssuer, "orders.read") + if err != nil { + t.Fatal(err) + } + groups, err := authz.CognitoGroups(cognitoIssuer, "operators") + if err != nil { + t.Fatal(err) + } + jwtPath, err := authz.All(scopes, groups) + if err != nil { + t.Fatal(err) + } + iamPath, err := authz.IAMPrincipal("arn:aws:iam::123456789012:user/Alice") + if err != nil { + t.Fatal(err) + } + permission, err := authz.Any(jwtPath, iamPath) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name, scheme, action, grant string + scope, group bool + status, lookups int + }{ + {"jwt_allowed", "Bearer", "orders.read", "allow", true, true, 200, 1}, + {"iam_allowed", "EdgeIAM", "orders.read", "allow", false, false, 200, 1}, + {"scope_denied", "Bearer", "orders.read", "allow", false, true, 403, 0}, + {"group_denied", "Bearer", "orders.read", "allow", true, false, 403, 0}, + {"unregistered", "Bearer", "orders.read", "unregistered", true, true, 403, 1}, + {"suspended", "EdgeIAM", "orders.read", "suspended", false, false, 403, 1}, + {"wrong_resource", "Bearer", "orders.other", "allow", true, true, 403, 1}, + {"grant_outage", "Bearer", "orders.read", "outage", true, true, 503, 1}, + {"iam_grant_outage", "EdgeIAM", "orders.read", "outage", false, false, 503, 1}, + {"unknown_action", "Bearer", "orders.unknown", "allow", true, true, 404, 0}, + {"malformed_action", "Bearer", "orders.read,orders.delete", "allow", true, true, 400, 0}, + } { + t.Run(tc.name, func(t *testing.T) { + claims := accessClaims() + if !tc.scope { + delete(claims, "scope") + } + if !tc.group { + delete(claims, "cognito:groups") + } + token := signedToken(t, key, "authorization", claims) + if tc.scheme == "EdgeIAM" { + token = proof.Value() + } + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { + t.Run(format, func(t *testing.T) { + var lookups, executions atomic.Int64 + grant, err := authz.Check(func(ctx context.Context, req authz.Request) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + lookups.Add(1) + if req.Action != tc.action { + t.Errorf("grant checked action %q", req.Action) + } + if j, ok := req.Caller.JWT(); ok { + if _, present := j.Claims().Lookup("grants"); present { + t.Error("application grants injected into token facts") + } + if req.Caller.Source() != identity.SourceLocallyVerifiedToken { + t.Error("JWT was not verified") + } + } else if req.Caller.Source() != identity.SourceVerifiedIAMProof { + t.Error("IAM proof was not verified") + } + if tc.grant == "outage" { + return true, errors.New("private tenant and grant database details") + } + return tc.grant == "allow" && req.Resource == "tenant/42/order/7", nil + }) + if err != nil { + t.Fatal(err) + } + policy, err := authz.All(permission, grant) + if err != nil { + t.Fatal(err) + } + type entry struct { + rule authz.Rule + resource string + execute func(http.ResponseWriter, *http.Request, authz.Request) + } + execute := func(w http.ResponseWriter, _ *http.Request, facts authz.Request) { + executions.Add(1) + if facts.Action != "orders.read" || facts.Resource != "tenant/42/order/7" { + t.Error("executed a different action or resource") + } + w.Header().Set("Content-Type", "application/json") + if err := json.MarshalWrite(w, map[string]string{"action": facts.Action, "resource": facts.Resource}); err != nil { + t.Error(err) + } + } + registry := map[string]entry{ + "orders.read": {policy, "tenant/42/order/7", execute}, + "orders.other": {policy, "tenant/99/order/8", execute}, + } + dispatch := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, err := selector.Parse(r.Header) + if err != nil { + authorizationResponse(t, w, 400, "invalid_action") + return + } + selected, ok := registry[action] + if !ok { + authorizationResponse(t, w, 404, "unknown_action") + return + } + facts := authz.Request{Caller: identity.FromContext(r.Context()), Action: action, Resource: selected.resource} + // Downstream header changes must not change authorization or execution. + r.Header.Set("Action", "orders.delete") + r.Header.Set("Resource", "tenant/99/order/8") + if err := selected.rule.Authorize(r.Context(), facts); err != nil { + switch { + case errors.Is(err, authz.ErrDenied): + // This route supports Bearer; no claim about the denial's cause. + w.Header().Set("WWW-Authenticate", `Bearer realm="edge"`) + authorizationResponse(t, w, 403, "forbidden") + case errors.Is(err, authz.ErrUnavailable): + authorizationResponse(t, w, 503, "unavailable") + default: + authorizationResponse(t, w, 500, "internal_error") + } + return + } + selected.execute(w, r, facts) + }) + h := handler(t, a, dispatch) + withCORS := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Access-Control-Allow-Origin", "https://app.example") + h.ServeHTTP(w, r) + }) + status, headers, body := serveFormatBody(t, withCORS, format, []string{tc.scheme + " " + token}, tc.action) + if status != tc.status || lookups.Load() != int64(tc.lookups) { + t.Fatalf("status=%d lookups=%d; want %d %d", status, lookups.Load(), tc.status, tc.lookups) + } + wantExecutions := int64(0) + if status == 200 { + wantExecutions = 1 + } + if executions.Load() != wantExecutions { + t.Errorf("executions=%d; want %d", executions.Load(), wantExecutions) + } + challenge := "" + if status == 403 { + challenge = `Bearer realm="edge"` + } + if headers.Get("WWW-Authenticate") != challenge || headers.Get("Access-Control-Allow-Origin") != "https://app.example" { + t.Errorf("incorrect challenge/CORS: %v", headers) + } + if headers.Get("Content-Type") != "application/json" || status != 200 && headers.Get("Cache-Control") != "no-store" { + t.Errorf("unsafe response headers: %v", headers) + } + var envelope map[string]string + if err := json.Unmarshal(body, &envelope); err != nil { + t.Fatal(err) + } + if status != 200 && (len(envelope) != 1 || envelope["error"] == "" || strings.Contains(string(body), "private")) { + t.Errorf("unsafe error envelope: %s", body) + } + }) + } + }) + } +} + +func authorizationResponse(t *testing.T, w http.ResponseWriter, status int, code string) { + t.Helper() + w.Header().Set("Content-Type", "application/json") + w.Header().Set("Cache-Control", "no-store") + w.WriteHeader(status) + if err := json.MarshalWrite(w, map[string]string{"error": code}); err != nil { + t.Error(err) + } +} diff --git a/authn/gateway_test.go b/authn/gateway_test.go index fc96ee7..a4be7a1 100644 --- a/authn/gateway_test.go +++ b/authn/gateway_test.go @@ -2,6 +2,7 @@ package authn_test import ( "context" + "encoding/base64" "encoding/json/v2" "errors" "io" @@ -17,6 +18,7 @@ import ( "github.com/asteroid-computing/go-lambda-edge" "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/authz" "github.com/asteroid-computing/go-lambda-edge/iamproof" "github.com/asteroid-computing/go-lambda-edge/identity" ) @@ -72,23 +74,27 @@ func TestAuthenticationAcrossTransports(t *testing.T) { if err != nil { t.Fatal(err) } + jwtRule, err := authz.JWTSubject(cognitoIssuer, "alice") + if err != nil { + t.Fatal(err) + } + iamRule, err := authz.IAMPrincipal("arn:aws:iam::123456789012:user/Alice") + if err != nil { + t.Fatal(err) + } + permission, err := authz.Any(jwtRule, iamRule) + if err != nil { + t.Fatal(err) + } dispatcher := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { action, err := selector.Parse(r.Header) if err != nil { http.Error(w, "invalid action", 400) return } - caller := identity.FromContext(r.Context()) - allowed := false - if iam, ok := caller.IAM(); ok { - allowed = iam.PrincipalARN() == "arn:aws:iam::123456789012:user/Alice" - } - if jwt, ok := caller.JWT(); ok { - sub, _ := jwt.Subject() - allowed = jwt.Issuer() == cognitoIssuer && sub == "alice" - } // Deny by default and authorize this exact action before executing it. - if !allowed || action != "orders.read" { + if action != "orders.read" || permission.Authorize(r.Context(), authz.Request{Caller: identity.FromContext(r.Context()), Action: action}) != nil { + w.Header().Set("WWW-Authenticate", `Bearer realm="edge"`) http.Error(w, "forbidden", 403) return } @@ -115,8 +121,8 @@ func TestAuthenticationAcrossTransports(t *testing.T) { {name: "malformed", values: []string{"Bearer a=b"}, status: 400, challenges: []string{`Bearer realm="edge", error="invalid_request"`, `EdgeIAM realm="edge"`}}, {name: "unsupported", values: []string{"Basic abc"}, status: 401, challenges: []string{`Bearer realm="edge"`, `EdgeIAM realm="edge"`}}, {name: "dependency", values: []string{"Bearer " + jwt}, status: 503}, - {name: "authz_denial", values: []string{"Bearer " + jwt}, action: "orders.delete", status: 403}, - {name: "iam_authz_denial", values: []string{"EdgeIAM " + proof.Value()}, action: "orders.delete", status: 403}, + {name: "authz_denial", values: []string{"Bearer " + jwt}, action: "orders.delete", status: 403, challenges: []string{`Bearer realm="edge"`}}, + {name: "iam_authz_denial", values: []string{"EdgeIAM " + proof.Value()}, action: "orders.delete", status: 403, challenges: []string{`Bearer realm="edge"`}}, {name: "field_size", values: []string{"Bearer " + strings.Repeat("A", 16384)}, status: 431}, } { t.Run(tc.name, func(t *testing.T) { @@ -152,6 +158,12 @@ func TestAuthenticationAcrossTransports(t *testing.T) { } func serveFormat(t *testing.T, h http.Handler, format string, values []string, action string) (int, http.Header) { + t.Helper() + status, headers, _ := serveFormatBody(t, h, format, values, action) + return status, headers +} + +func serveFormatBody(t *testing.T, h http.Handler, format string, values []string, action string) (int, http.Header, []byte) { t.Helper() if format == "native" { server := httptest.NewTestServer(t, h) @@ -165,10 +177,11 @@ func serveFormat(t *testing.T, h http.Handler, format string, values []string, a t.Fatal(err) } defer resp.Body.Close() - if _, err := io.Copy(io.Discard, resp.Body); err != nil { + body, err := io.ReadAll(resp.Body) + if err != nil { t.Fatal(err) } - return resp.StatusCode, resp.Header + return resp.StatusCode, resp.Header, body } adapter, err := edge.New(h) if err != nil { @@ -220,9 +233,21 @@ func serveFormat(t *testing.T, h http.Handler, format string, values []string, a for name, value := range resp2.Headers { headers.Set(name, value) } - return resp2.StatusCode, headers + return resp2.StatusCode, headers, responseBody(t, resp2.Body, resp2.IsBase64Encoded) + } + return resp1.StatusCode, http.Header(resp1.MultiValueHeaders), responseBody(t, resp1.Body, resp1.IsBase64Encoded) +} + +func responseBody(t *testing.T, body string, base64Encoded bool) []byte { + t.Helper() + if !base64Encoded { + return []byte(body) + } + out, err := base64.StdEncoding.DecodeString(body) + if err != nil { + t.Fatal(err) } - return resp1.StatusCode, http.Header(resp1.MultiValueHeaders) + return out } func TestIAMErrorMappingIsExplicit(t *testing.T) { diff --git a/authz/doc.go b/authz/doc.go new file mode 100644 index 0000000..07217cf --- /dev/null +++ b/authz/doc.go @@ -0,0 +1,16 @@ +// Package authz authorizes explicit caller, action and resource facts using +// immutable rules. It does not authenticate, evaluate AWS IAM policies, resolve +// application grants automatically or write HTTP responses. +// +// Construct rules once and share them across requests. Custom checks must be +// concurrency-safe, honor their context and return false, nil for ordinary +// denial. Every callback error becomes ErrUnavailable. All and Any evaluate +// left to right and stop on an observed error; keep mandatory application guards +// outside alternative permission branches. +// +// Authorize requires a nonanonymous caller even for a permissive custom check. +// Identity constructors validate facts but do not authenticate them: establish +// the caller through a trusted producer before authorization. Complete selection +// and authorization before executing the same action/resource or committing an +// HTTP response, including a stream. +package authz diff --git a/authz/example_test.go b/authz/example_test.go new file mode 100644 index 0000000..f64a84b --- /dev/null +++ b/authz/example_test.go @@ -0,0 +1,217 @@ +package authz_test + +import ( + "context" + "encoding/json/v2" + "errors" + "fmt" + "net/http" + "net/http/httptest" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/authz" + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +const orderIssuer = "https://issuer.example/pool" + +// grantStore is an application interface, not a universal edge principal model. +// Its implementation chooses the authority and enrollment keys, resolves tenant +// membership/suspension and checks the exact action/resource in one operation. +type grantStore interface { + Allowed(context.Context, identity.Caller, string, string) (bool, error) +} + +// orderHandler owns an immutable registry. This example has one fixed resource; +// a real application validates/resolves its target before calling Authorize and +// keeps that same object/version for execution, using a transaction if needed. +func orderHandler(authentication *authn.Authenticator, grants grantStore, issuer string) (http.Handler, error) { + selector, err := edge.NewActionHeader("Action") + if err != nil { + return nil, err + } + scopes, err := authz.JWTScopes(issuer, "orders.read") + if err != nil { + return nil, err + } + iam, err := authz.IAMRoleSessions("aws", "123456789012", "OrderReader") + if err != nil { + return nil, err + } + permission, err := authz.Any(scopes, iam) + if err != nil { + return nil, err + } + applicationGrant, err := authz.Check(func(ctx context.Context, req authz.Request) (bool, error) { + if req.Resource == "" { + return false, nil + } + return grants.Allowed(ctx, req.Caller, req.Action, req.Resource) + }) + if err != nil { + return nil, err + } + policy, err := authz.All(permission, applicationGrant) + if err != nil { + return nil, err + } + type entry struct { + policy authz.Rule + resource string + execute func(http.ResponseWriter, *http.Request, authz.Request) + } + registry := map[string]entry{ + "orders.read": { + policy: policy, + resource: "tenant/42/order/7", + execute: func(w http.ResponseWriter, _ *http.Request, facts authz.Request) { + w.Header().Set("Content-Type", "application/json") + // A write failure cannot be replaced after response commitment. + _ = json.MarshalWrite(w, struct { + Resource string `json:"resource"` + }{facts.Resource}) + }, + }, + } + dispatch := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, err := selector.Parse(r.Header) + if err != nil { + writePolicyError(w, 400, "invalid_action") + return + } + selected, ok := registry[action] + if !ok { + writePolicyError(w, 404, "unknown_action") + return + } + facts := authz.Request{Caller: identity.FromContext(r.Context()), Action: action, Resource: selected.resource} + if err := selected.policy.Authorize(r.Context(), facts); err != nil { + switch { + case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded), errors.Is(err, authz.ErrUnavailable): + writePolicyError(w, 503, "unavailable") + case errors.Is(err, authz.ErrDenied): + // This route accepts Bearer. A generic challenge reveals no policy + // reason and does not mislabel an IAM or grant denial as a scope error. + w.Header().Set("WWW-Authenticate", `Bearer realm="orders"`) + writePolicyError(w, 403, "forbidden") + case errors.Is(err, authz.ErrUnauthenticated): + w.Header().Add("WWW-Authenticate", `Bearer realm="orders"`) + w.Header().Add("WWW-Authenticate", `EdgeIAM realm="orders"`) + writePolicyError(w, 401, "unauthenticated") + default: + writePolicyError(w, 500, "internal_error") + } + return + } + selected.execute(w, r, facts) + }) + return authentication.Handler(dispatch) +} + +func writePolicyError(w http.ResponseWriter, status int, code string) { + w.Header().Set("Content-Type", "application/json") + w.Header().Set("Cache-Control", "no-store") + w.WriteHeader(status) + _ = json.MarshalWrite(w, struct { + Error string `json:"error"` + }{code}) +} + +// exampleGrants is a server-owned, read-only fixture. A database implementation +// can return false,nil for suspended/unregistered principals and an error when +// its dependency cannot decide. These grants are never put in identity.Claims. +type exampleGrants struct{} + +func (exampleGrants) Allowed(ctx context.Context, caller identity.Caller, action, resource string) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + if action != "orders.read" || resource != "tenant/42/order/7" { + return false, nil + } + if j, ok := caller.JWT(); ok { + subject, present := j.Subject() + return present && j.Issuer() == orderIssuer && subject == "alice", nil + } + if i, ok := caller.IAM(); ok { + // This fixture explicitly enrolls a full session. Applications can use a + // different reviewed enrollment key; account or session name alone is not one. + return i.PrincipalARN() == "arn:aws:sts::123456789012:assumed-role/OrderReader/session-one", nil + } + return false, nil +} + +func ExampleCheck() { + // These are validated fixture facts, not credential verification. A request + // handler obtains the caller from its configured authentication producer. + claims, err := identity.NewClaims(map[string]any{"iss": orderIssuer, "sub": "alice"}) + if err != nil { + panic(err) + } + caller, err := identity.NewJWT(claims, identity.SourceCustomAssertion) + if err != nil { + panic(err) + } + grants := exampleGrants{} + policy, err := authz.Check(func(ctx context.Context, req authz.Request) (bool, error) { + return grants.Allowed(ctx, req.Caller, req.Action, req.Resource) + }) + if err != nil { + panic(err) + } + request := authz.Request{Caller: caller, Action: "orders.read", Resource: "tenant/42/order/7"} + fmt.Println(policy.Authorize(context.Background(), request)) + request.Resource = "tenant/99/order/8" + fmt.Println(errors.Is(policy.Authorize(context.Background(), request), authz.ErrDenied)) + // Output: + // + // true +} + +func Example_dispatcher() { + // Construction performs no network I/O. Configure real pool/client/resource + // and STS audience values for deployment. The example sends no credentials, + // so neither verifier contacts AWS when run. + bearer, err := authn.NewCognitoVerifier(authn.CognitoConfig{ + Issuer: orderIssuer, + Clients: []authn.CognitoClient{{ClientID: "orders-client", Audience: "https://orders.example"}}, + }) + if err != nil { + panic(err) + } + proof, err := iamproof.NewVerifier("eu-west-2", "orders.production") + if err != nil { + panic(err) + } + authentication, err := authn.New(authn.Config{ + Realm: "orders", + Bearer: bearer.Verify, + IAMProof: func(ctx context.Context, token string) (identity.Caller, error) { + caller, err := proof.Verify(ctx, token) + if errors.Is(err, iamproof.ErrInvalidProof) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + return caller, err + }, + }) + if err != nil { + panic(err) + } + h, err := orderHandler(authentication, exampleGrants{}, orderIssuer) + if err != nil { + panic(err) + } + // h is an ordinary http.Handler. For Lambda, edge.New(h) uses the same + // dispatcher, with gateway identity left disabled for these local verifiers. + r := httptest.NewRequest(http.MethodGet, "/", nil) + r.Header.Set("Action", "orders.read") + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + fmt.Println(w.Code) + fmt.Println(w.Header().Values("WWW-Authenticate")) + // Output: + // 401 + // [Bearer realm="orders" EdgeIAM realm="orders"] +} diff --git a/authz/predicates.go b/authz/predicates.go new file mode 100644 index 0000000..55d619c --- /dev/null +++ b/authz/predicates.go @@ -0,0 +1,159 @@ +package authz + +import ( + "context" + "slices" + "strings" + "unicode/utf8" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Sources matches any supplied nonanonymous producer. Empty lists, SourceNone +// and unknown sources return ErrInvalidConfiguration. Source is trusted-code +// attribution, not an unforgeable credential. Combine this broad predicate with +// permission checks. The input slice is copied. +func Sources(allowed ...identity.Source) (Rule, error) { + if len(allowed) == 0 { + return Rule{}, ErrInvalidConfiguration + } + for _, source := range allowed { + switch source { + case identity.SourceGatewayAssertion, identity.SourceLocallyVerifiedToken, identity.SourceCustomAssertion, identity.SourceVerifiedIAMProof: + default: + return Rule{}, ErrInvalidConfiguration + } + } + sources := slices.Clone(allowed) + return Check(func(_ context.Context, r Request) (bool, error) { + return slices.Contains(sources, r.Caller.Source()), nil + }) +} + +// JWTSubject matches an exact issuer and subject. Both must be nonempty UTF-8 +// or construction returns ErrInvalidConfiguration. A subject does not prove a +// human identity. Wrong kind or missing subject does not match. +func JWTSubject(issuer, subject string) (Rule, error) { + if !validIdentifier(issuer) || !validIdentifier(subject) { + return Rule{}, ErrInvalidConfiguration + } + return Check(func(_ context.Context, r Request) (bool, error) { + j, ok := r.Caller.JWT() + value, present := j.Subject() + return ok && j.Issuer() == issuer && present && value == subject, nil + }) +} + +// JWTClient matches an exact issuer and client_id, without interpreting audience +// or subject as a client ID or inferring a machine grant. Both arguments must +// be nonempty UTF-8 or construction returns ErrInvalidConfiguration. +func JWTClient(issuer, clientID string) (Rule, error) { + if !validIdentifier(issuer) || !validIdentifier(clientID) { + return Rule{}, ErrInvalidConfiguration + } + return Check(func(_ context.Context, r Request) (bool, error) { + j, ok := r.Caller.JWT() + value, present := j.ClientID() + return ok && j.Issuer() == issuer && present && value == clientID, nil + }) +} + +// JWTScopes requires all supplied exact OAuth scope tokens from the exact issuer. +// It does not interpret wildcards. Missing or unavailable scopes do not match. +// Issuer must be nonempty UTF-8 and required must contain at least one valid +// RFC 6749 scope token, otherwise construction returns ErrInvalidConfiguration. +// The input slice is copied. Use Any for alternative scope requirements. +func JWTScopes(issuer string, required ...string) (Rule, error) { + return jwtSet(issuer, required, identity.JWT.Scopes, validScope) +} + +// CognitoGroups requires all supplied exact cognito:groups values from the exact +// issuer. Flattened gateway strings remain unavailable and never match. Groups +// are distinct from scopes and application grants. Issuer and each required +// value must be nonempty UTF-8; an empty list or invalid value returns +// ErrInvalidConfiguration. The input slice is copied. +func CognitoGroups(issuer string, required ...string) (Rule, error) { + return jwtSet(issuer, required, identity.JWT.CognitoGroups, validIdentifier) +} + +func jwtSet(issuer string, required []string, values func(identity.JWT) ([]string, bool), valid func(string) bool) (Rule, error) { + if !validIdentifier(issuer) || len(required) == 0 { + return Rule{}, ErrInvalidConfiguration + } + for _, value := range required { + if !valid(value) { + return Rule{}, ErrInvalidConfiguration + } + } + want := slices.Clone(required) + slices.Sort(want) + want = slices.Compact(want) + return Check(func(_ context.Context, r Request) (bool, error) { + j, ok := r.Caller.JWT() + if !ok || j.Issuer() != issuer { + return false, nil + } + got, known := values(j) + if !known { + return false, nil + } + for _, value := range want { + if _, found := slices.BinarySearch(got, value); !found { + return false, nil + } + } + return true, nil + }) +} + +func validIdentifier(s string) bool { return s != "" && utf8.ValidString(s) } + +// RFC 6749 scope-token = 1*(%x21 / %x23-5B / %x5D-7E), as in identity. +func validScope(s string) bool { + if s == "" { + return false + } + for i := range len(s) { + b := s[i] + if b < 0x21 || b > 0x7e || b == '"' || b == '\\' { + return false + } + } + return true +} + +// IAMPrincipal matches the full caller ARN literally, including path and session. +// A root ARN matches only that root caller, not the account. Valid literal * and +// ? in user paths retain their meaning as characters. Unsupported caller forms +// (including bare role ARNs) return ErrInvalidConfiguration, following NewIAM's +// grammar. This does not evaluate AWS IAM permissions or bind a unique identity +// across deletion and recreation of the principal. +func IAMPrincipal(principalARN string) (Rule, error) { + if _, err := identity.NewIAM(principalARN, identity.SourceCustomAssertion); err != nil { + return Rule{}, ErrInvalidConfiguration + } + return Check(func(_ context.Context, r Request) (bool, error) { + i, ok := r.Caller.IAM() + return ok && i.PrincipalARN() == principalARN, nil + }) +} + +// IAMRoleSessions matches assumed-role callers with the exact partition, account +// and role name, allowing any valid session name. Invalid components, wildcards +// and role paths return ErrInvalidConfiguration. This is a role-name policy: +// recreating the same role name can match again. It does not bind a unique role +// ID, identify the original actor, fabricate a role ARN or evaluate IAM policies. +func IAMRoleSessions(partition, accountID, roleName string) (Rule, error) { + prefix := "arn:" + partition + ":sts::" + accountID + ":assumed-role/" + roleName + "/" + // Validate a sample session with the identity grammar rather than maintaining + // a second ARN parser. Cross-check components to exclude delimiter injection. + caller, err := identity.NewIAM(prefix+"validation", identity.SourceCustomAssertion) + i, ok := caller.IAM() + if err != nil || !ok || i.PrincipalType() != identity.IAMPrincipalAssumedRole || i.Partition() != partition || i.AccountID() != accountID { + return Rule{}, ErrInvalidConfiguration + } + return Check(func(_ context.Context, r Request) (bool, error) { + i, ok := r.Caller.IAM() + return ok && i.PrincipalType() == identity.IAMPrincipalAssumedRole && strings.HasPrefix(i.PrincipalARN(), prefix), nil + }) +} diff --git a/authz/predicates_test.go b/authz/predicates_test.go new file mode 100644 index 0000000..37cacef --- /dev/null +++ b/authz/predicates_test.go @@ -0,0 +1,312 @@ +package authz_test + +import ( + "context" + "errors" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/authz" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +const issuer = "https://issuer.example/pool" + +func jwtCaller(t testing.TB, values map[string]any, source identity.Source) identity.Caller { + t.Helper() + claims, err := identity.NewClaims(values) + if err != nil { + t.Fatal(err) + } + c, err := identity.NewJWT(claims, source) + if err != nil { + t.Fatal(err) + } + return c +} + +func TestJWTPredicates(t *testing.T) { + makeRule := func(name string) authz.Rule { + t.Helper() + var r authz.Rule + var err error + switch name { + case "subject": + r, err = authz.JWTSubject(issuer, "alice") + case "client": + r, err = authz.JWTClient(issuer, "app") + case "scopes": + r, err = authz.JWTScopes(issuer, "orders/read", "orders/write", "orders/read") + case "groups": + r, err = authz.CognitoGroups(issuer, "operators", "tenant-A") + } + return rule(t, r, err) + } + for _, source := range []identity.Source{identity.SourceCustomAssertion, identity.SourceGatewayAssertion, identity.SourceLocallyVerifiedToken} { + for _, tc := range []struct { + name string + claims map[string]any + allowed []string + }{ + {"all", map[string]any{"iss": issuer, "sub": "alice", "client_id": "app", "scope": "orders/write orders/read", "cognito:groups": []string{"tenant-A", "operators"}}, []string{"subject", "client", "scopes", "groups"}}, + {"issuer_collision", map[string]any{"iss": issuer + "/other", "sub": "alice", "client_id": "app", "scope": "orders/read orders/write", "cognito:groups": []string{"operators", "tenant-A"}}, nil}, + {"unavailable", map[string]any{"iss": issuer, "sub": "alice"}, []string{"subject"}}, + {"client_not_subject", map[string]any{"iss": issuer, "client_id": "alice", "aud": []string{"app"}}, nil}, + {"client_only", map[string]any{"iss": issuer, "client_id": "app"}, []string{"client"}}, + {"subject_not_client", map[string]any{"iss": issuer, "sub": "app"}, nil}, + {"partial_sets", map[string]any{"iss": issuer, "sub": "bob", "scope": "orders/read", "cognito:groups": []string{"operators"}}, nil}, + {"empty_sets", map[string]any{"iss": issuer, "sub": "bob", "scp": []string{}, "cognito:groups": []string{}}, nil}, + {"wrong_case", map[string]any{"iss": issuer, "sub": "Alice", "client_id": "App", "scope": "Orders/read orders/write", "cognito:groups": []string{"Operators", "tenant-A"}}, nil}, + {"unrelated_groups", map[string]any{"iss": issuer, "sub": "bob", "groups": []string{"operators", "tenant-A"}, "roles": []string{"operators"}}, nil}, + } { + t.Run(tc.name+"/"+sourceName(source), func(t *testing.T) { + caller := jwtCaller(t, tc.claims, source) + for _, name := range []string{"subject", "client", "scopes", "groups"} { + want := authz.ErrDenied + for _, allowed := range tc.allowed { + if name == allowed { + want = nil + } + } + if err := makeRule(name).Authorize(t.Context(), authz.Request{Caller: caller, Action: "orders.read"}); !errors.Is(err, want) { + t.Errorf("%s = %v; want %v", name, err, want) + } + } + }) + } + } + for _, name := range []string{"subject", "client", "scopes", "groups"} { + if err := makeRule(name).Authorize(t.Context(), request(t)); !errors.Is(err, authz.ErrDenied) { + t.Error(err) + } + } +} + +func sourceName(s identity.Source) string { + switch s { + case identity.SourceGatewayAssertion: + return "gateway" + case identity.SourceCustomAssertion: + return "custom" + case identity.SourceLocallyVerifiedToken: + return "local" + default: + return "iam" + } +} + +func TestFlattenedCollectionsAndLiteralSets(t *testing.T) { + claims, err := identity.NewTextClaims(map[string]string{"iss": issuer, "sub": "alice", "scp": `["orders/*"]`, "cognito:groups": "[operators,tenant-A]"}) + if err != nil { + t.Fatal(err) + } + caller, err := identity.NewJWT(claims, identity.SourceGatewayAssertion) + if err != nil { + t.Fatal(err) + } + scope, err := authz.JWTScopes(issuer, "orders/*") + scope = rule(t, scope, err) + group, err := authz.CognitoGroups(issuer, "operators") + group = rule(t, group, err) + for _, r := range []authz.Rule{scope, group} { + if err := r.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); !errors.Is(err, authz.ErrDenied) { + t.Error(err) + } + } + // A separate dedicated gateway collection is usable without parsing text. + caller, err = identity.NewJWT(claims, identity.SourceGatewayAssertion, identity.WithGatewayScopes([]string{"orders/*"})) + if err != nil { + t.Fatal(err) + } + if err := scope.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); err != nil { + t.Fatal(err) + } + for _, value := range []string{"orders/*", "orders/read"} { + caller := jwtCaller(t, map[string]any{"iss": issuer, "sub": "alice", "scope": value}, identity.SourceCustomAssertion) + want := authz.ErrDenied + if value == "orders/*" { + want = nil + } + if err := scope.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); !errors.Is(err, want) { + t.Errorf("scope %q = %v", value, err) + } + } +} + +func TestIAMExactAndRoleFamilies(t *testing.T) { + const account = "123456789012" + for _, arn := range []string{ + "arn:aws:iam::" + account + ":root", + "arn:aws:iam::" + account + ":user/team*?/Alice", + "arn:aws:sts::" + account + ":assumed-role/Reader/session-one", + "arn:aws:sts::" + account + ":federated-user/Alice", + } { + r, err := authz.IAMPrincipal(arn) + r = rule(t, r, err) + for _, got := range []string{arn, "arn:aws:iam::" + account + ":user/Alice", "arn:aws:iam::" + account + ":user/teamAB/Alice"} { + want := authz.ErrDenied + if got == arn { + want = nil + } + if err := r.Authorize(t.Context(), authz.Request{Caller: iamCaller(t, got), Action: "read"}); !errors.Is(err, want) { + t.Errorf("principal %q caller %q = %v", arn, got, err) + } + } + } + family, err := authz.IAMRoleSessions("aws", account, "Reader") + family = rule(t, family, err) + exact, err := authz.IAMPrincipal("arn:aws:sts::" + account + ":assumed-role/Reader/session-one") + exact = rule(t, exact, err) + for _, tc := range []struct { + arn string + family, exact bool + }{ + {"arn:aws:sts::" + account + ":assumed-role/Reader/session-one", true, true}, + {"arn:aws:sts::" + account + ":assumed-role/Reader/session-two", true, false}, + {"arn:aws-cn:sts::" + account + ":assumed-role/Reader/session-one", false, false}, + {"arn:aws:sts::000000000000:assumed-role/Reader/session-one", false, false}, + {"arn:aws:sts::" + account + ":assumed-role/Readers/session-one", false, false}, + {"arn:aws:sts::" + account + ":assumed-role/reader/session-one", false, false}, + {"arn:aws:iam::" + account + ":user/Reader", false, false}, + {"arn:aws:iam::" + account + ":root", false, false}, + } { + for _, principalID := range []string{"OLDROLEID:session-one", "NEWROLEID:session-one"} { + caller := iamCaller(t, tc.arn, identity.WithIAMPrincipalID(principalID)) + for _, test := range []struct { + rule authz.Rule + allow bool + }{{family, tc.family}, {exact, tc.exact}} { + want := authz.ErrDenied + if test.allow { + want = nil + } + if err := test.rule.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); !errors.Is(err, want) { + t.Errorf("caller %q = %v; want %v", tc.arn, err, want) + } + } + } + } + jwt := jwtCaller(t, map[string]any{"iss": issuer, "sub": "alice"}, identity.SourceCustomAssertion) + if err := family.Authorize(t.Context(), authz.Request{Caller: jwt, Action: "read"}); !errors.Is(err, authz.ErrDenied) { + t.Error(err) + } +} + +func TestPredicateConfigurationAndSliceOwnership(t *testing.T) { + for _, construct := range []func() (authz.Rule, error){ + func() (authz.Rule, error) { return authz.Sources() }, + func() (authz.Rule, error) { return authz.Sources(identity.SourceNone) }, + func() (authz.Rule, error) { return authz.Sources(identity.Source(255)) }, + func() (authz.Rule, error) { return authz.JWTSubject("", "alice") }, + func() (authz.Rule, error) { return authz.JWTSubject(issuer, "\xff") }, + func() (authz.Rule, error) { return authz.JWTClient(issuer, "") }, + func() (authz.Rule, error) { return authz.JWTClient("\xff", "app") }, + func() (authz.Rule, error) { return authz.JWTScopes(issuer) }, + func() (authz.Rule, error) { return authz.CognitoGroups(issuer) }, + func() (authz.Rule, error) { return authz.CognitoGroups(issuer, "") }, + func() (authz.Rule, error) { return authz.CognitoGroups("", "operators") }, + func() (authz.Rule, error) { return authz.CognitoGroups(issuer, "\xff") }, + func() (authz.Rule, error) { return authz.IAMPrincipal("arn:aws:iam::123456789012:role/Reader") }, + func() (authz.Rule, error) { return authz.IAMRoleSessions("aws", "*", "Reader") }, + func() (authz.Rule, error) { return authz.IAMRoleSessions("aws", "123456789012", "Read*") }, + func() (authz.Rule, error) { return authz.IAMRoleSessions("aws", "123456789012", "path/Reader") }, + func() (authz.Rule, error) { return authz.IAMRoleSessions("*", "123456789012", "Reader") }, + func() (authz.Rule, error) { return authz.IAMRoleSessions("aws:sts", "123456789012", "Reader") }, + } { + if _, err := construct(); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Errorf("constructor = %v", err) + } + } + for _, scope := range []string{"", "two scopes", "with\ttab", "quote\"", "slash\\", "\x7f", "日本語"} { + if _, err := authz.JWTScopes(issuer, scope); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Errorf("scope %q = %v", scope, err) + } + } + values := []string{"operators"} + r, err := authz.CognitoGroups(issuer, values...) + r = rule(t, r, err) + values[0] = "admins" + sources := []identity.Source{identity.SourceCustomAssertion} + sourceRule, err := authz.Sources(sources...) + sourceRule = rule(t, sourceRule, err) + sources[0] = identity.SourceGatewayAssertion + for _, source := range []identity.Source{identity.SourceCustomAssertion, identity.SourceGatewayAssertion, identity.SourceLocallyVerifiedToken} { + caller := jwtCaller(t, map[string]any{"iss": issuer, "sub": "alice", "cognito:groups": []string{"operators"}}, source) + if err := r.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); err != nil { + t.Error(err) + } + want := authz.ErrDenied + if source == identity.SourceCustomAssertion { + want = nil + } + if err := sourceRule.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); !errors.Is(err, want) { + t.Error(err) + } + } +} + +func TestMandatoryGuardAndInapplicableBranch(t *testing.T) { + var calls int + provider := checked(t, func(context.Context, authz.Request) (bool, error) { calls++; return true, nil }) + jwtOnly, err := authz.JWTSubject(issuer, "alice") + jwtOnly = rule(t, jwtOnly, err) + jwtPath, err := authz.All(jwtOnly, provider) + jwtPath = rule(t, jwtPath, err) + iamPath, err := authz.IAMPrincipal("arn:aws:iam::123456789012:user/Alice") + iamPath = rule(t, iamPath, err) + paths, err := authz.Any(jwtPath, iamPath) + paths = rule(t, paths, err) + if err := paths.Authorize(t.Context(), request(t)); err != nil || calls != 0 { + t.Fatalf("inapplicable lookup: error=%v calls=%d", err, calls) + } + guard := checked(t, func(context.Context, authz.Request) (bool, error) { return false, nil }) + guarded, err := authz.All(guard, paths) + guarded = rule(t, guarded, err) + if err := guarded.Authorize(t.Context(), request(t)); !errors.Is(err, authz.ErrDenied) { + t.Fatal(err) + } +} + +func FuzzPolicyInputs(f *testing.F) { + for _, s := range []string{"", "orders/*", "日本語", "arn:aws:iam::123456789012:user/a*?/Alice", "arn:aws:sts::123456789012:assumed-role/Reader/session"} { + f.Add(s, "aws", "123456789012") + } + f.Fuzz(func(t *testing.T, value, partition, account string) { + for _, construct := range []func() (authz.Rule, error){ + func() (authz.Rule, error) { return authz.JWTSubject(issuer, value) }, + func() (authz.Rule, error) { return authz.JWTClient(issuer, value) }, + func() (authz.Rule, error) { return authz.JWTScopes(issuer, value) }, + func() (authz.Rule, error) { return authz.CognitoGroups(issuer, value) }, + func() (authz.Rule, error) { return authz.IAMPrincipal(value) }, + func() (authz.Rule, error) { return authz.IAMRoleSessions(partition, account, value) }, + } { + r, err := construct() + if err != nil { + if !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Fatal(err) + } + continue + } + if err := r.Authorize(t.Context(), authz.Request{Action: "read"}); !errors.Is(err, authz.ErrUnauthenticated) { + t.Fatalf("anonymous allowed: %v", err) + } + } + caller, callerErr := identity.NewIAM(value, identity.SourceCustomAssertion) + principal, policyErr := authz.IAMPrincipal(value) + if (callerErr == nil) != (policyErr == nil) { + t.Fatal("principal policy disagrees with identity grammar") + } + if policyErr == nil { + if err := principal.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); err != nil { + t.Fatal(err) + } + } + family, err := authz.IAMRoleSessions(partition, account, value) + if err == nil { + arn := "arn:" + partition + ":sts::" + account + ":assumed-role/" + value + "/session" + caller := iamCaller(t, arn) + if err := family.Authorize(t.Context(), authz.Request{Caller: caller, Action: "read"}); err != nil { + t.Fatal(err) + } + } + }) +} diff --git a/authz/rule.go b/authz/rule.go new file mode 100644 index 0000000..87553db --- /dev/null +++ b/authz/rule.go @@ -0,0 +1,156 @@ +package authz + +import ( + "context" + "errors" + "slices" + "unicode/utf8" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Authorization errors contain no request facts or callback diagnostics. +var ( + ErrInvalidConfiguration = errors.New("authz: invalid configuration") + ErrInvalidRequest = errors.New("authz: invalid request") + ErrUnauthenticated = errors.New("authz: unauthenticated") + ErrDenied = errors.New("authz: denied") + ErrUnavailable = errors.New("authz: unavailable") +) + +// Request supplies facts passed by value to each check. Action must +// be nonempty UTF-8; Resource is optional UTF-8. Both compare exactly without +// normalization and have application-defined meaning. A resource-aware check +// must reject an absent Resource. Caller is explicit, not read from context. +type Request struct { + Caller identity.Caller + Action string + Resource string +} + +// CheckFunc decides whether a request matches. Return false, nil for denial; +// any error becomes ErrUnavailable, even when accompanied by true. Checks run +// synchronously, must honor ctx and must be safe for concurrent use when shared. +// The caller owns dependency deadlines. Panics propagate. +type CheckFunc func(context.Context, Request) (bool, error) + +// Rule holds immutable private configuration. Copies may be shared concurrently. +// The zero value is unconfigured and never allows. Construction performs no I/O +// and never invokes callbacks. A rule is limited to depth 32 (a leaf has depth 1) +// and 1,024 expanded nodes, counting each occurrence of a shared subtree. +type Rule struct { + node *node +} + +type operation uint8 + +const ( + opCheck operation = iota + opAll + opAny + maxDepth = 32 + maxNodes = 1024 +) + +type node struct { + op operation + check CheckFunc + children []Rule + depth int + nodes int +} + +// Check constructs a rule from a nonnil callback. A nil callback returns +// ErrInvalidConfiguration. Reusing a check does not deduplicate its evaluation. +func Check(check CheckFunc) (Rule, error) { + if check == nil { + return Rule{}, ErrInvalidConfiguration + } + return Rule{node: &node{op: opCheck, check: check, depth: 1, nodes: 1}}, nil +} + +// All requires every child to match, evaluating left to right. It stops on the +// first nonmatch or error. Empty, unconfigured or oversized trees return +// ErrInvalidConfiguration. The input slice is copied. +func All(rules ...Rule) (Rule, error) { return combine(opAll, rules) } + +// Any requires one child to match, evaluating left to right. It stops on the +// first match or error; an error never falls through to another permission path. +// Empty, unconfigured or oversized trees return ErrInvalidConfiguration. The +// input slice is copied. Mandatory guards belong in an outer All. +func Any(rules ...Rule) (Rule, error) { return combine(opAny, rules) } + +func combine(op operation, rules []Rule) (Rule, error) { + if len(rules) == 0 || len(rules) >= maxNodes { + return Rule{}, ErrInvalidConfiguration + } + n := &node{op: op, depth: 1, nodes: 1} + for _, rule := range rules { + child := rule.node + if child == nil || child.depth >= maxDepth || child.nodes > maxNodes-n.nodes { + return Rule{}, ErrInvalidConfiguration + } + n.depth = max(n.depth, child.depth+1) + n.nodes += child.nodes + } + n.children = slices.Clone(rules) + return Rule{node: n}, nil +} + +// Authorize returns nil only for an explicit allow. It checks rule/context +// configuration, cancellation, request validity and a nonanonymous caller in +// that order. No check runs for an anonymous or invalid request. Nonmatches +// return ErrDenied; all callback errors are sanitized to ErrUnavailable. +// Cancellation is checked between nodes and before returning, and returns +// ctx.Err(), never context.Cause. No results are cached or installed in context. +func (r Rule) Authorize(ctx context.Context, request Request) error { + if r.node == nil || ctx == nil { + return ErrInvalidConfiguration + } + if err := ctx.Err(); err != nil { + return err + } + var allowed bool + var err error + switch { + case request.Action == "" || !utf8.ValidString(request.Action) || !utf8.ValidString(request.Resource): + err = ErrInvalidRequest + case request.Caller.Kind() == identity.KindAnonymous: + err = ErrUnauthenticated + default: + allowed, err = r.node.evaluate(ctx, request) + } + if canceled := ctx.Err(); canceled != nil { + return canceled + } + if err != nil { + return err + } + if !allowed { + return ErrDenied + } + return nil +} + +func (n *node) evaluate(ctx context.Context, request Request) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + if n.op == opCheck { + allowed, err := n.check(ctx, request) + if err != nil { + return false, ErrUnavailable + } + return allowed, nil + } + for _, child := range n.children { + allowed, err := child.node.evaluate(ctx, request) + if err != nil { + return false, err + } + if allowed == (n.op == opAny) { + return allowed, nil + } + } + return n.op == opAll, nil +} diff --git a/authz/rule_test.go b/authz/rule_test.go new file mode 100644 index 0000000..e9ef3ab --- /dev/null +++ b/authz/rule_test.go @@ -0,0 +1,267 @@ +package authz_test + +import ( + "context" + "errors" + "fmt" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + + "github.com/asteroid-computing/go-lambda-edge/authz" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func rule(t testing.TB, r authz.Rule, err error) authz.Rule { + t.Helper() + if err != nil { + t.Fatal(err) + } + return r +} + +func checked(t testing.TB, f authz.CheckFunc) authz.Rule { + t.Helper() + r, err := authz.Check(f) + return rule(t, r, err) +} + +func iamCaller(t testing.TB, arn string, opts ...identity.IAMOption) identity.Caller { + t.Helper() + c, err := identity.NewIAM(arn, identity.SourceCustomAssertion, opts...) + if err != nil { + t.Fatal(err) + } + return c +} + +func request(t testing.TB) authz.Request { + t.Helper() + return authz.Request{Caller: iamCaller(t, "arn:aws:iam::123456789012:user/Alice"), Action: "orders.read", Resource: "tenant/42/order/7"} +} + +func TestAuthorizeValidationOrder(t *testing.T) { + var calls int + r := checked(t, func(context.Context, authz.Request) (bool, error) { calls++; return true, nil }) + valid := request(t) + ctx, cancel := context.WithCancelCause(t.Context()) + cancel(errors.New("private cancellation cause")) + for _, tc := range []struct { + name string + rule authz.Rule + ctx context.Context + request authz.Request + want error + }{ + {"zero_before_cancellation", authz.Rule{}, ctx, valid, authz.ErrInvalidConfiguration}, + {"nil_context", r, nil, valid, authz.ErrInvalidConfiguration}, + {"canceled_before_request", r, ctx, authz.Request{}, context.Canceled}, + {"invalid_before_anonymous", r, t.Context(), authz.Request{}, authz.ErrInvalidRequest}, + {"anonymous", r, t.Context(), authz.Request{Action: "orders.read"}, authz.ErrUnauthenticated}, + {"invalid_action", r, t.Context(), authz.Request{Caller: valid.Caller, Action: "\xff"}, authz.ErrInvalidRequest}, + {"invalid_resource", r, t.Context(), authz.Request{Caller: valid.Caller, Action: valid.Action, Resource: "\xff"}, authz.ErrInvalidRequest}, + } { + t.Run(tc.name, func(t *testing.T) { + if err := tc.rule.Authorize(tc.ctx, tc.request); !errors.Is(err, tc.want) { + t.Errorf("Authorize = %v; want %v", err, tc.want) + } + }) + } + if calls != 0 { + t.Fatalf("invalid requests invoked callback %d times", calls) + } + // authz preserves application strings; it does not impose the action header grammar. + valid.Action, valid.Resource = " Read 日本語 ", "" + if err := r.Authorize(t.Context(), valid); err != nil || calls != 1 { + t.Fatalf("valid UTF-8 request = %v, calls=%d", err, calls) + } +} + +func TestCallbackOutcomes(t *testing.T) { + private := errors.New("private grant service diagnostic") + for _, callbackErr := range []error{nil, private, context.Canceled, context.DeadlineExceeded, authz.ErrDenied, authz.ErrInvalidConfiguration, authz.ErrInvalidRequest, authz.ErrUnauthenticated, authz.ErrUnavailable, fmt.Errorf("private wrapper: %w", authz.ErrDenied)} { + for _, allow := range []bool{false, true} { + r := checked(t, func(context.Context, authz.Request) (bool, error) { return allow, callbackErr }) + want := authz.ErrDenied + if callbackErr != nil { + want = authz.ErrUnavailable + } else if allow { + want = nil + } + err := r.Authorize(t.Context(), request(t)) + if !errors.Is(err, want) || err != nil && (errors.Is(err, private) || strings.Contains(err.Error(), "private")) { + t.Errorf("allow=%t callback=%v: got %v; want sanitized %v", allow, callbackErr, err, want) + } + } + } +} + +func TestOrderedCombinators(t *testing.T) { + for _, op := range []string{"all", "any"} { + for _, first := range []string{"allow", "deny", "error"} { + for _, second := range []string{"allow", "deny", "error"} { + t.Run(op+"/"+first+"/"+second, func(t *testing.T) { + var visited []string + makeCheck := func(label, outcome string) authz.Rule { + return checked(t, func(context.Context, authz.Request) (bool, error) { + visited = append(visited, label) + if outcome == "error" { + return true, errors.New("private") + } + return outcome == "allow", nil + }) + } + children := []authz.Rule{makeCheck("first", first), makeCheck("second", second)} + combine := authz.All + if op == "any" { + combine = authz.Any + } + r, err := combine(children...) + r = rule(t, r, err) + if len(visited) != 0 { + t.Fatal("constructor called a check") + } + wantVisits := []string{"first"} + outcome := first + if op == "all" && first == "allow" || op == "any" && first == "deny" { + wantVisits = append(wantVisits, "second") + outcome = second + } + want := map[string]error{"allow": nil, "deny": authz.ErrDenied, "error": authz.ErrUnavailable}[outcome] + if err := r.Authorize(t.Context(), request(t)); !errors.Is(err, want) || !slices.Equal(visited, wantVisits) { + t.Errorf("Authorize = %v, visits=%v; want %v, %v", err, visited, want, wantVisits) + } + }) + } + } + } +} + +func TestCancellationWinsAndStopsChecks(t *testing.T) { + for _, allow := range []bool{false, true} { + for _, callbackErr := range []error{nil, errors.New("private dependency failure")} { + for _, combine := range []func(...authz.Rule) (authz.Rule, error){authz.All, authz.Any} { + ctx, cancel := context.WithCancelCause(t.Context()) + first := checked(t, func(context.Context, authz.Request) (bool, error) { + cancel(errors.New("secret cause")) + return allow, callbackErr + }) + second := checked(t, func(context.Context, authz.Request) (bool, error) { + t.Error("check ran after cancellation") + return true, nil + }) + r, err := combine(first, second) + r = rule(t, r, err) + if err := r.Authorize(ctx, request(t)); err != context.Canceled { + t.Errorf("Authorize = %v; want context.Canceled", err) + } + } + } + } +} + +func TestConfigurationBounds(t *testing.T) { + leaf := checked(t, func(context.Context, authz.Request) (bool, error) { return true, nil }) + if _, err := authz.Check(nil); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Fatal(err) + } + for _, combine := range []func(...authz.Rule) (authz.Rule, error){authz.All, authz.Any} { + for _, children := range [][]authz.Rule{nil, {authz.Rule{}}, {leaf, authz.Rule{}}} { + if _, err := combine(children...); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Fatal(err) + } + } + deep := leaf + for range 31 { + next, err := combine(deep) + deep = rule(t, next, err) + } + if err := deep.Authorize(t.Context(), request(t)); err != nil { + t.Fatal(err) + } + if _, err := combine(deep); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Fatalf("depth 33 = %v", err) + } + wide := slices.Repeat([]authz.Rule{leaf}, 1023) + r, err := combine(wide...) + r = rule(t, r, err) + if err := r.Authorize(t.Context(), request(t)); err != nil { + t.Fatal(err) + } + if _, err := combine(append(wide, leaf)...); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Fatalf("1025 nodes = %v", err) + } + // This shared DAG has only ten distinct nodes but 1,023 expanded visits. + dag := leaf + for range 9 { + next, err := combine(dag, dag) + dag = rule(t, next, err) + } + if _, err := combine(dag); err != nil { + t.Fatal(err) + } + if _, err := combine(dag, leaf); !errors.Is(err, authz.ErrInvalidConfiguration) { + t.Fatalf("expanded DAG bound = %v", err) + } + } +} + +func TestOwnershipAndConcurrentReuse(t *testing.T) { + var calls atomic.Int64 + want := request(t) + leaf := checked(t, func(_ context.Context, got authz.Request) (bool, error) { + calls.Add(1) + if got.Action != want.Action || got.Resource != want.Resource || got.Caller.Kind() != want.Caller.Kind() { + t.Errorf("check received changed request: action=%q resource=%q", got.Action, got.Resource) + } + got.Action, got.Resource, got.Caller = "changed", "changed", identity.Caller{} + return true, nil + }) + children := []authz.Rule{leaf, leaf} + r, err := authz.All(children...) + r = rule(t, r, err) + children[0] = authz.Rule{} + var wg sync.WaitGroup + for range 32 { + wg.Go(func() { + if err := r.Authorize(t.Context(), want); err != nil { + t.Error(err) + } + }) + } + wg.Wait() + if calls.Load() != 64 { + t.Fatalf("shared check calls=%d; want 64", calls.Load()) + } +} + +func TestCallbackPanicPropagates(t *testing.T) { + r := checked(t, func(context.Context, authz.Request) (bool, error) { panic("programmer bug") }) + defer func() { + if got := recover(); got != "programmer bug" { + t.Errorf("panic = %v", got) + } + }() + _ = r.Authorize(t.Context(), request(t)) +} + +func TestExplicitCallerAndCallerContext(t *testing.T) { + facts := request(t) + other := iamCaller(t, "arn:aws:iam::123456789012:user/Bob") + ctx, err := identity.WithCaller(t.Context(), other) + if err != nil { + t.Fatal(err) + } + r := checked(t, func(got context.Context, req authz.Request) (bool, error) { + if got != ctx { + t.Error("callback did not receive caller context") + } + i, _ := req.Caller.IAM() + return i.PrincipalARN() == "arn:aws:iam::123456789012:user/Alice", nil + }) + if err := r.Authorize(ctx, facts); err != nil { + t.Fatal(err) + } +} diff --git a/authz/transport_test.go b/authz/transport_test.go new file mode 100644 index 0000000..d2a029c --- /dev/null +++ b/authz/transport_test.go @@ -0,0 +1,89 @@ +package authz_test + +import ( + "context" + "encoding/json/v2" + "errors" + "net/http" + "net/http/httptest" + "testing" + + "github.com/aws/aws-lambda-go/events" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authz" +) + +func TestAuthorizationCancellationAcrossTransports(t *testing.T) { + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { + t.Run(format, func(t *testing.T) { + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + calls := 0 + policy := checked(t, func(context.Context, authz.Request) (bool, error) { + calls++ + cancel(errors.New("private cancellation cause")) + return true, errors.New("private grant outage") + }) + facts := request(t) + h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := policy.Authorize(r.Context(), facts); err != nil { + if err != context.Canceled { + t.Errorf("policy error=%v; want context.Canceled", err) + } + writePolicyError(w, 503, "unavailable") + return + } + t.Error("canceled authorization executed the action") + }) + if format == "native" { + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequestWithContext(ctx, "GET", "/", nil)) + if w.Code != 503 || calls != 1 { + t.Fatalf("native response=%d checks=%d", w.Code, calls) + } + return + } + adapter, err := edge.New(h) + if err != nil { + t.Fatal(err) + } + v1 := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}} + v2 := events.APIGatewayV2HTTPRequest{Version: "2.0", RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}} + switch format { + case "typed_v1": + response, invokeErr := adapter.HandleV1(ctx, v1) + err = invokeErr + if response.StatusCode != 0 { + t.Error("canceled invocation returned a usable response") + } + case "typed_v2": + response, invokeErr := adapter.HandleV2(ctx, v2) + err = invokeErr + if response.StatusCode != 0 { + t.Error("canceled invocation returned a usable response") + } + default: + var event any = v1 + if format == "raw_http_v2" { + event = v2 + } + wire, marshalErr := json.Marshal(event) + if marshalErr != nil { + t.Fatal(marshalErr) + } + if format == "raw_http_v1" { + wire = append([]byte(`{"version":"1.0",`), wire[1:]...) + } + response, invokeErr := adapter.Invoke(ctx, wire) + err = invokeErr + if len(response) != 0 { + t.Error("canceled invocation returned a usable response") + } + } + if !errors.Is(err, context.Canceled) || calls != 1 { + t.Errorf("invocation=%v checks=%d; want canceled after one check", err, calls) + } + }) + } +} diff --git a/docs/authn.md b/docs/authn.md index 5964656..2d96319 100644 --- a/docs/authn.md +++ b/docs/authn.md @@ -9,8 +9,9 @@ with ordinary Go HTTP servers and the root edge adapter's raw/typed entry points with direct JSON v2 and bounded JWKS caching. Supply its Verify method, another fully verifying Bearer function, IAM proofs alone, or both mechanisms. See the [Cognito guide](cognito.md) for issuer/client/audience restrictions and cache -ownership. The authz package remains future work; applications must authorize -actions. Default gateway identity remains disabled. +ownership. The implemented [authz rules](authz.md) authorize explicit caller, +action and resource facts; applications own their dispatch and grant resolution. +Default gateway identity remains disabled. ## Configure both mechanisms @@ -135,6 +136,10 @@ Authorize that caller for that exact selection, then execute it. Both IAM and JWT callers can reach the same dispatcher. Successful IAM authentication does not permit every AWS identity to execute your actions; deny by default. +The [authorization dispatcher example](../authz/example_test.go) pairs each rule +with its handler, keeps the selected resource consistent and maps 403 versus 503 +without exposing policy diagnostics. It configures production verifier types. + Complete authentication and authorization before writing or flushing a response. The caller is a snapshot and is not automatically refreshed during a stream. Application-specific ongoing authorization remains explicit. diff --git a/docs/authz.md b/docs/authz.md new file mode 100644 index 0000000..bcf16f4 --- /dev/null +++ b/docs/authz.md @@ -0,0 +1,191 @@ +# Application authorization + +`authz` checks an authenticated caller's permission to perform an application +action on an optional resource. It is implemented with only standard-library and +`identity` dependencies. It does not authenticate credentials, evaluate AWS IAM +policies, resolve grants automatically or write HTTP responses. + +The [runnable examples](../authz/example_test.go) show an application grant check +and a header-selected HTTP dispatcher with Cognito and IAM-proof authentication. +The dispatcher example configures real verifiers but sends no credentials, so +running it makes no network calls. Integration tests separately exercise signed +Cognito tokens and production IAM-proof verification against synthetic JWKS/STS +responses, then authorize requests across native HTTP and raw/typed Gateway +entry points. No live AWS interoperability test has run. + +## Construct rules once + +Every constructor returns `(authz.Rule, error)`. Handle configuration errors +before serving requests. Rules are immutable; value copies share private nodes. +Constructor slices are copied. Custom callbacks must be safe for concurrent use. + +```go +scopes, err := authz.JWTScopes(issuer, "orders.read") +if err != nil { + return nil, err +} +role, err := authz.IAMRoleSessions("aws", "123456789012", "OrderReader") +if err != nil { + return nil, err +} +permission, err := authz.Any(scopes, role) +if err != nil { + return nil, err +} +``` + +Pass explicit facts after parsing input and selecting the operation: + +```go +facts := authz.Request{ + Caller: identity.FromContext(r.Context()), + Action: selectedAction, + Resource: selectedResource, +} +err := permission.Authorize(r.Context(), facts) +``` + +Only nil means allow. Action must be nonempty UTF-8; Resource is optional UTF-8. +Neither is trimmed, case-folded or interpreted as a path, URL or ARN. The action +header parser has its own stricter HTTP-token grammar. A resource-aware check +must reject an absent resource. The core uses `Request.Caller` and does not read +a second identity from context or install authorization state there. + +## Exact predicates + +| Constructor | Matches | +| --- | --- | +| `Sources(sources...)` | Any listed nonanonymous identity producer | +| `JWTSubject(issuer, subject)` | Exact issuer and subject | +| `JWTClient(issuer, clientID)` | Exact issuer and client_id | +| `JWTScopes(issuer, required...)` | Exact issuer and every required OAuth scope | +| `CognitoGroups(issuer, required...)` | Exact issuer and every required Cognito group | +| `IAMPrincipal(arn)` | One full, supported caller ARN | +| `IAMRoleSessions(partition, accountID, roleName)` | Any valid session of that exact role name/account/partition | + +Identifiers must be nonempty UTF-8. Source and collection lists must be nonempty; +SourceNone and unknown sources are invalid. Scopes follow RFC 6749's scope-token +grammar. Duplicate required values do not add requirements. All comparisons are +case-sensitive. Use `Any` of separate rules for alternative scope/group choices. + +Wrong caller kinds, missing claims and unavailable collections do not match. +Flattened gateway groups or `scp` text are never split or reparsed. Dedicated +gateway scopes can be used when supplied through the identity producer. A scope +named `orders/*` matches that literal value, with no wildcard expansion. Cognito +groups, OAuth scopes and server-resolved application grants remain separate facts. + +JWT predicates are always issuer-qualified. Subject does not imply a human; +client_id does not imply a machine grant and is not an audience. Predicates do +not check signature, expiry, audience, token_use or revocation themselves. The +configured authentication producer must establish those guarantees. + +Predicates accept every supported producer of their matching kind. An outer +`All(Sources(...), permission)` can restrict provenance. Sources alone is a broad +allow for those producers. Source is trusted-code attribution: constructing an +identity with a source label does not establish credential possession. + +`IAMPrincipal` uses `identity.NewIAM`'s caller grammar. A root ARN allows only +that root caller, not its whole account. An assumed-role ARN includes its exact +session name. Literal `*` and `?` in valid IAM user paths remain literal. Bare +IAM role ARNs are invalid caller forms. + +`IAMRoleSessions` deliberately broadens matching to a role's sessions. It rejects +role paths and wildcard configuration. It preserves the caller's full session +ARN without inventing an IAM role ARN. This is a **role-name policy**: deleting +and recreating the role with the same name can match again. It does not bind a +unique role ID, identify the original actor, evaluate session policies or prove +execute-api permission. Lifecycle-sensitive enrollment needs a consumer-owned +stable identifier and resolution policy; PrincipalID remains opaque. + +## Ordered composition and application grants + +`All` stops at the first nonmatch or error. `Any` stops at the first match or +error. An error never becomes a nonmatch followed by another permission path. +Thus `Any(allow, unavailable)` allows without running the second check, while +`Any(unavailable, allow)` fails without running the second check. Ordering can +change dependency work and outcomes. + +Put mandatory application guards outside alternative paths: + +```text +All(mandatoryApplicationGuard, Any(jwtPermissionPath, iamPermissionPath)) +``` + +Within a branch, put cheap identity predicates before its provider lookup. A +custom check receives the caller context and explicit action/resource: + +```go +grant, err := authz.Check(func(ctx context.Context, req authz.Request) (bool, error) { + if req.Resource == "" { + return false, nil + } + return store.Allowed(ctx, req.Caller, req.Action, req.Resource) +}) +``` + +The application chooses that store and its enrollment keys. Use issuer plus +subject for subject identity, or explicitly issuer plus client_id for a supported +client identity. Do not silently fall back from missing subject to a user named +by client_id, or use email, account or session name as an inferred global key. +Resolve tenant membership, suspension and resource grants in one logical check. +Known unregistered/suspended/unauthorized callers return false, nil. A dependency +that cannot decide returns an error. Never obtain grants from client-supplied +headers or inject them into authenticated claims. + +If execution needs a resolved object, resolve it once before the policy and keep +the immutable result in the application's own execution plan. The core provides +no automatic grant model, cache or deduplication. A reused check can run once per +evaluated occurrence. It is not an audit callback. Checks run synchronously with +the caller context; applications own dependency timeouts and must honor +cancellation. Programmer panics propagate. + +Empty All/Any and zero child rules are invalid. The maximum depth is 32 with a +leaf at depth 1. A rule can contain at most 1,024 expanded nodes, counting every +occurrence of a shared subtree. Constructors reject excess work before evaluation. +These are policy configuration bounds, not AWS quotas or limits on callback work. +There is no Not, anonymous allow rule, retry or parallel evaluation. Public routes +intentionally bypass protected-route authorization. + +## Outcomes and HTTP dispatch + +Authorize checks rule/non-nil context configuration, cancellation, request +validity and nonanonymous identity in that order, then evaluates checks. Anonymous +callers never reach a custom callback, even a permissive one. Cancellation is +checked between nodes and before returning; it returns `ctx.Err()`, never the +private cancellation cause. It cannot preempt arbitrary callback code. + +Every nonnil callback error becomes `ErrUnavailable`, even if the callback returns +true, a package sentinel such as ErrDenied, or an internal context error while +the caller context remains live. Return false, nil for denial. Public errors +contain no callback error chain or caller/action/resource diagnostics. + +| Result (`errors.Is`) | Consumer HTTP treatment | +| --- | --- | +| nil | Execute the selected action/resource | +| `ErrUnauthenticated` | 401 with configured challenges; normally handled earlier by authn | +| `ErrDenied` | 403; include the applicable Bearer challenge on Bearer-protected routes | +| `ErrUnavailable` | 503 without an authentication challenge | +| `ErrInvalidConfiguration`, `ErrInvalidRequest` | 500; reject malformed client input earlier with an appropriate 4xx | +| Caller cancellation | Never execute; best-effort 503 if transport still permits a response | + +For denial on a route configured to accept Bearer, a generic +`WWW-Authenticate: Bearer realm="orders"` avoids exposing the reason. Only use +`insufficient_scope` and a scope hint when that is accurately the requirement. +Do not describe every group, tenant or IAM denial as an OAuth scope error. The +example includes a generic Bearer challenge because its route accepts Bearer, +including when the current caller used IAM. The core does not infer a wire +authentication scheme from Caller.Kind. Gateway-only routes own their configured +challenge policy. Preserve CORS, use safe error codes and no-store responses. + +The consumer owns an immutable registry entry pairing a rule with its handler. +Parse the action header once, select once, resolve and validate the target, then +authorize and execute that same entry/target. Unknown actions have no unprotected +fallback. Do not reread mutable headers for policy or execution. Resource version +and transaction consistency remain the application's responsibility. + +The same handler works through ordinary net/http or `edge.New(handler)`. Complete +authorization before response commitment or streaming flush. This milestone does +not introduce public streaming entry points or ongoing stream reauthorization. + +See [decision 0021](decisions/0021-authorization-rules.md) for the approved design, +alternatives and official AWS/IETF evidence. diff --git a/docs/decisions/0021-authorization-rules.md b/docs/decisions/0021-authorization-rules.md index 87427be..0c91aff 100644 --- a/docs/decisions/0021-authorization-rules.md +++ b/docs/decisions/0021-authorization-rules.md @@ -1,6 +1,7 @@ # 0021: Application authorization rules and dispatcher integration -Status: proposed; no dependent implementation approved. +Status: accepted; rules, examples and local validation implemented. +The user approved Z1–Z7 on 2026-09-17. Reviewed on 2026-09-16 against the implemented identity/authn/edge boundaries. ## Evidence and scope @@ -316,7 +317,32 @@ JSON v2 for JSON examples. An error response does not execute an action. Finish authorization before any response commitment or streaming flush. This proposal does not add mid-stream reauthorization or a framework-owned audit stream. -## Implementation after approval +## Implementation and validation + +Implemented on 2026-09-17 in `authz`: explicit request facts, immutable Check, +All/Any, all seven predicate constructors, sanitized errors and policy tree +bounds. Production imports only the standard library and identity. IAM policy +configuration reuses identity.NewIAM's grammar; the role-family constructor +validates a sample session and never exposes or fabricates an IAM role ARN. + +Contract tests cover the criteria below, including concurrent reuse, copied +configuration, expanded shared-tree bounds, literal IAM paths/scopes, unavailable +gateway collections, identity-source and issuer boundaries, callback error +sanitization and cancellation. Application fixtures verify grant lookup counts, +403 versus 503, safe JSON v2 envelopes and Bearer challenges, CORS preservation, +and authorization/execution of the same action/resource despite header mutation. +Cognito signature verification and IAM-proof processing run before policy in +native HTTP plus five raw/typed Gateway paths, using synthetic JWKS/STS responses. +Cancellation during policy evaluation is also checked across those paths. + +Full race tests, vet, formatting and Linux arm64/amd64 builds pass on Go 1.27.1. +A 20-second policy fuzz run completed 45,257 cases without failure. The +[consumer guide](../authz.md) and [runnable examples](../../authz/example_test.go) +document HTTP ownership, resource consistency and grant resolution. No live AWS +calls, deployment or GitHub CI run occurred. Public streaming and generic +principal/grant models remain deferred as described in this decision. + +Completed acceptance criteria: 1. Implement core immutable rules, constructor validation and sanitized outcomes. 2. Add exact JWT/IAM/source predicates and ordered All/Any with bounded trees. @@ -334,7 +360,8 @@ does not add mid-stream reauthorization or a framework-owned audit stream. ## Resolution -Awaiting review of Z1-Z7. The main choices are explicit action/resource inputs, +The user approved Z1–Z7 on 2026-09-17. The accepted choices are explicit action/resource inputs, exact issuer-qualified predicates, a deliberately named role-session-family check, ordered error-stopping combinators, and consumer-owned grant resolution -and HTTP dispatch for the initial release. No production behavior changes here. +and HTTP dispatch for the initial release. These contracts are now implemented +and locally validated without expanding the approved public API. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 0911753..3f5b301 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -29,4 +29,4 @@ Keep related decisions small enough to review and implement independently. | [0018: Gateway identity recognition](0018-gateway-identity-recognition.md) | Accepted; extraction and buffered invocation implemented | Which native field mappings, presence rules and fixture qualifications should precede public invocation wiring? | | [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | | [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | -| [0021: Authorization rules](0021-authorization-rules.md) | Proposed; design review only | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | +| [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | diff --git a/docs/plan.md b/docs/plan.md index bb0e06b..eb83a0e 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -63,7 +63,8 @@ default; `WithGatewayIdentity(true)` opts in for both IAM and JWT/Cognito. Local verification is explicitly composed. Constructor details and producer conflicts are settled in decisions 0012–0018; decision 0019 accepts the now-implemented common HTTP authentication selector. Decision 0020 accepts the now-implemented -Cognito access-token verifier and JWKS cache. Authz remains a separate review. +Cognito access-token verifier and JWKS cache. Decision 0021 accepts authorization +rules and the consumer-owned resolution/dispatch boundary. ## Planned package boundaries @@ -73,7 +74,7 @@ Cognito access-token verifier and JWKS cache. Authz remains a separate review. | `identity` | Validated callers, claim representation, context transport; standard library only | | `authn` | Explicit IAM-proof/OAuth selection, Cognito verification and JWKS caching | | `iamproof` | Exported client proof generator and bounded online STS verifier | -| `authz` | Decisions, rules, combinators, middleware, application-principal resolution | +| `authz` | Exact identity predicates, ordered rules and context-aware application checks; consumers own grant resolution and HTTP dispatch | | `edgetest` | Consumer identity fixtures and gateway event helpers | Avoid a broad root-package facade that reexports every companion type. Keep AWS @@ -349,23 +350,32 @@ storms, and dependency outages remain distinguishable from invalid credentials. ### 5. Authorization and consumer ergonomics -[Decision 0021](decisions/0021-authorization-rules.md) records the proposed initial -authz API and design graph. It recommends immutable rules over explicit caller, +[Decision 0021](decisions/0021-authorization-rules.md), approved on 2026-09-17, +accepts the initial authz API and design graph: immutable rules over explicit caller, action and resource facts, exact predicates and ordered combinators. Application principal/grant resolution and HTTP dispatch initially remain consumer-owned -through context-aware custom checks and runnable examples. That narrowing and -the public/error contracts await user review; no authz code is implemented yet. - -- Implement reviewed rules/combinators, scope/group/grant separation, principal - resolution, and explicit authentication precedence. -- Review missing/invalid credential responses (ordinarily 401), authenticated - denial (403), and dependency unavailability (503), including bearer challenges - and behavior on public routes. -- Define ARN matching precisely, including separators, partitions, account - boundaries, and STS role sessions. Do not approximate IAM policy evaluation. -- Validate nil/empty authorizers and invalid identities consistently. -- Provide consumer fixtures, runnable examples, and limited structured logging - that excludes raw credentials and unnecessary profile claims. +through context-aware custom checks and runnable examples. Z1–Z7 are implemented +and locally validated. Production authz uses only the standard library and identity. + +- [x] Implement immutable checks, ordered All/Any, explicit action/resource + requests, exact JWT/source/IAM predicates and bounded policy trees. +- [x] Preserve scope/group/grant separation and consumer-owned principal + resolution through context-aware custom checks. +- [x] Document/test 401, 403 and 503 boundaries, safe Bearer challenges and + cancellation. Public routes bypass protected-route authorization explicitly. +- [x] Match exact caller ARNs and explicit role-session families, including + partition/account/path boundaries and role-name recreation limitations. +- [x] Reject zero/empty/nil rules and anonymous callers; sanitize callback errors. +- [x] Provide consumer fixtures, runnable examples and the + [authorization guide](authz.md). Logging hooks and a generic exported + principal/grant resolver remain deferred under decision 0021. + +Full race tests, vet, formatting and Linux arm64/amd64 builds pass on Go 1.27.1. +Policy fuzzing completed 45,257 cases without failure. Native HTTP and five +raw/typed Gateway paths exercise authorization after Cognito signature and IAM +proof verification using local synthetic provider responses. Tests cover grant +denials/outages, provider call counts, immutable action/resource selection, safe +responses and cancellation. No live AWS or GitHub CI run occurred. Acceptance: examples work both in Lambda and ordinary HTTP applications, while authorization decisions remain inspectable and deny by default. From 24244ce8a2bbf89936b4f42787d42738e73f77b9 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Thu, 17 Sep 2026 07:11:13 +0100 Subject: [PATCH 35/44] docs: review streaming writer policies and SDK compatibility --- docs/decisions/0009-streaming-boundaries.md | 7 + .../decisions/0022-streaming-writer-review.md | 224 ++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 10 + internal/streamprobe/http_test.go | 62 +++++ internal/streamprobe/runtime_test.go | 19 +- 6 files changed, 317 insertions(+), 6 deletions(-) create mode 100644 docs/decisions/0022-streaming-writer-review.md create mode 100644 internal/streamprobe/http_test.go diff --git a/docs/decisions/0009-streaming-boundaries.md b/docs/decisions/0009-streaming-boundaries.md index 42afc96..60a933d 100644 --- a/docs/decisions/0009-streaming-boundaries.md +++ b/docs/decisions/0009-streaming-boundaries.md @@ -2,6 +2,13 @@ Status: accepted by the user on 2026-09-14; implementation in progress. +The [2026-09-17 writer review](0022-streaming-writer-review.md) applies this +accepted contract to the completed shared foundations. Two additional policies +await review there: inferred streaming lengths and empty Content-Encoding +sniffing. Its fresh local SDK probe also observes response-connection reuse, +extending the existing runtime compatibility question below. Decision 0010 +already resolved the conservative metadata-prefix implementation limit. + ## Recommended public boundary Keep a distinct adapter in the root edge package. Accepted signatures: diff --git a/docs/decisions/0022-streaming-writer-review.md b/docs/decisions/0022-streaming-writer-review.md new file mode 100644 index 0000000..dca96dd --- /dev/null +++ b/docs/decisions/0022-streaming-writer-review.md @@ -0,0 +1,224 @@ +# 0022: Streaming writer completion and SDK compatibility review + +Status: design review completed on 2026-09-17. S1 and S2 are proposed refinements +awaiting user approval. S3 records fresh evidence for the existing release gate. +No production streaming writer or public streaming entry point was added. + +## Current boundaries and official evidence + +Decisions 0009/0010 already approve the separate adapter, raw Handle and typed +HandleV1, one producer, bounded sniffing, multivalue JSON v2 metadata, handoff, +cleanup, Flush/FlushError and terminal-error reporting. Decision 0013 supplies +sanitized error categories; decision 0021 now supplies application authorization. + +Implemented foundations are `stream.go`, `stream_encoding.go`, +`response_headers.go`, `adapter_invoke.go` and `invocation.go`. They provide the +bridge, framing, header ownership, request conversion, identity and cleanup. +The remaining work is the HTTP writer and approved public wiring. No new event +types, authentication APIs or framework-owned dispatcher are needed. + +Official documentation was checked through `aws-proxy-public` on 2026-09-17: + +- [API Gateway streaming](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode.html) + remains REST-only for HTTP_PROXY/AWS_PROXY integrations, with transfer mode + STREAM. Request streaming is unsupported. Gateway does not perform content + encoding for streams; compression belongs in the integration. A disconnected + client or timed-out connection need not stop Lambda execution. +- [Lambda proxy streaming format](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html) + requires InvokeWithResponseStream, the streaming integration URI, JSON metadata + and eight NUL bytes followed by raw payload. Multivalue headers can carry every + extra field, including cookies. If Content-Length and Transfer-Encoding are + both absent, Gateway adds chunked transfer encoding. +- The same guide still leaves decimal/binary KB and delimiter start/end + accounting unspecified. The approved 16,000-byte complete-prefix bound remains + edge's conservative policy; there is no reason to reopen or relax it. +- [Lambda custom runtimes](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html) + still require the streaming-mode and chunked-encoding headers and closing the + underlying response connection. Midstream errors use declared runtime trailers. + +Both the Go module proxy and AWS's GitHub latest-release endpoint still report +[aws-lambda-go v1.55.0](https://github.com/aws/aws-lambda-go/releases/tag/v1.55.0), +published 2026-08-27, as latest stable. It is already required here; no dependency +update is needed. The downloaded SDK's runtime client and invoke loop were read. + +Go rules, cached official Go 1.27 notes, and installed Go 1.27.1 godoc/source were +reviewed. [ResponseWriter](https://pkg.go.dev/net/http#ResponseWriter) permits +logical commitment before physical transmission and documents sniffing, header +suppression and buffered length inference. +[ResponseController.Flush](https://pkg.go.dev/net/http#ResponseController.Flush) +prefers FlushError over Flusher and can follow wrapper Unwrap methods. Source +comparisons used Go 1.27.1's GOROOT as reported by go env. + +## State graph: applying existing approvals + +```mermaid +stateDiagram-v2 + [*] --> Uncommitted + Uncommitted --> Committed: first final WriteHeader / Write / Flush + Committed --> Committed: retain up to 512 bytes when sniffing is needed + Committed --> Published: valid prefix + sniff filled / Flush / completion + Committed --> Published: first nonempty Write when sniffing is unnecessary + Uncommitted --> FailedBeforeHandoff: validation failure + Committed --> FailedBeforeHandoff: writer or finalization failure + Published --> Published: write raw bytes through the pipe + Published --> Terminal: handler completion + cleanup + reporting + Published --> Terminal: fault / cancellation / Close + join + FailedBeforeHandoff --> [*] + Terminal --> [*] +``` + +Committed means the application's status/header snapshot cannot change. +Published means the bridge acknowledged reader ownership, not that the SDK or +remote client consumed a byte. This handoff determines the failure path. + +The following apply the accepted design rather than reopen it: + +1. WriteHeader freezes the response but need not publish. Empty Write commits + 200 if needed and accepts zero bytes without requiring publication or sniffing. + Later ordinary header changes are ignored; trailer attempts remain faults. +2. Retain at most 512 bytes only when sniffing is needed. Publish when that prefix + fills, explicitly flushes or the handler finishes. No timer, heartbeat or body + queue. With a supplied/suppressed type or actual encoding, the first nonempty + Write can publish immediately. +3. Flush commits 200 if needed, finalizes sniffing from pending bytes and drains + them after publication. Empty Flush publishes without inventing a type; later + writes cannot revise it. Repeated Flush without pending bytes only checks + cancellation and recorded failure. It cannot promise client receipt or chunk + boundaries. Compression wrappers must flush their encoder first. +4. FlushError returns failures. Flush uses the same implementation and retains + failures for finalization. The first writer fault remains sticky if ignored. + ResponseController-compatible wrappers preserve FlushError or Unwrap support. +5. One handler goroutine owns the writer. Header/Write/Flush are not promised + concurrent safety; Read/Close synchronization remains the bridge's concern. +6. Validate a complete Write against declared length before copying/forwarding. + Reject an overrun without forwarding its excess; retain accurate counts for + partial pipe writes. Short non-HEAD bodies fail at completion, directly before + handoff or terminally afterward. Reaching declared length does not mean the + handler has completed or replace Flush as an explicit publication request. +7. HEAD accepts/counts representation writes but emits no body. Honor a supplied + representation length without requiring an omitted HEAD body to fill it. + Preserve the accepted 204/205/304 suppression and nonfatal ErrBodyNotAllowed + behavior, including the documented 205 difference from net/http. +8. Before publication, faults return nil reader plus the classified error. After + publication, no replacement status/body is possible; cleanup precedes terminal + error/reporting. Existing operation/cleanup faults precede concurrent transport + cancellation. This differs intentionally from authz's callback error boundary. +9. Reuse existing constructor/options rules and the approved streaming reporter. + It receives the invocation context, not an inferred request principal. Factor + request preparation without changing buffered semantics or original-body and + multipart ownership. Default-off gateway identity and inherited-caller + rejection apply to streaming too. +10. Raw Handle rejects explicitly versioned HTTP API envelopes. Typed V1 cannot + prove REST deployment configuration. Preserve that qualification and the SDK + capture-versus-edge JSON v2 distinction in registration examples. + +## S1. Proposed: never infer streaming Content-Length + +Earlier approvals prohibit whole-body buffering and inference after handoff, +but do not settle whether a tiny response completing before handoff should gain +an inferred length. Buffered edge and native net/http can infer one. + +**Recommend:** preserve valid caller-supplied Content-Length when the method and +status permit it; otherwise omit it consistently, including small completed +responses, empty responses and HEAD. Respect explicit suppression. Do not generate +Transfer-Encoding metadata; Gateway owns its transport framing. + +**Reasoning:** AWS explicitly supports omitted length. Inferring it only before +publication makes output depend on sniffing, Content-Type presence and Flush +placement. One rule keeps small and large responses on a predictable streaming +contract without extra buffering or configuration. + +**Alternative:** infer from the small prefix on successful completion before +handoff, closer to net/http's optimization. This saves chunk framing in some +cases but creates timing-dependent headers and an additional HEAD rule. + +**Consequence:** an unflushed tiny stream may omit a length that buffered edge or +native HTTP would infer. Applications needing a representation length supply +one explicitly. Buffered behavior is unchanged. This is edge policy, not an AWS +prohibition on inferred lengths. + +## S2. Proposed: share a nonblank Content-Encoding sniffing test + +`bufferedWriter.finish` currently suppresses sniffing whenever Content-Encoding +has any values, including a single empty string. Go 1.27.1 checks for a nonblank +encoding. The local HTTP probe confirms an empty Content-Encoding permits HTML +content detection, while an actual encoding suppresses it. + +**Recommend:** use a small shared private predicate in both writers. Suppress +sniffing if any committed, trimmed Content-Encoding value is nonempty. Nil/empty +slices and all-empty values do not suppress it. Preserve the field through +existing snapshot rules, without introducing an encoding grammar parser. +Content-Type's explicit nil/empty-slice suppression remains unchanged. + +**Reasoning:** an empty encoding does not say the bytes are encoded. Conversely, +a later nonempty repeated value must still suppress sniffing even if the first +value is blank; looking only at Header.Get misses that defensive multivalue case. +One helper prevents drift between the two writers. + +**Alternative:** retain the any-value check in both writers. It preserves the +current edge quirk but loses intended Go compatibility for empty encoding fields. +A universal encoding parser would be unnecessary additional scope. + +**Consequence:** buffered responses with absent Content-Type and explicitly empty +Content-Encoding may now gain a detected type. Actual encodings, explicit type +suppression and header representation remain unchanged. This compatibility +correction needs approval before changing production behavior. + +## S3. SDK issue: retain the existing deployment release gate + +All five existing local Runtime API cases were rerun with race detection against +v1.55.0. They still show incremental raw/typed delivery, reader closure and late +error trailers. Edge's byte-plus-error normalization remains necessary to preserve +data that the SDK's errorCapturingReader otherwise discards. + +The extended probe also observes: + +- Lambda-Runtime-Function-Response-Mode is absent. +- The response request does not ask to close its connection. +- All five cases reused the response TCP connection for the subsequent /next + request. Closing the returned io.ReadCloser is distinct from closing that + Runtime API connection. + +Both observations differ from the current custom-runtime guide. The mock accepts +them to observe SDK behavior. This proves neither rejection nor successful +streaming by the real Lambda/API Gateway services. + +[Upstream issue 500](https://github.com/aws/aws-lambda-go/issues/500) raised the +mode-header question. Collaborator comments support the reader/lambdaurl design +but do not answer it. Closing that issue does not resolve the mismatch or verify +today's REST API Gateway behavior. + +**Recommendation remains SDK reuse.** Continue the locally testable implementation +after S1/S2 approval, retaining the deployment gate for both header and connection +handling. Do not patch the module cache, change global HTTP transport, fork the SDK +or add a custom runtime as an implicit workaround. + +Resolution still needs authoritative AWS clarification, a suitable stable SDK +change or a separately approved deployed integration test. Such a test should +check client-observed incremental delivery and truncation/error behavior with the +exact SDK and REST integration configuration. This review used no account APIs, +deployments or upstream posts. Deployment support remains unverified. + +## Implementation sequence and validation + +1. Record S1/S2 resolution. If S2 is accepted, add the shared predicate and focused + buffered regressions before implementing streaming sniffing. +2. Implement the private writer over the existing bridge, testing split sniffing, + suppression, empty writes/flushes, publication, lengths, bodyless responses, + sticky faults, late trailers and panics. +3. Factor shared preparation and wire the approved constructor, reporter and + raw/typed REST entry points. +4. Exercise the actual edge writer through the SDK subprocess: first delivery + before producer completion, backpressure, cleanup/Close and invocation reuse. +5. Add SSE, direct JSON v2 NDJSON, binary and gzip examples, slow readers/early + closure and authn/authz denials/outages before commitment. Heartbeats and + application error records remain consumer-owned. +6. Run race tests, vet, formatting and both Lambda builds. Report local completion + separately from deployment compatibility. + +This review changes only documentation and local observation probes. The Go +1.27.1 HTTP reference probe passes for nil/empty/actual Content-Encoding, +Content-Type suppression and empty Flush. The latter publishes without a type +or inferred length. Both reference and SDK probes pass with race detection. +Public streaming implementation and the proposed policy corrections remain +pending; existing accepted contracts stay in force. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 3f5b301..7d0b6d8 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -30,3 +30,4 @@ Keep related decisions small enough to review and implement independently. | [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | | [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | +| [0022: Streaming writer review](0022-streaming-writer-review.md) | S1/S2 proposed; SDK release gate reconfirmed | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | diff --git a/docs/plan.md b/docs/plan.md index eb83a0e..ea12459 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -162,6 +162,16 @@ failures. Full race tests (including the existing SDK probe), vet, formatting, and Linux arm64/amd64 builds pass on Go 1.27.1. HTTP streaming commitment, sniffing, framing validation, and public entry points remain to be implemented. +The [2026-09-17 streaming writer review](decisions/0022-streaming-writer-review.md) +confirms the approved state/ownership graph against completed authn/authz and the +current AWS/Go documentation. S1 proposes no automatic streaming Content-Length; +S2 proposes shared sniffing behavior for empty Content-Encoding. These await +user approval. SDK v1.55.0 remains latest stable. Local probes reconfirm incremental +delivery and the missing streaming-mode header, and additionally observe Runtime +API connection reuse, another difference from the custom-runtime guide. Retain +the existing deployment release gate for both observations; no live AWS test has +run. The streaming writer and public entry points remain pending. + Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary errors, cleans up on failures/panics, and checks parent cancellation. The HTTP diff --git a/internal/streamprobe/http_test.go b/internal/streamprobe/http_test.go new file mode 100644 index 0000000..8eb833d --- /dev/null +++ b/internal/streamprobe/http_test.go @@ -0,0 +1,62 @@ +package streamprobe + +import ( + "io" + "net/http" + "net/http/httptest" + "testing" +) + +// Record the Go 1.27 reference behavior separately from the proposed edge writer. +func TestHTTPStreamingHeaderReference(t *testing.T) { + for _, tc := range []struct { + name string + encoding []string + suppressType bool + flushFirst bool + wantType string + }{ + {name: "nil_encoding", wantType: "text/html; charset=utf-8"}, + {name: "empty_encoding", encoding: []string{""}, wantType: "text/html; charset=utf-8"}, + {name: "encoded", encoding: []string{"gzip"}}, + {name: "suppressed_type", suppressType: true}, + {name: "empty_flush", flushFirst: true}, + } { + t.Run(tc.name, func(t *testing.T) { + server := httptest.NewTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header()["Content-Encoding"] = tc.encoding + if tc.suppressType { + w.Header()["Content-Type"] = nil + } + if tc.flushFirst { + if err := http.NewResponseController(w).Flush(); err != nil { + t.Error(err) + return + } + } + if _, err := io.WriteString(w, "example"); err != nil { + t.Error(err) + } + })) + r, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://example.com/", nil) + if err != nil { + t.Fatal(err) + } + // Only header behavior is under test; disable automatic decompression + // for the synthetic encoding fixture. + r.Header.Set("Accept-Encoding", "identity") + response, err := server.Client().Do(r) + if err != nil { + t.Fatal(err) + } + defer response.Body.Close() + if _, err := io.Copy(io.Discard, response.Body); err != nil { + t.Fatal(err) + } + if got := response.Header.Get("Content-Type"); got != tc.wantType { + t.Errorf("Content-Type=%q; want %q", got, tc.wantType) + } + t.Logf("content-type=%q content-length=%q transfer-encoding=%v", response.Header.Get("Content-Type"), response.Header.Get("Content-Length"), response.TransferEncoding) + }) + } +} diff --git a/internal/streamprobe/runtime_test.go b/internal/streamprobe/runtime_test.go index 7bc7198..501ca86 100644 --- a/internal/streamprobe/runtime_test.go +++ b/internal/streamprobe/runtime_test.go @@ -150,14 +150,17 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { closed := make(chan struct{}, 1) next := make(chan struct{}, 1) type result struct { - body []byte - header http.Header - trailer http.Header - chunked bool - err error + body []byte + header http.Header + trailer http.Header + chunked bool + closeConnection bool + connection string + err error } results := make(chan result, 1) var invokes atomic.Int32 + var nextConnection atomic.Value server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { case "/continue": @@ -171,6 +174,7 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { w.WriteHeader(http.StatusNoContent) case "/2018-06-01/runtime/invocation/next": if invokes.Add(1) > 1 { + nextConnection.Store(r.RemoteAddr) next <- struct{}{} <-r.Context().Done() return @@ -191,7 +195,7 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { } firstRead <- struct{}{} rest, err := io.ReadAll(r.Body) - results <- result{body: append(one, rest...), header: r.Header.Clone(), trailer: r.Trailer.Clone(), chunked: slices.Contains(r.TransferEncoding, "chunked"), err: err} + results <- result{body: append(one, rest...), header: r.Header.Clone(), trailer: r.Trailer.Clone(), chunked: slices.Contains(r.TransferEncoding, "chunked"), closeConnection: r.Close, connection: r.RemoteAddr, err: err} w.WriteHeader(http.StatusAccepted) default: data, err := io.ReadAll(r.Body) @@ -267,6 +271,9 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { t.Fatal("runtime did not close stream before requesting next invocation") } } + // Observe compatibility details without asserting that a future SDK + // must retain today's omissions or connection reuse behavior. + t.Logf("runtime-request-close=%v response-connection-reused-for-next=%v", got.closeConnection, nextConnection.Load() == got.connection) }) } } From 7f481e96a8d86c45604359445e20f5654975399f Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Thu, 17 Sep 2026 08:27:37 +0100 Subject: [PATCH 36/44] feat: implement REST response streaming adapter --- README.md | 24 +- adapter.go | 35 +- adapter_invoke.go | 18 +- authn/authorization_test.go | 2 +- authn/gateway_test.go | 51 ++- authz/transport_test.go | 26 +- buffered.go | 18 +- docs/authz.md | 7 +- docs/decisions/0009-streaming-boundaries.md | 17 +- docs/decisions/0010-response-header-design.md | 16 +- .../decisions/0022-streaming-writer-review.md | 55 ++- docs/decisions/README.md | 2 +- docs/plan.md | 41 +- docs/streaming.md | 160 ++++++++ internal/streamprobe/runtime_test.go | 118 +++++- response_headers.go | 24 ++ streaming_adapter.go | 119 ++++++ streaming_adapter_test.go | 384 ++++++++++++++++++ streaming_example_test.go | 165 ++++++++ streaming_writer.go | 189 +++++++++ streaming_writer_test.go | 294 ++++++++++++++ 21 files changed, 1664 insertions(+), 101 deletions(-) create mode 100644 docs/streaming.md create mode 100644 streaming_adapter.go create mode 100644 streaming_adapter_test.go create mode 100644 streaming_example_test.go create mode 100644 streaming_writer.go create mode 100644 streaming_writer_test.go diff --git a/README.md b/README.md index 3a640aa..e79d365 100644 --- a/README.md +++ b/README.md @@ -52,12 +52,16 @@ resource selection and HTTP responses; see the [authorization guide](docs/authz. and [runnable dispatcher examples](authz/example_test.go). No live Cognito/STS interoperability test has run. -The accepted streaming design now has a private bridge with incremental delivery, -backpressure, cancellation, cleanup, and terminal-error handling. Its lifecycle -is covered by race-enabled synthetic concurrency tests. Bounded JSON v2 metadata -prefix encoding is implemented. HTTP streaming commitment and public streaming -entry points remain under construction; deployed API Gateway -streaming has not been verified. +REST response streaming is implemented through `edge.NewStreaming(handler)` and +`lambda.Start(adapter.Handle)` or `lambda.Start(adapter.HandleV1)`. It provides +bounded sniffing, Flush/FlushError, raw body delivery, backpressure, cancellation, +cleanup and sanitized terminal errors. Streaming never infers Content-Length. +See the [streaming guide](docs/streaming.md) and +[SSE, NDJSON, binary and gzip examples](streaming_example_test.go). +Local SDK tests pass, but deployment support remains unverified: SDK v1.55.0's +Runtime API mode header and connection handling differ from current AWS guidance. +The [deployment qualification](docs/decisions/0022-streaming-writer-review.md#s3-sdk-issue-retain-the-existing-deployment-release-gate) +remains open; no live API Gateway streaming test has run. ## Header-selected actions @@ -86,9 +90,10 @@ value costs `len(name) + len(value) + 32`; nil/empty slices still cost a name pl 32, and generated headers consume remaining budget. This is separate from the complete buffered-envelope limit and the 16,000-byte streaming metadata prefix (including its delimiter). It is neither an AWS quota nor a total heap bound. -The buffered writer applies the allowance at commitment. Raw invocation also +Both writers apply the allowance at commitment. Raw buffered invocation also checks the complete serialized response against 6 MiB; typed callers own their -final envelope serialization and size. The streaming HTTP writer remains pending. +final envelope serialization and size. Streaming checks the complete metadata +prefix before handing off the reader; its body does not use the buffered limit. ## Owned claims @@ -125,7 +130,8 @@ allowance to gateway claims and dedicated scopes. Handlers receive an owned request body and canceled context after invocation. Multipart files parsed on the served request are removed on success, error or panic. Middleware that parses multipart on a separate request copy owns cleanup -of files attached only to that copy. Handler panics propagate after cleanup. +of files attached only to that copy. Buffered and pre-handoff streaming panics +propagate after cleanup; post-handoff streaming panics become terminal errors. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) diff --git a/adapter.go b/adapter.go index ea3d73f..86716c2 100644 --- a/adapter.go +++ b/adapter.go @@ -1,9 +1,11 @@ // Package edge adapts AWS API Gateway Lambda proxy events to net/http handlers. // It supports raw JSON v2 and already typed AWS events with optional native -// gateway identity. Buffered responses use one shared HTTP translation policy. +// gateway identity. Buffered and REST streaming responses share request and +// identity policies; each has its own response and invocation boundary. package edge import ( + "context" "errors" "fmt" "net/http" @@ -23,9 +25,11 @@ type config struct { gatewayIdentity bool identityClaimsBudget int responseHeaderBudget int + streamReporter func(context.Context, error) + streamReporterSet bool } -// Option configures an [Adapter] during construction. +// Option configures an [Adapter] or [StreamingAdapter] during construction. // Use the option functions provided by this package. A nil Option is invalid. // Options apply in argument order; the last assignment to a scalar setting wins. type Option func(*config) @@ -42,7 +46,7 @@ func WithGatewayIdentity(enabled bool) Option { } // WithIdentityClaimsBudget sets the gateway identity claims allowance in bytes. -// The default is 256 KiB; New rejects a final setting outside 1..6 MiB. +// The default is 256 KiB; New and NewStreaming reject settings outside 1..6 MiB. // Each claim value costs 64 bytes plus name, string and exact-number text bytes. // Dedicated gateway scopes consume the same allowance. Raw claim JSON is also // limited to this byte length. This is a resource policy, not a heap cap or AWS @@ -54,7 +58,7 @@ func WithIdentityClaimsBudget(bytes int) Option { } // WithResponseHeaderBudget sets the response header resource budget in bytes. -// The default is 256 KiB. New rejects a final configured value outside 1..6 MiB; +// The default is 256 KiB. Both constructors reject settings outside 1..6 MiB; // zero does not disable the bound. Each original value costs its name length, // value length and 32 bytes; nil/empty slices cost their name length and 32. // Generated headers also consume the budget. This is a library resource policy, @@ -73,8 +77,19 @@ func WithResponseHeaderBudget(bytes int) Option { // The caller retains ownership of handler and is responsible for its concurrency // safety when sharing it. New does not use [http.DefaultServeMux]. func New(handler http.Handler, opts ...Option) (*Adapter, error) { + cfg, err := configureAdapter(handler, opts) + if err != nil { + return nil, err + } + if cfg.streamReporterSet { + return nil, errors.New("edge: stream error reporter requires NewStreaming") + } + return &Adapter{handler: handler, config: cfg}, nil +} + +func configureAdapter(handler http.Handler, opts []Option) (config, error) { if handler == nil { - return nil, errors.New("edge: nil HTTP handler") + return config{}, errors.New("edge: nil HTTP handler") } // An interface containing a typed nil is non-nil. Check nil-capable kinds // before accepting a handler that would defer a startup fault to invocation. @@ -82,22 +97,22 @@ func New(handler http.Handler, opts ...Option) (*Adapter, error) { switch v.Kind() { case reflect.Chan, reflect.Func, reflect.Map, reflect.Pointer, reflect.Slice: if v.IsNil() { - return nil, errors.New("edge: nil HTTP handler") + return config{}, errors.New("edge: nil HTTP handler") } } cfg := config{identityClaimsBudget: 256 * 1024, responseHeaderBudget: defaultResponseHeaderBudget} for i, opt := range opts { if opt == nil { - return nil, fmt.Errorf("edge: nil option at index %d", i) + return config{}, fmt.Errorf("edge: nil option at index %d", i) } opt(&cfg) } if cfg.responseHeaderBudget <= 0 || cfg.responseHeaderBudget > maxResponseBytes { - return nil, errors.New("edge: response header budget must be between 1 and 6291456 bytes") + return config{}, errors.New("edge: response header budget must be between 1 and 6291456 bytes") } if cfg.identityClaimsBudget <= 0 || cfg.identityClaimsBudget > maxResponseBytes { - return nil, errors.New("edge: identity claims budget must be between 1 and 6291456 bytes") + return config{}, errors.New("edge: identity claims budget must be between 1 and 6291456 bytes") } - return &Adapter{handler: handler, config: cfg}, nil + return cfg, nil } diff --git a/adapter_invoke.go b/adapter_invoke.go index f4e6540..0b768d3 100644 --- a/adapter_invoke.go +++ b/adapter_invoke.go @@ -100,6 +100,16 @@ func (a *Adapter) valid() error { } func (a *Adapter) buffered(ctx context.Context, inv *invocation, event decodedEvent, raw bool) (*bufferedResponse, error) { + request, err := prepareRequest(ctx, inv, event, raw, a.config) + if err != nil { + return nil, err + } + return serveBuffered(a.handler, request, a.config.responseHeaderBudget) +} + +// prepareRequest owns the transport body before identity production can fail. +// Both invocation modes serve the same prepared request and cleanup owner. +func prepareRequest(ctx context.Context, inv *invocation, event decodedEvent, raw bool, cfg config) (*http.Request, error) { var request *http.Request var err error if event.v1 != nil { @@ -111,12 +121,12 @@ func (a *Adapter) buffered(ctx context.Context, inv *invocation, event decodedEv return nil, err } inv.ownRequest(request) - if a.config.gatewayIdentity { + if cfg.gatewayIdentity { var caller identity.Caller if raw { - caller, err = rawGatewayCaller(event, a.config.identityClaimsBudget) + caller, err = rawGatewayCaller(event, cfg.identityClaimsBudget) } else { - caller, err = typedGatewayCaller(event, a.config.identityClaimsBudget) + caller, err = typedGatewayCaller(event, cfg.identityClaimsBudget) } if err != nil { return nil, err @@ -130,5 +140,5 @@ func (a *Adapter) buffered(ctx context.Context, inv *invocation, event decodedEv // original transport body separately even if the handler replaces it. inv.request = request } - return serveBuffered(a.handler, request, a.config.responseHeaderBudget) + return request, nil } diff --git a/authn/authorization_test.go b/authn/authorization_test.go index 3435033..fa6c9db 100644 --- a/authn/authorization_test.go +++ b/authn/authorization_test.go @@ -78,7 +78,7 @@ func TestAuthorizationAcrossTransports(t *testing.T) { if tc.scheme == "EdgeIAM" { token = proof.Value() } - for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2", "stream_raw", "stream_typed"} { t.Run(format, func(t *testing.T) { var lookups, executions atomic.Int64 grant, err := authz.Check(func(ctx context.Context, req authz.Request) (bool, error) { diff --git a/authn/gateway_test.go b/authn/gateway_test.go index a4be7a1..3d56051 100644 --- a/authn/gateway_test.go +++ b/authn/gateway_test.go @@ -1,6 +1,7 @@ package authn_test import ( + "bytes" "context" "encoding/base64" "encoding/json/v2" @@ -134,7 +135,7 @@ func TestAuthenticationAcrossTransports(t *testing.T) { }))) a := authenticator(t, authn.Config{Bearer: verifier.Verify, IAMProof: mapProofErrors(proofVerifier)}) h := handler(t, a, dispatcher) - for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2", "stream_raw", "stream_typed"} { t.Run(format, func(t *testing.T) { action := tc.action if action == "" { @@ -152,8 +153,8 @@ func TestAuthenticationAcrossTransports(t *testing.T) { } }) } - if requests.Load() != 12 { - t.Fatalf("STS requests=%d; want 12 for the two valid-proof cases across six transports", requests.Load()) + if requests.Load() != 16 { + t.Fatalf("STS requests=%d; want 16 for the two valid-proof cases across eight transports", requests.Load()) } } @@ -183,10 +184,6 @@ func serveFormatBody(t *testing.T, h http.Handler, format string, values []strin } return resp.StatusCode, resp.Header, body } - adapter, err := edge.New(h) - if err != nil { - t.Fatal(err) - } v1 := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}, Headers: map[string]string{"action": action, "custom-trace": "keep"}} v2 := events.APIGatewayV2HTTPRequest{Version: "2.0", RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}, @@ -196,6 +193,46 @@ func serveFormatBody(t *testing.T, h http.Handler, format string, values []strin v1.MultiValueHeaders = map[string][]string{"Authorization": values} v2.Headers["authorization"] = strings.Join(values, ",") } + if format == "stream_raw" || format == "stream_typed" { + adapter, err := edge.NewStreaming(h) + if err != nil { + t.Fatal(err) + } + var stream io.ReadCloser + if format == "stream_raw" { + wire, marshalErr := json.Marshal(v1) + if marshalErr != nil { + t.Fatal(marshalErr) + } + stream, err = adapter.Handle(t.Context(), wire) + } else { + stream, err = adapter.HandleV1(t.Context(), v1) + } + if err != nil { + t.Fatal(err) + } + defer stream.Close() + wire, err := io.ReadAll(stream) + if err != nil { + t.Fatal(err) + } + prefix, body, ok := bytes.Cut(wire, make([]byte, 8)) + if !ok { + t.Fatal("missing stream delimiter") + } + var metadata struct { + Status int `json:"statusCode"` + Headers http.Header `json:"multiValueHeaders"` + } + if err := json.Unmarshal(prefix, &metadata); err != nil { + t.Fatal(err) + } + return metadata.Status, metadata.Headers, body + } + adapter, err := edge.New(h) + if err != nil { + t.Fatal(err) + } var resp1 events.APIGatewayProxyResponse var resp2 events.APIGatewayV2HTTPResponse switch format { diff --git a/authz/transport_test.go b/authz/transport_test.go index d2a029c..ab2b9a9 100644 --- a/authz/transport_test.go +++ b/authz/transport_test.go @@ -15,7 +15,7 @@ import ( ) func TestAuthorizationCancellationAcrossTransports(t *testing.T) { - for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2"} { + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2", "stream_raw", "stream_typed"} { t.Run(format, func(t *testing.T) { ctx, cancel := context.WithCancelCause(t.Context()) defer cancel(nil) @@ -51,6 +51,30 @@ func TestAuthorizationCancellationAcrossTransports(t *testing.T) { v1 := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}} v2 := events.APIGatewayV2HTTPRequest{Version: "2.0", RawPath: "/", RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "example", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}} switch format { + case "stream_raw", "stream_typed": + streaming, configErr := edge.NewStreaming(h) + if configErr != nil { + t.Fatal(configErr) + } + if format == "stream_raw" { + wire, marshalErr := json.Marshal(v1) + if marshalErr != nil { + t.Fatal(marshalErr) + } + stream, invokeErr := streaming.Handle(ctx, wire) + err = invokeErr + if stream != nil { + _ = stream.Close() + t.Error("canceled authorization returned stream") + } + } else { + stream, invokeErr := streaming.HandleV1(ctx, v1) + err = invokeErr + if stream != nil { + _ = stream.Close() + t.Error("canceled authorization returned stream") + } + } case "typed_v1": response, invokeErr := adapter.HandleV1(ctx, v1) err = invokeErr diff --git a/buffered.go b/buffered.go index 5fc534c..95208f2 100644 --- a/buffered.go +++ b/buffered.go @@ -4,7 +4,6 @@ import ( "math" "net/http" "strconv" - "strings" "github.com/aws/aws-lambda-go/events" ) @@ -56,7 +55,7 @@ func (w *bufferedWriter) WriteHeader(status int) { if w.err != nil { return } - if !bufferedBodyAllowed(status) { + if !responseBodyAllowed(status) { w.headers.remove("Content-Length") if status == http.StatusNotModified { w.headers.remove("Content-Type") @@ -75,7 +74,7 @@ func (w *bufferedWriter) Write(p []byte) (int, error) { if len(p) == 0 { return 0, nil } - if !bufferedBodyAllowed(w.status) { + if !responseBodyAllowed(w.status) { return 0, http.ErrBodyNotAllowed } if w.hasLength && int64(len(p)) > w.declaredLength-w.written { @@ -98,7 +97,7 @@ func (w *bufferedWriter) Write(p []byte) (int, error) { return len(p), nil } -func bufferedBodyAllowed(status int) bool { +func responseBodyAllowed(status int) bool { return status != http.StatusNoContent && status != http.StatusResetContent && status != http.StatusNotModified } @@ -113,17 +112,14 @@ func (w *bufferedWriter) finish() (*bufferedResponse, error) { if w.err != nil { return nil, w.err } - for name, values := range w.header { - declared := len(name) == len("Trailer") && strings.EqualFold(name, "Trailer") && len(values) != 0 - if strings.HasPrefix(name, http.TrailerPrefix) || declared { - return nil, errResponseTrailers - } + if err := responseTrailers(w.header); err != nil { + return nil, err } if w.method != http.MethodHead && w.hasLength && w.written != w.declaredLength { return nil, invocationError("response", ErrResponse, "response body length mismatch", http.ErrContentLength) } - if bufferedBodyAllowed(w.status) { - if len(w.body.data) > 0 && len(w.headers.fields["Content-Encoding"]) == 0 { + if responseBodyAllowed(w.status) { + if len(w.body.data) > 0 && w.headers.canSniffType() { kind := http.DetectContentType(w.body.data[:min(len(w.body.data), 512)]) if err := w.headers.automatic("Content-Type", kind); err != nil { return nil, err diff --git a/docs/authz.md b/docs/authz.md index bcf16f4..025c0c1 100644 --- a/docs/authz.md +++ b/docs/authz.md @@ -183,9 +183,10 @@ authorize and execute that same entry/target. Unknown actions have no unprotecte fallback. Do not reread mutable headers for policy or execution. Resource version and transaction consistency remain the application's responsibility. -The same handler works through ordinary net/http or `edge.New(handler)`. Complete -authorization before response commitment or streaming flush. This milestone does -not introduce public streaming entry points or ongoing stream reauthorization. +The same handler works through ordinary net/http, `edge.New(handler)` or +`edge.NewStreaming(handler)`. Complete authentication, target resolution and +authorization before response commitment or streaming flush. Ongoing stream +reauthorization remains application-owned; see the [streaming guide](streaming.md). See [decision 0021](decisions/0021-authorization-rules.md) for the approved design, alternatives and official AWS/IETF evidence. diff --git a/docs/decisions/0009-streaming-boundaries.md b/docs/decisions/0009-streaming-boundaries.md index 60a933d..4342f45 100644 --- a/docs/decisions/0009-streaming-boundaries.md +++ b/docs/decisions/0009-streaming-boundaries.md @@ -1,11 +1,12 @@ # 0009: Streaming entry points, ownership, and failure boundaries -Status: accepted by the user on 2026-09-14; implementation in progress. +Status: accepted by the user on 2026-09-14; implemented with local validation. +Deployed SDK/API Gateway compatibility remains unverified. The [2026-09-17 writer review](0022-streaming-writer-review.md) applies this -accepted contract to the completed shared foundations. Two additional policies -await review there: inferred streaming lengths and empty Content-Encoding -sniffing. Its fresh local SDK probe also observes response-connection reuse, +accepted contract to the completed shared foundations. Its refinements were +approved on 2026-09-17 and implemented: no inferred streaming lengths and shared +empty Content-Encoding sniffing. Its local SDK probe observes connection reuse, extending the existing runtime compatibility question below. Decision 0010 already resolved the conservative metadata-prefix implementation limit. @@ -249,6 +250,8 @@ reporter panic isolation, and bytes-plus-error normalization. Go 1.27 synctest bubbles also require their goroutines to exit. Full race tests, the existing SDK probe, vet, formatting, and Linux arm64/amd64 builds pass on Go 1.27.1. -This is the ownership/transport foundation. It does not yet implement the HTTP -streaming writer, prefix codec/limit, constructor/options, or public entry points, -and it does not resolve the documented AWS Runtime API header mismatch. +That initial milestone supplied the ownership/transport foundation. Decision +0010 subsequently implemented the prefix codec/limit, and decision 0022 completed +the HTTP writer, constructor/options and raw/typed public entry points on +2026-09-17. The SDK probe now exercises that production path. The documented +Runtime API header/connection mismatch remains an open deployment qualification. diff --git a/docs/decisions/0010-response-header-design.md b/docs/decisions/0010-response-header-design.md index 7b32c95..9f6a96f 100644 --- a/docs/decisions/0010-response-header-design.md +++ b/docs/decisions/0010-response-header-design.md @@ -2,8 +2,9 @@ Status: accepted by the user on 2026-09-15, including the measured 256 KiB default and explicit override up to 6 MiB. Shared snapshots, projections and -streaming-prefix encoding are implemented and connected to the private buffered -HTTP writer. Streaming HTTP writer wiring follows. +streaming-prefix encoding are implemented and connected to both HTTP writers. +Decision 0022 records the approved shared sniffing refinement and streaming +Content-Length policy. Terminology: V1/V2 in this record mean API Gateway payload formats 1.0/2.0, not versions of this Go module. The field-combination table concerns outgoing @@ -293,12 +294,11 @@ arm64/amd64 builds pass. The default's measured retained snapshot matches the probe at approximately 0.48 MiB for the high-cardinality fixture. Projection allocations are reported separately; no new resource policy is needed. -The buffered writer now uses these primitives for status/method rules, sniffing, +Both writers now use these primitives for status/method rules, sniffing, length enforcement, commitment timing and late unsupported-trailer detection. -The streaming HTTP writer remains to be implemented. Constructor options and -private primitives do -not make the unfinished adapter usable as a Lambda handler yet. +The streaming HTTP writer and public entry points were completed on 2026-09-17 +under decision 0022, including the shared empty-encoding sniffing refinement. -Public invocation/identity work and the SDK Runtime API header compatibility -question remain separate. No account APIs or live AWS deployments were used for +The SDK Runtime API header/connection compatibility question remains a separate +deployment qualification. No account APIs or live AWS deployments were used for this proposal. diff --git a/docs/decisions/0022-streaming-writer-review.md b/docs/decisions/0022-streaming-writer-review.md index dca96dd..1991a43 100644 --- a/docs/decisions/0022-streaming-writer-review.md +++ b/docs/decisions/0022-streaming-writer-review.md @@ -1,8 +1,8 @@ # 0022: Streaming writer completion and SDK compatibility review -Status: design review completed on 2026-09-17. S1 and S2 are proposed refinements -awaiting user approval. S3 records fresh evidence for the existing release gate. -No production streaming writer or public streaming entry point was added. +Status: S1 and S2 approved by the user on 2026-09-17; implemented and locally validated. +S3 retains the existing deployment release gate. The user authorized completing +the streaming writer and approved public entry points under these refinements. ## Current boundaries and official evidence @@ -14,8 +14,9 @@ sanitized error categories; decision 0021 now supplies application authorization Implemented foundations are `stream.go`, `stream_encoding.go`, `response_headers.go`, `adapter_invoke.go` and `invocation.go`. They provide the bridge, framing, header ownership, request conversion, identity and cleanup. -The remaining work is the HTTP writer and approved public wiring. No new event -types, authentication APIs or framework-owned dispatcher are needed. +The HTTP writer and approved public wiring are now implemented in +`streaming_writer.go` and `streaming_adapter.go`. No new event types, +authentication APIs or framework-owned dispatcher were needed. Official documentation was checked through `aws-proxy-public` on 2026-09-17: @@ -112,7 +113,7 @@ The following apply the accepted design rather than reopen it: prove REST deployment configuration. Preserve that qualification and the SDK capture-versus-edge JSON v2 distinction in registration examples. -## S1. Proposed: never infer streaming Content-Length +## S1. Accepted: never infer streaming Content-Length Earlier approvals prohibit whole-body buffering and inference after handoff, but do not settle whether a tiny response completing before handoff should gain @@ -137,9 +138,9 @@ native HTTP would infer. Applications needing a representation length supply one explicitly. Buffered behavior is unchanged. This is edge policy, not an AWS prohibition on inferred lengths. -## S2. Proposed: share a nonblank Content-Encoding sniffing test +## S2. Accepted: share a nonblank Content-Encoding sniffing test -`bufferedWriter.finish` currently suppresses sniffing whenever Content-Encoding +Before this refinement, `bufferedWriter.finish` suppressed sniffing when Content-Encoding has any values, including a single empty string. Go 1.27.1 checks for a nonblank encoding. The local HTTP probe confirms an empty Content-Encoding permits HTML content detection, while an actual encoding suppresses it. @@ -162,7 +163,7 @@ A universal encoding parser would be unnecessary additional scope. **Consequence:** buffered responses with absent Content-Type and explicitly empty Content-Encoding may now gain a detected type. Actual encodings, explicit type suppression and header representation remain unchanged. This compatibility -correction needs approval before changing production behavior. +correction was approved on 2026-09-17. ## S3. SDK issue: retain the existing deployment release gate @@ -201,7 +202,7 @@ deployments or upstream posts. Deployment support remains unverified. ## Implementation sequence and validation -1. Record S1/S2 resolution. If S2 is accepted, add the shared predicate and focused +1. Record S1/S2 resolution, then add the shared predicate and focused buffered regressions before implementing streaming sniffing. 2. Implement the private writer over the existing bridge, testing split sniffing, suppression, empty writes/flushes, publication, lengths, bodyless responses, @@ -216,9 +217,37 @@ deployments or upstream posts. Deployment support remains unverified. 6. Run race tests, vet, formatting and both Lambda builds. Report local completion separately from deployment compatibility. -This review changes only documentation and local observation probes. The Go +The initial review changed only documentation and local observation probes. The Go 1.27.1 HTTP reference probe passes for nil/empty/actual Content-Encoding, Content-Type suppression and empty Flush. The latter publishes without a type or inferred length. Both reference and SDK probes pass with race detection. -Public streaming implementation and the proposed policy corrections remain -pending; existing accepted contracts stay in force. +The user subsequently approved S1/S2 and authorized implementation on 2026-09-17. +Existing accepted contracts and the deployment qualification stay in force. + +## Implementation completed on 2026-09-17 + +The writer, constructor, raw/typed REST methods and terminal reporter are now +implemented. Shared preparation reuses buffered request validation, body ownership +and opt-in native identity. Both writers use the approved sniffing predicate; +streaming consistently omits inferred lengths. The writer retains at most 512 +sniff bytes and uses the existing bridge for handoff and backpressure. + +Contract tests cover split sniffing, header ownership, empty writes/flushes, +explicit/suppressed type, encoding, lengths, HEAD/bodyless statuses, sticky faults, +prefix/header bounds, partial writes, trailers, early/late panics, cancellation, +Close, reporter faults, multipart cleanup and concurrent identity isolation. +An 8 MiB body test proves the streaming path does not apply the buffered envelope +limit. Existing Cognito/IAM authentication, authorization and cancellation matrices +now include both streaming entry points. + +The SDK subprocess suite has nine cases, including the production raw/typed +adapter and its late length error/panic paths. The local Runtime API server reads +the first body segment before releasing the producer, testing delivery beyond +metadata. It checks reader closure before the next invocation. The mode-header +and connection-reuse observations remain; successful mock delivery does not +resolve S3. + +SSE, direct JSON v2 NDJSON, binary and gzip examples are runnable. Full module +race tests, vet, formatting, and Linux arm64/amd64 builds pass on Go 1.27.1. +See the [consumer guide](../streaming.md). No AWS deployment or account API was +used, and no SDK workaround was introduced. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 7d0b6d8..28f68de 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -30,4 +30,4 @@ Keep related decisions small enough to review and implement independently. | [0019: HTTP authentication selector](0019-http-authentication-selector.md) | Accepted; selector, middleware and examples implemented | What public selector API, verifier/error boundary, header policy and HTTP challenges should compose IAM proof and Bearer authentication? | | [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | -| [0022: Streaming writer review](0022-streaming-writer-review.md) | S1/S2 proposed; SDK release gate reconfirmed | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | +| [0022: Streaming writer review](0022-streaming-writer-review.md) | Accepted; writer, public entry points and local validation complete; SDK deployment gate retained | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | diff --git a/docs/plan.md b/docs/plan.md index ea12459..202768b 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -45,7 +45,8 @@ port, source copy, or compatibility-preserving update of Beakley. types, sharing translation and identity policy with raw invocation. - Support REST API proxy events, HTTP API payload 1.0, and HTTP API payload 2.0. API product and payload version are distinct concepts. -- Start with buffered responses. Streaming is a separate future capability. +- Provide separate buffered and REST streaming adapters over shared request and + identity foundations. Both are implemented; deployed streaming remains unverified. - Keep payload dispatch and the response writer private initially. - Preserve separate IAM/JWT caller representations and an anonymous zero value; enforce invariants rather than relying on comments about a union. @@ -140,7 +141,7 @@ allocation probe revised the resource policy to a 256 KiB default with an explicit positive override up to 6 MiB; the user approved it on 2026-09-15. The shared snapshot, suppression/accounting logic, V1/V2 projections, option validation and bounded streaming metadata prefix are implemented. The buffered -writer now uses the shared header layer; streaming HTTP writer wiring remains. +and streaming writers now use the shared header layer. Streaming is specified in [decision 0008](decisions/0008-streaming-architecture.md): explicit REST response streaming alongside the buffered adapter, sharing HTTP/identity foundations. @@ -149,8 +150,8 @@ accepts concrete APIs and an ownership/state graph. The private bridge and share invocation ownership primitives are implemented. A local SDK Runtime API probe demonstrates incremental delivery, raw/typed capture, closure, and error trailers; it also exposes a bytes-plus-error reader issue and a missing header -relative to AWS's documented streaming contract. No production streaming API is -implemented, and deployed API Gateway behavior is not verified. +relative to AWS's documented streaming contract. The production streaming API +is now implemented; deployed API Gateway behavior is not verified. The private bridge publishes an owned prefix through an explicit handoff before body writes can block, uses an unbuffered pipe, and transfers cleanup to one @@ -160,17 +161,27 @@ panics become sanitized terminal errors, and reader results defer errors that accompany bytes. Synthetic concurrency tests cover these boundaries and reporter failures. Full race tests (including the existing SDK probe), vet, formatting, and Linux arm64/amd64 builds pass on Go 1.27.1. HTTP streaming commitment, -sniffing, framing validation, and public entry points remain to be implemented. +sniffing, framing validation, and public entry points are now implemented under +the subsequent writer review. The [2026-09-17 streaming writer review](decisions/0022-streaming-writer-review.md) confirms the approved state/ownership graph against completed authn/authz and the -current AWS/Go documentation. S1 proposes no automatic streaming Content-Length; -S2 proposes shared sniffing behavior for empty Content-Encoding. These await -user approval. SDK v1.55.0 remains latest stable. Local probes reconfirm incremental +current AWS/Go documentation. The user approved S1 (no automatic streaming +Content-Length) and S2 (shared sniffing behavior for empty Content-Encoding) on +2026-09-17 and authorized implementation. SDK v1.55.0 remains latest stable. +Local probes reconfirm incremental delivery and the missing streaming-mode header, and additionally observe Runtime API connection reuse, another difference from the custom-runtime guide. Retain the existing deployment release gate for both observations; no live AWS test has -run. The streaming writer and public entry points remain pending. +run. The streaming writer, NewStreaming, Handle, HandleV1 and +WithStreamErrorReporter are implemented. Shared request preparation preserves +buffered behavior, identity policy and request cleanup. The SDK probe now drives +the production adapter in raw/typed and late-error/panic cases, observes body +delivery before producer completion and checks reader closure before reuse. +Runnable SSE, direct JSON v2 NDJSON, binary and gzip examples accompany the +[consumer guide](streaming.md). Authentication/authorization integration tests +cover both streaming entry points. Full module race tests, vet, formatting and +Linux arm64/amd64 builds pass locally on Go 1.27.1. Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary @@ -179,8 +190,8 @@ writer now uses the shared header implementation from [decision 0010](decisions/0010-response-header-design.md), enforces commitment and body/length/status rules, and returns typed AWS response projections without JSON work. Tests compare ordinary behavior with net/http and exercise request -conversion, mux/range/gzip handling and invocation cleanup. Public wiring still awaits -the minimum identity contracts. [Initial codec benchmarks](benchmarks.md) record +conversion, mux/range/gzip handling and invocation cleanup. Public buffered wiring +subsequently completed with the identity contracts. [Initial codec benchmarks](benchmarks.md) record allocation baselines; they do not establish a peak-memory bound. Research and present a contract covering: @@ -221,6 +232,10 @@ presenting an entire frozen public API at once. - [x] Implement bounded envelope/body encoding and shared invocation cleanup. - [x] Extract shared invocation ownership and implement the private streaming bridge, with handoff, backpressure, cancellation, cleanup, and error tests. +- [x] Implement the streaming HTTP writer, raw/typed REST entry points, reporter, + examples and authentication/authorization integration tests under 0022. +- [ ] Resolve the streaming SDK deployment qualification through authoritative + clarification, a suitable stable SDK fix or a separately approved live test. - [x] Implement the minimum reviewed identity/context and native gateway producers needed to honor WithGatewayIdentity(true) before publishing invocation. - [x] Implement raw invocation after the remaining transport contracts are reviewed. @@ -241,8 +256,8 @@ presenting an entire frozen public API at once. CI configuration is present for Go 1.27.0/latest 1.27 patch tests, race detection, vet, and Lambda-target builds. It has not run on GitHub yet. Local Go 1.27.1 race -tests, vet, formatting checks, and both target builds pass for the public buffered -adapter. SDK registration, gateway recognition/fidelity, owned claims, concurrent +tests, vet, formatting checks, and both target builds pass for both public +adapters. SDK registration, gateway recognition/fidelity, owned claims, concurrent identity isolation, multipart cleanup, panic/cancellation and raw envelope limits are exercised. Runnable examples and initial public allocation benchmarks are present. HTTP 1.0 authentication coverage is explicitly qualified in diff --git a/docs/streaming.md b/docs/streaming.md new file mode 100644 index 0000000..7a759a6 --- /dev/null +++ b/docs/streaming.md @@ -0,0 +1,160 @@ +# REST response streaming + +`edge.NewStreaming` adapts a REST API Lambda proxy event to an ordinary +`http.Handler`. The returned reader contains bounded JSON metadata, eight NUL +bytes, then raw body bytes. AWS consumes that prefix; it is not application body +content. Streaming input requests and API Gateway HTTP API responses are not +supported. + +The local implementation is complete. **Deployment compatibility remains +unverified.** AWS Lambda Go v1.55.0 is the latest stable release checked on +2026-09-17. Local Runtime API probes show incremental delivery and reader closure, +but the SDK omits the documented streaming-mode header and reuses the response +connection. See [decision 0022](decisions/0022-streaming-writer-review.md#s3-sdk-issue-retain-the-existing-deployment-release-gate). +These observations neither prove real-service rejection nor establish deployed +support. No live AWS test has run, and edge does not patch the SDK or transport. + +## Registration and configuration + +In a Lambda main, construct the adapter, handle its startup error, and register +one method: + +```go +adapter, err := edge.NewStreaming(handler, + edge.WithStreamErrorReporter(func(ctx context.Context, err error) { + logger.ErrorContext(ctx, "response stream failed", "error", err) + }), +) +if err != nil { + log.Fatal(err) +} +lambda.Start(adapter.Handle) +``` + +`Handle(context.Context, jsontext.Value)` captures raw event JSON through the SDK, +then validates/decodes it using direct JSON v2 inside edge. Explicitly versioned +HTTP API envelopes are rejected. Alternatively register +`lambda.Start(adapter.HandleV1)` for an `events.APIGatewayProxyRequest`; this skips +request-envelope JSON work inside edge, with input codec validation and numeric +fidelity owned by the caller/SDK. The V1 type also represents HTTP API payload +1.0, so its type cannot prove that the deployment is a REST streaming integration. +Both entry points encode output metadata using direct JSON v2. + +Do not register the StreamingAdapter object or wrap it in a byte-returning +Invoke method. The SDK must receive the reader to consume incrementally. Direct +callers own reading and closing it. There is no HandleV2 streaming entry point. + +The AWS configuration requires a REST proxy integration with response transfer +mode STREAM and an InvokeWithResponseStream integration URI. The buffered +integration configuration alone does not enable streaming. See the official +[REST streaming overview](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode.html) +and [Lambda streaming integration format](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html). + +Existing gateway-identity, claims-budget and response-header-budget options also +apply. Gateway identity remains disabled by default; inherited callers are +rejected. The adapter is immutable and reusable; the application owns handler +concurrency safety. Each invocation has its own request, context and writer. + +## Commitment, flushing and limits + +The first final WriteHeader, Write or Flush commits status and an owned header +snapshot. Later ordinary header mutations cannot change the response. An empty +Write commits 200 when needed but does not force publication. WriteHeader alone +does not necessarily hand off a reader. + +When Content-Type needs detection, edge retains at most 512 body bytes. Filling +that prefix, flushing or completing the handler publishes metadata and drains +pending bytes through an unbuffered pipe. An explicit or suppressed Content-Type, +or a nonblank Content-Encoding, permits immediate publication on the first +nonempty Write. Empty Content-Encoding values do not suppress detection. + +Use `http.NewResponseController(w).Flush()` to observe flush errors. The writer +also supports `http.Flusher`; its void Flush records failures for finalization. +An empty Flush publishes without inventing a type, and later writes cannot +change it. Flush provides an application publication boundary, not a promise of +client receipt or preserved network chunk boundaries. Middleware wrappers must +preserve FlushError/Flusher or expose ResponseController-compatible Unwrap. + +Streaming never infers Content-Length, including tiny, empty and HEAD responses. +A valid supplied length is honored when status permits it. A whole Write that +would exceed it fails before any bytes from that Write are accepted. A short +non-HEAD body fails at handler completion. Edge does not generate Transfer-Encoding; +Gateway owns its transport framing. This length policy intentionally differs +from buffered edge and native net/http length inference. + +HEAD counts/discards representation writes, may detect their type and permits a +supplied representation length without requiring that many writes. Statuses +204, 205 and 304 suppress bodies and Content-Length; 304 also suppresses +Content-Type. Nonempty writes for these statuses return http.ErrBodyNotAllowed +without invalidating the response. The 205 rule is an intentional difference +from net/http. Informational responses, application trailers, Hijacker, Pusher, +deadline controls and full duplex are unsupported. + +The shared default header resource budget is 256 KiB, adjustable through +WithResponseHeaderBudget. Separately, the complete metadata prefix including +delimiter must fit 16,000 bytes, edge's conservative interpretation of the AWS +metadata constraint. Validation happens before reader handoff. MultiValueHeaders +preserves repeated response fields, including Set-Cookie. + +The streaming body is raw, without base64 or a JSON envelope. It does not use the +buffered adapter's 6 MiB envelope limit; edge does not add an arbitrary body cap. +This is not a claim to exceed AWS service size, bandwidth or timeout limits. +Retained sniff bytes and metadata are bounded, not the application's allocations. + +## Authentication and application protocols + +Authenticate, resolve the action/resource and authorize before writing or +flushing. The same `authn` and `authz` components compose with both adapters. +Denied access, dependency outages, challenges, CORS and JSON error bodies remain +ordinary HTTP responses selected by application middleware. A valid 401, 403 or +503 response is a successful transport outcome. + +After commitment, a later authorization failure cannot replace the status or +earlier bytes. The application owns ongoing authorization, heartbeats and +protocol-specific terminal messages such as an SSE error event. Edge does not +turn a late transport failure into an application record or retry a stream. + +The [runnable examples](../streaming_example_test.go) cover SSE, direct JSON v2 +NDJSON, binary and gzip bodies. Their local reader collector is only for displaying +test output; a Lambda wrapper must return the reader directly. For gzip, negotiate +Accept-Encoding in the application, set the response type/encoding before writing, +flush the encoder before the HTTP writer, and close the encoder to finish its +footer. Gateway does not perform streaming content compression for the integration. + +## Errors and ownership + +Logical HTTP commitment and reader handoff are different boundaries. Before the +handoff, invocation faults return a nil reader and error; early producer panics +propagate after cleanup. After handoff, failures preserve status and prior bytes +and become sanitized terminal reader errors. Late panics are recovered as ErrStream. +Use errors.Is for public categories and context.Canceled/context.DeadlineExceeded; +errors.AsType can inspect sanitized InvocationError details. + +WithStreamErrorReporter runs once for a terminal failure, after request cleanup +and before completion. Pre-handoff failures are returned directly and are not +reported. The callback receives the invocation context, possibly canceled, and +must return promptly. It must not read or close its own stream, which would wait +for the callback itself to finish. A callback panic adds a sanitized ErrStream +without replacing the original failure. New rejects this streaming-only option; +NewStreaming rejects a final nil reporter. Options use their existing ordered, +last-assignment-wins policy. + +The bridge uses one producer and backpressure, not a goroutine or queued body per +Write. Read and Close coordinate with producer completion. Terminal reads wait +for cleanup/reporting; Close cancels the request, unblocks pipe writes and joins +the producer. Handlers must observe request cancellation and stop on write/flush +errors. A handler that ignores both can delay Close until invocation termination. +Do not abandon a returned reader without closing it. + +Request bodies and multipart files on the served request remain owned until +producer cleanup; multipart files attached only to a middleware-created request +copy remain that middleware's responsibility. The request context is canceled on +completion. Local SDK probes verify reader Close before the next invocation. +Client disconnection need not immediately cancel Lambda execution; this API +cannot promise disconnection notification that AWS does not provide. + +The SDK probe also covers its handling of readers returning bytes plus an error: +edge exposes those bytes first and the error on the next read, preserving the +payload despite the current SDK reader wrapper. See the accepted +[ownership design](decisions/0009-streaming-boundaries.md) and +[writer review](decisions/0022-streaming-writer-review.md) for evidence and limits. diff --git a/internal/streamprobe/runtime_test.go b/internal/streamprobe/runtime_test.go index 501ca86..fe15758 100644 --- a/internal/streamprobe/runtime_test.go +++ b/internal/streamprobe/runtime_test.go @@ -1,8 +1,9 @@ // Package streamprobe tests the pinned SDK transport against a local Runtime API. -// It is a design probe, not an implementation of edge's streaming API. +// It covers SDK behavior and the actual edge streaming writer without live AWS. package streamprobe import ( + "bufio" "bytes" "context" "encoding/json/jsontext" @@ -23,6 +24,8 @@ import ( "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" + + "github.com/asteroid-computing/go-lambda-edge" ) const eventJSON = `{"httpMethod":"GET","path":"/","requestContext":{"apiId":"example"},"number":9007199254740993,"duplicate":1,"duplicate":2}` @@ -57,11 +60,15 @@ func (s *probeStream) MarshalJSON() ([]byte, error) { return nil, errors.New("stream is not a JSON response") } func (s *probeStream) Close() error { + var closeErr error + if body, ok := s.body.(io.Closer); ok { + closeErr = body.Close() + } response, err := (&http.Client{Timeout: 3 * time.Second}).Get(s.url + "/closed") if err != nil { - return err + return errors.Join(closeErr, err) } - return response.Body.Close() + return errors.Join(closeErr, response.Body.Close()) } type gatedTail struct { @@ -102,6 +109,10 @@ func TestRuntimeStreamingProbeProcess(t *testing.T) { if mode == "" { return } + if strings.HasPrefix(mode, "edge_") { + runEdgeStream(t, mode) + return + } makeStream := func(ctx context.Context) (io.ReadCloser, error) { prefix, err := json.Marshal(struct { StatusCode int `json:"statusCode"` @@ -140,8 +151,66 @@ func TestRuntimeStreamingProbeProcess(t *testing.T) { }) } +func runEdgeStream(t *testing.T, mode string) { + t.Helper() + url := "http://" + os.Getenv("AWS_LAMBDA_RUNTIME_API") + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain") + if mode == "edge_late_error" { + w.Header().Set("Content-Length", "99") + } + if _, err := io.WriteString(w, "first\n"); err != nil { + return + } + if err := http.NewResponseController(w).Flush(); err != nil { + return + } + request, err := http.NewRequestWithContext(r.Context(), http.MethodGet, url+"/continue", nil) + if err != nil { + panic(err) + } + response, err := http.DefaultClient.Do(request) + if err != nil { + panic(err) + } + _ = response.Body.Close() + if mode == "edge_late_error" { + return + } + if mode == "edge_late_panic" { + panic("synthetic private panic") + } + _, _ = io.WriteString(w, "last\n") + })) + if err != nil { + t.Fatal(err) + } + observe := func(s io.ReadCloser, err error) (io.ReadCloser, error) { + if err != nil { + return nil, err + } + content, ok := s.(interface{ ContentType() string }) + if !ok || content.ContentType() != integrationContentType { + _ = s.Close() + return nil, errors.New("edge stream lost integration content type") + } + // Forward the production reader unchanged; the wrapper only records that + // SDK Close joined edge cleanup before the next invocation is requested. + return &probeStream{body: s, url: url}, nil + } + if mode == "edge_typed" { + lambda.Start(func(ctx context.Context, event events.APIGatewayProxyRequest) (io.ReadCloser, error) { + return observe(a.HandleV1(ctx, event)) + }) + return + } + lambda.Start(func(ctx context.Context, event jsontext.Value) (io.ReadCloser, error) { + return observe(a.Handle(ctx, event)) + }) +} + func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { - for _, mode := range []string{"raw", "typed", "late_error", "data_and_error", "guarded_data_and_error"} { + for _, mode := range []string{"raw", "typed", "late_error", "data_and_error", "guarded_data_and_error", "edge_raw", "edge_typed", "edge_late_error", "edge_late_panic"} { t.Run(mode, func(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), 15*time.Second) defer cancel() @@ -161,6 +230,7 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { results := make(chan result, 1) var invokes atomic.Int32 var nextConnection atomic.Value + var streamClosed atomic.Bool server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { case "/continue": @@ -170,10 +240,14 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { case <-r.Context().Done(): } case "/closed": + streamClosed.Store(true) closed <- struct{}{} w.WriteHeader(http.StatusNoContent) case "/2018-06-01/runtime/invocation/next": if invokes.Add(1) > 1 { + if !streamClosed.Load() { + t.Error("next invocation requested before stream Close completed") + } nextConnection.Store(r.RemoteAddr) next <- struct{}{} <-r.Context().Done() @@ -181,21 +255,39 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { } w.Header().Set("Lambda-Runtime-Aws-Request-Id", "probe") w.Header().Set("Lambda-Runtime-Deadline-Ms", strconv.FormatInt(time.Now().Add(time.Minute).UnixMilli(), 10)) - if _, err := io.WriteString(w, eventJSON); err != nil { + input := eventJSON + if strings.HasPrefix(mode, "edge_") { + input = strings.Replace(input, `,"duplicate":1,"duplicate":2`, "", 1) + } + if _, err := io.WriteString(w, input); err != nil { t.Error(err) } case "/2018-06-01/runtime/invocation/probe/response": - // Read only one byte before releasing the second part of the - // stream. A fully buffered implementation cannot pass this. - one := make([]byte, 1) - _, err := io.ReadFull(r.Body, one) + // Require the first application body segment, not just metadata, + // before releasing the producer's second segment. + reader := bufio.NewReader(r.Body) + var initial []byte + var err error + for len(initial) < 16000 && !bytes.HasSuffix(initial, make([]byte, 8)) { + var b byte + b, err = reader.ReadByte() + if err != nil { + break + } + initial = append(initial, b) + } + if err == nil { + first := make([]byte, len("first\n")) + _, err = io.ReadFull(reader, first) + initial = append(initial, first...) + } if err != nil { results <- result{err: err} return } firstRead <- struct{}{} - rest, err := io.ReadAll(r.Body) - results <- result{body: append(one, rest...), header: r.Header.Clone(), trailer: r.Trailer.Clone(), chunked: slices.Contains(r.TransferEncoding, "chunked"), closeConnection: r.Close, connection: r.RemoteAddr, err: err} + rest, err := io.ReadAll(reader) + results <- result{body: append(initial, rest...), header: r.Header.Clone(), trailer: r.Trailer.Clone(), chunked: slices.Contains(r.TransferEncoding, "chunked"), closeConnection: r.Close, connection: r.RemoteAddr, err: err} w.WriteHeader(http.StatusAccepted) default: data, err := io.ReadAll(r.Body) @@ -256,8 +348,8 @@ func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { t.Fatalf("invalid integration framing: %q", got.body) } wantBody := "first\nlast\n" - wantError := mode == "late_error" || mode == "data_and_error" || mode == "guarded_data_and_error" - if mode == "late_error" || mode == "data_and_error" { + wantError := mode == "late_error" || mode == "data_and_error" || mode == "guarded_data_and_error" || mode == "edge_late_error" || mode == "edge_late_panic" + if mode == "late_error" || mode == "data_and_error" || mode == "edge_late_error" || mode == "edge_late_panic" { wantBody = "first\n" } if string(body) != wantBody || (got.trailer.Get("Lambda-Runtime-Function-Error-Type") != "") != wantError { diff --git a/response_headers.go b/response_headers.go index 3e07398..9f6f075 100644 --- a/response_headers.go +++ b/response_headers.go @@ -30,6 +30,30 @@ type responseHeaders struct { noContentLength bool } +// canSniffType uses the committed, trimmed fields. A later nonblank encoding +// value still suppresses detection if an earlier repeated value was empty. +func (h *responseHeaders) canSniffType() bool { + if _, present := h.fields["Content-Type"]; present || h.noContentType { + return false + } + for _, encoding := range h.fields["Content-Encoding"] { + if encoding != "" { + return false + } + } + return true +} + +func responseTrailers(fields http.Header) error { + for name, values := range fields { + declared := len(name) == len("Trailer") && strings.EqualFold(name, "Trailer") && len(values) != 0 + if strings.HasPrefix(name, http.TrailerPrefix) || declared { + return errResponseTrailers + } + } + return nil +} + // chargeResponseHeaders bounds work and copying before sorting or allocation. // Empty slices count as one entry so suppression markers also have a cost. func chargeResponseHeaders(fields http.Header, remaining int) (int, error) { diff --git a/streaming_adapter.go b/streaming_adapter.go new file mode 100644 index 0000000..f33f48c --- /dev/null +++ b/streaming_adapter.go @@ -0,0 +1,119 @@ +package edge + +import ( + "context" + "encoding/json/jsontext" + "errors" + "io" + "net/http" + + "github.com/aws/aws-lambda-go/events" +) + +// StreamingAdapter serves REST API proxy events through an ordinary HTTP handler +// and returns a metadata-prefixed body stream. Construct it with NewStreaming; +// its zero value is unusable. Configuration is immutable; the application owns +// handler concurrency safety. HTTP APIs and request streaming are unsupported. +// +// Deployment requires a REST STREAM integration using InvokeWithResponseStream. +// SDK Runtime API compatibility remains unverified against live AWS; see the +// module's streaming guide for the documented SDK/header/connection qualification. +type StreamingAdapter struct { + handler http.Handler + config config +} + +// WithStreamErrorReporter reports a sanitized terminal failure once after +// producer cleanup and before stream completion. It applies only to NewStreaming; +// New rejects it. A final nil callback is invalid. The callback receives the +// invocation context, possibly canceled, and must return promptly. Pre-handoff +// failures are returned directly and not reported. A callback panic is recovered +// and adds a sanitized ErrStream fault without replacing the original failure. +// It must not read or close the same stream: reporting precedes its completion. +func WithStreamErrorReporter(report func(context.Context, error)) Option { + return func(c *config) { + c.streamReporter, c.streamReporterSet = report, true + } +} + +// NewStreaming validates handler and options without starting I/O or goroutines. +// It shares New's handler, header-budget, identity-budget and gateway defaults. +// Streaming never infers Content-Length; supply it explicitly when needed. +// Authenticate and authorize before writing or flushing any response. +func NewStreaming(handler http.Handler, opts ...Option) (*StreamingAdapter, error) { + cfg, err := configureAdapter(handler, opts) + if err != nil { + return nil, err + } + if cfg.streamReporterSet && cfg.streamReporter == nil { + return nil, errors.New("edge: nil stream error reporter") + } + return &StreamingAdapter{handler: handler, config: cfg}, nil +} + +// Handle validates raw REST event JSON with JSON v2 and returns a stream that +// callers must consume and Close. Register lambda.Start(adapter.Handle), not the +// adapter object or the buffered Handler.Invoke boundary. The SDK owns its initial +// jsontext.Value capture; edge owns strict event decoding and prefix encoding. +// Explicit HTTP API version envelopes are rejected. Do not mutate event during +// the call. Request resources remain owned until producer cleanup completes. +// +// Pre-handoff failures return a nil stream. Early producer panics propagate after +// cleanup; late panics become sanitized terminal errors. After handoff, failures +// cannot replace status or bytes. Close cancels and joins the producer; a handler +// ignoring cancellation/write errors can delay Close until invocation timeout. +func (a *StreamingAdapter) Handle(ctx context.Context, event jsontext.Value) (io.ReadCloser, error) { + if err := a.valid(); err != nil { + return nil, err + } + return a.start(ctx, func() (decodedEvent, error) { + decoded, err := decodeEvent(event) + if err != nil { + return decodedEvent{}, err + } + if decoded.version != "" || decoded.v1 == nil { + return decodedEvent{}, invocationError("validate", ErrUnsupportedEvent, "streaming requires a REST proxy event") + } + return decoded, nil + }, true) +} + +// HandleV1 accepts an already typed REST event, avoiding request envelope JSON +// work inside edge. Register lambda.Start(adapter.HandleV1). The caller/upstream +// codec owns wire validation and numeric fidelity. The V1 type also represents +// HTTP API 1.0; it cannot prove the deployment is a REST streaming integration. +// Its stream ownership and error contract match Handle. Inputs must not change +// during the call. Streaming metadata still requires direct JSON v2 encoding. +func (a *StreamingAdapter) HandleV1(ctx context.Context, event events.APIGatewayProxyRequest) (io.ReadCloser, error) { + if err := a.valid(); err != nil { + return nil, err + } + return a.start(ctx, func() (decodedEvent, error) { return decodedEvent{v1: &event}, nil }, false) +} + +func (a *StreamingAdapter) valid() error { + if a == nil || a.handler == nil { + return invocationError("validate", ErrInvalidInvocation, "uninitialized streaming adapter") + } + return nil +} + +func (a *StreamingAdapter) start(ctx context.Context, decode func() (decodedEvent, error), raw bool) (io.ReadCloser, error) { + var request *http.Request + stream, err := startStream(ctx, func(ctx context.Context, inv *invocation) error { + event, err := decode() + if err != nil { + return err + } + request, err = prepareRequest(ctx, inv, event, raw, a.config) + return err + }, func(ctx context.Context, out *streamOutput) error { + w := newStreamingWriter(ctx, request.Method, a.config.responseHeaderBudget, out) + a.handler.ServeHTTP(w, request) + return w.finish() + }, a.config.streamReporter) + if err != nil { + return nil, err // Do not put a typed nil stream in the returned interface. + } + return stream, nil +} diff --git a/streaming_adapter_test.go b/streaming_adapter_test.go new file mode 100644 index 0000000..13e3de1 --- /dev/null +++ b/streaming_adapter_test.go @@ -0,0 +1,384 @@ +package edge_test + +import ( + "bytes" + "context" + "encoding/json/jsontext" + "encoding/json/v2" + "errors" + "io" + "mime/multipart" + "net/http" + "os" + "strings" + "sync" + "sync/atomic" + "testing" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-lambda-go/lambda" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func restEvent() events.APIGatewayProxyRequest { + return events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/orders", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}} +} + +func streamWire(t testing.TB, s io.ReadCloser) (int, http.Header, []byte, error) { + t.Helper() + if s == nil { + t.Fatal("nil stream") + } + wire, err := io.ReadAll(s) + closeErr := s.Close() + if !errors.Is(closeErr, err) { + t.Errorf("Close=%v read=%v", closeErr, err) + } + prefix, body, ok := bytes.Cut(wire, make([]byte, 8)) + if !ok { + t.Fatalf("missing delimiter: %q", wire) + } + var meta struct { + Status int `json:"statusCode"` + Headers http.Header `json:"multiValueHeaders"` + } + if decodeErr := json.Unmarshal(prefix, &meta); decodeErr != nil { + t.Fatal(decodeErr) + } + return meta.Status, meta.Headers, body, err +} + +func TestStreamingConstructorsAndValidation(t *testing.T) { + var mux *http.ServeMux + for _, h := range []http.Handler{nil, http.HandlerFunc(nil), mux} { + if a, err := edge.NewStreaming(h); err == nil || a != nil { + t.Errorf("nil handler accepted: %v %v", a, err) + } + } + h := http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Error("invalid invocation reached handler") }) + for _, opts := range [][]edge.Option{{nil}, {edge.WithResponseHeaderBudget(0)}, {edge.WithIdentityClaimsBudget(0)}, {edge.WithStreamErrorReporter(nil)}} { + if a, err := edge.NewStreaming(h, opts...); err == nil || a != nil { + t.Errorf("invalid options accepted: %v %v", a, err) + } + } + report := edge.WithStreamErrorReporter(func(context.Context, error) { t.Error("invalid invocation reported as late failure") }) + if a, err := edge.New(h, report); err == nil || a != nil { + t.Fatal("buffered constructor accepted stream reporter") + } + a, err := edge.NewStreaming(h, edge.WithStreamErrorReporter(nil), report) + if err != nil { + t.Fatal(err) + } + wire, err := json.Marshal(restEvent()) + if err != nil { + t.Fatal(err) + } + for _, input := range []jsontext.Value{nil, []byte("null"), []byte(`{"httpMethod":"GET","httpMethod":"POST"}`), append([]byte(`{"version":"1.0",`), wire[1:]...), []byte(`{"version":"2.0","rawPath":"/","requestContext":{"apiId":"api","http":{"method":"GET"}}}`)} { + if s, err := a.Handle(t.Context(), input); err == nil || s != nil { + t.Errorf("invalid/unsupported event returned stream=%v err=%v", s, err) + } + } + if s, err := a.HandleV1(nil, restEvent()); !errors.Is(err, edge.ErrInvalidInvocation) || s != nil { + t.Errorf("nil context: %v %v", s, err) + } + if s, err := a.HandleV1(t.Context(), events.APIGatewayProxyRequest{}); !errors.Is(err, edge.ErrInvalidEvent) || s != nil { + t.Errorf("invalid typed event: %v %v", s, err) + } + caller, err := identity.NewIAM("arn:aws:iam::123456789012:user/Alice", identity.SourceCustomAssertion) + if err != nil { + t.Fatal(err) + } + ctx, err := identity.WithCaller(t.Context(), caller) + if err != nil { + t.Fatal(err) + } + if s, err := a.Handle(ctx, wire); !errors.Is(err, identity.ErrConflict) || s != nil { + t.Errorf("inherited identity: %v %v", s, err) + } + for _, bad := range []*edge.StreamingAdapter{nil, new(edge.StreamingAdapter)} { + if s, err := bad.HandleV1(t.Context(), restEvent()); !errors.Is(err, edge.ErrInvalidInvocation) || s != nil { + t.Errorf("zero adapter: %v %v", s, err) + } + } +} + +func TestStreamingSDKRegistrationAndBinary(t *testing.T) { + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/orders" { + t.Error("wrong path") + } + w.Header().Set("Content-Type", "application/octet-stream") + w.WriteHeader(400) + _, _ = w.Write([]byte{0, 255, 1}) + })) + if err != nil { + t.Fatal(err) + } + wire, err := json.Marshal(restEvent()) + if err != nil { + t.Fatal(err) + } + // This collector verifies registration/serialization, not incremental delivery. + // The actual Runtime API subprocess probe covers incremental transport. + for _, handler := range []any{a.Handle, a.HandleV1} { + out, err := lambda.NewHandler(handler).Invoke(t.Context(), wire) + if err != nil { + t.Fatal(err) + } + status, headers, body, err := streamWire(t, io.NopCloser(bytes.NewReader(out))) + if err != nil || status != 400 || headers.Get("Content-Length") != "" || !bytes.Equal(body, []byte{0, 255, 1}) { + t.Fatalf("stream response=%d %v %v %v", status, headers, body, err) + } + } +} + +func TestStreamingLateErrorsAndReporting(t *testing.T) { + for _, mode := range []string{"short", "trailer", "panic", "reporter_panic"} { + t.Run(mode, func(t *testing.T) { + var reports atomic.Int64 + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain") + if mode == "short" { + w.Header().Set("Content-Length", "99") + } + if _, err := io.WriteString(w, "first"); err != nil { + return + } + if mode == "panic" || mode == "reporter_panic" { + panic("private application detail") + } + if mode == "trailer" { + w.Header().Set("Trailer", "X-Private") + } + }), edge.WithStreamErrorReporter(func(_ context.Context, err error) { + reports.Add(1) + if err == nil || strings.Contains(err.Error(), "private") { + t.Errorf("unsafe report: %v", err) + } + if mode == "reporter_panic" { + panic("private reporter detail") + } + })) + if err != nil { + t.Fatal(err) + } + s, err := a.HandleV1(t.Context(), restEvent()) + if err != nil { + t.Fatal(err) + } + status, _, body, readErr := streamWire(t, s) + want := edge.ErrStream + if mode == "short" { + want = http.ErrContentLength + } + if mode == "trailer" { + want = http.ErrNotSupported + } + if status != 200 || string(body) != "first" || !errors.Is(readErr, want) || reports.Load() != 1 { + t.Fatalf("status=%d body=%q err=%v reports=%d", status, body, readErr, reports.Load()) + } + }) + } +} + +func TestStreamingEarlyFailuresAndPanic(t *testing.T) { + for _, mode := range []string{"panic", "prefix", "short"} { + t.Run(mode, func(t *testing.T) { + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch mode { + case "panic": + panic("early panic") + case "prefix": + w.Header().Set("Large", strings.Repeat("x", 16000)) + w.(http.Flusher).Flush() + case "short": + w.Header().Set("Content-Length", "10") + _, _ = io.WriteString(w, "short") + } + }), edge.WithStreamErrorReporter(func(context.Context, error) { t.Error("early failure was reported") })) + if err != nil { + t.Fatal(err) + } + defer func() { + got := recover() + if mode == "panic" && got != "early panic" || mode != "panic" && got != nil { + t.Errorf("panic=%v", got) + } + }() + s, err := a.HandleV1(t.Context(), restEvent()) + if s != nil || err == nil { + t.Errorf("early failure returned %v %v", s, err) + } + }) + } +} + +func TestStreamingCloseAndParentCancellation(t *testing.T) { + for _, parentCancel := range []bool{false, true} { + ctx, cancel := context.WithCancelCause(t.Context()) + var finished, reports atomic.Int64 + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + defer finished.Add(1) + w.Header().Set("Content-Type", "application/octet-stream") + if _, err := w.Write(bytes.Repeat([]byte("x"), 1024*1024)); !errors.Is(err, context.Canceled) { + t.Errorf("interrupted write=%v", err) + } + }), edge.WithStreamErrorReporter(func(_ context.Context, err error) { + reports.Add(1) + if !errors.Is(err, context.Canceled) || strings.Contains(err.Error(), "secret") { + t.Errorf("report=%v", err) + } + })) + if err != nil { + t.Fatal(err) + } + s, err := a.HandleV1(ctx, restEvent()) + if err != nil { + t.Fatal(err) + } + if parentCancel { + cancel(errors.New("secret cause")) + } + if err := s.Close(); !errors.Is(err, context.Canceled) { + t.Errorf("Close=%v", err) + } + if err := s.Close(); !errors.Is(err, context.Canceled) { + t.Error("Close not idempotent") + } + cancel(nil) + if finished.Load() != 1 || reports.Load() != 1 { + t.Fatalf("handler/reporter incomplete: %d %d", finished.Load(), reports.Load()) + } + } +} + +func TestStreamingGatewayIdentityAndIsolation(t *testing.T) { + for _, enabled := range []bool{false, true} { + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + j, ok := identity.FromContext(r.Context()).JWT() + value := "anonymous" + if ok { + value, _ = j.Subject() + } + _, _ = io.WriteString(w, value) + }), edge.WithGatewayIdentity(enabled)) + if err != nil { + t.Fatal(err) + } + var wg sync.WaitGroup + for range 10 { + for _, subject := range []string{"Alice", "Bob"} { + wg.Go(func() { + for _, raw := range []bool{true, false} { + event := restEvent() + event.RequestContext.Authorizer = map[string]any{"claims": map[string]any{"iss": "issuer", "sub": subject}} + var s io.ReadCloser + var err error + if raw { + wire, marshalErr := json.Marshal(event) + if marshalErr != nil { + t.Error(marshalErr) + return + } + s, err = a.Handle(t.Context(), wire) + } else { + s, err = a.HandleV1(t.Context(), event) + } + if err != nil { + t.Error(err) + return + } + _, _, body, err := streamWire(t, s) + want := "anonymous" + if enabled { + want = subject + } + if err != nil || string(body) != want { + t.Errorf("identity body=%q err=%v; want %q", body, err, want) + } + } + }) + } + } + wg.Wait() + } +} + +func TestStreamingMultipartCleanupBeforeEOF(t *testing.T) { + var body bytes.Buffer + form := multipart.NewWriter(&body) + part, err := form.CreateFormFile("upload", "file.txt") + if err != nil { + t.Fatal(err) + } + _, _ = io.WriteString(part, "temporary file content") + if err := form.Close(); err != nil { + t.Fatal(err) + } + var path string + var served context.Context + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + served = r.Context() + if err := r.ParseMultipartForm(0); err != nil { + t.Error(err) + return + } + file, err := r.MultipartForm.File["upload"][0].Open() + if err != nil { + t.Error(err) + return + } + if disk, ok := file.(*os.File); ok { + path = disk.Name() + } + _ = file.Close() + _, _ = io.WriteString(w, "done") + })) + if err != nil { + t.Fatal(err) + } + event := restEvent() + event.HTTPMethod, event.Body = "POST", body.String() + event.Headers = map[string]string{"Content-Type": form.FormDataContentType()} + s, err := a.HandleV1(t.Context(), event) + if err != nil { + t.Fatal(err) + } + _, _, output, err := streamWire(t, s) + if err != nil || string(output) != "done" || path == "" { + t.Fatalf("body=%q path=%q err=%v", output, path, err) + } + if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("multipart file remains: %v", err) + } + if served.Err() != context.Canceled { + t.Error("invocation context survived consumption") + } +} + +func TestStreamingDoesNotUseBufferedEnvelopeLimit(t *testing.T) { + chunk := bytes.Repeat([]byte("x"), 1024*1024) + a, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/octet-stream") + for range 8 { + if _, err := w.Write(chunk); err != nil { + t.Error(err) + return + } + } + })) + if err != nil { + t.Fatal(err) + } + s, err := a.HandleV1(t.Context(), restEvent()) + if err != nil { + t.Fatal(err) + } + defer s.Close() + n, err := io.Copy(io.Discard, s) + if err != nil || n <= 8*1024*1024 || n > 8*1024*1024+16000 { + t.Fatalf("stream bytes=%d err=%v; want 8 MiB body plus bounded metadata", n, err) + } +} diff --git a/streaming_example_test.go b/streaming_example_test.go new file mode 100644 index 0000000..f9c7033 --- /dev/null +++ b/streaming_example_test.go @@ -0,0 +1,165 @@ +package edge_test + +import ( + "bytes" + "compress/gzip" + "context" + "encoding/json/jsontext" + "encoding/json/v2" + "fmt" + "io" + "net/http" + + "github.com/aws/aws-lambda-go/events" + + "github.com/asteroid-computing/go-lambda-edge" +) + +func ExampleStreamingAdapter_Handle() { + adapter, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // Protected applications authenticate and authorize before this point. + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + controller := http.NewResponseController(w) + for _, message := range []string{"ready", "done"} { + if r.Context().Err() != nil { + return + } + if _, err := fmt.Fprintf(w, "data: %s\n\n", message); err != nil { + return + } + if err := controller.Flush(); err != nil { + return + } + } + })) + if err != nil { + panic(err) + } + // A Lambda main registers lambda.Start(adapter.Handle). This direct call + // demonstrates reader ownership locally; it makes no AWS calls. + stream, err := adapter.Handle(context.Background(), jsontext.Value(`{"httpMethod":"GET","path":"/","requestContext":{"apiId":"example"}}`)) + if err != nil { + panic(err) + } + fmt.Print(string(readExampleStream(stream))) + // Output: + // data: ready + // + // data: done +} + +func ExampleStreamingAdapter_HandleV1() { + adapter, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/x-ndjson") + for _, value := range []string{"ready", "done"} { + if r.Context().Err() != nil { + return + } + if err := json.MarshalWrite(w, struct { + State string `json:"state"` + }{value}); err != nil { + return + } + if _, err := io.WriteString(w, "\n"); err != nil { + return + } + if err := http.NewResponseController(w).Flush(); err != nil { + return + } + } + })) + if err != nil { + panic(err) + } + // A Lambda main may register lambda.Start(adapter.HandleV1). Input envelope + // decoding then belongs to the SDK; output metadata still uses JSON v2 in edge. + stream, err := adapter.HandleV1(context.Background(), exampleRESTEvent()) + if err != nil { + panic(err) + } + fmt.Print(string(readExampleStream(stream))) + // Output: + // {"state":"ready"} + // {"state":"done"} +} + +func ExampleNewStreaming_binary() { + adapter, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/octet-stream") + _, _ = w.Write([]byte{0, 255, 1}) + })) + if err != nil { + panic(err) + } + stream, err := adapter.HandleV1(context.Background(), exampleRESTEvent()) + if err != nil { + panic(err) + } + fmt.Printf("%v\n", readExampleStream(stream)) + // Output: [0 255 1] +} + +func ExampleNewStreaming_gzip() { + adapter, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain") + w.Header().Set("Content-Encoding", "gzip") + // The example always emits gzip; a real route negotiates Accept-Encoding. + compressed := gzip.NewWriter(w) + defer compressed.Close() + for _, chunk := range []string{"first\n", "last\n"} { + if r.Context().Err() != nil { + return + } + if _, err := io.WriteString(compressed, chunk); err != nil { + return + } + // Flush the encoder first, then the underlying HTTP response. + if err := compressed.Flush(); err != nil { + return + } + if err := http.NewResponseController(w).Flush(); err != nil { + return + } + } + })) + if err != nil { + panic(err) + } + stream, err := adapter.HandleV1(context.Background(), exampleRESTEvent()) + if err != nil { + panic(err) + } + reader, err := gzip.NewReader(bytes.NewReader(readExampleStream(stream))) + if err != nil { + panic(err) + } + defer reader.Close() + data, err := io.ReadAll(reader) + if err != nil { + panic(err) + } + fmt.Print(string(data)) + // Output: + // first + // last +} + +func exampleRESTEvent() events.APIGatewayProxyRequest { + return events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "example"}} +} + +// This collector is only for the small local examples. Lambda's runtime consumes +// the reader incrementally; do not collect the stream in a production wrapper. +func readExampleStream(stream io.ReadCloser) []byte { + defer stream.Close() + wire, err := io.ReadAll(stream) + if err != nil { + panic(err) + } + _, body, ok := bytes.Cut(wire, make([]byte, 8)) + if !ok { + panic("missing stream delimiter") + } + return body +} diff --git a/streaming_writer.go b/streaming_writer.go new file mode 100644 index 0000000..31f375c --- /dev/null +++ b/streaming_writer.go @@ -0,0 +1,189 @@ +package edge + +import ( + "context" + "io" + "math" + "net/http" +) + +// streamingWriter has one handler owner and retains at most 512 body bytes. +// Its context belongs to that invocation, not the reusable adapter. Logical +// HTTP commitment freezes headers; publish transfers ownership to the reader. +type streamingWriter struct { + ctx context.Context + out *streamOutput + header http.Header + headers *responseHeaders + headerBudget int + method string + status int + written int64 + declaredLength int64 + hasLength bool + published bool + sniff [512]byte + used int + err error +} + +func newStreamingWriter(ctx context.Context, method string, headerBudget int, out *streamOutput) *streamingWriter { + return &streamingWriter{ctx: ctx, out: out, header: make(http.Header), method: method, headerBudget: headerBudget} +} + +func (w *streamingWriter) Header() http.Header { return w.header } + +func (w *streamingWriter) WriteHeader(status int) { + if w.status != 0 || w.err != nil { + return + } + if status >= 100 && status <= 199 { + w.err = errInformationalResponse + return + } + if status < 200 || status > 599 { + w.err = invocationError("response", ErrResponse, "invalid final response status") + return + } + w.status = status + w.headers, w.err = snapshotResponseHeaders(w.header, w.headerBudget) + if w.err != nil { + return + } + if !responseBodyAllowed(status) { + w.headers.remove("Content-Length") + if status == http.StatusNotModified { + w.headers.remove("Content-Type") + } + } + w.declaredLength, w.hasLength, w.err = responseContentLength(w.headers.fields) +} + +func (w *streamingWriter) ready() error { + if w.status == 0 { + w.WriteHeader(http.StatusOK) + } + if w.err == nil { + w.err = w.ctx.Err() + } + return w.err +} + +func (w *streamingWriter) Write(p []byte) (int, error) { + if err := w.ready(); err != nil { + return 0, err + } + if len(p) == 0 { + return 0, nil + } + if !responseBodyAllowed(w.status) { + return 0, http.ErrBodyNotAllowed + } + if w.hasLength && int64(len(p)) > w.declaredLength-w.written { + w.err = invocationError("response", ErrResponse, "response exceeds declared content length", http.ErrContentLength) + return 0, w.err + } + if int64(len(p)) > math.MaxInt64-w.written { + w.err = invocationError("response", ErrResponse, "response byte count overflow") + return 0, w.err + } + if w.method == http.MethodHead { + if !w.published && w.headers.canSniffType() { + w.used += copy(w.sniff[w.used:], p) + } + w.written += int64(len(p)) + if !w.published && (!w.headers.canSniffType() || w.used == len(w.sniff)) { + return len(p), w.publish() + } + return len(p), nil + } + accepted := 0 + if !w.published && w.headers.canSniffType() { + accepted = copy(w.sniff[w.used:], p) + w.used += accepted + w.written += int64(accepted) + p = p[accepted:] + if w.used < len(w.sniff) { + return accepted, nil + } + } + if err := w.publish(); err != nil { + return accepted, err + } + if len(p) == 0 { + return accepted, nil + } + n, err := w.writeBody(p) + w.written += int64(n) + return accepted + n, err +} + +// publish validates all metadata before acknowledging handoff, then drains any +// previously accepted sniff bytes. A pipe error cannot undo accepted bytes. +func (w *streamingWriter) publish() error { + if err := w.ready(); err != nil { + return err + } + if w.err = responseTrailers(w.header); w.err != nil { + return w.err + } + if w.published { + return nil + } + if w.used != 0 && w.headers.canSniffType() { + w.err = w.headers.automatic("Content-Type", http.DetectContentType(w.sniff[:w.used])) + if w.err != nil { + return w.err + } + } + prefix, err := encodeStreamPrefix(w.status, w.headers) + if err != nil { + w.err = err + return err + } + if w.err = w.out.publish(prefix); w.err != nil { + return w.err + } + w.published = true + if w.used != 0 && w.method != http.MethodHead { + _, err = w.writeBody(w.sniff[:w.used]) + } + w.used = 0 + return err +} + +func (w *streamingWriter) writeBody(p []byte) (int, error) { + n, err := w.out.body.Write(p) + if err == nil && n != len(p) { + err = io.ErrShortWrite + } + if err != nil { + switch { + case w.ctx.Err() != nil: + w.err = w.ctx.Err() + case err == io.ErrClosedPipe: + w.err = err // The bridge translates its own interruption safely. + default: + w.err = invocationError("stream", ErrStream, "response body write failed") + } + } + return n, w.err +} + +func (w *streamingWriter) Flush() { _ = w.FlushError() } + +func (w *streamingWriter) FlushError() error { return w.publish() } + +func (w *streamingWriter) finish() error { + if err := w.ready(); err != nil { + return err + } + if w.err = responseTrailers(w.header); w.err != nil { + return w.err + } + if w.method != http.MethodHead && w.hasLength && w.written != w.declaredLength { + w.err = invocationError("response", ErrResponse, "response body length mismatch", http.ErrContentLength) + return w.err + } + return w.publish() +} diff --git a/streaming_writer_test.go b/streaming_writer_test.go new file mode 100644 index 0000000..642b7df --- /dev/null +++ b/streaming_writer_test.go @@ -0,0 +1,294 @@ +package edge + +import ( + "bytes" + "context" + "encoding/json/v2" + "errors" + "io" + "math" + "net/http" + "strings" + "testing" + "testing/synctest" +) + +type streamMetadata struct { + StatusCode int `json:"statusCode"` + Headers http.Header `json:"multiValueHeaders"` +} + +func parseStream(t testing.TB, wire []byte) (streamMetadata, []byte) { + t.Helper() + prefix, body, ok := bytes.Cut(wire, make([]byte, 8)) + if !ok { + t.Fatalf("stream has no delimiter: %q", wire) + } + var meta streamMetadata + if err := json.Unmarshal(prefix, &meta); err != nil { + t.Fatal(err) + } + return meta, body +} + +func TestContentEncodingSniffingShared(t *testing.T) { + for _, tc := range []struct { + name string + encoding []string + suppress bool + want string + }{ + {"absent", nil, false, "text/html; charset=utf-8"}, + {"empty", []string{""}, false, "text/html; charset=utf-8"}, + {"blank", []string{" \t "}, false, "text/html; charset=utf-8"}, + {"encoded", []string{"gzip"}, false, ""}, + {"later_encoding", []string{"", "gzip"}, false, ""}, + {"suppressed_type", []string{""}, true, ""}, + } { + t.Run(tc.name, func(t *testing.T) { + h := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Encoding"] = tc.encoding + if tc.suppress { + w.Header()["Content-Type"] = nil + } + _, _ = io.WriteString(w, "example") + }) + buffered, err := serveBuffered(h, &http.Request{Method: "GET"}, defaultResponseHeaderBudget) + if err != nil { + t.Fatal(err) + } + var wire bytes.Buffer + out := &streamOutput{publish: func(p []byte) error { _, err := wire.Write(p); return err }, body: &wire} + w := newStreamingWriter(t.Context(), "GET", defaultResponseHeaderBudget, out) + h.ServeHTTP(w, nil) + if err := w.finish(); err != nil { + t.Fatal(err) + } + meta, _ := parseStream(t, wire.Bytes()) + if got := meta.Headers.Get("Content-Type"); got != tc.want || buffered.headers.fields.Get("Content-Type") != tc.want { + t.Errorf("type: stream=%q buffered=%q; want %q", got, buffered.headers.fields.Get("Content-Type"), tc.want) + } + }) + } +} + +func TestStreamingWriterCommitmentAndSniffing(t *testing.T) { + var wire bytes.Buffer + publications := 0 + out := &streamOutput{publish: func(p []byte) error { publications++; _, err := wire.Write(p); return err }, body: &wire} + w := newStreamingWriter(t.Context(), "GET", defaultResponseHeaderBudget, out) + values := []string{"before"} + w.Header()["X-Value"] = values + w.Header().Add("Set-Cookie", "a=1") + w.Header().Add("Set-Cookie", "b=2") + w.WriteHeader(201) + values[0] = "after" + w.Header().Set("Content-Type", "ignored/after-commit") + w.WriteHeader(202) + _, _ = w.Write(nil) + _, _ = io.WriteString(w, "<") + _, _ = io.WriteString(w, "html>example") + if publications != 0 { + t.Fatal("small sniff prefix published before flush/completion") + } + if err := http.NewResponseController(w).Flush(); err != nil { + t.Fatal(err) + } + _, _ = io.WriteString(w, "tail") + w.Flush() + if err := w.finish(); err != nil { + t.Fatal(err) + } + meta, body := parseStream(t, wire.Bytes()) + if publications != 1 || meta.StatusCode != 201 || meta.Headers.Get("X-Value") != "before" || meta.Headers.Get("Content-Type") != "text/html; charset=utf-8" || len(meta.Headers.Values("Set-Cookie")) != 2 || string(body) != "exampletail" { + t.Fatalf("committed response: publications=%d meta=%+v body=%q", publications, meta, body) + } + if meta.Headers.Get("Content-Length") != "" || meta.Headers.Get("Transfer-Encoding") != "" { + t.Fatal("stream inferred transport framing") + } +} + +func TestStreamingWriterLengthAndBodyPolicies(t *testing.T) { + for _, tc := range []struct { + name, method, length string + status int + payload string + flush bool + wantErr error + wantBody, wantLength string + }{ + {name: "empty"}, + {name: "small", payload: "body", wantBody: "body"}, + {name: "explicit", length: "4", payload: "body", wantBody: "body", wantLength: "4"}, + {name: "zero", length: "0", wantLength: "0"}, + {name: "overflow", length: "2", payload: "body", wantErr: http.ErrContentLength}, + {name: "short_before", length: "9", payload: "body", wantErr: http.ErrContentLength}, + {name: "short_after", length: "9", payload: "body", flush: true, wantErr: http.ErrContentLength, wantBody: "body", wantLength: "9"}, + {name: "head", method: "HEAD", payload: "representation"}, + {name: "head_explicit", method: "HEAD", length: "99", payload: "representation", wantLength: "99"}, + {name: "head_omitted", method: "HEAD", length: "99", wantLength: "99"}, + {name: "head_overflow", method: "HEAD", length: "2", payload: "representation", wantErr: http.ErrContentLength}, + {name: "no_content", status: 204, payload: "forbidden"}, + {name: "reset", status: 205, payload: "forbidden"}, + {name: "not_modified", status: 304, payload: "forbidden"}, + } { + t.Run(tc.name, func(t *testing.T) { + var wire bytes.Buffer + out := &streamOutput{publish: func(p []byte) error { _, err := wire.Write(p); return err }, body: &wire} + w := newStreamingWriter(t.Context(), tc.method, defaultResponseHeaderBudget, out) + if tc.length != "" { + w.Header().Set("Content-Length", tc.length) + } + if tc.status != 0 { + w.WriteHeader(tc.status) + } + n, writeErr := io.WriteString(w, tc.payload) + if tc.status != 0 && (n != 0 || !errors.Is(writeErr, http.ErrBodyNotAllowed)) { + t.Errorf("bodyless write=%d %v", n, writeErr) + } + if tc.flush { + _ = w.FlushError() + } + err := w.finish() + if !errors.Is(err, tc.wantErr) { + t.Fatalf("finish=%v; want %v", err, tc.wantErr) + } + if tc.wantErr != nil && !tc.flush { + if wire.Len() != 0 { + t.Fatal("invalid unpublished response leaked data") + } + return + } + meta, body := parseStream(t, wire.Bytes()) + if string(body) != tc.wantBody || meta.Headers.Get("Content-Length") != tc.wantLength { + t.Errorf("body=%q length=%q; want %q %q", body, meta.Headers.Get("Content-Length"), tc.wantBody, tc.wantLength) + } + }) + } +} + +func TestStreamingWriterStickyFaults(t *testing.T) { + for _, tc := range []struct { + name string + setup func(*streamingWriter) + want error + }{ + {"informational", func(w *streamingWriter) { w.WriteHeader(103) }, http.ErrNotSupported}, + {"invalid_status", func(w *streamingWriter) { w.WriteHeader(600) }, ErrResponse}, + {"bad_header", func(w *streamingWriter) { w.Header().Set("Bad Header", "x"); w.WriteHeader(200) }, ErrResponse}, + {"bad_length", func(w *streamingWriter) { w.Header().Set("Content-Length", "-1"); w.WriteHeader(200) }, ErrResponse}, + {"header_budget", func(w *streamingWriter) { w.headerBudget = 1; w.Header().Set("X-Test", "x"); w.WriteHeader(200) }, ErrLimitExceeded}, + {"generated_type_budget", func(w *streamingWriter) { w.headerBudget = 1; _, _ = w.Write([]byte("body")); w.Flush() }, ErrLimitExceeded}, + {"prefix_size", func(w *streamingWriter) { w.Header().Set("Large", strings.Repeat("x", 16000)); w.Flush() }, ErrLimitExceeded}, + {"trailer", func(w *streamingWriter) { w.WriteHeader(200); w.Header().Set("Trailer", "X-Late"); w.Flush() }, http.ErrNotSupported}, + {"overflow_counter", func(w *streamingWriter) { w.written = math.MaxInt64; _, _ = w.Write([]byte("x")) }, ErrResponse}, + } { + t.Run(tc.name, func(t *testing.T) { + var wire bytes.Buffer + w := newStreamingWriter(t.Context(), "GET", defaultResponseHeaderBudget, &streamOutput{publish: func(p []byte) error { _, err := wire.Write(p); return err }, body: &wire}) + tc.setup(w) + w.WriteHeader(200) + if _, err := w.Write([]byte("ignored failure")); !errors.Is(err, tc.want) { + t.Errorf("Write=%v; want %v", err, tc.want) + } + if err := w.finish(); !errors.Is(err, tc.want) || wire.Len() != 0 { + t.Errorf("finish=%v published=%d", err, wire.Len()) + } + }) + } +} + +func TestStreamingEmptyFlushAndCapabilities(t *testing.T) { + var wire bytes.Buffer + w := newStreamingWriter(t.Context(), "GET", defaultResponseHeaderBudget, &streamOutput{publish: func(p []byte) error { _, err := wire.Write(p); return err }, body: &wire}) + controller := http.NewResponseController(w) + if err := controller.Flush(); err != nil { + t.Fatal(err) + } + _, _ = io.WriteString(w, "after flush") + if err := w.finish(); err != nil { + t.Fatal(err) + } + meta, body := parseStream(t, wire.Bytes()) + if meta.Headers.Get("Content-Type") != "" || meta.Headers.Get("Content-Length") != "" || len(body) == 0 { + t.Fatalf("empty flush invented metadata: %+v", meta) + } + if err := controller.EnableFullDuplex(); !errors.Is(err, http.ErrNotSupported) { + t.Fatal(err) + } + if _, _, err := controller.Hijack(); !errors.Is(err, http.ErrNotSupported) { + t.Fatal(err) + } + if _, ok := any(w).(http.Pusher); ok { + t.Fatal("unexpected push capability") + } +} + +func TestStreamingWriterBackpressureAndSniffThreshold(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var completed bool + s, err := startStream(t.Context(), prepareEmptyStream, func(ctx context.Context, out *streamOutput) error { + w := newStreamingWriter(ctx, "GET", defaultResponseHeaderBudget, out) + if n, err := w.Write(bytes.Repeat([]byte("a"), 511)); n != 511 || err != nil { + return errors.New("initial write failed") + } + // Filling byte 512 must publish, then block until the reader drains it. + if _, err := w.Write([]byte("bc")); err != nil { + return err + } + completed = true + return w.finish() + }, nil) + if err != nil { + t.Fatal(err) + } + defer s.Close() + synctest.Wait() + if completed { + t.Fatal("writer completed without a body consumer") + } + wire, err := io.ReadAll(s) + if err != nil { + t.Fatal(err) + } + _, body := parseStream(t, wire) + if len(body) != 513 || !completed || body[511] != 'b' || body[512] != 'c' { + t.Fatalf("body length=%d completed=%v", len(body), completed) + } + }) +} + +type partialStreamWriter struct{} + +func (partialStreamWriter) Write(p []byte) (int, error) { + return min(2, len(p)), errors.New("private write detail") +} + +func TestStreamingPartialWriteCountsAndSanitization(t *testing.T) { + for _, sniff := range []bool{false, true} { + w := newStreamingWriter(t.Context(), "GET", defaultResponseHeaderBudget, &streamOutput{ + publish: func([]byte) error { return nil }, body: partialStreamWriter{}, + }) + if sniff { + if n, err := w.Write(bytes.Repeat([]byte("x"), 511)); n != 511 || err != nil { + t.Fatalf("buffered Write=%d %v", n, err) + } + } else { + w.Header().Set("Content-Type", "text/plain") + } + n, err := w.Write([]byte("tail")) + want := 2 + if sniff { + want = 1 + } // The accepted sniff byte belongs to this call; the tail does not. + if n != want || !errors.Is(err, ErrStream) || strings.Contains(err.Error(), "private") { + t.Fatalf("Write=%d %v; want %d sanitized stream error", n, err, want) + } + if n, err := w.Write([]byte("again")); n != 0 || !errors.Is(err, ErrStream) { + t.Fatalf("sticky Write=%d %v", n, err) + } + if err := w.finish(); !errors.Is(err, ErrStream) { + t.Fatal(err) + } + } +} From 5400066c0b437c9ddd87fd226e2310f068e8f62e Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Thu, 17 Sep 2026 10:01:24 +0100 Subject: [PATCH 37/44] docs: verify SDK streaming contract and prepare AWS questions --- .../decisions/0022-streaming-writer-review.md | 11 ++ docs/plan.md | 6 + .../2026-09-17-sdk-streaming-follow-up.md | 169 ++++++++++++++++++ internal/streamprobe/runtime_test.go | 16 +- 4 files changed, 201 insertions(+), 1 deletion(-) create mode 100644 docs/reviews/2026-09-17-sdk-streaming-follow-up.md diff --git a/docs/decisions/0022-streaming-writer-review.md b/docs/decisions/0022-streaming-writer-review.md index 1991a43..ef11b27 100644 --- a/docs/decisions/0022-streaming-writer-review.md +++ b/docs/decisions/0022-streaming-writer-review.md @@ -251,3 +251,14 @@ SSE, direct JSON v2 NDJSON, binary and gzip examples are runnable. Full module race tests, vet, formatting, and Linux arm64/amd64 builds pass on Go 1.27.1. See the [consumer guide](../streaming.md). No AWS deployment or account API was used, and no SDK workaround was introduced. + +## Documentation and SDK follow-up + +The user's requested [2026-09-17 follow-up](../reviews/2026-09-17-sdk-streaming-follow-up.md) +rechecks current AWS docs, version-specific SDK Go docs and upstream discussions. +All direct AWS dependencies remain latest stable. A tenth probe case now returns +the SDK's APIGatewayProxyStreamingResponse directly and reproduces the missing +mode header and connection reuse without edge's adapter or encoder. The SDK's +explicit REST streaming documentation strengthens the intended-support evidence, +but the transport discrepancy remains unresolved. The follow-up includes a draft +of six precise questions for the user to send to AWS. No production policy changes. diff --git a/docs/plan.md b/docs/plan.md index 202768b..b8b2dd4 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -183,6 +183,12 @@ Runnable SSE, direct JSON v2 NDJSON, binary and gzip examples accompany the cover both streaming entry points. Full module race tests, vet, formatting and Linux arm64/amd64 builds pass locally on Go 1.27.1. +The [SDK documentation follow-up](reviews/2026-09-17-sdk-streaming-follow-up.md) +reconfirmed latest stable versions and reproduced the transport observations with +the SDK's own REST streaming response type. Ten local SDK probe cases now pass. +It records the evidence and questions prepared for the user's AWS outreach; +the deployment qualification remains open. + Implemented under decision 0006: bounded direct JSON v2 envelope encoding, text/base64 body selection, and a shared invocation scope that preserves primary errors, cleans up on failures/panics, and checks parent cancellation. The HTTP diff --git a/docs/reviews/2026-09-17-sdk-streaming-follow-up.md b/docs/reviews/2026-09-17-sdk-streaming-follow-up.md new file mode 100644 index 0000000..495619d --- /dev/null +++ b/docs/reviews/2026-09-17-sdk-streaming-follow-up.md @@ -0,0 +1,169 @@ +# AWS Lambda Go streaming: verification and AWS questions + +Checked 2026-09-17 using AWS public MCP documentation, version-specific Go docs, +downloaded SDK source, AWS's public GitHub repository and local Runtime API tests. +No AWS resources were deployed or invoked; this draft has not been sent to AWS. + +## Result + +All direct AWS dependencies match the latest stable versions returned by the Go +module proxy: + +| Module | Required and latest stable | Version timestamp | +| --- | --- | --- | +| github.com/aws/aws-lambda-go | v1.55.0 | 2026-08-27 | +| github.com/aws/aws-sdk-go-v2 | v1.47.0 | 2026-09-09 | +| github.com/aws/aws-sdk-go-v2/service/sts | v1.51.0 | 2026-09-14 | + +AWS's [latest Lambda Go release](https://github.com/aws/aws-lambda-go/releases/tag/v1.55.0) +independently confirms v1.55.0 and its Go 1.27 compatibility work. The Lambda Go +runtime library owns this streaming transport; the SDK v2 core and STS modules +serve our IAM-proof path. No dependency change is necessary. + +The strongest positive evidence is the SDK's own +[APIGatewayProxyStreamingResponse documentation](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.55.0/events#APIGatewayProxyStreamingResponse). +It explicitly describes AWS_PROXY, the response-streaming-invocations URI and +STREAM transfer mode. [PR 603](https://github.com/aws/aws-lambda-go/pull/603), merged +on 2025-12-05, introduced that type; its review discusses API Gateway testing. +This was already noted in decision 0008 and is reconfirmed here. It supports the +interpretation that this is an intended SDK path, not an accidental use of a +Function URL-only API. + +The reader contract is also documented by +[lambda.Start](https://pkg.go.dev/github.com/aws/aws-lambda-go@v1.55.0/lambda#Start): +reader returns, closing responses that implement io.Closer, reporting read errors, +and JSON serialization precedence. The response type's +[implementation](https://github.com/aws/aws-lambda-go/blob/v1.55.0/events/apigw.go#L37-L93) +uses the same content type, metadata/delimiter structure and deliberate JSON +serialization rejection as edge. Edge retains its own bounded JSON v2 encoder +and lifecycle; replacing it with the SDK response wrapper would not change the +underlying Runtime API transport. + +## Remaining discrepancy and local control + +The current [custom-runtime guide](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html#runtimes-custom-response-streaming) +still requires an explicit streaming-mode header, chunked encoding and closing +the underlying connection after writing the response. The SDK's +[runtime client](https://github.com/aws/aws-lambda-go/blob/v1.55.0/lambda/runtime_api_client.go#L128-L160) +uses Go's chunked request transport, but does not set the mode header or request +connection closure. Closing an io.ReadCloser or an HTTP response body is distinct +from closing the underlying TCP connection. + +The local probe was extended with `sdk_gateway`, returning +`*events.APIGatewayProxyStreamingResponse` directly through lambda.Start. It uses +neither edge's adapter nor its prefix encoder. The body contains a gated reader; +the local Runtime API server must receive the first body segment before it +releases the remaining segment. A body wrapper only observes Close. + +Run from the repository: + +```sh +go test -race -count=1 -v ./internal/streamprobe -run '^TestSDKStreamsThroughRuntimeAPI$' +``` + +All ten cases pass on Go 1.27.1. The direct SDK control and edge cases all show: + +```text +incremental=true response-mode="" +runtime-request-close=false response-connection-reused-for-next=true +``` + +The test also checks chunked transfer, integration content type, metadata/body +framing and Close before the next invocation. Fault cases emit runtime error +trailers. It verifies behavior against a permissive local server, not AWS's +acceptance or client-visible delivery. A real Lambda endpoint might close the +connection itself; the local result cannot establish what the service does. + +No reviewed source explains whether invocation mode, content type or another +service-side rule makes the header/connection requirements optional. That is a +hypothesis to ask AWS about, not an established explanation. + +## Upstream findings and limits + +- [Issue 500](https://github.com/aws/aws-lambda-go/issues/500) explicitly asks about + the missing mode header. Its collaborator responses and eventual closure + address reader/writer API design, not that protocol question. +- [Issue 565](https://github.com/aws/aws-lambda-go/issues/565#issuecomment-2221292052) + contains a collaborator's working Function URL example on provided.al2023. + This supports Go streaming generally but is not a test of our REST integration. +- [PR 603 review](https://github.com/aws/aws-lambda-go/pull/603#discussion_r2593680120) + resolves a default-status detail. It does not discuss the mode header or TCP + lifetime, so it cannot settle those questions. +- [Issue 610](https://github.com/aws/aws-lambda-go/issues/610), updated in September + 2026, concerns prelude and mixed-output expectations. Its comments do not + provide a maintainer resolution of this transport discrepancy. +- The SDK's [errorCapturingReader](https://github.com/aws/aws-lambda-go/blob/v1.55.0/lambda/runtime_api_client.go#L176-L188) + discards `n` when Read returns bytes together with a non-EOF error. The local + unguarded case loses those bytes; edge's existing guard returns bytes first and + the error on the next read. No reviewed issue/release supplied a fix. + +The [Gateway integration guide](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-transfer-mode-lambda.html) +confirms the metadata format and omitted-length fallback, but does not define +the exact byte accounting of its 16KB delimiter constraint. +The [troubleshooting guide](https://docs.aws.amazon.com/apigateway/latest/developerguide/response-streaming-troubleshoot.html) +provides streaming timing log fields and recommends deployed curl tests with +`--no-buffer`. Its console/TestInvokeMethod path buffers responses. It discusses +timeout truncation but does not establish how Lambda runtime error trailers map +to an API Gateway client's response or to failure metrics. + +## Draft for AWS + +Subject: Clarify aws-lambda-go REST response streaming contract and runtime documentation + +We are building a Go 1.27 net/http adapter for API Gateway REST Lambda proxy +response streaming, using the latest stable aws-lambda-go v1.55.0. Our intended +deployment is provided.al2023 with AWS_PROXY, response transfer mode STREAM and +the InvokeWithResponseStream integration URI. We reuse the SDK runtime transport. +We have not deployed this integration yet. + +The SDK documents APIGatewayProxyStreamingResponse for this configuration. +However, its Runtime API client omits Lambda-Runtime-Function-Response-Mode and +does not request TCP connection closure, both required by the custom-runtime +guide. A local Runtime API control using the SDK response type directly confirms +incremental chunked output, no mode header and TCP reuse for the next invocation. + +Could the Lambda runtime/Go SDK and API Gateway teams clarify: + +1. **Supported configuration:** Is unmodified aws-lambda-go v1.55.0 on + provided.al2023, returning APIGatewayProxyStreamingResponse through + lambda.Start, the supported REST streaming path? Does an equivalent io.ReadCloser + with the same content type and wire framing have the same support contract? + Are any additional runtime settings or build requirements necessary? + +2. **Mode header:** Is Lambda-Runtime-Function-Response-Mode: streaming required + for this path? If the SDK may omit it, what precise condition enables + streaming, and can AWS document that exception? If it is required, is this an + SDK defect, and which release will correct it? + +3. **Connection lifetime:** Does the documented connection-close requirement + mean a TCP close, or just completing the chunked request? Is keep-alive reuse + explicitly supported? If the runtime must close TCP, does the Lambda endpoint + enforce closure itself, or must the Go SDK change? + +4. **Midstream failure contract:** After metadata and body bytes reach API + Gateway, how are Lambda-Runtime-Function-Error-Type/Body trailers exposed to + the HTTP client? Does the client see an incomplete/reset transfer, forwarded + trailers, or a normally completed response with the original status? Which + access-log fields and Lambda/API Gateway metrics reliably identify the + failure, especially for an unknown-length response? + +5. **Exact prefix limit:** Does the first 16KB mean 16,000 or 16,384 bytes? Must + the entire eight-NUL delimiter fit within that bound, or only its first byte? + Please specify the maximum accepted byte offset/total prefix length. + +6. **Reader error handling:** Is the SDK's loss of `n > 0` bytes when Read also + returns a non-EOF error a known issue, and is a fix planned? We currently + preserve bytes by returning them before surfacing the error on the next read. + +Questions 1–3 are the immediate deployment-contract qualification. Question 4 +defines reliable production failure detection. Questions 5–6 do not block our +local implementation: we currently cap the entire prefix at 16,000 bytes and +guard the reader error case. + +## Recommendation + +Keep the approved implementation and SDK reuse. The public SDK type and upstream +work strengthen the case that Go REST streaming is intended to work. They do not +justify silently waiving the documented transport mismatch. Retain the deployment +qualification pending AWS's answers or a separately approved deployed test. +No new public API, trust-boundary or error-policy decision is proposed here. diff --git a/internal/streamprobe/runtime_test.go b/internal/streamprobe/runtime_test.go index fe15758..c0ac2f6 100644 --- a/internal/streamprobe/runtime_test.go +++ b/internal/streamprobe/runtime_test.go @@ -113,6 +113,20 @@ func TestRuntimeStreamingProbeProcess(t *testing.T) { runEdgeStream(t, mode) return } + if mode == "sdk_gateway" { + // Exercise the SDK's documented REST response type directly, without + // edge's adapter or prefix encoder, to isolate Runtime API behavior. + lambda.Start(func(ctx context.Context) (*events.APIGatewayProxyStreamingResponse, error) { + url := "http://" + os.Getenv("AWS_LAMBDA_RUNTIME_API") + body := io.MultiReader(strings.NewReader("first\n"), &gatedTail{ctx: ctx, url: url, tail: bytes.NewReader([]byte("last\n"))}) + return &events.APIGatewayProxyStreamingResponse{ + StatusCode: 200, + Headers: map[string]string{"Content-Type": "text/plain"}, + Body: &probeStream{body: body, url: url}, + }, nil + }) + return + } makeStream := func(ctx context.Context) (io.ReadCloser, error) { prefix, err := json.Marshal(struct { StatusCode int `json:"statusCode"` @@ -210,7 +224,7 @@ func runEdgeStream(t *testing.T, mode string) { } func TestSDKStreamsThroughRuntimeAPI(t *testing.T) { - for _, mode := range []string{"raw", "typed", "late_error", "data_and_error", "guarded_data_and_error", "edge_raw", "edge_typed", "edge_late_error", "edge_late_panic"} { + for _, mode := range []string{"raw", "typed", "late_error", "data_and_error", "guarded_data_and_error", "edge_raw", "edge_typed", "edge_late_error", "edge_late_panic", "sdk_gateway"} { t.Run(mode, func(t *testing.T) { ctx, cancel := context.WithTimeout(t.Context(), 15*time.Second) defer cancel() From 96a9552cc64452220d4ffd2cffbdc78df4d206e5 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Sun, 20 Sep 2026 14:29:54 +0100 Subject: [PATCH 38/44] docs: propose consumer-readiness milestone --- docs/decisions/0023-consumer-readiness.md | 278 ++++++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 25 ++ 3 files changed, 304 insertions(+) create mode 100644 docs/decisions/0023-consumer-readiness.md diff --git a/docs/decisions/0023-consumer-readiness.md b/docs/decisions/0023-consumer-readiness.md new file mode 100644 index 0000000..dfeb43c --- /dev/null +++ b/docs/decisions/0023-consumer-readiness.md @@ -0,0 +1,278 @@ +# 0023: Consumer readiness and the next implementation milestone + +Status: proposed on 2026-09-20. The user authorized this design review, not the +dependent public API or release-policy changes below. R1–R6 await review. + +## Outcome and scope + +The public packages already compose into the intended HTTP, authentication and +authorization pipeline. This review found no missing root facade or dispatcher +interface needed to demonstrate that composition. The next useful deliverable +is one complete consumer application with tests and a clear support matrix. + +The significant proposed refinement is to defer exporting `edgetest`. It is in +the original package plan, but no concrete helper API was previously accepted. +Existing identity constructors, standard httptest tools, explicit SDK event +literals and local provider transports already cover the basic test boundaries. +Prove the remaining repetition in the reference application before freezing a +new exported fixture API. + +This is a consumer composition and release-readiness review, not a new security +audit or a claim of deployed AWS interoperability. The Beakley comparison below +uses the nine findings preserved in the implementation plan; it does not assert +an exhaustive new source/API parity review of Beakley. + +## Evidence inspected + +- `authz/example_test.go` composes ActionHeader, Cognito configuration, IAM proof + verification/error mapping, ordered rules and application grant resolution. + Its runnable dispatcher example exercises missing credentials, not both + successful credential paths or Lambda registration. +- `authn/example_test.go` demonstrates a successful synthetic Bearer path and a + rejected proof. `iamproof/example_test.go` demonstrates client proof generation. + Registration and streaming examples live in separate root-package files. +- `authn/gateway_test.go` and `authn/authorization_test.go` exercise production + Cognito signature verification and IAM proof processing with synthetic local + JWKS/STS dependencies across native HTTP, five buffered entry formats and two + REST streaming entry points. This is stronger composition coverage than the + short consumer examples, but it is spread through test-only helpers. +- `authn.Authenticator.Handler` deliberately writes generic text failures. + Custom JSON failures are supported through Authenticate, Error.StatusCode, + Error.Challenges and identity.WithCaller, but the custom-envelope example only + demonstrates its failure branch. A complete success/error pipeline is missing + from the consumer walkthrough. +- `identity.NewJWT`, NewIAM and WithCaller already validate and construct fixture + facts. The adapter and authenticator reject inherited callers. A convenience + fixture must not quietly bypass that boundary or imply cryptographic proof. +- Public root godoc, package imports, README, guides, fixture inventory, current + CI workflow and decisions 0005/0011/0018–0022 were inspected. + +Official documentation and Go skills checked for this review: + +- [AWS HTTP API payload differences](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html), + through public AWS MCP: 2.0 combines repeated header/query values, handles + cookies separately, and omits the custom-domain API mapping from rawPath. +- [REST proxy input/output](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html), + through public AWS MCP: multivalue maps, deployment-dependent requestContext + and distinct IAM/Cognito/custom-authorizer assertions. +- Installed Go 1.27.1 godoc via go-docs for httptest.NewRequestWithContext and + ResponseRecorder.Result. The former invents test host/protocol defaults and + dummy TLS state for HTTPS; the latter is a buffered observation whose body + does not report late streaming failures. Neither simulates API Gateway. +- Go rules for API design, testing, errors, packages and runnable examples, + together with the previously loaded official Go 1.27 release notes. +- [Go release workflow](https://go.dev/doc/modules/release-workflow) and + [module source layout](https://go.dev/doc/modules/managing-source). A v0 + milestone can precede API stability; v1 establishes compatibility commitments. + +## R1. Build one complete reference application with existing APIs + +**Recommendation:** add a reference application under `examples/dispatcher`, +with shared application code internal to that example and small explicit native, +buffered Lambda and REST streaming entry points. Keep it in the root module so +normal tests and builds include it. Add an external-module compile check using +a local replace directive to verify consumption through public imports. + +Use a small orders domain and the configured Action header. The complete +pipeline should be visible in one place: + +```mermaid +flowchart LR + Transport[Native HTTP or edge adapter] --> Routes[Explicit public routes and CORS preflight] + Routes --> Authenticate[Select and verify Bearer or EdgeIAM] + Authenticate --> Caller[Install one caller] + Caller --> Select[Parse action and select registry entry once] + Select --> Target[Resolve and validate resource] + Target --> Authorize[Check identity policy and application grant] + Authorize --> Execute[Execute the same entry and resource] + Execute --> Output[JSON response or explicitly configured REST stream] +``` + +Show actual CognitoVerifier and iamproof.Verifier wiring, including the required +ErrInvalidProof-to-ErrInvalidCredentials mapping. Use Authenticate for consistent +application-owned JSON errors, preserve challenges/no-store/CORS and install the +caller only on success. Keep public routes/preflight explicit. Do not change the +existing convenience middleware's text response policy. + +Use a normal read action across all transports. Register a streaming action only +in the streaming application variant; do not select adapter mode from a header, +detect it with a failed Flush, or imply HTTP APIs support response streaming. +Authorize before the first write/flush. Stop on context/write/flush failures. +Grant storage and action/resource consistency remain application-owned. + +**Alternative:** add an edge dispatcher/resolver/error-rendering framework. +**Reasoning:** the existing public APIs already support the composition, and +decision 0021 deliberately keeps application policy and dispatch with consumers. +**Consequence:** the example contains some explicit wiring; its application +types and configuration are not new module-wide public contracts. + +## R2. Defer exported edgetest helpers until the example proves their value + +**Recommendation:** the next milestone adds no exported edgetest API. Start with +private fixture helpers in the example's tests and a consumer-testing guide. +This is a proposed change to the original sequencing, requiring user approval. + +The smallest test vocabulary currently needed is already available: + +| Test boundary | Existing tools | What the test does not prove | +| --- | --- | --- | +| Business authorization | identity constructors, authz.Request/Rule | Credential verification | +| Handler after authentication | httptest plus explicitly installed fixture caller | Adapter/authn admission of that caller | +| Full authn pipeline | Real verifier plus test-local JWKS/STS transport | AWS acceptance of synthetic credentials | +| Gateway conversion | Explicit SDK events or authored raw JSON and public adapter | Deployed routing/authorizer configuration | +| Incremental stream | Public streaming reader, controlled producer/consumer, Close | Remote client timing or AWS termination semantics | + +For native business-handler tests, use SourceCustomAssertion unless the test +specifically models a different source. Do not inject a caller into the parent +context passed to the adapter or authenticator. Provider stubs may return the +required verified-source fixture only when explicitly labeled as stubs; real +cryptographic/proof tests use the real verifiers with local dependencies. + +Keep distinct private REST, HTTP API 1.0 and HTTP API 2.0 fixtures. SDK V1 lacks +the raw version discriminator, so HTTP 1.0 raw fixtures must explicitly encode +`version: "1.0"` using JSON v2. Do not infer API product from the V1 Go type. +Avoid generic HTTP-request-to-Gateway conversion: an ordinary request lacks +deployment mapping, authorizer output and the original Gateway field choices. + +**Alternatives considered:** exported NewJWT/NewIAM convenience wrappers; +REST/HTTPV1/HTTPV2 JSON-marshaling wrappers; a universal request builder; +response collectors returning a common http.Response. The first two mostly +duplicate existing APIs. The latter two introduce fidelity, error, resource and +ownership policies that have not earned a public compatibility commitment. + +**Reasoning:** tests should expose meaningful protocol differences. A helper +that normalizes away repeated headers, claims fidelity or late read errors can +make incorrect application assumptions appear valid. +**Consequence:** some small explicit fixture construction remains. Revisit +exported helpers after the reference application and an actual consumer identify +a stable repeated operation; review exact signatures and policy at that point. + +## R3. Demonstrate realistic consumer outcomes, preserving fixture provenance + +**Recommendation:** test the reference application's public boundary with both +successful credential kinds, rather than only adding more constructor examples. +Keep the current transport fixtures as independent adapter contract tests. + +Acceptance for the reference application: + +- Both credential kinds can execute the same authorized action/resource. +- Missing/rejected credentials, duplicate Authorization or Action, unknown + action, grant denial, grant outage and cancellation produce the selected safe + outcomes; denied requests never execute. Preserve challenges and CORS. +- Resource resolution/authorization/execution use the same selected target. +- Normal actions work through native HTTP and all supported buffered formats; + streaming examples additionally demonstrate first delivery before completion, + a slow reader, early Close and a late failure after successful authorization. +- Custom consumer headers survive; meaningful multivalue/cookie/binary cases + remain visible instead of being hidden by a universal response normalizer. +- Test dependencies cannot accidentally fall through to live AWS/network calls. + Test keys and STS responses are synthetic and explicitly described that way. + +**Alternative:** export fixture machinery first or treat passing package tests +as a complete consumer walkthrough. **Reasoning:** existing tests establish +composition, but consumers still have to assemble the successful production +configuration and consistent error handling from several files. +**Consequence:** no new authentication guarantees or relaxed source checks. + +## R4. Publish a precise initial support matrix and retain deferred scope + +**Recommendation:** document what is implemented, locally tested, deferred and +deployment-unverified separately. Retain custom Lambda-authorizer mapping, +Gateway metadata accessors, mTLS identity, Function URLs/ALB, generic grant +resolvers and ongoing stream reauthorization outside this initial milestone. +No current required consumer case justifies silently introducing their contracts. + +The root package currently exposes no Gateway request-ID/stage/route accessor. +Decision 0005 explicitly deferred that API. The plan's broad package-responsibility +description must not be read as a claim that every metadata field is available. +Custom authorizers are not native IAM/Cognito/JWT assertions and remain rejected +when encountered by opt-in identity extraction under the accepted rules. + +**Alternative:** expand the next milestone to all deferred integrations. +**Reasoning:** each has independent representation and trust-boundary questions. +**Consequence:** feature exclusions become explicit consumer information, and +any specific unmet consumer requirement returns for its own design review. + +### Original Beakley failure cases traced to current coverage + +| Preserved finding | Implemented response and representative evidence | +| --- | --- | +| Unverified header JWT reconciled with gateway identity | No fallback/mixing; gateway recognition/conflict tests, authn context/source tests, Cognito signature tests | +| Repeated status/header mutation changed committed output | Buffered commitment tests and TestStreamingWriterCommitmentAndSniffing | +| V1 path delimiters interpreted as query/fragment | TestRequestPathsMatchHTTPParsing and decoder fixture rest.json | +| Single-value maps discarded by unrelated multivalue maps | TestRequestMergingAndOwnership and cross-format action-header tests | +| Simultaneous IAM/JWT arms could authorize | Immutable caller union; TestGatewayRecognitionMatrix conflicts; mandatory authz guard | +| Cold JWKS outage caused repeated fetches | TestCognitoColdFetchSharingAndBackoff and cancellation/rotation tests | +| Anonymous fixture blocked later verification | Anonymous WithCaller is a no-op; inherited nonanonymous caller rejects; context and transport tests | +| Broad ARN wildcard matching | TestIAMExactAndRoleFamilies and validated IAM caller forms; no glob matcher | +| Client ID/audience conflation and unrestricted clients | TestCognitoClientAudienceAndClaims and required explicit client configuration | + +These map the recorded defects to contracts/tests, not to a claim that AWS +produces every synthetic fixture or that Beakley's entire API was copied. + +## R5. Complete release engineering before a first v0 milestone + +**Recommendation:** prepare a v0 consumer milestone after the example and review +are complete. Do not claim v1 stability yet. Keep the streaming deployment +qualification in force and track real Cognito/STS interoperability separately. +Authoritative AWS answers or an approved live fixture are still needed for the +streaming transport questions; local tests cannot close that item. + +Current readiness observations: + +- Five public packages exist: edge, identity, authn, authz and iamproof. + Identity remains standard-library-only; authn/authz depend on identity and the + standard library. AWS dependencies stay in transport and iamproof. +- The identity package's encoding/json import is only for Number compatibility; + module-owned JSON processing remains direct JSON v2/jsontext. +- CI defines Go 1.27.0/latest 1.27 tests/race/vet and Linux arm64/amd64 builds. + Referenced checkout@v7 and setup-go@v7 tags resolve. No GitHub CI result is + recorded for this working branch; local success does not establish it. +- The historical nested `docs/probes/iamproof` module is excluded from root + `go test ./...`. Label it historical or run it explicitly when relying on its + evidence. Keep new runnable consumer examples in the root module. +- Initial buffered allocation baselines exist. Add a streaming baseline that + consumes incrementally and excludes fixture setup; record environment and + allocations, without inventing performance thresholds or Lambda latency claims. +- All direct AWS modules are still latest stable on 2026-09-20. Indirect + smithy-go v1.28.2 is now available, released 2026-09-18, versus required v1.28.1. + Its [upstream comparison](https://github.com/aws/smithy-go/compare/v1.28.1...v1.28.2) + includes HTTP path/host and event-stream/decoder changes. Recommend an isolated + patch update with IAM-proof contract tests, full race/vet and target builds. + This is not a fix for aws-lambda-go's separate streaming transport. +- A newer testify appears in the module graph through upstream SDK tests, not + this module's production imports. Do not blanket-update all listed modules. + +**Alternative:** declare release readiness from local unit tests alone. +**Reasoning:** the reference application, CI execution and accurate support +claims validate different consumer concerns. **Consequence:** prepare these +artifacts locally; pushing, tagging, publishing and deploying are separate actions. + +## R6. Obtain the owner's distribution/license choice before publication + +There is no tracked LICENSE/COPYING/NOTICE file in this repository. No existing +organization license was supplied by the task, and public/private repository +visibility was not investigated by this review. + +**Recommendation:** use the owner's existing organization-standard license if +one applies, with the exact license and copyright holder confirmed by the owner. +Otherwise defer external publication until that choice is made. Do not infer a +license from dependencies or Beakley, and do not change repository visibility. + +**Alternative:** choose a license during technical implementation. +**Reasoning:** this is an ownership/distribution decision, not an adapter API +default. **Consequence:** it does not block the example or local validation. +The exact license remains an open owner question, even if R1–R5 are approved. + +## Validation and next execution order + +On Go 1.27.1 darwin/arm64, the current implementation passes `go test -race ./...`, +`go vet ./...`, `go mod verify` and CGO-disabled Linux arm64/amd64 builds. +This review changes documentation only. It does not implement a new package, +update dependencies, publish a release or contact AWS. + +After approval: build/test the reference application and consumer-testing guide; +reconcile support/status documentation; make the isolated Smithy patch update; +record streaming baselines and prepare CI/release evidence. Resolve any new +public API or trust-boundary decision before implementing it. The existing AWS +questions can proceed independently through the user's outreach. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 28f68de..3c4f141 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -31,3 +31,4 @@ Keep related decisions small enough to review and implement independently. | [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | | [0022: Streaming writer review](0022-streaming-writer-review.md) | Accepted; writer, public entry points and local validation complete; SDK deployment gate retained | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | +| [0023: Consumer readiness](0023-consumer-readiness.md) | Proposed; R1–R6 await review | What complete consumer example, test support, deferred scope and release preparation should follow the implemented packages? | diff --git a/docs/plan.md b/docs/plan.md index b8b2dd4..d99400b 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -411,6 +411,31 @@ responses and cancellation. No live AWS or GitHub CI run occurred. Acceptance: examples work both in Lambda and ordinary HTTP applications, while authorization decisions remain inspectable and deny by default. +### 6. Consumer readiness and release preparation — proposed + +The user authorized a consumer-readiness design review on 2026-09-20. +[Decision 0023](decisions/0023-consumer-readiness.md) records the code/documentation +evidence, the Beakley failure-case coverage matrix and recommendations R1–R6. +The recommendations await approval; the earlier `edgetest` package direction is +not silently replaced by the proposed deferral. + +- [x] Review public composition, existing examples, fixture provenance, deferred + features, package dependencies and CI/release readiness. +- [ ] Review R1–R6: complete reference application; defer edgetest exports until + concrete use; consumer outcome tests; precise support matrix; v0 release + preparation; owner-selected distribution/license policy. +- [ ] After approval, implement the reference application and consumer-testing + guide using the existing public APIs, with local provider fixtures. +- [ ] After approval, reconcile support/status documentation, update the indirect + Smithy patch in isolation, and record incremental streaming baselines. +- [ ] Obtain actual GitHub CI evidence and resolve publication prerequisites + before a release. No push, tag, publication or deployment was authorized by + this review. Streaming deployment and live identity interoperability remain + separate qualifications. + +The review's fresh local race tests, vet, module verification and Linux +arm64/amd64 builds pass on Go 1.27.1. No runtime behavior changed. + ## Findings carried forward from the Beakley review These are requirements to test independently, not instructions to preserve old From dd9fd806c077dac060a91a796595a38b3603e2c5 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 21 Sep 2026 11:04:36 +0100 Subject: [PATCH 39/44] deps: update smithy-go to v1.28.2 --- go.mod | 2 +- go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 4a25dfc..ed4819a 100644 --- a/go.mod +++ b/go.mod @@ -14,5 +14,5 @@ require ( github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.3 // indirect github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 // indirect github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 // indirect - github.com/aws/smithy-go v1.28.1 // indirect + github.com/aws/smithy-go v1.28.2 // indirect ) diff --git a/go.sum b/go.sum index 3e12e6a..240dc24 100644 --- a/go.sum +++ b/go.sum @@ -14,8 +14,8 @@ github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3 h1:bON1rJf67 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.3/go.mod h1:c5BBpjJcQXpfeq9iASyVKA3T6vX6B6LEXY4mL/gklDY= github.com/aws/aws-sdk-go-v2/service/sts v1.51.0 h1:Zpnqa6XtrNzXZnwbdCqHOXpXhMsa01ql/pcRQ1sb4hk= github.com/aws/aws-sdk-go-v2/service/sts v1.51.0/go.mod h1:/8JRcdTt//hG0Q4BTmGbuOplT7ABe+5rdtqUHqXvYIM= -github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ= -github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= +github.com/aws/smithy-go v1.28.2 h1:myhcykQcatTul2B/zITjDk203G7t0awUAs1hVry5Bvg= +github.com/aws/smithy-go v1.28.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= From 230367c0d3e0b9d9ae6173433f799672b501cde8 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 21 Sep 2026 11:05:00 +0100 Subject: [PATCH 40/44] feat: add complete dispatcher consumer example --- .github/workflows/go.yml | 4 + README.md | 6 + docs/benchmarks.md | 34 +++ docs/consumer-testing.md | 51 ++++ docs/decisions/0023-consumer-readiness.md | 50 +++- docs/decisions/README.md | 2 +- docs/plan.md | 33 ++- docs/probes/iamproof/README.md | 7 +- docs/release-readiness.md | 34 +++ docs/support.md | 41 +++ examples/dispatcher/README.md | 113 +++++++ examples/dispatcher/buffered/main.go | 28 ++ examples/dispatcher/internal/orders/app.go | 276 ++++++++++++++++++ .../dispatcher/internal/orders/app_test.go | 270 +++++++++++++++++ .../dispatcher/internal/orders/environment.go | 93 ++++++ .../internal/orders/environment_test.go | 61 ++++ .../internal/orders/fixtures_test.go | 260 +++++++++++++++++ .../dispatcher/internal/orders/stream_test.go | 171 +++++++++++ examples/dispatcher/native/main.go | 25 ++ examples/dispatcher/streaming/main.go | 34 +++ scripts/check-consumer.sh | 12 + streaming_bench_test.go | 93 ++++++ testdata/consumer/consumer_test.go.txt | 82 ++++++ 23 files changed, 1757 insertions(+), 23 deletions(-) create mode 100644 docs/consumer-testing.md create mode 100644 docs/release-readiness.md create mode 100644 docs/support.md create mode 100644 examples/dispatcher/README.md create mode 100644 examples/dispatcher/buffered/main.go create mode 100644 examples/dispatcher/internal/orders/app.go create mode 100644 examples/dispatcher/internal/orders/app_test.go create mode 100644 examples/dispatcher/internal/orders/environment.go create mode 100644 examples/dispatcher/internal/orders/environment_test.go create mode 100644 examples/dispatcher/internal/orders/fixtures_test.go create mode 100644 examples/dispatcher/internal/orders/stream_test.go create mode 100644 examples/dispatcher/native/main.go create mode 100644 examples/dispatcher/streaming/main.go create mode 100644 scripts/check-consumer.sh create mode 100644 streaming_bench_test.go create mode 100644 testdata/consumer/consumer_test.go.txt diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 8939ae9..03e3401 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -27,6 +27,10 @@ jobs: run: go test -race ./... - name: Vet run: go vet ./... + - name: Verify modules + run: go mod verify + - name: Check external consumer + run: sh scripts/check-consumer.sh lambda-build: runs-on: ubuntu-latest diff --git a/README.md b/README.md index e79d365..e19c443 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,12 @@ HTTP API payload 1.0 and HTTP API payload 2.0. It uses AWS Lambda Go v1.55.0 typ and direct encoding/json/v2 processing. Raw and already typed entry points share HTTP conversion, response capture, identity policy, cancellation and cleanup. +Start with the [complete orders dispatcher](examples/dispatcher/README.md) for +verifier wiring, application grants, JSON errors and three transport entry points. +The [support matrix](docs/support.md) distinguishes implemented behavior, deferred +scope and deployment qualifications; the [testing guide](docs/consumer-testing.md) +shows how to test each consumer boundary. + Create an adapter with `edge.New(handler, options...)`, handle the constructor error, then register it with `lambda.Start(adapter)`. Pass the object, not `adapter.Invoke`, to retain the raw JSON v2 boundary. For a fixed payload family, diff --git a/docs/benchmarks.md b/docs/benchmarks.md index c6e2f10..1c6dc4c 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -234,3 +234,37 @@ timings vary with GC and host load; comparisons should use longer repeated runs. Initial three-iteration measurements included noticeable cold codec setup costs; the table instead uses the repeated 100 ms runs above. Native identity costs are additional and depend on claim shape and source fidelity. + +## Public incremental streaming (2026-09-21) + +Go 1.27.1, darwin/arm64, Apple M2, default GOMAXPROCS=8. Three 100 ms samples per +case exercise the public REST adapter, including stream completion and Close. +The application reuses a chunk of at most 4 KiB, flushing after each write. The +consumer reuses an 8 KiB buffer and reads incrementally; it never collects the +body. Adapter/event construction, raw fixture encoding and reusable buffers are +outside the timed loop. Gateway identity and local authentication are disabled. + +```sh +go test -run '^$' -bench '^BenchmarkStreamingPublic$' -benchmem -benchtime=100ms -count=3 . +``` + +Medians across the three samples: + +| Response body | Raw REST B/op | Raw allocs/op | Typed V1 B/op | Typed allocs/op | +| --- | ---: | ---: | ---: | ---: | +| 128 bytes | 7,876 | 80 | 6,116 | 56 | +| 64 KiB | 7,905 | 80 | 6,117 | 56 | +| 5 MiB | 7,877 | 80 | 6,124 | 56 | + +Raw calls include input decoding; typed calls exclude upstream input decoding. +Both include metadata serialization, the producer/reader bridge and cleanup. +The roughly constant allocated bytes in this fixture are evidence that the +adapter does not accumulate its response body. They do not establish a universal +heap bound: retained application data, metadata, request bodies, authentication +and concurrency add costs. One 5 MiB raw sample measured 8,170 B/op and 81 +allocations, illustrating short-run scheduling/allocation noise. + +These are cumulative allocations, not peak memory or RSS, and not an SDK Runtime +API or deployed Lambda benchmark. The buffered body-echo fixtures above also +carry large request bodies; do not treat these as a controlled timing comparison +between modes. No performance thresholds or AWS bandwidth claims are inferred. diff --git a/docs/consumer-testing.md b/docs/consumer-testing.md new file mode 100644 index 0000000..0ff802a --- /dev/null +++ b/docs/consumer-testing.md @@ -0,0 +1,51 @@ +# Testing an edge consumer + +Keep each test's trust boundary explicit. Identity constructors validate facts, +not credentials. Gateway fixtures describe shapes, not deployed authorizers. + +| Boundary | Tools | Evidence | +| --- | --- | --- | +| Business rule | Identity constructors and `authz.Request` | Exact action/resource policy outcomes | +| Authenticated handler | `httptest`, `identity.WithCaller` | Behavior after admission | +| Authentication | Real verifier with local JWKS/STS transport | Local signature/profile/proof processing | +| Gateway conversion | Explicit SDK events or raw JSON v2 | Chosen payload format's behavior | +| Stream lifecycle | Public reader, controlled producer, incremental reads | Delivery, backpressure, cancellation and errors | +| External module | Local replace and public imports | Package and registration boundaries | + +For business-handler fixtures, prefer `SourceCustomAssertion` unless modeling a +different source deliberately. Use issuer-qualified JWT claims and valid IAM +caller ARNs, checking all constructor/context errors. Never inject a caller into +an adapter's or authenticator's parent context: both reject inherited callers. +An explicitly labeled verifier stub must return its contract's required source; +the source label alone is not cryptographic evidence. + +For authentication tests, use real verifiers, locally signed JWTs and explicit +local JWKS transports. Generate IAM proofs with synthetic credentials and return +synthetic STS responses. Keep rejection distinct from dependency outage. An +unexpected test URL must fail, never fall through to a network transport. The +production `iamproof` suite separately checks reconstruction/signatures; a canned +STS response cannot establish AWS acceptance. + +Keep REST, HTTP API 1.0 and HTTP API 2.0 fixtures distinct. The shared SDK V1 type +does not include the raw HTTP 1.0 discriminator: explicitly encode `version: "1.0"`. +V2 combines repeated header/query values and represents cookies separately. Do +not infer a lossless Gateway event from an ordinary request, or split all commas +into header values. See the [fixture inventory](gateway-fixtures.md). + +`httptest.ResponseRecorder` is useful for small native outcomes. It does not +reproduce AWS mapping, network cancellation, incremental delivery or terminal +stream errors. Go 1.27's `httptest.NewTestServer(t, handler)` uses an in-memory +network and can participate in `testing/synctest` without wall-clock sleeps. + +Always close a returned stream. Read metadata through its eight-NUL delimiter, +then consume body chunks incrementally and inspect the terminal read error. +Synchronize with a controlled producer to assert delivery before completion, +block consumption to prove backpressure, and close early to prove producer exit. +A generic response collector or final-body assertion cannot establish these. + +The [dispatcher](../examples/dispatcher/README.md) demonstrates these boundaries +with private helpers. Small-response assertions retain V2 header/cookie differences; +separate tests exercise streaming lifecycle. `sh scripts/check-consumer.sh` runs +a temporary consumer module with public imports and a local replace; CI runs it +explicitly. There is no exported `edgetest` package: decision 0023 defers it until +actual consumers establish useful stable operations beyond existing tools. diff --git a/docs/decisions/0023-consumer-readiness.md b/docs/decisions/0023-consumer-readiness.md index dfeb43c..f24d856 100644 --- a/docs/decisions/0023-consumer-readiness.md +++ b/docs/decisions/0023-consumer-readiness.md @@ -1,7 +1,8 @@ # 0023: Consumer readiness and the next implementation milestone -Status: proposed on 2026-09-20. The user authorized this design review, not the -dependent public API or release-policy changes below. R1–R6 await review. +Status: R1–R5 accepted on 2026-09-20; local implementation and validation completed +on 2026-09-21. External release prerequisites remain open. R6 remains an owner +choice; no license, publication, push, tag or deployment is authorized. ## Outcome and scope @@ -10,7 +11,7 @@ authorization pipeline. This review found no missing root facade or dispatcher interface needed to demonstrate that composition. The next useful deliverable is one complete consumer application with tests and a clear support matrix. -The significant proposed refinement is to defer exporting `edgetest`. It is in +The accepted refinement is to defer exporting `edgetest`. It was in the original package plan, but no concrete helper API was previously accepted. Existing identity constructors, standard httptest tools, explicit SDK event literals and local provider transports already cover the basic test boundaries. @@ -110,7 +111,7 @@ types and configuration are not new module-wide public contracts. **Recommendation:** the next milestone adds no exported edgetest API. Start with private fixture helpers in the example's tests and a consumer-testing guide. -This is a proposed change to the original sequencing, requiring user approval. +The user approved this change to the original sequencing on 2026-09-20. The smallest test vocabulary currently needed is already available: @@ -264,15 +265,50 @@ license from dependencies or Beakley, and do not change repository visibility. default. **Consequence:** it does not block the example or local validation. The exact license remains an open owner question, even if R1–R5 are approved. -## Validation and next execution order +## Original review validation and accepted execution order On Go 1.27.1 darwin/arm64, the current implementation passes `go test -race ./...`, `go vet ./...`, `go mod verify` and CGO-disabled Linux arm64/amd64 builds. -This review changes documentation only. It does not implement a new package, -update dependencies, publish a release or contact AWS. +The original review changed documentation only. It did not implement a new +package, update dependencies, publish a release or contact AWS. After approval: build/test the reference application and consumer-testing guide; reconcile support/status documentation; make the isolated Smithy patch update; record streaming baselines and prepare CI/release evidence. Resolve any new public API or trust-boundary decision before implementing it. The existing AWS questions can proceed independently through the user's outreach. + + +## Implementation and validation (2026-09-21) + +R1–R5's local work is complete. The [reference application](../../examples/dispatcher/README.md) +provides shared internal orders code and native/buffered/REST-streaming mains. +It composes production verifiers, explicit IAM error mapping, one caller/action/ +resource, application enrollment and consistent JSON failures. No library public +API was added. The finite stream uses Go's documented http.ErrAbortHandler for +post-commit producer failures, which edge converts through its existing ErrStream +contract; it does not expose dependency diagnostics or invent replacement status. + +The example tests both credentials across eight transport entry paths, checking +repeated headers, separate V2 cookies, grant failures, cancellation and no execution +on denial. Local RSA/JWKS and synthetic STS transports cannot fall through to the +network. Separate synctest cases prove delivery before completion, slow-reader +backpressure, early Close and late failure. The actual demo enrollment policy is +also tested: exact issuer/subject or full IAM ARN, action and target all matter. + +The [consumer-testing guide](../consumer-testing.md), [support matrix](../support.md) +and [release checklist](../release-readiness.md) distinguish these local contracts +from deployment evidence. Exported edgetest remains deferred. The external-module +check uses only public imports and local replace; CI now runs it and module +verification. Historical nested IAM research is explicitly labeled as such. + +The approved Smithy v1.28.2 patch changes only its module requirement/checksums. +Production IAM-proof tests pass with that graph. [Streaming allocation baselines](../benchmarks.md#public-incremental-streaming-2026-09-21) +cover incremental raw/typed reads with setup excluded and no performance threshold. + +Full race tests, vet, module verification, formatting and CGO-disabled Linux +arm64/amd64 builds pass on Go 1.27.1; the external-module check also passes. +GitHub returned no workflow runs for this branch. A passing release-commit CI run, +the owner's license choice and existing AWS interoperability qualifications remain +open. Nothing was pushed, published or deployed. The user's AWS outreach can +proceed independently. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 3c4f141..f61339f 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -31,4 +31,4 @@ Keep related decisions small enough to review and implement independently. | [0020: Cognito verification](0020-cognito-verification.md) | Accepted; verifier, cache and local validation implemented | How should direct JSON v2 access-token verification, client/resource restrictions and bounded JWKS caching compose with authn? | | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | | [0022: Streaming writer review](0022-streaming-writer-review.md) | Accepted; writer, public entry points and local validation complete; SDK deployment gate retained | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | -| [0023: Consumer readiness](0023-consumer-readiness.md) | Proposed; R1–R6 await review | What complete consumer example, test support, deferred scope and release preparation should follow the implemented packages? | +| [0023: Consumer readiness](0023-consumer-readiness.md) | R1–R5 accepted; local implementation complete; external release prerequisites open | What complete consumer example, test support, deferred scope and release preparation should follow the implemented packages? | diff --git a/docs/plan.md b/docs/plan.md index d99400b..4795be4 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -71,12 +71,12 @@ rules and the consumer-owned resolution/dispatch boundary. | Package | Responsibility | | --- | --- | -| `edge` (root) | Invocation adapter, payload decoding, HTTP translation, gateway metadata, optional gateway identity extraction | +| `edge` (root) | Invocation adapter, payload decoding, HTTP translation and optional gateway identity; metadata accessors deferred | | `identity` | Validated callers, claim representation, context transport; standard library only | | `authn` | Explicit IAM-proof/OAuth selection, Cognito verification and JWKS caching | | `iamproof` | Exported client proof generator and bounded online STS verifier | | `authz` | Exact identity predicates, ordered rules and context-aware application checks; consumers own grant resolution and HTTP dispatch | -| `edgetest` | Consumer identity fixtures and gateway event helpers | +| `edgetest` (deferred) | Revisit exported fixtures after concrete consumer needs; decision 0023 | Avoid a broad root-package facade that reexports every companion type. Keep AWS event types out of transport-independent packages. Reassess any additional @@ -411,30 +411,39 @@ responses and cancellation. No live AWS or GitHub CI run occurred. Acceptance: examples work both in Lambda and ordinary HTTP applications, while authorization decisions remain inspectable and deny by default. -### 6. Consumer readiness and release preparation — proposed +### 6. Consumer readiness and release preparation — local work complete The user authorized a consumer-readiness design review on 2026-09-20. [Decision 0023](decisions/0023-consumer-readiness.md) records the code/documentation evidence, the Beakley failure-case coverage matrix and recommendations R1–R6. -The recommendations await approval; the earlier `edgetest` package direction is -not silently replaced by the proposed deferral. +The user approved R1–R5 on 2026-09-20, including deferring exported `edgetest` +helpers. The license/copyright choice remains open and does not block local work. - [x] Review public composition, existing examples, fixture provenance, deferred features, package dependencies and CI/release readiness. -- [ ] Review R1–R6: complete reference application; defer edgetest exports until - concrete use; consumer outcome tests; precise support matrix; v0 release - preparation; owner-selected distribution/license policy. -- [ ] After approval, implement the reference application and consumer-testing +- [x] Approve R1–R5: complete reference application, deferred edgetest exports, + consumer outcomes, precise support matrix and v0 preparation. The license + choice remains open. +- [x] Implement the reference application and consumer-testing guide using the existing public APIs, with local provider fixtures. -- [ ] After approval, reconcile support/status documentation, update the indirect +- [x] Reconcile support/status documentation, update the indirect Smithy patch in isolation, and record incremental streaming baselines. - [ ] Obtain actual GitHub CI evidence and resolve publication prerequisites before a release. No push, tag, publication or deployment was authorized by this review. Streaming deployment and live identity interoperability remain separate qualifications. -The review's fresh local race tests, vet, module verification and Linux -arm64/amd64 builds pass on Go 1.27.1. No runtime behavior changed. +The [complete dispatcher](../examples/dispatcher/README.md), [testing guide](consumer-testing.md), +[support matrix](support.md) and [release checklist](release-readiness.md) are +implemented. Local RSA/JWKS and synthetic STS fixtures exercise both credentials; +streaming lifecycle tests consume incrementally. The external-module check and +CI wiring are present. Smithy is updated to v1.28.2, and streaming allocation +baselines are recorded. No library API or transport policy changed. + +Full local race tests, vet, module verification, formatting, external consumption +and Linux arm64/amd64 builds pass on Go 1.27.1 (2026-09-21). GitHub has no runs for +this branch; its actual CI evidence, the license and deployment qualifications +remain open. No push, publication or AWS deployment occurred. ## Findings carried forward from the Beakley review diff --git a/docs/probes/iamproof/README.md b/docs/probes/iamproof/README.md index e108203..fbde1a8 100644 --- a/docs/probes/iamproof/README.md +++ b/docs/probes/iamproof/README.md @@ -1,7 +1,8 @@ -# IAM proof design probe +# Historical IAM proof design probe -This standalone Go 1.27 module is research for proposed decision 0017, not a -production helper/verifier. It uses synthetic credentials and makes no AWS calls. +This standalone Go 1.27 module preserves research before decision 0017's +acceptance and implementation. Current production code/tests live in the root +module's `iamproof` package. This probe uses synthetic credentials, with no AWS calls. Its SDK requirements do not change the parent module's dependencies. Root `go test ./...` and current CI do not traverse this nested module. diff --git a/docs/release-readiness.md b/docs/release-readiness.md new file mode 100644 index 0000000..2162e17 --- /dev/null +++ b/docs/release-readiness.md @@ -0,0 +1,34 @@ +# First v0 milestone + +Decision 0023 accepts preparing a consumer milestone before a v1 compatibility +commitment. This checklist records preparation, not authorization to publish: + +- Complete application with native/buffered/streaming registration. +- Local tests for both credentials, grant outcomes and streaming lifecycle. +- External-module check using public imports. +- Accurate [support matrix](support.md), [testing guide](consumer-testing.md) and + [allocation baselines](benchmarks.md). +- Reproducible SDK versions and an isolated Smithy patch update. +- Race tests, vet, module verification, formatting and Linux arm64/amd64 builds. + +## Outstanding evidence and owner choices + +GitHub returned no runs for `work/edge-foundation` during this milestone. The +workflow covers Go 1.27.0/latest 1.27, race tests, vet, module verification, +external consumer checks and both Linux architectures. Obtain a passing run for +the actual release commit after an authorized push. Local tests do not establish +the Go 1.27.0 matrix result. + +The owner's license and copyright holder remain undecided. No license, repository +visibility change, push, tag or publication has been performed in this milestone. +Resolve the license before external publication. + +REST streaming's SDK deployment qualification remains open: local probes cannot +establish AWS acceptance of the observed mode-header/connection behavior. +Authoritative clarification or a separately approved live test is still needed. +Live Cognito/STS interoperability is also unverified; local fixtures are synthetic. +No cloud resources were deployed. + +The nested `docs/probes/iamproof` module is historical research. Root tests and +CI do not traverse it. Run its tests explicitly if relying on new probe evidence; +current production proof tests live in the root module's `iamproof` package. diff --git a/docs/support.md b/docs/support.md new file mode 100644 index 0000000..b9a78e8 --- /dev/null +++ b/docs/support.md @@ -0,0 +1,41 @@ +# Support and evidence + +The module targets Go 1.27, processing module-owned JSON directly with JSON v2 +and jsontext. Its five public libraries are `edge`, `identity`, `authn`, `authz` +and `iamproof`. The dispatcher's internal application and commands are examples. + +| Capability | Implementation | Evidence / qualification | +| --- | --- | --- | +| Buffered REST proxy | Raw object Invoke; typed HandleV1 | Local contracts and SDK registration | +| Buffered HTTP API 1.0 | Raw version 1.0; typed HandleV1 | Shared SDK type cannot identify API product | +| Buffered HTTP API 2.0 | Raw version 2.0; typed HandleV2 | Combined headers/query values cannot be recovered; separate cookies | +| REST streaming | NewStreaming, raw Handle, typed HandleV1 | Local reader/SDK tests pass; AWS deployment qualification open | +| Custom headers/actions | Ordinary HTTP middleware, ActionHeader | Strict selection tested across transports | +| Gateway identity | Opt-in native IAM/JWT/Cognito mapping | Synthetic recognition/fidelity tests; no credential re-verification | +| Local OAuth | Cognito RS256 access-token verifier | Local signatures, independent vector, cache/rotation/outage tests; no live Cognito run | +| Local IAM | Signed GetCallerIdentity proof generator and verifier | Local protocol/HTTP contracts; no live STS run; not raw execute-api SigV4 validation | +| Mixed authentication | Bearer or EdgeIAM on one route | Both paths locally tested; AWS_IAM must not preempt this route | +| Authorization | Exact predicates, ordered rules, grant callbacks | Deny-by-default, outage/cancellation and dispatch consistency | +| Consumer composition | Complete orders example, three mains | Local application matrix and external-module check | + +Gateway identity is disabled by default. Enabling it is a deployment trust choice; +custom authorizer output is not a native IAM/JWT assertion. Flattened claims +cannot recover original arrays or exact numbers. Typed SDK inputs inherit their +upstream decoding fidelity. See [fixture qualifications](gateway-fixtures.md). + +REST buffered binary-media configuration remains the application's deployment +responsibility. Streaming emits raw bytes and does not imply HTTP API streaming +support. The SDK mode-header/connection questions remain in the +[streaming guide](streaming.md) and [AWS follow-up](reviews/2026-09-17-sdk-streaming-follow-up.md). + +Deferred: custom Lambda-authorizer identity mappers; Gateway request-ID, stage +and route metadata accessors; mTLS identity; Function URLs and ALB; generic grant +resolvers; ongoing stream reauthorization; exported `edgetest` helpers. The root +package currently has no Gateway metadata accessor. Broad original package +responsibilities are not a claim that those APIs exist. + +Beakley supplied requirements/failure cases, not copied code or compatibility +targets. [Decision 0023](decisions/0023-consumer-readiness.md#original-beakley-failure-cases-traced-to-current-coverage) +maps its nine recorded defects to current tests; this is not an exhaustive API +parity audit. Local success differs from deployed interoperability and a GitHub +CI result. The [release checklist](release-readiness.md) tracks remaining items. diff --git a/examples/dispatcher/README.md b/examples/dispatcher/README.md new file mode 100644 index 0000000..f9cfc3a --- /dev/null +++ b/examples/dispatcher/README.md @@ -0,0 +1,113 @@ +# Orders dispatcher + +This consumer application uses edge's existing public APIs. The code in +`internal/orders` is example application code, not a new edge API. It configures +real Cognito and IAM-proof verification, application grants and a tiny read-only +order store. + +| Entry point | Registration | Actions | +| --- | --- | --- | +| `native` | HTTP on `127.0.0.1:8080` | `orders.read` | +| `buffered` | `lambda.Start(adapter)` | `orders.read` | +| `streaming` | `lambda.Start(adapter.Handle)` | `orders.read`, `orders.watch` | + +Streaming is for REST API only and retains the [AWS deployment qualification](../../docs/streaming.md). +An Action header cannot change the transport selected at startup. For typed +inputs, register buffered `HandleV1`/`HandleV2` or streaming `HandleV1` instead. +The SDK then owns input decoding; edge skips an envelope JSON round trip. + +## Configuration + +Set all eight variables explicitly; there are no credential or enrollment defaults: + +| Variable | Meaning | +| --- | --- | +| `COGNITO_ISSUER` | Exact trusted HTTPS issuer including pool path | +| `COGNITO_CLIENT_ID` | Allowed access-token client | +| `COGNITO_AUDIENCE` | Required resource-bound audience for this client | +| `IAM_PROOF_REGION` | Regional STS endpoint, for example `eu-west-2` | +| `IAM_PROOF_AUDIENCE` | Audience shared with the proof generator | +| `ORDER_JWT_SUBJECT` | Exact subject enrolled under this issuer | +| `ORDER_IAM_PRINCIPAL` | Exact full caller ARN enrolled for this example | +| `CORS_ORIGIN` | Exact browser origin without a trailing slash | + +The example accepts resource-bound Cognito access tokens with `orders.read` +scope; it does not configure the distinct unbound M2M profile. Both actions +require read permission plus an application grant for their exact action and +target. An IAM assumed-role session ARN enrolls that exact session, not every +session of the role. Replace the demonstration grant store with application +enrollment and resource rules before expanding the data set. + +After setting configuration, run from the repository root: + +```sh +go run ./examples/dispatcher/native +``` + +`GET /healthz` is public. The protected endpoint is `GET /tenants/42/orders/7`, +with `Action: orders.read` and either `Authorization: Bearer ` or +`Authorization: EdgeIAM `. The streaming variant also accepts +`Action: orders.watch`, emitting two finite NDJSON records. Construction does no +network I/O; credential-bearing requests use the real configured JWKS/STS +endpoints. These commands are not offline simulators. + +Generate a proof with `iamproof.NewGenerator(region, audience, credentials)` and +`Generate(ctx)`, then set Authorization to `"EdgeIAM " + token.Value()`. Supply +the AWS credentials provider already configured by your client application. +See the [helper example](../../iamproof/example_test.go) and +[protocol guide](../../docs/iamproof.md). The server needs no AWS credentials to +forward the signed proof. Do not configure AWS_IAM authorization on this mixed +credential route; the Lambda mains leave gateway identity disabled. + +Build a Lambda binary without deploying it: + +```sh +CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o /tmp/orders-bootstrap ./examples/dispatcher/buffered +``` + +Use the `streaming` directory for a separately configured REST streaming +integration. The examples provision no routes, authorizers or AWS resources. + +## Request flow + +Public health and narrowly scoped CORS preflight are explicit. Protected +requests must use GET and a valid order path. The dispatcher authenticates once, +installs the caller, parses Action once, selects one registry entry, checks its +identity policy and application grant, then executes the same entry and target. +The demo data is immutable; a mutable store must preserve object/version +consistency between authorization and execution, for example with a transaction. + +The application uses `Authenticate` to own its JSON error envelope. It preserves +challenges, no-store and CORS. Missing/rejected credentials yield 401; malformed +credentials/actions 400 (oversized credentials 431); unknown actions 404; denied +grants 403; unavailable dependencies 503. Cancellation prevents execution, but +the transport may terminate instead of delivering the attempted 503. + +Application-owned CORS permits the configured origin, GET, and Authorization, +Action and Custom-Trace headers. Avoid a competing Gateway CORS policy. CORS is +not authentication: direct clients still need credentials and grants. Additional +consumer middleware uses ordinary `http.Handler` and `r.Header`. + +`orders.watch` authorizes before flushing and stops on context/write/flush +failures. A later producer failure uses Go's documented `http.ErrAbortHandler`; +edge converts it to a sanitized terminal stream error. It cannot rewrite the +committed status or earlier records. Abrupt termination is not a complete result; +replay/resume protocols belong to the application. The Lambda main returns the +reader directly and logs only edge's sanitized stream diagnostic, never tokens. + +## Local tests + +```sh +go test -race ./examples/dispatcher/... +sh scripts/check-consumer.sh +``` + +Tests sign synthetic access tokens locally and run production verifiers with +closed local JWKS/STS transports. A synthetic STS response does not prove AWS +accepted a signature. No test falls through to a live provider. Fixtures retain +REST/HTTP API distinctions, V2 combined headers and separate response cookies. + +Streaming tests consume incrementally and use `testing/synctest` to prove first +delivery before completion, slow-reader backpressure, early Close and late errors. +Small JSON/error tests collect bodies only for assertions. See the +[consumer testing guide](../../docs/consumer-testing.md). diff --git a/examples/dispatcher/buffered/main.go b/examples/dispatcher/buffered/main.go new file mode 100644 index 0000000..a2b738c --- /dev/null +++ b/examples/dispatcher/buffered/main.go @@ -0,0 +1,28 @@ +// Command buffered registers the orders application for REST or HTTP API events. +package main + +import ( + "log/slog" + "os" + + "github.com/aws/aws-lambda-go/lambda" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/examples/dispatcher/internal/orders" +) + +func main() { + h, err := orders.FromEnvironment(false) + if err != nil { + slog.Error("configure orders", "error", err) + os.Exit(1) + } + adapter, err := edge.New(h) + if err != nil { + slog.Error("configure buffered adapter", "error", err) + os.Exit(1) + } + // Register the object to retain JSON v2 envelope ownership. For a fixed + // typed input, register adapter.HandleV1 or adapter.HandleV2 instead. + lambda.Start(adapter) +} diff --git a/examples/dispatcher/internal/orders/app.go b/examples/dispatcher/internal/orders/app.go new file mode 100644 index 0000000..128d564 --- /dev/null +++ b/examples/dispatcher/internal/orders/app.go @@ -0,0 +1,276 @@ +// Package orders is application code for the dispatcher example, not an edge API. +package orders + +import ( + "context" + "encoding/json/v2" + "errors" + "io" + "net/http" + "strings" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/authz" + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Grants resolves application enrollment and permissions, independently of token +// scopes and AWS IAM policies. Implementations must support concurrent requests. +type Grants interface { + Allowed(context.Context, identity.Caller, string, string) (bool, error) +} + +// Store reads the exact authorized resource. Events calls yield synchronously, +// stops on its first error, and observes context cancellation. A database-backed +// application must also preserve object/version consistency through execution. +type Store interface { + Read(context.Context, string) (string, error) + Events(context.Context, string, func(string) error) error +} + +// Config belongs to this example. Required strings have no implicit defaults. +// Transports are trusted infrastructure; nil selects the verifier's default. +type Config struct { + Cognito authn.CognitoConfig + Region string + ProofAudience string + IAMPrincipal string + Origin string + STSTransport http.RoundTripper + Grants Grants + Store Store + Streaming bool +} + +// New constructs the complete application without network I/O. Streaming is a +// startup choice: orders.watch is absent from the buffered variant's registry. +func New(cfg Config) (http.Handler, error) { + if cfg.Grants == nil || cfg.Store == nil || cfg.Origin == "" { + return nil, errors.New("orders: missing application configuration") + } + bearer, err := authn.NewCognitoVerifier(cfg.Cognito) + if err != nil { + return nil, err + } + var proofOptions []iamproof.VerifierOption + if cfg.STSTransport != nil { + proofOptions = append(proofOptions, iamproof.WithTransport(cfg.STSTransport)) + } + proof, err := iamproof.NewVerifier(cfg.Region, cfg.ProofAudience, proofOptions...) + if err != nil { + return nil, err + } + authentication, err := authn.New(authn.Config{ + Realm: "orders", Bearer: bearer.Verify, + IAMProof: func(ctx context.Context, credential string) (identity.Caller, error) { + caller, err := proof.Verify(ctx, credential) + if errors.Is(err, iamproof.ErrInvalidProof) { + return identity.Caller{}, authn.ErrInvalidCredentials + } + return caller, err + }, + }) + if err != nil { + return nil, err + } + selector, err := edge.NewActionHeader("Action") + if err != nil { + return nil, err + } + jwtPermission, err := authz.JWTScopes(cfg.Cognito.Issuer, "orders.read") + if err != nil { + return nil, err + } + iamPermission, err := authz.IAMPrincipal(cfg.IAMPrincipal) + if err != nil { + return nil, err + } + permission, err := authz.Any(jwtPermission, iamPermission) + if err != nil { + return nil, err + } + grant, err := authz.Check(func(ctx context.Context, req authz.Request) (bool, error) { + return cfg.Grants.Allowed(ctx, req.Caller, req.Action, req.Resource) + }) + if err != nil { + return nil, err + } + policy, err := authz.All(permission, grant) + if err != nil { + return nil, err + } + type entry struct { + policy authz.Rule + execute func(http.ResponseWriter, *http.Request, string) + } + registry := map[string]entry{ + "orders.read": {policy: policy, execute: func(w http.ResponseWriter, r *http.Request, resource string) { + state, err := cfg.Store.Read(r.Context(), resource) + if err != nil { + writeError(w, 503, "unavailable") + return + } + writeJSON(w, 200, map[string]string{"resource": resource, "state": state}) + }}, + } + if cfg.Streaming { + registry["orders.watch"] = entry{policy: policy, execute: func(w http.ResponseWriter, r *http.Request, resource string) { + watch(w, r, cfg.Store, resource) + }} + } + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // This example owns CORS. Disable conflicting Gateway-managed CORS when + // deploying it, or move this policy wholly into that deployment layer. + w.Header().Add("Vary", "Origin") + if origins := r.Header.Values("Origin"); len(origins) == 1 && origins[0] == cfg.Origin { + w.Header().Set("Access-Control-Allow-Origin", cfg.Origin) + w.Header().Set("Access-Control-Expose-Headers", "WWW-Authenticate") + } + if r.Method == http.MethodGet && r.URL.Path == "/healthz" { + writeJSON(w, 200, map[string]string{"status": "ok"}) + return + } + resource, ok := resourcePath(r.URL.Path) + if !ok { + writeError(w, 404, "not_found") + return + } + if r.Method == http.MethodOptions { + w.Header().Add("Vary", "Access-Control-Request-Method") + w.Header().Add("Vary", "Access-Control-Request-Headers") + if w.Header().Get("Access-Control-Allow-Origin") == "" || r.Header.Get("Access-Control-Request-Method") != http.MethodGet { + writeError(w, 403, "forbidden") + return + } + w.Header().Set("Access-Control-Allow-Methods", "GET") + w.Header().Set("Access-Control-Allow-Headers", "Authorization, Action, Custom-Trace") + w.WriteHeader(http.StatusNoContent) + return + } + if r.Method != http.MethodGet { + w.Header().Set("Allow", "GET, OPTIONS") + writeError(w, 405, "method_not_allowed") + return + } + caller, err := authentication.Authenticate(r.Context(), r.Header) + if err != nil { + writeAuthenticationError(w, err) + return + } + ctx, err := identity.WithCaller(r.Context(), caller) + if err != nil { + writeError(w, 500, "internal_error") + return + } + r = r.WithContext(ctx) + action, err := selector.Parse(r.Header) + if err != nil { + writeError(w, 400, "invalid_action") + return + } + selected, ok := registry[action] + if !ok { + writeError(w, 404, "unknown_action") + return + } + facts := authz.Request{Caller: caller, Action: action, Resource: resource} + if err := selected.policy.Authorize(ctx, facts); err != nil { + switch { + case errors.Is(err, authz.ErrDenied): + w.Header().Set("WWW-Authenticate", `Bearer realm="orders"`) + writeError(w, 403, "forbidden") + case errors.Is(err, authz.ErrUnavailable), errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded): + writeError(w, 503, "unavailable") + default: + writeError(w, 500, "internal_error") + } + return + } + selected.execute(w, r, resource) + }), nil +} + +// The application accepts canonical decimal IDs only; no aliases, escaping or +// query parameters can change the target between authorization and execution. +func resourcePath(path string) (string, bool) { + parts := strings.Split(path, "/") + if len(parts) != 5 || parts[0] != "" || parts[1] != "tenants" || parts[3] != "orders" { + return "", false + } + for _, id := range []string{parts[2], parts[4]} { + if len(id) == 0 || len(id) > 18 || id[0] == '0' { + return "", false + } + for _, c := range id { + if c < '0' || c > '9' { + return "", false + } + } + } + return path, true +} + +func writeAuthenticationError(w http.ResponseWriter, err error) { + status := http.StatusInternalServerError + if failure, ok := errors.AsType[*authn.Error](err); ok { + status = failure.StatusCode() + for _, challenge := range failure.Challenges() { + w.Header().Add("WWW-Authenticate", challenge) + } + } else if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { + status = http.StatusServiceUnavailable + } + code := "internal_error" + switch status { + case 400, 431: + code = "invalid_credentials" + case 401: + code = "unauthenticated" + case 503: + code = "unavailable" + } + writeError(w, status, code) +} + +func writeError(w http.ResponseWriter, status int, code string) { + writeJSON(w, status, map[string]string{"error": code}) +} + +func writeJSON(w http.ResponseWriter, status int, payload map[string]string) { + w.Header().Set("Content-Type", "application/json") + w.Header().Set("Cache-Control", "no-store") + w.WriteHeader(status) + // Status is committed. A client write failure cannot be replaced by JSON. + _ = json.MarshalWrite(w, payload) +} + +func watch(w http.ResponseWriter, r *http.Request, store Store, resource string) { + w.Header().Set("Content-Type", "application/x-ndjson") + w.Header().Set("Cache-Control", "no-store") + controller := http.NewResponseController(w) + // Publish only after authorization; this also makes an empty stream's + // commitment explicit before entering a potentially slow producer. + if err := controller.Flush(); err != nil { + return + } + err := store.Events(r.Context(), resource, func(state string) error { + if err := r.Context().Err(); err != nil { + return err + } + if err := json.MarshalWrite(w, map[string]string{"resource": resource, "state": state}); err != nil { + return err + } + if _, err := io.WriteString(w, "\n"); err != nil { + return err + } + return controller.Flush() + }) + if err != nil { + // net/http's documented abort sentinel prevents a truncated result from + // appearing complete. Edge converts this post-handoff panic to ErrStream; + // it never exposes the store's diagnostic. Do not recover it in middleware. + panic(http.ErrAbortHandler) + } +} diff --git a/examples/dispatcher/internal/orders/app_test.go b/examples/dispatcher/internal/orders/app_test.go new file mode 100644 index 0000000..861560d --- /dev/null +++ b/examples/dispatcher/internal/orders/app_test.go @@ -0,0 +1,270 @@ +package orders_test + +import ( + "context" + "encoding/json/v2" + "errors" + "net/http" + "net/http/httptest" + "slices" + "strings" + "sync/atomic" + "testing" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +type grantFunc func(context.Context, identity.Caller, string, string) (bool, error) + +func (f grantFunc) Allowed(ctx context.Context, caller identity.Caller, action, resource string) (bool, error) { + return f(ctx, caller, action, resource) +} + +type spyStore struct { + reads atomic.Int64 + events func(context.Context, string, func(string) error) error +} + +func (s *spyStore) Read(ctx context.Context, target string) (string, error) { + s.reads.Add(1) + if err := ctx.Err(); err != nil { + return "", err + } + // The response exposes which target actually executed, independently of the + // grant callback's observation. A changed target makes the test fail. + return "read:" + target, nil +} + +func (s *spyStore) Events(ctx context.Context, target string, yield func(string) error) error { + s.reads.Add(1) + return s.events(ctx, target, yield) +} + +func TestConsumerOutcomes(t *testing.T) { + cfg, credentials := providerFixtures(t) + for _, kind := range []string{"bearer", "iam"} { + for _, tc := range []struct { + name string + authorization []string + actions []string + grant string + status int + code string + challenges []string + }{ + {name: "success", status: 200}, + {name: "missing", authorization: []string{}, status: 401, code: "unauthenticated", challenges: []string{`Bearer realm="orders"`, `EdgeIAM realm="orders"`}}, + {name: "invalid_bearer", authorization: []string{"Bearer invalid"}, status: 401, code: "unauthenticated", challenges: []string{`Bearer realm="orders", error="invalid_token"`}}, + {name: "invalid_proof", authorization: []string{"EdgeIAM invalid"}, status: 401, code: "unauthenticated", challenges: []string{`EdgeIAM realm="orders"`}}, + {name: "duplicate_authorization", authorization: []string{credentials[kind], credentials[kind]}, status: 400, code: "invalid_credentials", challenges: []string{`Bearer realm="orders"`, `EdgeIAM realm="orders"`}}, + {name: "duplicate_action", actions: []string{"orders.read", "orders.read"}, status: 400, code: "invalid_action"}, + {name: "missing_action", actions: []string{}, status: 400, code: "invalid_action"}, + {name: "unknown_action", actions: []string{"orders.delete"}, status: 404, code: "unknown_action"}, + {name: "watch_not_registered", actions: []string{"orders.watch"}, status: 404, code: "unknown_action"}, + {name: "grant_denied", grant: "denied", status: 403, code: "forbidden", challenges: []string{`Bearer realm="orders"`}}, + {name: "grant_outage", grant: "outage", status: 503, code: "unavailable"}, + } { + for _, format := range []string{"native", "typed_v1", "typed_v2", "raw_rest", "raw_http_v1", "raw_http_v2", "stream_raw", "stream_typed"} { + t.Run(kind+"/"+tc.name+"/"+format, func(t *testing.T) { + cfg := cfg + store := new(spyStore) + cfg.Store = store + var granted atomic.Int64 + cfg.Grants = grantFunc(func(ctx context.Context, caller identity.Caller, action, target string) (bool, error) { + granted.Add(1) + if ctx.Err() != nil || caller.Kind() == identity.KindAnonymous || action != "orders.read" || target != resource { + t.Error("grant did not receive the authenticated caller and selected action/resource") + return false, nil + } + if kind == "bearer" && caller.Source() != identity.SourceLocallyVerifiedToken || kind == "iam" && caller.Source() != identity.SourceVerifiedIAMProof { + t.Errorf("caller source = %v for %s", caller.Source(), kind) + } + if tc.grant == "outage" { + return false, errors.New("sensitive dependency diagnostic") + } + return tc.grant != "denied", nil + }) + authorization := tc.authorization + if authorization == nil { + authorization = []string{credentials[kind]} + } + actions := tc.actions + if actions == nil { + actions = []string{"orders.read"} + } + headers := http.Header{"Authorization": authorization, "Action": actions, "Origin": {origin}, "Custom-Trace": {"first", "second"}, "Cookie": {"client=example"}} + h := application(t, cfg) + // Consumer middleware sees ordinary Go headers/cookies, including + // the intentionally lossy V2 comma combination. + observed := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + wantTrace := []string{"first", "second"} + if strings.HasSuffix(format, "v2") { + wantTrace = []string{"first,second"} + } + if !slices.Equal(r.Header.Values("Custom-Trace"), wantTrace) { + t.Errorf("custom header = %q; want %q", r.Header.Values("Custom-Trace"), wantTrace) + } + cookie, err := r.Cookie("client") + if err != nil || cookie.Value != "example" { + t.Errorf("request cookie = %v, %v", cookie, err) + } + http.SetCookie(w, &http.Cookie{Name: "one", Value: "1"}) + http.SetCookie(w, &http.Cookie{Name: "two", Value: "2"}) + h.ServeHTTP(w, r) + }) + out := serve(t, observed, format, resource, headers) + if out.status != tc.status { + t.Errorf("status = %d; want %d", out.status, tc.status) + } + wantChallenges := tc.challenges + cookies := out.headers.Values("Set-Cookie") + if strings.HasSuffix(format, "v2") { + cookies = out.cookies + if len(wantChallenges) != 0 { + wantChallenges = []string{strings.Join(wantChallenges, ", ")} + } + } + if !slices.Equal(cookies, []string{"one=1", "two=2"}) { + t.Errorf("response cookies = %q", cookies) + } + if !slices.Equal(out.headers.Values("WWW-Authenticate"), wantChallenges) { + t.Errorf("challenges = %q; want %q", out.headers.Values("WWW-Authenticate"), wantChallenges) + } + if out.headers.Get("Access-Control-Allow-Origin") != origin || out.headers.Get("Cache-Control") != "no-store" || out.headers.Get("Content-Type") != "application/json" { + t.Errorf("missing CORS/cache/content-type headers: %v", out.headers) + } + var body map[string]string + if err := json.Unmarshal(out.body, &body); err != nil { + t.Fatal(err) + } + if tc.status == 200 { + if body["resource"] != resource || body["state"] != "read:"+resource || store.reads.Load() != 1 || granted.Load() != 1 { + t.Errorf("execution did not match authorized target: %v, reads=%d grants=%d", body, store.reads.Load(), granted.Load()) + } + } else if body["error"] != tc.code || len(body) != 1 || store.reads.Load() != 0 { + t.Errorf("failure = %v, executions=%d; want only %q and no execution", body, store.reads.Load(), tc.code) + } + }) + } + } + } +} + +func TestPublicRoutesAndPreflight(t *testing.T) { + cfg, _ := providerFixtures(t) + cfg.Store = new(spyStore) + cfg.Grants = grantFunc(func(context.Context, identity.Caller, string, string) (bool, error) { + t.Error("public route reached grant resolution") + return false, nil + }) + h := application(t, cfg) + for _, tc := range []struct { + method, path, origin string + status int + }{ + {"GET", "/healthz", origin, 200}, + {"OPTIONS", resource, origin, 204}, + {"OPTIONS", "/unknown", origin, 404}, + {"OPTIONS", resource, "https://untrusted.example", 403}, + {"POST", resource, origin, 405}, + {"GET", "/tenants/042/orders/7", origin, 404}, + } { + r := httptest.NewRequestWithContext(t.Context(), tc.method, tc.path, nil) + r.Header.Set("Origin", tc.origin) + r.Header.Set("Access-Control-Request-Method", "GET") + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != tc.status { + t.Errorf("%s %s = %d; want %d", tc.method, tc.path, w.Code, tc.status) + } + if tc.status == 204 && (w.Header().Get("Access-Control-Allow-Headers") != "Authorization, Action, Custom-Trace" || w.Header().Get("Access-Control-Allow-Methods") != "GET") { + t.Errorf("preflight headers = %v", w.Header()) + } + } +} + +func TestVerificationOutagesNeverExecute(t *testing.T) { + cfg, credentials := providerFixtures(t) + for _, kind := range []string{"bearer", "iam"} { + t.Run(kind, func(t *testing.T) { + cfg := cfg + store := new(spyStore) + cfg.Store = store + cfg.Grants = grantFunc(func(context.Context, identity.Caller, string, string) (bool, error) { + t.Error("verification outage reached grant resolution") + return true, nil + }) + outage := transportFunc(func(*http.Request) (*http.Response, error) { + return nil, errors.New("private provider diagnostic") + }) + if kind == "bearer" { + cfg.Cognito.Transport = outage + } else { + cfg.STSTransport = outage + } + headers := http.Header{"Authorization": {credentials[kind]}, "Action": {"orders.read"}, "Origin": {origin}} + out := serve(t, application(t, cfg), "native", resource, headers) + var body map[string]string + if err := json.Unmarshal(out.body, &body); err != nil { + t.Fatal(err) + } + if out.status != 503 || len(body) != 1 || body["error"] != "unavailable" || len(out.headers.Values("WWW-Authenticate")) != 0 || store.reads.Load() != 0 { + t.Errorf("provider outage = %d %v, executions=%d", out.status, body, store.reads.Load()) + } + }) + } +} + +func TestCancellationDuringGrantNeverExecutes(t *testing.T) { + cfg, credentials := providerFixtures(t) + for _, format := range []string{"native", "buffered", "streaming"} { + t.Run(format, func(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + cfg := cfg + store := new(spyStore) + cfg.Store = store + cfg.Grants = grantFunc(func(context.Context, identity.Caller, string, string) (bool, error) { + cancel() + return true, nil + }) + h := application(t, cfg) + headers := http.Header{"Authorization": {credentials["bearer"]}, "Action": {"orders.read"}} + switch format { + case "native": + r := httptest.NewRequestWithContext(ctx, "GET", resource, nil) + r.Header = headers + w := httptest.NewRecorder() + h.ServeHTTP(w, r) + if w.Code != 503 { + t.Errorf("canceled native status = %d; want 503", w.Code) + } + case "buffered": + a, err := edge.New(h) + if err != nil { + t.Fatal(err) + } + _, err = a.HandleV1(ctx, restEvent(resource, headers)) + if !errors.Is(err, context.Canceled) { + t.Errorf("buffered error = %v; want cancellation", err) + } + case "streaming": + a, err := edge.NewStreaming(h) + if err != nil { + t.Fatal(err) + } + stream, err := a.HandleV1(ctx, restEvent(resource, headers)) + if stream != nil { + stream.Close() + } + if !errors.Is(err, context.Canceled) { + t.Errorf("stream error = %v; want cancellation", err) + } + } + if store.reads.Load() != 0 { + t.Error("canceled grant executed an action") + } + }) + } +} diff --git a/examples/dispatcher/internal/orders/environment.go b/examples/dispatcher/internal/orders/environment.go new file mode 100644 index 0000000..4e24349 --- /dev/null +++ b/examples/dispatcher/internal/orders/environment.go @@ -0,0 +1,93 @@ +package orders + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/url" + "os" + + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// FromEnvironment configures real verification and a small, explicitly enrolled +// read-only demonstration store. It never obtains server-side AWS credentials. +func FromEnvironment(streaming bool) (http.Handler, error) { + values := make(map[string]string) + for _, name := range []string{"COGNITO_ISSUER", "COGNITO_CLIENT_ID", "COGNITO_AUDIENCE", "IAM_PROOF_REGION", "IAM_PROOF_AUDIENCE", "ORDER_JWT_SUBJECT", "ORDER_IAM_PRINCIPAL", "CORS_ORIGIN"} { + values[name] = os.Getenv(name) + if values[name] == "" { + return nil, fmt.Errorf("orders: missing %s", name) + } + } + origin, err := url.Parse(values["CORS_ORIGIN"]) + if err != nil || origin.Host == "" || origin.User != nil || origin.Path != "" || origin.RawQuery != "" || origin.ForceQuery || origin.Fragment != "" || (origin.Scheme != "https" && origin.Scheme != "http") { + return nil, errors.New("orders: invalid CORS_ORIGIN") + } + return New(Config{ + Cognito: authn.CognitoConfig{ + Issuer: values["COGNITO_ISSUER"], + Clients: []authn.CognitoClient{{ClientID: values["COGNITO_CLIENT_ID"], Audience: values["COGNITO_AUDIENCE"]}}, + }, + Region: values["IAM_PROOF_REGION"], ProofAudience: values["IAM_PROOF_AUDIENCE"], + IAMPrincipal: values["ORDER_IAM_PRINCIPAL"], Origin: values["CORS_ORIGIN"], + Grants: demoGrants{issuer: values["COGNITO_ISSUER"], subject: values["ORDER_JWT_SUBJECT"], principal: values["ORDER_IAM_PRINCIPAL"]}, + Store: demoStore{}, Streaming: streaming, + }) +} + +// These application grants are server configuration, never client claims. This +// example enrolls one exact JWT subject and one full IAM caller ARN. It does not +// infer IAM role paths, role lifecycle identity or a human behind a session. +type demoGrants struct { + issuer string + subject string + principal string +} + +func (g demoGrants) Allowed(ctx context.Context, caller identity.Caller, action, resource string) (bool, error) { + if err := ctx.Err(); err != nil { + return false, err + } + if resource != "/tenants/42/orders/7" || (action != "orders.read" && action != "orders.watch") { + return false, nil + } + if jwt, ok := caller.JWT(); ok { + subject, present := jwt.Subject() + return present && jwt.Issuer() == g.issuer && subject == g.subject, nil + } + if iam, ok := caller.IAM(); ok { + return iam.PrincipalARN() == g.principal, nil + } + return false, nil +} + +type demoStore struct{} + +func (demoStore) Read(ctx context.Context, resource string) (string, error) { + if err := ctx.Err(); err != nil { + return "", err + } + if resource != "/tenants/42/orders/7" { + return "", errors.New("orders: unavailable resource") + } + return "shipped", nil +} + +func (s demoStore) Events(ctx context.Context, resource string, yield func(string) error) error { + state, err := s.Read(ctx, resource) + if err != nil { + return err + } + for _, update := range []string{"accepted", state} { + if err := ctx.Err(); err != nil { + return err + } + if err := yield(update); err != nil { + return err + } + } + return nil +} diff --git a/examples/dispatcher/internal/orders/environment_test.go b/examples/dispatcher/internal/orders/environment_test.go new file mode 100644 index 0000000..4ad53c0 --- /dev/null +++ b/examples/dispatcher/internal/orders/environment_test.go @@ -0,0 +1,61 @@ +package orders + +import ( + "testing" + + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +// Test the actual demonstration enrollment policy separately from cryptographic +// verification. All identities here are explicitly custom-assertion fixtures. +func TestDemoEnrollment(t *testing.T) { + const issuer = "https://issuer.example/pool" + const principal = "arn:aws:iam::123456789012:user/Alice" + grants := demoGrants{issuer: issuer, subject: "alice", principal: principal} + for _, tc := range []struct { + name string + issuer string + subject string + arn string + allowed bool + }{ + {name: "enrolled_subject", issuer: issuer, subject: "alice", allowed: true}, + {name: "wrong_issuer", issuer: "https://other.example/pool", subject: "alice"}, + {name: "wrong_subject", issuer: issuer, subject: "bob"}, + {name: "missing_subject", issuer: issuer}, + {name: "enrolled_iam", arn: principal, allowed: true}, + {name: "different_iam", arn: "arn:aws:iam::123456789012:user/Bob"}, + } { + t.Run(tc.name, func(t *testing.T) { + var caller identity.Caller + var err error + if tc.arn != "" { + caller, err = identity.NewIAM(tc.arn, identity.SourceCustomAssertion) + } else { + // A subjectless caller still needs a client identity. Enrollment + // must not silently treat that client as the configured subject. + claims := map[string]any{"iss": tc.issuer, "client_id": "app"} + if tc.subject != "" { + claims["sub"] = tc.subject + } + facts, parseErr := identity.NewClaims(claims) + if parseErr != nil { + t.Fatal(parseErr) + } + caller, err = identity.NewJWT(facts, identity.SourceCustomAssertion) + } + if err != nil { + t.Fatal(err) + } + for _, action := range []string{"orders.read", "orders.watch", "orders.delete"} { + for _, resource := range []string{"/tenants/42/orders/7", "/tenants/99/orders/7"} { + allowed, err := grants.Allowed(t.Context(), caller, action, resource) + want := tc.allowed && action != "orders.delete" && resource == "/tenants/42/orders/7" + if err != nil || allowed != want { + t.Errorf("Allowed(%q, %q) = %v, %v; want %v", action, resource, allowed, err, want) + } + } + } + }) + } +} diff --git a/examples/dispatcher/internal/orders/fixtures_test.go b/examples/dispatcher/internal/orders/fixtures_test.go new file mode 100644 index 0000000..9a7992d --- /dev/null +++ b/examples/dispatcher/internal/orders/fixtures_test.go @@ -0,0 +1,260 @@ +package orders_test + +import ( + "bytes" + "context" + "crypto" + "crypto/rand" + "crypto/rsa" + "crypto/sha256" + "encoding/base64" + "encoding/json/v2" + "errors" + "io" + "math/big" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/aws/aws-lambda-go/events" + "github.com/aws/aws-sdk-go-v2/aws" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/examples/dispatcher/internal/orders" + "github.com/asteroid-computing/go-lambda-edge/iamproof" +) + +const ( + issuer = "https://issuer.example/pool" + principal = "arn:aws:iam::123456789012:user/Alice" + resource = "/tenants/42/orders/7" + origin = "https://app.example" +) + +type transportFunc func(*http.Request) (*http.Response, error) + +func (f transportFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +// providerFixtures uses production verifiers and local cryptographic signatures. +// STS is a synthetic response, not an AWS signature or enrollment assertion. +// Neither transport ever delegates to a network-capable RoundTripper. +func providerFixtures(t *testing.T) (orders.Config, map[string]string) { + t.Helper() + key, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + t.Fatal(err) + } + jwks := marshal(t, map[string]any{"keys": []any{map[string]any{ + "kty": "RSA", "alg": "RS256", "use": "sig", "kid": "fixture", + "n": base64.RawURLEncoding.EncodeToString(key.N.Bytes()), + "e": base64.RawURLEncoding.EncodeToString(big.NewInt(int64(key.E)).Bytes()), + }}}) + now := time.Now().Unix() + claims := marshal(t, map[string]any{"iss": issuer, "sub": "alice", "client_id": "app", "aud": "https://orders.example", "token_use": "access", "scope": "orders.read", "iat": now, "exp": now + 3600}) + header := marshal(t, map[string]string{"alg": "RS256", "kid": "fixture"}) + input := base64.RawURLEncoding.EncodeToString(header) + "." + base64.RawURLEncoding.EncodeToString(claims) + digest := sha256.Sum256([]byte(input)) + signature, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:]) + if err != nil { + t.Fatal(err) + } + jwt := input + "." + base64.RawURLEncoding.EncodeToString(signature) + generator, err := iamproof.NewGenerator("eu-west-2", "orders.test", aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) { + return aws.Credentials{AccessKeyID: "AKIAIOSFODNN7EXAMPLE", SecretAccessKey: "synthetic-only"}, nil + })) + if err != nil { + t.Fatal(err) + } + proof, err := generator.Generate(t.Context()) + if err != nil { + t.Fatal(err) + } + cfg := orders.Config{ + Cognito: authn.CognitoConfig{Issuer: issuer, Clients: []authn.CognitoClient{{ClientID: "app", Audience: "https://orders.example"}}, + Transport: transportFunc(func(r *http.Request) (*http.Response, error) { + if r.URL.String() != issuer+"/.well-known/jwks.json" { + return nil, errors.New("unexpected JWKS request") + } + return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(bytes.NewReader(jwks))}, nil + }), + }, + Region: "eu-west-2", ProofAudience: "orders.test", IAMPrincipal: principal, Origin: origin, + STSTransport: transportFunc(func(r *http.Request) (*http.Response, error) { + if r.URL.Host != "sts.eu-west-2.amazonaws.com" || r.Header.Get("X-Edge-IAM-Audience") != "orders.test" || r.URL.Query().Get("Action") != "GetCallerIdentity" || r.URL.Query().Get("X-Amz-Signature") == "" { + return nil, errors.New("unexpected STS request") + } + body := `` + principal + `123456789012AIDAEXAMPLE` + return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader(body))}, nil + }), + } + return cfg, map[string]string{"bearer": "Bearer " + jwt, "iam": "EdgeIAM " + proof.Value()} +} + +func marshal(t testing.TB, value any) []byte { + t.Helper() + wire, err := json.Marshal(value) + if err != nil { + t.Fatal(err) + } + return wire +} + +func application(t *testing.T, cfg orders.Config) http.Handler { + t.Helper() + h, err := orders.New(cfg) + if err != nil { + t.Fatal(err) + } + return h +} + +// Explicitly authored fixtures retain the product/version distinction. The SDK +// V1 request type cannot distinguish REST from HTTP API payload 1.0 by itself. +func restEvent(path string, headers http.Header) events.APIGatewayProxyRequest { + return events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: path, MultiValueHeaders: headers.Clone(), RequestContext: events.APIGatewayProxyRequestContext{APIID: "synthetic-rest"}} +} + +func httpV2Event(path string, headers http.Header) events.APIGatewayV2HTTPRequest { + combined := make(map[string]string) + var cookies []string + for name, values := range headers { + if len(values) == 0 { + continue + } + if strings.EqualFold(name, "Cookie") { + cookies = append(cookies, values...) + continue + } + combined[strings.ToLower(name)] = strings.Join(values, ",") + } + return events.APIGatewayV2HTTPRequest{Version: "2.0", RawPath: path, Headers: combined, Cookies: cookies, + RequestContext: events.APIGatewayV2HTTPRequestContext{APIID: "synthetic-http", HTTP: events.APIGatewayV2HTTPRequestContextHTTPDescription{Method: "GET"}}} +} + +// Small responses only. Repeated V2 headers remain combined; cookies are kept +// separate. Incremental/late-error assertions use the reader directly elsewhere. +type response struct { + status int + headers http.Header + cookies []string + body []byte +} + +func serve(t *testing.T, h http.Handler, format, path string, headers http.Header) response { + t.Helper() + if format == "native" { + server := httptest.NewTestServer(t, h) + r, err := http.NewRequestWithContext(t.Context(), "GET", "http://example.com"+path, nil) + if err != nil { + t.Fatal(err) + } + r.Header = headers.Clone() + out, err := server.Client().Do(r) + if err != nil { + t.Fatal(err) + } + defer out.Body.Close() + body, err := io.ReadAll(out.Body) + if err != nil { + t.Fatal(err) + } + return response{status: out.StatusCode, headers: out.Header, body: body} + } + if format == "stream_raw" || format == "stream_typed" { + stream := openStream(t, h, format, path, headers) + defer stream.Close() + wire, err := io.ReadAll(stream) + if err != nil { + t.Fatal(err) + } + prefix, body, ok := bytes.Cut(wire, make([]byte, 8)) + if !ok { + t.Fatal("stream missing metadata delimiter") + } + var metadata struct { + Status int `json:"statusCode"` + Headers http.Header `json:"multiValueHeaders"` + } + if err := json.Unmarshal(prefix, &metadata); err != nil { + t.Fatal(err) + } + return response{status: metadata.Status, headers: metadata.Headers, body: body} + } + adapter, err := edge.New(h) + if err != nil { + t.Fatal(err) + } + v1, v2 := restEvent(path, headers), httpV2Event(path, headers) + var out1 events.APIGatewayProxyResponse + var out2 events.APIGatewayV2HTTPResponse + switch format { + case "typed_v1": + out1, err = adapter.HandleV1(t.Context(), v1) + case "typed_v2": + out2, err = adapter.HandleV2(t.Context(), v2) + case "raw_rest", "raw_http_v1", "raw_http_v2": + wire := marshal(t, v1) + if format == "raw_http_v1" { + // HTTP API 1.0's version is present only at the wire boundary. + wire = append([]byte(`{"version":"1.0",`), wire[1:]...) + } + if format == "raw_http_v2" { + wire = marshal(t, v2) + } + var out []byte + out, err = adapter.Invoke(t.Context(), wire) + if err == nil { + if format == "raw_http_v2" { + err = json.Unmarshal(out, &out2) + } else { + err = json.Unmarshal(out, &out1) + } + } + default: + t.Fatalf("unknown fixture format %q", format) + } + if err != nil { + t.Fatal(err) + } + if strings.HasSuffix(format, "v2") { + h := make(http.Header) + for name, value := range out2.Headers { + h.Set(name, value) + } + return response{status: out2.StatusCode, headers: h, cookies: out2.Cookies, body: decodeBody(t, out2.Body, out2.IsBase64Encoded)} + } + return response{status: out1.StatusCode, headers: http.Header(out1.MultiValueHeaders), body: decodeBody(t, out1.Body, out1.IsBase64Encoded)} +} + +func decodeBody(t *testing.T, body string, encoded bool) []byte { + t.Helper() + if !encoded { + return []byte(body) + } + out, err := base64.StdEncoding.DecodeString(body) + if err != nil { + t.Fatal(err) + } + return out +} + +func openStream(t *testing.T, h http.Handler, format, path string, headers http.Header) io.ReadCloser { + t.Helper() + adapter, err := edge.NewStreaming(h) + if err != nil { + t.Fatal(err) + } + var stream io.ReadCloser + if format == "stream_raw" { + stream, err = adapter.Handle(t.Context(), marshal(t, restEvent(path, headers))) + } else { + stream, err = adapter.HandleV1(t.Context(), restEvent(path, headers)) + } + if err != nil { + t.Fatal(err) + } + return stream +} diff --git a/examples/dispatcher/internal/orders/stream_test.go b/examples/dispatcher/internal/orders/stream_test.go new file mode 100644 index 0000000..8fc9a83 --- /dev/null +++ b/examples/dispatcher/internal/orders/stream_test.go @@ -0,0 +1,171 @@ +package orders_test + +import ( + "bufio" + "context" + "encoding/json/v2" + "errors" + "io" + "net/http" + "strings" + "testing" + "testing/synctest" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func TestStreamLifecycle(t *testing.T) { + for _, format := range []string{"stream_raw", "stream_typed"} { + for _, outcome := range []string{"complete", "early_close", "late_failure"} { + t.Run(format+"/"+outcome, func(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + cfg, credentials := providerFixtures(t) + cfg.Streaming = true + cfg.Grants = grantFunc(func(_ context.Context, caller identity.Caller, action, target string) (bool, error) { + return caller.Kind() != identity.KindAnonymous && action == "orders.watch" && target == resource, nil + }) + continueProduction := make(chan struct{}) + finished := make(chan struct{}) + cfg.Store = &spyStore{events: func(ctx context.Context, target string, yield func(string) error) error { + defer close(finished) + if target != resource { + return errors.New("incorrect execution target") + } + if err := yield("ready"); err != nil { + return err + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-continueProduction: + } + if outcome == "late_failure" { + return errors.New("private database diagnostic") + } + return yield("done") + }} + headers := http.Header{"Authorization": {credentials["iam"]}, "Action": {"orders.watch"}} + stream := openStream(t, application(t, cfg), format, resource, headers) + defer stream.Close() + reader := bufio.NewReader(stream) + metadata := readPrefix(t, reader) + if metadata.Status != 200 || metadata.Headers.Get("Content-Type") != "application/x-ndjson" { + t.Fatalf("stream metadata = %+v", metadata) + } + line, err := reader.ReadString('\n') + if err != nil { + t.Fatal(err) + } + var record map[string]string + if err := json.Unmarshal([]byte(line), &record); err != nil { + t.Fatal(err) + } + if record["resource"] != resource || record["state"] != "ready" { + t.Errorf("first record = %v", record) + } + // The first record arrived before production was allowed to finish. + synctest.Wait() + select { + case <-finished: + t.Fatal("producer finished before release") + default: + } + if outcome == "early_close" { + _ = stream.Close() + select { + case <-finished: + default: + t.Error("Close returned before producer stopped") + } + return + } + close(continueProduction) + synctest.Wait() + if outcome == "complete" { + // No consumer has read the second record. The unbuffered bridge + // must hold the writer here instead of collecting the body. + select { + case <-finished: + t.Fatal("slow reader did not apply backpressure") + default: + } + } + rest, err := io.ReadAll(reader) + if outcome == "late_failure" { + if !errors.Is(err, edge.ErrStream) || len(rest) != 0 || strings.Contains(err.Error(), "database") { + t.Errorf("late failure = %q, %v; want sanitized terminal error", rest, err) + } + } else { + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(rest, &record); err != nil { + t.Fatal(err) + } + if record["state"] != "done" { + t.Errorf("last record = %v", record) + } + } + select { + case <-finished: + default: + t.Error("terminal read returned before producer stopped") + } + }) + }) + } + } +} + +type streamMetadata struct { + Status int `json:"statusCode"` + Headers http.Header `json:"multiValueHeaders"` +} + +func readPrefix(t *testing.T, reader *bufio.Reader) streamMetadata { + t.Helper() + var prefix []byte + zeros := 0 + for len(prefix) < 16000 { + b, err := reader.ReadByte() + if err != nil { + t.Fatal(err) + } + prefix = append(prefix, b) + if b == 0 { + zeros++ + } else { + zeros = 0 + } + if zeros == 8 { + var metadata streamMetadata + if err := json.Unmarshal(prefix[:len(prefix)-8], &metadata); err != nil { + t.Fatal(err) + } + return metadata + } + } + t.Fatal("stream metadata exceeds prefix limit") + return streamMetadata{} +} + +func TestWatchAuthorizationBeforeStreaming(t *testing.T) { + cfg, credentials := providerFixtures(t) + cfg.Streaming = true + for _, kind := range []string{"bearer", "iam"} { + for _, format := range []string{"stream_raw", "stream_typed"} { + t.Run(kind+"/"+format, func(t *testing.T) { + cfg := cfg + store := new(spyStore) + cfg.Store = store + cfg.Grants = grantFunc(func(context.Context, identity.Caller, string, string) (bool, error) { return false, nil }) + headers := http.Header{"Authorization": {credentials[kind]}, "Action": {"orders.watch"}, "Origin": {origin}} + out := serve(t, application(t, cfg), format, resource, headers) + if out.status != 403 || out.headers.Get("Content-Type") != "application/json" || out.headers.Get("Access-Control-Allow-Origin") != origin || store.reads.Load() != 0 { + t.Errorf("denied watch = %+v, executions=%d", out, store.reads.Load()) + } + }) + } + } +} diff --git a/examples/dispatcher/native/main.go b/examples/dispatcher/native/main.go new file mode 100644 index 0000000..271cb11 --- /dev/null +++ b/examples/dispatcher/native/main.go @@ -0,0 +1,25 @@ +// Command native serves the buffered orders application on loopback. +package main + +import ( + "errors" + "log/slog" + "net/http" + "os" + "time" + + "github.com/asteroid-computing/go-lambda-edge/examples/dispatcher/internal/orders" +) + +func main() { + h, err := orders.FromEnvironment(false) + if err != nil { + slog.Error("configure orders", "error", err) + os.Exit(1) + } + server := &http.Server{Addr: "127.0.0.1:8080", Handler: h, ReadHeaderTimeout: 5 * time.Second} + if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + slog.Error("serve orders", "error", err) + os.Exit(1) + } +} diff --git a/examples/dispatcher/streaming/main.go b/examples/dispatcher/streaming/main.go new file mode 100644 index 0000000..19f5be7 --- /dev/null +++ b/examples/dispatcher/streaming/main.go @@ -0,0 +1,34 @@ +// Command streaming registers the orders application for REST response streaming. +// Deployed interoperability remains subject to docs/streaming.md's qualification. +package main + +import ( + "context" + "log/slog" + "os" + + "github.com/aws/aws-lambda-go/lambda" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/examples/dispatcher/internal/orders" +) + +func main() { + h, err := orders.FromEnvironment(true) + if err != nil { + slog.Error("configure orders", "error", err) + os.Exit(1) + } + adapter, err := edge.NewStreaming(h, edge.WithStreamErrorReporter(func(_ context.Context, err error) { + // Edge supplies sanitized invocation diagnostics. Never log a request's + // Authorization header, token, proof or raw provider error here. + slog.Error("orders stream terminated", "error", err) + })) + if err != nil { + slog.Error("configure streaming adapter", "error", err) + os.Exit(1) + } + // Return the reader directly; do not collect or JSON-encode the stream. + // For already typed REST events, register adapter.HandleV1 instead. + lambda.Start(adapter.Handle) +} diff --git a/scripts/check-consumer.sh b/scripts/check-consumer.sh new file mode 100644 index 0000000..bd7b7f7 --- /dev/null +++ b/scripts/check-consumer.sh @@ -0,0 +1,12 @@ +#!/bin/sh +# Compile and run a separate consumer module using only public edge imports. +set -eu +repo_root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd) +consumer_dir=$(mktemp -d) +trap 'rm -rf "$consumer_dir"' EXIT HUP INT TERM +cp "$repo_root/testdata/consumer/consumer_test.go.txt" "$consumer_dir/consumer_test.go" +cd "$consumer_dir" +go mod init example.com/edge-consumer +go mod edit -go=1.27.0 -require=github.com/asteroid-computing/go-lambda-edge@v0.0.0 +go mod edit "-replace=github.com/asteroid-computing/go-lambda-edge=$repo_root" +go test -mod=mod . diff --git a/streaming_bench_test.go b/streaming_bench_test.go new file mode 100644 index 0000000..c06ef14 --- /dev/null +++ b/streaming_bench_test.go @@ -0,0 +1,93 @@ +package edge_test + +import ( + "bytes" + "encoding/json/v2" + "io" + "net/http" + "testing" + + "github.com/aws/aws-lambda-go/events" + + "github.com/asteroid-computing/go-lambda-edge" +) + +func BenchmarkStreamingPublic(b *testing.B) { + for _, size := range []struct { + name string + bytes int + }{ + {name: "128B", bytes: 128}, + {name: "64KiB", bytes: 64 * 1024}, + {name: "5MiB", bytes: 5 * 1024 * 1024}, + } { + b.Run(size.name, func(b *testing.B) { + // Reuse one bounded application chunk; never allocate a whole body. + chunk := bytes.Repeat([]byte("x"), min(size.bytes, 4096)) + adapter, err := edge.NewStreaming(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/octet-stream") + controller := http.NewResponseController(w) + for remaining := size.bytes; remaining > 0; { + if r.Context().Err() != nil { + return + } + n := min(remaining, len(chunk)) + if _, err := w.Write(chunk[:n]); err != nil { + return + } + if err := controller.Flush(); err != nil { + return + } + remaining -= n + } + })) + if err != nil { + b.Fatal(err) + } + event := events.APIGatewayProxyRequest{HTTPMethod: "GET", Path: "/", RequestContext: events.APIGatewayProxyRequestContext{APIID: "synthetic"}} + wire, err := json.Marshal(event) + if err != nil { + b.Fatal(err) + } + for _, format := range []string{"rawREST", "typedV1"} { + b.Run(format, func(b *testing.B) { + buffer := make([]byte, 8192) + b.ReportAllocs() + b.SetBytes(int64(size.bytes)) + for b.Loop() { + var stream io.ReadCloser + var err error + if format == "rawREST" { + stream, err = adapter.Handle(b.Context(), wire) + } else { + stream, err = adapter.HandleV1(b.Context(), event) + } + if err != nil { + b.Fatal(err) + } + // Explicit incremental reads avoid io.ReadAll and io.Copy's + // optional ReaderFrom/WriterTo shortcuts or buffer pooling. + total := 0 + for { + n, readErr := stream.Read(buffer) + total += n + if readErr == io.EOF { + break + } + if readErr != nil { + stream.Close() + b.Fatal(readErr) + } + } + if err := stream.Close(); err != nil { + b.Fatal(err) + } + if total <= size.bytes { + b.Fatal("stream body or metadata truncated") + } + } + }) + } + }) + } +} diff --git a/testdata/consumer/consumer_test.go.txt b/testdata/consumer/consumer_test.go.txt new file mode 100644 index 0000000..678091b --- /dev/null +++ b/testdata/consumer/consumer_test.go.txt @@ -0,0 +1,82 @@ +package consumer_test + +import ( + "context" + "net/http" + "testing" + + "github.com/aws/aws-lambda-go/lambda" + + "github.com/asteroid-computing/go-lambda-edge" + "github.com/asteroid-computing/go-lambda-edge/authn" + "github.com/asteroid-computing/go-lambda-edge/authz" + "github.com/asteroid-computing/go-lambda-edge/iamproof" + "github.com/asteroid-computing/go-lambda-edge/identity" +) + +func TestPublicComposition(t *testing.T) { + // Explicit verifier stub: constructing this caller does not verify a token. + claims, err := identity.NewClaims(map[string]any{"iss": "https://issuer.example/pool", "sub": "alice"}) + if err != nil { + t.Fatal(err) + } + caller, err := identity.NewJWT(claims, identity.SourceLocallyVerifiedToken) + if err != nil { + t.Fatal(err) + } + authentication, err := authn.New(authn.Config{Bearer: func(context.Context, string) (identity.Caller, error) { + return caller, nil + }}) + if err != nil { + t.Fatal(err) + } + policy, err := authz.JWTSubject("https://issuer.example/pool", "alice") + if err != nil { + t.Fatal(err) + } + selector, err := edge.NewActionHeader("Action") + if err != nil { + t.Fatal(err) + } + h, err := authentication.Handler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + action, err := selector.Parse(r.Header) + if err != nil { + http.Error(w, "bad action", 400) + return + } + if action != "read" { + http.NotFound(w, r) + return + } + err = policy.Authorize(r.Context(), authz.Request{Caller: identity.FromContext(r.Context()), Action: action}) + if err != nil { + http.Error(w, "forbidden", 403) + return + } + w.WriteHeader(204) + })) + if err != nil { + t.Fatal(err) + } + adapter, err := edge.New(h) + if err != nil { + t.Fatal(err) + } + var registered lambda.Handler = adapter + _, err = registered.Invoke(t.Context(), []byte(`{"httpMethod":"GET","path":"/","headers":{"authorization":"Bearer fixture","action":"read"},"requestContext":{"apiId":"synthetic"}}`)) + if err != nil { + t.Fatal(err) + } + _ = lambda.NewHandler(adapter.HandleV1) + _ = lambda.NewHandler(adapter.HandleV2) + stream, err := edge.NewStreaming(h) + if err != nil { + t.Fatal(err) + } + _ = lambda.NewHandler(stream.Handle) + _ = lambda.NewHandler(stream.HandleV1) + // Construction performs no STS call or server-credential lookup. + if _, err := iamproof.NewVerifier("eu-west-2", "consumer.test"); err != nil { + t.Fatal(err) + } +} From a3587bf9157e6aaf9da323b970136dce0a31ccfb Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 21 Sep 2026 11:12:40 +0100 Subject: [PATCH 41/44] ci: target main pull requests and document release automation --- .github/workflows/go.yml | 7 ++ docs/decisions/0024-release-automation.md | 100 ++++++++++++++++++++++ docs/decisions/README.md | 1 + docs/plan.md | 5 ++ 4 files changed, 113 insertions(+) create mode 100644 docs/decisions/0024-release-automation.md diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 03e3401..2295990 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -2,7 +2,14 @@ name: Go on: push: + branches: [main] pull_request: + branches: [main] + types: [opened, synchronize, reopened, ready_for_review] + +concurrency: + group: go-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} permissions: contents: read diff --git a/docs/decisions/0024-release-automation.md b/docs/decisions/0024-release-automation.md new file mode 100644 index 0000000..3998899 --- /dev/null +++ b/docs/decisions/0024-release-automation.md @@ -0,0 +1,100 @@ +# 0024: Release Please and pull-request validation + +Status: Release Please plus its GitHub Action, and checks for PRs against main, +are user-directed on 2026-09-21. Credential and versioning recommendations below +await review. PR trigger changes are implemented independently. + +## Evidence + +The existing Go workflow already runs formatting, race tests, vet, module +verification, an external consumer check and Linux arm64/amd64 builds. It previously +ran for pushes and PRs against every branch. There are no local release tags or +release configuration files. The repository secret/variable name listings returned +no entries; no secret values were requested. Organization-scoped credentials may +need separate configuration. This review makes no repository-settings changes. + +Official sources checked: + +- [Release Please Action v5.0.0](https://github.com/googleapis/release-please-action/releases/tag/v5.0.0) + is the latest stable GitHub release on this date. The tagged action uses Node 24 + and supports manifest/config inputs, a token and target-branch. Its commit is + `45996ed1f6d02564a971a2fa1b5860e934307cf7`. +- [Manifest configuration](https://github.com/googleapis/release-please/blob/main/docs/manifest-releaser.md) + supports the root package, Go strategy, bootstrap version and pre-major bump + policies. The Go strategy updates a changelog; no generated version.go is needed. +- [GitHub workflow triggers](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/trigger-a-workflow) + now allow built-in-token PR opened/synchronize/reopened events to create runs + requiring manual approval. App/PAT tokens allow automatic runs. This is newer + than Release Please README wording that says such events do not trigger CI. +- [PR events](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request) + filter by base branch and normally check the proposed merge result. +- [GitHub App token action](https://github.com/actions/create-github-app-token) + supports short-lived installation tokens scoped to a repository and explicit + permissions. Latest stable release checked: v3.2.0. + +## Accepted PR checks + +Use pull_request against main, including opened, synchronize, reopened and +ready_for_review. Retain the existing validation matrix, read-only default token +and checkout without persisted credentials. Cancel superseded runs for the same +PR. Do not use path filters that could omit required checks on release-only PRs. +Keep current push-to-main validation until the release workflow is integrated. + +## Recommendation A: Release workflow and credential + +Use a dedicated push-to-main release workflow that first calls the Go workflow +as a reusable validation job, then runs Release Please only after success. The Go +workflow retains direct PR triggers and gains workflow_call; remove its direct +push trigger at that point to avoid duplicate main validation. Keep release jobs +serialized without canceling a running release. Human merging of the generated +release PR remains the publication decision; do not enable auto-merge. + +Recommend a repository-scoped GitHub App installation token, using a variable +RELEASE_PLEASE_CLIENT_ID and secret RELEASE_PLEASE_PRIVATE_KEY. The current token +action prefers client-id; app-id remains supported but deprecated. Give the app Contents, +Pull requests and Issues write permissions for Release Please operations. Mint +only those permissions for this repository in the release job. The PR validation +workflow needs no app credentials. Pin both new actions to reviewed stable commit +SHAs with version comments; retain existing checkout/setup-go choices. + +Alternative: a fine-grained PAT stored as RELEASE_PLEASE_TOKEN, with matching +repository permissions. It works but ties rotation/ownership to a user credential. +The built-in GITHUB_TOKEN avoids an extra credential, but release PR checks need +manual approval and other downstream events remain suppressed. The repository's +current workflow PR-creation/approval setting is disabled, so that route also +needs an owner settings change. Do not silently fall back between token types. + +No app, credential, remote PR or release is created during local wiring. App +installation/credential provisioning is an explicit setup prerequisite. + +## Recommendation B: Initial version and v0 evolution + +Recommend one root Go module with ordinary v-prefixed tags, starting at v0.1.0. +Use manifest mode, release-type go, include-component-in-tag false, an empty +initial manifest and initial-version 0.1.0. The v5 action bundles release-please +17.6.0; its tagged schema explicitly supports initial-version. This avoids +pretending an actual earlier release exists. +Set bump-minor-pre-major true and bump-patch-for-minor-pre-major false: breaking +changes and features bump the minor version during v0; fixes bump the patch. +Move to v1 deliberately in a separately reviewed release rather than by an +incidental breaking commit. Do not rewrite the Go toolchain directive or add a +runtime version API. Release Please owns CHANGELOG.md and the version manifest. + +Alternative: normal breaking-change-to-major behavior, which can unexpectedly +turn a v0 breaking change into v1.0.0, or permanently forcing release-as, which +must be removed after the intended release. The bundled release-please supports initial-version; use it only for bootstrap. + +Use Conventional Commits in merged history. Squash-merge feature PRs with suitable +feat/fix/chore titles; a release PR's generated title/body must be preserved for +Release Please recognition. Publishing creates a Go module tag/GitHub release, +not an npm artifact or deployment of the example binaries. + +The existing license and AWS interoperability qualifications remain release-review +items. Configuring release automation does not claim they are resolved, and does +not authorize publishing a release now. + +## Validation + +The main-targeted PR workflow passes local actionlint v1.7.12 and diff whitespace +checks. No Go source changed in this step; the existing Go test/build commands +remain in place. No workflow was dispatched or repository setting changed. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index f61339f..0d69d09 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -32,3 +32,4 @@ Keep related decisions small enough to review and implement independently. | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | | [0022: Streaming writer review](0022-streaming-writer-review.md) | Accepted; writer, public entry points and local validation complete; SDK deployment gate retained | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | | [0023: Consumer readiness](0023-consumer-readiness.md) | R1–R5 accepted; local implementation complete; external release prerequisites open | What complete consumer example, test support, deferred scope and release preparation should follow the implemented packages? | +| [0024: Release automation](0024-release-automation.md) | Release Please/PR checks directed; credential and v0 policy await review | How should release PRs, validation, credentials and initial versioning work? | diff --git a/docs/plan.md b/docs/plan.md index 4795be4..8b033ec 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -445,6 +445,11 @@ and Linux arm64/amd64 builds pass on Go 1.27.1 (2026-09-21). GitHub has no runs this branch; its actual CI evidence, the license and deployment qualifications remain open. No push, publication or AWS deployment occurred. +Release automation follow-up (2026-09-21): the user selected Release Please and +its GitHub Action, plus checks on PRs against main. [Decision 0024](decisions/0024-release-automation.md) +records the accepted trigger change and proposed credential/versioning choices. +Those choices require review before the dependent release workflow is wired. + ## Findings carried forward from the Beakley review These are requirements to test independently, not instructions to preserve old From 6eb316744b63c7b3e4c6f6139fdf5dccb11eae62 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 21 Sep 2026 11:22:57 +0100 Subject: [PATCH 42/44] ci: configure release-please with GitHub App authentication --- .github/workflows/go.yml | 3 +- .github/workflows/release-please.yml | 50 ++++++++++++ .release-please-manifest.json | 1 + README.md | 1 + docs/decisions/0024-release-automation.md | 38 ++++++--- docs/decisions/README.md | 2 +- docs/plan.md | 8 +- docs/release-readiness.md | 12 ++- docs/releases.md | 95 +++++++++++++++++++++++ release-please-config.json | 15 ++++ 10 files changed, 208 insertions(+), 17 deletions(-) create mode 100644 .github/workflows/release-please.yml create mode 100644 .release-please-manifest.json create mode 100644 docs/releases.md create mode 100644 release-please-config.json diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 2295990..5140433 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -1,8 +1,7 @@ name: Go on: - push: - branches: [main] + workflow_call: pull_request: branches: [main] types: [opened, synchronize, reopened, ready_for_review] diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml new file mode 100644 index 0000000..afcb295 --- /dev/null +++ b/.github/workflows/release-please.yml @@ -0,0 +1,50 @@ +name: Release Please + +on: + push: + branches: [main] + +permissions: + contents: read + +concurrency: + group: release-please-${{ github.ref }} + cancel-in-progress: false + +jobs: + validate: + uses: ./.github/workflows/go.yml + + release: + needs: validate + runs-on: ubuntu-latest + # Repository writes use the scoped App token, not GITHUB_TOKEN. + permissions: {} + steps: + - name: Check App configuration + env: + APP_CLIENT_ID: ${{ vars.RELEASE_PLEASE_CLIENT_ID }} + APP_PRIVATE_KEY: ${{ secrets.RELEASE_PLEASE_PRIVATE_KEY }} + run: | + if [ -z "$APP_CLIENT_ID" ] || [ -z "$APP_PRIVATE_KEY" ]; then + echo '::error::Set RELEASE_PLEASE_CLIENT_ID and RELEASE_PLEASE_PRIVATE_KEY; see docs/releases.md.' + exit 1 + fi + - name: Create repository-scoped App token + id: app-token + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + with: + client-id: ${{ vars.RELEASE_PLEASE_CLIENT_ID }} + private-key: ${{ secrets.RELEASE_PLEASE_PRIVATE_KEY }} + owner: ${{ github.repository_owner }} + repositories: ${{ github.event.repository.name }} + permission-contents: write + permission-issues: write + permission-pull-requests: write + - name: Update release PR or publish merged release + uses: googleapis/release-please-action@45996ed1f6d02564a971a2fa1b5860e934307cf7 # v5.0.0 + with: + token: ${{ steps.app-token.outputs.token }} + target-branch: main + config-file: release-please-config.json + manifest-file: .release-please-manifest.json diff --git a/.release-please-manifest.json b/.release-please-manifest.json new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/.release-please-manifest.json @@ -0,0 +1 @@ +{} diff --git a/README.md b/README.md index e19c443..408381e 100644 --- a/README.md +++ b/README.md @@ -142,4 +142,5 @@ propagate after cleanup; post-handoff streaming panics become terminal errors. - [Implementation plan](docs/plan.md) - [Design decisions](docs/decisions/README.md) - [Repository working agreement](AGENTS.md) +- [Release workflow and GitHub App setup](docs/releases.md) - [Initial codec benchmarks](docs/benchmarks.md) diff --git a/docs/decisions/0024-release-automation.md b/docs/decisions/0024-release-automation.md index 3998899..f281e01 100644 --- a/docs/decisions/0024-release-automation.md +++ b/docs/decisions/0024-release-automation.md @@ -1,8 +1,8 @@ # 0024: Release Please and pull-request validation -Status: Release Please plus its GitHub Action, and checks for PRs against main, -are user-directed on 2026-09-21. Credential and versioning recommendations below -await review. PR trigger changes are implemented independently. +Status: accepted and locally implemented on 2026-09-21. The user approved +GitHub App authentication and initial v0.1.0 following the recommended v0 policy. +Release Please, its GitHub Action and PR checks against main are user-directed. ## Evidence @@ -38,9 +38,9 @@ Use pull_request against main, including opened, synchronize, reopened and ready_for_review. Retain the existing validation matrix, read-only default token and checkout without persisted credentials. Cancel superseded runs for the same PR. Do not use path filters that could omit required checks on release-only PRs. -Keep current push-to-main validation until the release workflow is integrated. +Main validation moves to the reusable job called by the release workflow. -## Recommendation A: Release workflow and credential +## Accepted A: Release workflow and credential Use a dedicated push-to-main release workflow that first calls the Go workflow as a reusable validation job, then runs Release Please only after success. The Go @@ -67,7 +67,7 @@ needs an owner settings change. Do not silently fall back between token types. No app, credential, remote PR or release is created during local wiring. App installation/credential provisioning is an explicit setup prerequisite. -## Recommendation B: Initial version and v0 evolution +## Accepted B: Initial version and v0 evolution Recommend one root Go module with ordinary v-prefixed tags, starting at v0.1.0. Use manifest mode, release-type go, include-component-in-tag false, an empty @@ -93,8 +93,24 @@ The existing license and AWS interoperability qualifications remain release-revi items. Configuring release automation does not claim they are resolved, and does not authorize publishing a release now. -## Validation - -The main-targeted PR workflow passes local actionlint v1.7.12 and diff whitespace -checks. No Go source changed in this step; the existing Go test/build commands -remain in place. No workflow was dispatched or repository setting changed. +## Implementation and validation + +The release workflow calls the existing Go jobs before minting a scoped App token +and invoking Release Please. Go retains direct main-targeted PR events and uses +workflow_call for main validation, with no duplicate push trigger. Both release +and App-token actions use the verified stable commit SHAs. Missing configuration +fails explicitly without printing credentials or falling back to GITHUB_TOKEN. + +Root Go manifest configuration selects initial-version 0.1.0, an empty bootstrap +manifest, unprefixed v-tags and the accepted pre-major bump flags. The first +release PR creates CHANGELOG.md. [The release guide](../releases.md) documents +App setup, permissions, checks, human publication review and remaining limits. + +Both workflows pass local actionlint v1.7.12 and diff whitespace checks. JSON +configuration validates against the bundled release-please 17.6.0 schema. Local +structure checks confirm PR triggers, reusable validation, release's dependency +on validation, scoped credential use, and unchanged Go test/build jobs. No Go +source changed, so runtime suites were not rerun for this workflow-only change. +No App was installed, credentials provisioned, workflow dispatched, repository +settings changed, or release published. Remote activation still needs App setup +and a successful GitHub run. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 0d69d09..460ab1a 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -32,4 +32,4 @@ Keep related decisions small enough to review and implement independently. | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | | [0022: Streaming writer review](0022-streaming-writer-review.md) | Accepted; writer, public entry points and local validation complete; SDK deployment gate retained | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | | [0023: Consumer readiness](0023-consumer-readiness.md) | R1–R5 accepted; local implementation complete; external release prerequisites open | What complete consumer example, test support, deferred scope and release preparation should follow the implemented packages? | -| [0024: Release automation](0024-release-automation.md) | Release Please/PR checks directed; credential and v0 policy await review | How should release PRs, validation, credentials and initial versioning work? | +| [0024: Release automation](0024-release-automation.md) | Accepted; locally implemented and validated; App setup/remote run pending | How should release PRs, validation, credentials and initial versioning work? | diff --git a/docs/plan.md b/docs/plan.md index 8b033ec..4351dbb 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -447,8 +447,12 @@ remain open. No push, publication or AWS deployment occurred. Release automation follow-up (2026-09-21): the user selected Release Please and its GitHub Action, plus checks on PRs against main. [Decision 0024](decisions/0024-release-automation.md) -records the accepted trigger change and proposed credential/versioning choices. -Those choices require review before the dependent release workflow is wired. +records the approved GitHub App credential and initial v0.1.0, with pre-major +versioning. Release Please will run after reusable main validation; PRs continue +to use the same Go checks. Local wiring and validation are complete: both workflows +pass actionlint, release JSON passes the bundled schema, and existing Go jobs are +unchanged. See the [release guide](releases.md). App installation/credential setup +and actual GitHub execution remain pending; nothing was pushed or published. ## Findings carried forward from the Beakley review diff --git a/docs/release-readiness.md b/docs/release-readiness.md index 2162e17..48af5e3 100644 --- a/docs/release-readiness.md +++ b/docs/release-readiness.md @@ -11,14 +11,24 @@ commitment. This checklist records preparation, not authorization to publish: - Reproducible SDK versions and an isolated Smithy patch update. - Race tests, vet, module verification, formatting and Linux arm64/amd64 builds. +Release Please and GitHub App authentication are now configured for an initial +v0.1.0 under decision 0024. [The release guide](releases.md) documents the workflow +and required App setup. Merging a generated release PR is the publication decision; +automation does not resolve the remaining readiness items below. + ## Outstanding evidence and owner choices GitHub returned no runs for `work/edge-foundation` during this milestone. The workflow covers Go 1.27.0/latest 1.27, race tests, vet, module verification, -external consumer checks and both Linux architectures. Obtain a passing run for +external consumer checks and both Linux architectures. It runs directly for PRs +against main and is reused by the release workflow on main pushes. Obtain a passing run for the actual release commit after an authorized push. Local tests do not establish the Go 1.27.0 matrix result. +Install/configure the release App and provide RELEASE_PLEASE_CLIENT_ID and +RELEASE_PLEASE_PRIVATE_KEY before activating releases. No App credentials or +repository policies were changed during local implementation. + The owner's license and copyright holder remain undecided. No license, repository visibility change, push, tag or publication has been performed in this milestone. Resolve the license before external publication. diff --git a/docs/releases.md b/docs/releases.md new file mode 100644 index 0000000..e20f541 --- /dev/null +++ b/docs/releases.md @@ -0,0 +1,95 @@ +# Releases + +Release Please manages this single root Go module. Its first release is +`v0.1.0`. While versions remain below v1, features and breaking changes bump the +minor version; fixes bump the patch version. A v1 release is an explicit review +decision. The Go toolchain directive is independent of the module version. + +## Workflow + +1. Opening or updating a PR against main runs the Go workflow. Reopening a PR + or marking it ready for review also runs checks. Superseded PR runs cancel. +2. A push to main starts Release Please's workflow. It calls the same Go workflow + for formatting, race tests, vet, module verification, external consumption and + Linux arm64/amd64 builds. No separate push-triggered Go run duplicates it. +3. Only after validation succeeds does the release job mint a GitHub App token + and run Release Please. It creates or updates a release PR with CHANGELOG.md + and the version manifest. The App's PR events can run CI automatically. +4. Review and merge that release PR when ready to publish. After the main checks + succeed, Release Please creates the version tag and GitHub release. There is + no automatic merge, binary deployment or separate package-registry upload. + +Release runs share one concurrency group and do not cancel a running release. +GitHub may replace an older pending run with a newer one; this is not a guarantee +of one release operation per push. Release Please reconciles the main branch. + +The configuration is [release-please-config.json](../release-please-config.json). +The [initial manifest](../.release-please-manifest.json) is empty because no release +exists yet. `initial-version` selects 0.1.0 only for the first release. Release +Please subsequently maintains the manifest; it does not force every release +back to 0.1.0. Plain `vX.Y.Z` tags identify this root module, without a component +prefix. CHANGELOG.md is created by the first release PR. + +## GitHub App setup + +Use an organization-owned App installed on `asteroid-computing/go-lambda-edge`. +Its repository permissions must include: + +- Contents: read and write. +- Pull requests: read and write. +- Issues: read and write, for release labels and related operations. + +In this repository's **Settings → Secrets and variables → Actions**, configure: + +| Kind | Name | Value | +| --- | --- | --- | +| Variable | `RELEASE_PLEASE_CLIENT_ID` | The App's Client ID | +| Secret | `RELEASE_PLEASE_PRIVATE_KEY` | Its PEM private key | + +Organization-level settings are also usable when explicitly granted to this +repository. Use the Client ID, not the installation ID; the current token action +prefers `client-id` over its deprecated `app-id` input. Store the private key as +an Actions secret, not in this repository. + +The workflow explicitly requests only this repository and the three permissions +above. The token action revokes its short-lived token after the job. The release +job's built-in GITHUB_TOKEN has no permissions; validation gets contents read and +no inherited App secrets. Missing App configuration fails with a setup message +instead of silently using a different credential. + +An App must be installed and allowed by organization policy before this works. +The workflow does not create/install the App, provision secrets, change rulesets, +or bypass branch protection. Configure required PR checks using the check names +observed in the first successful run, and review any existing rules that restrict +the App's release-branch or version-tag writes. PR checks use the proposed merge +result; main checks validate the push that triggered the release workflow. + +## Commit and release review + +Use Conventional Commits in merged history: `feat:`, `fix:` and explicit `!` or +`BREAKING CHANGE:` markers where appropriate. Squash-merge feature PRs with an +appropriate conventional title. Preserve Release Please's generated release PR +title/body and labels so it can recognize the merged release. Ordinary docs/CI +changes alone need not create a release PR. + +Merging the release PR is the publication decision. Confirm the +[release-readiness prerequisites](release-readiness.md), including the license +choice and accurate AWS support claims, before merging it. Those are human +review items, not automatically verified by the workflow. No live AWS tests run +in this automation, and a passing CI run does not close deployment qualifications. + +## Maintenance + +Release Please Action v5.0.0 and Create GitHub App Token v3.2.0 are pinned to +reviewed commit SHAs with version comments. Review upstream changes when updating +them. The config schema is pinned to release-please 17.6.0, bundled by this action. +No runtime version API or generated version.go file is required. + +Validate workflow edits locally with `actionlint`; validate config edits against +the referenced JSON schema. A real GitHub run is still needed to verify App +installation, permissions, repository policies and remote release recognition. + +References: [Release Please action](https://github.com/googleapis/release-please-action/tree/v5.0.0), +[manifest configuration](https://github.com/googleapis/release-please/blob/v17.6.0/docs/manifest-releaser.md), +[App token action](https://github.com/actions/create-github-app-token/tree/v3.2.0), +[reusable workflows](https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows). diff --git a/release-please-config.json b/release-please-config.json new file mode 100644 index 0000000..b82f2df --- /dev/null +++ b/release-please-config.json @@ -0,0 +1,15 @@ +{ + "$schema": "https://raw.githubusercontent.com/googleapis/release-please/v17.6.0/schemas/config.json", + "packages": { + ".": { + "release-type": "go", + "package-name": "go-lambda-edge", + "include-component-in-tag": false, + "include-v-in-tag": true, + "initial-version": "0.1.0", + "bump-minor-pre-major": true, + "bump-patch-for-minor-pre-major": false, + "changelog-path": "CHANGELOG.md" + } + } +} From f6a2624469d14a150d89e0f5911644db86449e31 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Mon, 21 Sep 2026 16:33:10 +0100 Subject: [PATCH 43/44] ci: pin workflow actions to latest stable releases --- .github/workflows/go.yml | 8 ++++---- docs/plan.md | 14 ++++++++++++-- docs/releases.md | 19 +++++++++++++++---- 3 files changed, 31 insertions(+), 10 deletions(-) diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index 5140433..d66a32d 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -20,10 +20,10 @@ jobs: matrix: go: ['1.27.0', '1.27.x'] steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: actions/setup-go@v7 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version: ${{ matrix.go }} cache-dependency-path: go.sum @@ -44,10 +44,10 @@ jobs: matrix: arch: [arm64, amd64] steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: actions/setup-go@v7 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version: '1.27.x' cache-dependency-path: go.sum diff --git a/docs/plan.md b/docs/plan.md index 4351dbb..21fbe54 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -451,8 +451,18 @@ records the approved GitHub App credential and initial v0.1.0, with pre-major versioning. Release Please will run after reusable main validation; PRs continue to use the same Go checks. Local wiring and validation are complete: both workflows pass actionlint, release JSON passes the bundled schema, and existing Go jobs are -unchanged. See the [release guide](releases.md). App installation/credential setup -and actual GitHub execution remain pending; nothing was pushed or published. +unchanged. See the [release guide](releases.md). + +Subsequent authorized setup and publication of the working branch are complete: +the App client ID variable and private-key secret are configured, and +[PR #1](https://github.com/asteroid-computing/go-lambda-edge/pull/1) is open. +[The first PR run](https://github.com/asteroid-computing/go-lambda-edge/actions/runs/35619179518) +passed both Go test jobs and both Linux build jobs at commit 6eb3167. The action +review verified all four latest stable releases and pinned checkout/setup-go to +their release commits; those aliases already selected the same versions. App +installation, permissions and the main-branch release path still need execution +evidence. No release was published, and the license and deployment qualifications +remain open. ## Findings carried forward from the Beakley review diff --git a/docs/releases.md b/docs/releases.md index e20f541..ad409eb 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -80,10 +80,21 @@ in this automation, and a passing CI run does not close deployment qualification ## Maintenance -Release Please Action v5.0.0 and Create GitHub App Token v3.2.0 are pinned to -reviewed commit SHAs with version comments. Review upstream changes when updating -them. The config schema is pinned to release-please 17.6.0, bundled by this action. -No runtime version API or generated version.go file is required. +All published actions are pinned to reviewed commit SHAs with version comments. +The latest stable upstream releases were verified on 2026-09-21: + +| Action | Stable release | +| --- | --- | +| actions/checkout | [v7.0.1](https://github.com/actions/checkout/releases/tag/v7.0.1) | +| actions/setup-go | [v7.0.0](https://github.com/actions/setup-go/releases/tag/v7.0.0) | +| actions/create-github-app-token | [v3.2.0](https://github.com/actions/create-github-app-token/releases/tag/v3.2.0) | +| googleapis/release-please-action | [v5.0.0](https://github.com/googleapis/release-please-action/releases/tag/v5.0.0) | + +The previous checkout/setup-go v7 aliases resolved to these same commits when +checked; pinning makes that selection reproducible. Review upstream release notes +and resolve each release tag to its commit when updating the SHA and version +comment together. The config schema is pinned to release-please 17.6.0, bundled +by the release action. No runtime version API or generated version.go is required. Validate workflow edits locally with `actionlint`; validate config edits against the referenced JSON schema. A real GitHub run is still needed to verify App From f45c70dfa73317a879658be3304ca536c21c1874 Mon Sep 17 00:00:00 2001 From: AJ Beck Date: Sun, 27 Sep 2026 00:58:05 +0100 Subject: [PATCH 44/44] docs: refresh CI and release readiness evidence --- docs/decisions/0024-release-automation.md | 12 +++-- docs/decisions/0025-main-branch-protection.md | 52 +++++++++++++++++++ docs/decisions/README.md | 3 +- docs/plan.md | 19 +++---- docs/release-readiness.md | 31 +++++++---- 5 files changed, 93 insertions(+), 24 deletions(-) create mode 100644 docs/decisions/0025-main-branch-protection.md diff --git a/docs/decisions/0024-release-automation.md b/docs/decisions/0024-release-automation.md index f281e01..c9e1108 100644 --- a/docs/decisions/0024-release-automation.md +++ b/docs/decisions/0024-release-automation.md @@ -111,6 +111,12 @@ configuration validates against the bundled release-please 17.6.0 schema. Local structure checks confirm PR triggers, reusable validation, release's dependency on validation, scoped credential use, and unchanged Go test/build jobs. No Go source changed, so runtime suites were not rerun for this workflow-only change. -No App was installed, credentials provisioned, workflow dispatched, repository -settings changed, or release published. Remote activation still needs App setup -and a successful GitHub run. +That local implementation did not install an App, provision credentials, dispatch +a workflow, change repository settings or publish a release. + +Subsequent authorized setup, verified on 2026-09-27: the repository variable +RELEASE_PLEASE_CLIENT_ID and encrypted Actions secret RELEASE_PLEASE_PRIVATE_KEY +are provisioned. The working branch was pushed and PR #1 opened. +[All four PR checks at f6a2624 passed](https://github.com/asteroid-computing/go-lambda-edge/actions/runs/35619720590). +App installation/permissions and the main-branch release path still need execution +evidence after merge. No release has been published. diff --git a/docs/decisions/0025-main-branch-protection.md b/docs/decisions/0025-main-branch-protection.md new file mode 100644 index 0000000..bfdd1b2 --- /dev/null +++ b/docs/decisions/0025-main-branch-protection.md @@ -0,0 +1,52 @@ +# 0025: Enforce main-branch validation + +Status: proposed on 2026-09-27; awaiting owner review. No repository policy has +been changed. + +Owner direction on 2026-09-27: update readiness documentation, then the owner will +merge and return to the outstanding AWS qualifications. Branch protection was +not approved as part of that instruction; this proposal remains unimplemented. + +## Evidence + +PR #1 at f6a2624469d14a150d89e0f5911644db86449e31 is mergeable, with all four +[Go checks passing](https://github.com/asteroid-computing/go-lambda-edge/actions/runs/35619720590). +GitHub's main-branch endpoint reports protected=false, and the effective branch +rules endpoint returns no rules. The PR has no submitted GitHub reviews. + +The workflow implements PR validation, but cannot itself require success before +merge. The release workflow validates main again before running Release Please. +[GitHub's protected-branch documentation](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) +supports required PRs, required checks with an expected source, up-to-date branch +requirements, and restrictions on force pushes and deletion. Protection of this +private organization repository depends on the organization's GitHub plan; +availability has not been verified. + +## Recommendation and alternatives + +Before merging, require a PR and these four checks from GitHub Actions on main: +test (1.27.0), test (1.27.x), lambda-build (arm64), lambda-build (amd64). +Require the branch to be current with main, block force pushes and deletion, and +apply the policy without a routine administrator or release-App bypass. The App +creates release branches and PRs; it does not need to push directly to main. +Do not introduce a mandatory reviewer count without an owner decision about +reviewer availability. + +This makes the existing validation an enforced merge condition. The alternative +is to retain manual inspection of checks before every merge; it avoids repository +configuration but permits accidental merges with missing or failing checks. + +Consequences: direct main pushes are prevented, and updating a PR after main +advances can require another CI run. Check-name changes must also update policy. +An unavailable repository feature requires an explicit fallback decision. + +## Related readiness observations + +Refresh the release checklist's historical CI/credential status. Credentials +have been provisioned and PR CI has passed; App installation/permissions and the +main release workflow still require execution evidence. Squash-merge the +foundation with its Conventional Commit feat title, then verify the generated +release PR and its checks. The license and documented live AWS qualifications +remain separate release-readiness items under decisions 0023 and 0024. + +No merge, policy change, release or live AWS test is authorized by this record. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 460ab1a..107bca2 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -32,4 +32,5 @@ Keep related decisions small enough to review and implement independently. | [0021: Authorization rules](0021-authorization-rules.md) | Accepted; rules, examples and local validation implemented | What rule API, IAM/JWT predicates, failure semantics and consumer resolution/dispatch boundaries should authz expose? | | [0022: Streaming writer review](0022-streaming-writer-review.md) | Accepted; writer, public entry points and local validation complete; SDK deployment gate retained | Should streaming omit inferred lengths, how should empty Content-Encoding affect sniffing, and what SDK evidence remains missing? | | [0023: Consumer readiness](0023-consumer-readiness.md) | R1–R5 accepted; local implementation complete; external release prerequisites open | What complete consumer example, test support, deferred scope and release preparation should follow the implemented packages? | -| [0024: Release automation](0024-release-automation.md) | Accepted; locally implemented and validated; App setup/remote run pending | How should release PRs, validation, credentials and initial versioning work? | +| [0024: Release automation](0024-release-automation.md) | Accepted; credentials provisioned and PR CI passed; main release execution pending | How should release PRs, validation, credentials and initial versioning work? | +| [0025: Main-branch protection](0025-main-branch-protection.md) | Proposed; awaiting owner review | Should the existing four CI checks and PR workflow be enforced before merging to main? | diff --git a/docs/plan.md b/docs/plan.md index 21fbe54..ea63005 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -428,10 +428,11 @@ helpers. The license/copyright choice remains open and does not block local work guide using the existing public APIs, with local provider fixtures. - [x] Reconcile support/status documentation, update the indirect Smithy patch in isolation, and record incremental streaming baselines. -- [ ] Obtain actual GitHub CI evidence and resolve publication prerequisites - before a release. No push, tag, publication or deployment was authorized by - this review. Streaming deployment and live identity interoperability remain - separate qualifications. +- [x] Obtain actual GitHub CI evidence for the foundation PR; all four jobs + passed at f6a2624. Later commits require their own passing checks. +- [ ] Verify the main release workflow and resolve publication prerequisites + before a release. Streaming deployment and live identity interoperability + remain separate qualifications. The [complete dispatcher](../examples/dispatcher/README.md), [testing guide](consumer-testing.md), [support matrix](support.md) and [release checklist](release-readiness.md) are @@ -441,9 +442,9 @@ CI wiring are present. Smithy is updated to v1.28.2, and streaming allocation baselines are recorded. No library API or transport policy changed. Full local race tests, vet, module verification, formatting, external consumption -and Linux arm64/amd64 builds pass on Go 1.27.1 (2026-09-21). GitHub has no runs for -this branch; its actual CI evidence, the license and deployment qualifications -remain open. No push, publication or AWS deployment occurred. +and Linux arm64/amd64 builds pass on Go 1.27.1 (2026-09-21). The subsequent +authorized push and passing PR CI are recorded below. The license and deployment +qualifications remain open; no release publication or AWS deployment occurred. Release automation follow-up (2026-09-21): the user selected Release Please and its GitHub Action, plus checks on PRs against main. [Decision 0024](decisions/0024-release-automation.md) @@ -456,8 +457,8 @@ unchanged. See the [release guide](releases.md). Subsequent authorized setup and publication of the working branch are complete: the App client ID variable and private-key secret are configured, and [PR #1](https://github.com/asteroid-computing/go-lambda-edge/pull/1) is open. -[The first PR run](https://github.com/asteroid-computing/go-lambda-edge/actions/runs/35619179518) -passed both Go test jobs and both Linux build jobs at commit 6eb3167. The action +[The PR run at f6a2624](https://github.com/asteroid-computing/go-lambda-edge/actions/runs/35619720590) +passed both Go test jobs and both Linux build jobs, verified on 2026-09-27. The action review verified all four latest stable releases and pinned checkout/setup-go to their release commits; those aliases already selected the same versions. App installation, permissions and the main-branch release path still need execution diff --git a/docs/release-readiness.md b/docs/release-readiness.md index 48af5e3..30a0b82 100644 --- a/docs/release-readiness.md +++ b/docs/release-readiness.md @@ -16,21 +16,30 @@ v0.1.0 under decision 0024. [The release guide](releases.md) documents the workf and required App setup. Merging a generated release PR is the publication decision; automation does not resolve the remaining readiness items below. -## Outstanding evidence and owner choices +## Verified readiness (2026-09-27) + +[PR #1](https://github.com/asteroid-computing/go-lambda-edge/pull/1) is open on +`work/edge-foundation`. All four jobs in +[the run for f6a2624](https://github.com/asteroid-computing/go-lambda-edge/actions/runs/35619720590) +passed: test (1.27.0), test (1.27.x), lambda-build (arm64) and lambda-build (amd64). +This provides GitHub evidence for formatting, race tests, vet, module +verification, external consumer checks and both Linux builds. Later commits +require their own passing checks. The same workflow validates main pushes before +Release Please runs. -GitHub returned no runs for `work/edge-foundation` during this milestone. The -workflow covers Go 1.27.0/latest 1.27, race tests, vet, module verification, -external consumer checks and both Linux architectures. It runs directly for PRs -against main and is reused by the release workflow on main pushes. Obtain a passing run for -the actual release commit after an authorized push. Local tests do not establish -the Go 1.27.0 matrix result. +The repository variable RELEASE_PLEASE_CLIENT_ID and encrypted Actions secret +RELEASE_PLEASE_PRIVATE_KEY are provisioned; their names were verified on +2026-09-27 without reading the private key. No branch protection was configured. + +## Outstanding evidence and owner choices -Install/configure the release App and provide RELEASE_PLEASE_CLIENT_ID and -RELEASE_PLEASE_PRIVATE_KEY before activating releases. No App credentials or -repository policies were changed during local implementation. +After the foundation PR merges, verify main validation, App token creation and +the generated v0.1.0 release PR and its checks. Credential provisioning does not +establish App installation, granted permissions or successful release execution. +Obtain passing validation for the actual release commit before publication. The owner's license and copyright holder remain undecided. No license, repository -visibility change, push, tag or publication has been performed in this milestone. +visibility change, release tag or release publication has been performed. Resolve the license before external publication. REST streaming's SDK deployment qualification remains open: local probes cannot