fix: stream-router contract — SequenceNumber batch failures and REMOVE record guard - #86
Merged
Merged
Conversation
AWS matches ReportBatchItemFailures identifiers against the DynamoDB stream record SequenceNumber. HandleStreamEvent returned the EventID, so Lambda did not recognise the identifier and re-drove the whole batch -- with bisect_batch_on_function_error and maximum_retry_attempts = 3 that replayed already-successful records whose WriteJobEvent and PublishEvent side effects are not idempotent. Adds the first test file to internal/lambda/stream.
handleRecord ignored EventName, so a deleted row reached the write handlers with a nil NewImage. Deleting a RERUN_REQUEST# row (those rows have no TTL and are never cleaned up) started a rerun with reason "manual"; deleting a SENSOR# row -- including the postrun-baseline# delete the rerun path itself issues -- could publish POST_RUN_PASSED or POST_RUN_FAILED; 30-day JOB# TTL expiries invoked the Lambda for nothing. REMOVE records are now skipped and logged at info with the pk, sk, event id and whether the delete came from TTL expiry. SK = CONFIG is the one exception: it still invalidates the config cache.
Defense in depth behind the handleRecord REMOVE guard. Both stream mappings now carry filter_criteria; the OR-ed filters pass INSERT and MODIFY on both tables, plus REMOVE of the CONFIG row on the control table so the config cache is still invalidated when a pipeline config is deleted. The joblog table holds no CONFIG rows, so it gets the writes filter only. Filtered records advance the stream iterator without invoking or billing the function, which also removes the 30-day JOB# TTL-expiry invocations. Requires terraform apply to take effect on existing deployments.
Mirrors the filter_criteria blocks added to deploy/terraform/lambda.tf so local E2E runs exercise the same delivery contract as production: writes always pass, and REMOVE passes only for the control table's CONFIG row. Existing mappings are left alone (the function is still idempotent), so a running stack needs a teardown and redeploy to pick the filters up.
Documents the SequenceNumber batch-failure fix and the REMOVE guard, including the event source mapping filters and the terraform apply that operators need to run to pick them up.
github-actions
Bot
added
tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes in the stream-router Lambda, plus matching filters on the DynamoDB event source mappings.
manual, deleting a sensor row (including the baseline delete the rerun path itself issues) could publish a post-run verdict, and 30-day job-log expiries invoked the function for nothing. Deletes are now skipped and logged, with TTL expiries identified. The one exception is a deleted pipeline config, which still invalidates the config cache.deploy/terraform/lambda.tf: inserts and updates always pass; deletes pass only for the control table's config row; the job-log mapping gets the writes filter only. This is an in-place update. Filtered records advance the stream without invoking or billing the function.deploy/localstack/deploy.py, verified end to end: the full local deploy and smoke check passed, and both mappings show the expected filters.internal/lambda/streampackage: failure identifier, delete guard (verified failing before the fix: writes, a state machine start, and a published event were observed for a deleted row), config cache invalidation, TTL detection, key extraction ordering.The deprecated copy in
internal/lambda/stream_router.gois untouched because its tests assert the old identifier; its doc comment now flags the known problems. It is removed when the root test suite is ported to the live packages.Operator note
Run
terraform applyafter merging to pick up the new event source mapping filters. The handler guard protects deployments in the meantime, and deploy order does not matter. An existing LocalStack stack needs a teardown and redeploy to get the filters.Test plan
make auditgreen at HEAD (lint clean, race suite passes)terraform -chdir=deploy/terraform validatesucceedsterraform apply: confirm both stream mappings show filter criteria and that deleting a rerun-request row no longer starts an executionFollow-ups
internal/lambdatest suite to the live sub-packages and delete the deprecated copies.