fix(cfn): schedule the missing server tasks on the CloudFormation stack

cloudformation/stacks/CUDly/template.yaml

@@ -37,6 +37,22 @@ Metadata:
           - LogRetentionPeriod
           - ExecutionFrequency
 
+      - Label:
+          default: Scheduled Tasks
+        Parameters:
+          - EnableFinalizeRevocationsSchedule
+          - FinalizeRevocationsSchedule
+          - EnableAnalyticsCollectSchedule
+          - AnalyticsCollectSchedule
+          - EnableReapStuckPurchasesSchedule
+          - ReapStuckPurchasesSchedule
+          - EnableFireScheduledPurchasesSchedule
+          - FireScheduledPurchasesSchedule
+          - EnableSendNotificationsSchedule
+          - SendNotificationsSchedule
+          - EnableCleanupSchedule
+          - CleanupSchedule
+
     ParameterLabels:
       EmailAddress:
         default: Notification email address
@@ -140,6 +156,103 @@ Parameters:
     Default: "rate(1 day)"
     Description: How often to check for new recommendations
 
+  # Scheduled server tasks (#712). Defaults and cadences match the Terraform
+  # Lambda module (terraform/modules/compute/aws/lambda/variables.tf) except
+  # where noted. Opt-in tasks start purchases or retry them, so each stack
+  # enables them deliberately.
+  EnableFinalizeRevocationsSchedule:
+    Type: String
+    Default: "true"
+    AllowedValues:
+      - "true"
+      - "false"
+    Description: Run the finalize_revocations task on a schedule (idempotent audit repair of revoked purchases)
+
+  FinalizeRevocationsSchedule:
+    Type: String
+    Default: "rate(1 hour)"
+    Description: EventBridge schedule expression for finalize_revocations
+
+  EnableAnalyticsCollectSchedule:
+    Type: String
+    Default: "true"
+    AllowedValues:
+      - "true"
+      - "false"
+    Description: Run the analytics_collect task on a schedule (savings snapshot, partition maintenance, retention and view refresh)
+
+  AnalyticsCollectSchedule:
+    Type: String
+    Default: "rate(1 day)"
+    Description: EventBridge schedule expression for analytics_collect
+
+  EnableReapStuckPurchasesSchedule:
+    Type: String
+    Default: "false"
+    AllowedValues:
+      - "true"
+      - "false"
+    Description: >-
+      Run the reap_stuck_purchases task on a schedule. It marks executions stuck
+      in running (after 10 minutes) or approved (after 1h20m) as failed. Off by
+      default on this stack, unlike the Terraform Lambda module: the
+      PurchaseCheck rule runs process_scheduled_purchases only daily, so an
+      approved execution stranded by a crash would be failed before the next
+      process run re-drives it (#894).
+
+  ReapStuckPurchasesSchedule:
+    Type: String
+    Default: "rate(5 minutes)"
+    Description: EventBridge schedule expression for reap_stuck_purchases
+
+  EnableFireScheduledPurchasesSchedule:
+    Type: String
+    Default: "false"
+    AllowedValues:
+      - "true"
+      - "false"
+    Description: >-
+      Run the fire_scheduled_purchases task on a schedule. It executes
+      purchases in status scheduled whose scheduled time has passed (the
+      delayed auto-approve path), so it makes real purchases. Opt-in.
+
+  FireScheduledPurchasesSchedule:
+    Type: String
+    Default: "rate(15 minutes)"
+    Description: EventBridge schedule expression for fire_scheduled_purchases
+
+  EnableSendNotificationsSchedule:
+    Type: String
+    Default: "false"
+    AllowedValues:
+      - "true"
+      - "false"
+    Description: Run the send_notifications task on a schedule (upcoming-purchase emails). Opt-in.
+
+  SendNotificationsSchedule:
+    Type: String
+    Default: "rate(1 day)"
+    Description: EventBridge schedule expression for send_notifications
+
+  EnableCleanupSchedule:
+    Type: String
+    Default: "false"
+    AllowedValues:
+      - "true"
+      - "false"
+    Description: >-
+      Run the server cleanup task on a schedule. Opt-in. It deletes expired
+      sessions and, via CleanupOldExecutions(30), purchase executions that are
+      completed with scheduled_date older than 30 days, canceled with
+      updated_at older than 30 days, or in a status outside failed/canceled
+      with expires_at older than 30 days. The first run deletes the whole
+      backlog, so export any history you want to keep first.
+
+  CleanupSchedule:
+    Type: String
+    Default: "rate(1 day)"
+    Description: EventBridge schedule expression for cleanup
+
   CrossAccountTargetAccountIds:
     Type: CommaDelimitedList
     Default: ""
@@ -190,6 +303,36 @@ Conditions:
               - Ref: CrossAccountTargetAccountIds
           - ""
 
+  ScheduleFinalizeRevocations:
+    Fn::Equals:
+      - Ref: EnableFinalizeRevocationsSchedule
+      - "true"
+
+  ScheduleAnalyticsCollect:
+    Fn::Equals:
+      - Ref: EnableAnalyticsCollectSchedule
+      - "true"
+
+  ScheduleReapStuckPurchases:
+    Fn::Equals:
+      - Ref: EnableReapStuckPurchasesSchedule
+      - "true"
+
+  ScheduleFireScheduledPurchases:
+    Fn::Equals:
+      - Ref: EnableFireScheduledPurchasesSchedule
+      - "true"
+
+  ScheduleSendNotifications:
+    Fn::Equals:
+      - Ref: EnableSendNotificationsSchedule
+      - "true"
+
+  ScheduleCleanup:
+    Fn::Equals:
+      - Ref: EnableCleanupSchedule
+      - "true"
+
 Resources:
   # =============================================================================
   # DynamoDB Tables
@@ -733,6 +876,144 @@ Resources:
       Principal: events.amazonaws.com
       SourceArn: !GetAtt PurchaseCheckRule.Arn
 
+  # Remaining scheduled server tasks (#712), one rule and invoke permission
+  # each, switched by the Enable*Schedule parameters. Name suffixes are at most
+  # as long as "-ScheduledExecution", so they fit EventBridge's 64-character
+  # limit for any stack name the existing rules accept. Retries stay at the
+  # EventBridge default: a re-run is safe under the server's per-task advisory
+  # lock and row compare-and-swap.
+  FinalizeRevocationsRule:
+    Type: AWS::Events::Rule
+    Condition: ScheduleFinalizeRevocations
+    Properties:
+      Name: !Sub "${AWS::StackName}-Finalize"
+      Description: Scheduled finalize_revocations task
+      ScheduleExpression: !Ref FinalizeRevocationsSchedule
+      State: ENABLED
+      Targets:
+        - Id: LambdaTarget
+          Arn: !GetAtt LambdaFunction.Arn
+          Input: '{"action": "finalize_revocations"}'
+
+  FinalizeRevocationsPermission:
+    Type: AWS::Lambda::Permission
+    Condition: ScheduleFinalizeRevocations
+    Properties:
+      Action: lambda:InvokeFunction
+      FunctionName: !Ref LambdaFunction
+      Principal: events.amazonaws.com
+      SourceArn: !GetAtt FinalizeRevocationsRule.Arn
+
+  AnalyticsCollectRule:
+    Type: AWS::Events::Rule
+    Condition: ScheduleAnalyticsCollect
+    Properties:
+      Name: !Sub "${AWS::StackName}-AnalyticsCollect"
+      Description: Scheduled analytics_collect task
+      ScheduleExpression: !Ref AnalyticsCollectSchedule
+      State: ENABLED
+      Targets:
+        - Id: LambdaTarget
+          Arn: !GetAtt LambdaFunction.Arn
+          Input: '{"action": "analytics_collect"}'
+
+  AnalyticsCollectPermission:
+    Type: AWS::Lambda::Permission
+    Condition: ScheduleAnalyticsCollect
+    Properties:
+      Action: lambda:InvokeFunction
+      FunctionName: !Ref LambdaFunction
+      Principal: events.amazonaws.com
+      SourceArn: !GetAtt AnalyticsCollectRule.Arn
+
+  ReapStuckPurchasesRule:
+    Type: AWS::Events::Rule
+    Condition: ScheduleReapStuckPurchases
+    Properties:
+      Name: !Sub "${AWS::StackName}-ReapStuck"
+      Description: Scheduled reap_stuck_purchases task
+      ScheduleExpression: !Ref ReapStuckPurchasesSchedule
+      State: ENABLED
+      Targets:
+        - Id: LambdaTarget
+          Arn: !GetAtt LambdaFunction.Arn
+          Input: '{"action": "reap_stuck_purchases"}'
+
+  ReapStuckPurchasesPermission:
+    Type: AWS::Lambda::Permission
+    Condition: ScheduleReapStuckPurchases
+    Properties:
+      Action: lambda:InvokeFunction
+      FunctionName: !Ref LambdaFunction
+      Principal: events.amazonaws.com
+      SourceArn: !GetAtt ReapStuckPurchasesRule.Arn
+
+  FireScheduledPurchasesRule:
+    Type: AWS::Events::Rule
+    Condition: ScheduleFireScheduledPurchases
+    Properties:
+      Name: !Sub "${AWS::StackName}-FireScheduled"
+      Description: Scheduled fire_scheduled_purchases task
+      ScheduleExpression: !Ref FireScheduledPurchasesSchedule
+      State: ENABLED
+      Targets:
+        - Id: LambdaTarget
+          Arn: !GetAtt LambdaFunction.Arn
+          Input: '{"action": "fire_scheduled_purchases"}'
+
+  FireScheduledPurchasesPermission:
+    Type: AWS::Lambda::Permission
+    Condition: ScheduleFireScheduledPurchases
+    Properties:
+      Action: lambda:InvokeFunction
+      FunctionName: !Ref LambdaFunction
+      Principal: events.amazonaws.com
+      SourceArn: !GetAtt FireScheduledPurchasesRule.Arn
+
+  SendNotificationsRule:
+    Type: AWS::Events::Rule
+    Condition: ScheduleSendNotifications
+    Properties:
+      Name: !Sub "${AWS::StackName}-SendNotifications"
+      Description: Scheduled send_notifications task
+      ScheduleExpression: !Ref SendNotificationsSchedule
+      State: ENABLED
+      Targets:
+        - Id: LambdaTarget
+          Arn: !GetAtt LambdaFunction.Arn
+          Input: '{"action": "send_notifications"}'
+
+  SendNotificationsPermission:
+    Type: AWS::Lambda::Permission
+    Condition: ScheduleSendNotifications
+    Properties:
+      Action: lambda:InvokeFunction
+      FunctionName: !Ref LambdaFunction
+      Principal: events.amazonaws.com
+      SourceArn: !GetAtt SendNotificationsRule.Arn
+
+  CleanupRule:
+    Type: AWS::Events::Rule
+    Condition: ScheduleCleanup
+    Properties:
+      Name: !Sub "${AWS::StackName}-Cleanup"
+      Description: Scheduled cleanup task
+      ScheduleExpression: !Ref CleanupSchedule
+      State: ENABLED
+      Targets:
+        - Id: LambdaTarget
+          Arn: !GetAtt LambdaFunction.Arn
+          Input: '{"action": "cleanup"}'
+
+  CleanupPermission:
+    Type: AWS::Lambda::Permission
+    Condition: ScheduleCleanup
+    Properties:
+      Action: lambda:InvokeFunction
+      FunctionName: !Ref LambdaFunction
+      Principal: events.amazonaws.com
+      SourceArn: !GetAtt CleanupRule.Arn
+
   # =============================================================================
   # SNS Topic for Notifications
   # =============================================================================

docs/DEPLOYMENT.md

@@ -592,11 +592,12 @@ Each cloud provider has lifecycle policies configured via Terraform (`terraform/
 
 ### Database Cleanup Jobs
 
-The server's `cleanup` task deletes expired sessions and, via `CleanupOldExecutions(30)`, purchase executions that are completed with `scheduled_date` older than 30 days, canceled with `updated_at` older than 30 days, or in any status other than failed or canceled with `expires_at` older than 30 days. Failed executions are kept. Azure Container Apps schedules it by default; AWS Lambda can schedule it on request:
+The server's `cleanup` task deletes expired sessions and, via `CleanupOldExecutions(30)`, purchase executions that are completed with `scheduled_date` older than 30 days, canceled with `updated_at` older than 30 days, or in any status other than failed or canceled with `expires_at` older than 30 days. Failed executions are kept. Azure Container Apps schedules it by default; AWS Lambda and CloudFormation can schedule it on request:
 
 - **Azure Container Apps (when `enable_scheduled_tasks` is true):** a Logic App calls `/api/scheduled/cleanup` daily (`terraform/modules/compute/azure/container-apps/scheduled-tasks.tf`)
 - **AWS Lambda (opt-in):** set `enable_cleanup_schedule = true` (default false; `cleanup_schedule` defaults to `rate(1 day)`) in `terraform/environments/aws`. The first run deletes the whole backlog older than 30 days, so export any history you want to keep first.
-- **AWS Fargate and CloudFormation:** nothing schedules it yet. The `terraform/modules/compute/aws/cleanup-lambda/` module is not consumed by any environment.
+- **CloudFormation (opt-in):** set the `EnableCleanupSchedule` stack parameter to `true` (default `false`; `CleanupSchedule` defaults to `rate(1 day)`) on `cloudformation/stacks/CUDly`. The same backlog warning applies.
+- **AWS Fargate:** nothing schedules it yet. The `terraform/modules/compute/aws/cleanup-lambda/` module is not consumed by any environment.
 - **GCP:** nothing schedules it. The `terraform/modules/compute/gcp/cleanup-function/` module is not consumed by any environment.
 
 AKS has no scheduler (#893). Scheduling for AWS Fargate, CloudFormation and GCP is tracked by #712.

known_issues/33_executions_retention_cleanup.md

@@ -9,7 +9,7 @@
 > schedules cleanup on AWS. The same holds for the Terraform behind the
 > "matches what the cleanup-lambda Terraform actually configures" sentence
 > under "Original problem". Only Azure Container Apps schedules cleanup today. Retention
-> scheduling is tracked by #712; retention is scheduled via the server `cleanup` task on AWS Lambda when opted in.
+> scheduling is tracked by #712; retention is scheduled via the server `cleanup` task on AWS Lambda and CloudFormation when opted in.
 
 ## Resolution
 
@@ -21,14 +21,17 @@
   `terraform/modules/compute/aws/cleanup-lambda/main.tf:98`~~ (unconsumed module, see amendment)
 - **AWS Lambda (opt-in):** the `enable_cleanup_schedule` / `cleanup_schedule`
   sweep `<stack>-sweep-cleanup` runs the server cleanup task.
+- **CloudFormation (opt-in):** the `EnableCleanupSchedule` /
+  `CleanupSchedule` stack parameters add a `<stack>-Cleanup` rule that runs
+  the server cleanup task.
 - **Azure Container Apps:** a Logic App calls `/api/scheduled/cleanup` daily
   when `enable_scheduled_tasks` is true. (The `azure/cleanup-function` module
   is not consumed by any environment, and AKS has no scheduler, #893.)
 
 The `CleanupOldExecutions` Postgres implementation in
 `internal/config/store_postgres.go` runs the expected
 `DELETE FROM purchase_executions WHERE …`, so the table is bounded
-only where that task is scheduled (Azure Container Apps by default, AWS Lambda when opted in, see amendment).
+only where that task is scheduled (Azure Container Apps by default, AWS Lambda and CloudFormation when opted in, see amendment).
 
 ## Original problem (historical context)