Skip to content

Export concourse audit logs to s3 - #254

Merged
cweibel merged 7 commits into
mainfrom
export-concourse-audit-logs-to-s3
Feb 18, 2026
Merged

cweibel merged 7 commits into
mainfrom
export-concourse-audit-logs-to-s3

Conversation

@cweibel

@cweibel cweibel commented Feb 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes proposed in this pull request:

  • Drain the audit logs for Concourse to an s3 bucket. Stored in a YYYY/MM/DD/HH/MM/SS folder structure, similar to the cf events scraper that logs-opensearch performs. Files are actually gzipped unlike logs-opensearch.
  • Once opensearch for platform logs is available, an ingestor-s3 job can be configured to import them.
  • The s3 bucket (and iam creds to be later used by the ingestor), were created with Concourse audit logs bucket & iam user/role terraform-provision#2292.
  • The names of the buckets (dev/stage/prod) were copied to Concourse Credhub so they could be used by pipeline.yml in this repo.
  • The job runs hourly on a timer. If this interval changes, also need to change the duration logic in export-concourse-audit-logs-to-s3.sh
  • Part of https://github.com/cloud-gov/private/issues/2835

security considerations

Secrets are in Credhub. Logs are sent to private/encrypted bucket.

@cweibel
cweibel requested a review from a team as a code owner February 18, 2026 13:41

@ChrisMcGowan ChrisMcGowan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@cweibel
cweibel merged commit 3dcdaef into main Feb 18, 2026
4 checks passed
@cweibel
cweibel deleted the export-concourse-audit-logs-to-s3 branch February 18, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants