Redact well-known secret formats in logs and rendered fields - #70956
Open
dkranchii wants to merge 1 commit into
Open
Redact well-known secret formats in logs and rendered fields#70956dkranchii wants to merge 1 commit into
dkranchii wants to merge 1 commit into
Conversation
Airflow's SecretsMasker decides what to hide by key name, not by the shape of the value. As the current docs acknowledge, keys in a Connection extra whose names do not match a sensitive keyword are not redacted, and values that flow through XCom or any other side channel reach Task logs and Rendered fields as-is even on a correctly configured deployment. Users routinely leak live credentials this way by printing environment variables during debugging, logging stack traces that include a token or PEM block, or pushing values through XCom keys whose names do not happen to match the sensitive-keyword list. Registering secrets ahead of time via mask_secret(), Connections, and Variables only covers values Airflow was told about. This adds a second line of defense so that even when Airflow was never told a specific value was sensitive, values that match a small curated set of well-known credential formats are redacted before they reach logs or rendered fields. The feature is opt-in (default False) so no existing deployment changes behavior. The initial pattern set is deliberately narrow - each entry has a distinctive fixed prefix so a match is overwhelmingly likely to be a real credential, and formats with high false-positive rates in log data are left out of the built-ins. Deployments that want additional formats can register them via add_content_patterns() rather than editing Airflow. All patterns use bounded or fixed-width bodies so the regex engine's work stays strictly linear on any input. related: apache#58514
dkranchii
force-pushed
the
content-pattern-masking
branch
from
August 3, 2026 07:04
cb6dca6 to
863d632
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Airflow's SecretsMasker decides what to hide by key name today. Values that reach Task logs via XCom, a print of an environment variable, a stack trace, or a Connection extra whose key doesn't happen to match a sensitive keyword are not redacted — as mask-sensitive-values.rst documents and #58514 previously reported (closed as a docs-only clarification, so the underlying gap remained).
This PR adds an opt-in, default-off second pass in SecretsMasker that scans string values for a small, curated set of well-known credential formats — AWS access keys, GitHub / Slack / Google / Stripe tokens, PEM private-key blocks, JWTs — and redacts any match. Because it plugs into the existing masker, it runs everywhere the masker already runs (log filter, redact(), rendered fields).
related: #58514
Design
Opt-in behavior. Enable via
[core] mask_secrets_content_patterns(defaultFalse). No existing deployment changes behavior.Curated built-in patterns. Seven formats, each with a distinctive fixed prefix so match confidence is very high:
AKIA…/ASIA…gh[pousr]_…xox[baprs]-…AIza…sk_live_…-----BEGIN … PRIVATE KEY-----eyJ…eyJ…Formats with documented high false-positive rates in log data (credit cards, SSNs, email addresses) are deliberately left out of the defaults.
Pluggable. Deployments can register additional named regexes via
SecretsMasker.add_content_patterns({name: regex}). Invalid regexes are logged and skipped rather than raised, so a misconfigured entry can't disable the whole masker.ReDoS-audited. Every pattern uses fixed-width or bounded quantifiers — no nested quantifiers, no overlapping alternations, and no
\S*?+ negative-lookahead combo (the shape fixed in #70716). Worst case is a single linear scan of the input string.Extends existing surface.
SecretsMaskeris already alogging.Filtershared byairflow-coreand the task-SDK viashared/secrets_masker/. Adding a parallelSensitivityClassifiermodule would duplicate the recursive-walk path the masker already implements.Log-filter short-circuit widened. The
filter()guard wasif self.replacer:; it's nowif self.replacer or self.mask_content_patterns:. Without this, content-pattern masking would silently no-op in log paths when no explicitadd_mask()secrets are registered.Compatibility
No behavior change with the flag off.
Not a replacement for mask_secret() — this is defense in depth. Known secrets should still be registered explicitly.
No new runtime dependencies, no schema change, no new decorator.
Testing
25 new tests covering each built-in pattern, false-positive guardrails, add_content_patterns() registration and invalid-regex rejection, reset_masker() behavior, composition with existing key-name masking and with add_mask(), and the log-filter path when only content-pattern masking is enabled.
Full pre-existing suite: 167 passed, 1 skipped (pre-existing k8s import), 1 xfailed (pre-existing) — no regressions.
ruff format and ruff check clean.
Related
related: #58514 — same limitation, closed as docs-only clarification.
Independent context: #70716 (ReDoS fix in Spark provider) — reviewed my regexes against the same class of issue. #70890 (bulk audit-log masking) — different code path, no conflict.
Reviewer notes
Happy to trim the initial pattern set (e.g. drop JWTs, which have the highest false-positive rate of the seven) or flip the default in a follow-up PR once we have adoption data. Not adding a newsfragment yet; can add one on request.
Was generative AI tooling used to co-author this PR?
Yes — Cursor Agent (Opus 4.7)