docs(connectors): fix the inverted serde_secret redaction claim - #3803
Conversation
|
Thanks for the PR. It is labeled Slash commands (own line, regular comment) move it around the queue:
See CONTRIBUTING.md for details. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3803 +/- ##
============================================
- Coverage 76.75% 74.37% -2.39%
- Complexity 1021 1316 +295
============================================
Files 1365 1377 +12
Lines 172646 168303 -4343
Branches 142575 137433 -5142
============================================
- Hits 132522 125173 -7349
- Misses 36301 38953 +2652
- Partials 3823 4177 +354
🚀 New features to boost your workflow:
|
76a7612 to
fd1e1cc
Compare
|
/request-review @hubcio |
hubcio
left a comment
There was a problem hiding this comment.
follow-up outside this diff: runtime/src/api/config.rs Debug impl hardcodes "[REDACTED]", which now duplicates the new REDACTED const. same literal is also hardcoded in core/common auth credentials, server state models and the s3_sink URL redaction - worth a small sweep to the const later.
|
Thanks, both wording findings were correct and the second one was load-bearing rather than cosmetic. Pushed
The inventory. Scoped to plugin-side callers, and it now says explicitly that the others are not all mistakes, naming On the paired A Happy to add either form now if you would rather have it mechanical. On the Gate: fmt, sort, clippy |
|
/ready |
|
/request-review @hubcio |
`serialize_secret` and `serialize_optional_secret` write the plaintext, which leaves an author who needs `Serialize` on a struct holding a credential with no redacting option — so they reach for the exposing one and believe it protects them. That is the trap apache#3801 documents. `serialize_redacted` and `serialize_optional_redacted` write a placeholder instead. The optional form keeps `Some` distinguishable from `None`: whether a credential is configured is not itself secret, and collapsing it to null would report a configured field as unset. The module doc now leads with what these helpers actually do, since the absent `Serialize` impl on `SecretString` is the protection and any helper here is a decision to give it up.
The Secrets guidance told plugin authors that annotating a `SecretString` with `serialize_secret` made serialization redact. It does the opposite: the helper calls `expose_secret()`. `SecretString` has no `Serialize` impl precisely so a struct holding one cannot be serialized, so adding the attribute is what unblocks the derive and gives up the guarantee. Nine plugins followed that guidance. The claim appeared twice: in the Secrets prose and again as an "Auto-redact on Debug/Display + serialization" row in the patterns table, where it read as a recommended pairing. Both now say which half was true - `Debug` does redact - and the section gives the default for a plugin config struct: do not derive `Serialize` at all, since nothing needs it and leaving it off makes the property compiler-enforced. It also names the redacting helpers for structs that genuinely need serialization, and notes that none of this protects against the runtime control API returning plugin config verbatim, which is apache#3802 and not fixable from the plugin side. Corrects the in-tree list too: it named delta_sink, which does not use these helpers, and omitted s3_sink and surrealdb_sink, which do.
Review caught two wrong statements in the guidance this PR was fixing.
"The runtime never deserializes into a plugin's config struct" is false:
the SDK glue does exactly that, `serde_json::from_str::<C>` under a
`DeserializeOwned` bound in `sdk/src/{sink,source}.rs`, so `Deserialize`
stays required and the advice now says which half to drop. The property
that actually carries the argument is that nothing ever re-serializes
the struct. Plugin configuration also does not only come from TOML; the
control API accepts it as JSON and env vars can inject it.
The in-tree list read as an exhaustive inventory while covering only
plugins, which framed two deliberate uses as oversights: the runtime's
own `HttpConfig::api_key`, and the `core/common` wire payloads for
login, create-user, change-password and PAT, where the credential is the
payload. Scoped the sentence to plugin-side callers.
Also answers the round-trip question raised in review. The doc now names
the failure directly: deserializing redacted output hands back the
placeholder as the secret rather than failing. Left mechanical rather
than adding a paired `deserialize_with`, because that guard is itself
opt-in and a caller who forgets it is exactly the case it claims to
cover. A newtype owning both directions is the shape that cannot be
half-applied, and it should be designed against a real consumer.
3cd136f to
0ddc886
Compare
|
The red on this PR after the rebase is a Maven Central rate limit, not the diff. Flagging it so it does not read as a real failure.
It never reached a step; the failure is in This PR touches I cannot re-run it myself ( Local gate on the rebased branch is unchanged and green: fmt, sort, clippy |
Closes #3801.
What was wrong
.claude/skills/connectors-overview/SKILL.mdtold plugin authors that annotatinga
SecretStringfield withiggy_common::serde_secret::serialize_secretmadeDebugand serialization redact. Only theDebughalf is true. The helpercalls
expose_secret()and writes the plaintext:serde_secret.rs's own module doc already said the opposite of the skill("Do not add
serialize_withto fields that should remain redacted"), so thetwo documents contradicted each other and the skill is the one plugin authors
read.
The inversion matters more than a typo would, because
SecretStringhas noSerializeimpl by design — a struct holding one cannot deriveSerializeatall. Adding the attribute is what unblocks the derive. The guidance therefore
recommended the exact step that converts a compile-time guarantee into plaintext
output, while describing it as protection.
Nine plugins carry the annotation on credential fields. It is inert today (the
runtime keeps plugin config as
serde_json::Valueand never deserializes into aplugin's config struct, so nothing calls these serializers), so this is a
correctness-of-documentation fix rather than a live leak — but the protection
those authors believe they have does not exist.
What this changes
docs(connectors)— the Secrets section now says which half of the claim wastrue, and gives the default: do not derive
Serializeon a plugin configstruct at all, since nothing needs it and leaving it off makes the property
compiler-enforced rather than convention-enforced.
iggy_connector_http_source(#3798) does exactly that.It also notes that none of this protects the credential from the runtime's own
control API, which returns plugin configuration verbatim — that is #3802, and it
is not addressable from the plugin side.
While in there: the "In-tree uses" list named
delta_sink, which does not usethese helpers, and omitted
s3_sinkandsurrealdb_sink, which do.feat(common)— addsserialize_redactedandserialize_optional_redactedso the corrected guidance has something to point at. Suggestion 3 in the issue.
Without them, an author who genuinely needs
Serializehas no redacting optionand reaches for the exposing one, which is the trap. The optional form keeps
Somedistinguishable fromNone: whether a credential is configured is notitself secret, and collapsing it to
nullwould report a configured field asunset. Documented as not round-tripping, so nobody feeds redacted output back
into a config loader.
Deliberately not in scope
Suggestion 2 in the issue — dropping
Serializefrom the nine plugin configstructs that do not need it. It is the right follow-up and it is safe, but it
touches nine crates and each needs checking for a real serializing caller, so it
does not belong in the same review as the documentation fix. Happy to do it as a
separate PR; say the word and I will.
Verification
cargo fmt --all --check,cargo sort --check --no-format --workspace,cargo clippy -p iggy_common --all-features --all-targets -- -D warnings,cargo test -p iggy_common serde_secret(6 pass),taplo fmt --check,hawkeye check,typos,markdownlint, trailing whitespace/newline — all exit0.