Skip to content

docs(connectors): fix the inverted serde_secret redaction claim - #3803

Merged
hubcio merged 5 commits into
apache:masterfrom
mlevkov:serde-secret-guidance
Aug 10, 2026
Merged

docs(connectors): fix the inverted serde_secret redaction claim#3803
hubcio merged 5 commits into
apache:masterfrom
mlevkov:serde-secret-guidance

Conversation

@mlevkov

@mlevkov mlevkov commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Closes #3801.

What was wrong

.claude/skills/connectors-overview/SKILL.md told plugin authors that annotating
a SecretString field with iggy_common::serde_secret::serialize_secret made
Debug and serialization redact. Only the Debug half is true. The helper
calls expose_secret() and writes the plaintext:

pub fn serialize_secret<S: serde::Serializer>(
    secret: &SecretString,
    serializer: S,
) -> Result<S::Ok, S::Error> {
    serializer.serialize_str(secret.expose_secret())
}

serde_secret.rs's own module doc already said the opposite of the skill
("Do not add serialize_with to fields that should remain redacted"), so the
two documents contradicted each other and the skill is the one plugin authors
read.

The inversion matters more than a typo would, because SecretString has no
Serialize impl by design — a struct holding one cannot derive Serialize at
all. Adding the attribute is what unblocks the derive. The guidance therefore
recommended the exact step that converts a compile-time guarantee into plaintext
output, while describing it as protection.

Nine plugins carry the annotation on credential fields. It is inert today (the
runtime keeps plugin config as serde_json::Value and never deserializes into a
plugin's config struct, so nothing calls these serializers), so this is a
correctness-of-documentation fix rather than a live leak — but the protection
those authors believe they have does not exist.

What this changes

docs(connectors) — the Secrets section now says which half of the claim was
true, and gives the default: do not derive Serialize on a plugin config
struct at all
, since nothing needs it and leaving it off makes the property
compiler-enforced rather than convention-enforced.
iggy_connector_http_source (#3798) does exactly that.

It also notes that none of this protects the credential from the runtime's own
control API, which returns plugin configuration verbatim — that is #3802, and it
is not addressable from the plugin side.

While in there: the "In-tree uses" list named delta_sink, which does not use
these helpers, and omitted s3_sink and surrealdb_sink, which do.

feat(common) — adds serialize_redacted and serialize_optional_redacted
so the corrected guidance has something to point at. Suggestion 3 in the issue.
Without them, an author who genuinely needs Serialize has no redacting option
and reaches for the exposing one, which is the trap. The optional form keeps
Some distinguishable from None: whether a credential is configured is not
itself secret, and collapsing it to null would report a configured field as
unset. Documented as not round-tripping, so nobody feeds redacted output back
into a config loader.

Deliberately not in scope

Suggestion 2 in the issue — dropping Serialize from the nine plugin config
structs that do not need it. It is the right follow-up and it is safe, but it
touches nine crates and each needs checking for a real serializing caller, so it
does not belong in the same review as the documentation fix. Happy to do it as a
separate PR; say the word and I will.

Verification

cargo fmt --all --check, cargo sort --check --no-format --workspace,
cargo clippy -p iggy_common --all-features --all-targets -- -D warnings,
cargo test -p iggy_common serde_secret (6 pass), taplo fmt --check,
hawkeye check, typos, markdownlint, trailing whitespace/newline — all exit
0.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 2, 2026
@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.37%. Comparing base (4a8394a) to head (08b3168).

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3803      +/-   ##
============================================
- Coverage     76.75%   74.37%   -2.39%     
- Complexity     1021     1316     +295     
============================================
  Files          1365     1377      +12     
  Lines        172646   168303    -4343     
  Branches     142575   137433    -5142     
============================================
- Hits         132522   125173    -7349     
- Misses        36301    38953    +2652     
- Partials       3823     4177     +354     
Components Coverage Δ
Rust Core 73.91% <100.00%> (-1.88%) ⬇️
Java SDK 66.40% <ø> (+3.18%) ⬆️
C# SDK 56.49% <ø> (-19.64%) ⬇️
Python SDK 89.98% <ø> (ø)
PHP SDK 82.97% <ø> (ø)
Node SDK 96.37% <ø> (+0.09%) ⬆️
Go SDK 69.13% <ø> (ø)
Files with missing lines Coverage Δ
core/common/src/utils/serde_secret.rs 100.00% <100.00%> (ø)

... and 225 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mlevkov
mlevkov force-pushed the serde-secret-guidance branch from 76a7612 to fd1e1cc Compare August 2, 2026 21:36
@mlevkov

mlevkov commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@github-actions
github-actions Bot requested a review from hubcio August 3, 2026 03:05

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow-up outside this diff: runtime/src/api/config.rs Debug impl hardcodes "[REDACTED]", which now duplicates the new REDACTED const. same literal is also hardcoded in core/common auth credentials, server state models and the s3_sink URL redaction - worth a small sweep to the const later.

Comment thread .claude/skills/connectors-overview/SKILL.md Outdated
Comment thread .claude/skills/connectors-overview/SKILL.md Outdated
Comment thread core/common/src/utils/serde_secret.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 3, 2026

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mlevkov

mlevkov commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, both wording findings were correct and the second one was load-bearing rather than cosmetic. Pushed 3cd136fe3.

Deserialize vs re-serialize. You are right, and my sentence would have led an author to drop the wrong derive. sdk/src/{sink,source}.rs do serde_json::from_str::<C> under a DeserializeOwned bound, so the plugin's struct is exactly what gets deserialized into. The guidance now says derive Deserialize but not Serialize, and names the property that actually carries the argument: nothing ever re-serializes the struct. Also corrected that plugin_config reaches the runtime as TOML, as JSON posted to the control API, or from env.

The inventory. Scoped to plugin-side callers, and it now says explicitly that the others are not all mistakes, naming HttpConfig::api_key and the core/common login / create-user / change-password / PAT payloads where the credential is the payload by design. Framing those as oversights was the worse half of that sentence.

On the paired deserialize_with: I have left it as a documented warning, strengthened to name the failure directly rather than implying it, and I would rather add the mechanism with its first consumer. Reasoning, in case you disagree:

A deserialize_with guard is itself opt-in, so it does not close the hole mechanically. An author who writes serialize_with = serialize_redacted and forgets the matching deserialize_with is precisely the case it claims to cover, and they get the silent placeholder-as-secret anyway. The shape that cannot be half-applied is a newtype owning both directions, and I would rather design that against a real call site than guess at one. This PR already adds two helpers with no consumers; a third feels like the wrong direction until something needs it.

Happy to add either form now if you would rather have it mechanical.

On the REDACTED const sweep (api/config.rs Debug impl, core/common auth credentials, server state models, s3_sink URL redaction): agreed, and I read your note as out of scope here. Say the word and I will open it as its own PR rather than widen this one.

Gate: fmt, sort, clippy -D warnings, cargo test -p iggy_common serde_secret (6 pass), taplo, hawkeye, typos, markdownlint all exit 0.

@mlevkov

mlevkov commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 8, 2026
@mlevkov

mlevkov commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@github-actions
github-actions Bot requested a review from hubcio August 9, 2026 01:11
mlevkov added 3 commits August 8, 2026 18:20
`serialize_secret` and `serialize_optional_secret` write the plaintext,
which leaves an author who needs `Serialize` on a struct holding a
credential with no redacting option — so they reach for the exposing one
and believe it protects them. That is the trap apache#3801 documents.

`serialize_redacted` and `serialize_optional_redacted` write a
placeholder instead. The optional form keeps `Some` distinguishable from
`None`: whether a credential is configured is not itself secret, and
collapsing it to null would report a configured field as unset.

The module doc now leads with what these helpers actually do, since the
absent `Serialize` impl on `SecretString` is the protection and any
helper here is a decision to give it up.
The Secrets guidance told plugin authors that annotating a `SecretString`
with `serialize_secret` made serialization redact. It does the opposite:
the helper calls `expose_secret()`. `SecretString` has no `Serialize`
impl precisely so a struct holding one cannot be serialized, so adding
the attribute is what unblocks the derive and gives up the guarantee.
Nine plugins followed that guidance.

The claim appeared twice: in the Secrets prose and again as an
"Auto-redact on Debug/Display + serialization" row in the patterns table,
where it read as a recommended pairing. Both now say which half was true
- `Debug` does redact - and the section gives the default for a plugin
config struct: do not derive `Serialize` at all, since nothing needs it
and leaving it off makes the property compiler-enforced. It also names
the redacting helpers for structs that genuinely need serialization, and
notes that none of this protects against the runtime control API
returning plugin config verbatim, which is apache#3802 and not fixable from
the plugin side.

Corrects the in-tree list too: it named delta_sink, which does not use
these helpers, and omitted s3_sink and surrealdb_sink, which do.
Review caught two wrong statements in the guidance this PR was fixing.

"The runtime never deserializes into a plugin's config struct" is false:
the SDK glue does exactly that, `serde_json::from_str::<C>` under a
`DeserializeOwned` bound in `sdk/src/{sink,source}.rs`, so `Deserialize`
stays required and the advice now says which half to drop. The property
that actually carries the argument is that nothing ever re-serializes
the struct. Plugin configuration also does not only come from TOML; the
control API accepts it as JSON and env vars can inject it.

The in-tree list read as an exhaustive inventory while covering only
plugins, which framed two deliberate uses as oversights: the runtime's
own `HttpConfig::api_key`, and the `core/common` wire payloads for
login, create-user, change-password and PAT, where the credential is the
payload. Scoped the sentence to plugin-side callers.

Also answers the round-trip question raised in review. The doc now names
the failure directly: deserializing redacted output hands back the
placeholder as the secret rather than failing. Left mechanical rather
than adding a paired `deserialize_with`, because that guard is itself
opt-in and a caller who forgets it is exactly the case it claims to
cover. A newtype owning both directions is the shape that cannot be
half-applied, and it should be designed against a real consumer.
@mlevkov
mlevkov force-pushed the serde-secret-guidance branch from 3cd136f to 0ddc886 Compare August 9, 2026 01:28
@mlevkov

mlevkov commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

The red on this PR after the rebase is a Maven Central rate limit, not the diff. Flagging it so it does not read as a real failure.

BDD • bdd-java failed while Gradle resolved its own plugin classpath:

> Could not resolve org.jetbrains.kotlin:kotlin-stdlib:2.3.21.
   > Could not GET 'https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/
     kotlin-gradle-plugin-api/2.3.21/kotlin-gradle-plugin-api-2.3.21.pom'.
     Received status code 429 from server: Too Many Requests
BUILD FAILED in 26s

It never reached a step; the failure is in :java:buildSrc configuration. The five other BDD jobs are CANCELLED by fail-fast rather than failed on their own, and finalize_pr fails downstream of those. Everything else on the run is green: 91 passing, 1 skipped.

This PR touches .claude/skills/connectors-overview/SKILL.md and adds two additive pub fns plus a doc comment to iggy_common::utils::serde_secret. No Java, no wire format, no server behaviour. The same rebase onto 01a64b2e0 produced fully green runs on my other four PRs, which went through the same BDD matrix minutes earlier.

I cannot re-run it myself (Must have admin rights to Repository). A re-run should clear it. If you would rather not spend the click, say so and I will force-push an identical tree to re-trigger, though that would change the SHA I quoted above.

Local gate on the rebased branch is unchanged and green: fmt, sort, clippy -D warnings, cargo test -p iggy_common 271 passing, license-headers, typos, taplo, all exit 0.

@hubcio
hubcio merged commit f900e7a into apache:master Aug 10, 2026
99 checks passed
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(connectors): serde_secret helpers expose secrets, but the connectors guidance says they redact

4 participants