feat(dgraph): harden backup login and expose backup ports on the headless Service - #145
feat(dgraph): harden backup login and expose backup ports on the headless Service#145mlwelles wants to merge 1 commit into
Conversation
…less Service Declare http (8080) and grpc (9080) on the alpha headless Service. The backup CronJobs reach a pinned alpha-0 through this Service; under a STRICT mTLS mesh the client sidecar only builds an mTLS route for ports the Service declares, so without them the /admin login falls through to plaintext and the sidecar resets it. Gate both ports on backups.full.enabled or backups.incremental.enabled, since their only consumer is the backup CronJobs. Guard each backup.sh curl with require_json: capture the curl exit status errexit-safely and validate the body is JSON before parsing. A mesh or network failure returns a plaintext body (e.g. an Envoy connect error) that now surfaces as a clear curl-status / URL / body message instead of a cryptic parse failure or a silently empty token. Convert the login bodies and response parsing to jq: get_token_rest and get_token_graphql build their request bodies with 'jq -n --arg' instead of hand-interpolating user/password into a JSON string, so credentials containing quotes or other JSON-breaking characters can no longer corrupt or inject into the request. All four functions parse responses with 'jq -r ... // empty' instead of grep -oP/grep -q errors, removing the dependency on grep's PCRE mode. backup_graphql's destination stays hand-interpolated since it is operator-controlled, not user credentials. Reset HEADERS and CERTOPTS at the top of get_token and backup so a second call in the same shell does not accumulate headers left over from an earlier call. Quote the get_token arguments in the full and incremental CronJobs so an admin user or password containing whitespace is passed as a single argument.
| - name: grpc-alpha-int | ||
| port: 7080 | ||
| targetPort: 7080 | ||
| ## http-alpha (8080) and grpc-alpha (9080) are exposed here only when backups are |
There was a problem hiding this comment.
This block is YAML, not a template comment, and it sits outside the {{- if }} — so it renders on a stock backups-off install. Diffing the rendered headless Service against main:
26a27,32
> ## http-alpha (8080) and grpc-alpha (9080) are exposed here only when backups are
> ## enabled (backups.full.enabled or backups.incremental.enabled): the backup CronJobs
... (all six lines)
Harmless to Kubernetes, but it makes the PR's "no change to a stock (backups-off) render" not quite true, it shows up as a diff for every GitOps user on upgrade, and it explains ports that aren't in the document. Since the audience here is whoever next edits this template rather than an operator reading rendered output, I'd make it {{/* ... */}} so it never renders. Moving it inside the conditional works too.
Separate question on the content: the comment justifies both ports by the backup CronJobs, but the CronJobs only ever curl :8080 — I don't see anything in the chart that consumes 9080 through this Service. If there's a mesh reason to declare it anyway, worth saying so explicitly; otherwise it reads as exposure the stated rationale doesn't cover. The gate itself is right and matches configs.yaml exactly.
| echo "ERROR: empty response from ${URL}" >&2 | ||
| return 1 | ||
| fi | ||
| if ! jq -e . >/dev/null 2>&1 <<< "$BODY"; then |
There was a problem hiding this comment.
This makes jq a hard runtime dependency, and it's on every backup path now, not just login — require_json is called from backup_rest and backup_graphql too.
The stock image is fine; I pulled dgraph/dgraph:v25.3.1 and confirmed /usr/bin/jq. The problem is that values.yaml still says ## Backups image - image needs at least curl command, which actively invites a slim curl-only override for what is otherwise a fairly heavy cron pod.
And the failure mode misdirects. With jq absent, this catches 127 and reports:
ERROR: non-JSON response from alpha:8080/login: {"data":{"accessJWT":"abc"}}
Valid JSON, labelled non-JSON. Anyone debugging a custom backups.image chases the wrong thing. Two asks: update that values.yaml comment to "curl and jq", and consider a command -v jq preflight so the message names the real cause. While you're in the docs, README line 256 still lists backups.image.tag as v21.03.0 — it follows the *image anchor, so it's v25.3.1.
| ) || CURL_STATUS=$? | ||
| require_json "$RESULT" "$CURL_STATUS" "${ALPHA_HOST}:8080/login" || return 1 | ||
| ERROR=$(jq -r '.errors[0].message // empty' <<< "$RESULT") | ||
| if [[ -n "$ERROR" ]]; then echo "ERROR: $ERROR"; return 1; fi |
There was a problem hiding this comment.
This one needs >&2 (and the same line in get_token_graphql). Everything else in these two functions correctly writes to stderr — require_json's three messages and the "could not parse accessJWT" fallback — but the errors-array branch doesn't, and that's the branch a wrong backup password takes.
The caller is ACCESS_TOKEN=$(get_token ...) under set -e, so this stdout goes into the substitution and is discarded when the shell exits:
$ bash -c 'set -e; f() { echo "ERROR: invalid credentials"; return 1; }; T=$(f); echo unreachable'
exit=1 # nothing printed
So the operator gets a CronJob that exits 1 with an empty log, on the likeliest real failure. That's the "failing silently" this PR set out to remove, still present on the most common path.
| @@ -216,20 +236,18 @@ data: | |||
| backup_graphql() { | |||
| GQL="{\"query\": \"mutation { backup(input: {destination: \\\"${BACKUP_DESTINATION}\\\" forceFull: $FORCE_FULL }) { response { message code } } }\"}" | |||
There was a problem hiding this comment.
This is the one place the hand-built JSON survives, and it's the one interpolating an operator-supplied value that the script has already rewritten twice by this point (/${SUBPATH}, then ?secure=false). A destination containing a quote or backslash yields malformed JSON.
Not a privilege boundary, since backups.destination comes from values rather than an untrusted caller, so I won't push hard. But jq is a dependency now regardless, so the jq -n --arg conversion is nearly free here. If you'd rather keep the PR scoped to login, that's fine and the title supports it — but then the body's "injection-safe" framing is claiming more than the diff delivers.
2 participants
Harden the backup CronJob login: parse responses with
jqand build the login bodies withjq -n --arg(injection-safe) instead ofgrep -oPand hand-built JSON; guard each curl with an errexit-safe status check; quote theget_tokenarguments. Also declare http (8080) and grpc (9080) on the alpha headless Service when backups are enabled, so a pinned-pod/adminlogin works under a strict mTLS mesh. No change to a stock (backups-off) render.Part of splitting #140 into per-area PRs. The change was built and validated on that branch; the merge of all split PRs reproduces #140's tree byte-for-byte. #140 is being closed as superseded.