Stop requesting GSS credential delegation (TGT NOT FORWARDABLE storm)#674
Merged
Conversation
Fork of gssapi.js 2.0.1 (bitbucket:karoshealth/node-gssapi, MIT).
Upstream passes GSS_C_DELEG_FLAG to gss_init_sec_context
unconditionally, which makes libkrb5 request a forwarded TGT from the
KDC on every client context creation - one per Factory+ HTTP token and
MQTT connection. ACS service principals hold non-forwardable TGTs, so
the KDC refuses every request ('TGT NOT FORWARDABLE ... KDC can't
fulfill requested option') and libkrb5 silently drops the flag: a
wasted KDC round trip and a KDC log line per token, multiplying into
serious noise under reconnect storms.
Nothing in Factory+ consumes delegated credentials, so the fork simply
does not request delegation. No other changes from upstream.
Publish to npm with a release tagged js/gssapi/v2.1.0 (the existing
js-lib workflow handles lib/js-* directories).
service-client tries the delegation-free fork first and falls back to upstream gssapi.js, so consumers without the fork installed keep working. The fork is an optionalDependency alongside gssapi.js: npm skips it while unpublished (installs fall back cleanly) and picks it up once it is on the registry. service-api now takes GSS from the service-client re-export instead of importing gssapi.js directly, so both sides of an ACS service resolve the same implementation and the native dependency has a single owner. A side effect is that service-api now loads on platforms with no GSS support and fails at auth time instead of at import time.
AlexGodbehere
force-pushed
the
ago/gssapi-no-deleg
branch
from
July 9, 2026 11:48
7d0c28f to
42a88e0
Compare
AlexGodbehere
added a commit
that referenced
this pull request
Jul 17, 2026
…kes effect (#676) ## What was broken `v6.0.0-rc.2`'s release failed: the `acs-admin` image build errored with ``` [vite]: Rollup failed to resolve import "@amrc-factoryplus/gssapi" from /app/acs-admin/node_modules/@amrc-factoryplus/service-client/lib/deps.js ``` which fail-fast-cancelled the other 18 JS image builds and skipped the chart release. ## Root cause Two separate defects, both introduced by 42a88e0 ("Prefer @amrc-factoryplus/gssapi over gssapi.js for GSS"): 1. **The browser build.** `acs-admin` is the only bundled build in the tree, and `vite.config.js` aliases native GSS out of it (`'gssapi.js': EMPTY`). 42a88e0 introduced a *second* bare specifier, `@amrc-factoryplus/gssapi`, into `deps.js` — a module that gets bundled — without adding it to that alias list. `v5.1.0` and `v6.0.0-rc.1` are byte-identical here and built fine; rc.1 predates the vendoring by 3h40m and its `acs-admin` job passed. 2. **The fork was never installed.** It was declared as a *registry* dependency (`^2.1.0`) of a package that is not on npm. `bun install`/`npm install` logged the 404 as a warning and skipped it (it is optional), so `deps.js` fell through its `.catch()` to upstream `gssapi.js` — **with** credential delegation. #674's delegation fix has therefore been inert since it merged. The npm 404 was never the cause of the build failure. In the same rc.2 run, `acs-edge` (npm, same `deps.js` via `file:`) built successfully. ## The fix - `acs-admin/vite.config.js`: alias `@amrc-factoryplus/gssapi` to the empty stub, next to the existing `gssapi.js` entry. This alone unblocks the build — aliases intercept the specifier before node resolution, so the package need not be installed. It is also correct on its own merits: without it, an installed fork would drag the native addon loader (`bindings`, `file-uri-to-path`) into the browser bundle. - The 16 in-tree services that use GSS via `service-client` declare `"@amrc-factoryplus/gssapi": "file:../lib/js-gssapi"` in `optionalDependencies`, which is what actually installs the fork and makes #674 live. A `file:` ref declared *by* a `file:`-linked package does not install — npm resolves the nested path relative to the copied location, hits ENOENT, and silently skips it. `optionalDependencies` (not `dependencies`) matches how `gssapi.js` is declared and keeps `npm install` working without krb5 headers. - Excluded: `acs-admin` (browser; the alias handles it), `acs-edge` (never imports `service-client`), `acs-example-service` (not built). - Reverts #675. Its premise was wrong: `js/gssapi/v2.1.0` did not fail on missing Kerberos headers, it failed in 2s at workflow evaluation with `Secret NPM_TOKEN is required, but not provided`. No step ever ran. `js/gssapi/v2.1.1`, cut *with* the libkrb5-dev step, failed identically. The repo has no Actions secrets, and the js-lib workflow has never published anything — it was created 2025-06-16, four days *after* service-client 1.6.0 went to npm. The npm route is abandoned; the fork stays in-tree. ## Verification - rc.2's failure reproduced locally byte-identically, then `built in 22.14s` with the alias. The bundled GSS import compiles to `import('./emptyModule-...js')`, giving the browser the same `GSS === {}` it has had since before the fork — no behaviour change. - No native code in the bundle: `node-gssapi` and `file-uri-to-path` both 0 matches. - The real fork installs and compiles via `npm install --install-links` (`build/Release/node-gssapi.node` present, true GSS surface). The js-build base image already installs `krb5-dev` explicitly "to build gssapi.js", so the fork builds wherever `gssapi.js` already does. - With the per-service declaration, `deps.js` resolves the fork rather than upstream, even with `gssapi.js` also installed. - `acs-i3x` jest: 13 suites, 342 tests passing. ## How to test 1. Merge and cut `v6.0.0-rc.3`. 2. Confirm the `acs-admin` build passes and the chart release runs. 3. In a built service image, confirm `node_modules/@amrc-factoryplus/gssapi/build/Release/node-gssapi.node` exists. 4. On a cluster, confirm the KDC no longer logs a refused `TGT NOT FORWARDABLE` request per token/MQTT connect. ## Known gaps - `acs-edge` imports `@amrc-factoryplus/utilities`, not `service-client`, so edge agents keep the upstream delegating `gssapi.js`. If edge principals contribute to the KDC storm, this does not resolve it for them. - Registry consumers of `service-client` (`acs-mcp`, `acs-visualiser`, `historian-uns`, `lib/js-sparkplug-app`) resolve the published copy and keep delegating. ## Release notes Fixes the `acs-admin` image build, which broke in v6.0.0-rc.2. The GSS fork that removes Kerberos credential delegation is now actually installed in the services that use it, so the fix released in #674 takes effect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every Factory+ JS service floods the KDC with refused
TGT NOT FORWARDABLErequests - one per HTTP token fetch and MQTT connection. This PR removes the cause: thegssapi.jsnpm module hardcodesGSS_C_DELEG_FLAGingss_init_sec_context, so libkrb5 asks the KDC for a forwarded TGT on every client context. ACS service principals hold non-forwardable TGTs (straight from keytabs), so the KDC refuses every time:libkrb5 silently drops the flag and carries on, so nothing breaks - but every token costs an extra KDC round trip, and under reconnect storms (see #671) this multiplies into thousands of KDC log lines per minute. Nothing in Factory+ consumes delegated credentials; the server side discards them.
Changes
lib/js-gssapi: vendored fork of gssapi.js 2.0.1 (MIT, provenance in README) published as@amrc-factoryplus/gssapi2.1.0. One change: noGSS_C_DELEG_FLAG. Publishable via the existing js-lib release workflow with a release taggedjs/gssapi/v2.1.0.js-service-client:deps.jsprefers the fork and falls back togssapi.js; the fork is added tooptionalDependenciesalongside the existing entries.js-service-api: takesGSSfrom the service-client re-export instead of its owngssapi.jsimport, so the native dependency has one owner. service-api now loads on GSS-less platforms and fails at auth time rather than import time.Rollout
Safe to merge before the npm publish: npm skips unpublished optional dependencies (verified with npm 10.8.2), so installs fall back to upstream
gssapi.jsand behave exactly as today. Oncejs/gssapi/v2.1.0is released to npm, rebuilt images pick the fork up automatically. Services consuming@amrc-factoryplus/service-clientfrom the registry (historian-uns, acs-mcp) get the fix when service-client is next published and bumped.Verification
A/B tested both modules against a live ACS KDC (fpd-ago, via port-forward) from a Linux container (node:22-bookworm, MIT krb5, cmake - same toolchain as the Docker builds):
TGS_REQ ... ISSUEplusTGS_REQ (1 etypes ...) TGT NOT FORWARDABLETGS_REQ ... ISSUE, nothing refusedThe refused request's signature (single-etype TGS_REQ for krbtgt) exactly matches the storm seen from every sv1 principal in production KDC logs.
Also verified: the fork compiles on Linux and macOS; with the fork installed,
service-client'sGSSexport resolves to it; without it, the fallback togssapi.jsloads andservice-api's auth module imports cleanly.Decisions
gssapi.jsdeps in the handful of services that declare one (acs-i3x, uns-ingester-sparkplug, historian-uns): they are the fallback path and become vestigial once service-client ships the fork; removing them is trivial follow-up cleanup.