[pull] main from LibreChat-AI:main

docker-compose.yaml

@@ -15,6 +15,12 @@ services:
             - CODEAPI_BRIDGE_DYNAMIC_WORKERS=${CODEAPI_BRIDGE_DYNAMIC_WORKERS:-true}
             - CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS=${CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS:-1}
             - CODEAPI_BRIDGE_WORKER_ID=${CODEAPI_BRIDGE_WORKER_ID:-}
+            - CODEAPI_BRIDGE_RECOVERY_SERVER_ID=${CODEAPI_BRIDGE_RECOVERY_SERVER_ID:-}
+            - CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS=${CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS:-0}
+            - CODEAPI_BRIDGE_RECOVERY_CHALLENGE_TTL_SECONDS=${CODEAPI_BRIDGE_RECOVERY_CHALLENGE_TTL_SECONDS:-60}
+            - CODEAPI_BRIDGE_RECOVERY_MAX_CHALLENGES_PER_MINUTE=${CODEAPI_BRIDGE_RECOVERY_MAX_CHALLENGES_PER_MINUTE:-12}
+            - CODEAPI_BRIDGE_RECOVERY_MAX_ATTEMPTS_PER_MINUTE=${CODEAPI_BRIDGE_RECOVERY_MAX_ATTEMPTS_PER_MINUTE:-30}
+            - CODEAPI_BRIDGE_RECOVERY_MAX_UNTRUSTED_PER_MINUTE=${CODEAPI_BRIDGE_RECOVERY_MAX_UNTRUSTED_PER_MINUTE:-240}
             - CODEAPI_AUTH_PROVIDER=${CODEAPI_AUTH_PROVIDER:-}
             - CODEAPI_ALLOW_AUTH_PROVIDER_NONE=${CODEAPI_ALLOW_AUTH_PROVIDER_NONE:-}
             - CODEAPI_JWT_ISSUER=${CODEAPI_JWT_ISSUER:-}
@@ -64,6 +70,12 @@ services:
             - CODEAPI_BRIDGE_DYNAMIC_WORKERS=${CODEAPI_BRIDGE_DYNAMIC_WORKERS:-true}
             - CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS=${CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS:-1}
             - CODEAPI_BRIDGE_WORKER_ID=${CODEAPI_BRIDGE_WORKER_ID:-}
+            - CODEAPI_BRIDGE_RECOVERY_SERVER_ID=${CODEAPI_BRIDGE_RECOVERY_SERVER_ID:-}
+            - CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS=${CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS:-0}
+            - CODEAPI_BRIDGE_RECOVERY_CHALLENGE_TTL_SECONDS=${CODEAPI_BRIDGE_RECOVERY_CHALLENGE_TTL_SECONDS:-60}
+            - CODEAPI_BRIDGE_RECOVERY_MAX_CHALLENGES_PER_MINUTE=${CODEAPI_BRIDGE_RECOVERY_MAX_CHALLENGES_PER_MINUTE:-12}
+            - CODEAPI_BRIDGE_RECOVERY_MAX_ATTEMPTS_PER_MINUTE=${CODEAPI_BRIDGE_RECOVERY_MAX_ATTEMPTS_PER_MINUTE:-30}
+            - CODEAPI_BRIDGE_RECOVERY_MAX_UNTRUSTED_PER_MINUTE=${CODEAPI_BRIDGE_RECOVERY_MAX_UNTRUSTED_PER_MINUTE:-240}
             - CODEAPI_AUTH_PROVIDER=${CODEAPI_AUTH_PROVIDER:-}
             - CODEAPI_JWT_SINGLE_TENANT_ID=${CODEAPI_JWT_SINGLE_TENANT_ID:-}
             - CODEAPI_TENANT_ISOLATION_STRICT=${CODEAPI_TENANT_ISOLATION_STRICT:-}
@@ -213,9 +225,11 @@ services:
     redis:
         image: redis:7-alpine
         container_name: redis
-        command: redis-server --requirepass localdev
+        command: redis-server --requirepass localdev --appendonly yes
         ports:
             - ${CODEAPI_REDIS_PORT:-16379}:6379
+        volumes:
+            - redis_data:/data
 
     minio:
         image: quay.io/minio/minio
@@ -232,3 +246,4 @@ services:
 
 volumes:
     minio_data:
+    redis_data:

docs/adr/001-stateful-code-environments.md

@@ -52,8 +52,11 @@ worker replacement; the UI and operator documentation must not imply otherwise.
 - The VM requires no inbound internet listener.
 - Code API, not the worker, authenticates LibreChat users and normalizes work.
 - A stolen short-lived credential is insufficient without the worker private
-  key; a stolen private key is insufficient after credential expiry or
-  revocation.
+  key. In the original pairing-only model, a stolen private key is insufficient
+  after credential expiry or revocation. With optional durable machine
+  enrollment and signed credential recovery, the private key itself remains
+  a revocable long-lived credential: access-credential expiry alone does not
+  protect against theft of that key. Revocation invalidates both.
 - Pairing codes and credentials are stored by digest where lookup permits.
 - One configured worker has at most one active fenced assignment.
 - Sandbox isolation and default-deny egress remain the mandatory default;

docs/byom-worker-admission.md

@@ -5,8 +5,13 @@ The limit is 32 admitted requests per worker, including the active request. When
 the limit is reached, the workspace endpoint returns HTTP 429 with
 `WORKER_QUEUE_FULL`. A different worker has an independent admission queue.
 
-The Code API workspace HTTP endpoint allows up to the smaller of `JOB_TIMEOUT`
-and five minutes for admission while the caller stays connected. After admission
+The Code API workspace HTTP endpoint allows 30 seconds for admission when the
+request has no `X-LibreChat-Workspace-Queue-Wait-Ms` header. A caller can
+advertise a positive integer allowance in milliseconds through that header,
+up to five minutes and any configured server queue ceiling. An invalid or
+out-of-range value is rejected before dispatch. This queue budget is
+independent of `JOB_TIMEOUT`; a shorter client or proxy deadline still ends
+the wait. After admission
 and worker validation, a separate execution deadline starts. Commands receive
 their requested timeout (30 seconds by default, up to five minutes), capped by
 the operator's `JOB_TIMEOUT`, plus five seconds to settle the result. Other
@@ -29,22 +34,27 @@ Existing workers still execute one assignment at a time. Parallel execution acro
 workspaces requires separate lease claims and isolated native sandbox contexts;
 this admission change does not advertise that capability.
 
-Clients and reverse proxies must allow queue time plus execution/settlement time
-and five seconds for HTTP delivery. With the default five-minute `JOB_TIMEOUT`,
-that is at least 335 seconds for non-command tools, 340 seconds for default
-commands, and 610 seconds for five-minute commands. With a smaller `JOB_TIMEOUT`,
-use `min(JOB_TIMEOUT, 300s)` for the queue, plus `min(JOB_TIMEOUT, 30s)` for other
+Clients and reverse proxies must allow the admitted queue budget plus
+execution/settlement time and five seconds for HTTP delivery. With the default
+five-minute `JOB_TIMEOUT` and no queue header, that is at least 65 seconds for
+non-command tools, 70 seconds for default commands, and 340 seconds for
+five-minute commands. At the maximum advertised five-minute queue allowance,
+those totals become 335, 340, and 610 seconds respectively. With a smaller
+`JOB_TIMEOUT`, use the advertised allowance (or 30 seconds without a header),
+bounded by the server queue ceiling, plus `min(JOB_TIMEOUT, 30s)` for other
 operations or `min(JOB_TIMEOUT, requested command timeout) + 5s` for commands,
-plus five seconds for delivery.
+plus five seconds for delivery. The caller should advertise only the queue time
+left after reserving execution, settlement, and delivery under its own HTTP
+deadline; Code API does not receive that absolute deadline.
 
-At the time of this change, LibreChat's `getWorkspaceToolTimeoutMs` still budgets
-only 30 seconds for a single admission attempt (65/70/340 seconds in total).
-Its `maxQueueWaitMs` is a retry horizon after a typed capacity rejection, **not**
-a per-attempt HTTP timeout. Updating Code API alone therefore does not guarantee
-the full wait. An earlier client, tool, or proxy timeout disconnects the request;
-if work was already admitted, a mutation may have run and must not be blindly
-retried. Match LibreChat's per-attempt timeout and each intermediary to the new
-budget before relying on it. Existing workers do not need an update.
+LibreChat's `maxQueueWaitMs` is a retry horizon after a typed capacity
+rejection, **not** a per-attempt HTTP timeout. Without its opt-in
+`maxRequestTimeoutMs`, LibreChat keeps a 30-second admission allowance per
+attempt. Enabling a longer client budget requires LibreChat's header support on
+every API replica and a timed canary through each intermediary; changing Code
+API alone does not guarantee the full wait. An earlier client, tool, or proxy
+timeout disconnects the request; if work was already admitted, a mutation may
+have run and must not be blindly retried. Existing workers do not need an update.
 
 Focused regression coverage lives in `service/src/bridge/admission.test.ts`,
 `service/src/bridge/worker-admission.test.ts`,

docs/remote-bridge/README.md

@@ -31,6 +31,42 @@ CODEAPI_BRIDGE_TOKEN=<strong-administrator-bootstrap-secret>
 CODEAPI_BRIDGE_AUTH_MODE=paired
 ```
 
+To opt in to durable machine authorization on every Code API replica, set a
+single stable public **Code API** origin (not the LibreChat URL):
+
+```dotenv
+CODEAPI_BRIDGE_RECOVERY_SERVER_ID=https://code.example.com
+# 0 (default): enrolled machine keys remain authorized until revoked.
+# CODEAPI_BRIDGE_ENROLLMENT_TTL_SECONDS=0
+# CODEAPI_BRIDGE_RECOVERY_CHALLENGE_TTL_SECONDS=60
+# CODEAPI_BRIDGE_RECOVERY_MAX_CHALLENGES_PER_MINUTE=12
+# CODEAPI_BRIDGE_RECOVERY_MAX_ATTEMPTS_PER_MINUTE=30
+# CODEAPI_BRIDGE_RECOVERY_MAX_UNTRUSTED_PER_MINUTE=240
+```
+
+Omitting the server ID retains the existing pairing and refresh behavior and
+hides the recovery routes. Deploy the compatible Code API version to **all**
+replicas before setting this value and enrolling workers again. Older Code API
+replicas can still pair or refresh a worker but do not write durable enrollment;
+they must not serve device login or recovery requests. Only a pairing redeemed
+after this option is enabled has a recoverable key. Updating Code API alone
+does not make old workers reconnect automatically: the CLI must also implement
+this recovery protocol in the later worker release.
+
+Store Redis state durably across restarts. The primary `docker-compose.yaml`
+now uses Redis AOF and a named `/data` volume; preserve that volume when
+recreating the stack. If upgrading a running stack with an in-memory Redis,
+migrate its state before recreating the container: mounting an empty volume
+does **not** preserve active assignments, fences, or earlier revocations. Other
+deployments must provide equivalent durable Redis (for example, a managed
+persistent Redis service and backups). Revocation and
+machine enrollment share that state across replicas; do not configure eviction
+of authorization keys. If enrollment state is missing, credentials minted under
+that enrollment fail closed, and the worker must be explicitly enrolled again.
+Restoring a backup from *before* a revocation can revive trust; reconcile
+revocations after recovery from backup. Use a distinct server ID for each Code
+API deployment and keep it stable when the endpoint changes behind a proxy.
+
 Use `strict` instead of `affinity` if every request must include a runtime
 session hint. In hardened mode, startup requires the bridge token to be at least
 32 bytes. `PTC_MODE=blocking` is rejected; replay mode is required because a
@@ -226,7 +262,40 @@ execution.
   atomically on their first redemption attempt.
 - Worker credentials expire after fifteen minutes and are bound to an Ed25519
   public key. Exact-request signatures include the HTTP method, path, body
-  digest, timestamp, nonce, and credential.
+  digest, timestamp, nonce, and credential. With recovery enabled, redeeming a
+  pairing also persists a separate machine authorization and its public key in
+  Redis without a TTL by default; an operator can instead set a bounded
+  enrollment lifetime.
+- `POST /v1/bridge/workers/:workerId/credentials/challenge` does not require
+  an administrator token or an existing access credential, but **does** require
+  the enrolled key. Its JSON body contains `protocolVersion: 1`,
+  `operation: "credential.challenge"`, the configured `serverId`, the matching
+  `workerId`, a fresh UTC ISO `timestamp`, a random 32-byte base64url `nonce`,
+  and `signature` computed with `signBridgeRecoveryStart(privateKey, fields)`
+  from `@librechat/code/identity`. Code API verifies the signed fields and
+  consumes the nonce once before charging the machine's shared challenge
+  budget; a fabricated request cannot exhaust another worker's budget.
+- The response is a short-lived, single-use challenge with the server ID,
+  worker ID, enrollment generation, operation and expiry. Sign those fields
+  with `signBridgeRecovery(privateKey, challenge)` and send the fields plus
+  `signature` to `POST .../credentials/recover` to obtain a new short-lived
+  credential. Invalid proofs are limited per high-entropy challenge; only
+  successfully verified, unused proofs consume the machine's shared recovery
+  budget. Separately, both recovery endpoints limit all incoming requests per
+  connection peer *before* key verification, including well-formed JSON with
+  malformed or forged proofs; forged headers and worker IDs cannot bypass
+  that limit or consume the signed machine budget. All limits live in shared
+  Redis; HTTP 429 means back off. When a reverse proxy connects to Code API,
+  its clients share that peer's limit. Restrict direct backend access and apply
+  client-IP and global
+  abuse limits at the trusted ingress to keep one proxy peer from becoming a
+  shared bottleneck; do not trust an arbitrary `X-Forwarded-For` on Code API.
+- Recovery and revocation are atomic Redis transitions across API replicas.
+  A missing, revoked, expired or superseded enrollment never creates new
+  credentials. Recovery only restores transport authentication. It does not
+  clear assignment fences, worker or workspace quarantine, or uncertain
+  execution state. The worker private key is a durable, revocable credential;
+  expiry of an access credential alone does **not** protect against key theft.
 - Accepted proof nonces cannot be replayed, credentials rotate before expiry,
   and an administrator can revoke the active worker identity immediately.
 - Assignment leases bind to a stable paired identity rather than an individual
@@ -239,12 +308,14 @@ execution.
   worker. The lower API or worker slot ceiling wins, and assignments sharing
   the same workspace isolation key remain serialized while independent
   conversation worktrees may run concurrently.
-- Workspace tool admission waits for capacity up to the smaller of `JOB_TIMEOUT`
-  and five minutes while the HTTP caller remains connected. Disconnects cancel
-  waiting, and admitted work receives a separate execution budget. A shorter
-  client or proxy timeout can end the wait sooner; Code API does not receive an
-  absolute caller deadline. See [BYOM worker admission](../byom-worker-admission.md)
-  for the caller and proxy timeout requirements.
+- Workspace tool admission waits for capacity up to 30 seconds without a
+  `X-LibreChat-Workspace-Queue-Wait-Ms` header. A caller can advertise a longer
+  per-request allowance, bounded by five minutes and any server queue ceiling.
+  Disconnects cancel waiting, and admitted work receives a separate execution
+  budget capped by `JOB_TIMEOUT`. A shorter client or proxy timeout can end the
+  wait sooner; Code API does not receive an absolute caller deadline. See
+  [BYOM worker admission](../byom-worker-admission.md) for the total-request
+  and proxy timeout requirements.
 - Dynamic workers are fenced to their server-issued tenant before assignment.
 - Each assignment has an absolute deadline, generation, and random lease token.
 - Settlements with the wrong worker, generation, token, or expired deadline are

docs/remote-bridge/worker-runbook.md

@@ -488,11 +488,16 @@ it still advertises named environments.
 
 ### Expired bridge credential
 
-A running worker refreshes its short-lived credential automatically. If a
-machine is offline long enough that refresh can no longer authenticate, issue
-a fresh one-time pairing for the same worker ID and redeem it with a newly
-generated keypair. Reusing the worker ID preserves the LibreChat environment
-record and its agent assignments; creating a new ID creates a new environment.
+A running worker refreshes its short-lived credential automatically. With
+Code API durable enrollment enabled, a worker that still has its enrolled
+private key can request a short-lived challenge and recover a new access
+credential without manual re-pairing. The current CLI does **not** yet invoke
+that endpoint automatically; update it when worker reconnect support ships.
+Until then, or if enrollment is missing or revoked, use the one-time operator
+pairing fallback. A new pairing replaces the Code API worker identity and may
+require LibreChat environment reauthorization; reusing a worker ID alone does
+not guarantee preservation of its LibreChat environment or agent assignments.
+Never clear quarantine or workspace fences as part of credential recovery.
 
 ### Failed environment setup or uncertain mutation
 

packages/code/README.md

@@ -791,17 +791,22 @@ Legacy requests without a conversation identity continue to use the selected
 source root. Older Code API deployments do not negotiate the capability, so the
 worker omits it until every request path understands the isolation boundary.
 
-On an updated Code API, admission waits up to the smaller of `JOB_TIMEOUT` and
-five minutes while the HTTP caller remains connected; older Code API versions
-waited at most 30 seconds. A `WORKSPACE_QUEUE_TIMEOUT` response (HTTP 503,
+On an updated Code API, admission waits up to 30 seconds without the
+`X-LibreChat-Workspace-Queue-Wait-Ms` request header. A caller may advertise a
+positive integer millisecond allowance up to five minutes, capped by any server
+queue ceiling. This allowance is separate from the `JOB_TIMEOUT` execution
+budget and cannot outlast a shorter client or proxy timeout. A
+`WORKSPACE_QUEUE_TIMEOUT` response (HTTP 503,
 `Retry-After: 1`) means the operation was not assigned or started; wait for
 capacity before submitting it again. This is distinct from `ASSIGNMENT_EXPIRED`
 or a transport timeout after dispatch, where execution may have occurred and
 mutations must not be blindly retried. No automatic retry is added by this policy.
-Align the client's per-attempt timeout and any proxy with the queue **plus**
-execution budget before relying on the longer wait. See the
+Align the client's per-attempt timeout and every proxy with the queue **plus**
+execution, settlement, and delivery budget before relying on a longer wait.
+LibreChat keeps the 30-second admission allowance unless its longer total HTTP
+budget is explicitly enabled and the live ingress path is verified. See the
 [BYOM worker admission guide](../../docs/byom-worker-admission.md) for the
-current client limitation and the timeout calculations.
+timeout calculations.
 
 Keep the existing URL, pairing/identity, and network policy configuration.
 The primary root keeps its configured workspace ID (default `primary`). Repeat

packages/code/package.json

@@ -15,6 +15,10 @@
       "types": "./dist/protocol.d.ts",
       "import": "./dist/protocol.js"
     },
+    "./identity": {
+      "types": "./dist/identity.d.ts",
+      "import": "./dist/identity.js"
+    },
     "./worker": {
       "types": "./dist/worker.d.ts",
       "import": "./dist/worker.js"