diff --git a/CHANGELOG.md b/CHANGELOG.md index 9450ac7b..876f2561 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,11 @@ new version heading in the same commit. ## [Unreleased] +## [0.458.0] - 2026-10-12 +### Added +- **DataForSEO stops "disconnecting": a local, API-key connector, and the gate now notices when any remote connector loses its login.** Every tenant's DataForSEO connector pointed at the hosted `mcp.dataforseo.com` URL with no headers, which authorizes with an OAuth grant kept in Claude Code's own config dir. The grant expires, isn't there at all on a pooled runtime account, and when it's missing Claude Code silently swaps every tool for `authenticate`/`complete_authentication` stubs an unattended run can't complete (20 stub calls in one tenant's transcripts). initech's `composio-company` has failed the same way since 2026-09-08, taking GSC/GA4/DataForSEO/Stripe/Sheets with it, and nobody was told. Two catalog entries, **DataForSEO** (`dataforseo-mcp-server@3.1.3`: `api_request` + docs tools) and **DataForSEO (v2 tools)** (`@2.9.13`: the 89 per-endpoint tools existing prompts name), run the official server over stdio with `secret:DATAFORSEO_LOGIN`/`secret:DATAFORSEO_PASSWORD`. Each call is plain Basic auth, with nothing to expire. Versions are pinned because v2→v3 renamed every tool. A cold `npx` install measured 21–35s, close to the MCP startup timeout, so enabled `npx` connectors are pre-fetched at boot and on add/enable (`src/connectors/npx-warm.ts`, audited `connector.warmed`). And a call to any connector's OAuth stub now reaches the gate as what it is: audited `connector.auth.lost`, one admin card per connector per 6h naming the fix, and on an unattended run a refusal telling the agent to carry on without it instead of hunting for a login (`src/connectors/mcp-auth-stub.ts`). Runbook `docs/connectors/dataforseo.md`; pinned by `scripts/connector-auth-stub-test.cjs`. + **For admins:** Add DataForSEO from Connectors with your API login stored in Secrets. It no longer needs a browser login that expires. If any connector loses its login, you now get an Inbox card saying which one and how to fix it. [Open Connectors](#/connectors) + ## [0.457.0] - 2026-10-09 ### Added - **A proposed task names who should work it, and assigning it on the board accepts it.** Agents filed proposals with no owner a quarter of the time, and to propose work for ANOTHER agent they had to fight `task_create`'s agent→agent default (poke-on-done implies auto-dispatch, so naming `agent:qa` handed it off instead of proposing it). `task_create` gains `suggest` (`agent:` / `me` / a member id), required whenever the task lands `proposed` (refused with the roster otherwise). The pick is stored as `tasks.suggested_assignee`, not the assignee — advice the reviewer weighs; the agent re-pointing it with `task_update` only revises it. Accepting a proposal with no assignee adopts the suggestion (Inbox Accept, Accept & run, and moving it to `todo` on the board all agree); dismissing doesn't. On the board, assigning someone to a proposal moves it straight to `todo`, gated like the accept button and audited `task.proposal.accepted` (`via: 'assign'`); a board accept/dismiss is now audited at all. The Inbox card and the board's proposals strip show the pick marked "suggested", and the task page offers **Assign & accept**. Pinned by `scripts/task-proposals-test.cjs` §9. diff --git a/docs/connectors/README.md b/docs/connectors/README.md index 4c293111..7122fb08 100644 --- a/docs/connectors/README.md +++ b/docs/connectors/README.md @@ -12,6 +12,7 @@ bit us. | **ClickUp** | webhook (`POST /hooks/clickup`) | `clickup_reply` | **yes** | [clickup.md](clickup.md) | | **GitHub** | — (credential source) | `GH_TOKEN` in the shell + API bearer; `github_refresh` | no | [github.md](github.md) | | **Composio** | webhook (`POST /triggers/composio`) | the SaaS long tail, via a minted Tool Router URL | for triggers only | [composio.md](composio.md) | +| **DataForSEO** | — | `dataforseo-mcp-server` over stdio, API login from the vault | no | [dataforseo.md](dataforseo.md) | Design history and the five original use cases live in [`../connectors-and-triggers.md`](../connectors-and-triggers.md) — note its as-built banner: the shipped tree diverged from that spec, and these pages are the current diff --git a/docs/connectors/dataforseo.md b/docs/connectors/dataforseo.md new file mode 100644 index 00000000..d809d6ff --- /dev/null +++ b/docs/connectors/dataforseo.md @@ -0,0 +1,64 @@ +# DataForSEO + +SEO data (SERPs, keyword volumes, Labs, backlinks, on-page) for agents, via DataForSEO's **official MCP +server run locally over stdio**, authenticated with the API login and password from the vault. + +## Why not the hosted URL + +DataForSEO also hosts the server at `https://mcp.dataforseo.com/mcp` (v2) and `…/v3/mcp`. Added as a +`custom-remote` connector with no headers, that endpoint authorizes with **OAuth**, and the grant lives in +Claude Code's own per-config-dir store. That is what made DataForSEO "keep disconnecting": + +- the grant expires, and refreshing it needs a browser; +- it lives in ONE config dir, so a run on a pooled runtime account (or under + `AOS_CLAUDE_CONFIG_ISOLATION`) has no grant at all; +- when it is missing, Claude Code doesn't fail the server. It replaces every DataForSEO tool with + `authenticate` / `complete_authentication` stubs, and an unattended run can't finish them. One tenant's + transcripts held 20 such stub calls. + +On stdio there is no OAuth: each tool call is a plain Basic-auth REST call, with nothing to expire and no +remote session to drop. + +## Setup + +1. **Settings → Secrets**: store `DATAFORSEO_LOGIN` and `DATAFORSEO_PASSWORD` tenant-wide. These are the + **API** credentials from , not the dashboard login. +2. **Connectors → Add**, pick one of: + - **DataForSEO**: `dataforseo-mcp-server@3.1.3`. Four tools: `api_request` (`{method, path, data}`) + plus `docs_index` / `docs_list_sections` / `docs_search`. + - **DataForSEO (v2 tools)**: `dataforseo-mcp-server@2.9.13`. 89 per-endpoint tools + (`serp_organic_live_advanced`, `kw_data_google_ads_search_volume`, …). Use it when agent prompts or + skills already name those tools. +3. Fill both fields with `secret:DATAFORSEO_LOGIN` / `secret:DATAFORSEO_PASSWORD`. They are resolved from + the vault at launch, and an unresolvable reference is blanked and audited as + `connector.secret.unresolved`. + +**Keep the connector's NAME when replacing an old one.** The MCP server key is the connector id (a slug of +its label), and tool names are `mcp____`. Replacing a `DataforSEO` connector with one labelled +`DataForSEO` keeps `mcp__dataforseo__*`; a doosra-style `DFS` keeps `mcp__dfs__*`. + +## Versions are pinned on purpose + +v2 → v3 renamed the entire tool surface, so an unpinned `@latest` would silently break every prompt that +names a tool. Move to a new version deliberately: re-add the connector, then update the prompts. + +## Cold start + +`npx` installs the package on first use: 21s (v2) to 35s (v3) measured on the Mac Mini, against ~0.6s +warm. That is close enough to Claude Code's MCP startup timeout to drop the server from a session. So the +server pre-fetches every enabled `npx` connector's package at boot and when one is added or re-enabled +(`src/connectors/npx-warm.ts`, audited `connector.warmed` `{spec, ok}`). + +## When a remote connector loses auth anyway + +This applies to any remote MCP connector, Composio's Tool Router included. A call to an +`mcp____authenticate` / `__complete_authentication` stub reaches the gate, which: + +- audits `connector.auth.lost` `{connector, step, unattended, url?}`; +- posts ONE admin card per connector per 6h (`system:connector-auth:`), naming the connector and + the fix (for the hosted DataForSEO URL: switch to the catalog entry); +- on an **unattended** run (headless, not taken over), refuses the call and tells the agent to carry on + without the connector or report it as blocked. An attended session goes through the normal policy, since + its human can finish the login. + +`src/connectors/mcp-auth-stub.ts`; pinned by `scripts/connector-auth-stub-test.cjs`. diff --git a/package-lock.json b/package-lock.json index 4d6f4d19..41463640 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "agent-os", - "version": "0.457.0", + "version": "0.458.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "agent-os", - "version": "0.457.0", + "version": "0.458.0", "license": "MIT", "bin": { "agent-os": "bin/agent-os" diff --git a/package.json b/package.json index 022261f4..deedbf85 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "agent-os", - "version": "0.457.0", + "version": "0.458.0", "description": "A generic, governed operating system for running autonomous agents safely across brands. Ships with a local web console.", "license": "MIT", "type": "commonjs", @@ -27,7 +27,7 @@ "check-deps": "bash scripts/install-deps.sh --check", "dev": "ts-node src/cli.ts serve", "demo:dev": "ts-node src/demo.ts", - "test:governance": "node scripts/version-sync-test.cjs && node scripts/event-loop-freeze-test.cjs && node scripts/git-ceiling-test.cjs && node scripts/cron-supersede-test.cjs && node scripts/governance-conformance.cjs && node scripts/tier-a-policy-test.cjs && node scripts/policy-baseline-test.cjs && node scripts/heredoc-intent-test.cjs && node scripts/script-body-gate-test.cjs && node scripts/capability-registry-test.cjs && node scripts/composio-envelope-test.cjs && node scripts/composio-identity-test.cjs && node scripts/idle-reaper-test.cjs && node scripts/dm-continuity-test.cjs && node scripts/telegram-dm-lane-test.cjs && node scripts/cli-link-origin-test.cjs && node scripts/alert-staleness-test.cjs && node scripts/run-as-identity-test.cjs && node scripts/email-identity-guard-test.cjs && node scripts/deps-freshness-test.cjs && node scripts/runtime-account-test.cjs && node scripts/runtime-account-misattribution-test.cjs && node scripts/runtime-usage-refresh-test.cjs && node scripts/keychain-credential-test.cjs && node scripts/credential-preflight-test.cjs && node scripts/capacity-queue-test.cjs && node scripts/runtime-pool-signed-out-test.cjs && node scripts/login-expiry-test.cjs && node scripts/runtime-login-test.cjs && node scripts/rotate-on-reload-test.cjs && node scripts/headless-resumable-test.cjs && node scripts/session-revive-gates-test.cjs && node scripts/session-pause-test.cjs && node scripts/make-live-lock-test.cjs && node scripts/claude-config-seed-test.cjs && node scripts/claude-config-isolation-test.cjs && node scripts/output-style-test.cjs && node scripts/session-cost-test.cjs && node scripts/chain-model-test.cjs && node scripts/task-workers-test.cjs && node scripts/tuning-patch-test.cjs && node scripts/task-runs-test.cjs && node scripts/task-pr-links-test.cjs && node scripts/task-draft-delete-test.cjs && node scripts/task-discussion-delivery-test.cjs && node scripts/task-resume-test.cjs && node scripts/task-unblock-test.cjs && node scripts/unblock-run-dedupe-test.cjs && node scripts/audience-session-access-test.cjs && node scripts/warm-chat-test.cjs && node scripts/poke-warm-caller-test.cjs && node scripts/wakeup-queue-test.cjs && node scripts/stranded-human-stop-test.cjs && node scripts/inject-submit-test.cjs && node scripts/blocked-routing-test.cjs && node scripts/self-dispatch-guard-test.cjs && node scripts/task-proposals-test.cjs && node scripts/npm-boundary-test.cjs && node scripts/agent-edit-guard-test.cjs && node scripts/per-agent-context-test.cjs && node scripts/goal-update-guard-test.cjs && node scripts/insights-signal-test.cjs && node scripts/outcome-derivation-test.cjs && node scripts/episode-quality-test.cjs && node scripts/memory-upkeep-test.cjs && node scripts/automem-health-test.cjs && node scripts/memory-store-switch-test.cjs && node scripts/memory-preload-test.cjs && node scripts/turn-lifecycle-test.cjs && node scripts/resume-seed-test.cjs && node scripts/outcome-vocabulary-test.cjs && node scripts/skill-presets-test.cjs && node scripts/skill-edit-proposal-test.cjs && node scripts/notify-hook-route-test.cjs && node scripts/review-notify-test.cjs && node scripts/turn-idle-background-guard-test.cjs && node scripts/waiting-brief-test.cjs && node scripts/runtime-death-alert-test.cjs && node scripts/github-per-member-test.cjs && node scripts/github-multi-org-test.cjs && node scripts/card-measurement-test.cjs && node scripts/scheduler-admission-test.cjs && node scripts/tick-liveness-test.cjs && node scripts/audit-mirror-test.cjs && node scripts/request-metrics-test.cjs && node scripts/tool-usage-test.cjs && node scripts/sessions-list-perf-test.cjs && node scripts/summarizer-degradation-test.cjs && node scripts/agent-history-scope-test.cjs && node scripts/webhook-ingress-test.cjs && node scripts/slack-content-filter-test.cjs && node scripts/slack-ingress-test.cjs && node scripts/discord-ingress-test.cjs && node scripts/chat-attachments-test.cjs && node scripts/clickup-task-bridge-test.cjs && node scripts/agentric-commands-test.cjs && node scripts/whats-new-test.cjs && node scripts/opencode-gate-test.cjs && node scripts/protected-path-guard-test.cjs && node scripts/attach-grace-test.cjs && node scripts/attach-file-liveness-test.cjs && node scripts/feed-smoke.cjs && node scripts/activity-classify-test.cjs && node scripts/goal-room-test.cjs && node scripts/secret-rotation-test.cjs && node scripts/update-watch-test.cjs && node scripts/runtime-update-watch-test.cjs && node scripts/setup-wizard-test.cjs && node scripts/md-pdf-test.cjs && node scripts/proposal-surfacing-test.cjs && node scripts/process-janitor-test.cjs && node scripts/detached-work-steer-test.cjs && node scripts/statusline-install-test.cjs && node scripts/docs-create-agent-test.cjs && node scripts/agent-stats-rollup-test.cjs && node scripts/task-discussion-rollup-test.cjs && node scripts/session-insights-stamp-test.cjs && node scripts/loop-stall-attribution-test.cjs && node scripts/session-progress-test.cjs && node scripts/drift-nudge-test.cjs && node scripts/goal-metric-review-test.cjs && node scripts/goal-heartbeat-test.cjs && node scripts/question-durability-test.cjs && node scripts/ask-block-link-test.cjs && node scripts/bets-test.cjs && node scripts/capability-gap-test.cjs && node scripts/workflow-proposal-test.cjs && node scripts/automation-edit-proposal-test.cjs && node scripts/app-card-close-test.cjs", + "test:governance": "node scripts/version-sync-test.cjs && node scripts/event-loop-freeze-test.cjs && node scripts/git-ceiling-test.cjs && node scripts/cron-supersede-test.cjs && node scripts/governance-conformance.cjs && node scripts/tier-a-policy-test.cjs && node scripts/policy-baseline-test.cjs && node scripts/heredoc-intent-test.cjs && node scripts/script-body-gate-test.cjs && node scripts/capability-registry-test.cjs && node scripts/composio-envelope-test.cjs && node scripts/connector-auth-stub-test.cjs && node scripts/composio-identity-test.cjs && node scripts/idle-reaper-test.cjs && node scripts/dm-continuity-test.cjs && node scripts/telegram-dm-lane-test.cjs && node scripts/cli-link-origin-test.cjs && node scripts/alert-staleness-test.cjs && node scripts/run-as-identity-test.cjs && node scripts/email-identity-guard-test.cjs && node scripts/deps-freshness-test.cjs && node scripts/runtime-account-test.cjs && node scripts/runtime-account-misattribution-test.cjs && node scripts/runtime-usage-refresh-test.cjs && node scripts/keychain-credential-test.cjs && node scripts/credential-preflight-test.cjs && node scripts/capacity-queue-test.cjs && node scripts/runtime-pool-signed-out-test.cjs && node scripts/login-expiry-test.cjs && node scripts/runtime-login-test.cjs && node scripts/rotate-on-reload-test.cjs && node scripts/headless-resumable-test.cjs && node scripts/session-revive-gates-test.cjs && node scripts/session-pause-test.cjs && node scripts/make-live-lock-test.cjs && node scripts/claude-config-seed-test.cjs && node scripts/claude-config-isolation-test.cjs && node scripts/output-style-test.cjs && node scripts/session-cost-test.cjs && node scripts/chain-model-test.cjs && node scripts/task-workers-test.cjs && node scripts/tuning-patch-test.cjs && node scripts/task-runs-test.cjs && node scripts/task-pr-links-test.cjs && node scripts/task-draft-delete-test.cjs && node scripts/task-discussion-delivery-test.cjs && node scripts/task-resume-test.cjs && node scripts/task-unblock-test.cjs && node scripts/unblock-run-dedupe-test.cjs && node scripts/audience-session-access-test.cjs && node scripts/warm-chat-test.cjs && node scripts/poke-warm-caller-test.cjs && node scripts/wakeup-queue-test.cjs && node scripts/stranded-human-stop-test.cjs && node scripts/inject-submit-test.cjs && node scripts/blocked-routing-test.cjs && node scripts/self-dispatch-guard-test.cjs && node scripts/task-proposals-test.cjs && node scripts/npm-boundary-test.cjs && node scripts/agent-edit-guard-test.cjs && node scripts/per-agent-context-test.cjs && node scripts/goal-update-guard-test.cjs && node scripts/insights-signal-test.cjs && node scripts/outcome-derivation-test.cjs && node scripts/episode-quality-test.cjs && node scripts/memory-upkeep-test.cjs && node scripts/automem-health-test.cjs && node scripts/memory-store-switch-test.cjs && node scripts/memory-preload-test.cjs && node scripts/turn-lifecycle-test.cjs && node scripts/resume-seed-test.cjs && node scripts/outcome-vocabulary-test.cjs && node scripts/skill-presets-test.cjs && node scripts/skill-edit-proposal-test.cjs && node scripts/notify-hook-route-test.cjs && node scripts/review-notify-test.cjs && node scripts/turn-idle-background-guard-test.cjs && node scripts/waiting-brief-test.cjs && node scripts/runtime-death-alert-test.cjs && node scripts/github-per-member-test.cjs && node scripts/github-multi-org-test.cjs && node scripts/card-measurement-test.cjs && node scripts/scheduler-admission-test.cjs && node scripts/tick-liveness-test.cjs && node scripts/audit-mirror-test.cjs && node scripts/request-metrics-test.cjs && node scripts/tool-usage-test.cjs && node scripts/sessions-list-perf-test.cjs && node scripts/summarizer-degradation-test.cjs && node scripts/agent-history-scope-test.cjs && node scripts/webhook-ingress-test.cjs && node scripts/slack-content-filter-test.cjs && node scripts/slack-ingress-test.cjs && node scripts/discord-ingress-test.cjs && node scripts/chat-attachments-test.cjs && node scripts/clickup-task-bridge-test.cjs && node scripts/agentric-commands-test.cjs && node scripts/whats-new-test.cjs && node scripts/opencode-gate-test.cjs && node scripts/protected-path-guard-test.cjs && node scripts/attach-grace-test.cjs && node scripts/attach-file-liveness-test.cjs && node scripts/feed-smoke.cjs && node scripts/activity-classify-test.cjs && node scripts/goal-room-test.cjs && node scripts/secret-rotation-test.cjs && node scripts/update-watch-test.cjs && node scripts/runtime-update-watch-test.cjs && node scripts/setup-wizard-test.cjs && node scripts/md-pdf-test.cjs && node scripts/proposal-surfacing-test.cjs && node scripts/process-janitor-test.cjs && node scripts/detached-work-steer-test.cjs && node scripts/statusline-install-test.cjs && node scripts/docs-create-agent-test.cjs && node scripts/agent-stats-rollup-test.cjs && node scripts/task-discussion-rollup-test.cjs && node scripts/session-insights-stamp-test.cjs && node scripts/loop-stall-attribution-test.cjs && node scripts/session-progress-test.cjs && node scripts/drift-nudge-test.cjs && node scripts/goal-metric-review-test.cjs && node scripts/goal-heartbeat-test.cjs && node scripts/question-durability-test.cjs && node scripts/ask-block-link-test.cjs && node scripts/bets-test.cjs && node scripts/capability-gap-test.cjs && node scripts/workflow-proposal-test.cjs && node scripts/automation-edit-proposal-test.cjs && node scripts/app-card-close-test.cjs", "test:alert-staleness": "node scripts/alert-staleness-test.cjs", "test:deps": "node scripts/deps-freshness-test.cjs && node scripts/runtime-account-test.cjs && node scripts/runtime-account-misattribution-test.cjs && node scripts/runtime-login-test.cjs && node scripts/claude-config-seed-test.cjs && node scripts/claude-config-isolation-test.cjs", "test:dm-continuity": "node scripts/dm-continuity-test.cjs", diff --git a/scripts/connector-auth-stub-test.cjs b/scripts/connector-auth-stub-test.cjs new file mode 100644 index 00000000..1e75fc8d --- /dev/null +++ b/scripts/connector-auth-stub-test.cjs @@ -0,0 +1,121 @@ +#!/usr/bin/env node +/* + * A remote MCP connector that lost its authorization: caught at the gate, carded to the admins. + * + * When a remote MCP server 401s at session start, Claude Code replaces its tools with OAuth stubs + * (`mcp____authenticate` / `__complete_authentication`). Live, this was DataForSEO's hosted URL + * with no auth header (an OAuth grant that expired) and initech's `composio-company` Tool Router. Each + * time an unattended agent found an `authenticate` tool, could not finish a browser login, and nothing + * told an admin. The fix has two halves, pinned here: + * 1. the gate refuses a stub call on an unattended run with a reason the agent can act on, and posts + * ONE admin card per connector per cooldown (an attended session passes, since its human can log in); + * 2. DataForSEO gets catalog entries that run the official server locally over stdio with the API + * login from the vault, PINNED (v2→v3 renamed every tool), and npx connectors are pre-fetched + * so a cold install (21–35s measured) can't time out a session's MCP startup. + * + * Usage: npm run build && node scripts/connector-auth-stub-test.cjs + */ +const fs = require('fs'); +const os = require('os'); +const path = require('path'); + +const ROOT = path.resolve(__dirname, '..'); +const HOME = fs.mkdtempSync(path.join(os.tmpdir(), 'aos-authstub-test-')); +process.env.AGENT_OS_HOME = HOME; +process.env.AGENT_OS_TENANT = 'testco'; +process.env.AOS_NO_TTYD = '1'; +delete process.env.AGENT_OS_SECRET_KEY; + +let pass = 0, fail = 0; +const ok = (n) => { pass++; console.log(` \x1b[32m✓\x1b[0m ${n}`); }; +const bad = (n, d) => { fail++; console.log(` \x1b[31m✗ ${n}\x1b[0m${d ? `\n ${d}` : ''}`); }; +const assert = (c, n, d) => (c ? ok(n) : bad(n, d)); + +async function main() { + const { parseAuthStub, isHostedDataForSeo, authStubRemedy } = require(path.join(ROOT, 'dist/connectors/mcp-auth-stub.js')); + const { npxSpec, warmNpxConnectors } = require(path.join(ROOT, 'dist/connectors/npx-warm.js')); + const { CATALOG } = require(path.join(ROOT, 'dist/connectors/connectors.js')); + + console.log('\n\x1b[1m1) Recognising the stub\x1b[0m'); + assert(JSON.stringify(parseAuthStub('mcp__dataforseo__authenticate')) === '{"server":"dataforseo","step":"authenticate"}', 'mcp__dataforseo__authenticate → server dataforseo'); + assert(parseAuthStub('mcp__composio-company__complete_authentication')?.server === 'composio-company', 'a hyphenated server name parses whole'); + assert(parseAuthStub('mcp__dataforseo__serp_organic_live_advanced') === null, 'a real tool is not a stub'); + assert(parseAuthStub('mcp__agentos__authenticate') === null, "the OS's own server is never treated as a connector"); + assert(parseAuthStub(undefined) === null && parseAuthStub('authenticate') === null, 'no tool / bare name → null'); + assert(isHostedDataForSeo('https://mcp.dataforseo.com/v3/mcp') && isHostedDataForSeo('https://mcp.dataforseo.com/mcp'), 'both hosted DataForSEO URLs are recognised'); + assert(!isHostedDataForSeo('https://evil.example/mcp.dataforseo.com') && !isHostedDataForSeo(''), '…and nothing else is'); + assert(/catalog/.test(authStubRemedy({ url: 'https://mcp.dataforseo.com/mcp' })), 'the hosted-DataForSEO remedy points at the catalog entry'); + + console.log('\n\x1b[1m2) DataForSEO catalog entries run locally, pinned, with vault creds\x1b[0m'); + for (const [type, ver] of [['dataforseo', /^3\./], ['dataforseo-v2', /^2\./]]) { + const e = CATALOG.find((c) => c.type === type); + const spec = e && npxSpec({ transport: e.transport, command: e.command, args: e.args }); + const m = spec && /^dataforseo-mcp-server@(\d+\.\d+\.\d+)$/.exec(spec); + assert(e?.transport === 'stdio' && m && ver.test(m[1]), `${type}: stdio, exact pinned version (${spec})`); + assert(e?.fields.map((f) => f.key).join() === 'DATAFORSEO_LOGIN,DATAFORSEO_PASSWORD' && e.fields.every((f) => /secret:/.test(f.placeholder || '')), `${type}: API login + password fields, vault refs suggested`); + } + + console.log('\n\x1b[1m3) npx pre-fetch\x1b[0m'); + assert(npxSpec({ transport: 'stdio', command: 'npx', args: ['-y', 'resend-mcp'] }) === 'resend-mcp', 'reads the spec after -y'); + assert(npxSpec({ transport: 'stdio', command: 'node', args: ['x.js'] }) === undefined, 'not npx → nothing to warm'); + assert(npxSpec({ transport: 'stdio', command: 'npx', args: ['--registry', 'r', 'pkg'] }) === undefined, 'an option it does not model → no guess'); + const ran = []; + let release; + const gate = new Promise((r) => { release = r; }); + const fake = (spec) => { ran.push(spec); return gate.then(() => true); }; + const conn = (id, args, enabled = true) => ({ id, transport: 'stdio', command: 'npx', args, enabled }); + const started = warmNpxConnectors([conn('a', ['-y', 'p@1.0.0']), conn('b', ['-y', 'p@1.0.0']), conn('c', ['-y', 'q@2.0.0'], false)], undefined, fake); + assert(started.join() === 'p@1.0.0' && ran.length === 1, 'one fetch per spec; disabled connectors skipped'); + assert(warmNpxConnectors([conn('a', ['-y', 'p@1.0.0'])], undefined, fake).length === 0, 'a spec already in flight is not fetched twice'); + release(); await gate; await new Promise((r) => setImmediate(r)); + assert(warmNpxConnectors([conn('a', ['-y', 'p@1.0.0'])], undefined, fake).length === 1, '…and can be fetched again once it finished'); + + console.log('\n\x1b[1m4) The gate\x1b[0m'); + const { TenantRegistry } = require(path.join(ROOT, 'dist/tenant-registry.js')); + const { TerminalManager } = require(path.join(ROOT, 'dist/terminal.js')); + const registry = new TenantRegistry(ROOT, 0); + registry.bootAll(); + const osx = registry.get('testco').os; + const tm = new TerminalManager(osx, 'http://127.0.0.1:1', path.join(HOME, 'tmux.sock')); + const db = osx.db; + osx.connectors.add({ type: 'custom-remote', label: 'DataforSEO', transport: 'http', url: 'https://mcp.dataforseo.com/mcp' }); + const mkSession = (id, headless, claimedBy = null) => db.prepare( + 'INSERT INTO term_sessions (id, agent, title, task, tmux, status, spawned_by, run_as, secret, headless, claimed_by, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', + ).run(id, 'seo', id, '', id, 'running', 'automation:x', null, 's', headless, claimedBy, Date.now(), Date.now()); + mkSession('s_unattended', 1); + mkSession('s_unattended2', 1); + mkSession('s_attended', 0); + mkSession('s_claimed', 1, 'owner@testco.example'); + const call = (sid, tool) => tm.gate(sid, 'seo', 'connector.call', { tool, input: {} }, 'test'); + const cards = (server) => db.prepare("SELECT title, body, status FROM messages WHERE session_id = ? ORDER BY created_at").all(`system:connector-auth:${server}`); + const audits = (type) => db.prepare('SELECT data FROM audit_events WHERE type = ?').all(type).map((r) => JSON.parse(r.data)); + + const r1 = call('s_unattended', 'mcp__dataforseo__authenticate'); + assert(r1.decision === 'deny' && /lost its authorization/.test(r1.reason) && /Do not retry/.test(r1.reason), 'unattended run: the stub is refused with a reason the agent can act on', JSON.stringify(r1)); + const c1 = cards('dataforseo'); + assert(c1.length === 1 && /DataforSEO \(dataforseo\)/.test(c1[0].title) && /catalog/.test(c1[0].body), 'one admin card naming the connector and the catalog fix', JSON.stringify(c1)); + const lost = audits('connector.auth.lost'); + assert(lost.length === 1 && lost[0].connector === 'dataforseo' && lost[0].unattended === true && lost[0].url === 'https://mcp.dataforseo.com/mcp', 'audited as connector.auth.lost with the URL'); + + call('s_unattended2', 'mcp__dataforseo__complete_authentication'); + assert(cards('dataforseo').length === 1, 'a second hit inside the cooldown does not post a second card'); + assert(audits('connector.auth.lost').length === 2, '…but is still audited'); + + const r3 = call('s_attended', 'mcp__composio-company__authenticate'); + assert(r3.decision !== 'deny' || !/lost its authorization/.test(r3.reason || ''), 'attended session: not refused by the stub check (its human can log in)', JSON.stringify(r3)); + const cc = cards('composio-company'); + assert(cc.length === 1 && /Composio API key/.test(cc[0].body), '…but the admins still get a card, with the Composio remedy'); + + const r4 = call('s_claimed', 'mcp__other__authenticate'); + assert(!/lost its authorization/.test(r4.reason || ''), 'a taken-over (claimed) run counts as attended'); + + const r5 = call('s_unattended', 'mcp__dataforseo__serp_organic_live_advanced'); + assert(!/lost its authorization/.test(r5.reason || ''), 'a real connector tool is untouched'); + + console.log(`\n${fail === 0 ? '\x1b[32mPASS' : '\x1b[31mFAIL'}\x1b[0m ${pass} passed, ${fail} failed`); + registry.stopAll?.(); + fs.rmSync(HOME, { recursive: true, force: true }); + process.exit(fail === 0 ? 0 : 1); +} + +main().catch((e) => { console.error(e); fs.rmSync(HOME, { recursive: true, force: true }); process.exit(1); }); diff --git a/src/connectors/connectors.ts b/src/connectors/connectors.ts index e233ab41..925fa20d 100644 --- a/src/connectors/connectors.ts +++ b/src/connectors/connectors.ts @@ -119,6 +119,57 @@ export const CATALOG: CatalogEntry[] = [ }, ], }, + // DataForSEO runs LOCALLY over stdio with the API login/password, never the hosted + // `mcp.dataforseo.com` URL. That endpoint defaults to OAuth, and an OAuth grant lives in Claude Code's + // own per-config-dir store: it expires, it is absent on a pooled runtime account, and when it lapses + // Claude Code silently swaps every tool for `authenticate`/`complete_authentication` stubs an + // unattended run cannot complete — the "DataForSEO keeps disconnecting" reports. On stdio every tool + // call is a plain Basic-auth REST call with nothing to expire. Versions are PINNED: v2→v3 renamed the + // whole tool surface (89 per-endpoint tools → a generic `api_request`), and agent prompts name tools. + { + type: 'dataforseo', + label: 'DataForSEO', + description: 'SEO data (SERPs, keywords, backlinks, Labs) via the official DataForSEO MCP server, run locally with your API login — no OAuth to expire. v3 tool surface: api_request + docs_search.', + transport: 'stdio', + command: 'npx', + args: ['-y', 'dataforseo-mcp-server@3.1.3'], + fields: [ + { + key: 'DATAFORSEO_LOGIN', + label: 'API login', + placeholder: 'secret:DATAFORSEO_LOGIN', + help: 'The API login from app.dataforseo.com/api-access (not your dashboard login). Store it in Settings → Secrets and enter secret:DATAFORSEO_LOGIN here.', + }, + { + key: 'DATAFORSEO_PASSWORD', + label: 'API password', + placeholder: 'secret:DATAFORSEO_PASSWORD', + help: 'The API password from the same page. Enter secret:DATAFORSEO_PASSWORD to pull it from the encrypted vault.', + }, + ], + }, + { + type: 'dataforseo-v2', + label: 'DataForSEO (v2 tools)', + description: 'The same server pinned to v2: one tool per endpoint (serp_organic_live_advanced, kw_data_google_ads_search_volume, …). Pick this when agent prompts already name those tools.', + transport: 'stdio', + command: 'npx', + args: ['-y', 'dataforseo-mcp-server@2.9.13'], + fields: [ + { + key: 'DATAFORSEO_LOGIN', + label: 'API login', + placeholder: 'secret:DATAFORSEO_LOGIN', + help: 'The API login from app.dataforseo.com/api-access (not your dashboard login). Store it in Settings → Secrets and enter secret:DATAFORSEO_LOGIN here.', + }, + { + key: 'DATAFORSEO_PASSWORD', + label: 'API password', + placeholder: 'secret:DATAFORSEO_PASSWORD', + help: 'The API password from the same page. Enter secret:DATAFORSEO_PASSWORD to pull it from the encrypted vault.', + }, + ], + }, { type: 'custom', label: 'Custom (local)', diff --git a/src/connectors/mcp-auth-stub.ts b/src/connectors/mcp-auth-stub.ts new file mode 100644 index 00000000..912308cf --- /dev/null +++ b/src/connectors/mcp-auth-stub.ts @@ -0,0 +1,73 @@ +/** + * A remote MCP connector that lost its authorization, detected from the one signal that is certain. + * + * When a remote (http/sse) MCP server answers 401 at session start, Claude Code does not fail the + * server. It starts the OAuth discovery flow and replaces the server's tools with two stubs, + * `mcp____authenticate` and `mcp____complete_authentication`. The agent sees a tool + * list with nothing it asked for. The stubs return a browser URL, and an unattended run has nobody to + * open it. + * + * Seen live twice with two different causes: + * - DataForSEO's hosted URL (`mcp.dataforseo.com`) with no headers, so authorized by an OAuth grant + * in Claude Code's own store. The grant expires and is absent on a pooled runtime account. + * 20 stub calls in one tenant's transcripts. + * - initech's `composio-company` Tool Router came up unauthenticated from 2026-09-08 and took GSC, + * GA4, DataForSEO, Stripe and Sheets dark with it. Agents filed tasks about it; nothing told an + * admin it was down. + * + * Calling a stub IS the detection: it is a connector tool call, so it reaches the gate. The gate turns + * it into a refusal the agent can act on and one admin card per connector. + */ + +export interface AuthStub { + /** The MCP server name: the connector id, or `composio` / `composio-company`. */ + server: string; + step: 'authenticate' | 'complete_authentication'; +} + +const STUB = /^mcp__(.+)__(authenticate|complete_authentication)$/; + +/** Parse a tool name. Only Claude Code's `mcp____` spelling: the stubs are its own, and + * codex/opencode don't run an OAuth flow for a 401. */ +export function parseAuthStub(tool: unknown): AuthStub | null { + if (typeof tool !== 'string') return null; + const m = STUB.exec(tool); + if (!m || m[1] === 'agentos') return null; + return { server: m[1], step: m[2] as AuthStub['step'] }; +} + +export interface StubConnectorInfo { + /** Display label, when the server maps to a connector row. */ + label?: string; + /** Its URL, when it is a remote connector row. */ + url?: string; + /** Composio's minted servers, which have no connector row. */ + composio?: boolean; +} + +/** DataForSEO's hosted endpoint. On that URL the fix is the stdio catalog entry, not a re-login. */ +export function isHostedDataForSeo(url: string | undefined): boolean { + return !!url && /(^|\/\/)mcp\.dataforseo\.com(\/|$)/i.test(url); +} + +/** The fix to suggest, phrased for an admin. */ +export function authStubRemedy(info: StubConnectorInfo): string { + if (isHostedDataForSeo(info.url)) { + return 'This connector points at DataForSEO\'s hosted MCP URL, which authorizes with an OAuth login that expires. ' + + 'Replace it with the "DataForSEO" (or "DataForSEO (v2 tools)") connector from the catalog: it runs locally with your API login from the vault and has nothing to expire. ' + + 'Keep the same connector name so agents\' tool names don\'t change.'; + } + if (info.composio) { + return 'This is a Composio Tool Router session. Check the Composio API key in Settings → Integrations, and that the app\'s connection is still active in Composio.'; + } + return 'Check the credentials in the connector\'s auth headers. A remote connector with no auth header falls back to an OAuth login that expires and cannot be completed by an unattended run; give it a header (or an API-key based connector) instead.'; +} + +/** What the agent is told, in place of the stub. */ +export function authStubDenial(stub: AuthStub, unattended: boolean): string { + const lead = `Agentric: the "${stub.server}" connector has lost its authorization. Its real tools are unavailable in this session, and Claude Code put this ${stub.step} stub in their place.`; + if (unattended) { + return `${lead} This run is unattended, so there is nobody to complete a browser login, and the admins have been notified. Do not retry it or look for another way to log in. Carry on without "${stub.server}" if the task allows, otherwise say in your report that it was blocked on this connector.`; + } + return `${lead} The admins have been notified.`; +} diff --git a/src/connectors/npx-warm.ts b/src/connectors/npx-warm.ts new file mode 100644 index 00000000..7cb48f86 --- /dev/null +++ b/src/connectors/npx-warm.ts @@ -0,0 +1,75 @@ +/** + * Pre-fetch the npm package behind every `npx` connector, so a session's MCP server starts warm. + * + * Claude Code starts each stdio MCP server when the session launches and gives up on one that has not + * answered `initialize` in time. A cold `npx -y @` downloads and installs the package first: + * measured 21s (`dataforseo-mcp-server@2.9.13`) and 35s (`@3.1.3`) on the Mac Mini, against ~0.6s + * warm. So the first session after a connector is added, or after npm evicts its cache, comes up with + * the connector silently missing. That reads exactly like the connector being "flaky". + * + * `npx -y --package= -- node -e 0` installs into the SAME `_npx` cache entry a later + * `npx -y ` resolves (the cache key is the package list, not the command run), then exits at + * once without starting the server. Fire-and-forget, async (`spawn`, never `spawnSync`, which stalls + * the single-threaded server), and at most one in flight per spec. + */ +import { spawn } from 'node:child_process'; +import type { McpConnector } from './connectors'; +import type { AgentOS } from '../kernel'; + +/** The package spec an `npx` connector launches (`resend-mcp`, `dataforseo-mcp-server@3.1.3`), or + * undefined when it isn't an npx launch or the spec can't be read off its args. */ +export function npxSpec(c: Pick): string | undefined { + if (c.transport !== 'stdio' || c.command.trim() !== 'npx') return undefined; + for (const a of c.args) { + if (a === '-y' || a === '--yes') continue; + if (a.startsWith('--package=')) return a.slice('--package='.length) || undefined; + if (a.startsWith('-')) return undefined; // an option we don't model — don't guess + return a; + } + return undefined; +} + +const WARM_TIMEOUT_MS = 5 * 60_000; +const inFlight = new Set(); + +export type WarmRunner = (spec: string) => Promise; + +const npxRunner: WarmRunner = (spec) => new Promise((resolve) => { + let child; + try { + child = spawn('npx', ['-y', `--package=${spec}`, '--', 'node', '-e', '0'], { stdio: 'ignore', env: process.env }); + } catch { resolve(false); return; } + const t = setTimeout(() => { try { child.kill('SIGKILL'); } catch { /* gone */ } }, WARM_TIMEOUT_MS); + child.on('error', () => { clearTimeout(t); resolve(false); }); + child.on('exit', (code) => { clearTimeout(t); resolve(code === 0); }); +}); + +/** Warm every ENABLED npx connector. Returns the specs it started; `onDone` reports each outcome. */ +export function warmNpxConnectors( + connectors: McpConnector[], + onDone?: (spec: string, ok: boolean) => void, + run: WarmRunner = npxRunner, +): string[] { + const started: string[] = []; + for (const c of connectors) { + if (!c.enabled) continue; + const spec = npxSpec(c); + if (!spec || inFlight.has(spec)) continue; + inFlight.add(spec); + started.push(spec); + void run(spec) + .catch(() => false) + .then((ok) => { inFlight.delete(spec); try { onDone?.(spec, ok); } catch { /* advisory */ } }); + } + return started; +} + +/** Warm a tenant's npx connectors (all enabled ones, or just `only`), auditing each fetch's outcome as + * `connector.warmed`. A failure isn't fatal: the session's own `npx` will try the same install. */ +export function warmConnectorPackages(os: AgentOS, only?: McpConnector[]): string[] { + try { + return warmNpxConnectors(only ?? os.connectors.list(), (spec, ok) => { + os.audit.append({ ts: Date.now(), runId: '-', tenant: os.tenant, principal: 'system', type: 'connector.warmed', data: { spec, ok } }); + }); + } catch { return []; } +} diff --git a/src/server.ts b/src/server.ts index 32ea6c44..4f1286d5 100644 --- a/src/server.ts +++ b/src/server.ts @@ -79,6 +79,7 @@ import { UpdateWatch } from './edge/update-watch'; import { RuntimeUpdateWatch } from './edge/runtime-update-watch'; import { checkDeps, checkDepUpdates, installDeps, updateNpmDep } from './edge/deps'; import { CATALOG, redact } from './connectors/connectors'; +import { warmConnectorPackages } from './connectors/npx-warm'; import { GithubIdentity } from './edge/github-identity'; import { PrCache, taskPrRefs, taskPrRefsBulk, prSummary, type PrToken } from './edge/task-prs'; import { convertAppManifest, userInstallationStatus } from './connectors/github'; @@ -7559,6 +7560,7 @@ async function handle(os: AgentOS, tm: TerminalManager, autos: Automations, req: scope, ownerMemberId: scope === 'personal' ? me.id : undefined, }); + warmConnectorPackages(os, [created]); return sendJson(res, 200, redact(created)); } catch (e) { return sendJson(res, 400, { error: e instanceof Error ? e.message : String(e) }); @@ -7589,6 +7591,7 @@ async function handle(os: AgentOS, tm: TerminalManager, autos: Automations, req: return sendJson(res, updated ? 200 : 404, updated ? redact(updated) : { error: 'not found' }); } const updated = os.connectors.setEnabled(id, !!b.enabled); + if (updated?.enabled) warmConnectorPackages(os, [updated]); return sendJson(res, updated ? 200 : 404, updated ? redact(updated) : { error: 'not found' }); } } diff --git a/src/tenant-registry.ts b/src/tenant-registry.ts index 4c8e197e..d7ceb623 100644 --- a/src/tenant-registry.ts +++ b/src/tenant-registry.ts @@ -31,6 +31,7 @@ import { Audience, approvalAudience, resolveRecipients } from './governance/reci import { ChatPlatform, chatLink, consolePage } from './governance/chat-links'; import { controlHome, resolvePaths, resolveTenantPaths } from './home'; import { ensureNpmBoundary } from './edge/npm-boundary'; +import { warmConnectorPackages } from './connectors/npx-warm'; import { TenantRecord, TenantStore } from './state/control'; import { requestMetrics } from './edge/request-metrics'; @@ -268,6 +269,8 @@ export class TenantRegistry { const tm = new TerminalManager(os, this.loopbackBase, paths.tmuxSocket, consoleOrigin); const autos = new Automations(os, tm); autos.start(); + // Fetch npx connectors' packages now, so the first session doesn't time out on a cold install. + warmConnectorPackages(os); // Hosted apps: the supervisor spawns each app on demand + idle-reaps it. In-session/app loopback // calls carry the tenant via x-aos-tenant, so it's exported into every app's env. const apps = new AppSupervisor(os.apps, { diff --git a/src/terminal.ts b/src/terminal.ts index 9ed941cb..fab8d951 100644 --- a/src/terminal.ts +++ b/src/terminal.ts @@ -28,6 +28,7 @@ import { readExecutedScripts, freshSince, type ScriptBody } from './governance/s import { isolateClaudeConfig } from './edge/config-isolation'; import { resolveCapability } from './capabilities/normalize'; import { unwrapComposioEnvelope } from './capabilities/composio-envelope'; +import { parseAuthStub, authStubDenial, authStubRemedy, type AuthStub } from './connectors/mcp-auth-stub'; import { briefFor } from './governance/briefer'; import { ReliabilityMonitor } from './edge/reliability'; import { DriftMonitor, DriftJudgement, buildDriftInput, judgeDriftConfirmed, driftNote, driftEscalationBody, askGist } from './edge/drift'; @@ -5932,6 +5933,47 @@ export class TerminalManager { return readExecutedScripts(command, agentDir, freshSince(started)); } + private connectorAuthAlertAt = new Map(); + private static readonly CONNECTOR_AUTH_ALERT_COOLDOWN_MS = 6 * 60 * 60_000; + + /** Handle a call to a connector's OAuth stub: audit it, card the admins (once per connector per + * cooldown), and refuse it on an unattended run. Returns the gate result to use, or null to let an + * attended session through the normal policy path (a human at the keyboard can finish the login). */ + private connectorAuthLost(sessionId: string, agent: string, stub: AuthStub): GateResult | null { + const row = this.db.prepare('SELECT headless, claimed_by FROM term_sessions WHERE id = ?') + .get<{ headless: number; claimed_by: string | null }>(sessionId); + const unattended = !!row && row.headless === 1 && !row.claimed_by; + const conn = this.os.connectors.get(stub.server); + const composio = !conn && (stub.server === 'composio' || stub.server.startsWith('composio-')); + this.audit(sessionId, agent, 'connector.auth.lost', { + connector: stub.server, step: stub.step, unattended, + ...(conn ? { transport: conn.transport, url: conn.url || null } : {}), + ...(composio ? { composio: true } : {}), + }); + const now = Date.now(); + if (now - (this.connectorAuthAlertAt.get(stub.server) ?? 0) >= TerminalManager.CONNECTOR_AUTH_ALERT_COOLDOWN_MS) { + this.connectorAuthAlertAt.set(stub.server, now); + const topic = `connector-auth:${stub.server}`; + const name = conn?.label ? `${conn.label} (${stub.server})` : stub.server; + try { + this.closeSystemCards(topic); + this.postSystemCard({ + topic, + type: 'notification', + title: `Connector "${name}" has lost its authorization`, + body: `Agents can't use its tools: Claude Code replaced them with an OAuth "${stub.step}" stub, which ${agent} just hit (session ${sessionId}). ${unattended ? 'That run was unattended, so it was told to carry on without the connector.' : 'That session is attended, so its human can finish the login there.'}\n\n${authStubRemedy({ label: conn?.label, url: conn?.url, composio })}`, + audience: { kind: 'admins' }, + args: { connector: stub.server }, + link: { page: 'connectors', label: 'Connectors' }, + }); + } catch { /* the audit line is the durable record */ } + } + if (!unattended) return null; + const reason = authStubDenial(stub, true); + this.audit(sessionId, agent, 'gate.decision', { capability: 'connector.call', decision: { effect: 'deny', riskClass: 'deny', reason } }); + return { decision: 'deny', reason, capability: 'connector.call' }; + } + /** The gate. Same policy brain as the console — allow flows, ask → inbox approval (auto-cleared for * an attended approver), never → deny. Args are enriched into facts first (the single classifier). */ gate(sessionId: string, agent: string, capability: string, rawArgs: Record, reasoning: string, subagent?: { type?: string; id?: string }): GateResult { @@ -5945,6 +5987,14 @@ export class TerminalManager { this.audit(sessionId, agent, 'gate.killswitch', { capability }); return { decision: 'deny', reason: 'workspace emergency stop (kill switch) is engaged — every action is blocked until an owner disengages it', capability }; } + // A remote MCP connector that lost its authorization (mcp-auth-stub.ts): Claude Code swaps its tools + // for OAuth stubs, and calling one is the only certain signal that it happened. Unattended runs + // can't finish a browser login, so refuse there; either way, tell an admin which connector is down. + const stub = capability === 'connector.call' ? parseAuthStub(rawArgs.tool) : null; + if (stub) { + const verdict = this.connectorAuthLost(sessionId, agent, stub); + if (verdict) return verdict; + } // Host-egress governance (Phase 2b): OFF unless the workspace switch is on. When on, pass the // agent's granted host matchers (org + shared + the session's run-as member's personal) so the // enricher can parse the egress target and compute host facts. `null` = feature off.