Context
.github/workflows/docs-agent-eval-ci.yml runs the docs agent evaluation (scenarios 01 + 02) against a live Outpost. Each run bills Anthropic for the agent + LLM judge, so trigger scope matters.
Current state (file):
- ✅
workflow_dispatch
- ✅
push on main with paths filter
- ✅
pull_request with paths filter
- ❌ No scheduled cron — no drift safety net
- ⚠️ Path list includes
sdks/outpost-typescript/**, which fires on every Speakeasy bot SDK regen — likely too broad for the eval's value.
What we want
- Add a weekly cron as a low-frequency safety net so docs/eval regressions caught even if no triggering PR landed that week.
- Audit the path list — remove or scope
sdks/outpost-typescript/** so bot-driven SDK regen PRs don't fire the eval (which costs $ per run). Keep paths that genuinely affect docs content: docs/content/**, docs/apis/**, docs/agent-evaluation/**, docs/README.md, docs/AGENTS.md, the workflow file itself.
Suggested patch:
on:
workflow_dispatch:
schedule:
- cron: '0 6 * * 1' # Mondays 06:00 UTC — weekly safety net
push:
branches:
- main
paths:
- "docs/content/**"
- "docs/apis/**"
- "docs/agent-evaluation/**"
- "docs/README.md"
- "docs/AGENTS.md"
- ".github/workflows/docs-agent-eval-ci.yml"
# consider removing: - "sdks/outpost-typescript/**"
pull_request:
paths:
- "docs/content/**"
- "docs/apis/**"
- "docs/agent-evaluation/**"
- "docs/README.md"
- "docs/AGENTS.md"
- ".github/workflows/docs-agent-eval-ci.yml"
# consider removing: - "sdks/outpost-typescript/**"
Acceptance
- Workflow fires on PRs touching docs paths only
- Weekly cron runs successfully without manual intervention
workflow_dispatch still works for ad-hoc runs
- Cost savings observable in Anthropic billing (no eval runs from SDK-regen PRs)
🤖 Generated with Claude Code
Context
.github/workflows/docs-agent-eval-ci.ymlruns the docs agent evaluation (scenarios 01 + 02) against a live Outpost. Each run bills Anthropic for the agent + LLM judge, so trigger scope matters.Current state (file):
workflow_dispatchpushonmainwithpathsfilterpull_requestwithpathsfiltersdks/outpost-typescript/**, which fires on every Speakeasy bot SDK regen — likely too broad for the eval's value.What we want
sdks/outpost-typescript/**so bot-driven SDK regen PRs don't fire the eval (which costs $ per run). Keep paths that genuinely affect docs content:docs/content/**,docs/apis/**,docs/agent-evaluation/**,docs/README.md,docs/AGENTS.md, the workflow file itself.Suggested patch:
Acceptance
workflow_dispatchstill works for ad-hoc runs🤖 Generated with Claude Code