Skip to content

ci: scope docs-agent-eval triggers and add weekly cron safety net #923

@leggetter

Description

@leggetter

Context

.github/workflows/docs-agent-eval-ci.yml runs the docs agent evaluation (scenarios 01 + 02) against a live Outpost. Each run bills Anthropic for the agent + LLM judge, so trigger scope matters.

Current state (file):

  • workflow_dispatch
  • push on main with paths filter
  • pull_request with paths filter
  • ❌ No scheduled cron — no drift safety net
  • ⚠️ Path list includes sdks/outpost-typescript/**, which fires on every Speakeasy bot SDK regen — likely too broad for the eval's value.

What we want

  1. Add a weekly cron as a low-frequency safety net so docs/eval regressions caught even if no triggering PR landed that week.
  2. Audit the path list — remove or scope sdks/outpost-typescript/** so bot-driven SDK regen PRs don't fire the eval (which costs $ per run). Keep paths that genuinely affect docs content: docs/content/**, docs/apis/**, docs/agent-evaluation/**, docs/README.md, docs/AGENTS.md, the workflow file itself.

Suggested patch:

on:
  workflow_dispatch:
  schedule:
    - cron: '0 6 * * 1'   # Mondays 06:00 UTC — weekly safety net
  push:
    branches:
      - main
    paths:
      - "docs/content/**"
      - "docs/apis/**"
      - "docs/agent-evaluation/**"
      - "docs/README.md"
      - "docs/AGENTS.md"
      - ".github/workflows/docs-agent-eval-ci.yml"
      # consider removing: - "sdks/outpost-typescript/**"
  pull_request:
    paths:
      - "docs/content/**"
      - "docs/apis/**"
      - "docs/agent-evaluation/**"
      - "docs/README.md"
      - "docs/AGENTS.md"
      - ".github/workflows/docs-agent-eval-ci.yml"
      # consider removing: - "sdks/outpost-typescript/**"

Acceptance

  • Workflow fires on PRs touching docs paths only
  • Weekly cron runs successfully without manual intervention
  • workflow_dispatch still works for ad-hoc runs
  • Cost savings observable in Anthropic billing (no eval runs from SDK-regen PRs)

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions