diff --git a/.github/workflows/probe-issue.yaml b/.github/workflows/probe-issue.yaml index 025e7a1..3a8f477 100644 --- a/.github/workflows/probe-issue.yaml +++ b/.github/workflows/probe-issue.yaml @@ -1,6 +1,8 @@ name: Probe issue on: + issues: + types: [opened] workflow_dispatch: inputs: issue_number: @@ -14,14 +16,15 @@ permissions: pull-requests: write concurrency: - group: probe-issue-${{ github.event.inputs.issue_number }} + group: probe-issue-${{ github.event.issue.number || github.event.inputs.issue_number }} cancel-in-progress: false jobs: probe: runs-on: ubuntu-latest + if: github.event_name == 'workflow_dispatch' || github.event.issue != null env: - ISSUE_NUMBER: ${{ github.event.inputs.issue_number }} + ISSUE_NUMBER: ${{ github.event.issue.number || github.event.inputs.issue_number }} OPENCODE_VERSION: '1.18.16' steps: - name: Checkout diff --git a/AGENT_DESIGN.md b/AGENT_DESIGN.md index 04caf70..77882bd 100644 --- a/AGENT_DESIGN.md +++ b/AGENT_DESIGN.md @@ -2,7 +2,7 @@ ## Goal -A manually dispatched GitHub Action in `basic-charms` that reads an issue describing public charm-dev documentation, writes tests that verify how things actually behave, and opens a PR. The agent is skeptical of the docs: it forms its own understanding of how the code behaves, writes a test asserting that understanding, and deduces what the CI result means for the doc. The user reviews the PR, inspects the CI results, and reads the agent's reasoning to determine whether the doc was validated or refuted. The PR is an artifact for review, not for merging. +A GitHub Action in `basic-charms` that reads an issue describing public charm-dev documentation, writes tests that verify how things actually behave, and opens a PR. The agent is skeptical of the docs: it forms its own understanding of how the code behaves, writes a test asserting that understanding, and deduces what the CI result means for the doc. The user reviews the PR, inspects the CI results, and reads the agent's reasoning to determine whether the doc was validated or refuted. The PR is an artifact for review, not for merging. ## Files (all under `.github/`) @@ -45,7 +45,7 @@ The `fetch_url` tool is the other exception: it lets the agent fetch content fro ## Workflow flow -1. `workflow_dispatch` with `issue_number` (required). +1. Triggered automatically when an issue is opened (`issues: [opened]`), or manually via `workflow_dispatch` with `issue_number` (required) to re-run on an existing issue. 2. Checkout with `persist-credentials: false`, `fetch-depth: 0`. No git credentials in `.git/config` during the agent run. 3. `git config core.hooksPath /dev/null` — defense in depth, inert hooks. 4. Setup Node 24, install `opencode-ai` (version pinned in the `OPENCODE_VERSION` env var in the workflow), set up uv. @@ -147,7 +147,7 @@ Doc-fetch allowlist: only `canonical.com`, `ubuntu.com`, `raw.githubusercontent. Agent staging and cleanup: agent definition and tool files copied to `.opencode/` before the run, removed before diff collection. -Manual dispatch only: no automatic triggers. The user explicitly chooses to run this. +Triggered automatically on issue open, or manually via `workflow_dispatch` to re-run on an existing issue. Repository setting: the repo must have "Allow GitHub Actions to create and approve pull requests" enabled (Settings → Actions → General → Workflow permissions). This is the gate that lets the `GITHUB_TOKEN` create PRs. The workflow declares `pull-requests: write` in its `permissions:` block, but that alone is not enough — the repo-level flag must also be on. The default workflow permission should remain `read` (least privilege); each workflow declares its own `permissions:` block. @@ -223,7 +223,7 @@ OpenCode vulnerability allowing code execution despite `bash: deny`: low, outsid ## Dry-run mode (not implemented) -The workflow has no dry-run mode. The workflow is manually dispatched (a human already chose to run it), the agent can return `BLOCKED` when it cannot proceed, and an unwanted PR is cheap to close and delete. A dry-run mode would add complexity across the input, env vars, conditional steps, and issue-comment branches for a mode whose main use is during initial development of the agent script. +The workflow has no dry-run mode. The workflow is triggered automatically on issue open (or manually for re-runs), the agent can return `BLOCKED` when it cannot proceed, and an unwanted PR is cheap to close and delete. A dry-run mode would add complexity across the input, env vars, conditional steps, and issue-comment branches for a mode whose main use is during initial development of the agent script. If dry-run is wanted later, implement it as follows: diff --git a/README.md b/README.md index c13d099..6204c52 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ To perform an adversarial test: 2. Create an issue in `basic-charms` with a link to the documentation page and a description of the claim you want to test. This could be a quote from the page or an explanation of something the page implies. -3. Run the [Probe issue](https://github.com/dwilding/basic-charms/actions/workflows/probe-issue.yaml) workflow, entering the issue number in the **Run workflow** UI. + The [Probe issue](https://github.com/dwilding/basic-charms/actions/workflows/probe-issue.yaml) workflow runs automatically when an issue is opened. You can also trigger it manually from the Actions page (enter the issue number in the **Run workflow** UI) to re-run the probe on an existing issue. -4. Wait for a PR to be created. +3. Wait for a PR to be created. The PR will modify one or more of the basic charms (and possibly their unit tests or integration tests) to test the documentation claim you described in the issue. @@ -22,11 +22,11 @@ To perform an adversarial test: > I believe `` is true. I added a test asserting it, which is expected to pass. If CI passes, the doc is correct. -5. Review the PR to make sure the agent's changes are meaningful and trustworthy. Once you're satisfied, approve the PR's workflow runs to trigger CI. +4. Review the PR to make sure the agent's changes are meaningful and trustworthy. Once you're satisfied, approve the PR's workflow runs to trigger CI. -6. After the CI checks have completed, use the PR description to draw a conclusion about the documentation. +5. After the CI checks have completed, use the PR description to draw a conclusion about the documentation. -7. Decide how to fix the documentation — if needed. +6. Decide how to fix the documentation — if needed. The agentic workflow that creates the PR is explained in [AGENT_DESIGN.md](AGENT_DESIGN.md). It's a **highly experimental** workflow based on ideas explored in [SecondSkoll/generic-agentic-workflows](https://github.com/SecondSkoll/generic-agentic-workflows). It uses OpenCode, OpenRouter, and GLM-5.2.