Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .config/tend.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ workflows:
# GitHub only ever delays a scheduled run, so the default 09:17 slot
# has always started before the reset.
cron: "17 15 * * 0"
ci-fix:
ci-failures:
watched_workflows:
- CI
- Argos
282 changes: 282 additions & 0 deletions .github/workflows/tend-ci-failures.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,282 @@
# Generated by tend 0.3.16. Regenerate with: uvx tend@latest init
#
# Do not edit this file directly — it will be overwritten on regeneration.
# To change this workflow, edit .config/tend.yaml. To change what the agent
# does, edit the relevant skill (for example, `running-tend`) in this repo's
# .claude/skills/ directory, or open an issue at
# https://github.com/max-sixty/tend/issues for changes that need to
# happen upstream in the tend-ci-runner plugin.

name: tend-ci-failures
# Turns a watched workflow's verdict on a watched branch into its CI failure
# report, with no agent: a failure files the report or comments on it and
# dispatches the report's triage, and a pass closes it. `ci_failure_report.py`
# documents the report, and why it writes with the job's own token rather than
# the bot's.
on:
workflow_run:
workflows: ["CI", "Argos"]
types: [completed]
branches: ["main"]

jobs:
report:
# Only a verdict reaches the group below. On a busy branch most completions
# are runs a newer commit cancelled, and each would otherwise take the
# lane's single pending slot from a verdict waiting in it.
#
# Only this repository's own branches, too. A fork PR opened from the
# fork's `main` runs with `head_branch` set to `main`, which passes the
# branch filter; reporting it would file and close another lane's report,
# with a title and logs the fork's author controls.
if: >-
github.repository_owner == 'diffplug' && vars.TEND_ENABLED != 'false'
&& github.event.workflow_run.head_repository.full_name == github.repository
&& (
github.event.workflow_run.conclusion == 'success'
|| github.event.workflow_run.conclusion == 'failure'
|| github.event.workflow_run.conclusion == 'timed_out'
|| github.event.workflow_run.conclusion == 'startup_failure')
concurrency:
# One writer per report, so one lane's verdicts apply in turn and never
# file two reports. A newer pending verdict replaces an older one, which
# it supersedes anyway.
group: ${{ github.workflow }}-${{ github.event.workflow_run.name }}-${{ github.event.workflow_run.event }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: false
runs-on: ubuntu-24.04
# `actions: write` is what the triage dispatch needs; the job holds no
# secret, and runs only the pinned script below.
permissions:
actions: write
issues: write
steps:
- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
id: tend_uv
env:
UV_NO_MODIFY_PATH: "1"
with:
version: "0.13.5"
checksum: ${{ runner.arch == 'ARM64' && '4bb69cf95ed13f23713bb7bc09f241896ea0c75b7c557e615a20604b82f9c403' || 'f68d291c75783122dad58ab77def0e13f3233f358731db5d6d7e8715603b928f' }}
enable-cache: false
ignore-empty-workdir: true
- name: File, update or close the CI failure report
env:
GH_TOKEN: ${{ github.token }}
TRIAGE_WORKFLOW: "tend-triage.yaml"
TEND_UV: ${{ steps.tend_uv.outputs.uv-path }}
run: |
"$TEND_UV" run --script - <<'TEND_PY'
# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""File, update or close the CI failure report a finished watched run calls for.

`tend-ci-failures` runs this, with no agent, when a watched workflow finishes
on a watched branch of this repository with a verdict. A report is one issue
per workflow, branch and triggering event, the unit a run's verdict speaks
for: a scheduled suite can fail on tests that push runs never reach.

- A failed run files the report, or comments on the one already open, and
dispatches `tend-triage` for it.
- A passing run closes the open report. The cause issues filed under it stay
as they are: a pass doesn't show that a flaky cause is gone, and a cause's
fix may still be in flight, so each cause's own triage closes it.
- A run that a newer finished run of the same lane supersedes changes nothing.
Runs can finish out of order, and the newer verdict is the lane's state.

The job's own `GITHUB_TOKEN` writes, as `github-actions[bot]`, rather than the
bot's token, so the bot is notified of the report and of each later comment;
GitHub notifies nobody of their own activity. `GITHUB_TOKEN` starts no workflow
by opening an issue, but it may dispatch one, and the dispatched triage run is
the fast path. The notification is the fallback, for a dispatch that failed or
a run the rate limit refused: the run marks it read only once it succeeds, and
the notifications poll re-dispatches whatever is left unread.

Inputs (env): ``GH_TOKEN``, ``GITHUB_REPOSITORY``, ``GITHUB_EVENT_PATH`` (a
`workflow_run` payload), and ``TRIAGE_WORKFLOW``, the triage workflow's file,
empty where triage is not generated.
"""

from __future__ import annotations

import json
import os
import subprocess
import sys
from pathlib import Path
from typing import Any
from urllib.parse import urlencode

LABEL = "tend-ci-failure"
LABEL_DESCRIPTION = "A watched CI workflow is failing; Tend tracks its causes here"
LABEL_COLOR = "b60205"
REPORTER = "github-actions[bot]"
# The verdicts that say something about the lane. The generated job's `if:`
# admits only these, so a cancelled run never takes the lane's pending slot
# from a verdict; everything but `success` reaching here is a failure.
VERDICTS = ("success", "failure", "timed_out", "startup_failure")
# How far back the newer-run check looks. A lane finishing more runs than this
# while this one waited for its job is a lane whose newest run reports itself.
RECENT_RUNS = 30


def api(path: str, *, method: str = "GET", body: dict | None = None) -> Any:
"""One `gh api` call; a failure raises, and fails the job with gh's error."""
args = ["gh", "api", "--method", method, path]
if body is not None:
args += ["--input", "-"]
# `gh` colorizes a piped response when the job's environment forces color;
# `CLICOLOR_FORCE=0` outranks a forced value where `NO_COLOR` alone loses.
env = {**os.environ, "NO_COLOR": "1", "CLICOLOR_FORCE": "0"}
result = subprocess.run(
args,
input=json.dumps(body) if body is not None else None,
capture_output=True,
text=True,
env=env,
check=False,
)
if result.returncode:
sys.stderr.write(result.stderr)
raise subprocess.CalledProcessError(
result.returncode, args, result.stdout, result.stderr
)
return json.loads(result.stdout) if result.stdout.strip() else None


def paginated(path: str) -> list[dict[str, Any]]:
"""Every item of a listing, page by page."""
items: list[dict[str, Any]] = []
page = 1
while True:
batch = api(f"{path}&page={page}")
items += batch
if len(batch) < 100:
return items
page += 1


def title(run: dict[str, Any]) -> str:
return f"`{run['name']}` failing on `{run['head_branch']}` ({run['event']})"


def run_link(run: dict[str, Any]) -> str:
return f"[run {run['id']}]({run['html_url']}) on `{run['head_sha'][:12]}`"


def superseded(repo: str, run: dict[str, Any]) -> bool:
"""Whether a newer run of the same lane has already finished with a verdict."""
query = urlencode(
{
"branch": run["head_branch"],
"event": run["event"],
"status": "completed",
"per_page": RECENT_RUNS,
}
)
listing = api(f"repos/{repo}/actions/workflows/{run['workflow_id']}/runs?{query}")
return any(
other["id"] > run["id"] and other.get("conclusion") in VERDICTS
for other in listing["workflow_runs"]
)


def open_report(repo: str, run: dict[str, Any]) -> int | None:
"""The lane's open report: the lowest open issue by `github-actions[bot]`
under the label and the lane's title. Any workflow's `GITHUB_TOKEN` writes
as that account, so this matches what no person and not the bot filed."""
issues = paginated(f"repos/{repo}/issues?labels={LABEL}&state=open&per_page=100")
numbers = sorted(
issue["number"]
for issue in issues
if "pull_request" not in issue
and issue["user"]["login"] == REPORTER
and issue["title"] == title(run)
)
return numbers[0] if numbers else None


def comment(repo: str, number: int, body: str) -> None:
api(f"repos/{repo}/issues/{number}/comments", method="POST", body={"body": body})


def file_report(repo: str, run: dict[str, Any]) -> int:
# Fails once the label exists, which is every run after the first.
try:
api(
f"repos/{repo}/labels",
method="POST",
body={
"name": LABEL,
"description": LABEL_DESCRIPTION,
"color": LABEL_COLOR,
},
)
except subprocess.CalledProcessError:
pass
body = (
f"`{run['name']}` failed on `{run['head_branch']}`: {run_link(run)}, "
f"triggered by `{run['event']}`.\n\n"
f"This issue tracks `{run['name']}`'s `{run['event']}` runs on "
f"`{run['head_branch']}` until one passes. Each later failed run adds a "
"comment, and the next passing run closes this issue. Tend triages each "
"failure, filing every distinct cause as a sub-issue and working it "
"there.\n"
)
issue = api(
f"repos/{repo}/issues",
method="POST",
body={"title": title(run), "body": body, "labels": [LABEL]},
)
return issue["number"]


def dispatch_triage(repo: str, run: dict[str, Any], number: int, ref: str) -> None:
"""Start the report's triage; a failure leaves it to the notification."""
workflow = os.environ.get("TRIAGE_WORKFLOW", "")
if not workflow:
return
try:
api(
f"repos/{repo}/actions/workflows/{workflow}/dispatches",
method="POST",
body={
"ref": ref,
"inputs": {"issue": str(number), "title": title(run), "report": "true"},
},
)
except subprocess.CalledProcessError:
print(
f"::warning::could not dispatch triage for #{number}; the "
"notifications poll picks it up"
)


def main() -> int:
repo = os.environ["GITHUB_REPOSITORY"]
event = json.loads(Path(os.environ["GITHUB_EVENT_PATH"]).read_text())
run = event["workflow_run"]
if superseded(repo, run):
print(f"A newer run of this lane has finished; run {run['id']} changes nothing")
return 0
number = open_report(repo, run)
if run["conclusion"] != "success":
if number is None:
number = file_report(repo, run)
else:
comment(repo, number, f"Failed again: {run_link(run)}.")
dispatch_triage(repo, run, number, event["repository"]["default_branch"])
elif number is not None:
comment(repo, number, f"{run_link(run)} passed. Closing.")
api(
f"repos/{repo}/issues/{number}",
method="PATCH",
body={"state": "closed", "state_reason": "completed"},
)
return 0


if __name__ == "__main__":
raise SystemExit(main())
TEND_PY
64 changes: 0 additions & 64 deletions .github/workflows/tend-ci-fix.yaml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/tend-mention-relay.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Generated by tend 0.3.14. Regenerate with: uvx tend@latest init
# Generated by tend 0.3.16. Regenerate with: uvx tend@latest init
#
# Do not edit this file directly — it will be overwritten on regeneration.
# To change this workflow, edit .config/tend.yaml. To change what the agent
Expand Down
Loading
Loading