Add Project Requirements PR gate (targeted review as a required status check) by devin-ai-integration[bot] · Pull Request #44 · NASA-AMMOS/tig

devin-ai-integration · 2026-10-07T17:16:03Z

Summary

Adds a dedicated PR gate for TIG's three project requirements: documentation parity, broad deployability, and test coverage. It does not depend on Devin Review's generic flags. The result is a Project Requirements commit status that branch protection can require.

Criteria moved out of REVIEW.md. The criteria from devin/1790713887-review-md now live in .github/project-requirements/criteria.md. Devin Review automatically picks up any **/REVIEW.md, so the file is renamed as well as moved; otherwise the general review would keep applying it. The scope section now reads "evaluate only these three requirements" instead of "perform the full default review first", which keeps the gate focused. The detailed rules for requirements 1–3 are unchanged.

Flow (.github/workflows/project-requirements.yml → .github/scripts/project_requirements_gate.py, standard library only):

pull_request_target [opened, synchronize, reopened, ready_for_review, labeled/unlabeled(waiver label only)]
  checkout BASE branch (never the PR head)
  if waiver label added by user with write/maintain/admin -> status success "Waived by @user"; stop
  if draft -> skip
  status pending
  hints = precheck_hints(changed_files)   # e.g. tig-cli/src changed, tig-cli/tests did not
  POST {DEVIN_API_URL}/v3/organizations/{DEVIN_ORG_ID}/sessions
       prompt = criteria.md + PR ref + hints, structured_output_schema = OUTPUT_SCHEMA (required)
  poll GET .../sessions/{id} until finished/exit (timeout 40 min)
  upsert one sticky PR comment (marker <!-- tig-project-requirements-gate -->)
  status = failure if any verdict == "fail" else success   # "concern" does not block
  any API/timeout/malformed-output problem -> status error + error comment

Design notes:

pull_request_target gives fork PRs access to DEVIN_API_KEY. The job checks out only the base branch, so the script and criteria can't be changed by the PR under review. One consequence is that this gate does not run on this PR itself; it applies to PRs opened after merge.
The script's unit tests run in a separate pull_request workflow (project-requirements-tests.yml, Python 3.9). That workflow installs pytest from the hash-pinned, wheel-only .github/scripts/requirements-test.txt.
The prompt tells the session the task is read-only (no pushes, comments, or labels) and to treat PR content as untrusted.
Concurrency is per PR with cancel-in-progress, set at job level so unrelated label events don't cancel a running review. A cancelled run's Devin session is not stopped; PROJECT_REQUIREMENTS_MAX_ACU caps the cost of each session.

Setup before requiring the check (also in docs/reference/project-requirements-gate.md):

Secret DEVIN_API_KEY (service user that can create sessions) and variable DEVIN_ORG_ID.
Optional variables: DEVIN_API_URL for dedicated deployments, PROJECT_REQUIREMENTS_MAX_ACU, and PROJECT_REQUIREMENTS_WAIVER_LABEL (default requirements-waived).
Create the requirements-waived label, then require the Project Requirements status on develop and master.

Testing: 25 unit tests pass on Python 3.9 and 3.12; ruff and actionlint are clean, and the SonarCloud quality gate passes with no open issues. The gate has not yet run against a live PR or the Devin API.

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/4be0d92fba4f4ec3b8e00342c356f8aa
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/4be0d92fba4f4ec3b8e00342c356f8aa?variant=devin
Requested by: @jl-0

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-07T17:16:07Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration

Devin Review found 4 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-10-07T17:20:40Z

+    pr = github.pull_request()
+    sha = pr["head"]["sha"]


🔴 Failed rerun retains passing status

When pull_request() fails on a rerun, run exits before replacing the previous status. The old success remains on the commit without a current verdict.

Learn more

The gate writes a commit status for the PR head on every successful evaluation. A rerun against the same commit can fail fetching the PR before its pending status and guarded evaluation start. The previous success remains the latest status, while the workflow reports failure separately. Other calls before the guarded block, including waiver checks, have the same failure mode.

Example: A commit previously received Project Requirements: success. A manual rerun encounters a transient GitHub API failure fetching that PR. The workflow fails, but the latest commit status remains success rather than error.

Recommended fix: Move initial GitHub calls into an error-handled path and use the event's PR-head SHA or another reliable source to publish an error status when fetching the PR fails. Include waiver checks in the guarded path.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-07T17:20:41Z

+            github.set_status(sha, "success", f"Waived by @{actor}", config.run_url)
+            print(f"Requirements waived by @{actor} via the {config.waiver_label} label.")
+            return 0


🟡 Waived gate retains failing comment

After a maintainer waives a failed gate, run sets success but leaves the sticky comment showing FAIL. Reviewers see conflicting verdicts for the same commit.

Learn more

The gate writes verdicts to a commit status and one sticky PR comment. When a failing PR receives an authorized waiver label, this branch updates only the status and returns. The previous comment still reports failure for the same commit.

Example: A PR has a FAIL comment for missing documentation. A maintainer adds requirements-waived. The commit status says Waived by @maintainer, while the gate comment still says Project Requirements: FAIL.

Recommended fix: Update the sticky comment on the waiver path to show the waiver and actor, or clearly mark the previous findings as waived.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-07T17:20:43Z

create_session

+    for attempt in range(1, attempts + 1):
+        request = urllib.request.Request(url, data=data, headers=headers, method=method)
+        result = _send(request, final=attempt == attempts)


🟡 Lost creation response starts duplicate sessions

If Devin creates a session but its response is lost, http_json retries the POST. Each retry can start another session that the gate never tracks or stops.

Learn more

The generic HTTP helper retries POST requests after network or server errors. Session creation uses it through create_session. Once Devin accepts creation, a lost response is indistinguishable from failed creation. Retrying can create multiple sessions, while the gate only polls the ID from one response.

Example: Devin creates session A, but the connection resets before the response reaches the workflow. The retry creates session B, which the gate polls. Session A continues untracked.

Recommended fix: Use an API-supported idempotency key for session creation, if available. Otherwise avoid automatic retries of the creation POST after ambiguous failures.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-10-07T17:20:44Z

+        for event in events:
+            if event.get("event") == "labeled" and event.get("label", {}).get("name") == label:
+                actor = (event.get("actor") or {}).get("login")
+        return actor


🟥 Reapplied waiver inherits old maintainer authorization

After a maintainer removes a waiver, last_labeler retains their earlier label event. A later relabel by another actor can receive the maintainer's authorization and skip the gate.

Was this helpful? React with 👍 or 👎 to provide feedback.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

sonarqubecloud · 2026-10-07T17:22:03Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Add Project Requirements PR gate with criteria moved out of REVIEW.md

9ecc66a

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot assigned jl-0 Oct 7, 2026

Address SonarCloud findings: split http_json, hash-pinned test deps

aaa61c8

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot commented Oct 7, 2026

View reviewed changes

Build per-requirement schema without shared mutable value

80fc839

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot mentioned this pull request Oct 7, 2026

Add REVIEW.md with project review guidelines #43

Closed

jl-0 merged commit 7b3f50b into develop Oct 7, 2026
4 checks passed

Navigation Menu

Repository navigation

Add Project Requirements PR gate (targeted review as a required status check) - #44

Add Project Requirements PR gate (targeted review as a required status check)#44
jl-0 merged 3 commits into
developfrom
devin/1791391658-project-requirements-gate

devin-ai-integration Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Oct 7, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

Uh oh!

sonarqubecloud Bot commented Oct 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

devin-ai-integration Bot commented Oct 7, 2026

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Oct 7, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Oct 7, 2026 •

edited

Loading

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading

devin-ai-integration Bot Oct 7, 2026 •

edited

Loading