diff --git a/CHANGELOG.md b/CHANGELOG.md index 9955e065..448e1f9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,43 +4,60 @@ All notable changes to ThrillhouseBot. ## [Unreleased] +## [0.6.0] — 2026-08-13 + +On-demand commands for improving a PR and generating tests, per-repository review +configuration, and cost controls that bound what one review may spend. Review +accuracy work concentrates on the verifier and on saying plainly when a file was +not read. + ### Added -- **Per-repo ignore patterns** (#51): a repository can now declare ignore globs of its own in an optional `.github/thrillhousebot.yml`, under `review.ignored-files`. They are **additive** — the effective set is the union of the deployment-wide `thrillhousebot.review.ignored-files` list and whatever the repository declared, so a repo can take more files out of review scope (generated dirs, vendored code, fixtures) but can never put back a file the deployment excludes. One install can therefore serve repositories with different layouts without everyone sharing a single global list. The file is read from the default branch and cached per repository for five minutes, alongside the existing instructions lookup, and fails soft in every direction: a missing file, invalid YAML, an unexpected shape, or an uncompilable glob is logged and skipped, leaving the global list in force. Operators who do not want repositories adjusting their own review scope can disable the mechanism with `THRILLHOUSEBOT_REVIEW_REPO_CONFIG_ENABLED=false` -- **`/improve` — whole-PR improvement pass** (#452): a new on-demand command that reads the whole change set under the token budget and posts committable suggestions for the improvements a review would otherwise only describe. Enabled by default; disable with `REVIEW_IMPROVE_ENABLED=false` -- **`/generate-tests` — unit tests for changed code** (#461): a new on-demand command that proposes tests for the code the PR changes. The generator is handed the review's own findings (#616) so it tests intended behavior instead of canonizing the defects the review just reported, and the emitted code is held to compiling, runnable shape (#590). Enabled by default; disable with `REVIEW_GENERATE_TESTS_ENABLED=false` -- **Per-review token spend ceiling** (#509): `REVIEW_MAX_TOKENS_PER_REVIEW` bounds the tokens one review may consume across every AI call it makes — actual input+output as the provider reports them, counting retries, the finding verifier (#521) and the final summary call. Once reached, remaining batches are disclosed as not reviewed and the summary degrades to counts rather than making further calls (#524, #530). `0` (the default) keeps the previous unbounded behavior -- **A `concise` named model for the fixed-shape calls** (#507): the final summary, the finding verifier and maintainer replies now run on their own model binding with its own response cap, `REVIEW_CONCISE_MAX_OUTPUT_TOKENS` (default `8192`), so they never share a cap sized for batch review output. Its reasoning effort is its own too — `AI_REASONING_EFFORT_CONCISE` (#574) — because reasoning tokens count against that cap -- **Models may declare an output budget separate from their input window** (#496): the new `separate-output-budget` per-model flag says whether the provider bills completion against the same window as the prompt, which is what the input-budget arithmetic needs to be right on shared-window providers. Ships with `xhigh`/`max` reasoning tiers and corrected `deepseek-v4-flash` defaults (#490, #619) -- **Patch coverage in the review context** (#467): when the PR's CI publishes a coverage report, the changed lines it does not cover are fed to the review so untested new code can be named as such. Enabled with `REVIEW_PATCH_COVERAGE_ENABLED` -- **Path-scoped review instructions** (#460): `.github/thrillhousebot.yml` can attach extra review guidance to path globs, so a repository can hold its API surface, its migrations and its test tree to different rules without one global instruction blob -- **Follow-up delta comment** (#459): a follow-up review can post a short "what changed since the last review" comment — new, resolved, and still-open findings — instead of leaving a maintainer to diff two summaries by eye. Enabled with `REVIEW_FOLLOW_UP_SUMMARY_ENABLED` -- **Large-PR nudge** (#466): a large PR whose review produced no inline finding now says so explicitly, rather than reading as a silent clean bill of health for a change nobody could have read in one sitting -- **Config-key documentation checks** (#109, #450, #465): the review pulls a documented config key's real definition out of the repository and reports documentation that is correct but incomplete — a missing default, an unstated unit, an undocumented sibling key -- **Data-structure tracing from producer to consumer** (#117): a change to a shared record or payload shape is followed to the code that reads it, so a field renamed or narrowed at the source is reported against the consumer that still expects the old shape -- **Decline re-check** (#453): when a maintainer declines a finding, the stated reason is re-checked against the reviewed code before the finding is recorded "justified" — it stays open one more round only when the code plainly contradicts the reason. `REVIEW_DECLINE_RECHECK_ENABLED=false` restores the unconditional close -- **Clearing a thread-less finding from the PR conversation** (#553, #563): a finding raised below the inline-posting bar has no review thread to reply on, so commenting `@thrillhousebot resolved path/to/File.java:42 — ` on the PR now closes it. An interrogative (`resolved?`) is deliberately not a directive, and a directive naming no locator is answered as such -- **Pacing for content-creating GitHub calls** (#597): `GITHUB_WRITE_MIN_INTERVAL` (default `1s`) spaces comments, review comments, thread replies and reviews process-wide so a burst never reaches GitHub's secondary rate limit in the first place, with `GITHUB_WRITE_MAX_WAIT` (default `60s`) capping any one caller's wait. A write GitHub does throttle anyway is backed off and logged with what it returned (#577), and a command dropped after that budget is spent now says so on the PR instead of vanishing (#541, #598) +- **`/improve`** (#452): reads the whole change set under the token budget and posts committable suggestions for improvements a review would otherwise only describe. On by default; disable with `REVIEW_IMPROVE_ENABLED=false` +- **`/generate-tests`** (#461): proposes tests for the code a PR changes. On by default; disable with `REVIEW_GENERATE_TESTS_ENABLED=false` +- **Per-repository ignore patterns** (#51): a repository can declare its own ignore globs in `.github/thrillhousebot.yml` under `review.ignored-files`. The effective set is the union of the deployment list and the repository's, so a repository can remove files from review scope but cannot restore one the deployment excludes. The file is read from the default branch and cached for five minutes. A missing file, invalid YAML, an unexpected shape or an uncompilable glob is logged and skipped, leaving the deployment list in force. Set `THRILLHOUSEBOT_REVIEW_REPO_CONFIG_ENABLED=false` to disable +- **Path-scoped review instructions** (#460): `.github/thrillhousebot.yml` can attach review guidance to path globs, so an API surface, a migration directory and a test tree can be held to different rules +- **Per-review token spend ceiling** (#509): `REVIEW_MAX_TOKENS_PER_REVIEW` bounds the tokens one review may consume across every AI call it makes, counting retries, the finding verifier and the summary call. Once reached, remaining batches are disclosed as not reviewed and the summary degrades to counts. `0`, the default, keeps the previous unbounded behaviour +- **A `concise` model binding for fixed-shape calls** (#507): the summary, the finding verifier and maintainer replies run on their own model with their own response cap, `REVIEW_CONCISE_MAX_OUTPUT_TOKENS` (default `8192`), so they no longer share a cap sized for batch review output +- **`separate-output-budget` per-model flag** (#496): declares whether a provider bills completion against the same window as the prompt, which the input-budget arithmetic needs on shared-window providers +- **Patch coverage in the review context** (#467): when a PR's CI publishes a coverage report, the changed lines it does not cover are given to the review, so untested new code can be named as such. Off by default; enable with `REVIEW_PATCH_COVERAGE_ENABLED=true` +- **Config-key documentation checks** (#109, #450, #465): the review reads a documented config key's real definition out of the repository and reports documentation that is incomplete, such as a missing default, an unstated unit, or an undocumented sibling key +- **Producer-to-consumer tracing** (#117): a change to a shared record or payload shape is followed to the code that reads it, so a field renamed or narrowed at the source is reported against the consumer that still expects the old shape +- **Decline re-check** (#453): when a maintainer declines a finding, the stated reason is checked against the reviewed code before the finding is recorded as justified. It stays open one more round only when the code contradicts the reason. `REVIEW_DECLINE_RECHECK_ENABLED=false` restores the unconditional close +- **Clearing a finding that has no review thread** (#553): a finding raised below the inline-posting bar has no thread to reply on, so commenting `@thrillhousebot resolved path/to/File.java:42 — <title>` on the PR closes it. A question (`resolved?`) is not treated as a directive, and a directive naming no location is answered as such +- **Follow-up delta comment** (#459): a follow-up review can post a short summary of what changed since the last one, listing new, resolved and still-open findings. Off by default; enable with `REVIEW_FOLLOW_UP_SUMMARY_ENABLED=true` +- **Salvage of a length-capped response** (#511): findings completed before a response hit the model's length cap are kept. Previously the whole response was discarded +- **Large-PR notice** (#466): a large PR whose review produced no inline finding now says so explicitly. Off by default; enable with `REVIEW_LARGE_PR_NUDGE_ENABLED=true`, with `REVIEW_LARGE_PR_NUDGE_MIN_FILES` and `REVIEW_LARGE_PR_NUDGE_MIN_CHANGED_LINES` setting what counts as large +- **Write pacing for GitHub calls** (#597): `GITHUB_WRITE_MIN_INTERVAL` (default `1s`) spaces comments, review comments, thread replies and reviews across the process so a burst does not reach GitHub's secondary rate limit. `GITHUB_WRITE_MAX_WAIT` (default `60s`) caps how long one caller waits ### Changed -- **`/describe`, `/changelog`, `/add-docs` and `/generate-tests` are planned as token-budgeted batches** (#463, #469): the on-demand generators no longer render one call capped by `REVIEW_MAX_DIFF_LINES`, so a large PR is covered whole instead of being reduced to its first N diff lines. `REVIEW_MAX_DIFF_LINES` now applies only to the remaining single-call renders (replies, base comparison, budgeting-disabled review) -- **The PR summary is grounded in the whole change set** (#335): the summary call gets the authoritative file and line totals and a directory breakdown, so a change spread across several packages can no longer be described as a single-file edit when the input budget clamps the file list -- **Security severity follows the defect class, not the reviewer's certainty** (#575, #594): an unmitigated injection sink is floored at high risk instead of drifting down when the surrounding context is only partly visible, and the floor's defeaters are scoped to clauses the material actually asserts (#610) -- **Chat memory is disabled on every AI service** (#585): each call is stateless by construction, so no round can inherit context from an unrelated one +- **`/describe`, `/changelog` and `/add-docs` are planned as token-budgeted batches** (#463, #469): these commands no longer render one call capped by `REVIEW_MAX_DIFF_LINES`, so a large PR is covered whole. `REVIEW_MAX_DIFF_LINES` now applies only to the remaining single-call renders, which are replies, base comparison, and review with budgeting disabled. The commands added in this release, `/improve` and `/generate-tests`, are planned the same way from the start +- **The summary is grounded in the whole change set** (#335): the summary call is given the authoritative file and line totals and a directory breakdown, so a change spread across several packages is no longer described as a single-file edit when the input budget clamps the file list +- **Security severity follows the defect class** (#575): an unmitigated injection sink is held at high risk even when the surrounding context is only partly visible +- **Chat memory is disabled on every AI service** (#585): each call is stateless, so no round inherits context from an unrelated one ### Fixed -- **A response cut at the model's length cap no longer discards the work already paid for** (#495, #504, #511, #582, #592, #617): findings complete before the cut are salvaged on both the multi-call and single-call lanes, the verifier keeps the verdicts its cut body already carried, a length stop is no longer retried at full price, and the truncation's remedy names the cap that actually applies to the call it came from (#595, #542) -- **Coverage gaps hold approval instead of passing silently** (#489): a file GitHub reported with changes but no patch text, and a file whose batch call failed, are disclosed by name and withhold APPROVE rather than counting as reviewed. An unnamed file keeps its slot in that accounting instead of crashing the plan or disappearing from it (#472, #551, #621) -- **Outgoing content stays inside GitHub's limits** (#487, #503): comment bodies are capped at 65,536 characters and check-run output fields at their own limits, so an oversized review is trimmed rather than rejected -- **The heuristic failure-mode review dimension runs again under token budgeting** (#618): it was gated on a diff string that budgeting leaves empty, so the whole dimension was silently inert in the default configuration -- **The previous-findings context is bounded to a share of the input budget** (#615): it grew without limit across rounds and could eventually crowd out the diff -- **Verifier precision** (#611, #612, #613, #614): a demonstrated sink and standard-library semantics are carved out of the downgrade pressure, an unmatched-quote finding is kept rather than deleted, a second defect in the same hunk is filed as its own finding, and every claim a withheld path licenses is refused, not just the first -- **The walkthrough and summary surfaces stop contradicting each other** (#522, #523, #544, #549, #596): file summaries are filled on large PRs, a row with no summary says why, ceiling-skipped files are dropped from the walkthrough rows, a summary-only cut no longer claims partial file coverage, and one observation is published on one surface instead of up to three -- **`quarkus:dev` boots usable on a fresh clone** (#591, #620): the dev-mode schema is created at startup and the README states the `.env` and PEM steps a fresh clone needs -- **An empty SSE frame is skipped rather than failing the stream on the event loop** (#555) -- **Build and image fixes**: the native binary is no longer stored twice and the website is out of the build context (#561), and the H2 driver is kept out of the runtime image (#564) +- **Oversized comments were rejected by GitHub** (#487, #503): comment bodies are capped at 65,536 characters and check-run output fields at their own limits, so an oversized review is trimmed and still posts +- **A file GitHub reported without patch text could crash a review** (#472, #551): null filenames reached null-hostile immutable collections. Such files are now counted and disclosed by name, and they withhold approval rather than passing as reviewed (#489) +- **A response cut at the length cap was retried at full price** (#495, #504): a length stop is detected on the blocking command paths and not retried +- **The finding verifier deleted correct findings** (#611, #612, #613, #614): all claims a withheld path licenses are refused, where previously only the first was, a second defect in the same hunk is filed as its own finding, an unmatched-quote finding is kept, and demonstrated sinks and standard-library semantics are carved out of the downgrade pressure +- **The previous-findings context grew without bound across rounds** (#615) and could crowd out the diff. It is now bounded to a share of the input budget +- **Walkthrough rows were missing their file summaries on large PRs** (#544), and a row without a summary now says why (#549) +- **One observation could be published on up to three surfaces of the same comment** (#596) +- **The heuristic failure-mode review dimension was inert** (#618): it was gated on a diff string that token budgeting leaves empty +- **`quarkus:dev` did not boot usable on a fresh clone** (#591, #620): the dev-mode schema is created at startup, and the README states the `.env` and PEM steps a fresh clone needs +- **An empty SSE frame failed the stream on the event loop** (#555). It is now skipped +- **Runtime image size and contents** (#561, #564): the native binary is no longer stored twice, the website is out of the build context, and the H2 driver is kept out of the runtime image + +### Dependencies + +- Bumped the Quarkus platform from 3.37.4 to 3.38.0 and `quarkus-langchain4j` from 1.12.0 to 1.12.2 +- Declared `jackson-dataformat-yaml` explicitly. It was already on the classpath transitively and is version-managed by the Jackson BOM; reading a repository's own `.github/thrillhousebot.yml` uses it directly +- Bumped the frontend `next` to 16.2.12 and the dev-only `jsdom` to 30.0.1, with `@types/node` on a patch release +- Bumped GitHub Actions `github/codeql-action` to v4.37.6, `actions/setup-java` to v5.7.0 and `docker/login-action` to v4.6.0 +- Bumped the Spotless Maven plugin to 3.9.0 ## [0.5.0] — 2026-07-26 diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 93564587..b46d15bf 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -205,9 +205,9 @@ sequenceDiagram | Package | Responsibility | Notable classes | |---|---|---| | `webhook/` | Receives GitHub events, verifies the HMAC signature, decides whether an event triggers a review (trigger filters, per-PR pause state, auto-review rate limit), acks slash/mention commands with 👀, runs the comment commands (`/help`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/improve`, `/generate-tests`, `/resolve`, `/pause`, `/resume`), and schedules finding-feedback capture on review-thread replies | `WebhookController`, `WebhookVerifier`, `TriggerDetector`, `ReviewTriggerFilter`, `AckReactionService`, `CommentCommandService`, `PrPauseService` | -| `review/` | Orchestrates a review: plans the token budget, calls the AI layer (single-call or map-reduce), maps findings to a risk level and review state, writes the summary comment, optionally labels the PR, answers maintainer replies/mentions in PR threads, and persists maintainer finding feedback (👍/👎 / reply heuristics) for a future learnings pipeline | `ReviewOrchestrator`, `ReviewDispatcher`, `DiffBudgetPlanner`, `FindingPipeline`, `AutoReviewRateLimiter`, `ReviewDiffFormatter`, `FollowUpAnalyzer`, `FindingFeedbackCaptureService`, `FindingFeedbackService`, `PrSummaryGenerator`, `PrLabeler`, `MaintainerReplyService`, `MaintainerReplyDispatcher` | -| `review/ai/` | The LangChain4j layer: streams or batches model responses, parses findings, runs a second pass to verify them, applies generation/reasoning customizers, and writes conversational replies | `PrReviewer`, `AiReviewService`, `ChatModelCustomizers`, `FindingVerifier`, `FindingVerificationService`, `ReviewResponseParser`, `ReplyAssistant` | -| `github/` | Talks to the GitHub REST and GraphQL APIs: app auth, pull requests, reviews, check runs, comments, labels, reactions (create + list), and reading the repo instructions file | `GitHubAuthClient`, `GitHubReviewClient`, `GitHubCheckRunClient`, `GitHubLabelClient`, `GitHubReactionClient`, `InstructionsResolver` | +| `review/` | Orchestrates a review: plans the token budget and the per-review spend ceiling, calls the AI layer (single-call or map-reduce), maps findings to a risk level and review state, re-checks a maintainer's decline against the reviewed code, writes the summary comment, optionally labels the PR, answers maintainer replies/mentions in PR threads, and persists maintainer finding feedback (👍/👎 / reply heuristics) for a future learnings pipeline | `ReviewOrchestrator`, `ReviewDispatcher`, `DiffBudgetPlanner`, `FindingPipeline`, `AutoReviewRateLimiter`, `ReviewDiffFormatter`, `FollowUpAnalyzer`, `FindingFeedbackCaptureService`, `FindingFeedbackService`, `PrSummaryGenerator`, `PrLabeler`, `MaintainerReplyService`, `MaintainerReplyDispatcher`, `PrImprovementService`, `PatchCoverage`, `ConfigKeyContextResolver`, `RebuttalContradiction`, `SummarySurfaceDeduplicator`, `VerdictBuilder` | +| `review/ai/` | The LangChain4j layer: streams or batches model responses, parses findings, runs a second pass to verify them, applies generation/reasoning customizers, and writes conversational replies | `PrReviewer`, `AiReviewService`, `ChatModelCustomizers`, `FindingVerifier`, `FindingVerificationService`, `ReviewResponseParser`, `ReplyAssistant`, `TruncatedResponseSalvager`, `FindingVerifierPrompts` | +| `github/` | Talks to the GitHub REST and GraphQL APIs: app auth, pull requests, reviews, check runs, comments, labels, reactions (create + list), and reading the repo instructions file | `GitHubAuthClient`, `GitHubReviewClient`, `GitHubCheckRunClient`, `GitHubLabelClient`, `GitHubReactionClient`, `InstructionsResolver`, `GitHubWriteRetry` | | `dashboard/` | The live UI backend: OAuth login (in-memory sessions), WebSocket broadcaster (`review.stream` / `review.batch`), review session persistence, and finding-feedback aggregates | `AuthResource`, `DashboardSessionStore`, `SessionEventBroadcaster`, `ReviewSessionRepository`, `DashboardResource` | | `config/` | Wiring: the outbound HTTP client, the review thread pool, typed config, active-model settings (caps, generation params), fail-fast startup validation, and the shared bot-identity used to recognize the bot's own activity | `HttpClientProducer`, `ReviewExecutorProducer`, `ThrillhouseConfig`, `ActiveModelSettings`, `StartupConfigValidator`, `BotIdentity` | | `frontend/` | The Next.js dashboard, built to a static export and served by Quarkus | — | @@ -232,6 +232,39 @@ dashboard; they emit `review.batch` progress events instead. Batches run concurrently on virtual threads; a failed batch is retried once after the parallel pass completes. +**Cost ceiling** — `REVIEW_MAX_TOKENS_PER_REVIEW` bounds the tokens one review may +spend across every call it makes, counting retries, the verifier and the summary. +Once reached, remaining batches are disclosed as not reviewed by name and the +summary degrades to counts rather than making further calls. `0`, the default, +leaves it unbounded. The summary, the verifier and maintainer replies run on a +separate `concise` model binding with its own response cap +(`REVIEW_CONCISE_MAX_OUTPUT_TOKENS`) and its own reasoning effort, so they never +share a cap sized for batch review output. + +**Coverage honesty** — a file the review never read does not pass silently. A +file GitHub reported with changes but no patch text, a file that did not fit any +batch, and a file whose batch call failed are each disclosed by name and withhold +APPROVE. A response the model cut at its length cap keeps the findings that +completed before the cut rather than being discarded whole. + +**Patch coverage as review context** — when a PR's CI publishes a coverage +report, `PatchCoverage` reads the changed lines it does not cover and gives them +to the review, so new code with no test behind it can be named as such. Off +unless `REVIEW_PATCH_COVERAGE_ENABLED` is set. + +**Repository-supplied configuration** — `.github/thrillhousebot.yml` carries a +repository's own ignore globs and path-scoped review instructions, read from the +default branch and cached for five minutes. Ignore globs are additive to the +deployment list; a repository can narrow its own review scope but cannot restore +a file the deployment excludes. Every failure mode (missing file, invalid YAML, +unexpected shape, uncompilable glob) is logged and skipped, leaving the +deployment configuration in force. + +**Write pacing** — content-creating GitHub calls are spaced process-wide by +`GITHUB_WRITE_MIN_INTERVAL` so a burst of comments never reaches the secondary +rate limit in the first place, with `GITHUB_WRITE_MAX_WAIT` capping how long any +one caller waits. + Each AI call is bounded by `AI_TIMEOUT` (LangChain4j) and `thrillhousebot.review.ai-timeout-seconds`. Cost and token metrics come from OpenTelemetry. OAuth login sessions are opaque IDs in cookies with tokens kept diff --git a/pom.xml b/pom.xml index 8ddcb781..6af70a7c 100644 --- a/pom.xml +++ b/pom.xml @@ -8,7 +8,7 @@ <groupId>dev.thiagogonzaga.thrillhousebot</groupId> <artifactId>thrillhousebot</artifactId> - <version>0.5.1-SNAPSHOT</version> + <version>0.6.0</version> <properties> <compiler-plugin.version>3.15.0</compiler-plugin.version> diff --git a/src/main/resources/application.properties b/src/main/resources/application.properties index 2860baaa..42e722cc 100644 --- a/src/main/resources/application.properties +++ b/src/main/resources/application.properties @@ -218,8 +218,9 @@ thrillhousebot.review.token-safety-margin=${REVIEW_TOKEN_SAFETY_MARGIN:0.9} # max-ai-calls only counts planned calls. Once reached, remaining batches are disclosed as not # reviewed and the summary degrades to counts-only rather than making further calls. 0 = off. thrillhousebot.review.max-tokens-per-review=${REVIEW_MAX_TOKENS_PER_REVIEW:0} -# Line cap on single-call renders (/describe, /changelog, /add-docs, replies, base comparison, -# budgeting-disabled review). Token-budgeted review calls ignore it; 0 = off (unbounded). +# Line cap on single-call renders (replies, base comparison, budgeting-disabled review). +# Token-budgeted reviews and the batched commands (/improve, /describe, /changelog, +# /generate-tests, /add-docs) ignore it; 0 = off (unbounded). thrillhousebot.review.max-diff-lines=${REVIEW_MAX_DIFF_LINES:5000} thrillhousebot.review.instructions-file=.github/thrillhousebot.md # Second-pass AI audit that drops/downgrades unverifiable findings before they are posted diff --git a/website/src/assets/0.5.0/icon.png b/website/src/assets/0.5.0/icon.png new file mode 100644 index 00000000..e523b074 Binary files /dev/null and b/website/src/assets/0.5.0/icon.png differ diff --git a/website/src/assets/0.5.0/live-streaming.png b/website/src/assets/0.5.0/live-streaming.png new file mode 100644 index 00000000..d6728e7d Binary files /dev/null and b/website/src/assets/0.5.0/live-streaming.png differ diff --git a/website/src/assets/0.5.0/pr-approval.png b/website/src/assets/0.5.0/pr-approval.png new file mode 100644 index 00000000..1767740c Binary files /dev/null and b/website/src/assets/0.5.0/pr-approval.png differ diff --git a/website/src/content/docs/0.5.0/architecture.md b/website/src/content/docs/0.5.0/architecture.md new file mode 100644 index 00000000..e02e0970 --- /dev/null +++ b/website/src/content/docs/0.5.0/architecture.md @@ -0,0 +1,263 @@ +--- +slug: 0.5.0/architecture +title: Architecture +description: One-page overview of how the bot is structured and how a review flows through it. +--- + + + + +One-page overview of how the bot is structured and how a review flows through it. + +ThrillhouseBot is a Quarkus application that runs as a GitHub App. A webhook +arrives when a pull request changes, the bot builds a review with an +OpenAI-compatible model, and it posts the result back as a PR review plus a +check run. A dashboard streams what is happening live. + +## Components + +```mermaid +flowchart TB + subgraph GH[GitHub.com] + IN[PR push, /review, @mention] + OUT[PR reviews · check runs · comments] + end + + subgraph BOT[ThrillhouseBot · Quarkus] + WH[webhook/<br/>WebhookController] + RO[review/<br/>ReviewOrchestrator] + AI[review/ai/<br/>AiReviewService] + GHC[github/<br/>REST clients] + DB[dashboard/ + frontend/] + + WH --> RO --> AI + RO --> GHC + AI -.->|live tokens / review.batch| DB + end + + IN -->|POST /api/webhook<br/>HMAC-verified| WH + GHC --> OUT +``` + +The `github/` clients wrap the GitHub REST surface the bot uses: installation +tokens, pull diffs and prior reviews, check runs, PR reviews with inline +comments, issue comments, and the instructions-file fallback chain +(`.github/thrillhousebot.md`, `.github/copilot-instructions.md`, `CLAUDE.md`, +`AGENTS.md`, `AGENT.md`). + +## Request flow + +```mermaid +flowchart TD + GH[GitHub: PR opened / synced / comment] -->|webhook| WH[webhook/] + WH -->|verify HMAC, filters, rate limit, 👀 ack| RO[review/ ReviewOrchestrator] + RO -->|fetch diff, instructions, prior findings| GHC[github/ API clients] + RO -->|budget plan · stream or batch| AI[review/ai/ LangChain4j] + AI -->|parse findings| RO + RO -->|verify findings — 2nd AI call per batch, on by default| AI + RO -->|post review + check run| GHC --> GH + AI -.->|live tokens or review.batch| DB[dashboard/ broadcaster] + DB -->|WebSocket| FE[frontend/ Next.js UI] + RO -->|persist session, cost, tokens| PG[(H2 / PostgreSQL)] + AI -->|traces, token & cost metrics| OT[(OpenTelemetry)] +``` + +Automatic triggers (`pull_request` opened / synchronize, and similar) are subject +to `AUTO_REVIEW_MIN_INTERVAL`: if the same PR was auto-reviewed too recently, +the webhook path skips the review silently. Manual `/review` always bypasses that +window. Slash and mention **commands** get a best-effort 👀 reaction before +pause/authorization; conversational `@thrillhousebot` mentions (no command word) +are answered without a reaction. + +## Review lifecycle + +### First review (PR opened) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + participant AI as AI Provider + + Dev->>GH: git push (PR opened) + GH->>TB: POST /api/webhook (pull_request: opened) + + Note over TB: Verify HMAC → JWT → install token + Note over TB: Auto-review rate limit (skip if within AUTO_REVIEW_MIN_INTERVAL) + + TB->>GH: POST check-runs → status: queued + TB->>GH: PATCH check-run → status: in_progress + + par Fetch context + TB->>GH: GET /pulls/{pr}/files (diff) + TB->>GH: GET /compare/{base}...{head} (regression context) + end + + TB->>GH: GET /pulls/{pr}/reviews (check if already reviewed this SHA) + GH-->>TB: no prior reviews → first run + + Note over TB: DiffBudgetPlanner — single-call or token-budgeted batches + + alt Diff fits one call + TB->>AI: POST chat (diff + base comparison + review prompt) — live tokens to dashboard + AI-->>TB: findings + risk levels + suggestions + opt Findings found and REVIEW_VERIFIER_ENABLED (default) + TB->>AI: POST chat (re-check each finding against the diff) + AI-->>TB: confirmed / downgraded / dropped findings + end + else Large diff — multi-call + loop Each batch in parallel (up to REVIEW_MAX_AI_CALLS − 1) + TB-->>TB: review.batch progress (no per-token stream) + TB->>AI: POST chat (batch diff) + AI-->>TB: batch findings + opt Findings and verifier on + TB->>AI: POST chat (verify batch findings) + end + end + TB->>AI: POST chat (summary rollup of aggregated findings) + AI-->>TB: PR-level summary + previous-findings status + Note over TB: Any files that still won't fit are disclosed by name + end + + alt AI fails + TB->>GH: PATCH check-run → conclusion: failure + TB->>GH: POST comment: retry hint (no internal details) + else AI succeeds + issues found + TB->>GH: POST PR review (REQUEST_CHANGES or COMMENT) with inline suggestions + TB->>GH: PATCH check-run → conclusion: failure (critical/high) or neutral + TB->>GH: POST comment: PR summary (risk table + key findings) + else AI succeeds + zero issues + TB->>GH: POST PR review → APPROVE (no body) + TB->>GH: PATCH check-run → conclusion: success + TB->>GH: POST comment: PR summary (celebration inside) + end +``` + +### Follow-up review (new push) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + participant AI as AI Provider + + Dev->>GH: git push (PR synchronize) + GH->>TB: POST /api/webhook (pull_request: synchronize) + + Note over TB: Verify → auth → rate limit → create check run (in_progress) + + par Fetch context + TB->>GH: GET /pulls/{pr}/files (diff) + TB->>GH: GET /compare/{base}...{head} + and Fetch prior review + TB->>GH: GET /pulls/{pr}/reviews (find ThrillhouseBot's last review) + GH-->>TB: previous findings + thread status + end + + Note over TB: Prompt includes diff + prior findings + "check if each was addressed" + Note over TB: Same single-call or map-reduce path as first review + + TB->>AI: POST chat (one or more review calls ± verifier ± summary) + AI-->>TB: resolved / unresolved / new findings + + alt AI fails + Note over TB: Same sanitized error path as first review + else AI succeeds + TB->>GH: POST PR review (suggestions for unresolved + new issues) + TB->>GH: PATCH check-run → conclusion based on risk + Note over TB: No summary comment on follow-up (only on first run) + end +``` + +### Manual trigger (`/review` or `@Thrillhousebot review`) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + + Dev->>GH: Comments "/review" + GH->>TB: POST /api/webhook (issue_comment: created) + + Note over TB: Verify → parse trigger → 👀 ack (bounded wait) → auth + Note over TB: Manual /review bypasses AUTO_REVIEW_MIN_INTERVAL + Note over TB: Fetch diff, compare, and prior reviews + Note over TB: Full re-review even if this SHA was already reviewed +``` + +### Conversational reply (`@thrillhousebot` mention) + +```mermaid +sequenceDiagram + actor Dev + participant GH as GitHub + participant TB as ThrillhouseBot + + Dev->>GH: Mentions @thrillhousebot (in a PR thread or finding reply) + GH->>TB: POST /api/webhook (pull_request_review_comment or issue_comment: created) + + Note over TB: Bot-loop guard → mention detected (no command) → ACK 200 + Note over TB: No 👀 reaction — conversational mentions are answered, not reacted to + Note over TB: Async on review executor: authorize (write access) + Note over TB: Build threaded prompt (finding + diff hunk + thread) + TB->>GH: POST reply in the review thread (or a PR comment) +``` + +## Packages + +| Package | Responsibility | Notable classes | +|---|---|---| +| `webhook/` | Receives GitHub events, verifies the HMAC signature, decides whether an event triggers a review (trigger filters, per-PR pause state, auto-review rate limit), acks slash/mention commands with 👀, runs the comment commands (`/help`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/resolve`, `/pause`, `/resume`), and schedules finding-feedback capture on review-thread replies | `WebhookController`, `WebhookVerifier`, `TriggerDetector`, `ReviewTriggerFilter`, `AckReactionService`, `CommentCommandService`, `PrPauseService` | +| `review/` | Orchestrates a review: plans the token budget, calls the AI layer (single-call or map-reduce), maps findings to a risk level and review state, writes the summary comment, optionally labels the PR, answers maintainer replies/mentions in PR threads, and persists maintainer finding feedback (👍/👎 / reply heuristics) for a future learnings pipeline | `ReviewOrchestrator`, `ReviewDispatcher`, `DiffBudgetPlanner`, `FindingPipeline`, `AutoReviewRateLimiter`, `ReviewDiffFormatter`, `FollowUpAnalyzer`, `FindingFeedbackCaptureService`, `FindingFeedbackService`, `PrSummaryGenerator`, `PrLabeler`, `MaintainerReplyService`, `MaintainerReplyDispatcher` | +| `review/ai/` | The LangChain4j layer: streams or batches model responses, parses findings, runs a second pass to verify them, applies generation/reasoning customizers, and writes conversational replies | `PrReviewer`, `AiReviewService`, `ChatModelCustomizers`, `FindingVerifier`, `FindingVerificationService`, `ReviewResponseParser`, `ReplyAssistant` | +| `github/` | Talks to the GitHub REST and GraphQL APIs: app auth, pull requests, reviews, check runs, comments, labels, reactions (create + list), and reading the repo instructions file | `GitHubAuthClient`, `GitHubReviewClient`, `GitHubCheckRunClient`, `GitHubLabelClient`, `GitHubReactionClient`, `InstructionsResolver` | +| `dashboard/` | The live UI backend: OAuth login (in-memory sessions), WebSocket broadcaster (`review.stream` / `review.batch`), review session persistence, and finding-feedback aggregates | `AuthResource`, `DashboardSessionStore`, `SessionEventBroadcaster`, `ReviewSessionRepository`, `DashboardResource` | +| `config/` | Wiring: the outbound HTTP client, the review thread pool, typed config, active-model settings (caps, generation params), fail-fast startup validation, and the shared bot-identity used to recognize the bot's own activity | `HttpClientProducer`, `ReviewExecutorProducer`, `ThrillhouseConfig`, `ActiveModelSettings`, `StartupConfigValidator`, `BotIdentity` | +| `frontend/` | The Next.js dashboard, built to a static export and served by Quarkus | — | + +## Notes + +PR reviews carry inline comments and suggestions; check runs carry pass/fail +status for branch protection (no inline annotations on the check run itself). + +**AI call budget** — a review that reports findings makes **two** model calls +by default: the review call plus a skeptical verification pass +(`FindingVerifier`) that re-sends the diff and each candidate finding, dropping +or downgrading what it can't confirm. It fails open — a verifier error keeps +the original findings, so a broken verifier can never block a review. Under +token-aware budgeting on large PRs this becomes N batch review calls + N +per-batch verification calls + one summary call. `REVIEW_VERIFIER_ENABLED=false` +skips only the AI pass (a deterministic hedging-language guard still runs) and +trades cost for more false positives. Expect two model spans per flagged +single-call review (or N+N+1 under budgeting) in the traces and in the +dashboard's session totals. Multi-call reviews do not stream tokens to the +dashboard; they emit `review.batch` progress events instead. Batches run +concurrently on virtual threads; a failed batch is retried once after the +parallel pass completes. + +Each AI call is bounded by `AI_TIMEOUT` (LangChain4j) and +`thrillhousebot.review.ai-timeout-seconds`. Cost and token metrics come from +OpenTelemetry. OAuth login sessions are opaque IDs in cookies with tokens kept +server-side; review history persists in the database. Maintainer finding +feedback (reactions and reply heuristics) is documented in +[Finding feedback](https://devops-thiago.github.io/ThrillhouseBot/feedback/) +(source: [docs/FEEDBACK.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/docs/FEEDBACK.md)). See +[SECURITY.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/SECURITY.md) +for the reporting process. + +## Adding an AI provider + +There is no provider-specific code. The model is reached through LangChain4j's +OpenAI-compatible client, so a new provider is configuration: point `AI_BASE_URL` +and `AI_MODEL` at it. Add a `thrillhousebot.ai.pricing.<model>.*` pair for cost +tracking (without it the bot warns once and flags sessions as "no pricing" +instead of `$0`). Optionally set `thrillhousebot.ai.models.<model>.*` for the +model's input cap and generation parameters, and `AI_REASONING_ENABLED` / +`AI_REASONING_EFFORT` when the model supports reasoning. See the +[provider table](https://devops-thiago.github.io/ThrillhouseBot/providers/) and +the [configuration reference](https://devops-thiago.github.io/ThrillhouseBot/configuration/). + + diff --git a/website/src/content/docs/0.5.0/commands.md b/website/src/content/docs/0.5.0/commands.md new file mode 100644 index 00000000..d9148334 --- /dev/null +++ b/website/src/content/docs/0.5.0/commands.md @@ -0,0 +1,45 @@ +--- +slug: 0.5.0/commands +title: Commands +description: Drive the bot directly from a PR with comment commands. +--- + + + +Drive the bot directly from a PR by commenting one of these. Each also accepts the +mention form, e.g. `@Thrillhousebot review`. The bot acknowledges every command +instantly with a 👀 reaction on your comment while the work runs in the background; +a conversational `@thrillhousebot` mention (no command word) gets an answer instead, +not a reaction. + +| Command | What it does | Access | +|---|---|---| +| `/help` | List the available commands | anyone | +| `/review` | Run (or re-run) a full review of the PR | write | +| `/summary` | Post the PR summary if it isn't already on the PR — regenerates it if the comment was deleted, otherwise no-op | write | +| `/describe` | Suggest an improved PR title and description generated from the diff, as a comment to copy in (never overwrites the PR) | write | +| `/changelog` | Draft a CHANGELOG entry for the PR from the diff (Added/Changed/Fixed/Security…), as a comment to copy into `CHANGELOG.md` (never commits) | write | +| `/add-docs` | Generate docstrings/inline docs for the symbols changed in the PR, posted as committable suggestions (or a note with the drafted docs when a multi-line declaration can't be pinned to a single diff hunk) | write | +| `/resolve` | Resolve ThrillhouseBot's outstanding finding threads on the PR | write | +| `/pause` | Silence the bot on the PR | write | +| `/resume` | Re-enable the bot on a paused PR | write | + +**Access** — every command except `/help` requires the commenter to hold write access to +the repository (or to be named in +`THRILLHOUSEBOT_REVIEW_MANUAL_TRIGGER_ALLOWED_LOGINS`), since reviews spend the operator's +AI budget. + +**Pause** — while a PR is paused, ThrillhouseBot skips automatic reviews on new commits, +ignores `/review`, `/summary`, `/describe`, `/changelog`, and `/add-docs`, and does not answer +`@thrillhousebot` mentions (it replies once to say it is paused). `/resume` lifts the pause. +`/help` and `/resolve` keep working while paused. + +**`/add-docs`** — on demand, the bot reads the diff and proposes documentation comments for +the public symbols changed in the PR, honoring the repository instructions and each file's +language. Each suggestion is a committable `suggestion` block placed on the symbol's +declaration (spanning the whole signature when it wraps), so it only inserts docs without +rewriting code. When a multi-line declaration can't be pinned to a single diff hunk, the bot +posts a note with the drafted docs to add manually instead of a committable suggestion. It +spends AI budget per run; operators can turn it off with `REVIEW_ADD_DOCS_ENABLED=false`. + + diff --git a/website/src/content/docs/0.5.0/comparison.md b/website/src/content/docs/0.5.0/comparison.md new file mode 100644 index 00000000..2e139661 --- /dev/null +++ b/website/src/content/docs/0.5.0/comparison.md @@ -0,0 +1,89 @@ +--- +slug: 0.5.0/comparison +title: How it compares +description: An honest look at where ThrillhouseBot sits next to other AI code-review tools. +--- + + + + +A quick, honest look at where ThrillhouseBot sits next to other AI code-review +tools. Facts verified June 2026 from the sources linked below; vendor +capabilities change, so check the source if a detail matters to you. + +The table covers ThrillhouseBot, [CodeRabbit][cr], the open-source +[PR-Agent][pra], [GitHub Copilot code review][cop], the [PRSense][prs] CLI, and +[Kit][kit] (cased-kit). PR-Agent's commercial hosted sibling, Qodo Merge, is +left out because its self-hosting and model details sit behind enterprise sales +and could not be verified from public docs. PRSense and Kit are included because +they overlap on self-hosting and bring-your-own-model workflows; they differ in +how reviews are triggered and how cost visibility is surfaced. + +| | ThrillhouseBot | CodeRabbit | PR-Agent | Copilot review | PRSense | Kit | +|---|---|---|---|---|---|---| +| License | Apache-2.0 | Proprietary | Apache-2.0 [^pra-lic] | Proprietary | Apache-2.0 [^prs-lic] | MIT [^kit-lic] | +| Self-host | Yes | Enterprise only, 500+ seats [^cr-sh] | Yes | Models no; self-hosted runners yes [^cop-run] | Yes (CLI) [^prs] | Yes (CLI / Actions) [^kit] | +| Bring your own model | Any OpenAI-compatible endpoint | Yes: OpenAI, Azure OpenAI, Bedrock, Anthropic [^cr-llm] | Yes, via OpenAI-compatible / LiteLLM | No, GitHub-managed | Yes [^prs] | Yes [^kit] | +| Local models (Ollama) | Yes | No [^cr-llm] | Yes, with config caveats | No | Yes [^prs] | Yes [^kit] | +| Cost / token dashboard | Built-in web UI | No | No | GitHub billing only | Token counts in CLI only [^prs] | Per-review cost in CLI [^kit] | +| Footprint | ~50 MB native binary | SaaS or self-hosted container | Python app / container | SaaS plus Actions minutes [^cop-bill] | Node.js CLI [^prs] | Python app [^kit] | +| Cost model | Free; you pay your own API usage | Freemium | Free | Paid [^cop-bill] | Free; you pay your own API usage | Free; you pay your own API usage | + +## Where ThrillhouseBot fits + +Among the tools in the table above, ThrillhouseBot is the only one that combines +a GitHub App (webhook-driven reviews with inline PR comments), a built-in web +dashboard for cost and token analytics, token-budgeted map-reduce reviews for +large PRs, and a small native footprint, all under an OSI-approved license. If you want reviews to run on your own infrastructure +against a local Ollama model so that no code leaves your network, and you want +ongoing cost visibility in a dashboard rather than per-run CLI output, that is +the niche it targets. + +CodeRabbit and Copilot are more polished hosted products with broader reach. +ThrillhouseBot reviews GitHub pull requests only, with no GitLab or Bitbucket +support, and there is no managed hosting option: you run it yourself. + +PR-Agent, PRSense, and Kit are the closest in spirit for self-hosting with your +own model. PR-Agent is the most established open-source GitHub review bot in this +set. PRSense emphasizes grounded findings tied to the diff and ships as a CLI. +Kit adds rich repository context, local diff review, and per-run cost output in +the terminal or CI logs. None of those three ship a GitHub App plus web +dashboard in one package the way ThrillhouseBot does. + +Pick CodeRabbit or Copilot if you want a hosted product and don't need to bring +your own model. Pick PR-Agent, PRSense, or Kit if you prefer CLI or workflow +integration. Pick ThrillhouseBot if you want a GitHub App with automatic reviews, +a live dashboard, and a small native binary. + +[cr]: https://www.coderabbit.ai/ +[pra]: https://github.com/qodo-ai/pr-agent +[cop]: https://docs.github.com/en/copilot/concepts/agents/code-review +[prs]: https://prsense.org/ +[kit]: https://kit.cased.com/pr-reviewer/ + +[^pra-lic]: PR-Agent was moved to a community-owned GitHub organization and + relicensed under Apache-2.0 in April 2026. +[^cr-sh]: CodeRabbit docs: "The self-hosted option is only available for + CodeRabbit Enterprise customers with 500 user seats or more." + <https://docs.coderabbit.ai/self-hosted/github> +[^cr-llm]: Self-hosted CodeRabbit requires your own LLM credentials for OpenAI, + Azure OpenAI, AWS Bedrock, or Anthropic; its docs do not list local models + or Ollama. <https://docs.coderabbit.ai/self-hosted/github> +[^cop-run]: Copilot code review uses GitHub-managed models; you cannot supply + your own. It can run on self-hosted or larger GitHub-hosted runners. + <https://docs.github.com/en/copilot/concepts/agents/code-review> +[^cop-bill]: From June 1, 2026, Copilot code review is billed as AI credits for + token use plus GitHub Actions minutes for the review infrastructure. + <https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/> +[^prs]: PRSense ships as the `@prsense/cli` npm package (Apache-2.0). Reviews + run locally or in CI; output includes token counts but there is no web + dashboard. Supports Ollama and cloud providers. + <https://prsense.org/> · <https://www.npmjs.com/package/@prsense/cli> +[^prs-lic]: `@prsense/cli` is published under Apache-2.0 on npm. +[^kit]: Kit (cased-kit) is a MIT-licensed Python CLI and library. Reviews can + run from the terminal, against local diffs, or from GitHub Actions; each run + reports LLM cost in stdout, not in a separate dashboard. + <https://kit.cased.com/pr-reviewer/> · <https://github.com/cased/kit> +[^kit-lic]: cased-kit is published under MIT on PyPI and GitHub. + + diff --git a/website/src/content/docs/0.5.0/configuration.md b/website/src/content/docs/0.5.0/configuration.md new file mode 100644 index 00000000..00c21a98 --- /dev/null +++ b/website/src/content/docs/0.5.0/configuration.md @@ -0,0 +1,200 @@ +--- +slug: 0.5.0/configuration +title: Configuration +description: Every environment variable the bot reads, with defaults. +--- + + + +Configuration is read from environment variables (see `.env.example`). Short +names (`AI_*`, `REVIEW_*`, `WEBHOOK_*`, ...) are explicit aliases; every other +`thrillhousebot.*` key is settable through the standard Quarkus env-var mapping +— uppercase with `.`/`-` replaced by `_` (e.g. `thrillhousebot.review.ignored-files` +→ `THRILLHOUSEBOT_REVIEW_IGNORED_FILES`). The AI variables are the ones you +will change per provider: + +| Variable | Purpose | Default | +|---|---|---| +| `AI_API_KEY` | API key for the AI provider | _(required)_ | +| `AI_BASE_URL` | OpenAI-compatible base URL | `https://api.deepseek.com/v1` | +| `AI_MODEL` | Chat model name | `deepseek-chat` | +| `AI_PROVIDER` | Provider label for telemetry (`gen_ai.provider.name`); derived from `AI_BASE_URL` when unset | _(derived)_ | +| `AI_TIMEOUT` | Per-request timeout | `300s` | +| `AI_REASONING_ENABLED` | Send a reasoning hint to reasoning-capable models; when `false` no reasoning parameter is sent and the provider default applies | `false` | +| `AI_REASONING_EFFORT` | Effort sent while enabled: `none`/`low`/`medium`/`high` (`none` explicitly asks the model not to reason); reasoning tokens are billed as output tokens | `low` | +| `GITHUB_APP_ID` | GitHub App ID | _(required)_ | +| `GITHUB_PRIVATE_KEY` | GitHub App private key (PEM) | _(required)_ | +| `GITHUB_WEBHOOK_SECRET` | Webhook HMAC secret | _(required)_ | +| `GITHUB_BOT_LOGINS` | Comma-separated bot account login(s) the bot skips to avoid replying to itself; override when deployed under a different App slug (`<app-slug>[bot]`) | `thrillhousebot[bot],thrillhouse-bot[bot]` | +| `WEBHOOK_DEDUP_TTL` | Webhook deduplication time-to-live for GitHub redeliveries | `24h` | +| `THRILLHOUSEBOT_REVIEW_MANUAL_TRIGGER_ALLOWED_LOGINS` | Comma-separated allowlist of logins permitted to trigger manual `/review` without repo access | _(empty)_ | +| `MANUAL_TRIGGER_AUTH_TIMEOUT` | Upper bound on the manual-trigger write-access check on the webhook ACK thread; fails closed (denies) if GitHub is slower | `5s` | +| `ACK_REACTION_TIMEOUT` | Upper bound on the 👀 command-ack reaction on the webhook ACK thread; the wait is abandoned (reaction may land late) if GitHub is slower | `3s` | +| `AUTO_REVIEW_MIN_INTERVAL` | Minimum interval between automatic reviews of the same PR — pushes within the window are skipped silently, even on a new head SHA (in-memory, per replica). A manual `/review` always bypasses; unset or `0` reviews every push | `0` (disabled) | +| `REVIEW_CI_GATING` | How strictly CI status factors into APPROVE: `strict` holds approval while required CI is pending, failing, or unreadable (fail-closed, safest); `warn` allows APPROVE but notes CI uncertainty in the summary/check; `off` skips CI entirely (findings-only). Prefer `strict` unless flaky CI or incomplete required-context resolution makes soft modes necessary | `strict` | +| `WEBHOOK_SKIP_DRAFTS` | Skip auto-review while a PR is a draft (reviewed once marked ready / on later pushes) | `false` | +| `WEBHOOK_REQUIRED_LABELS` | Comma-separated labels; only auto-review PRs carrying at least one (case-insensitive) | _(empty — no gate)_ | +| `WEBHOOK_EXCLUDED_LABELS` | Comma-separated labels; skip auto-review of PRs carrying any (wins over required) | _(empty)_ | +| `WEBHOOK_BASE_BRANCHES` | Comma-separated globs; only auto-review PRs whose base branch matches one (e.g. `main,release/*`). Globs are gitignore-style: `*` does **not** cross `/`, so use `**` to span slashes (`**` alone matches every branch) | _(empty — all branches)_ | +| `WEBHOOK_IGNORED_BASE_BRANCHES` | Comma-separated globs; skip auto-review of PRs whose base branch matches one (wins over allowlist; same `*`/`**` rule — match nested branches with `**`, e.g. `dependabot/**`) | _(empty)_ | +| `REVIEW_VERIFIER_ENABLED` | Second, skeptical AI pass that re-checks each finding against the diff before posting, dropping or downgrading what it can't confirm (see [AI call budget](#ai-call-budget)); fails open — a verifier error keeps the original findings | `true` | +| `REVIEW_BLOCKING_STRICTNESS` | When findings escalate to `REQUEST_CHANGES`: `balanced` (CRITICAL/HIGH + HIGH confidence), `strict` (any CRITICAL/HIGH), or `lenient` (CRITICAL + HIGH confidence only). See [Blocking strictness](#blocking-strictness) | `balanced` | +| `REVIEW_CONVERSATIONAL_REPLIES_ENABLED` | Answer `@thrillhousebot` mentions in PR threads (including finding replies) with an AI reply | `true` | +| `REVIEW_ADD_DOCS_ENABLED` | Allow the on-demand `/add-docs` command to generate docstrings as committable suggestions | `true` | +| `REVIEW_DIAGRAM_ENABLED` | Include an opt-in Mermaid control-flow diagram in the PR summary | `false` | +| `REVIEW_MAX_INPUT_TOKENS` | Per-call input-token budget for review calls; large PRs are split into batches that each fit it. Bounded by the active model's input cap (see [Per-model AI settings](#per-model-ai-settings)). `0` disables token budgeting | `48000` | +| `REVIEW_OUTPUT_BUFFER_TOKENS` | Tokens reserved out of the input budget for the model's response | `8192` | +| `REVIEW_MAX_AI_CALLS` | Cap on AI calls per review (batch calls plus the final summary call); files that still don't fit are reported by name as omitted | `6` | +| `REVIEW_TOKEN_SAFETY_MARGIN` | Fraction of the input budget actually used, absorbing token-estimate error | `0.9` | +| `REVIEW_MAX_DIFF_LINES` | Line cap on single-call diff renders (`/describe`, `/changelog`, `/add-docs`, replies, budgeting-disabled review). Token-budgeted reviews ignore it (planner owns coverage by tokens); `0` disables the cap | `5000` | +| `THRILLHOUSEBOT_REVIEW_MAX_REVIEW_COMMENTS` | Maximum inline comments posted per review; findings over the cap are surfaced in the summary instead of dropped | `50` | +| `THRILLHOUSEBOT_REVIEW_MAX_AI_RETRIES` | Attempts per failed AI call before the review errors out | `5` | +| `THRILLHOUSEBOT_REVIEW_AI_RETRY_BASE_DELAY_MS` | Base delay of the exponential retry backoff, in milliseconds | `2000` | +| `THRILLHOUSEBOT_REVIEW_AI_TIMEOUT_SECONDS` | Client-side wait per AI streaming attempt; keep it >= `AI_TIMEOUT` so timed-out attempts don't leave orphaned provider streams | `300` | +| `THRILLHOUSEBOT_REVIEW_INSTRUCTIONS_FILE` | Repo-relative path of the per-repo instructions file read on each review | `.github/thrillhousebot.md` | +| `THRILLHOUSEBOT_REVIEW_IGNORED_FILES` | Comma-separated gitignore-style globs excluded from review — lockfiles, generated code, build output. `*` does not cross `/`; use `**` to span directories. Replaces (not extends) the default list, so re-include the defaults you still want | `**/pom.xml,**/package-lock.json,**/*.lock,**/*.generated.*,**/target/**` | +| `REVIEW_LABELS_ENABLED` | Opt in to context-aware PR labels (see [PR labels](#pr-labels)) | `false` | +| `REVIEW_LABELS_APPLY` | When labels are enabled, add them to the PR instead of only suggesting them in a comment | `false` | +| `REVIEW_LABELS_ALLOW_CREATE` | Allow the bot to create suggested labels that don't exist yet | `false` | +| `REVIEW_LABELS_MAX` | Maximum labels applied or suggested per PR | `3` | +| `GITHUB_CLIENT_ID` / `GITHUB_CLIENT_SECRET` | OAuth credentials for dashboard login | _(required for dashboard)_ | +| `DASHBOARD_URL` | Public dashboard URL (OAuth callback base) | `http://localhost:8080` | +| `DATASOURCE_DB_KIND` | `h2` or `postgresql` | `h2` (dev), `postgresql` (`%prod`) | +| `HTTP_CONNECT_TIMEOUT` | Outbound HTTP connect timeout (GitHub API, OAuth) | `10s` | +| `HTTP_REQUEST_TIMEOUT` | Outbound HTTP request timeout (GitHub API, OAuth) | `10s` | +| `WEBSOCKET_KEEPALIVE_MS` | Dashboard WebSocket keepalive interval in ms; `0` or negative disables it (and stale replay-buffer eviction) | `25000` | + +### AI call budget + +A review that reports findings makes **two** model calls by default, not one: +the review call itself plus a verification call that re-sends the diff and the +candidate findings, so budget roughly **2× tokens** per flagged review. On +large PRs under token-aware budgeting this becomes N batch review calls + N +per-batch verification calls + one summary call. Set +`REVIEW_VERIFIER_ENABLED=false` to skip only the AI verifier — cheaper, at the +cost of more false positives; a deterministic hedging guard still runs, and a +verifier failure never blocks the review (it fails open, keeping the original +findings). + +### Blocking strictness + +By default (`REVIEW_BLOCKING_STRICTNESS=balanced`), only **CRITICAL** or **HIGH** +risk findings that the model reports at **HIGH** confidence escalate the PR +review to `REQUEST_CHANGES` (and fail the check run). Medium/low-confidence +severity findings still post as comments with a neutral check. + +| Mode | Blocks merge when | +|---|---| +| `balanced` (default) | CRITICAL or HIGH risk **and** HIGH confidence | +| `strict` | CRITICAL or HIGH risk, **any** confidence | +| `lenient` | CRITICAL risk **and** HIGH confidence only | + +**Security-team recommendation:** use `strict` so a CRITICAL/HIGH finding cannot +slip through as a comment just because confidence was demoted. Be aware that +under `strict`, the finding verifier's confidence demotions (hedged claims → +medium/low) no longer prevent a merge block — only risk reduction or dropping +the finding does. Stay on `balanced` if you want verifier demotions to keep +speculative severity findings non-blocking. + +This knob controls the **review verdict** only. Where findings are posted +(inline thread vs summary) is separate confidence gating tracked in +[#105](https://github.com/devops-thiago/ThrillhouseBot/issues/105); when that +lands, low-confidence findings can move out of the inline stream without +changing these blocking rules. + +The app validates configuration at startup and **fails fast** if a required value +(`GITHUB_APP_ID`, `GITHUB_PRIVATE_KEY`, `GITHUB_WEBHOOK_SECRET`, `AI_API_KEY`) is missing or — for +the private key — not a valid PEM RSA key, naming every offending variable in one message instead of +surfacing later on the first webhook or review. Dashboard OAuth (`GITHUB_CLIENT_ID` / +`GITHUB_CLIENT_SECRET`) is optional: leave both unset and the dashboard login is simply disabled. + +Cost tracking uses per-model pricing keyed by the model name, for example: + +```properties +thrillhousebot.ai.pricing.deepseek-chat.input-per-1k=0.00014 +thrillhousebot.ai.pricing.deepseek-chat.output-per-1k=0.00028 +``` + +If you switch to a different `AI_MODEL`, add a matching +`thrillhousebot.ai.pricing.<model>.*` pair so the dashboard can compute cost. +Without an entry the bot still records tokens, but warns once and flags sessions +as "no pricing" instead of showing `$0`. + +### Per-model AI settings + +Model-specific settings live under `thrillhousebot.ai.models.<model>.*`, keyed +by the model name (the `AI_MODEL` value) like the pricing map. Only the active +model's entry is read, so you can keep entries for every model you use and +switch `AI_MODEL` freely: + +```properties +# Input hard cap (the model's context window). The effective review budget is +# min(REVIEW_MAX_INPUT_TOKENS, cap); models without an entry get a 128000 cap. +thrillhousebot.ai.models.deepseek-chat.max-input-tokens=64000 +# Per-model overrides of REVIEW_OUTPUT_BUFFER_TOKENS / REVIEW_TOKEN_SAFETY_MARGIN +thrillhousebot.ai.models.deepseek-chat.output-buffer-tokens=8192 +thrillhousebot.ai.models.deepseek-chat.token-safety-margin=0.9 +# Generation parameters, sent on every chat call when set +thrillhousebot.ai.models.deepseek-chat.temperature=0.2 +thrillhousebot.ai.models.deepseek-chat.top-p=0.95 +thrillhousebot.ai.models.deepseek-chat.max-output-tokens=8192 +thrillhousebot.ai.models.deepseek-chat.frequency-penalty=0.1 +thrillhousebot.ai.models.deepseek-chat.presence-penalty=0.1 +thrillhousebot.ai.models.deepseek-chat.seed=42 +``` + +Notes: + +- **`max-input-tokens` is a cap, not the budget.** `REVIEW_MAX_INPUT_TOKENS` + stays the spend knob; the per-model value keeps it from overshooting the + model's real window. To use a large-context model beyond 128k, raise both. + Startup logs a warning whenever the cap lowers your configured budget. +- **Quote keys with `.` or `/`** (`thrillhousebot.ai.models."gpt-5.5".…`), the + same rule as the pricing map. Override via env — hyphen-only keys use underscores + (`THRILLHOUSEBOT_AI_MODELS_DEEPSEEK_V4_PRO_MAX_INPUT_TOKENS=1000000`); dotted keys use the + quoted-key form (`THRILLHOUSEBOT_AI_MODELS__GPT_5_5__MAX_INPUT_TOKENS=256000`). + `application.properties` ships empty stubs for known models so SmallRye can + disambiguate hyphenated keys — [Quarkus env mapping](https://quarkus.io/guides/config-reference#environment-variables). + For a model without a stub, add an empty + `thrillhousebot.ai.models."<model>".max-input-tokens=` line (external + `application.properties` or `-D`) alongside the env var. +- **`top_k` is not available** on the OpenAI-compatible wire; it becomes + relevant only with native provider integrations. +- **`max-output-tokens` vs `output-buffer-tokens`**: `max-output-tokens` is the + hard response-length cap sent to the provider; `output-buffer-tokens` only + reserves input-budget headroom for the map-reduce budgeter. Keep the buffer + at least as large as the output cap so a response the model is allowed to + produce always has reserved room — set both when capping output. +- **`seed`** is a best-effort determinism hint (same seed + same parameters aims + for the same sampling) on providers that support it; unsupported providers + ignore it. For deterministic reviews, prefer a low `temperature` first. +- **Generation-parameter validation** happens at boot: temperature must be in + `[0, 2]`, `top-p` in `(0, 1]`, penalties in `[-2, 2]`, token counts positive — + a typo in any entry (even an inactive model's) fails startup with a message + naming the key. + + + + + +## PR labels + +ThrillhouseBot can suggest context-aware labels (area, change type, risk) drawn +from the diff. The feature is **off by default**; turn it on with +`REVIEW_LABELS_ENABLED=true`. + +When enabled, the model is shown the repository's existing labels and picks the +few that best describe the PR — it only ever chooses from labels that already +exist, so it respects whatever label scheme the repo already uses. What happens +next depends on `REVIEW_LABELS_APPLY`: + +- `false` (default): the suggestions are posted as a one-line comment on the + first review, leaving the decision to a maintainer. +- `true`: the labels are added to the PR automatically. + +Set `REVIEW_LABELS_ALLOW_CREATE=true` to let the bot create a suggested label +that doesn't exist yet (off by default, so it never invents labels), and +`REVIEW_LABELS_MAX` to cap how many labels it applies or suggests (default `3`). +Labelling is best-effort — a failure here never blocks or fails the review. + + diff --git a/website/src/content/docs/0.5.0/contributing.md b/website/src/content/docs/0.5.0/contributing.md new file mode 100644 index 00000000..d038bc60 --- /dev/null +++ b/website/src/content/docs/0.5.0/contributing.md @@ -0,0 +1,137 @@ +--- +slug: 0.5.0/contributing +title: Contributing +description: Development setup and the CI bar for contributions. +--- + + + + +This project follows the [Contributor Covenant](https://github.com/devops-thiago/ThrillhouseBot/blob/main/CODE_OF_CONDUCT.md). By taking +part you agree to uphold it. + +## Where to start + +- **[GitHub Discussions](https://github.com/devops-thiago/ThrillhouseBot/discussions)** — questions, setup help, and general conversation. Start with the pinned [welcome post](https://github.com/devops-thiago/ThrillhouseBot/discussions/1) if you are new. +- Issues labeled [`good first issue`](https://github.com/devops-thiago/ThrillhouseBot/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) are well-scoped and explained in detail. +- Issues labeled [`help wanted`](https://github.com/devops-thiago/ThrillhouseBot/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) are larger but ready to be picked up. +- For bugs and feature requests, use the [issue templates](https://github.com/devops-thiago/ThrillhouseBot/issues/new/choose). For larger changes, open an issue first so we can discuss the approach before you invest time. + +## Development setup + +Prerequisites: **Java 25+**, **Node.js 22+** (dashboard only), **Docker** (native builds only). Maven is bundled via the wrapper. + +Follow the [README Quick Start](https://github.com/devops-thiago/ThrillhouseBot#quick-start) for cloning, credentials, dev mode, and webhook forwarding — it is the single source of truth for setup commands. Contributor-specific extras: + +```bash +cd frontend && npm install && npm run dev # dashboard with hot reload +``` + +## Before you open a PR + +Run the Java verification locally before opening a PR: + +```bash +./mvnw spotless:apply # google-java-format; CI runs spotless:check +./mvnw verify # tests + JaCoCo coverage gate + SpotBugs +``` + +If you changed the dashboard, also run `cd frontend && npm ci && npm run test && npm run build`. + +To exercise the UI without a backend, run `cd frontend && npm run dev:mock`. +CI additionally runs Dependency Review (required), SonarCloud, Trivy, and a Docker +build check on pull requests — see **Dual-gate merge policy** below. +To build a disposable test image for a branch or PR (JVM or native, without +touching `latest`), use the **Docker test image** workflow in Actions +(`.github/workflows/docker-test-image.yml`). + +The bar, enforced by CI: + +- **All tests pass** — and new code comes with tests. Test behavior, not implementation: prefer asserting observable outcomes over `assertDoesNotThrow`, and avoid reflection on private members (relax to package-private instead if a test genuinely needs the seam). +- **Coverage doesn't drop** — JaCoCo, Codecov patch coverage, and the SonarCloud quality gate all run on every PR. +- **SpotBugs clean** at `effort=Max`/`threshold=Low`. If you hit a false positive, prefer restructuring; document any exclusion in `config/spotbugs-exclude.xml` with a justification comment. +- **No new Sonar issues** — the quality gate requires an A reliability/security rating on new code. + +## Dual-gate merge policy + +ThrillhouseBot's LLM review and the repo's static CI gates are **complementary**, not substitutes. Merge only when **both** are green (or explicitly waived by a maintainer with a written reason on the PR). + +| Gate | What it catches | Required on this repo | +|------|-----------------|------------------------| +| **Static** | Dependency CVEs/license regressions ([Dependency Review](https://github.com/devops-thiago/ThrillhouseBot/blob/main/.github/workflows/dependency-review.yml)), filesystem CVEs (Trivy), format/tests/frontend, SpotBugs/Sonar/CodeQL | Yes — `format`, `test`, `frontend`, `trivy`, and `dependency-review` are required status checks on `main`/`develop` | +| **ThrillhouseBot** | Diff narrative, incomplete fixes, framework-context issues static tools miss | Soft gate — wait for the **ThrillhouseBot Review** check / posted review when the App is installed; do not merge solely because CI is green if the bot flagged open threads | + +Optional third signal: Cursor Bugbot (or similar) may run on PRs. It is **not** a merge requirement here — useful when present, never a replacement for the two gates above. + +This policy does **not** fold linters into the LLM prompt ([#34](https://github.com/devops-thiago/ThrillhouseBot/issues/34)); that work stays separate. The static gate is the CI safety net for supply-chain and compile-time classes of bug the bot can miss. + +### When the gates disagree + +Maintainers triage; do not "average" the signals away. + +| Static | ThrillhouseBot | What to do | +|--------|----------------|------------| +| Red | Green / quiet | **Fix or formally suppress the static finding.** LLM silence is not evidence the CVE is safe. Prefer a real version bump; if the advisory does not apply to this codebase, document the hold with the file the failing tool actually reads — Dependency Review: `allow-ghsas` or `.github/dependency-review-config.yml`; Trivy: `.trivyignore`; OpenSSF Scorecard / osv-scanner: `osv-scanner.toml` (Jackson GHSA pattern in [#308](https://github.com/devops-thiago/ThrillhouseBot/pull/308)). | +| Green | Red / open threads | **Address or refute the bot findings** in the PR discussion before merge. Static green does not clear logic/copy/incomplete-fix issues. | +| Red | Red | Clear the **static** gate first (often blocks merge already), then resolve bot threads. | +| Same area, conflicting severity | — | Prefer the **static** tool for dependency/CVE/license facts; prefer the **bot** for intent, call-site context, and "did the PR actually finish the fix?" Prefer neither blindly — leave a short maintainer note on the PR. | + +Waivers (rare): only a maintainer may merge with a red soft gate or a documented static suppression, and the PR must record who waived what and why. + +## Commit messages + +The project uses [Conventional Commits](https://www.conventionalcommits.org/): `feat:`, `fix:`, `perf:`, `test:`, `docs:`, `ci:`, `deps:`. One logical change per commit; explain the *why* in the body when it isn't obvious. + +## Architecture + +See the [architecture overview](https://devops-thiago.github.io/ThrillhouseBot/architecture/) +(source: [docs/ARCHITECTURE.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/docs/ARCHITECTURE.md)). Package flow: +`webhook/` → `review/` (`review/ai/`) → `github/` → `dashboard/` (`frontend/`). + +## Adding an AI provider + +There is no provider-specific code to write. The model is reached through +LangChain4j's OpenAI-compatible client, so a new provider is configuration: point +`AI_BASE_URL` and `AI_MODEL` at the endpoint, and add a matching +`thrillhousebot.ai.pricing.<model>.input-per-1k` / `.output-per-1k` pair in +`application.properties` if you want the dashboard to track its cost. The +[README provider table](https://github.com/devops-thiago/ThrillhouseBot#provider-support) lists the ones that are known +to work. + +## Prompt eval corpus + +Changes to the review or verifier prompts (`PrReviewPrompts`, `FindingVerifierPrompts`) +should be checked against the labeled regression corpus in +`src/test/resources/evalcorpus/` before they ship. Each case directory pins a real +dogfood outcome — a `case.json` spec plus `diff.txt` in the exact format the review +pipeline sends to the model: + +- **verifier** cases feed a candidate finding through the second-pass audit and assert + the expected verdict (`confirmed` / `downgraded` / `rejected`); +- **generator** cases run the first-pass review prompt over a diff and assert that a + finding matching the case's keywords is (`must-find`) or is not (`must-not-find`) + raised on the target file. + +The corpus schema is validated by `EvalCorpusTest` in every build. The live suite is +opt-in — it calls the configured AI provider: + +```bash +QUARKUS_LANGCHAIN4J_OPENAI_API_KEY=<key> ./mvnw test -Peval -Dtest=PromptEvalTest +``` + +Point `AI_BASE_URL` / `AI_MODEL` at the provider you want to evaluate (defaults apply +otherwise). Each case is sampled `-Deval.samples` times (default 3) and judged by +majority; `-Deval.tolerated` (default 0) allows a known-unfixed label — one tracked by +an open prompt-hardening issue — to fail without blocking unrelated prompt work. + +To add a case, create a new directory under `evalcorpus/` with `case.json` and +`diff.txt` (see an existing case for the shape). Source new cases from resolved PR +review threads: a refuted false positive becomes `expectedVerdicts: ["rejected"]`, a +confirmed true positive `["confirmed"]`, and note the provenance in `why`. + +## Reporting security issues + +Please **do not** open a public issue — see +[SECURITY.md](https://github.com/devops-thiago/ThrillhouseBot/blob/main/SECURITY.md). + + diff --git a/website/src/content/docs/0.5.0/feedback.md b/website/src/content/docs/0.5.0/feedback.md new file mode 100644 index 00000000..3a235c08 --- /dev/null +++ b/website/src/content/docs/0.5.0/feedback.md @@ -0,0 +1,93 @@ +--- +slug: 0.5.0/feedback +title: Finding feedback +description: Data model, privacy, and retention for maintainer finding feedback signals. +--- + + + + +ThrillhouseBot records lightweight maintainer signals about review findings so a +future cross-review learnings store ([#38](https://github.com/devops-thiago/ThrillhouseBot/issues/38)) +has training data. This is the precursor shipped for +[#324](https://github.com/devops-thiago/ThrillhouseBot/issues/324); it does **not** +yet inject preferences into review prompts. + +## Why poll instead of a reaction webhook? + +GitHub Apps do not receive a `reaction` webhook event. The bot therefore lists +👍 (`+1`) and 👎 (`-1`) on finding comments via the +[Reactions REST API](https://docs.github.com/en/rest/reactions/reactions) when: + +1. A human **replies** on an inline review thread (`pull_request_review_comment` + with `in_reply_to_id`), or +2. A **follow-up review** already loaded inline comments — every bot finding-root + comment across prior rounds is scanned (capped, ordered by comment id), not + only findings from the immediately previous AI response. + +Capture is best-effort and never fails the webhook `200` or the review. + +## Signals + +| Signal | Source | Meaning | +|--------|--------|---------| +| `useful` | `reaction` (`+1`) | Maintainer marked the finding comment 👍 | +| `not_useful` | `reaction` (`-1`) | Maintainer marked the finding comment 👎 | +| `not_useful` | `reply_heuristic` | Reply body matched a conservative phrase (`not useful`, `false positive`, `noise`, 👎, `:-1:`) | + +Only comments that carry the hidden `<!-- thrillhousebot:finding=N -->` marker are +eligible. The bot's own reactions (e.g. 👀 command ack) are ignored. + +## Data model + +Table `finding_feedback` (Hibernate schema-update; no Flyway): + +| Column | Type | Notes | +|--------|------|-------| +| `id` | bigint | Panache / sequence PK | +| `repository` | string | `owner/repo` | +| `prNumber` | int | PR on that repository | +| `githubCommentId` | bigint | Finding root review-comment id | +| `findingIndex` | int (nullable) | 1-based index from the marker | +| `signal` | string | `useful` or `not_useful` | +| `source` | string | `reaction` or `reply_heuristic` | +| `reactorLogin` | string | GitHub login only (lower-cased) | +| `githubReactionId` | bigint (nullable) | Unique when present (idempotent re-poll) | +| `createdAt` | instant | Insert time | + +Unique constraints: + +- `githubReactionId` (when non-null) — reaction redeliveries / re-polls +- `(githubCommentId, reactorLogin, signal, source)` — one logical event per actor + +## Privacy + +Stored PII is limited to the **GitHub login** already present on webhook and API +payloads. No email, display name, IP, or reaction text beyond the fixed emoji +content codes (`+1` / `-1`) is persisted. Finding title/description are not +copied into this table. + +## Retention + +Rows are retained for the lifetime of the deployment database. There is no +automatic purge. Operators may `DELETE` rows or drop the table when +decommissioning an installation. Uninstalling the GitHub App does not currently +auto-delete feedback rows (same posture as `ReviewSession` history). + +## Aggregation API (ContextProvider seam) + +`FindingFeedbackService.summarize(repository)` and `summarizeAll()` return +per-repo `useful` / `not_useful` counts for a future `ContextProvider`. The +dashboard exposes the same aggregates at `GET /api/dashboard/feedback` (session +cookie required; optional `?repository=owner/repo`). + +## Notable classes + +- `FindingFeedback` / `FindingFeedbackRepository` / `FindingFeedbackService` +- `FindingFeedbackCaptureService` — poll + heuristics +- `GitHubReactionClient.listReviewCommentReactions` +- `WebhookController` — schedules capture on review-thread replies +- `ReviewOrchestrator` — capture pass on follow-up reviews + + + diff --git a/website/src/content/docs/0.5.0/getting-started.md b/website/src/content/docs/0.5.0/getting-started.md new file mode 100644 index 00000000..719d788d --- /dev/null +++ b/website/src/content/docs/0.5.0/getting-started.md @@ -0,0 +1,125 @@ +--- +slug: 0.5.0/getting-started +title: Getting started +description: Create the GitHub App, configure the bot, and start it with Docker Compose. +--- + +Everything you need to go from zero to a bot reviewing your pull requests: +create the GitHub App, configure the bot, and start it with Docker Compose. + +## Prerequisites + +- [Docker & Docker Compose](https://docs.docker.com/compose/install/) +- An API key for any [OpenAI-compatible provider](/ThrillhouseBot/providers/) +- A public hostname where GitHub can reach the bot (for local development, + a [Smee.io](https://smee.io/) channel works) + +## 1. Create the GitHub App + +The bot authenticates as a GitHub App, so the app must exist before the bot +starts. The easiest path is the +**[hosted installer](/ThrillhouseBot/install.html)**: type the public hostname +where the bot will run and click **Create ThrillhouseBot GitHub App** — the +page builds the app manifest for you and sends it to GitHub. No local server +needed. + +After GitHub creates the app: + +1. Note the **App ID** (app settings → About). +2. Generate a **private key** (downloads a `.pem` file). +3. Set a **webhook secret**. +4. Copy the **Client ID** and **Client secret** from *Identifying and + authorizing users* (needed for dashboard login). +5. Install the app on your account or organization. + +Alternatively, grab the `code` query parameter GitHub appends to the +post-creation redirect and generate `.env` automatically from the manifest +conversion response: + +```bash +gh api --method POST /app-manifests/<code>/conversions \ + | java scripts/GenEnv.java --host <your-host> +``` + +:::note[Smee setups] +The hosted installer registers Smee webhooks at the channel root and the +OAuth callback at `http://localhost:8080/api/auth/callback`. If you use +`GenEnv.java`, don't pass the Smee URL as `--host` (that writes +`DASHBOARD_URL=https://<host>`); run it without `--host` and then set +`DASHBOARD_URL=http://localhost:8080` in the generated `.env` — dashboard +login only works when `DASHBOARD_URL` matches the registered callback. + +Webhooks land on the Smee channel, so also run the relay client to +forward them to the bot (keep it running alongside the bot): + +```bash +npx smee-client -u https://smee.io/YOUR_CHANNEL -t http://localhost:8080/api/webhook +``` +::: + +:::note[Manual registration instead] +Prefer to register the app by hand? Create it at +<https://github.com/settings/apps/new> with these settings: + +| Setting | Value | +|---|---| +| Webhook URL | `https://<your-host>/api/webhook` | +| Webhook Secret | Random string | +| Repository Permissions | Pull Requests: R/W, Checks: R/W, Contents: Read, Issues: R/W, Actions: Read, Commit Statuses: Read | +| Subscribe to Events | Pull Request, Issue comment, Pull request review comment | +| Identifying & authorizing users | Enabled (for dashboard login) | +| Callback URL | `https://<your-host>/api/auth/callback` | + +For a Smee channel, register the channel URL itself as the Webhook URL (no +`/api/webhook` suffix — Smee delivers only at the channel root) and +`http://localhost:8080/api/auth/callback` as the Callback URL, since OAuth +redirects happen in your browser against the local bot. +::: + +:::tip[Re-registering later] +Once the bot is running, `install.html` on the bot's own URL +(`https://<your-host>/install.html` behind a reverse proxy, or +`http://localhost:8080/install.html` when hitting it directly) builds the +manifest from the detected host automatically — handy for adding the app to +another account. Smee-based dev setups should re-register through the +[hosted installer](/ThrillhouseBot/install.html) instead: the bot's own page +registers the origin it's served from, without the Smee webhook-root handling. +::: + +## 2. Clone and configure + +```bash +git clone https://github.com/devops-thiago/ThrillhouseBot.git && cd ThrillhouseBot +cp .env.example .env +``` + +Edit `.env` with the credentials from step 1: + +| Variable | Value | +|---|---| +| `GITHUB_APP_ID` | From GitHub App settings → About | +| `GITHUB_PRIVATE_KEY` | Downloaded when you generated a private key | +| `GITHUB_WEBHOOK_SECRET` | The webhook secret you set | +| `GITHUB_CLIENT_ID` | From app settings → Identifying and authorizing users | +| `GITHUB_CLIENT_SECRET` | From app settings → Identifying and authorizing users | +| `AI_API_KEY` | Your AI provider's API key | + +## 3. Start the bot + +```bash +docker compose up -d +``` + +The bot is running on `http://localhost:8080`. Point your reverse proxy at it +and you're done — the next pull request on an installed repository gets +reviewed automatically. + +## Next steps + +- Tune the bot with the + [configuration reference](/ThrillhouseBot/configuration/) — auto-review + triggers, labels, timeouts, and more. +- Drive it from a PR with the [comment commands](/ThrillhouseBot/commands/). +- Customize reviews per repository with a `.github/thrillhousebot.md` + instructions file (fallback chain: `.github/copilot-instructions.md` → + `CLAUDE.md` → `AGENTS.md` → `AGENT.md`). diff --git a/website/src/content/docs/0.5.0/index.md b/website/src/content/docs/0.5.0/index.md new file mode 100644 index 00000000..76ff8db0 --- /dev/null +++ b/website/src/content/docs/0.5.0/index.md @@ -0,0 +1,75 @@ +--- +slug: 0.5.0 +title: ThrillhouseBot +description: Self-hosted AI pull-request reviewer — a GraalVM-native GitHub App built with Quarkus. +--- + +> **"Everything's coming up Thrillhouse!"** + +A self-hosted, GraalVM-native PR review bot, built as a GitHub App with Quarkus. +It reviews pull requests using any OpenAI-compatible chat API, so the review is +language-agnostic and you can pick the provider that suits you — including a +local Ollama model, so no code has to leave your network. + +![ThrillhouseBot approving a clean pull request](../../../assets/0.5.0/pr-approval.png) + +## Features + + + +- Reviews diffs for correctness, security, regressions, stale comments, and code quality +- Token-budgeted whole-PR review for large diffs — split into parallel map-reduce batches with omitted files named, not silently dropped +- Configurable auto-review triggers — skip drafts, gate on labels, or filter by base branch — plus an optional per-PR auto-review interval (`AUTO_REVIEW_MIN_INTERVAL`) when you want to cap spend on noisy PRs (off by default; use `/pause` to silence a PR) +- Inline code suggestions on review comments that you can apply with one click +- Every finding is tagged `critical`, `high`, `medium`, or `low` +- Follow-up reviews track whether earlier findings were addressed or justified +- Maintainer 👍/👎 (and "not useful" replies) on finding comments are recorded for a future learnings pipeline — see [Finding feedback](https://devops-thiago.github.io/ThrillhouseBot/feedback/) +- Conversational replies: `@thrillhousebot` it in a PR thread or finding reply and the bot answers in context +- A summary comment on the first run, with a risk breakdown and a changed-files walkthrough +- Operable from the PR with comment commands — `/help`, `/review`, `/summary`, `/describe`, `/changelog`, `/add-docs`, `/resolve`, `/pause`, `/resume` +- Live dashboard (Next.js) with a WebSocket activity feed, cost charts, and token tracking +- OpenTelemetry traces, token histograms, cost counters, and latency metrics +- Optional reasoning-effort dial and per-model generation/budget caps for OpenAI-compatible endpoints +- Reads per-repo instructions from `.github/thrillhousebot.md`, falling back to Copilot/Claude/Agents files +- Compiles ahead-of-time with GraalVM/Mandrel, so it starts fast and stays small + + + +## Where to go next + +- **[Getting started](/ThrillhouseBot/getting-started/)** — create the GitHub + App with the [hosted installer](/ThrillhouseBot/install.html) and run the bot + with Docker Compose. +- **[Commands](/ThrillhouseBot/commands/)** — drive the bot from a PR: + `/review`, `/describe`, `/changelog`, `/add-docs`, and more. +- **[Configuration](/ThrillhouseBot/configuration/)** — every environment + variable, with defaults. +- **[AI providers](/ThrillhouseBot/providers/)** — point the bot at the + OpenAI-compatible endpoint of your choice. +- **[Architecture](/ThrillhouseBot/architecture/)** — how a review flows + through the system. +- **[Finding feedback](/ThrillhouseBot/feedback/)** — maintainer 👍/👎 capture + for the learnings pipeline. +- **[How it compares](/ThrillhouseBot/comparison/)** — an honest look at where + ThrillhouseBot sits next to other AI code-review tools. +- **[Contributing](/ThrillhouseBot/contributing/)** — development setup and the + CI bar. + +## Dashboard + +The built-in dashboard (Next.js, served by the bot itself) shows summary cards, +a live activity feed that streams the model's output as a review runs, cost +charts by model, token breakdowns, and a paginated session history: + +![Dashboard Overview with summary cards, live model-output panel, and recent activity](../../../assets/0.5.0/live-streaming.png) + +## Community and license + +Questions and setup help belong in +[GitHub Discussions](https://github.com/devops-thiago/ThrillhouseBot/discussions); +bugs and feature requests in +[Issues](https://github.com/devops-thiago/ThrillhouseBot/issues/new/choose). + +Licensed under the +[Apache License 2.0](https://github.com/devops-thiago/ThrillhouseBot/blob/main/LICENSE) +(SPDX: `Apache-2.0`). diff --git a/website/src/content/docs/0.5.0/providers.md b/website/src/content/docs/0.5.0/providers.md new file mode 100644 index 00000000..ebad528e --- /dev/null +++ b/website/src/content/docs/0.5.0/providers.md @@ -0,0 +1,30 @@ +--- +slug: 0.5.0/providers +title: AI providers +description: Point the bot at any OpenAI-compatible chat endpoint. +--- + + + +ThrillhouseBot talks to any endpoint that implements the OpenAI chat-completions +API. Point `AI_BASE_URL` and `AI_MODEL` at your provider of choice: + +| Provider | `AI_BASE_URL` | Example `AI_MODEL` | +|---|---|---| +| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` | +| OpenRouter | `https://openrouter.ai/api/v1` | `openai/gpt-4o-mini` | +| Alibaba Cloud (Model Studio) | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `qwen-plus` | +| OpenAI | `https://api.openai.com/v1` | `gpt-4o-mini` | +| Ollama (local) | `http://localhost:11434/v1` | `llama3.2` | + +The default is DeepSeek, used only because it is inexpensive; nothing in the bot +is tied to it. + + + +There is no provider-specific code in the bot — a new provider is just +configuration. See +[Adding an AI provider](/ThrillhouseBot/architecture/#adding-an-ai-provider) +in the architecture notes, and add a `thrillhousebot.ai.pricing.<model>.*` pair +(see [Configuration](/ThrillhouseBot/configuration/)) if you want cost tracking +for the model. diff --git a/website/src/content/versions/0.5.0.json b/website/src/content/versions/0.5.0.json new file mode 100644 index 00000000..3dd7a284 --- /dev/null +++ b/website/src/content/versions/0.5.0.json @@ -0,0 +1,36 @@ +{ + "sidebar": [ + { + "label": "Home", + "slug": "index" + }, + { + "label": "Getting started", + "slug": "getting-started" + }, + { + "label": "Commands", + "slug": "commands" + }, + { + "label": "Configuration", + "slug": "configuration" + }, + { + "label": "AI providers", + "slug": "providers" + }, + { + "label": "Architecture", + "slug": "architecture" + }, + { + "label": "How it compares", + "slug": "comparison" + }, + { + "label": "Contributing", + "slug": "contributing" + } + ] +} diff --git a/website/versions.json b/website/versions.json index dee450a0..89c517ee 100644 --- a/website/versions.json +++ b/website/versions.json @@ -1,8 +1,12 @@ { "current": { - "label": "v0.5.0" + "label": "v0.6.0" }, "versions": [ + { + "slug": "0.5.0", + "label": "v0.5.0" + }, { "slug": "0.4.0", "label": "v0.4.0"