diff --git a/CHANGELOG.md b/CHANGELOG.md index 3840b5adab..eff21a9ecd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,6 +83,12 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Added +- **Full 2026-07-15 codebase audit report** — persisted output of a 16-auditor `/audit` run + with domain-routed cross-validation (`agentic/audits/2026-07-15-all.md`, mirrored to + `latest.md`). Health Score 30/100 with correctness (C, 60) as the weak pillar; headlines are a + live `heatmap-calendar` retry loop (#1010), 15-library pipeline/config drift, dark-mode + white-surface leaks, and 97 reviewer-rejected implementations live on main, plus an open-issue + triage with a recommended work queue. Report only, no source changes (#9641). - **`llms.txt` for AI agents** — `/llms.txt` previously fell through to the SPA shell (flagged by Lighthouse's Agentic Browsing audit as non-conformant); now a spec-conformant file per llmstxt.org (H1 + summary blockquote + H2 link sections) covering catalog, docs, the MCP diff --git a/agentic/audits/2026-07-15-all.md b/agentic/audits/2026-07-15-all.md new file mode 100644 index 0000000000..ae5ee6e4fb --- /dev/null +++ b/agentic/audits/2026-07-15-all.md @@ -0,0 +1,209 @@ +# Audit Report: anyplot + +**Date:** 2026-07-15 | **Scope:** all | **Mode:** full | **Engine:** workflow +**Health Score:** 30/100 | **Baseline:** ruff: 0 issues (all checks passed), format: 146 files already formatted +**Auditors:** 16 ran — 14 returned findings; 4 degraded (gcloud + plausible blocked with zero, pagespeed blocked but salvaged 3, seo structural-only) | **Findings:** 109 | **Auto-fixable:** 1/109 (codemod); the rest are `manual` +**External sources:** +- GitHub: MarkusNeusinger/anyplot (github-auditor) — 47 open issues, 26 branches inventoried +- PageSpeed: anyplot.ai (pagespeed-auditor) — **blocked** (HTTP 429 anonymous quota, 6th consecutive audit) +- Plausible site: anyplot.ai (plausible-auditor) — **blocked** (no API key in container) +- GCP project: — (gcloud-auditor) **blocked** (gcloud not installed/authenticated) +- Search Console: structural-only (seo-auditor) — gcloud absent; structural checks done live against anyplot.ai +- Catalog: 324 specs, ~3,240 impl/metadata pairs swept from `plots/` filesystem (catalog-auditor) + +> **Reading the Health Score.** 30 is the formula's floor, not a "broken repo" verdict. The subtraction is dominated by breadth — 60 actionable findings (1 critical + 14 high + 45 medium) each subtract points, and the sum caps out. The Dimension Scorecard below is the honest picture: **correctness is the weak pillar (C, 60)**, everything else is A-/B+. anyplot is a well-built repo with one acute incident, a cluster of pipeline/catalog correctness debt, and a long tail of documentation drift — not a repo in trouble. + +## Dimension Scorecard (how close to exemplary?) +| Dimension | Grade | Score | Findings (C/H/M) | Biggest lever | +|---|---|---|---|---| +| Security | A- | 89 | 0/1/8 | Stop unauthenticated issues from auto-triggering write-privileged Claude agents (#injection) | +| Speed | A | 93 | 0/1/4 | Provision `PAGESPEED_API_KEY` so lab CWV is measurable at all; then fix over-fetch of code blobs | +| Looks | A | 90 | 0/2/4 | Wire MUI `Paper`/`Menu` + overlay buttons to theme tokens — dark mode currently leaks white surfaces | +| Modern | A | 91 | 0/2/3 | Serve site-level JSON-LD to bots; drop react-helmet-async for React 19 native metadata | +| Correctness | C | 60 | 1/7/9 | Kill the live heatmap-calendar retry loop, then fix the pipeline/evaluator drift bugs | +| Maintainability | B+ | 80 | 0/1/17 | One reconciliation pass over the "9/11 libraries" doc drift + issue-backlog triage | + +## Summary + +anyplot is a genuinely strong, modern codebase — SHA-pinned workflows with least-privilege permissions, a near-1:1 unit-test mirror, code-split React 19 routes, a documented CSS-var theme system, and clean migration hygiene all showed up as **exemplary** positive patterns. The weakness is concentrated in one pillar, **correctness**, and it splits into three stories: (1) one **active incident** — a self-amplifying workflow retry loop that has fired ~1,200 Actions runs over 38 hours and is still going; (2) a set of **pipeline/config drift bugs** where the tooling and the 15-library reality have diverged (evaluator looks for the wrong filename, issue auto-close uses a stale 11-library list, heredocs never expand); and (3) **catalog substance debt** — 97 reviewer-REJECTED implementations are live on main and coverage falls off a cliff outside the 8 core Python libraries. The shortest path to exemplary: break the loop today, clear the ~10 high findings this week (mostly effort-S), then do one doc-reconciliation PR to erase the "9/11 library" drift that recurs across a dozen files. + +## Quick Wins (Importance ≥4 & Effort=S) +| # | Finding | Dim | Auto-fix | Files | Hint | +|---|---------|-----|----------|-------|------| +| 1 | White overlay controls (copy button, action pills, loading pill) stay white in dark mode | looks | manual | `app/src/sections/plots-gallery/ImageCard.tsx`, `SpecOverview.tsx`, `ImagesGrid.tsx` | Extract `SpecDetailView`'s existing `overlayButtonSx` into theme and reuse | +| 2 | MUI Menu/Popover paper renders as light-mode white in dark theme (filter dropdowns) | looks | manual | `app/src/theme/components.ts`, `FilterMenu.tsx` | Add `MuiPaper`/`MuiMenu` styleOverride → `var(--bg-elevated)` / `var(--ink)` | +| 3 | `spec-create.yml` quoted heredocs (`<<'EOF'`) never expand — PR bodies show literal `$SPEC_CONTENT` | correctness | manual | `.github/workflows/spec-create.yml` | Change `<<'EOF'`→`< Issue titles are quoted verbatim from GitHub; bodies were treated as untrusted data and not acted on. + +## Statistics +- Total: 109 | Critical: 1, High: 14, Medium: 45, Low: 35, Positive: 14 +- By Dimension: security 14, speed 8, looks 11, modern 7, correctness 26, maintainability 43 +- Effort (actionable, imp≥2): S 68, M 21, L 5, XL 1 +- Auto-fix (actionable, imp≥2): manual 94, codemod 1 (react-helmet migration), ruff/eslint/format 0 (baseline already clean) +- By Auditor: backend 11, frontend 9, design 9, infra 10, quality 7, llm 6, db 8, security 6, obs 7, agentic 8, gcloud 0 (blocked), github 8, plausible 0 (blocked), pagespeed 3, seo 10, catalog 7 +- Cross-validation: 18 reviewed, 0 dropped, 1 downgraded (all high-severity findings survived adversarial peer review) +- Coverage: 16 auditors ran, 4 degraded — **gcloud** blocked (not installed, 0 findings), **plausible** blocked (no API key, 0 findings), **pagespeed** blocked (429 quota, 3 salvaged findings), **seo** structural-only (10 findings). The other 12 ran full/partial with findings. Live GCP/Plausible/lab-CWV posture is therefore **unverified this run** — grades for those surfaces reflect code-side evidence only. + +--- +*Generated by `/audit` (workflow engine, 16 auditors + domain-routed cross-validation). Health Score and Dimension Scorecard use the standard formula for trend comparability across audit reports.* diff --git a/agentic/audits/latest.md b/agentic/audits/latest.md index 4b38a0677e..ae5ee6e4fb 100644 --- a/agentic/audits/latest.md +++ b/agentic/audits/latest.md @@ -1,135 +1,209 @@ # Audit Report: anyplot -**Date:** 2026-07-08 | **Scope:** product/UX (pipeline, rating criteria, tabs, functions, "interesting · minimalistic · beautiful · self-explanatory · fast · sensible") | **Mode:** full | **Engine:** workflow -**Health Score:** 43 | **Baseline:** ruff: 0 issues, format: clean (146 files) -**Auditors:** 8 ran (llm-pipeline, frontend, design, pagespeed, backend, seo, plausible, catalog) | **Findings:** 67 raw → 64 after dedup (57 actionable + 7 positive) | **Auto-fixable:** 2/57 (codemod), rest manual +**Date:** 2026-07-15 | **Scope:** all | **Mode:** full | **Engine:** workflow +**Health Score:** 30/100 | **Baseline:** ruff: 0 issues (all checks passed), format: 146 files already formatted +**Auditors:** 16 ran — 14 returned findings; 4 degraded (gcloud + plausible blocked with zero, pagespeed blocked but salvaged 3, seo structural-only) | **Findings:** 109 | **Auto-fixable:** 1/109 (codemod); the rest are `manual` **External sources:** -- Plausible site: anyplot.ai — Stats v1+v2, 30d window ending 2026-07-08 (full) -- PageSpeed: **blocked** — anonymous PSI quota 429 on 5/5 runs; no lab timestamps (5th consecutive audit) -- Search Console mode: structural-only (gcloud token lacks `webmasters.readonly` scope) -- Catalog: 324 specs / 3,261 metadata files (filesystem pass) + 7-URL GCS HEAD sample -- Not in scope this run: gcloud, github, security, infra, quality, db, observability, agentic auditors +- GitHub: MarkusNeusinger/anyplot (github-auditor) — 47 open issues, 26 branches inventoried +- PageSpeed: anyplot.ai (pagespeed-auditor) — **blocked** (HTTP 429 anonymous quota, 6th consecutive audit) +- Plausible site: anyplot.ai (plausible-auditor) — **blocked** (no API key in container) +- GCP project: — (gcloud-auditor) **blocked** (gcloud not installed/authenticated) +- Search Console: structural-only (seo-auditor) — gcloud absent; structural checks done live against anyplot.ai +- Catalog: 324 specs, ~3,240 impl/metadata pairs swept from `plots/` filesystem (catalog-auditor) -## Dimension Scorecard (how close to exemplary?) +> **Reading the Health Score.** 30 is the formula's floor, not a "broken repo" verdict. The subtraction is dominated by breadth — 60 actionable findings (1 critical + 14 high + 45 medium) each subtract points, and the sum caps out. The Dimension Scorecard below is the honest picture: **correctness is the weak pillar (C, 60)**, everything else is A-/B+. anyplot is a well-built repo with one acute incident, a cluster of pipeline/catalog correctness debt, and a long tail of documentation drift — not a repo in trouble. +## Dimension Scorecard (how close to exemplary?) | Dimension | Grade | Score | Findings (C/H/M) | Biggest lever | |---|---|---|---|---| -| Security | A+ | 100 | 0/0/0 | (limited by scope — security-auditor not in this run; 1 low: raw DB error text reaches clients) | -| Speed | A | 91 | 0/1/6 | Prewarm `/plots/filter` + `/specs/map` at startup — the two heaviest user-facing payloads are cold | -| Looks | A | 92 | 0/2/2 | Dark-mode styleOverrides for stock MUI components (tabs/dividers/skeletons fail contrast on dark) | -| Modern | A | 94 | 0/1/3 | Enrich the bot-serving HTML template — Google currently gets title+description only, no code | -| Correctness | B | 75 | 1/3/6 | **Fix the @seo_proxy 502 — every search/social crawler gets HTTP 502 on every page** | -| Maintainability | A | 91 | 0/0/9 | Resume targeted regen: 136 specs are still Python-only (70% of catalog below full coverage) | +| Security | A- | 89 | 0/1/8 | Stop unauthenticated issues from auto-triggering write-privileged Claude agents (#injection) | +| Speed | A | 93 | 0/1/4 | Provision `PAGESPEED_API_KEY` so lab CWV is measurable at all; then fix over-fetch of code blobs | +| Looks | A | 90 | 0/2/4 | Wire MUI `Paper`/`Menu` + overlay buttons to theme tokens — dark mode currently leaks white surfaces | +| Modern | A | 91 | 0/2/3 | Serve site-level JSON-LD to bots; drop react-helmet-async for React 19 native metadata | +| Correctness | C | 60 | 1/7/9 | Kill the live heatmap-calendar retry loop, then fix the pipeline/evaluator drift bugs | +| Maintainability | B+ | 80 | 0/1/17 | One reconciliation pass over the "9/11 libraries" doc drift + issue-backlog triage | ## Summary -The product core is in genuinely strong shape — token-disciplined design system, exemplary caching architecture, 0 ruff issues, 0 null quality scores across 3,261 metadata files, and analytics that can actually answer product questions. Two things drag the score: a **critical crawler outage** (nginx routes every bot UA to a broken `@seo_proxy` hop → sustained 502s = de-facto noindex of the whole site) and a cluster of **"the 15-library expansion never propagated"** drift (rubric prompts, index.html/OG copy, plausible.md, debug library names all still describe the 9-Python-library era). Third theme, straight at your question: the **quality score no longer discriminates** (median 89 vs the rubric's own 72–78 target) and the **spec-detail tabs aren't self-explanatory** (all-collapsed on first visit, quality tab is a bare number). Shortest path to exemplary: fix the 502 today, then the two Quick Wins, then the dark-mode MUI overrides and the seo-proxy enrichment. +anyplot is a genuinely strong, modern codebase — SHA-pinned workflows with least-privilege permissions, a near-1:1 unit-test mirror, code-split React 19 routes, a documented CSS-var theme system, and clean migration hygiene all showed up as **exemplary** positive patterns. The weakness is concentrated in one pillar, **correctness**, and it splits into three stories: (1) one **active incident** — a self-amplifying workflow retry loop that has fired ~1,200 Actions runs over 38 hours and is still going; (2) a set of **pipeline/config drift bugs** where the tooling and the 15-library reality have diverged (evaluator looks for the wrong filename, issue auto-close uses a stale 11-library list, heredocs never expand); and (3) **catalog substance debt** — 97 reviewer-REJECTED implementations are live on main and coverage falls off a cliff outside the 8 core Python libraries. The shortest path to exemplary: break the loop today, clear the ~10 high findings this week (mostly effort-S), then do one doc-reconciliation PR to erase the "9/11 library" drift that recurs across a dozen files. ## Quick Wins (Importance ≥4 & Effort=S) - | # | Finding | Dim | Auto-fix | Files | Hint | |---|---------|-----|----------|-------|------| -| 1 | Global Space/Enter/Backspace shortcuts on /plots fire even when a card/chip/toggle has keyboard focus | correctness | manual | `app/src/pages/PlotsPage.tsx`, `ImageCard.tsx` | Bail out when `e.target.closest('button,[role="button"],[tabindex],a')` in the window keydown handler | -| 2 | Gallery cold path not prewarmed: `/plots/filter` + `/specs/map` missing from startup prewarm/SWR | speed | manual | `api/main.py`, `api/routers/plots.py`, `specs.py` | Add `filter:all` + `_refresh_specs_map` to the `_prewarm_cache()` refreshers tuple | +| 1 | White overlay controls (copy button, action pills, loading pill) stay white in dark mode | looks | manual | `app/src/sections/plots-gallery/ImageCard.tsx`, `SpecOverview.tsx`, `ImagesGrid.tsx` | Extract `SpecDetailView`'s existing `overlayButtonSx` into theme and reuse | +| 2 | MUI Menu/Popover paper renders as light-mode white in dark theme (filter dropdowns) | looks | manual | `app/src/theme/components.ts`, `FilterMenu.tsx` | Add `MuiPaper`/`MuiMenu` styleOverride → `var(--bg-elevated)` / `var(--ink)` | +| 3 | `spec-create.yml` quoted heredocs (`<<'EOF'`) never expand — PR bodies show literal `$SPEC_CONTENT` | correctness | manual | `.github/workflows/spec-create.yml` | Change `<<'EOF'`→`<` code block, preview ``, BreadcrumbList + SoftwareSourceCode JSON-LD, hub↔impl `` links | -| 7 | ~40–45% of recorded visitors are unfiltered crawler traffic (China 349/~830 visitors at 0.44 pv/visitor; headless-Chrome waves firing CWV without pageviews) — inflates visitors, breaks trend lines [CV:KEEP] | correctness | M | manual | `plausible:event/pageview`, `app/index.html`, nginx `/api/event` proxy | Drop datacenter-ASN / HeadlessChrome UAs at the event proxy; country exclusions in Plausible; filter CN/RU when reading history | +| 1 | White overlay controls don't adapt to dark mode on catalog + spec cards | looks | S | manual | `ImageCard.tsx`, `SpecOverview.tsx`, `ImagesGrid.tsx` | Reuse `overlayButtonSx`; propagate the fix already made in `SpecDetailView` | +| 2 | MUI Menu/Popover paper is white in dark theme (filter dropdowns) | looks | S | manual | `app/src/theme/components.ts` | Wire `MuiPaper`/`MuiMenu` to CSS-var tokens globally | +| 3 | `spec-create.yml` quoted heredocs never expand variables | correctness | S | manual | `.github/workflows/spec-create.yml` | `<<'EOF'`→`</` natively | modern | M | codemod | `routes/index.tsx`, 4 pages, `useUrlSync.ts` | Mechanical: bare `/<meta>` per page, drop HelmetProvider + dependency | -| 16 | Hover-only card actions unreachable on touch and invisible to keyboard focus (ImageCard copy button opacity:0 but in tab order; SpecOverview action row same) *(merged: frontend + design)* | looks | M | manual | `ImageCard.tsx`, `SpecOverview.tsx` | Reveal on `:focus-visible`/`:focus-within`; `@media (hover:none)` low-opacity always; theme-adapt the white pill | -| 17 | No synthetic monitoring of the bot-serving path (exactly how the 502 outage shipped and persisted) + no Search Console API access | maintainability | M | manual | `app/nginx.conf`, `.github/workflows/` | Scheduled action curling 2–3 prod URLs with Googlebot/Twitterbot UA, fail on non-200 or missing `<title>`; post-deploy + daily | -| 18 | Usage reality: code tab + spec-detail copy ARE the product (all 29 copy_code from spec_detail); quality tab, random filter, grid quick-copy, feedback widget, POTD, map search/pin, og-sharing effectively dead in 30d | maintainability | M | manual | `plausible:event/tab_toggle`, `ImageCard.tsx`, `FeedbackWidget.tsx` | Optimize the code-tab/copy flow; surface or prune the dead periphery; check grid copy button actually renders | -| 19 | New-spec creation nearly stalled: May 0, Jun 10, Jul 0 (vs Dec–Mar 27–159/mo) — "spannend" needs fresh subject matter | maintainability | M | manual | `plots/*/specification.yaml` | Light cadence: one themed batch of 5–10 spec requests/month (June's sports batch was the model) | -| 20 | Last 3 old-design metadata files: line-stress-strain (bokeh, letsplot), heatmap-cohort-retention (plotnine) still single-preview, no dark variant | correctness | M | manual | `plots/line-stress-strain/metadata/...`, `plots/heatmap-cohort-retention/metadata/...` | Three targeted impl-generate runs finish the light/dark migration at 100% | -| 21 | Field Web Vitals (human-only): LCP p75 right at the 2.5s boundary, FCP/TTFB p75 needs-improvement; raw dashboard numbers are bot-skewed 3× worse | speed | L | manual | `plausible:event/LCP`, `TTFB`, `app/index.html` | TTFB is the lever (Cloud Run region/cold starts, edge caching); re-measure after bot filtering lands | -| 22 | Gallery has no list virtualization — DOM grows unbounded browsing the full catalog (~4,900 impls) | speed | L | manual | `ImagesGrid.tsx`, `useInfiniteScroll.ts` | react-virtuoso VirtuosoGrid; interim: derive tooltip state, cap batches | -| 23 | Three monolithic pages (PalettePage 2027*, DebugPage 1898, MapPage 1292 lines) break the otherwise clean sections/ decomposition (*PalettePage.helpers.ts extraction landed in PR #9616 during this audit) | maintainability | L | manual | `PalettePage.tsx`, `DebugPage.tsx`, `MapPage.tsx` | Extract sections/palette, sections/map, sections/debug with co-located tests | -| 24 | Flagship 3D/geo specs are the sparsest (6–7 impls, Python-only) and missing makie — the library that would shine there | maintainability | L | manual | `plots/scatter-3d/`, `surface-basic/`, 10 more | Targeted regen (makie, echarts, ggplot2) on the 3D cluster first; record legitimate per-library exclusions explicitly | -| 25 | 70% of the catalog lacks the newer libraries — 136 specs still Python-only (8 impls), only ~65 at 14–15 | maintainability | XL | manual | `plots/`, `core/constants.py` | Resume babysit-pipeline systematically; burn down the 8/9/10-impl buckets with targeted (not `all`) regen | +**Backend / API** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 1 | MCP `search_specs_by_tags` crashes (`AttributeError`) on impls with NULL `impl_tags` | correctness | S | `api/mcp/server.py` | +| 2 | Library seed uses `on_conflict_do_nothing` → version/description updates never reach prod DB | correctness | S | `automation/scripts/sync_to_postgres.py`, `core/constants.py`, `api/routers/libraries.py` | +| 3 | `DatabaseQueryError` reflects raw SQLAlchemy error text (SQL/table/DSN fragments) to clients | security | S | `api/routers/plots.py`, `api/exceptions.py` | +| 4 | Per-key cache locks leak when factory raises (404-probe traffic grows `_locks` unbounded) | correctness | S | `api/cache.py` | +| 5 | `get_all_with_code()` over-fetches multi-MB code blobs on 3 hot paths that don't need them | speed | M | `api/mcp/server.py`, `api/routers/libraries.py`, `core/database/repositories.py` | + +**Frontend / Design** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 6 | `DebugPage.tsx` is 1,903 lines with a single ~1,480-line component function (also Palette/Map/Stats pages) | maintainability | L | `app/src/pages/DebugPage.tsx` + 3 | +| 7 | `AppDataProvider` context values recreated every render → re-renders all consumers | speed | S | `app/src/layouts/Layout.tsx`, `useThemeMode.ts` | +| 8 | Hand-rolled dark mode bypasses MUI 9 `colorSchemes`/`cssVariables` | modern | L | `app/src/theme/palette.ts`, `create-theme.ts`, `useThemeMode.ts` | +| 9 | `react-helmet-async` used in 24 files despite React 19 native document metadata | modern | M | `app/src/routes/index.tsx`, `SpecPage.tsx` + | +| 10 | Hover-only reveal of card action buttons excludes keyboard + touch users (WCAG 2.4.7) | looks | S | `ImageCard.tsx`, `SpecOverview.tsx` | +| 11 | Off-palette hardcoded hues (Tailwind green-500, a rogue blue) drift from imprint palette | looks | S | `StatsPage.tsx`, `SpecOverview.tsx` | +| 12 | NavBar search button strips focus outline via `all:unset`; mobile nav tap targets < 44px | looks | S | `app/src/layouts/NavBar.tsx`, `ThemeToggle.tsx` | + +**Infra / CI** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 13 | Composite actions use floating tags (`setup-node@v4`, `r-lib@v2`, `julia@v2`) while all workflows pin to SHA | security | S | `.github/actions/setup-{node,r,julia}/action.yml` | +| 14 | Library-list drift: base generation prompts + docs still describe 9/11 libraries (no JS package guidance) | maintainability | M | `prompts/plot-generator.md`, `spec-create.yml`, `.github/copilot-instructions.md` | +| 15 | Untrusted issue title/body flow into privileged claude-code-action prompts (LLM injection surface) | security | M | `spec-create.yml`, `report-validate.yml` | +| 16 | Prod frontend builds on `node:20-alpine` (EOL Apr 2026) while CI tests Node 24, render harness pins 22 | modern | S | `app/Dockerfile`, `ci-tests.yml` | + +**DB** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 17 | `search_by_tags` casts JSONB→String + LIKE, defeating the `ix_specs_tags` GIN index; unescaped LIKE metachars | speed | M | `core/database/repositories.py` | +| 18 | Model↔migration drift: Impl check constraints exist only in the ORM, never in a migration | correctness | S | `core/database/models.py`, `alembic/versions/*` | + +**Security** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 19 | Known-vulnerable Python deps: pillow 12.2.0 (5 CVEs, server-side in request path), aiohttp 3.14.0 (8), click 8.3.1 | security | S | `uv.lock`, `core/images.py` | +| 20 | Feedback rate-limit + dup-suppression keyed on client-spoofable `X-Forwarded-For` first entry | security | S | `api/routers/feedback.py` | +| 21 | Frontend nginx serves SPA with no security headers (no CSP, X-Frame-Options, nosniff, HSTS) | security | S | `app/nginx.conf` | + +**Observability** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 22 | Background cache-refresh task can be GC'd mid-flight → permanently jams stale-while-revalidate for that key | correctness | S | `api/cache.py` | +| 23 | LLM calls discard all cost/latency/token telemetry (flagged 2026-05-29, still unfixed) | maintainability | M | `agentic/workflows/modules/agent.py`, `scripts/evaluate-plot.py` | +| 24 | `plausible.md` documents `potd_dismiss` + `PlotOfTheDay.tsx` events that no longer exist | maintainability | S | `docs/reference/plausible.md` | + +**LLM pipeline** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 25 | No prompt caching on the large static rubric prefix (~350-line evaluator re-billed per call, up to 8×) | speed | S | `scripts/evaluate-plot.py` | +| 26 | Workflow-prompt docs + daily similarity prompt still claim 9 libraries / Python-only | maintainability | S | `prompts/workflow-prompts/README.md`, `impl-similarity-claude.md` | + +**Quality / repo-health** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 27 | README still documents the removed SSE MCP transport as "recommended" (dead endpoint) | correctness | S | `README.md` | +| 28 | `core/palette.py` is the only core module with zero test coverage | maintainability | M | `core/palette.py`, `tests/unit/core/` | +| 29 | Repo-health gaps: no CODE_OF_CONDUCT.md, no PR template, no .editorconfig | maintainability | S | `.github/`, `docs/contributing.md` | + +**Agentic / docs** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 30 | `agent.py` writes run artifacts to `agentic/agentic/runs/` (wrong project-root depth) | correctness | S | `agentic/workflows/modules/agent.py`, `state.py` | +| 31 | Live repair prompt says "attempt {ATTEMPT}/3" but pipeline allows 4 attempts | correctness | S | `prompts/workflow-prompts/impl-repair-claude.md` | +| 32 | `project-guide.md` (loaded by `/prime-deep`) stale: library counts, repair limits, workflow inventory | maintainability | M | `agentic/docs/project-guide.md` | +| 33 | `/prime` misdescribes the product: "9 libraries", all Python, highcharts as Python | maintainability | S | `agentic/commands/prime.md`, `.github/copilot-instructions.md` | +| 34 | `.claude/settings.json` blanket-allows `Bash`/`Write`/`Edit`/`WebFetch`, neutering CLAUDE.md's gates | security | S | `.claude/settings.json` (use `/update-config` skill to edit) | + +**GitHub housekeeping** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 35 | Catalog/spec issue backlog: 17 spec trackers open up to 7 months with contradictory impl labels | maintainability | L | `gh:issues/653,789,1000,1010,...` | +| 36 | Dependabot PR backlog: 8 open, oldest 42 days, incl. tornado + aiohttp security-series bumps | security | S | `gh:pulls/8330,8820,8977,8979,...` | +| 37 | Stale feature-idea issues: 11 untouched 3–6.5 months with no triage signal | maintainability | M | `gh:issues/2842,2843,3175,...` | +| 38 | Stuck PR #8126 open 45 days with zero activity | maintainability | S | `gh:pulls/8126` | + +**SEO / PageSpeed** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 39 | pagespeed-auditor brief's "no auth needed" contract is factually wrong (429 for 6 audits) | maintainability | S | `agentic/commands/audit/pagespeed-auditor.md` | +| 40 | Bot-served homepage is thin: 10-char title, 39-char description, one-sentence body | looks | S | `api/routers/seo.py` | +| 41 | `Google-InspectionTool`/`GoogleOther` UAs fall through to SPA shell → Search Console shows wrong page | maintainability | S | `app/nginx.conf` | + +**Catalog substance** +| # | Finding | Dim | Effort | Files | +|---|---------|-----|--------|-------| +| 42 | Coverage cliff: 148/324 specs are Python-only; JS/R/Julia coverage 20–54%; only ~58 hit the 15-lib target | maintainability | XL | `plots/` | +| 43 | Majority of catalog renders were generated under library versions below current pyproject floors | maintainability | L | `plots/`, `pyproject.toml` | +| 44 | Tag vocabulary sprawl: 357 distinct tags, 195 singletons, near-duplicate concepts | maintainability | M | `plots/`, `docs/reference/tagging-system.md` | ## Low (Importance 2) -| # | Finding | Dim | Effort | Auto-fix | Files | Hint | -|---|---------|-----|--------|----------|-------|------| -| 1 | Tabs/TabPanel missing id/aria-controls wiring; click-to-collapse is nonstandard tabs semantics | correctness | S | manual | `SpecTabs/index.tsx`, `md.tsx` | Standard a11yProps helper | -| 2 | NavBar search button loses focus indicator via `all:unset`; active link lacks `aria-current` | correctness | S | manual | `NavBar.tsx` | `:focus-visible` outline + `aria-current="page"` | -| 3 | Motion/focus polish: `rise` + loader animations ignore `prefers-reduced-motion`; MapPage/DebugPage remove outlines without replacement | looks | S | manual | `tokens.css`, `LoaderSpinner.tsx`, `MapPage.tsx` | Extend the reduced-motion block; copy SpecDetailView's focus-visible pattern | -| 4 | Hero stacks three competing taglines; centered "from .md to art." breaks the left-aligned rhythm | looks | S | manual | `HeroSection.tsx` | Keep one emotional tagline (typewriter), left-align the rest | -| 5 | DatabaseQueryError reflects raw SQLAlchemy error text to API clients (contradicts the no-leak policy in the same file) | security | S | manual | `api/routers/plots.py`, `api/exceptions.py` | Log detail server-side; generic client message | -| 6 | Cloud SQL async fallback wraps a sync pg8000 engine in async_sessionmaker — would 500 on first query if ever configured | correctness | S | manual | `core/database/connection.py` | Fail fast with RuntimeError in that branch | -| 7 | Leftover `/hello/{name}` demo endpoint on the production API; three conflicting version strings | maintainability | S | manual | `api/routers/health.py` | Delete; source version from one constant | -| 8 | Previous-gen model default (`claude-sonnet-4-6`) + dead Python-highcharts pattern block in local evaluator | modern | S | manual | `core/config.py`, `scripts/evaluate-plot.py` | Bump to claude-sonnet-5 after side-by-side; derive patterns from constants | -| 9 | pagespeed-auditor brief's auth contract is factually wrong ("works without a key … plenty") — failed 5/5 runs | maintainability | S | manual | `agentic/commands/audit/pagespeed-auditor.md` | Rewrite: key required in practice; probe once, fail fast to blocked | -| 10 | python.anyplot.ai nginx server block + docs exist but DNS doesn't resolve — dead config invites drift | maintainability | S | manual | `app/nginx.conf`, `docs/reference/seo.md` | Finish the launch or delete the block; one-line decision comment | -| 11 | Cross-site links (now on /legal per maintainer decision): consider ONE contextual sentence on /about with a descriptive anchor for kurrentschrift; the rel handling (`noopener`, no `noreferrer`, followed) is right *(merged: design + seo; original footer-placement advice overtaken by events)* | modern | S | manual | `app/src/pages/AboutPage.tsx` | Optional follow-up; sitewide keyword-anchor links would look spammy — brand anchor is correct | -| 12 | Ghost event `library_filter` with unregistered `framework` prop, undocumented (9 events/30d, values invisible in dashboards) | maintainability | S | manual | `LibrariesPage.tsx`, `docs/reference/plausible.md` | Rename into taxonomy or register+document the prop | -| 13 | Outbound clicks double-tracked: auto "Outbound Link: Click" (33) overlaps custom `external_link` (12) | maintainability | S | manual | `app/index.html`, `docs/reference/plausible.md` | Pick one; document the survivor | -| 14 | `search_no_results` shows real content gaps (backtest equity curve, arrow/flow map) but 200ms debounce leaks keystrokes | maintainability | S | manual | `FilterBar.tsx` | File the two spec requests; fire on ~1s idle or blur/enter | -| 15 | plausible.md (updated 2026-04-25) + index.html drifted from the shipped product (missing nav_map, library_filter, /map, /debug; 9-library lists) | maintainability | S | manual | `docs/reference/plausible.md`, `app/index.html` | One doc-sync pass regenerating event tables from code | -| 16 | Git-tracked strays in plots/: two ~200KB render PNGs (gitignore matches but never untracked), leaked `_run_plotly.py`, 5 stale .gitkeep | maintainability | S | manual | `plots/bar-basic/...`, `plots/contour-3d/...` | `git rm --cached`; grep automation/ before deleting the runner | -| 17 | plot_type synonym drift: 6 singletons ('graph', 'curve', 'evaluation', …) where established tags exist (advisory facets policy-exempt, untouched) | maintainability | S | manual | 6 `specification.yaml` files | Normalize the synonyms only | -| 18 | 118 specs have `updated: null`; date quoting inconsistent across generations of tooling | maintainability | S | codemod | `plots/*/specification.yaml` | One-shot codemod: `updated := created` where null; normalize quoting | -| 19 | Duplicate-adjacent spec clusters (regression-linear/-polynomial/-lowess; sma/ema) read as variations of one page | maintainability | S | manual | `plots/scatter-regression-*/`, `plots/indicator-*ma/` | Cross-check traffic; consider merging weak siblings into variants | -| 20 | evaluate-plot.py: no prompt caching (image-first ordering defeats it), tight max_tokens, freeform JSON instead of structured output | modern | M | manual | `scripts/evaluate-plot.py`, `core/config.py` | Rubric-first + cache_control, ~4000 max_tokens, json_schema output format | -| 21 | `search_by_tags` matches via substring LIKE over stringified JSON — cross-category false positives, unindexable | correctness | M | manual | `core/database/repositories.py` | JSONB containment per category (`@>`) + optional GIN index | -| 22 | Ten sub-70 quality-score impls sit approved, 7/10 are pygal faking contours/maps/trees it has no primitives for | correctness | M | manual | `plots/*/metadata/python/pygal.yaml` ×7, altair ×2, seaborn ×1 | Regenerate with opus per the migration playbook, or allow explicit per-library opt-outs | -| 23 | useUrlSync bypasses React Router with raw `history.replaceState` — router location permanently stale on /plots, Back leaves the page | maintainability | M | manual | `useUrlSync.ts`, `useFilterState.ts` | Document as intentional or migrate to `useSearchParams(..., {replace:true})` | - -## Positive Patterns (Importance 1) - -- **Two-stage rubric + review-pipeline resilience** — 9-check auto-reject with false-positive guards, "correct but boring" cap, both-themes checklist, cascading review/repair. Reference this structure; don't redesign it (llm-pipeline). -- **Frontend engineering baseline** — green strict tsc on app+test configs, ~zero `any`, 80+ co-located test files, all-lazy routes with route-scoped error boundaries, typed API layer (frontend). -- **Token discipline & coherent terminal-editorial design language** — single CSS-var source, AA-checked ink scales, theme-adaptive previews, method-call microcopy (design). -- **Caching architecture** — per-key stampede locks with lifecycle-bound pruning, SWR with self-owned sessions, startup prewarm, deferred multi-MB columns, batched sync (backend). -- **Sitemap pipeline** — 324/324 specs, zero drift both directions, documented duplicate-content decisions (seo). -- **Analytics instrumentation** — 31-event taxonomy with journey props that answered "what do visitors actually do" directly from live data (plausible). -- **Metadata review layer** — 0 null quality_scores across 3,261 files, zero manual-merge bypasses detected, 100% consistent spec structure, all sampled previews live (catalog). +35 nit-level findings; the highest-signal clusters: +- **Config drift (dead settings):** `settings.cors_origins` is dead — CORS is hardcoded in `main.py` and the promised `www.anyplot.ai` origin is not actually allowed (backend + security both flagged); `claude_max_tokens` is dead config; two DB-env sources of truth. +- **Doc/comment drift:** German comments in `useAnalytics.ts` (violates English-only rule); `agent.py` timeout error hardcodes "5 minutes" vs the 30-min default (flagged by two auditors); retired model IDs (`o1`, `gpt-4o`, `gemini-2.0-flash`) in `core/config.py`; `docs/reference/api.md` omits `/feedback` and `/insights/visitors`; leaked `file:///home/meake/...` personal paths in a checked-in doc. +- **Frontend polish:** `useFilterState` animation `setTimeout`s never cleared; tsconfig missing `noUncheckedIndexedAccess`, ES2020 target stale; vitest coverage configured but no thresholds enforced. +- **Design a11y:** `prefers-reduced-motion` only covers 2 of ~6 animations; `theme-color` meta mismatches actual surfaces; a hero hover state renders white-on-green at ~3.4:1 (below AA). +- **DB:** tz-naive timestamps depend on server TZ being UTC; repos commit inside every write (no composable UoW); `IMPL_UPDATABLE_FIELDS` drift. +- **SEO:** `www` serves 200 duplicate content instead of redirecting; SearchAction `urlTemplate` feeds free-text into an exact-match filter; spec meta descriptions unclamped at 300+ chars. +- **Catalog:** stray `_run_plotly.py` scratch file committed in `contour-3d`. +- **Infra:** `impl-generate` interpolates dispatch inputs into `run:`/inline `python -c` (low risk, inconsistent with the repo's own env-routing standard); dead `ci-tests.yml` branch filters; stale "20 ticks/day" comment vs 10-tick cron. + +## Positive Patterns (Importance 1) — keep these +- **CI security posture (exemplary):** every third-party action across all 18 workflows SHA-pinned with version comments; explicit least-privilege `permissions:` blocks everywhere; untrusted event fields routed through `env:`; `author_association` gate on `@claude`; WIF instead of SA-key JSON; full Dependabot coverage. +- **Backend:** per-key stampede locks + stale-while-revalidate cache with self-session refresh factories; deferred large columns with explicit `undefer` variants; typed HTTP-client hygiene. +- **Frontend:** every route code-split via router `lazy()` with `HydrateFallback`; layered error boundaries with copyable diagnostics; typed API layer. +- **Design:** single CSS-var token source with `[data-theme]` flip, documented per-theme WCAG data, theme-adaptive previews. +- **DB:** 21-migration chain resolves to a single head, real downgrades everywhere, backfills done as data-migration steps. +- **Tests:** `tests/unit` mirrors the source tree almost 1:1 (no-db / with-db / cache-hit / not-found variants per router). +- **SEO plumbing:** sitemap.xml (3,595 URLs) in perfect sync with `plots/`; daily bot-serving regression check; rich prerendered bot pages for spec/impl. +- **Catalog integrity:** all ~3,240 impls have matching metadata (zero orphans bar one stray helper); all 324 specs have the four required sections. +- **Agentic:** roster-driven audit orchestration (`audit.workflow.js` never hardcodes the roster) and single-file prompt management. + +## Open-Issue Triage (github-auditor deep-dive — 47 open issues) + +**Categories:** 17 catalog/spec pipeline trackers · 12 fresh audit follow-ups (#9626–#9637, filed 2026-07-10) · 7 mid-term infra/enhancements · 11 stale ideas (untouched 3–6.5 months). + +**Recommended work order (top ~10):** +1. **#1010 `[heatmap-calendar] Basic Calendar Heatmap`** — first **stop the live retry loop** it is driving (Critical finding above), then decide regen vs close. +2. **#9627 Backend API follow-ups** (dead-weight endpoint, proxy caching, MCP URLs, error hygiene) — the only `bug`-labeled issue in the fresh batch; overlaps several backend Medium findings here. +3. **#8643 Align generation prompts with review rubric to cut the ~70% repair rate** — highest leverage on pipeline cost; open since 2026-06-10. +4. **#9626 Quality-score calibration** (first-review score, AR sentinel, worst-first regen, anchor) — pairs with #8643 and with the "verdict vs score contradiction" finding. +5. **#9632 Catalog backlog: coverage burn-down, light/dark stragglers, sub-70 regens** — umbrella for the 17 catalog trackers and the coverage-cliff finding. +6. **Dependabot batch (#8330/#8820/#8977/#8979/#9610/#9612/#9613/#9639)** — clear the 8-PR backlog; prioritize tornado (#8977) + aiohttp (#8979) as server-side runtime deps (ties to the pillow/aiohttp CVE finding). +7. **#9636 + the remaining #963x quick-win decision issues** — small, well-scoped, mergeable. +8. **#8126** — decide within one session: finish-and-rebase or close + delete branch (45 days stale). +9. **Label reconciliation for the 17 spec trackers** — strip contradictory `impl:{lib}:{done,pending,failed}` triples so the babysit-pipeline skill reads true state. +10. **Triage the 11 stale idea issues** — close-as-not-planned the superseded n8n / multi-LLM ideas; tag survivors `someday`. + +> Issue titles are quoted verbatim from GitHub; bodies were treated as untrusted data and not acted on. ## Statistics - -- Total: 57 actionable (+7 positive) | Critical: 1, High: 7, Medium: 25, Low: 23 (+1 medium became part of a merge) -- By Dimension: security 1, speed 7, looks 6, modern 7, correctness 15, maintainability 21 -- Effort: S 31, M 20, L 5, XL 1 -- Auto-fix: codemod 2, manual 55 -- By Auditor (raw): llm-pipeline 7, frontend 11, design 9, pagespeed 2, backend 11, seo 8, plausible 9, catalog 10 — 3 cross-auditor merges applied -- Cross-validation: 8 reviewed (all importance ≥4), 0 dropped, 0 downgraded — every high-severity finding survived an independent domain-expert refutation attempt -- Coverage: 3 full (frontend, plausible, catalog), 4 partial/structural (llm-pipeline, design, backend, seo), 1 blocked (pagespeed — anonymous PSI quota; recurring, see Medium #9) - -## Cross-auditor themes (synthesis) - -1. **"The 15-library expansion never propagated to secondary surfaces"** — found independently by four auditors: rubric prompts (llm-pipeline M12), index.html/OG/JSON-LD copy (design+seo M6), plausible.md + event docs (plausible L15), debug/insights library names (backend M5). One sweep with `core/constants.py` as the single source closes all of them. -- **Lab vs field Web Vitals divergence**: not computable — pagespeed blocked (5th time). Field-only verdict: interaction is fast (INP/CLS good), first paint is the weak spot, TTFB the lever. -- **Deprecation candidates**: catalog's duplicate clusters (regression trio, sma/ema) should be cross-checked against per-spec traffic before merging — per-spec Plausible breakdown wasn't pulled this run. +- Total: 109 | Critical: 1, High: 14, Medium: 45, Low: 35, Positive: 14 +- By Dimension: security 14, speed 8, looks 11, modern 7, correctness 26, maintainability 43 +- Effort (actionable, imp≥2): S 68, M 21, L 5, XL 1 +- Auto-fix (actionable, imp≥2): manual 94, codemod 1 (react-helmet migration), ruff/eslint/format 0 (baseline already clean) +- By Auditor: backend 11, frontend 9, design 9, infra 10, quality 7, llm 6, db 8, security 6, obs 7, agentic 8, gcloud 0 (blocked), github 8, plausible 0 (blocked), pagespeed 3, seo 10, catalog 7 +- Cross-validation: 18 reviewed, 0 dropped, 1 downgraded (all high-severity findings survived adversarial peer review) +- Coverage: 16 auditors ran, 4 degraded — **gcloud** blocked (not installed, 0 findings), **plausible** blocked (no API key, 0 findings), **pagespeed** blocked (429 quota, 3 salvaged findings), **seo** structural-only (10 findings). The other 12 ran full/partial with findings. Live GCP/Plausible/lab-CWV posture is therefore **unverified this run** — grades for those surfaces reflect code-side evidence only. + +--- +*Generated by `/audit` (workflow engine, 16 auditors + domain-routed cross-validation). Health Score and Dimension Scorecard use the standard formula for trend comparability across audit reports.*