Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ bin/gstack-global-discover*
.opencode/
.slate/
.cursor/
.grok/
.openclaw/
.hermes/
.gbrain/
Expand Down
49 changes: 49 additions & 0 deletions TODOS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,55 @@

## NEXT PRIORITY

### P2: Grok Build full-compat follow-ups (after `feat/grok-build-host`)

**What:** Finish / harden the Grok-native packaging work landed on
`feat/grok-build-host` (runtime root, SETUP path fix, host-bleed cleanup,
bridges, `bin/gstack-grok-compat-audit`, section pointers for token ceiling).
These are the remaining product/ship items — not blockers for local Grok use.

**Context (landed on branch):**
- U1–U3 packaging + U4–U7 bridges + U9 audit gate
- Setup host-scoping (no always-on Codex `.agents/` regen / Claude plan-tune
prompt on `--host grok-build`)
- Option A: Grok carved skills use STOP-Read section pointers
(`~/.grok/skills/gstack-*/sections/`) — `/ship` always-loaded ~20k tokens
(was ~45k)

**Follow-ups to track:**

1. **Upstream PR** — open/update PR against `garrytan/gstack` for
`feat/grok-build-host` (VERSION/CHANGELOG/title deferred until local DoD
green on main merge path).
2. **U4 behavioral smoke** — if claiming `/spec --execute` COMPATIBLE (not
file-only), record multi-turn tool-use / cwd isolation / fail-closed auth
smoke or fixture; flag-string greps alone are not enough.
3. **Section pointers for other external hosts (optional)** — Codex ship is
still monolith ~39k (near soft ceiling). Reuse
`hostUsesSectionPointers` for Codex/Cursor/Factory when package paths are
portable (Codex: `.agents/skills/gstack-*` or global
`~/.codex/skills/gstack-*`).
4. **R10 / U8 optional `gstack-codex`** — only if product wants suite
completeness; keep bare `/codex` = OpenAI plugin; deny bare `codex` alias
on Grok install (partial deny already in `link_grok_skill_dirs`).
5. **Gbrain Grok MCP stretch** — optional `~/.grok/config.toml` gbrain stanza
(absolute path, TOML RMW, atomic write + backup; non-interactive refuse
overwrite without `--force`). Success path remains CLI + AGENTS.md.
6. **Audit harness productization** — keep `bin/gstack-grok-compat-audit` as
long-lived multi-host CI later, or document as branch/DoD gate only
(open question from plan ce-doc-review).
7. **Name collision note** — document `/review` vs compound-engineering
`code-review` routing in `~/.grok/AGENTS.md` (do not delete CE skill).

**Why:** Branch is implementation-ready for local Grok; these items close
honesty gaps, upstream ship, and optional suite completeness without
reopening Phase A packaging.

**Depends on / blocked by:** Nothing for local use. Upstream PR waits on
clean validation + VERSION/CHANGELOG if that is the release process.

---

### P1: #1882 — portable skill-install prefix (non-`gstack` install dirs break silently)

**What:** Every generated SKILL.md hardcodes the literal `~/.claude/skills/gstack/...`
Expand Down
21 changes: 12 additions & 9 deletions benchmark-models/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@ allowed-tools:
## When to invoke this skill

Runs the same prompt through Claude,
GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost,
and optionally quality via LLM judge. Answers "which model is actually best
for this skill?" with data instead of vibes. Separate from /benchmark, which
measures web page performance. Use when: "benchmark models", "compare models",
"which model is best for X", "cross-model comparison", "model shootout".
GPT (via Codex CLI), Gemini, and Grok side-by-side — compares latency, tokens,
cost, and optionally quality via LLM judge. Answers "which model is actually
best for this skill?" with data instead of vibes. Separate from /benchmark,
which measures web page performance. Use when: "benchmark models", "compare
models", "which model is best for X", "cross-model comparison", "model
shootout".

Voice triggers (speech-to-text aliases): "compare models", "model shootout", "which model is best".

Expand Down Expand Up @@ -578,20 +579,21 @@ If C: ask for the path. Verify it exists. Use as positional argument.
## Step 2: Choose providers

```bash
"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini --dry-run
"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini,grok --dry-run
```

Show the dry-run output. The "Adapter availability" section tells the user which providers will actually run (OK) vs skip (NOT READY — remediation hint included).

If ALL three show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest `claude login`, `codex login`, or `gemini login` / `export GOOGLE_API_KEY`.
If ALL show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest `claude login`, `codex login`, `gemini login` / `export GOOGLE_API_KEY`, or `grok` login / `export XAI_API_KEY`.

If at least one is OK: AskUserQuestion:
- **Simplify:** "Which models should we include? The dry-run above showed which are authed. Unauthed ones will be skipped cleanly — they won't abort the batch."
- **RECOMMENDATION:** A (all authed providers) because running as many as possible gives the richest comparison.
- **Options:**
- A) All authed providers. Completeness: 10/10.
- B) Only Claude. Completeness: 6/10 (no cross-model signal — use /ship's review for solo claude benchmarks instead).
- C) Pick two — specify on next turn. Completeness: 8/10.
- C) Only Grok (Grok-only machine). Completeness: 7/10.
- D) Pick two — specify on next turn. Completeness: 8/10.

---

Expand All @@ -608,7 +610,8 @@ If judge is available, AskUserQuestion:
- A) Enable judge (adds ~$0.05). Completeness: 10/10.
- B) Skip judge — speed/cost/tokens only. Completeness: 7/10.

If judge is NOT available, skip this question and omit the `--judge` flag.
If judge is NOT available (Grok-only / no ANTHROPIC_API_KEY): skip this question
and omit `--judge` (Grok-or-skip). Speed/cost/tokens still work.

---

Expand Down
21 changes: 12 additions & 9 deletions benchmark-models/SKILL.md.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,12 @@ preamble-tier: 1
version: 1.0.0
description: |
Cross-model benchmark for gstack skills. Runs the same prompt through Claude,
GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost,
and optionally quality via LLM judge. Answers "which model is actually best
for this skill?" with data instead of vibes. Separate from /benchmark, which
measures web page performance. Use when: "benchmark models", "compare models",
"which model is best for X", "cross-model comparison", "model shootout". (gstack)
GPT (via Codex CLI), Gemini, and Grok side-by-side — compares latency, tokens,
cost, and optionally quality via LLM judge. Answers "which model is actually
best for this skill?" with data instead of vibes. Separate from /benchmark,
which measures web page performance. Use when: "benchmark models", "compare
models", "which model is best for X", "cross-model comparison", "model
shootout". (gstack)
voice-triggers:
- "compare models"
- "model shootout"
Expand Down Expand Up @@ -69,20 +70,21 @@ If C: ask for the path. Verify it exists. Use as positional argument.
## Step 2: Choose providers

```bash
"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini --dry-run
"$BIN" --prompt "unused, dry-run" --models claude,gpt,gemini,grok --dry-run
```

Show the dry-run output. The "Adapter availability" section tells the user which providers will actually run (OK) vs skip (NOT READY — remediation hint included).

If ALL three show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest `claude login`, `codex login`, or `gemini login` / `export GOOGLE_API_KEY`.
If ALL show NOT READY: stop with a clear message — benchmark can't run without at least one authed provider. Suggest `claude login`, `codex login`, `gemini login` / `export GOOGLE_API_KEY`, or `grok` login / `export XAI_API_KEY`.

If at least one is OK: AskUserQuestion:
- **Simplify:** "Which models should we include? The dry-run above showed which are authed. Unauthed ones will be skipped cleanly — they won't abort the batch."
- **RECOMMENDATION:** A (all authed providers) because running as many as possible gives the richest comparison.
- **Options:**
- A) All authed providers. Completeness: 10/10.
- B) Only Claude. Completeness: 6/10 (no cross-model signal — use /ship's review for solo claude benchmarks instead).
- C) Pick two — specify on next turn. Completeness: 8/10.
- C) Only Grok (Grok-only machine). Completeness: 7/10.
- D) Pick two — specify on next turn. Completeness: 8/10.

---

Expand All @@ -99,7 +101,8 @@ If judge is available, AskUserQuestion:
- A) Enable judge (adds ~$0.05). Completeness: 10/10.
- B) Skip judge — speed/cost/tokens only. Completeness: 7/10.

If judge is NOT available, skip this question and omit the `--judge` flag.
If judge is NOT available (Grok-only / no ANTHROPIC_API_KEY): skip this question
and omit `--judge` (Grok-or-skip). Speed/cost/tokens still work.

---

Expand Down
Loading