Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- **`10x bench-kit` is generally available.** The `TENX_CLI_EXPERIMENTAL`
opt-in is gone: `init` and `update` are registered unconditionally, appear
in `--help`, and are documented in the README. The `experimental_locked`
error envelope no longer exists (the gating module has been removed).

### Added

- **`10x sync` — bulk download & update with change visibility.** One command to
Expand Down
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Once installed, just tell your agent to **set up 10x-cli** and it will pick up t
| `10x sync` | Bulk-download / refresh lessons and report what changed upstream |
| `10x doctor` | Diagnose auth, API connectivity, and local config |
| `10x bench` | Live top-10 AI model leaderboard from [10xbench.ai](https://10xbench.ai) — no login needed |
| `10x bench-kit <action>` | Create (`init`) and update (`update`) a company benchmark instance from the [10x-bench-kit](https://github.com/przeprogramowani/10x-bench-kit) template |

### `10x get` Flags

Expand Down Expand Up @@ -180,6 +181,40 @@ color-coded score bars in your terminal. Public data, works without logging in.
The data refreshes whenever new benchmark results are published on 10xbench.ai.
Colors honor `NO_COLOR` and are disabled automatically when output is piped.

### `10x bench-kit`

Creates and maintains a **company benchmark instance** from the
[10x-bench-kit](https://github.com/przeprogramowani/10x-bench-kit) template —
a self-hosted benchmark that scores AI agents on tasks embedded in your own
repositories. The CLI is a thin installer/updater; tasks, assertions, and
scoring live in the template and the instance.

| Action | Description |
|--------|-------------|
| `init [dir]` | Materialize a fresh instance from the template (no git history, fresh `git init`), register the detected base repo, install runner dependencies |
| `update [dir]` | Upgrade the instance to a newer template: runtime zone replaced wholesale, skills proposed as a reviewable diff, company content untouched |

| Flag | Description |
|------|-------------|
| `--template-version <tag>` | Template tag to install (default: latest) |
| `--tool <id>` | Agent tool for skill placement (`claude-code`, `cursor`, `copilot`, `codex`, `windsurf`, `gemini`, `generic`) |
| `--yes` | Run non-interactively, accepting defaults |

```bash
# Create an instance next to your product repo (run inside it to auto-register)
10x bench-kit init my-benchmark

# Pin the template version
10x bench-kit init my-benchmark --template-version v0.8.0

# Upgrade an existing instance to the latest template
10x bench-kit update
```

Re-running `init` on an existing instance is a **repair**: missing template
files are restored, company content is never touched. After `update`, run the
instance's `bench validate` to confirm the content still matches the new kit.

### Global Flags

- `--json` — Machine-readable JSON output (auto-detected when piped)
Expand Down
11 changes: 1 addition & 10 deletions src/commands/bench-kit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ import {
import { tmpdir } from "node:os";
import { basename, join, resolve, sep } from "node:path";
import type { CAC } from "cac";
import { experimentalEnabled, requireExperimental } from "../lib/experimental";
import {
ExitCodes,
type GlobalFlags,
Expand Down Expand Up @@ -121,23 +120,15 @@ export interface BenchKitDeps {
// bench-kit follows the `auth` precedent: one command dispatching on an
// action argument.
export function registerBenchKitCommand(cli: CAC): void {
// Experimental commands are all-or-nothing: without the opt-in the
// command is not registered at all — absent from help and behaving like
// any unknown command — instead of showing up half-locked.
if (!experimentalEnabled()) return;
cli
.command(
"bench-kit <action> [dir]",
"Manage a benchmark instance (actions: init, update; experimental)",
)
.command("bench-kit <action> [dir]", "Manage a benchmark instance (actions: init, update)")
.option("--template-version <tag>", "Template tag to install (default: latest)")
.option("--tool <id>", `Agent tool for skill placement (${Object.keys(PROFILES).join(", ")})`)
.option("--yes", "Run non-interactively, accepting defaults")
.example("10x bench-kit init my-benchmark")
.example("10x bench-kit init my-benchmark --template-version v0.1.0")
.example("10x bench-kit update")
.action(async (action: string, dir: string | undefined, options: BenchKitFlags) => {
requireExperimental(`bench-kit ${action}`, options);
const ctx = resolveContext(options);
if (action === "init") {
await runBenchKitInit(ctx, dir, options);
Expand Down
39 changes: 0 additions & 39 deletions src/lib/experimental.ts

This file was deleted.

51 changes: 2 additions & 49 deletions tests/bench-kit-command.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ import {
runBenchKitUpdate,
toHttpsUrl,
} from "../src/commands/bench-kit";
import { EXPERIMENTAL_ENV, experimentalEnabled } from "../src/lib/experimental";
import type { OutputContext } from "../src/lib/output";

interface CaptureResult {
Expand Down Expand Up @@ -725,58 +724,12 @@ describe("toHttpsUrl", () => {
});
});

describe("experimental gate", () => {
const savedEnv = process.env[EXPERIMENTAL_ENV];

afterEach(() => {
if (savedEnv === undefined) {
delete process.env[EXPERIMENTAL_ENV];
} else {
process.env[EXPERIMENTAL_ENV] = savedEnv;
}
});

it("is off by default and accepts 1 / true", () => {
expect(experimentalEnabled({})).toBe(false);
expect(experimentalEnabled({ [EXPERIMENTAL_ENV]: "0" })).toBe(false);
expect(experimentalEnabled({ [EXPERIMENTAL_ENV]: "1" })).toBe(true);
expect(experimentalEnabled({ [EXPERIMENTAL_ENV]: "true" })).toBe(true);
});

it("keeps bench-kit fully hidden without the opt-in", async () => {
delete process.env[EXPERIMENTAL_ENV];
const cli = cac("10x");
registerBenchKitCommand(cli);
expect(cli.commands.map((c) => c.name)).not.toContain("bench-kit");

// Invoking it behaves like any unknown command: no output, no error.
const result = await runCli(["bench-kit", "init", "some-dir", "--json"]);
expect(result.exitCode).toBeUndefined();
expect(result.stdout).toBe("");
});

it("registers bench-kit when the opt-in is set", () => {
process.env[EXPERIMENTAL_ENV] = "1";
describe("10x bench-kit dispatch", () => {
it("registers bench-kit unconditionally", () => {
const cli = cac("10x");
registerBenchKitCommand(cli);
expect(cli.commands.map((c) => c.name)).toContain("bench-kit");
});
});

describe("10x bench-kit dispatch", () => {
const savedEnv = process.env[EXPERIMENTAL_ENV];

beforeEach(() => {
process.env[EXPERIMENTAL_ENV] = "1";
});

afterEach(() => {
if (savedEnv === undefined) {
delete process.env[EXPERIMENTAL_ENV];
} else {
process.env[EXPERIMENTAL_ENV] = savedEnv;
}
});

it("routes 'update' to the real implementation", async () => {
// A temp dir that is not an instance — proves dispatch reaches update.
Expand Down