Skip to content

docs(bench): re-run every row of the llms-benchmark under oddyssey 1.13.0 #636

Description

@using-system

Problem

Every row of .llms-benchmark/README.md was measured under oddyssey 1.12.0. Release 1.13.0 (e7fd9fa) changed what a run reads and runs: one observation depth (#620 - the mission no longer names a depth), environment-scoped stack_config and the persisted grafana context in the MCP server (#621, #622), and the two-runs-per-model protocol of launch-llms-benchmark (#634). Rows taken under 1.12.0 are not comparable with a row taken under 1.13.0.

What to do

Re-run the whole table under 1.13.0 with the launch-llms-benchmark protocol as it stands at e7fd9fa - every model on its CLI, two runs each, the better run in the table - on one branch, and propose the re-sorted ranking in one PR.

Rows (14): z-ai/glm-5.3, deepseek/deepseek-v4.1-flash, z-ai/glm-5.3-flashx, google/gemini-3.7-flash, google/gemini-3.8-flash, z-ai/glm-5.3-flash, qwen/qwen3.8-27b, qwen/qwen3.8-max-0902 on opencode; openai/gpt-5.6-luna, openai/gpt-5.6-terra, openai/gpt-5.6-sol on copilot; anthropic/claude-opus-5, anthropic/claude-fable-5.1, anthropic/claude-sonnet-5 on claude.

A model that fails a run moves to the end of the queue and is replayed; a model that fails twice keeps its 1.12.0 row marked provisional, with the failure recorded in the PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationlocalLocal otel-lgtm stackpriority: mediumReal friction, workaround existsrelease: v1.13.1Shipped in release v1.13.1

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions