Kilo auto efficient

README.md

@@ -194,6 +194,160 @@ The script loads your prediction file, makes API calls using the models specifie
 > [!NOTE]
 > - For robustness evaluation, we only measure the model-selection flip ratio after adding noise to the original prompt, so no additional LLM inference is required for this stage.
 
+### Kilo Auto Efficient: combined routing and generation
+
+The standalone driver uses Kilo's public OpenAI-compatible endpoint with
+`model: "kilo-auto/efficient"`. It does not train or tune on RouterArena.
+Prepare the official datasets from the repository root first:
+
+```bash
+python -m pip install 'datasets>=4.0.0' 'pandas>=2.3.2'
+python scripts/process_datasets/prep_datasets.py
+```
+
+The preparation script downloads RouterArena and LiveCodeBench data. The driver
+itself uses only Python's standard library and this repository's model registry.
+Set `KILO_API_KEY` securely in the environment; do not put it in a command, config,
+or prediction file. To bill authorized organization credits, optionally set
+`KILO_ORGANIZATION_ID` securely in the environment; it is sent only as the
+`x-kilocode-organizationid` request header, not recorded in submission metadata.
+Set a unique `KILO_RUN_ID` for each new run.
+Keep that identifier unchanged when you resume or retry the run.
+Start each new run in a workspace with no previous predictions or checkpoints.
+Set the endpoint through the environment:
+
+```bash
+export KILO_API_URL=https://api.kilo.ai/api/gateway/chat/completions
+export KILO_RUN_ID=routerarena-20261008-a
+python scripts/submit_kilo_auto_efficient.py full --concurrency 4
+python scripts/submit_kilo_auto_efficient.py robustness --concurrency 4
+```
+
+To ramp concurrency, use timed outcome windows:
+
+```bash
+python scripts/submit_kilo_auto_efficient.py full --concurrency 16 --adaptive-concurrency --order-seed 841
+```
+
+The order seed changes scheduling only, not the generation seed.
+Supply the same order seed on every resume and provider retry.
+The ramp doubles concurrency after healthy 45-second windows with at least eight outcomes.
+It reduces concurrency when provider errors or successful fallback responses exceed its thresholds.
+After a transient error or fallback limit, two error-free and fallback-free windows restart the ramp.
+Two consecutive low-gain windows stop the ramp at the preceding concurrency.
+Slow requests remain active across windows; they do not block the ramp.
+The final `adaptive:` report identifies the concurrency to retain on resume.
+
+Each request sends the prepared prompt unchanged as one user message.
+Streaming transport avoids the gateway's non-streaming header timeout.
+The driver does not override temperature, token limits, seeds, or reasoning settings.
+Routing decisions use the response's concrete `model`, not the router alias.
+Unknown or out-of-pool models stop the run; the driver does not guess aliases.
+Full submissions contain the returned answer and authoritative token usage.
+Each query has a fresh run-scoped session.
+The session stays fixed across retries and does not reuse a previous run's classifier cache.
+
+The gateway's Vercel provider returns Alibaba IDs for Qwen models.
+The registry uses these exact naming maps:
+
+- `alibaba/qwen3.8-flash` maps to `qwen/qwen3.8-flash`.
+- `alibaba/qwen3.8-max-0902` maps to `qwen/qwen3.8-max-0902`.
+- `zai/glm-5.3` maps to `z-ai/glm-5.3`.
+- `zai/glm-5.3-flash` maps to `z-ai/glm-5.3-flash`.
+
+The maps preserve the exact model suffix.
+Generation metadata and the raw journal preserve the original response ID.
+
+The current production table selects reasoning variants.
+The driver preserves those server choices without changing generation settings or reusing cached base-model answers.
+The response exposes the base model ID.
+RouterArena's model-only robustness score does not distinguish reasoning variants.
+
+Robustness streams close after the first concrete model appears.
+Partial inference can still incur costs; the submission omits `generated_result`.
+Completion duration is not routing-only latency.
+The stock evaluator excludes classifier overhead, robustness calls, and discarded retry attempts.
+
+Provider errors, empty answers, and missing usable usage remain `success: false` rows.
+The driver records each failure and does not omit the row.
+If final usage is missing or unusable, `token_usage` stays null.
+The driver does not invent tokens; the stock evaluator cannot account for those costs.
+
+Include the gateway's successful static fallback responses without retrying them.
+The current fallback is `z-ai/glm-5.3-flash`.
+The gateway uses it when its routing request fails or exceeds five seconds.
+The configured submission pool includes this model, although the ranked production pool does not.
+
+An older checkpoint can add only its declared static fallback to the accepted model names.
+The driver preserves the original journal header and every captured response.
+
+Raw completions, stream events, and early selection chunks are checkpointed in
+`cached_results/kilo-auto-efficient/<split>.jsonl`. Rerun the same command to
+resume without repeating checkpointed calls. Do not edit the dataset or pool
+during a run. A request without a valid model selection stops new work.
+In-flight calls are drained and checkpointed. Preserve the journal with
+the submission for auditing. A forced process/VM termination can lose an
+in-flight response and therefore cause another paid request on resume.
+
+An initial HTTP 429/500/502/503/504 transport error remains an unfinished query with an audited attempt.
+The driver does not invent a selected model, answer, or token usage.
+Nonadaptive batches drain active calls and return exit code 75.
+Adaptive mode reduces concurrency and continues unrelated queries.
+Resume unfinished queries at the reported concurrency.
+Other transport errors require investigation before the driver can resume.
+
+After the full first pass, explicitly retry provider errors:
+
+```bash
+python scripts/submit_kilo_auto_efficient.py full --retry-provider-errors --concurrency 4
+```
+
+Each pass makes one additional attempt per eligible query.
+Repeat provider-only passes until those requests succeed.
+Explicit provider timeouts and HTTP 429/500/502/503/504 errors qualify.
+The provider can report these codes as numbers, decimal strings, or a `statusCode` field.
+Serialized `AI_InvalidResponseDataError` stream failures also qualify when no contradictory HTTP status exists.
+Empty answers and missing usage alone do not qualify.
+Correctness scores are never read.
+Successful generations are never retried, even when their answers are incorrect.
+Every explicit retry remains in the journal, including transport errors.
+Final rows retain the first success or the last captured generation failure.
+
+Only explicit operator approval permits incomplete-response exceptions.
+Pass `--retry-approved-indices` with the exact approved global indices.
+
+The exception applies only to failed empty-answer or missing-usage rows at the listed indices.
+Successful answers and other failure types remain locked.
+Repeat the same approved indices on every subsequent full checkpoint replay.
+The journal records the retry basis and preserves each original response.
+Disclose any approved exceptions in the submission config.
+No prompt, generation setting, or correctness-based selection changes.
+
+Use `--limit 1 --concurrency 1` for a one-request smoke run; it produces an
+explicitly incomplete file and that response is reused on the subsequent full
+run. Exported JSON is rebuilt from the journal on resume, so evaluate only after
+generation finishes. Do not run the stock LLM inference script afterward:
+it would bypass Kilo's router and generate again.
+
+Registered concrete models also use `ModelInference`'s Kilo provider.
+For separate model inference, set `KILO_API_KEY` and optionally `KILO_ORGANIZATION_ID`.
+That path calls the concrete model without Auto Efficient's selected reasoning settings.
+This submission does not use that path.
+
+Validate the completed files:
+
+```bash
+python router_inference/check_config_prediction_files.py kilo-auto-efficient full --check-generated-result
+python router_inference/check_config_prediction_files.py kilo-auto-efficient robustness
+```
+
+Official submission requires 8,400 full rows and 420 robustness rows. Local
+evaluation follows Step 4 below; it needs the evaluation dependencies and ground
+truth, and LiveCodeBench executes generated code, so use an isolated environment.
+
+Use `--num-workers 1` if `filelock` reports an unsafe fork during parallel grading.
+Regrade the unchanged answers; do not regenerate them.
+
 ## 4. Run Router Evaluation
 
 As the last step, run the evaluation script:

llm_inference/model_inference.py

@@ -83,6 +83,8 @@ def infer(
                     return self._call_perplexity(model_name, prompt)
                 elif provider == "openrouter":
                     return self._call_openrouter(model_name, prompt)
+                elif provider == "kilo":
+                    return self._call_kilo(model_name, prompt)
                 elif provider == "replicate":
                     return self._call_replicate(model_name, prompt)
                 elif provider == "aws":
@@ -198,6 +200,17 @@ def _get_provider(self, model_name: str) -> str:
             "mistralai/devstral-2512:free": "openrouter",
             "meta-llama/llama-3.3-70b-instruct": "openrouter",
             "meta-llama/llama-3.1-405b-instruct": "openrouter",
+            # Kilo Auto Efficient's concrete production pool
+            "anthropic/claude-haiku-5.5": "kilo",
+            "deepseek/deepseek-v4.1-flash": "kilo",
+            "meta/muse-spark-1.3-contributor": "kilo",
+            "moonshotai/kimi-k3": "kilo",
+            "openai/gpt-6-luna": "kilo",
+            "qwen/qwen3.8-flash": "kilo",
+            "qwen/qwen3.8-max-0902": "kilo",
+            "xiaomi/mimo-v2.6-pro": "kilo",
+            "z-ai/glm-5.3": "kilo",
+            "z-ai/glm-5.3-flash": "kilo",
             # Replicate
             "meta/codellama-34b-instruct": "replicate",
             # AWS Bedrock
@@ -313,6 +326,73 @@ def _call_replicate(self, model_name: str, prompt: str) -> Dict[str, Any]:
             "provider": "replicate",
         }
 
+    def _call_kilo(self, model_name: str, prompt: str) -> Dict[str, Any]:
+        """Generate with a concrete Kilo model, without invoking the router."""
+        from universal_model_names import ModelNameManager
+
+        api_key = os.getenv("KILO_API_KEY")
+        if not api_key:
+            raise ValueError("KILO_API_KEY is required for Kilo model inference")
+        headers = {}
+        organization_id = os.getenv("KILO_ORGANIZATION_ID")
+        if organization_id:
+            headers["x-kilocode-organizationid"] = organization_id
+        content = []
+        usage = None
+        served_model = None
+        finish_reason = None
+        with OpenAI(
+            base_url="https://api.kilo.ai/api/gateway",
+            api_key=api_key,
+            timeout=600,
+            max_retries=0,
+        ) as client:
+            stream = client.chat.completions.create(
+                model=model_name,
+                messages=[{"role": "user", "content": prompt}],
+                stream=True,
+                stream_options={"include_usage": True},
+                extra_headers=headers,
+            )
+            with stream:
+                for chunk in stream:
+                    if chunk.model:
+                        served_model = chunk.model
+                    if chunk.usage is not None:
+                        usage = chunk.usage
+                    for choice in chunk.choices:
+                        if choice.index != 0:
+                            continue
+                        if choice.delta.content is not None:
+                            content.append(choice.delta.content)
+                        if choice.finish_reason is not None:
+                            finish_reason = choice.finish_reason
+        if not served_model:
+            raise ValueError("Kilo stream did not identify the served model")
+        answer = "".join(content)
+        success = (
+            bool(answer.strip()) and usage is not None and usage.completion_tokens > 0
+        )
+        return {
+            "response": answer,
+            "success": success,
+            "token_usage": {
+                "input_tokens": usage.prompt_tokens,
+                "output_tokens": usage.completion_tokens,
+                "total_tokens": usage.total_tokens,
+            }
+            if usage is not None
+            else None,
+            "model_used": ModelNameManager.get_universal_name(served_model),
+            "response_model": served_model,
+            "usage_raw": usage.model_dump() if usage is not None else None,
+            "finish_reason": finish_reason,
+            "provider": "kilo",
+            "error": None
+            if success
+            else "Provider omitted an answer or usable token accounting",
+        }
+
     def _call_openrouter(self, model_name: str, prompt: str) -> Dict[str, Any]:
         """Call OpenRouter API."""
         openrouter_api_key = os.getenv("OPENROUTER_API_KEY")

model_cost/model_cost.json

@@ -383,6 +383,46 @@
     "input_token_price_per_million": 1.25,
     "output_token_price_per_million": 10.0
   },
+  "anthropic/claude-haiku-5.5": {
+    "input_token_price_per_million": 0.1,
+    "output_token_price_per_million": 0.5
+  },
+  "deepseek/deepseek-v4.1-flash": {
+    "input_token_price_per_million": 0.3,
+    "output_token_price_per_million": 1.2
+  },
+  "meta/muse-spark-1.3-contributor": {
+    "input_token_price_per_million": 0.1,
+    "output_token_price_per_million": 0.2
+  },
+  "openai/gpt-6-luna": {
+    "input_token_price_per_million": 0.1,
+    "output_token_price_per_million": 0.5
+  },
+  "qwen/qwen3.8-flash": {
+    "input_token_price_per_million": 0.15,
+    "output_token_price_per_million": 0.47
+  },
+  "qwen/qwen3.8-max-0902": {
+    "input_token_price_per_million": 2.0,
+    "output_token_price_per_million": 6.0
+  },
+  "moonshotai/kimi-k3": {
+    "input_token_price_per_million": 3.0,
+    "output_token_price_per_million": 15.0
+  },
+  "z-ai/glm-5.3": {
+    "input_token_price_per_million": 1.4,
+    "output_token_price_per_million": 4.4
+  },
+  "xiaomi/mimo-v2.6-pro": {
+    "input_token_price_per_million": 0.43,
+    "output_token_price_per_million": 0.87
+  },
+  "z-ai/glm-5.3-flash": {
+    "input_token_price_per_million": 0.15,
+    "output_token_price_per_million": 0.5
+  },
   "google/gemma-4-31b-it": {
     "input_token_price_per_million": 0.08,
     "output_token_price_per_million": 0.35

router_inference/config/kilo-auto-efficient.json

@@ -0,0 +1,54 @@
+{
+  "pipeline_params": {
+    "router_name": "kilo-auto-efficient",
+    "models": [
+      "anthropic/claude-haiku-5.5",
+      "deepseek/deepseek-v4.1-flash",
+      "meta/muse-spark-1.3-contributor",
+      "moonshotai/kimi-k3",
+      "openai/gpt-6-luna",
+      "qwen/qwen3.8-flash",
+      "qwen/qwen3.8-max-0902",
+      "xiaomi/mimo-v2.6-pro",
+      "z-ai/glm-5.3",
+      "z-ai/glm-5.3-flash"
+    ],
+    "gateway_model": "kilo-auto/efficient",
+    "model_pool_source": "https://api.kilo.ai/api/gateway/models",
+    "model_pool_snapshot_date": "2026-10-08",
+    "pricing_source": "https://api.kilo.ai/api/gateway/models",
+    "pricing_snapshot_date": "2026-10-08",
+    "description": "Kilo Auto Efficient, queried through the production gateway without RouterArena training or tuning. The pool preserves the gateway's provider-prefixed model IDs; predictions must record the selected model rather than substitute a cached model. Generate submissions with the standalone driver, not the stock router-class generator.",
+    "routing_table_version": "registry-cdf46fd6-variants",
+    "classifier_model": "~typesafe/jev-latest",
+    "fallback_model": "z-ai/glm-5.3-flash",
+    "fallback_policy": "Include the gateway's successful fallback responses unchanged. A routing timeout is not permission to retry a successful generation. The gateway falls back after its 5-second routing deadline; preserve the actual served model and answer.",
+    "production_model_pool": [
+      "~anthropic/claude-haiku-latest",
+      "deepseek/deepseek-v4.1-flash",
+      "~moonshotai/kimi-latest",
+      "~openai/gpt-luna-latest",
+      "~z-ai/glm-latest",
+      "meta/muse-spark-1.3-contributor",
+      "qwen/qwen3.8-flash",
+      "qwen/qwen3.8-max-0902",
+      "xiaomi/mimo-v2.6-pro"
+    ],
+    "alias_targets_observed": {
+      "~anthropic/claude-haiku-latest": "anthropic/claude-haiku-5.5",
+      "~openai/gpt-luna-latest": "openai/gpt-6-luna",
+      "~moonshotai/kimi-latest": "moonshotai/kimi-k3",
+      "~z-ai/glm-latest": "z-ai/glm-5.3"
+    },
+    "retry_policy": "Retry explicit provider timeouts, HTTP 429/500/502/503/504 reported through code or statusCode, and serialized AI_InvalidResponseDataError upstream stream failures without a contradictory HTTP status. Numeric and decimal-string status codes are equivalent. Keep the first successful generation, including incorrect answers. Do not retry empty answers or missing usage without provider-error evidence or separately recorded operator approval. Preserve prompts, production settings, and every captured attempt. Do not consult correctness scores during retries.",
+    "evaluation_limitations": [
+      "RouterArena computes generation cost from generated_result.token_usage at the selected or actually served model's registered input/output rates. It does not include classifier or other routing overhead; these costs must be disclosed separately and must not be represented as fabricated generation tokens.",
+      "The registered prices are uncached USD rates per million tokens from the production catalog snapshot. RouterArena does not account for separate input-cache read/write prices or per-request and web-search charges.",
+      "The live smoke used BYOK and reported usage.cost=0 with a nonzero usage.market_cost. Evaluation uses registered catalog market rates rather than treating the BYOK account charge as free inference. Reported market_cost is not evidence that classifier overhead is included.",
+      "Reasoning settings are not part of RouterArena's generic cache identity. Preserve production reasoning settings and authoritative token usage in each generated result; do not alias a reasoning variant to an existing cached base model.",
+      "Streaming changes delivery only. Each query has a fresh run-scoped session, which stays fixed across retries. No previous answers or checkpoints are reused. Robustness streams close after a concrete model appears.",
+      "Provider generation errors remain success=false rows and count as failures. If the provider omits final token usage, token_usage is null. The stock evaluator excludes discarded retry attempts and cannot account for missing failed-inference usage, so its reported cost is incomplete.",
+      "Latest aliases resolve to concrete response models. Registered rates use each concrete model's catalog prices; alias discounts can differ. Preserve usage.market_cost separately."
+    ]
+  }
+}