fix: remove the text_label bulk labeling tool and add a cost side-effects rule

.github/pull_request_template.md

@@ -17,6 +17,12 @@ Please provide a description of the issue, the changes you made to fix it, and w
 
 ### How did you verify your code works?
 
+<!-- altimate_change start — required cost side-effects section (see CONTRIBUTING.md) -->
+### Cost side effects
+
+_Does this change add model calls, warehouse queries or cloud resources whose cost grows with user data (rows, files, items)? Describe each one and how it is capped, estimated and confirmed, or write "None"._
+
+<!-- altimate_change end -->
 ### Screenshots / recordings
 
 _If this is a UI change, please include a screenshot or recording._
@@ -25,5 +31,8 @@ _If this is a UI change, please include a screenshot or recording._
 
 - [ ] I have tested my changes locally
 - [ ] I have not included unrelated changes in this PR
+<!-- altimate_change start — cost side-effects check (see CONTRIBUTING.md) -->
+- [ ] I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"
+<!-- altimate_change end -->
 
 _If you do not follow this template your PR will be automatically rejected._

.github/workflows/pr-standards.yml

@@ -252,14 +252,26 @@ jobs:
               }
             }
 
+            // altimate_change start — every PR states its cost side effects, even when there are none
+            const costMatch = body.match(/^### Cost side effects[ \t]*\r?\n([\s\S]*?)(?=^#{1,3}[ \t]|(?![\s\S]))/m);
+            const costContent = costMatch
+              ? costMatch[1].replace(/<!--[\s\S]*?-->/g, '').replace(/^_Does this change add[\s\S]*?_\s*$/m, '').trim()
+              : '';
+            if (!costContent) {
+              issues.push('"Cost side effects" section is missing or empty. Describe any model calls, queries or cloud resources whose cost grows with user data, or write "None".');
+            }
+            // altimate_change end
+
             // Check: checklist boxes are checked
             if (hasChecklistSection) {
               const checklistMatch = body.match(/### Checklist\s*\n([\s\S]*?)(?=###|$)/);
               const checklistContent = checklistMatch ? checklistMatch[1] : '';
               const unchecked = (checklistContent.match(/- \[ \]/g) || []).length;
               const checked = (checklistContent.match(/- \[x\]/gi) || []).length;
-              if (checked < 2) {
-                issues.push('Not all checklist items are checked. Please confirm you have tested locally and have not included unrelated changes.');
+              // altimate_change start — require all three checklist boxes; a deleted line must not pass
+              if (unchecked > 0 || checked < 3) {
+                issues.push('Not all checklist items are checked. Please confirm you have tested locally, have not included unrelated changes and have checked for cost side effects.');
+              // altimate_change end
               }
             }
 

.opencode/skills/data-question/SKILL.md

@@ -11,7 +11,7 @@ Use the available read-only tools to answer the user's question. Source database
 
 1. **Inspect.** Use `warehouse_list`, `schema_inspect` and small `sql_execute` queries as needed. Inspect actual keys and text before choosing joins, casts or parsing rules: schemas alone hide whitespace, case, prefixes and mixed formats. Preserve meaningful leading zeros; check that normalization does not merge distinct identities.
 2. **Plan.** State the population, row grain, denominator, units and material assumptions. Decide which joins and aggregation levels preserve them. Keep this brief; a trivial lookup does not need a checklist or several validation scans.
-3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate. For exact final answers, aggregate the full intended population in SQL.
+3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate, or an exact answer would need per-row model labeling of a large population (see "Batch semantic labels, then count"); then say the answer is an estimate. For exact final answers, aggregate the full intended population in SQL.
 4. **Verify relevant invariants.** Check key uniqueness, before/after join counts, missing matches and null share when they affect the answer. Use `sql_check` for these invariants and extraction coverage; for cross-warehouse comparisons, first materialize both sides into `scratch` with `sql_materialize`. Reconcile components with totals and preserve weighting. `SUM(group_mean * contributing_count) / SUM(contributing_count)` recovers the overall mean for disjoint, complete groups using unrounded means and counts of the same non-null measure; `AVG(group_mean)` gives each group equal weight. `sql_analyze` is optional. If available, a read-only `data-explorer` via `task` can independently re-derive a difficult result.
 5. **Answer.** Explain the basis in one or two sentences (population, grain, caveats), retaining useful tables or explanation before it. Finish with `**Answer:** <requested value(s), with units>`. If evidence or available tools are insufficient, finish with `**Answer:** unable to determine — <why>`. Never guess a missing result.
 
@@ -139,9 +139,9 @@ Dialect references: [DuckDB query_table](https://www.duckdb.org/docs/current/gui
 
 Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
 
-Inspect text, then define the complete population of stable IDs. When `text_label` is available, materialize that population in `scratch` with `sql_materialize`, then call `text_label` with SQL over scratch, the stable ID and text column names, a fixed label set with a description for each label, and any rubric/tie-breaking instructions. It classifies every row through the session's LLM in bounded batches and persists the results in scratch; do not substitute regex or keyword guesses for semantic labels. `text_label` accepts at most 30,000 rows per call. For a larger population, split it into disjoint, deterministic id ranges (one call per range, each into a distinct result table), combine the persisted result tables in SQL, and verify that combined coverage equals the population count before claiming results. Check `rows_in`, `labeled`, `failed` and `unparseable`, and retrieve failed IDs before claiming complete coverage. Optional spot-check agreement measures repeatability, not correctness. Review uncertain cases consistently; batching makes counting reproducible, not classification inherently correct.
+Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself. Without the user's confirmation, label at most 500 rows in total by any path: your own reading, subagents, or warehouse AI functions (for example classification functions in Snowflake, Databricks or BigQuery), which are billed per row too. To label more, first tell the user the row count, how the labels would be produced and the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions), and wait for their confirmation. When the population is larger than 500 rows, by default label a documented random sample of up to 500 rows (roughly ±4.5 percentage points at 95% confidence for a proportion) and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
 
-Use the persisted `text_label` output for SQL counts. It retains the source ID column name, so use `message_id` as the ID column for this example. To reuse the checks below, select `message_id`, `label` and your declared rubric version as `rubric_version` in a SELECT/CTE named `labels`; `population` must cover the same intended IDs. If `text_label` is unavailable, small audited annotation batches can use an available permitted ingestion path to form `labels(message_id, label, rubric_version)`. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
+To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
 
 ```sql
 WITH annotations(message_id, label, rubric_version) AS (
@@ -152,9 +152,9 @@ WITH annotations(message_id, label, rubric_version) AS (
 SELECT * FROM annotations;
 ```
 
-These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. Use `text_label` for large labeling jobs; if it is unavailable and there is no permitted way to ingest complete returned annotation records, report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
+These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. When the full population cannot be labeled within these limits, label a sample or report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
 
-**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
+**Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs (the sampled IDs for a sample) and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
 
 ```sql
 -- Must return no rows: includes duplicates across batches.
@@ -179,7 +179,7 @@ SELECT label, COUNT(*) AS messages
 FROM labels GROUP BY label ORDER BY label;
 ```
 
-Verify the label counts sum to the full population count. Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries.
+Verify the label counts sum to the `population` count (the sample size when you labeled a sample). Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries. For a sample, report proportions with the sample size and the size of the full population, and do not present scaled-up counts as exact.
 
 ## Reading spilled results
 

CONTRIBUTING.md

@@ -204,6 +204,19 @@ Some debug methods that can be problematic:
 
 With that said, you may want to try these methods, as they might work for you.
 
+## Cost side effects
+
+Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR template's "Cost side effects" section, even when the answer is "None". The PR-standards check flags a missing or empty section on contributor PRs (it skips team members), and the review bots check every PR against `REVIEW.md` focus area 12. Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
+
+Such a path must be:
+
+- Off by default, or bounded by a hard cap that does not scale with data size.
+- Above the cap, shown as a cost estimate and run only after the user confirms.
+- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
+- Counted in session cost and traces, so its spend is visible.
+
+Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
+
 ## AI Contributions Policy
 
 We welcome contributions from developers who use AI tools as part of their workflow. However, we require that all contributions reflect genuine understanding and effort from the contributor.

REVIEW.md

@@ -49,6 +49,8 @@ Before flagging: trace the actual caller and the actual shipped surface. This re
 
 11. **Test isolation leaks under parallel CI** (Warning). `mock.module`/dispatcher `reset()` leaked state across test files (52 CI failures fixed via #460). **Flag** new tests using global mocks/dispatchers without confirmed teardown under parallel `bun test`.
 
+12. **Unbounded cost side effects** (Critical; always check, every PR). A feature whose spend grows with user data rather than with the conversation. `text_label` shipped on by default and labeled every row of a table with one session-model call per ~50-row batch: one evaluation day billed ~$680 for it, about 7x the agent sessions themselves, none of it visible in traces or session cost. On a million-row table that is ~20,000 model calls per question. **Flag when** a change can, without an explicit user decision: make model calls per row, batch, file or item (fan-out inside a tool, loops over query results, per-item summarization or classification); run unbounded or user-data-sized warehouse queries, scans or exports; start cloud resources; or add hidden model calls (sub-calls inside tools, background or automatic calls). **Require** all of: (a) off by default or bounded by a hard cap that does not scale with data size; (b) above the cap, an estimate shown and the user's confirmation required before the work runs; (c) the cheapest adequate model by default, with no cache breakpoint on one-off content; (d) its usage counted in session cost and traces. A PR that adds such a path without these is not mergeable; say so even when the feature itself is correct.
+
 ## Repo invariants & landmines
 
 - Effect (`effect` v4 beta) is used pervasively; `Effect.promise`/`Effect.sync`-returning functions look like normal async functions but must be run via `AppRuntime.runPromise` (or equivalent) — a bare `await` silently no-ops the effect.

packages/opencode/AGENTS.md

@@ -12,6 +12,19 @@
 - Capture the current TUI output with: `tmux capture-pane -pt opencode-dev`.
 - Stop the session explicitly when done: `tmux kill-session -t opencode-dev`.
 
+# Cost side effects
+
+Check every feature for spend that grows with user data instead of with the conversation, and flag it in the PR template's "Cost side effects" section, even when the answer is "None". The PR-standards check flags a missing or empty section on contributor PRs (it skips team members), and the review bots check every PR against `REVIEW.md` focus area 12. Examples: model calls per row, batch, file or item inside a tool; hidden model calls (sub-calls in tools, background or automatic calls); unbounded warehouse queries, scans or exports; starting cloud resources. Users' tables can hold millions of rows, so a per-row design that looks cheap in a test can cost thousands of dollars per question.
+
+Such a path must be:
+
+- Off by default, or bounded by a hard cap that does not scale with data size.
+- Above the cap, shown as a cost estimate and run only after the user confirms.
+- On the cheapest adequate model by default, with no prompt-cache breakpoint on one-off content (a cache write costs more than uncached input and is never read back).
+- Counted in session cost and traces, so its spend is visible.
+
+Bulk labeling of text through the session model is the cautionary case: on by default, it cost about 7x the agent sessions themselves on one evaluation day, and none of it appeared in traces.
+
 # Module shape
 
 Do not use `export namespace Foo { ... }` for module organization. It is not

packages/opencode/src/agent/agent.ts

@@ -281,9 +281,6 @@ export const layer = Layer.effect(
           tool_lookup: "allow",
           // Materialized scratch tables for multi-step analysis (tool added separately; key is harmless if absent)
           sql_materialize: "allow",
-          // altimate_change start — model labels are persisted only in session-local scratch tables.
-          text_label: "allow",
-          // altimate_change end
           bash: "deny",
         } as const
         // Cap every configured ruleset at the subagent's read-only allow-list.

packages/opencode/src/altimate/prompts/analyst.txt

@@ -4,7 +4,7 @@ Inspect connections, schemas and actual rows before joining or parsing; key form
 
 Derive large intermediate sets in-engine. For a cross-engine join, move the smaller side: a few hundred keys go into an IN list or VALUES against the big table; materialize into `scratch` only the side that is small after filtering, never a multi-million-row table when a filter or aggregate could shrink it first. Preserve requested weighting: group means weighted by contributing counts can be exact; unweighted averages change the weighting. For many similar tables, prefer catalog-driven SQL (information_schema, DuckDB query_table/UNION ALL BY NAME) after confirming dialect support.
 
-Treat inferred text labels as data: fixed rubric, stable IDs, full coverage and duplicate checks, then deterministic counting, never by eye. When available, `task` with read-only `data-explorer` can batch work or verify reasoning.
+Treat inferred text labels as data: fixed rubric, stable IDs, coverage and duplicate checks, then deterministic counting, never by eye. Count the population first. Labeling grows in cost with the row count: without the user's confirmation, label at most 500 rows in total by any path (your own reading, `data-explorer` subagents or warehouse AI functions). For a larger population, label a documented random sample of up to 500 rows and report an estimate with its sample size; label more only after telling the user the row count, the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions) and getting their confirmation. When available, `task` with read-only `data-explorer` can verify reasoning.
 
 ## Cost-conscious exploration
 Use selected columns, partition filters and LIMIT/sampling to inspect cheaply; estimates are not exact final computations. Read JSONL spill paths with `read`; spills contain returned rows, not necessarily all matches. Aggregate in SQL for exact answers. On timeout, revise the query. Report costs only from evidence.