Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 71 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,7 @@ Supported values:
| `llm.auth` | `subscription`, `api_key` |
| `llm.adapter` | `claude_cli`, `anthropic_api`, `openai_api`, `pi_rpc`, and `codex_cli` are usable for review. `codex_cli` requires `provider: openai` and `auth: subscription`, and is currently best-effort/beta because Codex does not yet expose an explicit all-tools-disabled flag. |
| `llm.model_map` keys | `small`, `medium`, `large` |
| `llm_runtimes.<name>.max_effort` keys | `small`, `medium`, `large`; values `low`, `medium`, `high` |
| `llm.reviewer_model_tier` | `small`, `medium`, `large` |
| `review_policy.major_event` | `comment`, `request_changes` |
| `review_policy.resolve_threads` | `auto`, `never` |
Expand Down Expand Up @@ -664,17 +665,85 @@ Migration note: older releases treated reviewer `model_tier` as a direct map
lookup. Current releases treat it as a minimum acceptable tier, so profiles can
raise the reviewer baseline without editing shared agent catalogs.

### Model-Tier Floors and Effort Ceilings

Agent catalogs declare an absolute `effort` (`low`, `medium`, `high`) that
becomes the provider's reasoning-effort setting. For reviewer resolution,
`agent.model_tier` and `llm.reviewer_model_tier` are minimum floors for model
selection. The selected runtime's `max_effort`
(`llm_runtimes.<name>.max_effort`) is a ceiling for default effort at the final
resolved tier; it does not raise effort or select a model by itself.

For reviewer resolution, `cr` applies this order:

1. Resolve the effective reviewer tier as the higher of the agent tier and the
profile reviewer-tier floor.
2. Resolve `model_map[effective tier]`, including provider built-ins.
3. Cap the default effort with `max_effort[effective tier]`.
4. Apply an explicit effort override, which wins over the ceiling.

Other tier-resolved internal stages use their own stage tier; `max_effort` is
applied at that stage's final resolved tier, without the reviewer floors.

Configure ceilings manually in `config.yml`:

```yaml
llm_runtimes:
review:
provider: openai
auth: subscription
adapter: codex_cli
model_map:
large: openai-codex/gpt-5.6-sol
medium: openai-codex/gpt-5.6-terra
max_effort:
large: medium
profiles:
default:
llm_runtime: review
```

A tier absent from `max_effort` is uncapped. The cap is a ceiling only: an agent
declaring `low` under a `medium` ceiling still runs at `low`. Reviewer floors
apply only to reviewer resolution; other tier-resolved internal stages use
their own final tier for the ceiling.

The complete precedence and bypass table is:

| Input or path | Model selection | Default effort | `max_effort` on selected runtime |
|---------------|-----------------|----------------|-----------------|
| Reviewer resolution with no explicit override | `max(llm.reviewer_model_tier, agent.model_tier)`, then `model_map` | Agent effort | Caps the default at the final reviewer tier |
| `--reviewer-model-tier` | Raises the reviewer baseline before the agent floor is applied | Agent effort | Caps at the final resolved tier |
| Other tier-resolved internal stage | That stage's own tier, then `model_map` | Stage effort | Caps the default at the stage's final tier |
| `--selection-effort` or `--reviewer-effort` | Normal tier or exact-model selection | Requested effort | Explicit effort wins after the ceiling |
| `--selection-model` or `--reviewer-model` | Exact requested model ID | Stage/agent effort or explicit effort | Bypassed; exact model overrides intentionally bypass tier resolution and the cap |
| Agent `model_id` | Exact agent model ID | Agent effort | Bypassed; exact model selection intentionally bypasses tier resolution and the cap |
| `cr benchmark run` stage model/effort overrides | Exact benchmark model when supplied; otherwise normal tier selection | Explicit benchmark effort when supplied | Explicit benchmark overrides bypass the profile ceiling |

For example, with `agent.model_tier: small`, `effort: high`,
`llm.reviewer_model_tier: large`, and the selected runtime's
`max_effort.large: medium`, the reviewer
runs with the large model at medium effort. Adding `--reviewer-effort high`
runs that same large model at high effort. Adding
`--reviewer-model my-provider/model` selects that exact model and keeps high
effort without applying the tier ceiling. `--selection-effort high` follows
the same post-ceiling override rule for selection.

`cr init` preserves runtime `max_effort` but cannot yet edit it; set
`llm_runtimes.<name>.max_effort` by hand in `config.yml`.

Dry-run and no-post runs also record selected reviewer runtime resolution in
`agent-sources.json` for auditability. Each selected agent may include
`reviewer_runtime` with:

| Field | Meaning |
|-------|---------|
| `mode` | `tier_floor` for portable tier resolution, `exact_model` for agent `model_id` passthrough |
| `mode` | `tier_floor` for portable tier resolution, `exact_model` for agent `model_id` passthrough, or `override` for `--reviewer-model` |
| `floor_tier` | Declared agent `model_tier` floor when `mode=tier_floor` |
| `baseline_tier` | Effective operator baseline tier used for this run |
| `effective_tier` | Higher of baseline and agent floor |
| `resolved_model` | Resolved provider model, from the active model map for `tier_floor` or from agent `model_id` for `exact_model` |
| `resolved_model` | Actual provider model, from the active model map for `tier_floor`, agent `model_id` for `exact_model`, or `--reviewer-model` for `override` |
| `resolved_effort` | Actual reviewer effort after the tier ceiling and any explicit reviewer-effort override |
| `model_map_source` | `built_in` or `config` for the resolved tier mapping |

Built-in model maps:
Expand Down
35 changes: 27 additions & 8 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,28 +43,47 @@ session row, but they still use the same metadata schema and lifecycle runner.

Runtime model choice must be resolved through `internal/stagemodel`. Code that
executes an LLM stage must not hard-code model IDs and must not call
`config.ResolveModelTier` directly.
`config.ResolveModelTier` or `config.ResolveMaxEffort` directly.

`stagemodel.ResolveStageModel` is the single runtime path from profile
preferences and command overrides to a concrete model and effort. The request
must include the named stage, requested tier, default effort, and any explicit
operator override. The resolver applies user profile `llm.model_map` values,
operator override. The resolver applies the selected runtime's `model_map` values,
built-in provider defaults, and configured tier floors before returning the
concrete runtime choice.

This boundary exists so model catalog data, provider capabilities, token costs,
and profile-level tier floors can be added without touching individual review
stages. Runtime hard-coding bypasses user preference and is a bug.
and reviewer-resolution tier floors can be added without touching individual
review stages. Runtime hard-coding bypasses user preference and is a bug.

For reviewer tier-based requests, the resolver's authoritative ordering is:
resolve the effective tier after applying the profile reviewer-tier floor and
agent floor; resolve the model for that tier; cap the default effort with the
Comment thread
rianjs marked this conversation as resolved.
Comment thread
rianjs marked this conversation as resolved.
selected runtime's `max_effort` entry for that final tier; then apply
`EffortOverride`. This means `--reviewer-model-tier` is still capped at the tier
it ultimately resolves, while `--selection-effort` and `--reviewer-effort` win
after the ceiling. Other tier-resolved internal stages use their own stage
tier before applying `max_effort` at that final tier.

An explicit `ModelOverride` returns with its requested effort or default effort
while intentionally bypassing tier resolution and the `max_effort` cap.
`--selection-model`, `--reviewer-model`, and agent `model_id` use this
exact-model path. Benchmark stage model and effort overrides are explicit
runtime inputs and retain the same ceiling bypass.

Reviewer `agent.model_id` is an exact provider-specific model override. It must
still enter runtime execution through `stagemodel.ResolveStageModel` as a model
override rather than bypassing the resolver, but it intentionally bypasses the
tier map because the agent author selected a concrete model.

The direct `config.ResolveModelTier` exception is config inspection and the
resolver implementation itself.
`internal/architecture/model_resolution_test.go` enforces that direct
`config.ResolveModelTier` calls stay inside approved packages. Hard-coded
The reviewer execution request, cohort member, session row, and
`agent-sources.json` reviewer provenance must all use the same final resolved
model and effort, including explicit reviewer model and effort overrides.

Direct `config.ResolveModelTier` and `config.ResolveMaxEffort` calls are
allowed only for config inspection and inside the resolver implementation.
`internal/architecture/model_resolution_test.go` enforces that both direct
calls stay inside approved packages. Hard-coded
runtime model IDs remain a code-review concern until model-catalog guardrails
exist.

Expand Down
19 changes: 19 additions & 0 deletions docs/init-config-surface.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,25 @@ Retention is global config under `data.retention`, not profile config.
#178 must add retention command tests for omitted/default vs explicit zero.
#184 must use the same validation and reset behavior in interactive init.

## LLM Effort Ceiling Ownership

The canonical ceiling path is `llm_runtimes.<name>.max_effort`, and
`profiles.<name>.llm_runtime` selects that runtime. The legacy
`profiles.<name>.llm.max_effort` path is compatibility/projection only, not the
canonical storage location. The map accepts `small`, `medium`, and `large`
tier keys with `low`, `medium`, or `high` ceiling values. Interactive and
non-interactive `cr init` must preserve an existing map, including when the
profile or selected LLM runtime is staged and saved; init does not edit or
remove it. Configure it by editing `config.yml` directly. Model-map JSON-row
parity and init editing for this field are out of scope.

At review time, reviewer floors apply only to reviewer resolution. Other
tier-resolved internal stages use their own stage tier, and default effort is
capped only after that final tier is resolved. Explicit `--selection-effort`
and `--reviewer-effort` values win after the cap. Exact
`--selection-model`, `--reviewer-model`, agent `model_id`, and benchmark stage
model/effort overrides intentionally bypass tier resolution and the cap.

## Scripted Install Ownership

Scripted installs should remain readable. The intended shape is:
Expand Down
4 changes: 2 additions & 2 deletions internal/architecture/model_resolution_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,15 @@ func TestRuntimeModelResolutionGoesThroughStageResolver(t *testing.T) {
return true
}
selector, ok := call.Fun.(*ast.SelectorExpr)
if !ok || selector.Sel.Name != "ResolveModelTier" {
if !ok || selector.Sel.Name != "ResolveModelTier" && selector.Sel.Name != "ResolveMaxEffort" {
return true
}
ident, ok := selector.X.(*ast.Ident)
if !ok || !configAliases[ident.Name] {
return true
}
pos := fset.Position(selector.Pos())
t.Fatalf("%s calls config.ResolveModelTier directly; runtime model selection must use internal/stagemodel", pos)
t.Fatalf("%s calls config.%s directly; runtime model and effort resolution must use internal/stagemodel", pos, selector.Sel.Name)
return false
})
return nil
Expand Down
53 changes: 49 additions & 4 deletions internal/cmd/initcmd/initcmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,8 @@ type initDraft struct {
Routes []configedit.RepositoryRouteSpec
ModelMapSet bool
ModelMap config.ModelMap
MaxEffortSet bool
MaxEffort config.EffortMap
AgentSourcesSet bool
AgentSources []string
ReviewPolicySet bool
Expand Down Expand Up @@ -430,6 +432,7 @@ type initLLMRuntimeDraft struct {
CredentialStore string
CredentialRef string
ModelMap config.ModelMap
MaxEffort config.EffortMap
ReviewerModelTier config.ModelTier
}

Expand Down Expand Up @@ -1116,6 +1119,12 @@ func completeInteractiveInitProfileV2Draft(ctx initPromptContext, draft initDraf
}
draft.ModelMapSet = true
}
if !draft.MaxEffortSet {
if ctx.ExistingProfile != nil {
draft.MaxEffort = copyEffortMap(ctx.ExistingProfile.LLM.MaxEffort)
}
draft.MaxEffortSet = true
}
if !draft.AgentSourcesSet {
if ctx.ExistingProfile != nil {
draft.AgentSources = append([]string(nil), ctx.ExistingProfile.AgentSources...)
Expand Down Expand Up @@ -2333,10 +2342,11 @@ func initReviewerModelTierOptions() []huh.Option[string] {

func initProfileEditorModelMapLLM(draft initDraft, selectedLLMRuntime string, runtimes map[string]initLLMRuntimeDraft) config.LLMConfig {
llm := config.LLMConfig{
Provider: config.LLMProvider(draft.LLMProvider),
Auth: config.LLMAuth(draft.LLMAuth),
Adapter: config.LLMAdapter(draft.LLMAdapter),
ModelMap: copyModelMap(draft.ModelMap),
Provider: config.LLMProvider(draft.LLMProvider),
Auth: config.LLMAuth(draft.LLMAuth),
Adapter: config.LLMAdapter(draft.LLMAdapter),
ModelMap: copyModelMap(draft.ModelMap),
MaxEffort: copyEffortMap(draft.MaxEffort),
}
if runtime, ok := runtimes[selectedLLMRuntime]; ok {
llm.Provider = runtime.Provider
Expand Down Expand Up @@ -2540,6 +2550,7 @@ func initLLMRuntimeDraftFromSeedDraft(draft initDraft) initLLMRuntimeDraft {
Adapter: config.LLMAdapter(draft.LLMAdapter),
Credential: initCredentialLocationIfName(draft.LLMCredentialStore, draft.LLMCredentialRef),
ModelMap: copyModelMap(draft.ModelMap),
MaxEffort: copyEffortMap(draft.MaxEffort),
ReviewerModelTier: config.ModelTier(strings.TrimSpace(draft.LLMReviewerModelTier)),
})
}
Expand Down Expand Up @@ -2699,6 +2710,8 @@ func applyLLMRuntimeInventorySelection(draft *initDraft, selection string, runti
draft.LLMAdapter = string(runtime.Adapter)
draft.ModelMap = copyModelMap(runtime.ModelMap)
draft.ModelMapSet = true
draft.MaxEffort = copyEffortMap(runtime.MaxEffort)
draft.MaxEffortSet = true
draft.LLMReviewerModelTier = string(runtime.ReviewerModelTier)
if !draft.AdvancedStorageLabels {
draft.LLMCredentialStore = initCredentialStoreDraftValue(runtime.CredentialStore)
Expand Down Expand Up @@ -3105,6 +3118,8 @@ func seedInteractiveInitDraft(requestedProfileName string, existingProfileName s
draft.LLMCredentialStore = initCredentialStoreDraftValue(existingProfile.LLM.Credential.Store)
draft.LLMCredentialRef = existingProfile.LLM.Credential.Name
draft.ModelMap = copyModelMap(existingProfile.LLM.ModelMap)
draft.MaxEffort = copyEffortMap(existingProfile.LLM.MaxEffort)
draft.MaxEffortSet = true
draft.AgentSources = append([]string(nil), existingProfile.AgentSources...)
draft.ReviewPolicy = existingProfile.ReviewPolicy
if existingProfile.Reviewer.GitHubAppInstallation != nil {
Expand Down Expand Up @@ -3404,6 +3419,9 @@ func buildNonInteractiveInitPlan(cmd *cobra.Command, opts *root.Options, flags i
}
profile.LLM.ModelMap = modelMap
}
if previousProfile.LLM.MaxEffort != nil {
profile.LLM.MaxEffort = copyEffortMap(previousProfile.LLM.MaxEffort)
}
if !cmd.Flags().Changed("agent-source") {
profile.AgentSources = append([]string(nil), previousProfile.AgentSources...)
}
Expand Down Expand Up @@ -4541,6 +4559,7 @@ func initLLMRuntimeDraftFromConfig(llm config.LLMConfig) initLLMRuntimeDraft {
CredentialStore: initCredentialStoreDraftValue(llm.Credential.Store),
CredentialRef: strings.TrimSpace(llm.Credential.Name),
ModelMap: copyModelMap(llm.ModelMap),
MaxEffort: copyEffortMap(llm.MaxEffort),
ReviewerModelTier: llm.ReviewerModelTier,
}
if spec, ok := config.FindLLMRuntimeSpec(runtime.Provider, runtime.Auth, runtime.Adapter); ok &&
Expand All @@ -4559,6 +4578,7 @@ func (runtime initLLMRuntimeDraft) exportConfig() config.LLMConfig {
Auth: runtime.Auth,
Adapter: runtime.Adapter,
ModelMap: copyModelMap(runtime.ModelMap),
MaxEffort: copyEffortMap(runtime.MaxEffort),
ReviewerModelTier: runtime.ReviewerModelTier,
}
if runtime.Auth == config.LLMAuthAPIKey {
Expand All @@ -4577,13 +4597,23 @@ func (runtime initLLMRuntimeDraft) identityKey() string {
for _, tier := range modelKeys {
models = append(models, tier+"="+strings.TrimSpace(runtime.ModelMap[tier]))
}
effortKeys := make([]string, 0, len(runtime.MaxEffort))
for tier := range runtime.MaxEffort {
effortKeys = append(effortKeys, tier)
}
sort.Strings(effortKeys)
efforts := make([]string, 0, len(effortKeys))
for _, tier := range effortKeys {
efforts = append(efforts, tier+"="+strings.TrimSpace(runtime.MaxEffort[tier]))
}
return strings.Join([]string{
string(runtime.Provider),
string(runtime.Auth),
string(runtime.Adapter),
initCredentialStoreDraftValue(runtime.CredentialStore),
strings.TrimSpace(runtime.CredentialRef),
strings.Join(models, "\x1f"),
strings.Join(efforts, "\x1f"),
string(runtime.ReviewerModelTier),
}, "\x00")
}
Expand Down Expand Up @@ -4762,6 +4792,7 @@ func cloneInitLLMConfig(llm config.LLMConfig) config.LLMConfig {
cloned.ModelMap[tier] = model
}
}
cloned.MaxEffort = copyEffortMap(llm.MaxEffort)
return cloned
}

Expand Down Expand Up @@ -4848,6 +4879,9 @@ func synthesizeInteractiveProfile(flags initOptions, profileName string, previou
profile.LLM.Auth = config.LLMAuth(draft.LLMAuth)
profile.LLM.Adapter = config.LLMAdapter(draft.LLMAdapter)
profile.LLM.ReviewerModelTier = config.ModelTier(strings.TrimSpace(draft.LLMReviewerModelTier))
if draft.MaxEffortSet {
profile.LLM.MaxEffort = copyEffortMap(draft.MaxEffort)
}
if profile.LLM.Auth == config.LLMAuthAPIKey {
llmRef := strings.TrimSpace(draft.LLMCredentialRef)
if llmRef == "" {
Expand Down Expand Up @@ -5814,6 +5848,17 @@ func initCredentialWritePlanSatisfiesEntry(entry initCredentialPlanEntry, target
return true
}

func copyEffortMap(effortMap config.EffortMap) config.EffortMap {
if len(effortMap) == 0 {
return nil
}
copied := make(config.EffortMap, len(effortMap))
for tier, ceiling := range effortMap {
copied[tier] = ceiling
}
return copied
}

func copyModelMap(modelMap config.ModelMap) config.ModelMap {
if len(modelMap) == 0 {
return nil
Expand Down
Loading
Loading