feat: odek v2 merge-blocker gaps (cache usage, idle timeout, reasoning replay)

AGENTS.md

@@ -48,7 +48,7 @@ The canonical type system is OpenAI-shaped; `gemini.go`/`anthropic.go` translate
 ## Testing conventions
 
 - **RED-first TDD**: failing test first, then the fix. Table-driven tests; `httptest` servers for hermetic coverage; `newTestClient` helper pins `backoffUnit` to 1ms — restore package vars in `t.Cleanup`.
-- Two timing knobs are package vars for tests: `backoffUnit`, `streamIdleTimeout`.
+- Two timing knobs are package vars for tests: `backoffUnit`, `streamIdleTimeout`. Operators override the idle watchdog via `SetStreamIdleTimeout` (positive values only).
 - The e2e suite (`e2e_test.go`) is behind a `e2e` build tag and hits **live APIs**. It must never lose that tag. Keys come from env or a gitignored `.env`; contents are never logged; tests skip when a key is absent. Adding a provider = one `e2eTarget` entry; models overridable via `<ID>_E2E_MODEL`.
 - Live-provider behavior (e.g. DeepSeek eliding `reasoning_content`) is **not an SDK contract** — probe softly, assert only what the SDK guarantees (call success, parsing, canonical finish). Model answer correctness is never an assertion.
 - Timer hygiene: since Go 1.23 no drain-before-`Reset` is needed for `time.Timer`. Note: on some dev machines `time.After` + select-default spin loops have hung — prefer deadline loops in tests.

README.md

@@ -84,22 +84,22 @@ Requests and results are provider-neutral. Unknown message roles are rejected at
 ```go
 type ChatRequest struct {
     Model          string         // optional; ChatClient's model wins when both set
-    Messages       []Message      // RoleUser | RoleAssistant | RoleSystem | RoleTool
+    Messages       []Message      // RoleUser | RoleAssistant | RoleSystem | RoleTool; Message.Cache → Anthropic user-block cache_control
     System         []SystemBlock  // {Text, Cache} — Cache marks Anthropic prompt-cache blocks
     Tools          []ToolDef      // {Name, Description, Parameters json.RawMessage}
-    Thinking       string         // "", "enabled", "disabled", "low", "medium", "high"
+    Thinking       string         // "", "enabled", "disabled", "low", "medium", "high", "max"
     ThinkingBudget int            // explicit token budget where the provider supports it
     MaxTokens      int            // routed to max_completion_tokens on o-series/gpt-5
     Temperature    float64        // 0 = provider default; negative = explicit 0
 }
 
 type ChatResult struct {
     Content           string
-    ReasoningContent  string      // provider thinking text (advisory)
+    ReasoningContent  string      // provider thinking text; replayed as reasoning_content on OpenAI-format assistant turns
     ThinkingSignature string      // Anthropic: replay via Message.ThinkingSignature
     ToolCalls         []ToolCall  // {ID, Name, Arguments}
     FinishReason      string      // stop | length | tool_calls | content_filter | ""
-    Usage             Usage       // {PromptTokens, CompletionTokens, ReasoningTokens}
+    Usage             Usage       // PromptTokens is uncached-only; cache volumes in CacheRead/Creation/CachedTokens
 }
 ```
 
@@ -109,7 +109,7 @@ Finish reasons are canonical: anything a provider reports outside the vocabulary
 
 `CallStream` enforces four guarantees, each covered by regression tests:
 
-1. **Idle watchdog** — a stream silent longer than `streamIdleTimeout` (120s default) fails with `ErrIdleTimeout`. Keepalive comments reset it.
+1. **Idle watchdog** — a stream silent longer than `StreamIdleTimeout()` (120s default; override with `SetStreamIdleTimeout`, positive values only) fails with `ErrIdleTimeout`. Keepalive comments reset it.
 2. **Hard wall-clock deadline** — the whole stream is bounded by the per-request timeout (`WithRequestTimeout`, default 120s; per-client via `SetRequestTimeout`).
 3. **Retries never duplicate output** — retries happen only before the first emitted delta. A failure after partial output returns the partial `*ChatResult` plus a wrapped error and is never retried.
 4. **No silent empty successes** — a provider that closes the stream before its completion signal (before `[DONE]` / `message_stop`) yields a retryable error, not an empty result. Gemini, whose streams legitimately end at EOF, is exempt.
@@ -143,7 +143,7 @@ On Gemini, a tool result's `ToolName` may be omitted — the SDK recovers the fu
 ## Extended thinking
 
 - **Anthropic** — `thinking` blocks are parsed in both buffered and streaming modes. `ChatResult.ThinkingSignature` carries the provider signature; for tool loops, replay it on the assistant message (`Message.ReasoningContent` + `Message.ThinkingSignature`) — the SDK re-serializes it as the first block, as Anthropic's API requires. Omitting it makes extended-thinking tool loops fail mid-conversation.
-- **DeepSeek / GLM** — reasoning streams as `DeltaReasoning` fragments and lands in `ReasoningContent` (advisory; not replayed).
+- **DeepSeek / GLM** — reasoning streams as `DeltaReasoning` fragments and lands in `ReasoningContent`. Assistant-turn replay echoes it as `reasoning_content` (required for DeepSeek/GLM tool loops). GLM maps thinking `medium` → `reasoning_effort` `high` (no medium level) and `max` → `max`.
 - **Gemini** — `thought: true` parts map to reasoning deltas; `thinkingConfig` is derived from `Thinking`/`ThinkingBudget`.
 
 ## Learn-once fallbacks
@@ -159,10 +159,11 @@ When a provider rejects a request pattern, the SDK learns the constraint **once
 
 ## Retry policy
 
-8 attempts, exponential backoff capped at 30s with ±20% jitter, `Retry-After` (seconds or HTTP-date) honored, context cancellation honored between and during attempts. Persistent 429s surface as `*llm.RateLimitError{Attempts, RetryAfter}` — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. `RateLimitError` unwraps to `*APIError` for `errors.As` access to `Status`/`Retryable`.
+8 attempts, exponential backoff capped at 30s with ±20% jitter, `Retry-After` (seconds or HTTP-date) honored and capped at 120s, context cancellation honored between and during attempts. Retryable statuses include 408/429/5xx plus Cloudflare 520–524 and Anthropic 529. Persistent 429s surface as `*llm.RateLimitError{Attempts, RetryAfter}` — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. A 429 whose body is billing exhaustion (`insufficient_quota`, `exceeded your current quota`, `insufficient balance`, `no resource package`) is not retryable and fails on the first attempt. `RateLimitError` unwraps to `*APIError` for `errors.As` access to `Status`/`Retryable`.
 
 ## Timeouts & cancellation
 
+- The pooled transport honors `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` via `http.ProxyFromEnvironment`.
 - Buffered calls: per-request timeout on the HTTP client (`WithRequestTimeout`, per-client `SetRequestTimeout` — race-safe, swap is atomic).
 - Streaming: the same timeout becomes the hard wall-clock deadline via context; per-attempt SSE reads are additionally bounded by the idle watchdog.
 - Every wait (backoff, Retry-After, stream reads) selects on the caller's context — cancellation propagates everywhere, and a cancelled call never misreports as "retry exhausted".

anthropic.go

@@ -28,6 +28,8 @@ type anSysBlock struct {
 type anBlock struct {
 	Type string `json:"type"` // "text" | "tool_use" | "tool_result"
 	Text string `json:"text,omitempty"`
+	// CacheControl is set on user text blocks when Message.Cache is true.
+	CacheControl *anCacheControl `json:"cache_control,omitempty"`
 	// thinking (replayed assistant turns; must be the FIRST block and
 	// carry the provider signature)
 	Thinking  string `json:"thinking,omitempty"`
@@ -151,9 +153,13 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte,
 		case RoleSystem:
 			out.System = append(out.System, anSysBlock{Type: "text", Text: m.Content})
 		case RoleUser:
+			blk := anBlock{Type: "text", Text: m.Content}
+			if m.Cache {
+				blk.CacheControl = &anCacheControl{Type: "ephemeral"}
+			}
 			out.Messages = append(out.Messages, anMessage{
 				Role:    "user",
-				Content: []anBlock{{Type: "text", Text: m.Content}},
+				Content: []anBlock{blk},
 			})
 		case RoleAssistant:
 			var blocks []anBlock
@@ -213,13 +219,33 @@ type anRespBlock struct {
 	Input     json.RawMessage `json:"input"`
 }
 
+type anUsage struct {
+	InputTokens         int `json:"input_tokens"`
+	OutputTokens        int `json:"output_tokens"`
+	CacheCreationTokens int `json:"cache_creation_input_tokens"`
+	CacheReadTokens     int `json:"cache_read_input_tokens"`
+}
+
 type anResponse struct {
 	Content    []anRespBlock `json:"content"`
 	StopReason string        `json:"stop_reason"`
-	Usage      struct {
-		InputTokens  int `json:"input_tokens"`
-		OutputTokens int `json:"output_tokens"`
-	} `json:"usage"`
+	Usage      anUsage       `json:"usage"`
+}
+
+// usageFromAnthropic maps Messages API usage onto canonical Usage.
+// Anthropic reports cache volumes exclusively — input_tokens is already
+// uncached-only — so PromptTokens is left alone.
+func usageFromAnthropic(u anUsage) Usage {
+	out := Usage{
+		PromptTokens:        u.InputTokens,
+		CompletionTokens:    u.OutputTokens,
+		CacheCreationTokens: u.CacheCreationTokens,
+		CacheReadTokens:     u.CacheReadTokens,
+	}
+	if u.CacheCreationTokens > 0 || u.CacheReadTokens > 0 {
+		out.CacheReported = true
+	}
+	return out
 }
 
 // mapAnthropicStopReason maps stop_reason to canonical values.
@@ -275,10 +301,7 @@ func parseAnthropicResponse(body []byte) (*ChatResult, error) {
 			res.ThinkingSignature = b.Signature
 		}
 	}
-	res.Usage = Usage{
-		PromptTokens:     r.Usage.InputTokens,
-		CompletionTokens: r.Usage.OutputTokens,
-	}
+	res.Usage = usageFromAnthropic(r.Usage)
 	return res, nil
 }
 
@@ -288,9 +311,7 @@ type anStreamEvent struct {
 	Type string `json:"type"`
 	// message_start
 	Message struct {
-		Usage struct {
-			InputTokens int `json:"input_tokens"`
-		} `json:"usage"`
+		Usage anUsage `json:"usage"`
 	} `json:"message"`
 	// content_block_start / content_block_stop
 	Index        int         `json:"index"`
@@ -326,7 +347,7 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro
 	var deltas []Delta
 	switch ev.Type {
 	case "message_start":
-		acc.usage.PromptTokens = ev.Message.Usage.InputTokens
+		acc.usage = usageFromAnthropic(ev.Message.Usage)
 	case "content_block_start":
 		if ev.ContentBlock.Type == "tool_use" {
 			c := acc.call(ev.Index)

anthropic_test.go

@@ -87,6 +87,37 @@ func TestBuildAnthropicRequest_Golden(t *testing.T) {
 	}
 }
 
+func TestBuildAnthropicRequest_UserCacheMarker(t *testing.T) {
+	req := &ChatRequest{Messages: []Message{
+		{Role: RoleUser, Content: "cached turn", Cache: true},
+		{Role: RoleUser, Content: "plain turn"},
+	}}
+	body, err := buildAnthropicRequest(req, "claude", false)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var m map[string]any
+	if err := json.Unmarshal(body, &m); err != nil {
+		t.Fatalf("decode: %v\n%s", err, body)
+	}
+	msgs := m["messages"].([]any)
+	if len(msgs) != 2 {
+		t.Fatalf("messages = %d, want 2", len(msgs))
+	}
+	cached := msgs[0].(map[string]any)["content"].([]any)[0].(map[string]any)
+	if cached["text"] != "cached turn" {
+		t.Errorf("cached text = %v", cached["text"])
+	}
+	cc, ok := cached["cache_control"].(map[string]any)
+	if !ok || cc["type"] != "ephemeral" {
+		t.Errorf("cached user block cache_control = %v, want {type:ephemeral}", cached["cache_control"])
+	}
+	plain := msgs[1].(map[string]any)["content"].([]any)[0].(map[string]any)
+	if _, ok := plain["cache_control"]; ok {
+		t.Errorf("plain user block must omit cache_control, got %v", plain["cache_control"])
+	}
+}
+
 func TestBuildAnthropicRequest_EmptyAssistantPlaceholder(t *testing.T) {
 	req := &ChatRequest{Messages: []Message{
 		{Role: RoleUser, Content: "q"},

chat.go

@@ -80,9 +80,23 @@ const (
 
 // streamIdleTimeout bounds the silence between SSE events. Thinking models
 // can legitimately spend minutes before their first event, so the default
-// is generous. Package var so tests can shorten it.
+// is generous. Package var so tests can shorten it; operators override
+// via SetStreamIdleTimeout.
 var streamIdleTimeout = 120 * time.Second
 
+// SetStreamIdleTimeout overrides the SSE idle watchdog. Call at startup,
+// before the first request; non-positive values are ignored.
+func SetStreamIdleTimeout(d time.Duration) {
+	if d > 0 {
+		streamIdleTimeout = d
+	}
+}
+
+// StreamIdleTimeout reports the active idle watchdog (introspection/tests).
+func StreamIdleTimeout() time.Duration {
+	return streamIdleTimeout
+}
+
 // errStreamStop is the internal sentinel for a clean stream end.
 var errStreamStop = errors.New("llm: stream complete")
 
@@ -315,7 +329,8 @@ func billingExhausted(e *APIError) bool {
 	m := strings.ToLower(e.Message)
 	return strings.Contains(m, "insufficient balance") ||
 		strings.Contains(m, "insufficient_quota") ||
-		strings.Contains(m, "no resource package")
+		strings.Contains(m, "no resource package") ||
+		strings.Contains(m, "exceeded your current quota")
 }
 
 // retryDelay picks Retry-After when present, else exponential backoff.

dispatch_edges_test.go

@@ -813,8 +813,8 @@ func TestBuildOpenAIRequestSystemAndToolRoles(t *testing.T) {
 		t.Fatal(err)
 	}
 	s := string(b)
-	if !strings.Contains(s, `"role":"system","content":"sys\nband"`) {
-		t.Errorf("system fold missing: %s", s)
+	if !strings.Contains(s, `"role":"system","content":"sys"`) || !strings.Contains(s, `"role":"system","content":"band"`) {
+		t.Errorf("separate system messages missing: %s", s)
 	}
 	if !strings.Contains(s, `"role":"tool","content":"r","tool_call_id":"t1"`) {
 		t.Errorf("tool role missing: %s", s)
@@ -1003,6 +1003,31 @@ func TestListModelsGeminiMidPageError(t *testing.T) {
 	}
 }
 
+func TestBillingExhausted_Markers(t *testing.T) {
+	for _, tc := range []struct {
+		name    string
+		status  int
+		message string
+		want    bool
+	}{
+		{"z.ai insufficient balance", 429, "Insufficient balance or no resource package. Please recharge.", true},
+		{"openai insufficient_quota code", 429, "insufficient_quota", true},
+		{"openai quota message", 429, "You exceeded your current quota, please check your plan and billing details.", true},
+		{"deepseek balance", 429, "Insufficient Balance", true},
+		{"plain rate limit", 429, "rate limit exceeded, retry later", false},
+		{"billing text on non-429", 500, "Insufficient balance", false},
+		{"nil error", 429, "", false},
+	} {
+		var e *APIError
+		if tc.name != "nil error" {
+			e = &APIError{Status: tc.status, Message: tc.message}
+		}
+		if got := billingExhausted(e); got != tc.want {
+			t.Errorf("%s: billingExhausted = %v, want %v", tc.name, got, tc.want)
+		}
+	}
+}
+
 // ── billing exhaustion: 429 that is really a permanent billing failure ───
 
 // Billing/resource exhaustion signalled as 429 is permanent — the SDK

idle_timeout_test.go

@@ -0,0 +1,25 @@
+package llm
+
+import (
+	"testing"
+	"time"
+)
+
+// SetStreamIdleTimeout pins the setter contract: positive values apply,
+// zero and negative are ignored so a misconfigured caller cannot disable
+// the watchdog by accident.
+func TestSetStreamIdleTimeout(t *testing.T) {
+	orig := StreamIdleTimeout()
+	t.Cleanup(func() { streamIdleTimeout = orig })
+
+	SetStreamIdleTimeout(5 * time.Second)
+	if got := StreamIdleTimeout(); got != 5*time.Second {
+		t.Fatalf("StreamIdleTimeout() = %v, want 5s", got)
+	}
+
+	SetStreamIdleTimeout(0)
+	SetStreamIdleTimeout(-1 * time.Second)
+	if got := StreamIdleTimeout(); got != 5*time.Second {
+		t.Fatalf("non-positive override applied; StreamIdleTimeout() = %v, want 5s", got)
+	}
+}

message.go

@@ -47,11 +47,12 @@ type ToolCall struct {
 // Message is one canonical chat message. For RoleTool messages, ToolCallID
 // and ToolName identify the call being answered and Content carries the
 // tool result. ReasoningContent is provider-reported thinking text
-// (deepseek-reasoner, anthropic thinking, gemini thoughts). It is consumer
-// metadata; the SDK replays it back only where a provider requires it for
-// conversation continuity — Anthropic, and only when ThinkingSignature is
-// also set (extended-thinking tool loops mandate the signed thinking block
-// as the first block of the replayed assistant turn).
+// (deepseek-reasoner, anthropic thinking, gemini thoughts). The SDK
+// replays it where a provider requires conversation continuity:
+// OpenAI-format assistant messages echo it as reasoning_content
+// (DeepSeek/GLM tool loops), and Anthropic re-serializes a signed
+// thinking block as the first content block when ThinkingSignature is
+// also set.
 type Message struct {
 	Role             Role
 	Content          string
@@ -62,6 +63,10 @@ type Message struct {
 	ToolCalls         []ToolCall
 	ToolCallID        string
 	ToolName          string
+	// Cache marks this user message for Anthropic prompt caching
+	// (cache_control ephemeral on the text block). Ignored on other
+	// formats and on non-user roles.
+	Cache bool
 }
 
 // SystemBlock is one system-prompt segment. On Anthropic each block maps to
@@ -82,15 +87,24 @@ type ToolDef struct {
 }
 
 // Usage reports token accounting. Fields the provider does not report stay 0.
+// PromptTokens is exclusive (uncached-only) after provider-specific
+// normalization: OpenAI cached_tokens and DeepSeek hit/miss are subsets of
+// prompt_tokens and are subtracted; Anthropic reports cache volumes
+// exclusively and is left alone. Cache volumes live in the cache fields so
+// budget enforcement can sum without double-counting.
 type Usage struct {
-	PromptTokens     int
-	CompletionTokens int
-	ReasoningTokens  int
+	PromptTokens        int
+	CompletionTokens    int
+	ReasoningTokens     int
+	CacheReadTokens     int
+	CacheCreationTokens int
+	CachedTokens        int
+	CacheReported       bool
 }
 
 // ChatRequest is the canonical request. Model is filled from the ChatClient
 // when empty. Thinking accepts "", "enabled", "disabled", "low", "medium",
-// "high" and is translated per provider format. Temperature: 0 means use
+// "high", "max" and is translated per provider format. Temperature: 0 means use
 // the provider default (field omitted); use a negative value to explicitly
 // send 0.
 type ChatRequest struct {