diff --git a/CHANGELOG.md b/CHANGELOG.md index 54e11b5d..4b1e4763 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,78 @@ target. ## [Unreleased] +## [3.4.0] — 2026-10-11 + +Conversations with reasoning models behind Chat-format upstreams keep their +reasoning from turn to turn, Gemini's tool loops keep their thought +signatures through a conversion, and a tool call that a Chat-format model +writes into its answer text reaches the caller as a tool call. The +thinkwatch-core crates move from v0.68.0 to v0.69.0. + +### Read before upgrading + +- **Converted answers from Chat-format upstreams can carry tool calls that + arrived as text before.** Self-hosted and relayed Chat-format models + (DeepSeek, GLM, Qwen, Llama and others) sometimes write a tool call into + `content` in the template they were trained on instead of answering with + `tool_calls`. On a route that converts the request to the Chat format, + such a call to a tool the request defined now reaches the caller as a + structured tool call, which the caller executes, where it used to be a + tagged piece of prose the caller showed and did nothing with. Tool-call + inspection sees these calls like any other and refuses or cuts them under + the same rules; the request log's response body records them as tool + calls, and a recovered call ends the answer with the tool-use stop reason + instead of `stop`. A request that defined no tools, a name the request + did not define, and a bare JSON object in prose stay text. A Chat caller + on a Chat-format route is forwarded as sent and unchanged. No setting, + schema, API route, Helm value or Redis key changes. + +### Fixed + +- **Gemini thought signatures are not counted as answer text.** When a + request's usage has to be estimated (`usage_estimated`), the + `thoughtSignature` on a Gemini thought or function call — an opaque blob + of up to several kilobytes — was counted as output text; it is left out + now, as the other formats' reasoning signatures were. + +### Changed + +- thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard) + v0.68.0 → v0.69.0. Only tw-dialect changes: + - **Reasoning from a Chat-format upstream round-trips** (tw-dialect). + Reasoning a Chat-format upstream returned (`reasoning_content`, or + `reasoning` as OpenRouter and vLLM call it) was dropped from the next + turn's request on every route, so a Claude Code or Codex conversation + with tools against DeepSeek, which requires the reasoning of a + tool-calling turn on the following requests of that turn, lost it every + turn. Such reasoning is now carried in the caller's own format with a + `tw1.ch.` signature, read back on the next turn, and written as + `reasoning_content` only to a Chat-format upstream; to an upstream of + another format it is left out as before, and reasoning another vendor + signed is still not sent to a Chat-format upstream. Consecutive + assistant messages are merged before a request is written in the Chat + format, so a Responses caller's reasoning, text and tool call of one + turn arrive as one message. The reasoning history a request carries + back counts as its input, so the input tokens of those turns grow by + it, as the upstream charges them. + - **Gemini thought signatures on function calls are kept** (tw-dialect). + Gemini with thinking puts a turn's thought signature on the first + `functionCall` part, and the conversion only read signatures from + thought parts, so a converted Gemini tool loop lost the signature and + with it the model's reasoning continuity. The signature on a function + call is now carried through a conversion and written back on the + `functionCall` part; a call with no real signature gets the + `skip_thought_signature_validator` placeholder as before. A Gemini + request forwarded as sent that carries a signature from another route + on a `functionCall` part keeps the part with the placeholder, where the + part was removed and the tool loop broke. + - **Tool calls written into a Chat-format upstream's answer text are + recognised** (tw-dialect): see Read before upgrading. In this edition + the answer assembled for the request log and the cache of a Chat + caller's stream forwarded as sent to a Chat-format upstream keeps such + a call as the text the caller received, rather than the structured call + the conversion would have made of it. + ## [3.3.1] — 2026-10-10 Streams are logged as they ended. A caller that hangs up once it has read @@ -1696,7 +1768,8 @@ unreleased builds should: stop the gateway, run `db/schema.sql` against PostgreSQL, restart against this tag. The schema is idempotent end-to-end, so the apply is safe to repeat. -[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.3.1...HEAD +[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.4.0...HEAD +[3.4.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.4.0 [3.3.1]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.3.1 [3.3.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.3.0 [3.2.1]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.2.1 diff --git a/Cargo.lock b/Cargo.lock index 61fbf483..7f2b54fb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4189,7 +4189,7 @@ dependencies = [ [[package]] name = "think-watch-auth" -version = "3.3.1" +version = "3.4.0" dependencies = [ "anyhow", "argon2", @@ -4219,7 +4219,7 @@ dependencies = [ [[package]] name = "think-watch-common" -version = "3.3.1" +version = "3.4.0" dependencies = [ "aes-gcm", "anyhow", @@ -4262,7 +4262,7 @@ dependencies = [ [[package]] name = "think-watch-gateway" -version = "3.3.1" +version = "3.4.0" dependencies = [ "anyhow", "arc-swap", @@ -4301,7 +4301,7 @@ dependencies = [ [[package]] name = "think-watch-mcp-gateway" -version = "3.3.1" +version = "3.4.0" dependencies = [ "anyhow", "arc-swap", @@ -4330,7 +4330,7 @@ dependencies = [ [[package]] name = "think-watch-server" -version = "3.3.1" +version = "3.4.0" dependencies = [ "anyhow", "arc-swap", @@ -4381,7 +4381,7 @@ dependencies = [ [[package]] name = "think-watch-test-support" -version = "3.3.1" +version = "3.4.0" dependencies = [ "anyhow", "async-stream", @@ -4837,8 +4837,8 @@ dependencies = [ [[package]] name = "tw-bedrock" -version = "0.68.0" -source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.68.0#0955f9906c88f9c8fba99a17e3898efa23c952b9" +version = "0.69.0" +source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.69.0#13d039d812f39445d44f59b46d33845715bb33ed" dependencies = [ "aws-credential-types", "aws-sigv4", @@ -4852,16 +4852,16 @@ dependencies = [ [[package]] name = "tw-breaker" -version = "0.68.0" -source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.68.0#0955f9906c88f9c8fba99a17e3898efa23c952b9" +version = "0.69.0" +source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.69.0#13d039d812f39445d44f59b46d33845715bb33ed" dependencies = [ "serde", ] [[package]] name = "tw-dialect" -version = "0.68.0" -source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.68.0#0955f9906c88f9c8fba99a17e3898efa23c952b9" +version = "0.69.0" +source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.69.0#13d039d812f39445d44f59b46d33845715bb33ed" dependencies = [ "memchr", "serde", @@ -4870,8 +4870,8 @@ dependencies = [ [[package]] name = "tw-guard" -version = "0.68.0" -source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.68.0#0955f9906c88f9c8fba99a17e3898efa23c952b9" +version = "0.69.0" +source = "git+https://github.com/ThinkWatchProject/ThinkWatch-Core.git?tag=v0.69.0#13d039d812f39445d44f59b46d33845715bb33ed" dependencies = [ "base64 0.22.1", "bytes", diff --git a/Cargo.toml b/Cargo.toml index 67ed8a67..bf8c74b3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,7 +10,7 @@ members = [ ] [workspace.package] -version = "3.3.1" +version = "3.4.0" edition = "2024" # The MSRV: the newest `rust-version` among the locked dependencies # (sqlx 0.9 and the aws-smithy crates under aws-sigv4). Without it, @@ -70,10 +70,10 @@ opt-level = 3 # never re-exported through a local shim. And the reverse: something only # this side uses (the at-rest crypto, IMDSv2 credentials, the gateway error) # lives here, not in core. -tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.68.0" } -tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.68.0" } -tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.68.0" } -tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.68.0" } +tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.69.0" } +tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.69.0" } +tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.69.0" } +tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.69.0" } # Web framework axum = { version = "0.8", features = ["macros", "ws"] } diff --git a/crates/gateway/src/proxy/generate.rs b/crates/gateway/src/proxy/generate.rs index e109ad81..90aa7600 100644 --- a/crates/gateway/src/proxy/generate.rs +++ b/crates/gateway/src/proxy/generate.rs @@ -357,6 +357,13 @@ impl Outbound { /// compaction would be recorded as a failed one. A request the /// conversion layer cannot read at all can still be forwarded, and its /// answer is assembled knowing only the model and whether it streams. + /// + /// A Chat upstream can write a tool call into its answer text instead + /// of `tool_calls`. A converted answer turns such a call, when the + /// request defined the tool, into a structured one (see + /// `tw_dialect::chat::text_calls`); a forwarded answer reaches the + /// caller as the upstream wrote it, text included, and is assembled as + /// that (`Session::keep_text_calls`). fn forwarded_session(&self, body: &Value, model: &str, target: &Target) -> Session { let client = self.surface.dialect; let request = @@ -368,7 +375,9 @@ impl Outbound { ..Default::default() }, }; - tw_dialect::convert::encode(&request, target).session + tw_dialect::convert::encode(&request, target) + .session + .keep_text_calls() } /// Address the request to `protocol`, naming `model` upstream. @@ -1273,6 +1282,63 @@ mod tests { assert!(converted.convert.as_ref().unwrap().is_compaction()); } + /// A Chat upstream that writes a tool call into its text: a converted + /// answer carries it as a structured call, a forwarded one is assembled + /// as the caller received it — text. + #[test] + fn a_tool_call_written_into_text_is_assembled_as_the_caller_received_it() { + const STREAM: &str = concat!( + "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,", + "\"delta\":{\"role\":\"assistant\",\"content\":\"{\\\"name\\\": \\\"ls\\\", ", + "\\\"arguments\\\": {\\\"path\\\": \\\"/\\\"}}\"}}]}\n\n", + "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,", + "\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n", + "data: [DONE]\n\n" + ); + let tools = serde_json::json!([{ + "type": "function", + "function": {"name": "ls", "parameters": {"type": "object"}} + }]); + let assemble = |session: &Session| -> Value { + let mut c = session.collector(); + c.process(STREAM.as_bytes()); + serde_json::from_slice(&c.finish().unwrap()).unwrap() + }; + + let ask = serde_json::json!({ + "model": "m", "stream": true, "tools": tools, + "messages": [{"role": "user", "content": "list /"}] + }); + let forwarded = outbound(CHAT, "/v1/chat/completions", ask, None) + .address(UpstreamProtocol::OpenAiChat, "m", false) + .unwrap_or_else(|e| panic!("{e:?}")); + assert!(forwarded.convert.is_none()); + let message = &assemble(&forwarded.collect)["choices"][0]["message"]; + assert!( + message["content"].as_str().unwrap().contains(""), + "{message}" + ); + assert!(message.get("tool_calls").is_none(), "{message}"); + + let ask = serde_json::json!({ + "model": "m", "stream": true, "max_tokens": 100, + "tools": [{"name": "ls", "input_schema": {"type": "object"}}], + "messages": [{"role": "user", "content": "list /"}] + }); + let converted = outbound(MESSAGES, "/v1/messages", ask, None) + .address(UpstreamProtocol::OpenAiChat, "m", false) + .unwrap_or_else(|e| panic!("{e:?}")); + let content = &assemble(converted.convert.as_ref().unwrap())["content"]; + assert!( + content + .as_array() + .unwrap() + .iter() + .any(|b| b["type"] == "tool_use" && b["name"] == "ls"), + "{content}" + ); + } + /// Only breakpoints the conversion added may be taken out when an /// upstream refuses them; the caller's own are its decision. #[test] diff --git a/crates/gateway/src/usage_estimate.rs b/crates/gateway/src/usage_estimate.rs index 8f6c669b..5c43297f 100644 --- a/crates/gateway/src/usage_estimate.rs +++ b/crates/gateway/src/usage_estimate.rs @@ -91,6 +91,8 @@ fn text_len(v: &Value, skip: &[&str]) -> usize { "call_id", "tool_call_id", "signature", + "thoughtSignature", + "thought_signature", "encrypted_content", ]; match v { @@ -175,6 +177,15 @@ mod tests { ], }); assert_eq!(answer_tokens(body.to_string().as_bytes()), 2); + // Gemini's, on a thought and on the function call it led to. + let body = json!({ + "candidates": [{"content": {"parts": [ + {"text": "abcd", "thought": true, "thoughtSignature": "x".repeat(4000)}, + {"functionCall": {"name": "ls", "args": {}}, "thoughtSignature": "x".repeat(4000)}, + {"functionCall": {"name": "ls", "args": {}}, "thought_signature": "x".repeat(4000)} + ]}}], + }); + assert_eq!(answer_tokens(body.to_string().as_bytes()), 2); } #[test] diff --git a/deploy/helm/think-watch/Chart.yaml b/deploy/helm/think-watch/Chart.yaml index c53611cb..7506d2cf 100644 --- a/deploy/helm/think-watch/Chart.yaml +++ b/deploy/helm/think-watch/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: think-watch description: Enterprise AI API Gateway & MCP Management Platform type: application -version: 3.3.1 -appVersion: "3.3.1" +version: 3.4.0 +appVersion: "3.4.0" keywords: - ai - gateway diff --git a/web/package.json b/web/package.json index 70e57c97..90048baa 100644 --- a/web/package.json +++ b/web/package.json @@ -1,7 +1,7 @@ { "name": "web", "private": true, - "version": "3.3.1", + "version": "3.4.0", "type": "module", "packageManager": "pnpm@12.9.1", "scripts": {