Feat/provider free compaction

AGENTS.md

@@ -26,6 +26,7 @@ When parts of this file is stale after your work, update this file.
 | Portable verified-row, source-capability, Go-lexical, and shell-header semantics | `internal/verifiedrow`, `internal/sourcekind`, `internal/golex`, `internal/shellsyntax` |
 | Versioned plugin shared-core adapter and private WASM bridge | `internal/router/toolplugin/core-v1.mjs`, `internal/router/toolplugin/core-v1.d.ts`, `internal/sharedwasm` |
 | Router lifecycle, launch flags, modes, and HTTP endpoints | `internal/router/server.go`, `internal/router/flags.go` |
+| Provider-free context pruning, local envelopes, and native-history restoration | `internal/router/context_compaction*.go` |
 | Third-party native subagent projection, Grok authentication/translation, and model-catalog metadata | `internal/router/subagent_bridge.go`, `internal/router/grok_*.go` |
 | Codex-facing WebSocket sessions, incremental history, and steering | `internal/router/server_websocket.go` |
 | Codex authentication and upstream Responses transport | `internal/router/client.go`, `internal/router/client_websocket.go` |
@@ -47,6 +48,7 @@ When parts of this file is stale after your work, update this file.
 | --- | --- |
 | Root engine | `go test .` |
 | Router request, response, recovery, workspace, plugin, or transport | `go test ./internal/router` |
+| Context compaction client compatibility | `MEKUGI_COMPACTION_CODEX_BIN="$(command -v codex)" go test ./internal/router -run '^TestCompactionInstalledCodex$'` (isolated loopback fixtures, no provider inference) |
 | Portable core or `mekugi:core/v1` adapter | `go generate ./internal/router/toolplugin`, then `go test ./...` and `bun test ./internal/router/toolplugin/tests/core.test.ts` |
 | TypeScript plugin source | `go generate ./internal/router/toolplugin`, then `bun test ./internal/router/toolplugin/tests` |
 | Router or shell-helper process entry point | `go test ./cmd/mekugi ./cmd/shell` |

README.md

@@ -147,6 +147,54 @@ go install github.com/yusing/mekugi/cmd/mekugi@latest \
 Add `$GOBIN`, or `$(go env GOPATH)/bin` when unset, to the `PATH` used by both
 Mekugi and Codex. The fixed `shell` helper must be available to Codex's executor.
 
+Context compaction is handled locally, without a provider-generated summary.
+Codex still decides when to compact using your settings. Mekugi preserves the
+retained native history in an encrypted item and restores it before the next
+provider request. The first compaction creates an owner-only key at
+`$XDG_CONFIG_HOME/mekugi/compaction.key` (normally
+`~/.config/mekugi/compaction.key` on Linux). Keep that key to resume compacted
+sessions, including when moving them to another installation.
+
+Compaction first discards unmarked historical details from older finished operations,
+even while the task is still open. It keeps factual execution records, requests,
+visible decisions, diagnostic excerpts, referenced evidence, and recent/live work.
+Terminal results from recognized direct `go test` calls are an exception: even when
+recent or referenced, they retain only pass/fail status and reported failed test
+names, while detailed runner output is discarded. Recognized applied patch bodies
+and associated older opaque reasoning can also be retired. Other older failed-command
+output can lose unreferenced bulk while retaining errors, warnings, and provenance.
+Truncated or oversized documentation can also lose unreferenced bulk; a single
+oversized evidence line retains bounded excerpts. Discarded details are not currently
+retrievable through Mekugi.
+
+The retained history targets 50,000 visible-string tokens, with at most 30,000
+tokens of overshoot. If the initial reductions cannot fit, a budget-first pass
+keeps prioritized excerpts and drops older material regardless of tool or content
+format. Developer/system/model instructions and canonical `AGENTS.md` context
+remain intact. The latest user request and recent/live execution evidence come
+next, but user messages and live output can be excerpted when oversized.
+Repeated text can collapse to representative occurrences with an omission count.
+The selector reserves space for the request chain and discussion before execution
+bulk, with an initial preference for the latest assistant or agent report.
+Unused shares are redistributed as small items fit. It need not fill the
+50,000-token target.
+This is deliberately lossy: it cannot guarantee that every omitted detail is
+unimportant. The model receives explicit omission markers, not an invented summary.
+
+During compaction, ordinary historical images become `[Image]` placeholders,
+keeping the surrounding text. This deliberately discards visual details rather
+than assuming earlier reasoning captured them. Fresh images on normal turns and
+mandatory instruction images remain unchanged. Image data is not tokenized as
+text, and historical images no longer need a separate allocation budget.
+
+Original user/agent messages needed to reconcile Codex's carried history remain
+encrypted in the capsule, but are not returned to the model or available through
+a retrieval tool. The budget measures the restored history, not ciphertext size
+or complete provider input; fresh instructions and later messages add to it.
+Required instructions alone exceeding 80,000 tokens, invalid input, unreadable
+envelopes, or a no-op on an already-small history can still produce an error.
+Mekugi never falls back to a provider summary.
+
 Then launch:
 
 ```sh
@@ -191,11 +239,13 @@ down when Codex exits. Multiple sessions can run independently. Codex handles
 terminal Ctrl-C, and its exit status is preserved.
 
 The wrapper uses the fixed Codex ChatGPT upstream and overrides provider
-selection for that invocation only. Standalone serving, fixed ports, custom
-providers, and provider-selection arguments such as `--oss` are not supported.
-It also forces `include_collaboration_mode_instructions=false` for the invocation,
-so Codex does not inject collaboration-mode instructions, even if enabled in your
-config or command-line overrides. No configuration files are changed.
+selection for that invocation only. Codex displays this provider as `OpenAI`;
+its requests still go through the private Mekugi router. Standalone serving,
+fixed ports, custom providers, and provider-selection arguments such as `--oss`
+are not supported. It also forces
+`include_collaboration_mode_instructions=false` for the invocation, so Codex
+does not inject collaboration-mode instructions, even if enabled in your config
+or command-line overrides. No configuration files are changed.
 
 The wrapper enables WebSockets between Codex and Mekugi for that invocation,
 without changing Codex configuration. Mekugi keeps the ChatGPT connection open
@@ -209,6 +259,9 @@ HTTP/SSE clients and can fall back to HTTP for those requests when ChatGPT
 explicitly rejects the WebSocket upgrade. It never silently replays a dropped
 request or accepted steering. Grok provider requests remain on HTTP.
 
+See the [context-compaction contract](doc/spec/compaction.md) for supported forms
+and preservation behavior. No compaction threshold or scope is overridden.
+
 ### Options
 
 | Flag | Default | Purpose |
@@ -224,14 +277,14 @@ request or accepted steering. Grok provider requests remain on HTTP.
 | `--metrics-output PATH` | Disabled | Write the final metrics snapshot on shutdown, overwriting the destination |
 | `--debug` | Disabled | Record diagnostics, capture, metrics, patched instructions, runtime reads, and an AX report; print all artifact paths on exit |
 
-For a transport-only session:
+To disable Mekugi tool and model-string transformations:
 
 ```sh
 mekugi --mode passthrough codex
 ```
 
 Passthrough does not load the plugin registry, so it does not require Node.js or
-plugin grammar validation. Capture remains available.
+plugin grammar validation. Local context compaction and capture remain available.
 
 ### Grok subagents
 
@@ -680,6 +733,34 @@ For focused checks, use `go test .` for the engine,
 `go test ./internal/router` for routing, or
 `go test ./cmd/mekugi ./cmd/shell` for process entry points.
 
+To inspect compaction loss using an installed Codex client:
+
+```sh
+MEKUGI_COMPACTION_CODEX_BIN="$(command -v codex)" go test ./internal/router -run '^TestCompactionInstalledCodex$' -count=1 -v
+```
+
+These loopback fixtures make no provider inference requests. Pressure cases log
+per-item token counts, retention decisions, and the next request's token count.
+They check specified continuation facts and client compatibility, not model
+reasoning quality on real task histories.
+
+For an offline replay of an existing session's first recorded compaction window,
+use the rollout check. The optional audit directory receives original/retained
+history and per-item diagnostics, so it must already exist with owner-only
+permissions. Original session files are read-only; no historical tool calls or
+provider inference are executed.
+
+```sh
+compaction_audit_dir="$(mktemp -d)"
+MEKUGI_COMPACTION_ROLLOUT="/absolute/path/rollout.jsonl" \
+MEKUGI_COMPACTION_AUDIT_DIR="$compaction_audit_dir" \
+go test ./internal/router -run '^TestCompactionRolloutReplay$' -count=1 -v
+```
+
+The replay measures recorded items, not omitted model instructions or request/tool
+framing. Passing its budget/restoration checks does not establish task-fact retention;
+inspect the exported before/after histories for that assessment.
+
 ## License
 
 MIT. See [LICENSE](LICENSE).

cmd/mekugi/wrap.go

@@ -132,15 +132,24 @@ func wrapCodex(ctx context.Context, routerArgs, args []string) (code int, runErr
 }
 
 func codexArgs(baseURL string, args []string) []string {
-	// Keep overrides in the final command's config layer: Codex subcommands
+	// Keep router overrides in the final command's config layer: Codex subcommands
 	// can replace pre-subcommand -c settings with their own. Never cross --.
 	index := slices.Index(args, "--")
 	if index < 0 {
 		index = len(args)
 	}
+	// Codex gates remote compaction on the OpenAI provider identity. Keep that
+	// identity for our fixed ChatGPT upstream so manual and automatic compact
+	// requests use the local compact interfaces instead of model summaries.
+	// Source: Codex model-provider/src/provider.rs ConfiguredModelProvider::capabilities.
+	// Disable both the feature toggle and final config value: Codex merges them
+	// in different layers for the TUI and subcommands.
+	// The local JSON boundary does not accept Codex's ChatGPT Zstd request bodies.
 	return slices.Insert(slices.Clone(args), index,
 		"-c", `model_provider="mekugi_wrap"`,
-		"-c", fmt.Sprintf(`model_providers.mekugi_wrap={name="mekugi",base_url=%q,wire_api="responses",requires_openai_auth=true,supports_websockets=true}`, baseURL),
+		"-c", fmt.Sprintf(`model_providers.mekugi_wrap={name="OpenAI",base_url=%q,wire_api="responses",requires_openai_auth=true,supports_websockets=true}`, baseURL),
+		"--disable", "enable_request_compression",
+		"-c", `features.enable_request_compression=false`,
 		"-c", `include_collaboration_mode_instructions=false`,
 	)
 }

cmd/mekugi/wrap_test.go

@@ -24,12 +24,16 @@ func TestCodexArgsPreservesArguments(t *testing.T) {
 	forwarded := []string{"exec", "-c", "model=\"example\"", "--", "a prompt with spaces"}
 	args := codexArgs("http://127.0.0.1:12345/v1", forwarded)
 	index := slices.Index(forwarded, "--")
-	if !slices.Equal(args[:index], forwarded[:index]) || !slices.Equal(args[index+6:], forwarded[index:]) {
+	if !slices.Equal(args[:index], forwarded[:index]) || !slices.Equal(args[index+10:], forwarded[index:]) {
 		t.Fatalf("forwarded arguments changed: %q", args)
 	}
+	if !slices.Equal(args[index+4:index+6], []string{"--disable", "enable_request_compression"}) {
+		t.Fatal("wrapped requests must disable the compression feature toggle")
+	}
 	var config struct {
-		IncludeCollaborationModeInstructions *bool  `toml:"include_collaboration_mode_instructions"`
-		ModelProvider                        string `toml:"model_provider"`
+		Features                             map[string]bool `toml:"features"`
+		IncludeCollaborationModeInstructions *bool           `toml:"include_collaboration_mode_instructions"`
+		ModelProvider                        string          `toml:"model_provider"`
 		Providers                            map[string]struct {
 			Name       string `toml:"name"`
 			BaseURL    string `toml:"base_url"`
@@ -39,7 +43,10 @@ func TestCodexArgsPreservesArguments(t *testing.T) {
 		} `toml:"model_providers"`
 	}
 	var settings []string
-	for i := index; i < index+6; i += 2 {
+	for i := index; i < index+10; i += 2 {
+		if i == index+4 {
+			continue
+		}
 		if args[i] != "-c" {
 			t.Fatalf("not a config override: %q", args)
 		}
@@ -51,8 +58,11 @@ func TestCodexArgsPreservesArguments(t *testing.T) {
 	if config.IncludeCollaborationModeInstructions == nil || *config.IncludeCollaborationModeInstructions {
 		t.Fatalf("collaboration mode instructions not disabled: %q", args)
 	}
+	if enabled, set := config.Features["enable_request_compression"]; !set || enabled {
+		t.Fatal("wrapped requests must remain uncompressed JSON")
+	}
 	provider := config.Providers[config.ModelProvider]
-	if provider.Name == "" || provider.BaseURL != "http://127.0.0.1:12345/v1" || provider.WireAPI != "responses" || !provider.Auth || !provider.WebSockets {
+	if provider.Name != "OpenAI" || provider.BaseURL != "http://127.0.0.1:12345/v1" || provider.WireAPI != "responses" || !provider.Auth || !provider.WebSockets {
 		t.Fatalf("provider = %+v", provider)
 	}
 	withoutDelimiter := []string{"exec", "-c", `model="example"`, "prompt"}