Skip to content

fix: caller-controlled scope IDs grow Runtime caches without bound - #1325

Draft
thunguo wants to merge 3 commits into
oceanbase:masterfrom
thunguo:fix/runtime-cache
Draft

fix: caller-controlled scope IDs grow Runtime caches without bound#1325
thunguo wants to merge 3 commits into
oceanbase:masterfrom
thunguo:fix/runtime-cache

Conversation

@thunguo

@thunguo thunguo commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Closes #1322.

Rationale for this change

Caller-controlled scope_id values could permanently populate several process-level Runtime and relational-provider caches. A long-running HTTP or MCP Server therefore exhibited linear memory growth when receiving valid requests with many one-off scope IDs, even when no Source or Memory data was persisted.

The Runtime needs a bounded lifecycle for inactive scope-local objects while preserving per-scope serialization for operations that currently hold or are waiting for a lock.

What changes are included in this PR?

  • Add a lease-aware LRU cache for scope-local Runtime resources.
    • Retain up to 128 inactive scopes by default.
    • Never evict a scope while an operation holds or waits for its serialization lock.
    • Allow concurrent active scopes to temporarily exceed the configured capacity, then trim the cache when operations complete.
  • Evict the corresponding relational context, source lock, activation lock, and experience lock together.
  • Clear cached scope resources when the Runtime closes.
  • Keep nested scoped operations within the same Runtime lifecycle so graceful shutdown waits for all related work.
  • Add RuntimeConfig.scope_cache_size and the POWERCONTEXT_SERVER_RUNTIME_SCOPE_CACHE_SIZE environment variable.
  • Add the low-cardinality Prometheus metric:
    • powercontext_server_runtime_scopes{state="active"}
    • powercontext_server_runtime_scopes{state="cached"}
  • Ensure metrics never use scope_id as a label.
  • Document the new configuration in the English and Chinese configuration references.
  • Add regression coverage for:
    • many one-off scope IDs through the public HTTP boundary;
    • cache convergence to the configured capacity;
    • lock holders and waiters remaining protected during cache pressure;
    • environment-based configuration.

Are there any user-facing changes?

Yes, all changes are additive:

  • RuntimeConfig.scope_cache_size defaults to 128.
  • Server deployments can configure it with POWERCONTEXT_SERVER_RUNTIME_SCOPE_CACHE_SIZE.
  • /metrics now exposes active and cached scope counts using a bounded state label.

There are no breaking HTTP, MCP, OpenAPI, or persisted-data changes. If an inactive scope is evicted, its ephemeral services are recomposed on the next request; persisted scope data is unaffected.

How was this change tested?

Targeted Runtime, provider, Server, and metrics tests:

.venv/bin/pytest -q \
  tests/builtin/runtime \
  tests/builtin/persistence/test_provider.py \
  tests/e2e/test_builtin_runtime.py \
  tests/test_server.py \
  tests/test_server_metrics.py

Result: 67 passed.

Repository validation:

UV_CACHE_DIR=/tmp/powercontext-uv-cache uv lock --locked
.venv/bin/prek run -a
.venv/bin/ty check
.venv/bin/zensical build -s
.venv/bin/python -m pytest --doctest-modules \
  --deselect tests/builtin/inference/test_pydantic_ai.py::test_instrumented_generation_spans_exclude_schema_retry_content

Result: 464 passed, 7 skipped, 1 deselected.

The deselected inference tracing test was also run independently and fails because the current Pydantic AI instrumentation includes schema-retry content in span attributes. No inference or tracing implementation is changed by this PR.

Manual validation sent 250 successful POST /v1/context/prepare requests using distinct scope IDs:

  • Before: Runtime locks, contexts, source locks, and activation locks each grew to 250.
  • After: cached Runtime scopes and the corresponding provider resources converged to the default limit of 128, with 0 active scopes after the requests completed.

AI usage statement

Copilot AI lite review requested due to automatic review settings August 23, 2026 09:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a bounded, lease-aware LRU cache for scope-local built-in Runtime resources to prevent unbounded memory growth when callers use many one-off scope_id values (especially via long-running HTTP/MCP server processes). It also adds configuration, metrics, documentation, and regression tests around the new behavior.

Changes:

  • Add a lease-aware ScopeCache to bound inactive scope-local Runtime resources (default 128) and evict corresponding relational/provider scope resources together.
  • Add RuntimeConfig.scope_cache_size (and env-based configuration via existing nested settings) plus /metrics reporting for active/cached scope counts without using scope_id labels.
  • Add regression tests covering one-off scope IDs, convergence to capacity, lock waiter/holder safety, and environment-driven configuration.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/test_server.py Verifies server settings load runtime.scope_cache_size from environment.
tests/test_server_metrics.py Adds regression test ensuring one-off scope IDs don’t produce unbounded Runtime scope retention and that metrics avoid scope_id labels.
tests/builtin/runtime/test_scope_cache.py Adds targeted concurrency/eviction regression test for the scope lease + lock interaction under cache pressure.
src/powercontext/server/metrics.py Adds powercontext_server_runtime_scopes{state=...} gauge and setter.
src/powercontext/server/factory.py Wires Runtime scope cache observer into server metrics.
src/powercontext/builtin/runtime/relational.py Adds evict() to drop per-scope cached contexts and locks together.
src/powercontext/builtin/runtime/config.py Adds RuntimeConfig.scope_cache_size with default and validation.
src/powercontext/builtin/runtime/composition.py Plumbs scope_cache_size, evictor, and observer into BuiltinRuntime composition.
src/powercontext/builtin/runtime/application.py Integrates ScopeCache, ensures Runtime close clears scope cache, and refines operation lifecycle tracking for nested operations.
src/powercontext/builtin/runtime/_scope_cache.py New lease-aware, bounded LRU scope cache implementation with eviction hooks and observer support.
docs/en/docs/reference/configuration.md Documents POWERCONTEXT_SERVER_RUNTIME_SCOPE_CACHE_SIZE.
docs/zh/docs/reference/configuration.md Documents POWERCONTEXT_SERVER_RUNTIME_SCOPE_CACHE_SIZE.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/powercontext/builtin/runtime/_scope_cache.py Outdated
Comment on lines +63 to +68
def observe(cached: int, active: int) -> None:
nonlocal same_scope_leases
if cached == 1 and active == 1:
same_scope_leases += 1
if same_scope_leases == 2:
both_same_scope_operations_started.set()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 当前 Head 仍未修复这个等待条件。ScopeCacheCounts.cached 现在只统计 inactive entries,因此两个 same-scope operation 都持有 lease 时观察值是 cached=0, active=1;这里等待的 cached=1, active=1 永远不会出现,both_same_scope_operations_started.wait() 没有超时并会挂住测试。请在修复 _locked() 的 P0 后同步更新 predicate(或改用直接的 lease/lock 同步信号),并给等待增加有界超时,避免回归再次把测试任务永久挂起。

thunguo and others added 2 commits August 23, 2026 17:58
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@thunguo
thunguo marked this pull request as draft August 23, 2026 10:26
yield context

def _lock(self, scope_id: str) -> asyncio.Lock:
return self._scope_cache.lock(validate_scope_id(scope_id))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P0] 删除 _locks_locked() 仍在访问它

这里把 scope lock 存储切到了 _scope_cache,但紧接着的 _locked() 仍执行 self._locks.setdefault(...)。exact Head 的公开 HTTP 用例在第一个 POST /v1/context/prepare 就稳定抛出 AttributeError: BuiltinRuntime has no attribute _locks;prepare、Memory 写入、Review、Skill 等使用 _locked() 的路径都会被阻断。请让 _locked() 在当前 scope lease 内取得 _scope_cache.lock()(并保留 tracing/acquire/release 语义),再用真实 HTTP 入口补回归测试。

@@ -0,0 +1,135 @@
"""Bounded lifecycle for scope-local Runtime resources."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 两个新增 Python 文件缺少许可证头

exact Head 的 License Check 已明确报告 invalid: 2,缺失文件是本文件和 tests/builtin/runtime/test_scope_cache.py,因此当前强制门禁退出 1。请按仓库现有 Python 文件格式在两个文件顶部补齐 Apache 2.0 许可证头,并重新运行 License Check。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: caller-controlled scope IDs grow Runtime caches without bound

3 participants