Skip to content

Registered project graph-runtime bind is silently skipped when graph activation is deferred, permanently disabling Work/workflow/semantic-evaluation invocation surfaces #752

Description

@ScriptedAlchemy

Symptom

On the operator daemon (merged build 0.1.0-beta.37+087e6674286e, branch codex/tracedecay-total-redesign-plan-reopened, project /fast/projects/tracedecay), every daemon-protocol invocation that requires the registered project runtime fails typed-Unavailable for the entire daemon lifetime, while MCP tools and the catalogued application surface (tracedecay git status, tracedecay tool status, search) stay healthy.

Observable loop in the journal, repeating every ~5 s from project open until shutdown, across every daemon incarnation tonight (clean starts included, 705+ occurrences):

WARN tracedecay::daemon::service::invocation::work_blocked_interval_recovery: Work blocked-interval recovery scan failed error=registered Work database: database error: project graph runtime is not bound (operation: attach registered Work topology)
ERROR tracedecay::daemon::service::invocation::work::workflow_fan_out::recovery: workflow fan-out startup recovery authority failed error=Database { message: "project graph runtime is not bound", operation: "attach registered workflow topology" } stage="workflow_application_services"

Downstream, tracedecay-search-eval-direct evaluate-and-publish --project-root /fast/projects/tracedecay --profile hybrid-conservative fails instantly with semantic evaluation publication rejected: Unavailable, and qualify-native with project route error (semantic_qualification_unavailable) — even when fired inside a current|fresh|ready code-index window. No semantic_evaluation_admission events are logged, so the rejection is the silent front-door denial in dispatch (requires_project() && !project_runtime_admitted). Net effect: semantic retrieval can never be evaluated/activated on the operator profile, and the Work/workflow surfaces are dead.

Root cause

bind_verified_project_graph_runtime silently no-ops when the memory graph runtime is not yet available at full-upgrade composition time, and nothing ever retries the bind:

// crates/tracedecay/src/daemon/project_composition/runtime.rs
pub(super) async fn bind_verified_project_graph_runtime(
    database: Arc<tracedecay_runtime_core::db::Database>,
    sessions: &RegisteredGlobalDb,
) -> tracedecay_runtime_core::errors::Result<()> {
    let Some(graph_proxy) = database.memory_graph_runtime() else {
        return Ok(());          // <-- silent success on missing runtime
    };
    sessions.bind_project_graph_runtime(graph_proxy)...
}

Database::memory_graph_runtime() returns None until the graph OnceLock is set by (deferred) graph activation. Since the deferred-restore change kept core query lanes mountable during restore, composition on a busy profile consistently reaches this bind before graph activation completes, so the registered session DB permanently lacks the proxy. RegisteredGlobalDb::work_application_services / workflow_application_services then fail forever with "project graph runtime is not bound", and the Work recovery task retries the attach every 5 s — but nothing retries the bind.

This violates the typed-state rule directly: a missing runtime is reported as Ok(()).

Reproduction observed

  1. Daemon opens /fast/projects/tracedecay while the code index is mid-rebuild (this profile rebuilds ~20 min per worktree change, so this is the common case).
  2. full_published code_index=warming logs; MCP/catalog surfaces work.
  3. work_blocked_interval_recovery + workflow_fan_out error loop begins and never clears; semantic_evaluate_and_publish/semantic_qualify return Unavailable for the daemon's lifetime.

Confirmed after both kernel-OOM restarts and a clean systemctl --user stop/start (02:30 UTC start showed the loop from 02:30 onward).

Suggested direction

Bind a lazily-resolving proxy (resolve memory_graph_runtime at use time, returning the existing typed "not bound" state until activation completes), or re-run the bind from the graph-activation completion path. A silent return Ok(()) on a missing runtime should not survive either way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions