Research action-conditioned mid-run memory recall

[sentry-junior]

Automatic memory recall today only runs at turn start from the latest user text. During multi-step tool loops, useful procedural or episodic memories often become relevant only after the agent proposes an action or sees a tool result, and relying on the model to call memory tools for that would create a lot of noise.

Current behavior

• Automatic recall is wired through the memory plugin userPrompt hook (packages/junior-memory/src/plugin.ts, packages/junior-memory/src/recall.ts).
• The recall query is the current user text (ctx.text): hybrid lexical + vector retrieval, personal-scope probes, then a relevance model that admits at most five memories into prompt context.
• Explicit searchMemories remains model-invoked and is separate from automatic recall (packages/junior-memory/README.md).
• Passive extraction already reviews completed sessions asynchronously; this issue is about recall timing, not write/capture quality.

Gap

In long tool-using runs, the information that would make memory useful often appears mid-loop:

• before a consequential tool/action, prior procedure/preferences for how that kind of work is usually done
• after a tool result or error, prior episodes that explain the observation or the next recovery step

Turn-start recall keyed only on the original user utterance often misses that. Pure tool-based recall forces the agent to decide when to search memory, which tends toward either under-use or many low-value memory calls.

Prior art

Closest work is observation/action-conditioned retrieval inside the agent decision loop, not richer turn-start query rewriting:

• Voyager — retrieves skills from task/environment state and folds execution feedback into later decisions without a separate user-facing memory request each time
• Reflexion — stores verbal episodic feedback and reuses it on later attempts/steps
• CoALA — frames memory retrieval as part of the observe → decide → act cycle (with lineage to ACT-R / Soar production retrieval)
• Adjacent but different: MemGPT/Letta archival tools (model-directed pull), ChatGPT dreaming / sleep-time memory synthesis (background state refresh), turn-start profile/core-memory pinning

General-purpose assistants still mostly ship turn-start retrieval or explicit memory tools; mid-loop automatic recall appears less productized.

Research questions

• Which mid-run signals are high enough value to justify automatic recall (proposed tool/action, errorful results, first use of a consequential tool, phase changes) versus noise on routine successful results?
• What should form the retrieval query once the trigger is not just the original user text (active goal, tool identity/args, observation, bounded recent trajectory)?
• How should mid-run recalls interact with turn-start recalls: dedupe by memory id, per-run budget, relevance gate reuse, cost/event accounting?
• When is automatic injection better than keeping memory as an explicit tool the model can choose?

Related

• Show recalled memories on transcript messages #1074 — surface recalled memories on transcript messages
• feat(memory): add recall-layer relevance gate to filter low-relevance vector matches #679 — recall-layer relevance gating (turn-start path)

Requested by David Cramer.

--

View Junior Session [Sentry]