Skip to content

fix(tests): make mock retain extraction deterministic#1220

Draft
JulienJBO wants to merge 1 commit intovectorize-io:mainfrom
JulienJBO:codex/mock-llm-fact-extraction
Draft

fix(tests): make mock retain extraction deterministic#1220
JulienJBO wants to merge 1 commit intovectorize-io:mainfrom
JulienJBO:codex/mock-llm-fact-extraction

Conversation

@JulienJBO
Copy link
Copy Markdown

@JulienJBO JulienJBO commented Apr 23, 2026

Summary

  • generate deterministic sentence-level fact extraction output for the mock LLM when retain uses scope=retain_extract_facts
  • default the shared memory test fixture to the mock provider when no local LLM env is configured
  • add coverage for the new mock retain behavior

Why

Running hindsight-api-slim tests locally without a populated .env currently falls back to a real provider (groq) and fails early, or, when forced to mock, retain-heavy integration tests extract zero facts and produce large misleading failure clusters. This keeps the generic mock behavior intact while making local retain/recall integration tests deterministic and useful.

Validation

  • uv run pytest tests/test_per_operation_llm_config.py::TestMockLLMProvider::test_mock_provider_generates_facts_for_retain_extract_scope -q
  • uv run pytest tests/test_retain_append_mode.py::test_append_mode_concatenates_content -q
  • uv run pytest tests/test_tags_visibility.py -q
  • uv run ruff check hindsight_api/engine/providers/mock_llm.py tests/conftest.py tests/test_per_operation_llm_config.py

Follow-up

A broader local no-env test pass still has a second cluster around richer mock behaviors (consolidation, causal_relations, entity_labels, and zero-fact edge cases). I kept this PR scoped to the first large retain/recall failure wave.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant