Podcast ingest pipeline + RRF/MMR/SKOS retrieval stack + kernel perfo…

[shitcoinsherpa]

…rmance bake

A 100-file, +3,950/-1,661 LoC change across the kernel, knowledge subsystem,
agent loop, bots, voice, and infra layers. Five themes:

1. Podcast ingestion end-to-end — TOOL_PODCAST tool, faster-whisper + pyannote.audio diarization, RSS/iTunes host & guest resolution, 7-stage MEDIA_REVIEW pipeline that produces structured KB articles per episode.

2. Retrieval stack overhaul — RRF (Reciprocal Rank Fusion) over BM25 + nomic + qwen with adaptive per-query weights; MMR diversification; HYBRID_SEARCH and KB_NEAREST kernel instructions; auto-RAG injection inside ASSIST/ESCALATE so agent answers ground in the corpus instead of hallucinating.

3. SKOS-style hierarchical tagging + typed link graph — 71-concept hand-built skos_hierarchy.json with broader/narrower/related closures (precomputed via Kahn's topo sort, O(1) lookups), markdown-aware inline #hashtag extraction, typed-edge LinkStore with kind+score+metadata sidecar, MMR-aware auto-link with hard cap.

4. Kernel performance bake — vectorized topic-centroid scoring (~50× via single BLAS GEMV), KB embedding-matrix cache + lazy-rebuild (~50× faster auto_link_entry at N≈1000), DisplacementLog _response_cache, NumPy (S×T) broadcast for diarization speaker-segment overlap, MMR rewritten as a single (N,N) similarity matmul with pre-normed vectors, per-gate reflex-arc diagnostics.

5. Bootstrap + operational hardening — run.bat/run.sh install ComfyUI, Ollama, GPU-aware torch wheels via gpu_detect; gpu_coordinator manages VRAM lease + Ollama eviction during transcription; orchestrator hard wallclock cap (orchestrator_hard_cap_minutes, default 30); dynamic-bot delegation (perceive_manifest + act_agent_session + spawn_from_script); encrypted token store (leos_secrets) + Settings credentials UI; tool_bridge dict→json normalization; reflection prompts split into lesson (gap introspection) and insight (content engagement) modes.

What's New

Knowledge — rrf_search.py, mmr.py, raptor_tree.py,
atomic_extractor.py, hashtag_extraction.py, links.py, skos.py,
topic_vocabulary.py, transcript_segmenter.py

Agent / Bots — bots/kb_reflection_listener.py,
subsystems/kb_void_scheduled.py, scripted-bot delegation primitives
(perceive_manifest, act_agent_session)

Infra — gpu_coordinator.py, gpu_detect.py, leos_secrets.py,
server_routes_credentials.py

Tools — tools/kb_nearest.py, tools/podcast.py

Pages — page_podcasts.html (search/feed/ingest UI),
Settings page extended for HF token entry

Maintenance scripts — _system-maintenance/{batch_all, dedup_records,probe_corpus,retag_relink}.py

Seed data — data/skos_hierarchy.json, infra/decision_trees/trees.json,
references/references.json

Notable Behavior Changes

• KnowledgeBase.search() now routes through _search_rrf when LEOS_RRF_SEARCH=1 (default). Falls back to the linear-blend hybrid on any error.
• qwen retrieval/re-ranking is opt-in via LEOS_RRF_QWEN=1 (default off) pending index-time qwen embedding generation; cold-cache pass blocked startup otherwise.
• Auto-link cap (MAX_AUTO_RELATED=8) and diversity weight (MMR_DIVERSITY=0.4) eliminate same-show clustering.
• Topic auto-tag threshold raised to 0.50 with a relative-margin filter and tiered fallback (0.50 → 0.45 → 0.40 single-tag).
• kernel.execute() envelope uses setdefault so handler ok=False responses are no longer silently overwritten.
• Reflection on KB save defaults to experience_type='insight' so monologues engage with article substance (facts, claims, themes) rather than narrating the act of saving.

Configuration Knobs

Key	Default	Purpose
LEOS_RRF_SEARCH	1	Enable RRF fusion in kb.search
LEOS_RRF_QWEN	0	Add qwen as third RRF retriever
orchestrator_hard_cap_minutes	30	Hard wallclock ceiling on plan runs
topic_llm_fallback	false	LLM-assisted tagging when cosine cascade returns empty

Migration Notes

• First boot rebuilds data/topic_centroids.json from the new SEED_TOPICS phrases; older cache is preserved as topic_centroids.json.old for rollback.
• links.jsonl is created on demand under data/; existing article.related[] arrays remain authoritative until the next auto_link_entry pass refreshes both views.
• Existing KB articles benefit from the new tag vocabulary + SKOS-aware auto-link via python -m _system-maintenance.retag_relink.

Verification

• _system-maintenance/probe_corpus.py runs the corpus-wide retrieval
  • agent-chat smoke test. All five retrieval paths return correct results in <1 s on a 50-article test corpus; ASSIST grounds answers in cited references.
• _system-maintenance/test_review.py exercises the 7-stage media review pipeline end-to-end.