bm reindex --full completes its work but never exits, pinning one CPU core

[benjaminfrombe]

Summary

bm reindex --full finishes all of its work but never exits. The process stays alive and pins exactly one CPU core indefinitely, doing zero syscalls. It has to be killed manually.

Environment

• Basic Memory 0.23.0 (ghcr.io/basicmachines-co/basic-memory:latest, FastMCP 4.0.0b1)
• Docker, SQLite backend
• Semantic search enabled: fastembed / bge-small-en-v1.5, sqlite-vec vector index
• Corpus: 573 markdown files (3.6 MB), 575 entities, 174 observations, 994 relations, 14440 vector chunks

Steps to reproduce

basic-memory reindex --full

Expected

The command completes its work and exits.

Actual

All work completes successfully:

• embeddings: 14440 of 14440 chunks embedded
• search_index: 1743 rows (= 575 entities + 174 observations + 994 relations, consistent)
• log line: watch_service:handle_changes:342 - Event-index file change processing completed, processed_files=573, failed_files=0, skipped_files=0, total_indexed_files=6303, duration_ms=1721

After that the process keeps running at ~100% CPU (one full core) with no further progress and no further log output.

Evidence that it is a spin loop, not remaining work

/proc/<pid>/io sampled several minutes apart is byte for byte identical, while the process continues to accrue CPU time:

sample 1: rchar: 2416681816 wchar: 928888478 syscr: 571216 syscw: 388900 read_bytes: 692224 write_bytes: 866467840
sample 2: rchar: 2416681816 wchar: 928888478 syscr: 571216 syscw: 388900 read_bytes: 692224 write_bytes: 866467840

Zero syscalls, zero reads, zero writes, yet CPU time keeps accumulating at ~101% (measured as ~1012 utime+stime ticks per 10 seconds).

Process state at that point:

• 38 threads
• main thread wchan = do_epoll_wait (asyncio loop idle)
• open fds: 15x memory.db, 5x memory.db-wal, 1x memory.db-shm, plus the log file and two sockets/pipes

Database counters are frozen during the whole period:

chunks 14440 | embedded 14440 | index 1743

So the main thread is parked in the event loop while some other thread spins.

Workaround

kill -TERM <pid> terminates it cleanly. PRAGMA integrity_check returns ok afterwards and all counters are intact, which supports the conclusion that the work really was finished.

Note on a separate observation

Mentioned only so the CPU usage during the run is not confused with the spin loop after completion, and in case the numbers are useful.

The embedding phase itself runs at roughly 570 chunks/minute at ~570% CPU on 8 cores (fastembed / bge-small-en-v1.5). I tried tuning it and found no measurable effect:

semantic_embedding_batch_size	semantic_embedding_request_concurrency	throughput
2	4	~570 chunks/min
32	1	~570 chunks/min
32	4	~570 chunks/min

So neither batch size nor request concurrency appears to influence embedding throughput on this setup. If those knobs are expected to matter, they may not be reaching the fastembed call path.

[phernandez]

Thanks for the exceptionally thorough diagnostics. Before we chase this as a code bug, we need one missing fact: the host platform. Please share:

1. uname -m on the host and inside the container
2. Whether Docker reports the image platform as native or emulated (docker inspect <container> --format '{{.Platform}}', or Docker Desktop's "amd64" badge on Apple Silicon)

Your embedding throughput (~570 chunks/min for bge-small on 8 cores) is 5–10× below what native x86 delivers — that plus a native thread spinning at exactly one core with zero syscalls is the classic fingerprint of the amd64 image running under QEMU emulation on an ARM host, where onnxruntime's spin-waits are known to burn a core indefinitely and can hang in-flight inference. The image publishes both linux/amd64 and linux/arm64, so if that's the case, pulling the native arm64 image should resolve both the throughput and the hang.

Two more things that would pin it conclusively:

3. py-spy dump --pid <pid> on the stuck process — this shows what the event loop is awaiting and every thread's stack in one shot
4. Confirmation of which process your /proc samples targeted — the bm reindex CLI or the container's main server process. Note that your quoted watch_service log line (total_indexed_files=6303 on a 573-file corpus) comes from the long-running server process sharing the same log file, not from bm reindex itself (the reindex command uses the scan-based indexer and never starts the watch service), so the two processes' evidence is interleaved. The 15 open memory.db file descriptors also suggest both processes were sharing the WAL concurrently during the run.

One thing your report did catch on our side: semantic_embedding_request_concurrency currently only applies to remote providers (OpenAI/LiteLLM), not FastEmbed — that knob doing nothing on your setup is expected behavior, and we'll document it. semantic_embedding_batch_size does reach FastEmbed, but with local ONNX compute as the bottleneck it won't move throughput much.