Align similarity threshold between base and docker profiles for MiniLM embeddings

[devops-thiago]

Background

query.similarity-threshold is inconsistent across profiles:

• src/main/resources/application.yml → 0.7
• src/main/resources/application-docker.yml → 0.50

RetrievalEngine.retrieveRelevantChunks filters results with score >= threshold using cosine similarity from the local all-MiniLM-L6-v2 embedding model (384 dims). MiniLM cosine scores for topically similar but non-verbatim chunks typically fall in the 0.2–0.6 range, so 0.7 frequently discards valid matches and returns zero chunks.

Impact

• With the default (non-Docker) profile, users get "no relevant documents found" for questions that do have answers in the corpus.
• Behavior differs between Docker and local runs, making the system unpredictable across environments.
• Empty retrieval triggers the out-of-scope guardrail, producing the fixed refusal reply even for valid questions.

How to reproduce

1. Run with the default profile (mvn spring-boot:run), threshold 0.7.
2. Ingest a Java 25 doc, e.g. POST /api/ingest?documentPath=docs/specs/<file>.html.
3. Ask a paraphrase-style question via POST /api/chat/query that matches a doc section semantically but not verbatim.
4. Observe RetrievalEngine logs "Found 0 documents above threshold" and the answer is the out-of-scope refusal.

Where the fix should land

Pick a single threshold tuned against the actual corpus with the MiniLM model and apply it to both profiles (or centralize it with a documented default). First measure the score distribution over the corpus for known-good queries (scores are already logged per chunk), then choose the value. Add a YAML comment explaining the rationale.

Files touched

• src/main/resources/application.yml
• src/main/resources/application-docker.yml
• src/main/java/br/com/arquivolivre/myjavagenie/config/QueryConfig.java (only if a constraint/validation is added)
• Retrieval tests under src/test/java/...

Acceptance criteria

• Both profiles use the same, tuned query.similarity-threshold with a documented rationale.
• A known-good query set returns non-empty chunks on both profiles.
• Retrieval tests updated and passing; mvn spotless:check passes.