Proposal: an optional on-device Memory & Answers module (local RAG) #850

Hi Jacob and Akram 👋 I've built an optional module that makes Cotabby more than autocomplete. Cotabby
remembers your conversations and documents, and drafts replies from them. Before opening ~14k
lines of PRs, I'd like to know whether you want this in Cotabby, and in what shape.

It runs today in my fork's builds (https://github.com/senadaruc/cotabby, branch senad, with
signed releases), so you can try it before deciding.

What it does

1. Memory: local RAG over your own conversations and documents.

• Sources (each opt-in, off by default):
  • Apple Mail, Outlook (New Outlook and classic), WhatsApp, the Microsoft Teams cache
  • Calendar (EventKit)
  • a notes folder, iCloud Drive, Google Drive and OneDrive (PDF, Office, RTF, text)
• Suggestions in a mail or chat can draw on what was said earlier in that conversation, or with
  exactly the same people. Memory is never shared across audiences.

2. Answer cards.

• When a message asks you something and your reply is still empty, Cotabby drafts an answer from
  memory and shows it on a card with its sources.
• Tab inserts it; Esc or typing dismisses it.
• Questions like "Are you free Thursday?" are answered from your calendar, with busy and free times
  only.

Together these turn Cotabby into an assistant that knows your context. It's still invisible until
it can help, and still Tab to accept.

Privacy design

• Local only.
  • Messages, vectors and keyword terms are encrypted at rest (AES-GCM, with the key in the
    Keychain).
  • Embeddings run on-device through llama.cpp.
  • Memory never reaches the OpenAI-compatible endpoint. Answers use Apple Intelligence or the local
    model only.
• Reads local files only. It uses the apps' own caches and files, with no Graph or Google APIs
  and no account access. Full Disk Access is needed only for the mail and chat sources.
• Defensive by default:
  • Mail counts as "from you" only when it's in Sent (no From-address spoofing), and Junk is skipped.
  • The question being answered is treated as untrusted text, and drafts are grounded: no
    names, numbers or dates the facts don't contain.
  • Calendar answers never reveal what a meeting is or who is in it.
• User controls: retention limits, excluded people and conversations, per-source "use for
  answers", Forget per source, and Delete All.

How it's built

It follows AGENTS.md's layering, so it can live as a self-contained module:

• Support/Memory, Support/Answers: pure, unit-tested rules. These cover question detection
  (English and Turkish), quoted-reply parsing, chunking, BM25 with reciprocal-rank fusion,
  grounding, availability facts, and Office and ZIP text extraction.
• Services/Memory:
  • the engine (encrypted SQLite store, half-precision vectors with an exact Accelerate scan)
  • background indexing (only on power and when the user is idle)
  • one reader per source
• App/Coordinators/Answers: the answer-card flow.
• UI/Settings/Panes/Memory: sources, index stats, privacy and a search Playground.
• CotabbyInference: one addition, an EmbeddingEngine with a reference-counted backend guard,
  so it can run next to autocomplete.

If you'd rather keep the core lean, it could also become its own Swift package or target that the
app links only when enabled.

Measured (M-series Mac)

Proposed PR sequence (each reviewable alone)

1. cotabbyinference: the embedding engine and backend guard.
2. Memory core: engine, store, indexing and the settings pane, with only the notes folder as a
   source.
3. Sources: Apple Mail and WhatsApp.
4. Sources: Teams and Outlook.
5. Sources: Calendar and the cloud drives.
6. Memory context in suggestions.
7. Answer cards.

Questions for you

• Do you want this in Cotabby, as a built-in module or as an optional add-on target?
• Are the reverse-engineered readers (the Teams IndexedDB and New Outlook's HxStore.hxd) acceptable
  to maintain, or should they stay out of upstream?
• Anything about the privacy model you'd want changed before PR 2?

Happy to adapt the shape to whatever you prefer. Thanks for Cotabby!