---

A normal cache keys on the exact bytes of the request. LLM traffic almost never repeats exactly — "how do I cancel my subscription" and "where's the cancel button" are the same question and the same answer, and an exact-match cache bills you twice for it.

This one keys on meaning. It embeds each stored output, and on a new query returns the cached result when cosine similarity clears a threshold.

output ──embed──▶ vector ──store──▶ pgvector (cosine index)
                                         ▲
new query ─embed─▶ vector ─nearest───────┘
                     │
            similarity ≥ threshold ?  ── yes ─▶ HIT  (return cached output + $ / ms saved)
                                       └─ no ──▶ MISS (caller runs the model, then /store)

Provider-agnostic — OpenAI, Anthropic, a local model, or a plain HTTP API. The cache never calls your model; it only remembers what you tell it.

Quick start

The default EMBEDDING_PROVIDER=local uses a deterministic offline embedding, so the whole service runs with no OpenAI key and no external services — just Docker.

cp .env.example .env
npm install
npm run db:up                 # pgvector Postgres on :5433
npx prisma migrate dev --name init
npm run dev                   # http://localhost:8080

# get a key
curl -XPOST localhost:8080/v1/auth/signup \
  -H 'content-type: application/json' -d '{"email":"me@example.com"}'

# store an output that cost you $0.004 and 800ms
curl -XPOST localhost:8080/v1/cache/store -H "authorization: Bearer $KEY" \
  -H 'content-type: application/json' \
  -d '{"output":"The capital of France is Paris.",
       "metadata":{"model":"gpt-4","cost_usd":0.004,"latency_ms":800}}'

How it works

Layer	What it does
src/services/embedding.service.ts	Embeds text. OpenAI or a deterministic local provider, behind one interface.
src/services/cache.service.ts	Nearest-neighbour lookup over pgvector, threshold comparison, TTL and GC.
src/services/redis.service.ts	Hot path for exact repeat queries — skips the embed + pgvector round trip entirely.
src/services/namespace.service.ts	Keeps tenants' caches from colliding.
src/services/analytics.service.ts	Tracks what each hit actually saved, in dollars and milliseconds.

Redis is an accelerator, not a dependency. If REDIS_URL is empty or the server is unreachable, everything degrades to pgvector and keeps serving.

The embedding provider has a circuit breaker. After EMBEDDING_BREAKER_THRESHOLD consecutive failures it stops calling out for EMBEDDING_BREAKER_COOLDOWN_MS, so a provider outage becomes a cache miss instead of a queue of hanging requests.

Security

There is a full threat model in the repo. The short version:

Risk	Control
Stolen database → key replay	Only SHA-256 hashes of API keys are stored. The raw key is shown once at signup and never persisted.
Leaked key	POST /v1/auth/rotate invalidates the old one immediately.
Over-broad key	Per-key scopes — cache:read, cache:write, stats:read — enforced per route.
Cross-tenant reads	Every query filters by user_id; entries cascade-delete with the user.
Injection	Parameterised SQL everywhere, including the raw pgvector queries. Zod validation on every input. Body capped at 2 MB.
Cost blowups	Per-user daily rate limit, Redis-backed so it holds across instances.

SDKs

sdks/javascript and sdks/python, both thin wrappers over the REST API. The full endpoint reference is in docs/api-spec.md.

Tests

npm test          # unit + integration
npm run typecheck

tests/unit covers the embedding provider and vector key handling; tests/integration covers the API surface, the response contract, and rate limiting.

Why I built it

The same waste kept showing up: an application pays a model full price for a question it already answered, because the second user phrased it differently. An ordinary cache cannot help — it keys on bytes, and in LLM traffic the bytes essentially never repeat.

Three decisions shaped it:

Redis is an accelerator, never a dependency. A cache that goes down and takes the application with it has made things worse than no cache at all. Empty REDIS_URL, unreachable host, wrong password — all of them degrade to pgvector and keep serving.

The embedding provider sits behind a circuit breaker. A provider outage should become a cache miss, not a queue of requests hanging on a timeout.

Raw API keys are never stored — only SHA-256 digests. A stolen copy of the database should be worth nothing. The plaintext key is shown once, at signup, and never written down.

License

MIT — see LICENSE.

_{Built by Rohit Maruri  ·  rohitmaruriats@gmail.com}