English | 简体中文

pgdocket

Durable, versioned, calibrated AI decisions for Postgres. Declare a semantic decision once ("is this complaint about debt collection?"). pgdocket keeps the answer stored as data, versioned, calibrated against your own labels, and honest about its uncertainty, in any Postgres and with any decision model.

Status: Phase 1 in progress. Working: pgdocket check measures how trustworthy an AI decision is on your table; the ledger stores decisions in Postgres and keeps them current as rows change; pgdocket estimate gives counts with confidence intervals; pgdocket compare tests a revised decision against the current one before switching. It works with any /v1/systemone server, and Laya, run on your own machine, has been measured against Jev. Not built yet: adapters for other APIs, such as OpenAI Decisions, and a published release. The repository also contains the design (REPORT.md, ADRs) and the Phase 0 benchmark (findings). Progress is tracked in issues and milestones.

The problem

pg-jev lets you filter rows in plain language, such as WHERE jev(tickets, 'the customer threatens to cancel'), using TypeSafe's Jev decision model. That's good for exploring. To rely on the answers in production, they need to be stable, comparable and trustworthy, and today they aren't:

• they disappear when the database session ends, so every session judges the whole table again;
• they change when someone adds a label option, or switches model or backend;
• they change when you simply ask again, or send the row alongside different rows;
• a score of 0.7 doesn't mean 70 %, and count(*) over AI labels has no error bar;
• they don't run on managed hosts like Supabase, Neon or RDS.

Model calls now cost roughly $6–25 per million rows (depending on text length and rows per request), so the bottleneck is no longer cost. It's whether people can trust and audit the answers.

What Phase 0 measured

2,000 real consumer-complaint narratives, judged by jev-1.13.0 through pg-jev, for about $0.30 in API calls in total.

Finding	Result
Scores are over-confident	9-way choice ECE 0.089; yes/no ECE 0.24–0.38
~100 labels fix most of it	Calibration on 100 labels removes 45–85 % of the error
Asking again changes answers	4.8 % of rows change top answer across 3 identical runs
Neighbours change answers	Re-batching the same rows changes ~16.5 % of answers (≈ 5× noise)
One row per request is better	Judging rows singly: +2.4 pp accuracy, +0.05 macro-F1, at 1.8× tokens
Adding an option moves answers	2.5 % of answers move to a new catch-all option
Scores are coarse	Probabilities come on a 0.01 grid with exact zeros

Write-up for a general audience: Same row, different answer. Details, methods and limits: docs/phase0/FINDINGS-jev-1.13.md.

Install

pip install "git+https://github.com/ElyasEbrahim/pgdocket"      # Python 3.10+; installed from GitHub
pgdocket --version

Or as a container: docker build -t pgdocket . from this repository. New here? The 5-minute quickstart runs Postgres, the worker and 60 synthetic support tickets with Docker, through to calibration and trying a change before switching it on. It costs well under $0.01 in API calls.

Try it on your own data: pgdocket check

Point it at a table, the text to send, a question and some labels (an existing column, or a CSV you label by hand). It reports agreement, calibration, a threshold with a stated precision guarantee, consistency and cost. It's read-only, and no row text goes in the report.

pgdocket check --table public.tickets --key id --text subject,body \
  --kind choice --question "Which team should handle this ticket?" \
  --options billing,technical,security,sales --label-column team --yes

On two years of Kubernetes issues, routing to 10 teams (SIGs) agreed with the human labels 88.8 % of the time. 90 % of issues could be auto-routed with precision ≥ 90 % guaranteed at 95 % confidence, or 56 % at ≥ 95 % (write-up, report). Full usage: pgdocket/README.md.

What pgdocket does, and will do

1. Stored decisions. Built. Answers live in an append-only ledger and a view you can join and index. New and changed rows are judged automatically, one row per request. 100,000 rows: one row reads in 0.14 ms; on a live stream, the median time from insert to answer was 0.7 s (details).
2. Calibration on your labels. Built. About 100–200 human labels per decision turn raw scores into probabilities that mean what they say. The view shows the calibrated confidence and an auto flag.
3. Honest numbers. Built. Thresholds come with stated guarantees ("precision ≥ 90 % at 95 % confidence"). Answers that fall short of the target are flagged one by one, and --per-answer gives each answer its own guarantee. For yes/no decisions, a recall target also lets rows be answered no automatically while keeping, say, 90 % of the real yeses (details). pgdocket plan says how many rows people must review for the whole column to be at most, say, 5 % wrong (details). Counts come with confidence intervals: pgdocket estimate combines the model's answers with a random sample of labels (prediction-powered inference). On real Kubernetes issues its intervals held 98–100 % of the time and were 30–50 % narrower than labels alone (details). --by region gives a share per group, each corrected with its own labels, and --target says how many more labels each group needs. On 2,000 CFPB complaints, labels drawn within groups brought every region to ± 5 points with 24–64 % fewer than labels drawn across the table (details). --weight handles queries that count a row many times, by what they sum or how often a join uses the row. There the rows that matter are the heaviest, so labels are drawn in proportion to weight. The 2,000 complaints, judged once as distinct texts, stand for 21,980 complaints. Drawing by weight brought their shares to ± 2 points with 112–192 labels, 53–94 % fewer than texts drawn at random (details). --with combines several models' answers with the labels (Multiple-PPI), and --label-cost finds the cheapest mix of labels and rows each model judges. For a share of 100 million rows, judging a random sample instead of every row cut the cost of ± 2 points by 68–93 % (details). With --by, each group gets its own cheapest mix. Bringing every region to ± 3 points cost 49–67 % less this way than with one design drawn across the table (details). With --weight, the second model judges a longer run of the labels' ranking by weight. On the CFPB join that cut the cost of ± 2 points by 10–19 % for the common answer, and saved little for rare ones (details).
4. Versions. Built. Changing the question, options or model creates a shadow version. pgdocket compare judges a sample under both versions and reports which answers changed beyond run-to-run noise. It also reports which version is more accurate. Only the rows where the versions disagree need labels for that. On Kubernetes routing, dropping the option descriptions saved 14 % of tokens and moved 3.3 % of answers beyond noise. Labelling the 13 disagreements showed the descriptions were worth keeping (v1 right 8 times, v2 twice). With all labels, they also let 95 % of issues be auto-routed at the guaranteed 90 % precision, instead of 90 %. promote recalibrates the new version, so automation carries over (details).
5. Drift checks. Built. pgdocket drift checks the two assumptions behind every guarantee. The first is that recent rows still look like the earlier ones; this check is free, from the answers already stored. The second is that the model still answers labelled rows as it did; this check asks a fixed panel of them again. Alarms share one false-alarm budget across repeated runs. On stored Jev answers, false alarms stayed at 0–2.6 %, and real shifts of 13–17 % in CFPB and Kubernetes labels were caught. Removing the option descriptions was caught from the model's scores (details). pgdocket audit checks the accepted answers themselves. People label a few auto-accepted rows per batch without seeing the AI's answer, and an anytime-valid test alarms once precision is demonstrably below target; it can be read after every batch without raising the false-alarm rate. On real streams it raised no false alarm, and a fall to 70 % was caught in 93 % of audits (details).
6. Rows that try to steer the model. Built. When outsiders write the rows, they can write text aimed at the model. On 300 Kubernetes issues, a counterfeit routing policy appended by the issue's author sent 98.7 % of them to the team the author chose, and auto accepted 93.3 %: steered answers come out fully confident.
   • pgdocket check --steer measures this on your own table.
   • --untrusted tells the model to ignore instructions in the rows; that policy then steered 1.3 % of issues.
   • calibrate --harden lets a row be auto only once erase-and-check has passed it: no answer may change when any one paragraph is removed. It caught every steered row auto would have accepted, including misleading claims the sentence can't stop. It costs 7 % of clean accepted rows, sent to review.
   • Together, every attack we tried got through on at most 0.3 % of issues, except a misleading claim written twice (7.3 %). They kept 86 % of issues auto instead of 95 % (details).
7. Runs anywhere. Built and tested on PostgreSQL 14–18 as a non-superuser. Plain SQL plus an external worker, no untrusted extension. Not yet tested on Supabase, Neon or RDS themselves.
8. Any backend. Built. pgdocket speaks the /v1/systemone API, so it can use TypeSafe's Jev or a model you run yourself; TypeSafe's key goes only to TypeSafe. On the same labelled rows, Laya (open weights, run locally, no API cost) was right 38 % of the time on Kubernetes routing, where Jev was right 88.8 %. The guarantees held: pgdocket would have sent all of Laya's answers to review rather than promise a precision it doesn't have. Calibrated on 2,000 CFPB labels, Laya's probabilities were as reliable as Jev's. An issue's author steered Laya far less than Jev, mostly because Laya reads only the start of a long issue; written first, a counterfeit policy still steered 21 %, and --untrusted never reached the model (details).

Architecture and SQL sketch: REPORT.md §4. Decisions so far: docs/adr/.

Quick start: the Phase 0 harness

Requires Docker. Everything runs in containers.

make up            # Postgres 16 + pg-jev v0.2.1, pg-jev's mock API, a simulated backend
make test          # harness unit tests
make smoke         # pg-jev's README queries against the mock
make fetch load    # 2,000 CFPB complaint narratives (CC0 mirror) → Postgres
make bench         # all experiments on the simulated backend → results/<run_id>/report.md

To run against the real Jev API, copy .env.example to .env, add your TypeSafe key, then:

make bench BACKEND=jev ARGS=--yes   # prints the estimated spend; refuses anything over $1

More commands (determinism, learning curve, batch context): harness/README.md.

Repository map

Path	What
REPORT.md	Project report: pg-jev teardown, research survey, candidates, design, evaluation plan, risks
ROADMAP.md	Phased plan with exit criteria
CHANGELOG.md	What changed, by date
docs/adr/	Architecture decision records
docs/phase0/	Phase 0 findings, status, draft upstream issue
docs/notes/	Public write-ups
docs/phase1/	Phase 1 results: the ledger, counts with error bars, version comparison, Kubernetes triage
pgdocket/	The product: pgdocket check, the ledger and worker, estimate, compare
examples/	The 5-minute quickstart, and option files for the example checks
pyproject.toml, Dockerfile	Packaging: the pgdocket command and the worker image (releasing)
harness/	Benchmark harness: jevbench CLI, simulated backend, tests
results/	Benchmark reports from the runs cited in the findings
docker-compose.yml, Makefile	The benchmark stack and its commands

Sources

Every source, grouped by the feature it shaped and with what each was used for, is in REFERENCES.md.

Contributing

This is a single-maintainer project. Issues and questions are welcome; pull requests are not accepted for now. See CONTRIBUTING.md.

License

Apache-2.0. Copyright 2026 ElyasEbrahim.