A Technical White Paper July 2026
Building a supervised computer-vision dataset is a slow, manual bottleneck: someone has to look at every image or document page and draw a box, then decide what it is. That work is linear in dataset size, and it's usually the single most expensive part of the machine-learning lifecycle — more expensive, in aggregate, than training itself.
Censor collapses that from "annotate everything" to "annotate a handful of examples." It is a native, universal SwiftUI app for iPadOS and macOS that lets a user draw and label a few boxes on a handful of files in a folder, then uses an on-device few-shot matching pipeline — built entirely on Apple's system frameworks, with no bundled models and no network calls — to find and label the same kinds of objects across the rest of the folder. The user reviews what the system proposes; every confirmation and rejection refines the system's understanding for the next pass. What comes out the other end is a clean, standards-compliant labeled dataset, ready to train wherever the user chooses.
This paper describes the problem Censor addresses, the design principles behind it, how its seed → propagate → review → export pipeline works end to end, and why an on-device few-shot approach was chosen over the more obvious alternative of calling a cloud vision-language model.
Supervised object detection and document field-extraction models need labeled training data — bounding boxes tied to categories, applied consistently across hundreds or thousands of examples. Producing that data has a few recurring failure modes:
- Manual annotation doesn't scale. Drawing boxes on every file in a folder is tedious, error-prone at volume, and doesn't get cheaper as the dataset grows.
- General-purpose labeling tools assume you'll do the work yourself, or hand the problem to a large cloud vision-language model. The latter solves the scaling problem but introduces new ones: API cost per image, network dependency, latency, and — critically for scanned business documents, medical images, or any personal or confidential material — the images have to leave the device.
- Fully automated cloud-based labeling is opaque. A large model's label for a given box is rarely traceable to a clear reason a user can inspect or correct systematically.
The founding constraint on this project, inherited from its predecessor app (ModelBuilder), is that everything must run on-device, on Apple hardware, universally across iPad and Mac — with no bundled foundation models, since large vision-language or multimodal models don't fit comfortably in iPad RAM regardless of chip generation. That constraint rules out the easy answer ("just call a cloud model") and shapes everything about how Censor is built.
On-device only. Vision, PDFKit, and SwiftData — all system frameworks. Zero external dependencies, zero network calls, zero bundled model weights. A user's documents never leave their device.
Few-shot, not zero-shot. Censor does not attempt to be a general open-vocabulary detector — that effectively requires a foundation vision-language model, which was ruled out for RAM reasons. Instead, it leans into the fact that a person is present and willing to label a handful of examples, and turns that into the mechanism itself: propagation works by visual similarity to what the user has already drawn, not by interpreting a typed description.
Human-in-the-loop, not human-replaced. Propagation drafts labels; it does not finalize them. The review step is not a rubber stamp — confirming or rejecting a proposed box actively changes what the system looks for next, on the same folder. The labeling agent's "understanding" of a category is never more than the sum of the exemplars the user has personally confirmed.
Export, don't lock in. Censor's job ends at producing a clean, standards-compliant dataset. It does not train a model itself. Training happens wherever the user wants — including in ModelBuilder, or any other tool that reads COCO, YOLO, or Create ML formats.
Censor's interface has four sections, and they map directly onto four pipeline stages: Import → Annotate → Review → Export.
| Stage | What Happens | Key Mechanism |
|---|---|---|
| Import | User points Censor at a folder. It's scanned for images and PDFs. | Security-scoped folder bookmark; PDFKit renders each PDF page as an independently labelable unit. |
| Annotate | User draws boxes and assigns labels on a handful of seed files. | Drawing and labeling are a single atomic action — draw, then immediately pick or create a label. |
| Propagate | The system proposes matching boxes across the rest of the folder. | Vision region proposal + feature-print embedding + nearest-exemplar matching (see §3.3). |
| Review | User confirms, rejects, or adjusts each proposed box. | Confirmations become new exemplars; rejections tighten that category's match threshold. |
| Export | The confirmed dataset is written out. | Cropped-by-category folders, or a COCO / YOLO / Create ML manifest. |
A workspace is just a folder the user grants Censor access to. Under the app sandbox, that access is persisted as a security-scoped bookmark, so the same folder can be reopened across launches without re-prompting. The folder is scanned recursively for images and PDFs; each PDF page becomes its own labelable unit, rendered to a raster image via PDFKit at annotation time.
On a handful of files, the user drags to draw a box, and finishing that drag immediately opens a label picker — pick an existing category or type a new one. This deliberately collapses what's normally a two-pass workflow (draw every box, then go back and tag everything) into a single pass, because the two-pass version is where most annotation fatigue and inconsistency creeps in.
This is the technical heart of the "agent" the original brief asked for, and it's built entirely without a foundation model:
- Candidate region proposal. For each unlabeled file, Censor generates candidate boxes without knowing what it's looking for, using
VNGenerateObjectnessBasedSaliencyImageRequest(general object-like regions, good for photos) alongsideVNDetectRectanglesRequestandVNRecognizeTextRequest(rectangular and text-block regions, good for forms and documents). - Feature embedding. Every seed crop the user drew, and every candidate region on every unlabeled file, is embedded using
VNGenerateImageFeaturePrintRequest— Apple's built-in perceptual feature print. There is no model to ship, download, or fine-tune. - Matching. Each candidate is compared by feature-print distance against every category's accumulated exemplars. The nearest category wins, but only if the distance is under that category's own threshold — so a candidate with no good match is correctly left unlabeled rather than forced into the closest available bucket.
Because there is no open-vocabulary foundation model on-device, Censor does not pretend to support "type a description, find it." Matching is grounded entirely in the visual exemplars the user has drawn. That is a real limitation — no zero-shot classes — but it is also a feature: every match is explainable in a single sentence ("it looked like this example you drew"), which a cloud vision-language model's classification generally is not.
This is how Censor satisfies the second idea from the original brief — an agent that "builds a dataset out of it based on the user's expected outcome" — without natural-language prompting:
- Confirming a proposed box promotes it to a real exemplar. The category's understanding of "what this looks like" widens to include it, so the next propagation pass is informed by it.
- Rejecting a proposed box does more than delete it — it tightens that category's match threshold, so future propagation for that category requires a closer visual match before it will guess again.
Run propagation, review the results, run it again: each pass is measurably better than the last, converging on the user's actual intent over the course of a folder, entirely from box-drawing interactions the user was already doing.
Detection datasets and classification datasets have different shapes, so Censor produces two distinct outputs rather than forcing one mold onto both:
- Cropped by category. Every confirmed box is saved as its own image file, sorted into a folder named for its label. This is immediately usable for training an image classifier, and it directly satisfies the third idea from the original brief — organizing messy input into a structure ready for training.
- Detection manifest. Whole labeled images are copied alongside a manifest — COCO JSON, YOLO (
.txt+classes.txt), or Create ML JSON — describing every box and its category, for training an object detector. The Create ML format in particular feeds directly into ModelBuilder's existing training pipeline.
Only reviewed annotations — user-drawn or user-confirmed — are ever exported. Boxes still sitting in the review queue are never mistaken for ground truth.
Censor is a single SwiftUI codebase targeting a shared multiplatform destination (supportedDestinations: [iOS, macOS]), generated via xcodegen from a project.yml, with a deployment target of iPadOS 26 and macOS 26. There is no iPhone or watch target — the annotation workflow assumes a larger canvas.
State is persisted with SwiftData across four models: Workspace (a folder, held as a security-scoped bookmark), DocumentItem (one image or PDF page), Category (a label, with its accumulated exemplars and its own match threshold), and Annotation (a box, tagged with its source — user-drawn, auto-propagated, or user-confirmed — so the pipeline always knows which boxes are ground truth and which are still drafts).
The propagation engine sits behind a RegionMatcher protocol, with OnDeviceRegionMatcher as its only current implementation. That seam is deliberate: it means a different backend — for instance, a cloud vision-language model, for users willing to trade privacy for open-vocabulary recall on a specific project — could be added later without reworking the rest of the app. Nothing beyond the protocol boundary exists yet; it is a design decision, not a shipped feature.
The more obvious design would call a hosted vision-language model with the seed crops and a description, and let it find matches anywhere. It would likely recognize a wider range of objects on the first try. It was ruled out for this project on the following grounds:
| Dimension | Fully Manual | Cloud Vision-Language Model | Censor (on-device few-shot) |
|---|---|---|---|
| Privacy | Full — human review only | Images leave the device | Full — never leaves the device |
| Cost per image | High (labor) | Per-call API cost | None (local compute only) |
| Works offline | Yes | No | Yes |
| Open-vocabulary recall | N/A (human judgment) | High | None — grounded in the user's own exemplars |
| Explainability | Full | Low (opaque model) | High — every match traces to a specific exemplar |
| Setup required | None | API key / account | None |
The trade Censor makes is explicit: it gives up open-vocabulary recall in exchange for privacy, zero marginal cost, offline operation, and explainability. For the target use case — a user with a folder of private documents or photos who is willing to label a handful of examples — that trade favors the on-device approach. The RegionMatcher seam exists precisely so that trade doesn't have to be permanent for every user.
In scope for v1: PDFs and images, as spatial bounding-box annotation targets. PDF pages and photos flow through the identical pipeline once rendered to a raster image, so supporting both was a natural extension rather than two separate features.
Deliberately out of scope for v1: tabular or spreadsheet data. Row/column labeling is a fundamentally different annotation paradigm from bounding boxes, and folding it into v1 would have diluted the core loop before it was proven. It remains a plausible future extension, not a rejected idea.
Deliberately excluded, not deferred: in-app model training. Censor's scope ends at producing a clean labeled dataset. Training is left to dedicated tools — including ModelBuilder, whose Create ML pipeline can consume Censor's export directly — so Censor stays a labeling specialist rather than duplicating a role another tool already fills.
The expensive part of building a computer-vision dataset was never really "training a model" — it was getting a human to look at enough examples and agree on what they mean. Censor doesn't try to remove the human from that loop; it tries to make each interaction in that loop count for more than one example, by turning a handful of drawn boxes into a matching signal that propagates across a folder, and turning every review decision into a correction that improves the next pass. Built entirely on Apple's system frameworks, it does this without a single image ever leaving the device.