Show it once. It coaches everyone after.
Film an expert doing a hands-on task once. Apprentice turns that video into a step-by-step procedure, then watches anyone else's attempt (recorded or live on a webcam) and points at the exact fold, part or edge that went wrong.
Built for the Meta Global AI Developer Hackathon ("Push the Frontier") with Muse Spark 1.3 and SAM 3.1 on the Meta Model API.
Most hands-on skills, like assembly, lab technique and repairs, are still taught by someone standing next to you. Written instructions lose the details that matter: which edge, which way, how far. Apprentice needs one recording of someone who knows how, and every learner after that gets a coach who has watched it.
| Stage | Model | What happens |
|---|---|---|
| Learn | Muse Spark 1.3 (video) | Reads the expert video and writes structured steps: instruction, the key object, what "done" looks like, the common mistake, and the best frame. |
| Illustrate | SAM 3.1 | Segments each step's key object on the expert frame, so every step card shows what to look at. |
| Watch blind | Muse Spark 1.3 (video, high reasoning) | Logs every action in the learner's video with timestamps before it is told the procedure. |
| Judge | Muse Spark 1.3 (both videos + log) | Matches procedure steps to logged actions. A step counts only if a logged action performs it, which stops the model "seeing" steps that never happened. |
| Point | SAM 3.1 | Masks the object that went wrong, side by side with the expert's frame at the same step. |
| Live | Muse Spark 1.3 (images, minimal reasoning) + SAM 3.1 | Every few seconds: how far through the procedure is the object in front of the camera? Advances the steps, speaks the next instruction, and marks mistakes once two checks agree. |
- Confirmation bias. Given the procedure first, the model reported a skipped fold as done, because the end shape looked right. The blind observation pass fixed it, and it now catches the skipped fold every time in our tests.
- Camera-relative directions. A creator who turned the paper sideways was told they folded the wrong edge. Prompts now describe every fold relative to the object (long edge, short edge, centre crease), and turning or flipping the object is never a mistake.
- Live lag. Checks trail the hands by a few seconds, so live mode asks "how far through is this object?" instead of "is step N done?", and it can jump ahead. One odd frame (a hand in the way, the paper mid-turn) can't trigger a correction: a mistake has to show up in two checks in a row.
Needs Node 20+ and a Meta Model API key from https://dev.meta.ai/api-keys/.
npm install
echo 'MODEL_API_KEY=your-key' > .env
npm start # http://localhost:5173ffmpeg-static converts any phone or webcam video (HEVC .mov, webm) to small H.264 mp4 before upload.
To try live mode without a camera, open /?sim=/path/to/clip.mp4: the clip plays as if it were the webcam.
scripts/eval.mjs <expert.mp4> <attempt.mp4> runs an expert/attempt pair through the API and prints the procedure, the blind log and the verdict.
It runs as one Express app on Vercel. Request bodies there are capped at 4.5 MB, so when BLOB_READ_WRITE_TOKEN is set, the browser uploads videos straight to Vercel Blob and /api/ingest pulls them from there, converts them and deletes the blob. Each IP is rate-limited (RATE_LIMIT, default 150 calls per 10 minutes), because every visitor spends the same API key.
server.mjs: API routes (upload/ingest, procedure, observe, check, live, segment)public/: the whole front end (no build step)scripts/: the eval harness and API probes