Feat/card capture by BosuBose132 · Pull Request #278 · mieweb/ui

BosuBose132 · 2026-06-20T19:12:09Z

Summary

This PR adds a reusable CardCapture component to @mieweb/ui.

The component runs an ONNX ID-card detection model directly in the browser and starts automatic capture only after an ID card is detected consistently and the existing image-quality checks pass.

Problem

The existing camera flow checks focus, brightness, and frame stability, but it cannot confirm that the object in front of the camera is actually an ID card.

This means other clear rectangular objects could pass the general image-quality checks without semantic card validation.

Solution

CardCapture combines:

Existing MIE camera handling
Focus, brightness, and frame-stability checks
Browser-based ONNX ID-card detection
Consecutive-frame validation
Automatic countdown and capture
Manual capture fallback

The model is executed through onnxruntime-web, and camera frames remain inside the browser during detection.

https://youtube.com/shorts/kOTIUUnLONU?feature=share (A small demo short of my output)

Implementation

The component:

Converts webcam frames into the model’s [1, 3, 640, 640] tensor format
Runs the ONNX model using the WebAssembly execution provider
Filters YOLO predictions by confidence
Applies non-maximum suppression
Requires consecutive positive detections
Prevents overlapping inference calls
Combines semantic detection with focus, brightness, and stability checks
Returns the confirmed image as a browser File

The visible interface is built using existing MIE UI components and utilities, including:

Modal
Button
Spinner
Alert
Text
MIE icons
useCamera
useDocumentDetection

Features Added

Browser-based ID-card detection
Automatic capture after stable detection
Manual capture fallback
Configurable confidence threshold
Configurable countdown
Camera switching
Permission and model error states
Captured-image preview
Retake and confirmation flow
Minor handheld-movement tolerance
Model, timer, camera, and object URL cleanup
Live Storybook demonstration
Versioned ONNX model for zero-setup testing
Unit and component test coverage
Tree-shakeable component entry
Maintainer documentation

Usage

import { CardCapture } from '@mieweb/ui/components/CardCapture';

<CardCapture
  open={isOpen}
  onOpenChange={setIsOpen}
  modelUrl="/models/id-card-detector-v1.onnx"
  wasmPaths="/ort-wasm/"
  onCapture={(file) => {
    // Handle the confirmed image.
  }}
/>;

Testing

Verified that:

Supported ID cards trigger detection and automatic capture
Phones, blank paper, and notebooks remain undetected
Manual capture works when automatic detection is unavailable
Retake restarts the camera and detection flow
Confirmed captures return a browser File
The model runs directly in Chrome without an inference server

Automated tests cover model loading, preprocessing, prediction processing, stable detection, capture flows, permission handling, model fallback, retake, confirmation, and cleanup.

Reviewer Instructions

pnpm install
pnpm build:esheet
pnpm storybook

Open:

Components → Images & Media → CardCapture → Live Browser Model

Suggested checks:

Open the camera and allow permission.
Show a supported ID card.
Confirm that the detected state and countdown appear.
Test automatic capture, retake, and confirmation.
Confirm manual capture works.
Test a phone, notebook, and blank paper as negative objects.

Current Model Scope

The included MVP model supports a single id_card class.

It performs best on ID formats represented in its training data. Some additional formats, such as certain student IDs or employee badges, may require future model calibration or retraining.

The model can be updated independently without changing the public CardCapture API.

Breaking Changes

None.

The existing DocumentScanner behavior remains unchanged by default.

Validation

Combine frame preprocessing, ONNX session execution, and YOLO post-processing. Return a simple detected/confidence result while keeping model geometry internal. Validate missing model inputs and outputs with actionable errors, and add unit coverage for positive detections, empty results, and invalid model output.

Load and reuse the browser ONNX session while sampling frames from the existing camera video reference. Prevent overlapping inference calls, require consecutive positive predictions, and tolerate a configurable number of temporary misses before resetting detection. Expose model loading, searching, detected, and error states with explicit start, stop, and reset controls. Add hook tests for model loading, stable detection, missed-frame resets, inference locking, session cleanup, and loading failures.

Compose the existing MIE Modal, Button, Spinner, Alert, Text, icon, and camera primitives into a reusable ID-card capture experience. Require both document-quality readiness and stable semantic ID-card detection before starting the automatic capture countdown. Preserve manual capture as a fallback when the model is loading, unavailable, or uncertain. Support camera switching, permission and model errors, captured-image preview, retake, confirmation, modal cleanup, and File-based capture output.

…pture thresholds for minor handheld movement

…issions, model fallback, confirmation, and cleanup

…rnalize ONNX Runtime Web

…ction pipeline, testing, and maintenance constraints

…orybook testing

… Storybook viewport

Copilot

Pull request overview

Adds a new reusable CardCapture component to @mieweb/ui that performs in-browser ONNX-based ID-card detection (via onnxruntime-web) and integrates it with the existing camera + image-quality checks to enable stable automatic capture with a manual fallback.

Changes:

Introduces CardCapture (component, hooks, inference pipeline, Storybook story) plus unit/component tests.
Extends useDocumentDetection with a configurable stabilityThreshold (default preserved) and updates tests.
Updates packaging/build config to ship CardCapture as a tree-shakeable subpath and keeps onnxruntime-web external + optional peer dependency; adds Storybook static assets mapping for ONNX runtime WASM files.

Reviewed changes

Copilot reviewed 21 out of 23 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tsup.config.ts	Adds `components/CardCapture/index` entry and marks `onnxruntime-web` as external.
src/components/DocumentScanner/useDocumentDetection.ts	Adds `stabilityThreshold` to config/defaults and uses it in stability comparison.
src/components/DocumentScanner/useDocumentDetection.test.ts	Adds a stability similarity test case for relaxed thresholds.
src/components/CardCapture/useCardDetection.ts	Implements polling-based detection loop, model lifecycle, and stable-detection logic.
src/components/CardCapture/useCardDetection.test.ts	Tests model loading, stable detections, misses handling, inference locking, and errors.
src/components/CardCapture/runCardInference.ts	Wires preprocessing + session run + postprocessing into a single inference step.
src/components/CardCapture/runCardInference.test.ts	Unit tests for inference orchestration and error cases.
src/components/CardCapture/preprocessCardFrame.ts	Implements letterbox resize + RGBA→NCHW float tensor conversion.
src/components/CardCapture/preprocessCardFrame.test.ts	Unit tests for letterboxing and tensor conversion behavior.
src/components/CardCapture/postprocessCardDetections.ts	Parses YOLO output, maps boxes to source coords, and applies NMS.
src/components/CardCapture/postprocessCardDetections.test.ts	Unit tests for parsing/layout support, IoU/NMS, and threshold validation.
src/components/CardCapture/MAINTAINERS.md	Adds maintainer-focused documentation for model/runtime requirements and tuning.
src/components/CardCapture/loadCardModel.ts	Loads ONNX model and configures ORT WASM asset path.
src/components/CardCapture/loadCardModel.test.ts	Unit tests for model URL validation/trim and WASM path handling.
src/components/CardCapture/index.ts	Exposes `CardCapture` and `useCardDetection` via subpath entry.
src/components/CardCapture/CardCapture.tsx	Implements modal UI, countdown auto-capture, preview/confirm/retake flow, and cleanup.
src/components/CardCapture/CardCapture.test.tsx	Component tests for open/start, manual capture, retake, permissions, auto-capture countdown, and cleanup.
src/components/CardCapture/CardCapture.stories.tsx	Adds live Storybook demo and documentation for model-based capture.
pnpm-lock.yaml	Locks `onnxruntime-web` and its transitive dependencies.
package.json	Adds `onnxruntime-web` as optional peer dependency and dev dependency.
eslint.config.js	Adds Canvas-related globals to avoid lint false-positives.
.storybook/main.ts	Serves `onnxruntime-web/dist` as static assets for local Storybook inference.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Comments suppressed due to low confidence (1)

src/components/CardCapture/MAINTAINERS.md:265

The file ends with extra triple-backtick fences (```), leaving unterminated/empty code blocks and breaking Markdown formatting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…eusable canvas preprocessing, and consistent model references

Copilot

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

…sary model loading

Copilot

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

+    if (!enabled) {
+      setStatus('idle');
+      setIsModelReady(false);
+      setError(null);
+      return;
+    }


+  const {
+    isModelReady,
+    isCardDetected,
+    error: cardDetectionError,
+    startDetection: startCardDetection,
+    stopDetection: stopCardDetection,
+    resetDetection: resetCardDetection,
+  } = useCardDetection(videoRef, {
+    modelUrl,
+    wasmPaths,
+    confidenceThreshold,
+    detectionIntervalMs: 500,
+    stableDetectionsRequired: 2,
+    allowedMisses: 1,
+  });


+      <div
+        className={cn(
+          'absolute inset-4 rounded-lg border-4 transition-colors duration-300',
+          ready ? 'border-success' : 'border-neutral-100/50'
+        )}
+      >


BosuBose132 added 14 commits June 17, 2026 16:44

chore(card-capture): add browser ONNX runtime dependency

1aa71fc

feat(card-capture): add browser ONNX model loader

15a5348

feat(card-capture): add YOLO frame preprocessing

0bca995

feat(card-capture): add YOLO prediction post-processing

2b860af

docs(card-capture): add live browser inference story

e252343

fix(card-capture): make frame stability configurable and relax CardCa…

093e6eb

…pture thresholds for minor handheld movement

test(card-capture): cover camera startup, capture flows, retake, perm…

6711c1e

…issions, model fallback, confirmation, and cleanup

build(card-capture): expose a tree-shakeable component entry and exte…

3ef0a50

…rnalize ONNX Runtime Web

docs(card-capture): document the model contract, runtime assets, dete…

203fd91

…ction pipeline, testing, and maintenance constraints

chore(card-capture): include a versioned ONNX model for zero-setup St…

0d4b834

…orybook testing

fix(card-capture): make the camera and preview fit within the visible…

0948304

… Storybook viewport

Copilot AI review requested due to automatic review settings June 20, 2026 19:12

Copilot started reviewing on behalf of BosuBose132 June 20, 2026 19:12 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

BosuBose132 added 2 commits June 20, 2026 13:19

fix(card-capture): address review feedback with lazy model loading, r…

74e77d7

…eusable canvas preprocessing, and consistent model references

chore: merge upstream main and resolve dependency conflicts

c20283d

Copilot AI review requested due to automatic review settings June 20, 2026 20:41

Copilot started reviewing on behalf of BosuBose132 June 20, 2026 20:42 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

Comment thread src/components/CardCapture/loadCardModel.ts Outdated

Comment thread src/components/CardCapture/useCardDetection.ts

Comment thread src/components/CardCapture/CardCapture.tsx Outdated

Comment thread package.json

Potential fix for pull request finding

b1c0f25

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 20, 2026 20:53

Copilot started reviewing on behalf of BosuBose132 June 20, 2026 20:53 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

Comment thread src/components/CardCapture/loadCardModel.ts Outdated

Comment thread package.json

Comment thread src/components/CardCapture/useCardDetection.ts Outdated

Comment thread src/components/CardCapture/useCardDetection.ts Outdated

BosuBose132 added 2 commits June 20, 2026 14:27

fix(card-capture): address review feedback and dependency cleanup

a3358f0

fix(card-capture): respect disabled detection state and avoid unneces…

9571ea6

…sary model loading

Copilot AI review requested due to automatic review settings June 20, 2026 21:36

Copilot started reviewing on behalf of BosuBose132 June 20, 2026 21:37 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/card capture#278

Feat/card capture#278
BosuBose132 wants to merge 19 commits into
mieweb:mainfrom
BosuBose132:feat/card-capture

BosuBose132 commented Jun 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

BosuBose132 commented Jun 20, 2026

Summary

Problem

Solution

Implementation

Features Added

Usage

Testing

Reviewer Instructions

Current Model Scope

Breaking Changes

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants