A Smarter and Faster On-Device AI Brain for Robots

English · 中文

MiniCPM-Robot is MiniCPM's embodied intelligence model family for real-world perception, decision-making, and action. The first models include:

• MiniCPM-RobotManip: 🦾 A 1.5B generalist VLA for robot manipulation (sim & real). One set of weights across tasks; beats larger models such as π₀.₅ (3B) and Qwen-VLA (5B+) on representative evals. Streaming inference preserves native memory capabilities and maintains the same response speed as before.

• MiniCPM-RobotTrack: 🎯 The first fully on-device embodied target tracker (0.9B). Covers static, dynamic, and adversarial targets; open-source SOTA on EVT-Bench; runs 5+ FPS / ~180 ms on Unitree Go2 EDU with fully local, vision-only natural-language tracking.

🎉 News

• [2026.07.19] 🔥🔥🔥 We release and open-source MiniCPM-Robot, MiniCPM's first embodied intelligence model family for interaction with the physical world. Its first releases are MiniCPM-RobotManip for generalist robot manipulation and MiniCPM-RobotTrack for embodied target tracking. Try it now!

• [2026.07.19] 🚀🚀🚀 PhyAI adds Day-0 support for MiniCPM-Robot, increasing inference throughput on NVIDIA H20 from 10 Hz to 37 Hz through CUDA Graph and custom Triton fused kernels.

Contents

• MiniCPM-RobotManip
  • Benchmark Results
  • Quick Start
  • Inference
• MiniCPM-RobotTrack
  • Benchmark Results
  • Quick Start
  • Cookbook
• Model Zoo

MiniCPM-RobotManip

MiniCPM-RobotManip is a 1.5B vision-language-action model for embodied manipulation with the following highlights:

• Generalist Manipulation: A unified 1.5B generalist policy that covers all downstream tasks with one set of weights.
• Streaming Context: Historical observations are incorporated into the model context through streaming inference. With 60 frames of history, traditional recomputation requires 125 TFLOPs per decision step, while streaming inference needs only 3.3 TFLOPs. The model supports up to 1 minute of visual context memory while keeping online cost comparable to traditional single-frame reactive inference.
• Efficient Inference: Inherits MiniCPM-V 4.6's visual token compression, reducing each frame from 256 to 64 visual tokens for 4× compression. With H100, BF16, and single-frame input, model-forward latency per decision step is 120 ms, compared with 234 ms for π0.5. The measurement excludes task autoregressive decoding.

Benchmark Results

Method	Eval Setting	Open Weights	Model Size	LIBERO	Calvin (ABC→D)	RoboTwin2 (clean+random)		RMBench
						easy	hard
π₀	Specialist	✅	3B	94.4	3.9	65.9	58.4	—
π₀.₅	Specialist	✅	3B	96.9	4.1	82.7	76.8	10.4
Abot-M0	Specialist	✅	4B+	98.6	—	86.1	85.1	—
StarVLA-α	Generalist	✅	4B+	97.8	—	88.7	87.8	—
Qwen-VLA	Generalist	❌	5B+	97.9	—	86.1	87.2	—
LingBot-VA	Specialist	✅	5B+	98.5	—	92.9	91.6	—
MiniCPM-RobotManip	Generalist	✅	1.5B	97.5	4.1	91.3	91.6	53.3

Quick Start

Install and initialize Conda first. The specification follows the tested Python 3.10 and PyTorch 2.6.0 (CUDA 12.4) setup.

cd MiniCPM-RobotManip
conda env create -f environment.yml
conda activate MiniCPM-RobotManip

Inference

Run single-sample inference with vla_infer.py. Provide at least one image and a language instruction; robot state defaults to zeros if omitted. The model returns an action chunk of shape (30, 80).

cd MiniCPM-RobotManip
python vla_infer.py \
    --image frame.jpg \
    --text "Pick up the red block." \
    --checkpoint ./checkpoint \
    --state-file state.npy \
    --embodiment-id 0 \
    --output action.npy

Use multiple --image flags for multi-view inputs. Without --output, the predicted action is printed as JSON.

cd MiniCPM-RobotManip
python vla_infer.py \
    --image cam_front.jpg \
    --image cam_wrist.jpg \
    --text "Pick up the red block." \
    --checkpoint ./checkpoint

MiniCPM-RobotTrack

MiniCPM-RobotTrack is a compact vision-language-action policy for embodied target tracking built on MiniCPM4-0.5B with following highlights:

• Quality-driven self-evolving data pipeline: automated checks and manual review remove abnormal trajectories, incorrect actions, and invalid interactions, while continual model-environment interaction adds high-value training samples.
• DAgger for embodied tracking: after learning from large-scale general scenarios, the model interacts with simulators and real robots to expose failures in long-tail cases such as target crossings, rapid turns, short occlusions, and multi-person intersections. Samples corrected by expert policies or rules are aggregated into the next training round to continuously improve tracking and generalization.
• End-to-end Go2 optimization: joint optimization across visual capture, input encoding, inference, action generation, command transmission, and execution delivers a stable 5+ FPS with approximately 180 ms end-to-end latency on the Unitree Go2's native onboard compute.
• One-command Go2 Edu deployment and launch: the local deployment workflow covers environment setup, dependencies, model loading, camera input, text commands, robot control, and a one-command deployment-and-launch workflow, enabling fully local, vision-only natural-language tracking without rebuilding a separate perception, planning, and control stack.

Examples

Outdoor Obstacle-aware Tracking	Elevator Tracking	Underground Parking Tracking

EVT-Bench Results

Results are reported as SR / TR / CR: success rate and tracking rate are higher-is-better, while collision rate is lower-is-better. All values are percentages.

Method	Open Weights	Model Size	STT			DT			AT
			SR ↑	TR ↑	CR ↓	SR ↑	TR ↑	CR ↓	SR ↑	TR ↑	CR ↓
TrackVLA	❌	7.8B+	85.1	78.6	1.7	57.6	63.2	5.8	50.2	63.7	17.1
TrackVLA++	❌	7.8B+	86.0	81.0	2.1	66.5	68.8	4.7	51.2	63.4	15.9
Qwen-RobotNav	❌	4.4B+	77.4	90.0	6.4	—	—	—	—	—	—
OmTrackVLA	✅	1.0B+	81.4	82.8	5.1	41.5	58.8	11.3	60.0	73.9	7.6
MiniCPM-RobotTrack	✅	0.9B	84.1	89.8	3.0	53.2	73.4	13.6	58.0	80.4	9.0

Among the open-source checkpoints shown above, MiniCPM-RobotTrack obtains the best STT SR/TR/CR and the best DT SR/TR. It also reaches 80.35 TR on AT with a 0.5B backbone.

Quick Start

1. Create the environment

Install and initialize Conda first. The specification follows the tested Python 3.9, Habitat-Sim 0.3.1, Bullet, PyTorch 2.4.1, and CUDA 12.1 setup.

cd MiniCPM-RobotTrack
conda env create -f environment.yml
conda activate MiniCPM-RobotTrack

2. Prepare simulator data and assets

Download HM3D, MP3D, humanoid, and robot assets following the Habitat-Sim data instructions and TrackVLA asset instructions. Preserve their original directory structure under the project's data/ directory.

data/
├── datasets/
├── scene_datasets/
├── humanoids/
└── robots/

Data Preparation

1. Unprocessed rollouts

Each Habitat rollout retains its video, per-step simulator state, and episode result. The runnable sim_data/raw_sample example includes per-step records in the following format:

{
  "base_velocity": [0.62, 0.00, 0.02],
  "collision": false,
  "target_distance": 1.72,
  "human_center_norm": [0.50, 0.48]
}

2. Generate trajectory data

The command below keeps successful episodes, extracts RGB frames, and converts future actions into trajectory JSONL for finetuning:

cd MiniCPM-RobotTrack
python tools/make_tracking_data.py \
  --input_root sim_data/raw_sample \
  --output_root sim_data/train/stt \
  --only_success \
  --history 31 \
  --horizon 8 \
  --dt 0.1 \
  --incremental

For the full dataset, place the raw STT, DT, and AT rollouts under sim_data/raw/<task> and run the same command for each task.

3. Pre-cache visual features

cd MiniCPM-RobotTrack
for task in stt dt at; do
  python tools/precompute_features.py \
    --json "sim_data/train/${task}/jsonl" \
    --data-root "sim_data/train/${task}" \
    --cache-root "sim_data/train/${task}/vision_cache"
done

The processed outputs contain frames/, jsonl/, and vision_cache/. sim_data/sample provides two processed records for each of STT, DT, and AT, showing the final training-data format.

Finetuning

After preparing data and visual caches for all three tasks, run the public finetuning entry point:

cd MiniCPM-RobotTrack
bash scripts/train.sh

Data paths, batch size, learning rates, and epoch count can be adjusted in scripts/train.sh.

Evaluation

Download the complete released Hugging Face snapshot to minicpm_robot_track/checkpoints/MiniCPM-RobotTrack/. It contains the fine-tuned MiniCPM4 backbone and is loaded with Transformers from_pretrained. Evaluation still requires DINOv3 ViT-S/16 and SigLIP So400m; existing local vision-model directories can be selected with environment variables:

export DINOV3_MODEL_PATH=/path/to/dinov3-vits16
export SIGLIP_MODEL_PATH=/path/to/siglip-so400m-patch14-384

After the environment, models, and simulator assets are ready, run the evaluation script for each task:

cd MiniCPM-RobotTrack
CKPT=minicpm_robot_track/checkpoints/MiniCPM-RobotTrack
bash scripts/eval_stt.sh "$CKPT" results/stt
bash scripts/eval_dt.sh  "$CKPT" results/dt
bash scripts/eval_at.sh  "$CKPT" results/at

Each command runs the evaluation for the corresponding task.

Go2 Deployment

The real-robot workflow targets a Unitree Go2 EDU with a Jetson Orin NX 16GB. The validated stack is Jetson Linux R36.5, CUDA 12.6, TensorRT 10.7, Python 3.10, ROS 2 Humble, and MAXN mode 0. Deployment uses separate Jetson dependencies and defaults to dry-run, which never sends motion commands.

Go2/D435i RGB -> TCP JPEG -> DINO + SigLIP TensorRT
              -> MiniCPM-RobotTrack -> waypoint -> rate-limited control

The complete setup uses a Go2 EDU, Orin NX 16GB, and D435i. The built-in Go2 front camera is the default validated source; each installation must validate the D435i RGB path separately. Detailed hardware parameters, network settings, asset download instructions, flashing instructions, and live-control procedures are kept in the deployment documentation:

• Go2 deployment and reproduction guide
• Model and deployment assets
• JetPack 6 upgrade guide

Quick Start

The following assumes JetPack 6 and the carrier-board patch are already installed:

Download the checkpoint manually from openbmb/MiniCPM-RobotTrack. Place the complete Hugging Face snapshot, including custom model code, tokenizer files, and model.safetensors, in MiniCPM-RobotTrack/minicpm_robot_track/checkpoints/MiniCPM-RobotTrack/ before running preflight.

cd MiniCPM-RobotTrack

python3 scripts/download_upstream_assets.py

python3 -m pip install --user -r requirements-build.txt
./scripts/export_onnx.sh

sudo nvpmodel -m 0
sudo jetson_clocks
./scripts/build_engines.sh

./scripts/preflight.sh
./go2_runtime.py run

Status and shutdown:

cd MiniCPM-RobotTrack
./go2_runtime.py status
./go2_runtime.py stop-control
./go2_runtime.py stop

Safety: Keep runtime.mode: dry-run for the first run. Before enabling live control, validate the camera, model, latency, and stop path on a stand with an on-site operator and a working remote/App emergency stop. Follow the Go2 deployment guide for the complete live-control procedure and release limits.

Model Zoo

Model	Description	Download
MiniCPM-RobotManip	A 1.5B vision-language-action model for Robot Manipulation	🤗
MiniCPM-RobotTrack	A 0.9B vision-language-action model for Target Tracking	🤗

License

Model weights and code are open-sourced under the Apache-2.0 license.

Acknowledgments

MiniCPM-Robot builds on and references MiniCPM, starVLA, LeRobot, DINOv3, SigLIP, Habitat-Lab, Habitat-Sim, EVT-Bench, and TrackVLA. We thank the authors and communities for their open-source contributions. Third-party models, simulator code, datasets, and assets retain their own licenses; see THIRD_PARTY_NOTICES.md.