Same Scene, Different Story:
Evaluating Robot Perception Across Scene Phases in the Wild

Jishnu Jaykumar P^1*  ·  Itay Kadosh^2*  ·  Narendhiran Vijayakumar^3*  ·  Srinanditha Kamath⁴  ·  Sai Haneesh Allu¹
Govind Tyagi Rangappa¹  ·  Animesh Maheshwari¹  ·  Jikai Wang¹  ·  Yu Xiang¹

¹UT Dallas    ²UF    ³IIIT Hyderabad    ⁴UT Austin
_{^*Equal contribution}

_{The same real scenes before, during and after manipulation, scored on six perception tasks.}

RPX is a benchmark and toolkit for robot perception in scenes that a robot is changing. We recorded 100 real scenes before (Clutter), during (Interaction) and after (Clean) manipulation, plus an egocentric view, and score six tasks on the same frames. Two diagnostics summarize every model: phase robustness Φ and worst-phase quality 𝒥_min.

0 of 65	2–4×	5–11×	0.23
evaluations both accurate and robust across phases	near-range video depth error during Interaction	more tracker identity switches during Interaction	best worst-phase score for in-context grounding

_{Results across 49 models and 65 evaluations in the paper.}

Citation  ·  Getting started  ·  Tasks  ·  Repository layout  ·  Documentation

Citation

@misc{rpx2026,
  title  = {Same Scene, Different Story: Evaluating Robot Perception Across
            Scene Phases in the Wild},
  author = {{Jishnu Jaykumar P} and Kadosh, Itay and Vijayakumar, Narendhiran and
            Kamath, Srinanditha and Allu, Sai Haneesh and Rangappa, Govind Tyagi and
            Maheshwari, Animesh and Wang, Jikai and Xiang, Yu},
  year   = {2026},
  note   = {Dataset: \url{https://huggingface.co/datasets/IRVLUTD/RPX}}
}

Getting started

The rpx-benchmark package provides the data, an adapter per task, the paper's benchmark protocol, and the Φ and 𝒥_min calculator. You write one function that runs your model.

pip install 'rpx-benchmark[hub,schemas]'

The example below scores an image-depth model on a 6-frame trial. The download fetches only this task's files (about 160 MB) and needs no login.

import numpy as np
import rpx_benchmark as rpx

# 1. Download the task's data
manifest = rpx.download_split("monocular_depth", "easy", max_samples=6)

# 2. Your model: RGB image (H, W, 3) in, depth (H, W) in metres out
def predict(rgb):
    return np.ones(rgb.shape[:2], np.float32)

# 3. Wrap it in the task's adapter and run the benchmark
model = rpx.make_numpy_depth_model(predict, name="my-model")
result, report, paths = rpx.run_monocular_depth(rpx.MonocularDepthRunConfig(
    model=model, manifest_path=str(manifest), output_dir="rpx_results/my-model",
))
print(result.aggregated["absrel"], result.aggregated["delta1"])

Φ and 𝒥_min compare phases, so they need all three phases of many scenes. Drop max_samples to download the full tier, run the benchmark again, then:

from rpx_benchmark.examples.summarize_metrics import calculate, read_records

metrics = ["absrel", "rmse", "silog", "delta1"]
summary = calculate(read_records(paths["json"]), metrics)

To try any task before writing code, rpx.benchmark("T2", rpx.download("T2", max_samples=2)) runs it with a placeholder model.

Tasks

Every task follows the steps above with its own download name, adapter and runner.

Task	Download	Adapter and runner	Paper metrics
T1 Image depth	"monocular_depth"	make_numpy_depth_model run_monocular_depth	AbsRel, RMSE, SILog, δ1
T2 Video depth	"video_depth"	make_numpy_video_depth_model run_video_depth	T1's, plus TGM, TGSE
T3 Object tracking	"object_tracking"	make_numpy_tracking_model run_object_tracking	HOTA, MOTA, IDSW
T4 Relative camera pose	"relative_camera_pose"*	make_numpy_pose_model run_relative_pose	AUC at 5°, 10°, 20°
T5 VQA	rpx.download("T5")	vqa.evaluate_vqa	Acc@0.5, GIoU, center-in-GT
T6 In-context VQA	rpx.download("T6")	vqa.evaluate_vqa	Acc@0.5, GIoU, center-in-GT

_{* Pass manifest_name="relative_pose" as well. Functions are under rpx; vqa is rpx_benchmark.vqa, and its models return text, so they need no adapter. rpx.download("T5") is a 58-question quick start; full=True gives the paper's 21,000 (10,500 for T6). dummy_models.py shows the inputs and outputs of every task.}

To add a task or a metric, see the toolkit docs on tasks and metrics.

Repository layout

Folder	Contents
benchmark/	the rpx-benchmark package, its tests and the docs source
docker/	images with the models evaluated in the paper
data/capture/	capture rig and recording scripts
data/mask_annotation/	the instance-mask annotation pipeline
tracking/	text prompts for text-prompted trackers (T3)
tools/	dataset card, dataset QA, docs and figure builders

Documentation

• Toolkit docs: model contracts per task, the evaluation protocol, Φ and 𝒥_min, annotation tools
• Dataset: 100 scenes, 70 objects and 133K frames on Hugging Face
• Docker images: which image holds which of the paper's models
• Python package: the rpx-benchmark API, also on PyPI

---

_{Code under the MIT License, dataset under CC BY 4.0. Questions: open an issue or contact Jishnu, Itay or Naren.
Built at the Intelligent Robotics and Vision Lab, The University of Texas at Dallas.}