HandUMI: A modular, robot-free capture system for bimanual arms

01 / Motivation

Why robot-free capture

The robot is valuable at deployment. It should not also be the scarce instrument that every demonstration depends on.

A conventional bimanual leader–follower setup consumes two follower arms and either two leader arms or a VR control rig. Every additional collector still needs access to the physical deployment system, its workspace, and its safety envelope. Moving to another room or site means moving the robot too.

HandUMI separates capture from execution. The human supplies the dexterity and task context; compact wearable hardware records the trajectory and observation; the robot re-enters only after the episode has passed quality checks and kinematic review. This changes the scaling unit from a robot cell to a wearable pair. The separation builds on the original Universal Manipulation Interface, which made portable capture and deployment-time retargeting distinct stages.^[8]Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild RobotsCheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song; Robotics: Science and Systems, 2024.

02 / Architecture

Capture first. Choose the robot later.

The core design decision is to keep the raw capture robot-agnostic. Controller poses, physical jaw widths, camera frames, calibration identities, source timing, and health flags are recorded before a target kinematic model is chosen. A robot profile then supplies TCP frames, joint ordering, limits, inverse-kinematics weights, gripper mapping, and—when available—a measured table-to-robot transform.^[2]HandUMI software repositoryApache-2.0 recorder, validation, conversion, replay, robot profiles, tests, and documentation; source snapshot fd0be87, 20 August 2026.

1. 01DemonstrateNatural bimanual motion
2. 02SynchronizePose, aperture, and video
3. 03ValidateTracking, timing, and continuity
4. 04RetargetCalibration plus robot profile
5. 05PreviewSimulation before hardware

03 / Hardware

A gripper worn as an instrument

HandUMI mounts across the thumb and index/middle fingers. The operator opens and closes the jaws through an ordinary pinch, retaining direct cutaneous and kinesthetic feedback instead of commanding a remote trigger. The structural body, camera mount, encoder actuation, and controller support remain fixed; only the contact tip changes for a new parallel-jaw gripper.^[1]HandUMI hardware repositoryApache-2.0 CAD, bill of materials, assembly files, and calibration resources; source snapshot e58de33, 20 August 2026.

PoseVR tracking

PICO 4 Ultra or Meta Quest 3 controllers supply wrist pose in a shared headset world frame.

ApertureDirect sensing

A Feetech servo encoder records the mechanical opening rather than estimating width from wrist-camera markers.

ObservationWrist RGB

A compact UVC fisheye camera stays task-facing while the VR system handles motion tracking.

ContactSwappable tips

Published CAD covers AgileX PiPER, ARX X5 2023, Dream Gripper, Trossen WidowX AI, and the original UMI gripper.

Reference bill of materials

At the current published prices, mechanical and structural parts total $14.88 and electronics total $94.98, for $109.86 per unit or $219.72 per bimanual pair. These are early reference estimates: they exclude the shared headset, workstation, shipping, taxes, assembly labor, and calibration time.

Listing prices are normalized by pack quantity; each row’s total is the amount allocated to one HandUMI unit. Prices and availability can change by region.

04 / Software

A traceable data lifecycle

The recorder aligns every row to one shared target timestamp. Camera, tracker, and aperture samples retain their own source times and health state, so short failures remain inspectable rather than being silently replaced. Sustained tracking, camera, or encoder loss discards the episode. A strict validator checks duration, synchronization, frozen signals, motion jumps, and invalid state before conversion.^[4]HandUMI technical documentationRecording and validation reference for synchronization, source timestamps, health flags, strict episode checks, and the LeRobot-compatible dataset layout.

handumi doctor --device meta
handumi record --output-dir outputs/handumi-demo
handumi validate outputs/handumi-demo --strict
handumi replay outputs/handumi-demo --robot openarmv1

Captures use a LeRobot v3-compatible layout. Wrist and workspace video, controller pose, gripper width, sensor health, calibration hashes, and coordinate conventions travel together. Conversion creates a target-specific derivative and leaves the robot-agnostic source intact. Controller-to-TCP calibration is selected from an explicit file, an identity-bound dataset snapshot, the robot/device profile, or a documented legacy fallback—in that order.

05 / Current evidence

What the public release supports today

The software ships simulation models for AgileX PiPER, OpenArm, TRLC-DK1, Axol, and I2RT YAM. PiPER and OpenArm also have real-robot teleoperation backends. The remaining entries are replay/simulation integrations unless their repository documentation states otherwise.

Capture

PICO + Meta QuestXRoboToolkit or the Quest compatibility app

Simulation

5 robot modelsOne profile interface for replay and conversion

Physical control

2 backendsAgileX PiPER and OpenArm

Published replay diagnostics

The repository documents target-specific replay on recorded validation episodes. The numbers below measure inverse- kinematics agreement in simulation under provisional table transforms; they are not external measurements of physical TCP accuracy and are not robot policy success rates.

Orientation error and configuration details remain essential: for example, the documented TRLC-DK1 replay reports 22.19° maximum orientation error even with low position error. Full diagnostics are in the replay guide.

06 / Claims boundary

Scope and limitations

07 / Availability

Inspect it. Build it. Extend it.

The HandUMI hardware and core software are released under Apache 2.0. The public repositories include printable STL and editable STEP files, the bill of materials, calibration and quality tooling, robot profiles, tests, and documentation. New fixed-base parallel-jaw embodiments can be added without changing the recorder.

Meta Quest support is provided through a separate public Unity compatibility project reconstructed from the YubiQuestApp APK. It is an independent, best-effort behavioral reconstruction, not recovered original source. The repository does not currently include a license file, so this paper does not label that app itself as open source.^[3]HandUMI Quest App v0.2.1Public Unity compatibility-source snapshot 1597c24, 13 July 2026; an independent behavioral reconstruction with no license file in the current repository.