PiHand: Latent Hand–Object Interaction Priors

Zero-shot tabletop manipulation of held-out objects along held-out trajectories. Two policies, one trained on GRAB and one trained on DexYCB, both on top of the same GRAB-trained interaction prior.

Abstract

TL;DR We distill physics-based mocap-tracking experts into a hand–object interaction prior. Exploring its latent space with RL, using a single object-tracking reward, yields dexterous manipulation of unseen objects and trajectories, and transfers to catching and bimanual handover.

Synthesizing physically grounded dexterous hand–object interactions is important for VR/AR, gaming, and robotics, yet remains challenging because subtle contact dynamics, diverse object geometries, and target trajectories must be jointly respected while maintaining natural grasps. Existing methods either track mocap trajectories with limited transfer to novel objects, or rely on hand-only generative priors that fail to capture coupled hand–object motion, sacrificing contact fidelity and sample efficiency.

To tackle these challenges, we introduce PiHand, a Physically grounded Interaction-aware Hand motion synthesis framework. PiHand first trains expert policies that track mocap demonstrations in simulation, distills them into a unified hand–object interaction prior using a goal-conditioned VAE, and finally solves diverse dexterous manipulation tasks through bounded residual exploration in its latent space. With only a single reward and no complex curriculum training, this yields general tabletop manipulation policies that generalize to unseen objects and trajectories on GRAB and DexYCB. The same prior transfers to object catching and bimanual handover, two tasks that are rarely seen in its training data. Across all tasks, PiHand consistently outperforms baselines in all metrics.

93.3%

Mocap imitation success

Stage I experts on GRAB50, vs. 69.6% for ManipTrans.

+14.0 / +16.9

Success points on unseen objects

Tabletop manipulation on GRAB / DexYCB, over the strongest baseline.

10×

Faster training

Reaches the best baseline's peak in 900 instead of 8900 epochs.

2.0× / 4.3×

New tasks

Success rate relative to the strongest baseline, on catching / handover.

Method

An interaction-first latent space

Instead of hand-only motion priors or reward engineering, PiHand embeds the hand–object interaction itself into the latent space, in three stages.

Stage I

Physics-grounded interaction tracking

PPO experts track GRAB mocap in simulation as smooth residual corrections around the reference, trained with a contact-aware reward (object surface samples, reference contacts, hand keypoints). Mocap records kinematics only; the experts turn it into physically valid interactions. One expert per object for single-handed and one for bimanual motions.

Stage II

Interaction prior distillation

All experts are distilled with DAgger into a conditional VAE. Its learnable prior sees only the future object trajectory; the privileged encoder also sees the reference hands. Prior and decoder together form an interaction-aware model of physically simulated hand–object motion that needs no reference hands at test time.

Stage III

Bounded latent exploration

For a new task, an RL policy outputs a residual on the frozen prior's mean, squashed to a norm bound B, and the frozen decoder turns it into hand actions. Exploring inside the prior keeps grasps natural and makes RL sample-efficient.

Stage I

High-fidelity imitation of human mocap

Our tracking experts reproduce GRAB interactions in physics, the data engine behind the interaction prior.

Method	Data	SR ↑	TC ↑	E_t (cm) ↓	E_r (°) ↓	MPJPE (cm) ↓
ManipTrans (per-seq.)	GRAB50	69.6	77.6	0.43	7.3	1.97
Ours (per-seq.)	GRAB50	93.3	95.8	0.28	4.9	1.01
Ours (per-object)	GRAB full	84.8	92.2	1.04	15.4	2.08

Physical imitation of mocap reference motions. GRAB50 is a random subset of 50 single-handed sequences, since ManipTrans does not support bimanual manipulation of one object. Training two experts per object (single-handed and bimanual) scales to the full dataset while still beating per-sequence ManipTrans.

Explore the expert rollouts

Per-object expert policies tracking GRAB sequences. The reference is shown as a translucent red ghost; toggle it below the viewer.

Drag to orbit, scroll to zoom, right-drag to panPick a sequence on the left

Stage III · Task 1

Tabletop manipulation of unseen objects

Given only a target object trajectory, a single policy must synthesize a physically plausible interaction for objects and trajectories it has never seen. One policy per dataset, both on top of the same GRAB-trained prior.

	GRAB · unseen objects				DexYCB · unseen objects
Method	SR ↑	TC ↑	E_t (cm) ↓	E_r (°) ↓	SR ↑	TC ↑	E_t (cm) ↓	E_r (°) ↓
Action Space Exploration	59.1	73.6	1.79	21.5	60.8	81.3	1.63	14.2
Hand-Only Prior	60.0	73.4	1.52	21.2	59.5	79.4	1.59	16.1
PiHand (ours)	74.0	82.9	1.36	20.0	77.7	87.4	1.52	13.8

Success rate (SR, %), tracking completion (TC, %) and object translation / rotation error on held-out objects and trajectories. The interaction prior is trained on GRAB only, so the DexYCB column also tests cross-dataset transfer. Both baselines additionally get a pre-grasp reward and a difficulty curriculum; ours needs neither.

Side-by-side comparison

Action Space Exploration explores the 102-dimensional joint space without a prior. Hand-Only Prior (as in Omnigrasp / PULSE) explores a latent space distilled from experts that track hand motion without the object. Six held-out sequences, three from GRAB and three from DexYCB; the cameras of the three panels are linked.

Rotating one panel rotates all three"Reference" shows the target object trajectory

More zero-shot rollouts

Our policy on held-out GRAB objects (camera, toothpaste, apple, mug and binoculars), single-handed and bimanual.

Grasp diversity

The interaction prior does not collapse to a single way of holding an object. All fifteen of our rollouts on held-out mug sequences, ordered by grasp: with two hands, by the handle, around the body from different sides, and from above at the rim. The final frame shows every hold pose around an identically oriented mug.

Stage III · Tasks 2 & 3

Transfer to catching and handover

Does the prior transfer to dynamics that are absent or rare in its training data? Same exploration policy, object split and baselines; the object-tracking reward is complemented by simple task-specific terms.

Catching

An object with random orientation and spin is launched on a ballistic arc towards an intercept point; the hands start open, palms facing, at a random offset. We train a bimanual and a right-hand-only variant.

Handover

The object starts in the left hand and the reference moves it in a straight line to the right side of the workspace. Each hand may only stay in its half of the workspace, so the object can only get there through a proper change of hands.

Method	Catching	Handover
Action Space Exploration	43.0	11.9
Hand-Only Prior	1.0	5.0
PiHand (right hand)	78.0	–
PiHand (ours)	87.0	51.9

Success rate (%) on held-out GRAB objects.

"Reference" shows the object's target path as a translucent red ghostTraining and held-out objects are labelled in the list

Analysis

What makes it work

All analyses on tabletop manipulation with held-out GRAB objects.

Stage III components

Variant	SR ↑
Direct prior rollout (no exploration)	30.8
Action-space residual	49.6
w/o random trajectories	67.9
PiHand (ours)	74.0

Stage III more than doubles the frozen prior's success. Exploring in the latent space beats correcting the decoded actions, and random trajectory augmentation adds about six points.

Stricter success criteria

Method	SR @ 10 cm	SR @ 7.5 cm	SR @ 5 cm
Action Space Exploration	59.1 ± 4.7	50.0 ± 4.0	37.2 ± 4.0
Hand-Only Prior	60.0 ± 3.1	55.0 ± 2.6	46.6 ± 2.4
PiHand (ours)	74.0 ± 0.9	69.9 ± 1.1	63.5 ± 1.1

Mean ± std over 3 seeds. With a tighter termination threshold our advantage grows from 14.0 to 16.9 points: the policy tracks the target closely rather than just staying inside the tolerance.

Residual bound: success vs. diversity

Success rises as the latent residual bound B is relaxed. Up to B = 5 the deployed latents stay within the prior's distribution and hand usage follows the human reference. Looser bounds gain little success, but the latents drift away from the prior and the policy collapses almost entirely to bimanual grasps, so we use B = 5.

Success rate as a function of the residual bound B. — Success rate

Hand usage as a function of the residual bound B. — Hand usage

PCA of deployed latents (blue) against the learned prior (grey) for different bounds. — Deployed latents (blue) vs. prior (grey), PCA

BibTeX

@misc{berndt2026pihandlatenthandobjectinteraction,
      title={PiHand: Latent Hand-Object Interaction Priors for Physics-Based Policy Learning}, 
      author={Niklas Berndt and Xianghui Xie and Enes Duran and Yixiao Wang and Gerard Pons-Moll},
      year={2026},
      eprint={2610.31299},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.31299}, 
}

PiHand Latent Hand–Object Interaction Priors for Physics-Based Policy Learning