Episode 3. HiPHI: Designing the Motion Space for Humanoid Intelligence
Humanoid intelligence requires learning over an enormous space of whole-body motions: walking, running, crawling, kneeling, carrying...
read articleEpisode 2. The World Compiler: A Noitom Robotics Blueprint for the Next Infrastructure Layer of Physical AI
Compared to conventional AI systems, embodied intelligence necessitates a considerably richer and more diverse set of data modalities...
read articleEpisode 1. ModalityNet: The Art of Modalities in Human-Centric Data
Compared to conventional AI systems, embodied intelligence necessitates a considerably richer and more diverse set of data modalities...
read articleEpisode 3. HiPHI: Designing the Motion Space for Humanoid Intelligence
Research Team, Noitom Robotics
We Are Open-Sourcing HiPHI
Humanoid intelligence requires learning over an enormous space of whole-body motions: walking, running, crawling, kneeling, carrying, pushing, pulling, kicking, and coordinating the whole body under diverse contact conditions. The fundamental challenge is therefore not simply how many hours of motion data we can collect, but whether the data systematically spans the motion space that matters for humanoid learning.
Existing data sources each capture only part of this goal. Internet videos provide enormous behavioral diversity but lack precise 3D physical states; conventional motion capture offers high kinematic fidelity but is often limited by manually designed action lists; human-object datasets add interaction states but are typically much smaller. Humanoid learning needs these properties together: scale, precision, motion diversity, and physically grounded interaction.
In Episode 1, we introduced HiPHI-MOV as the motion-oriented component of ModalityNet. We now release its research benchmark, HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction. HiPHI contains 617.5 hours of motion data, including 245.7 hours of human-object interaction, captured from 132 performers at 90 Hz. More importantly, HiPHI approaches data collection as a problem of systematically designing and expanding the motion space.
Figure 1. Overview of HiPHI. A large-scale, high-fidelity dataset for humanoid learning.
From Task Lists to Motion Space
Most motion datasets begin with manually written activity lists: walking, sitting, waving, carrying an object, and so on. But semantic diversity is not equivalent to motion diversity. Two semantically different actions may produce almost identical body motions, while a single action such as walking can generate a large family of motions by changing speed, direction, stride, posture, or rhythm. Simply adding more task descriptions therefore does not guarantee broader coverage of the actual motion distribution.
HiPHI addresses this problem using FrameNet [1] as a semantic scaffold. Similar to how ImageNet [2] used WordNet to structure the visual concept space, we use FrameNet to organize a structured motion concept space. Each selected Frame–Lexical Unit (LU) pair is treated as a motion seed, rather than one fixed action. We systematically expand each seed along factors that meaningfully alter motion, including direction, speed, rhythm, amplitude, posture, body-part involvement, support conditions, object properties, contact modes, and trajectories. For example, a Self_motion: walk seed can generate different speeds, directions, routes, strides and postures, while a Cause_motion: push seed can vary across objects, loads, contact locations and pushing trajectories. The objective therefore changes from collecting more task names to systematically expanding a physically meaningful motion manifold.
Figure 2
From FrameNet to HiPHI data construction. Frame–LU motion seeds are systematically expanded through controlled motion and interaction factors before optical motion capture.
The HiPHI Data
HiPHI contains 617.5 hours and approximately 200.1 million released motion frames, derived from 308.7 hours of original optical motion capture and augmented by left-right mirroring. The dataset includes: 371.8 hours of body-only motion, covering locomotion, posture transitions, dynamic motion, balance and whole-body coordination; 245.7 hours of human-object interaction, accounting for 39.8% of the release.
Its semantic layer contains 22 FrameNet frames and 214 Frame–LU motion units, with motions distributed across many performers and a deliberately retained long tail: 46.3% of the total duration lies outside the 50 most common Frame–LUs.
The human-object subset contains 40 real objects across 12 categories, with masses ranging from 0.45 to 6.25 kg. Human BVH motion is synchronized with the corresponding 6-DoF object trajectory and object mesh, preserving human and object states in a common coordinate system. This is important for embodied learning. Sitting without a chair, pulling without the object being pulled, or carrying without the corresponding load may look reasonable as animation, but is physically incomplete as a reference for robot control. Interaction geometry, load and contact all affect how the human body moves.
Motion instances can also be explored interactively through the HiPHI online motion viewer.
Quality Assessment
For motion data used in humanoid learning, scale alone is not sufficient. We evaluate HiPHI from three complementary perspectives: motion-space coverage, body-motion quality, and human-object interaction consistency.
Motion-Space Coverage
To compare different datasets fairly, we first eliminate the advantage caused simply by having more data.
All compared datasets are converted into a common 23-keypoint body representation, normalized in body scale and resampled to 30 FPS. We then sample at most 5,000 motion windows from each dataset and train one shared unsupervised temporal encoder using only reconstruction loss. Dataset labels and semantic annotations are never provided to the encoder. The resulting motion features are projected into a shared latent space for coverage analysis.
Under this dataset-balanced protocol, HiPHI occupies 1,620 motion-space grid cells, compared with 1,438 for BONES-SEED, the strongest large-scale baseline. More importantly, HiPHI achieves an effective occupancy of 1,443 versus 1,114, indicating that its samples are more evenly distributed rather than merely concentrated in a few common regions. Its long-tail share also increased from 10.7% for BONES-SEED to 14.1% for HiPHI. The same advantage remains under different random seeds, t-SNE configurations and grid resolutions.
In other words, the benefit of HiPHI is not simply that it contains more samples. Under the same sampling budget, it reaches more regions of the motion space and distributes data more broadly across them.
Motion-space diversity comparison. All datasets are represented using the same dataset-balanced unsupervised motion encoder and projected into one shared latent space. (a) Global projection with balanced sampled points per dataset. (b) Support-envelope comparison between HiPHI and BONES-SEED, the closest large-scale baseline. (c) Grid-based local-coverage statistics on the same embedding.
Motion Quality
We evaluate physical artifacts including jerk, acceleration, ground penetration, unsupported floating and support-point sliding. HiPHI achieves the best results across all reported body-motion quality metrics among datasets with compatible floor conventions. Its 95th-percentile ground penetration is only 8 mm, unsupported floating accounts for 0.015% of the evaluated duration, and support-point drift is 64 mm/s.
These properties matter because motion tracking is ultimately a physical control problem. Small artifacts in human trajectories can become unrealistic contacts, unstable balance, or unnecessary optimization difficulty after robot retargeting.
Physically Grounded Object Interaction
For human-object motion, the human and object trajectories must also remain geometrically consistent.
HiPHI achieves a 98.1% non-conflict rate and 95.7% near-surface grounding, compared with 79.0% for HIMO and 50.4% for OMOMO. The goal is not for the human and object to look independently plausible, but to form one physically coherent interaction.
How to Use HiPHI
HiPHI is designed as reference data for humanoid motion tracking, whole-body control and loco-manipulation learning. A typical pipeline first retargets human motion to a humanoid embodiment, then trains a physics-based RL policy to reproduce the trajectory while satisfying robot dynamics and contact constraints. To test whether dataset quality translates into robot learning performance, we retarget multiple datasets to the same Unitree G1 embodiment and train them with the same imitation-learning setup. Importantly, each dataset receives the same training-data budget.
Under both 3-hour and 20-hour matched-budget comparisons, HiPHI achieves the highest tracking success rate and fastest convergence. The advantage therefore does not come simply from having more data: HiPHI provides motion references that are more suitable for physical humanoid execution.
Whole-body humanoid tracking under matched training-data budgets. (a) Success rates. (b) Training failure-rate curves.
What Happens When Motion Data Scales?
We further increase HiPHI training data from 3 hours to 300 hours and evaluate the learned humanoid policy on motions from AMASS, BONES-SEED, Motion-X++ and LAFAN1. Across all four external benchmarks, increasing the amount of HiPHI data consistently reduces tracking MPJPE. This gives an encouraging result for large-scale humanoid learning: motion tracking continues to benefit from scaling high-quality human motion data, rather than rapidly saturating at a small dataset size.
Scaling HiPHI training data from 3 to 300 hours consistently improves cross-dataset humanoid tracking.
Open Release
The HiPHI release provides standardized BVH motion, synchronized object trajectories and meshes, Frame–LU indices, natural-language descriptions and performer metadata. The current release contains approximately 200.1 million frames and 215 GB of data.
HiPHI is available through Hugging Face under the ModalityNet Open Research License v1.0 for non-commercial scientific research, education and evaluation. We also provide open-source tools and an online interactive motion viewer.
HiPHI represents one layer of the broader ModalityNet data stack introduced in Episode 1. Its goal is specific: to build a large-scale, precise and physically grounded foundation for understanding how the human body moves and interacts with the physical world.
Our longer-term goal is to connect this motion foundation with richer sensing modalities, open-world human behavior and robot-specific data, enabling embodied models to learn increasingly general physical intelligence from scalable human-centric data.
If you are interested in HiPHI, ModalityNet, or research collaboration, please feel free to contact us.
References
[1] Baker C. F., Fillmore C. J., Lowe J. B. The Berkeley FrameNet Project. Proceedings of COLING-ACL, 1998.
[2] Deng J., Dong W., Socher R., Li L., Li K., Fei-Fei L. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
[3] Mahmood N., Ghorbani N., Troje N. F., Pons-Moll G., Black M. J. AMASS: Archive of Motion Capture as Surface Shapes. ICCV, 2019.
[4] Peng X. B., Abbeel P., Levine S., van de Panne M. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. ACM Transactions on Graphics, 2018.
[5] Yang L., Huang X., Wu Z., et al. OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction. arXiv preprint, 2025.
[6] Liao Q., Truong T. E., Huang X., et al. BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion. arXiv preprint, 2025.
Citation
@article{ji2026hiphi,
title={HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction},
author={Ji, Jiahao and Ma, Ji and Zhang, Runhan and Yu, Runyi and Wang, Wenjia and Chi, Weiheng and Peng, Qianqian and Yan, Weichao and Gu, Yongfei and Tian, Ye and Wu, Ting and Li, Longwei and Yuan, Chun and Dai, Ruoli and Han, Lei},
journal={arXiv preprint arXiv:2608.16222},
year={2026}
}