ContentsStart

CoRL 2026 · arXiv:2608.16222v2

HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction

Jiahao Ji^1,2,*, Ji Ma^1,5,*, Runhan Zhang^1,3,*, Runyi Yu^1,4, Wenjia Wang^1,5, Weiheng Chi¹, Qianqian Peng¹, Weichao Yan¹, Yongfei Gu¹, Ye Tian¹, Ting Wu¹, Longwei Li¹, Chun Yuan³, Ruoli Dai^1,†, Lei Han^1,†

¹Noitom Robotics²National University of Singapore³SIGS, Tsinghua University⁴The Hong Kong University of Science and Technology⁵The University of Hong Kong

* Equal contribution† Corresponding authors

Abstract

Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing embodied datasets remain fundamentally limited: internet-scale video data lack precise physical states and interaction grounding, while laboratory motion datasets provide high fidelity but only narrow behavioral coverage. This mismatch creates a critical bottleneck for scalable humanoid policy learning. We present HiPHI, a 600+ hour scale high-fidelity whole-body human motion dataset designed to systematically maximize coverage of the human motion and interaction manifold. HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives. Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories. We further introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications. Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets while maintaining high-fidelity interaction quality, and establishes a scalable data foundation for training, evaluating, and generalizing humanoid policies in real-world embodied tasks, where similar extensions are also applicable to motion prior models in computer graphics.

Figure 1: Overview of HiPHI. A large-scale, high-fidelity dataset for humanoid learning.

Keywords: Large-scale Motion Capture, Humanoid Robot Learning, Human-Object Interaction, Reinforcement Learning, Motion Tracking

1 Introduction

Humanoid learning requires reference motion that is both broad in coverage and physically faithful. Robot policies must acquire balance, posture transitions, limb coordination, contact, and load-dependent strategies. However, most existing data sources are not designed to capture high-precision human behavior for humanoid learning. Robot demonstrations and teleoperation data are limited by scale and cost, and are tied to specific embodiments [30, 4]. Human videos and egocentric data provide broad behavioral diversity at a large scale, but are limited to visual observations and therefore need to rely on reconstruction or proxy signals rather than directly measured physical states [6]. MoCap and HOI datasets provide accurate motion and sometimes object state, but are primarily designed for motion synthesis, language grounding, reconstruction, or interaction understanding [27, 7, 2, 23]. Humanoid policy learning, therefore, calls for data that are broad, precise, and grounded in object-constrained whole-body control.

We formulate human data collection for humanoid learning as a problem of motion-space design. Rather than manually curating behavioral scripts, HiPHI uses FrameNet [1, 36] as a semantic scaffold, where frames represent event types and lexical units (LUs) identify the word senses that evoke them. We select frames and LUs most relevant to embodied intelligence, and use each LU under a specific frame as a seed for performer-facing capture scripts. We then expand each seed along multiple dimensions, including direction, speed, amplitude, body posture, body-part involvement, and object/contact conditions, producing a diverse set of repeatable motion instances for systematic capture. This turns handcrafted script authoring into a structured and scalable process for expanding motion-space coverage.

Using this pipeline, we present HiPHI, a 600-hour-scale, high-precision human motion and object-interaction benchmark, all captured from 132 performers using a large optical motion capture (MoCap) system with sub-millimeter spatial tracking accuracy. Specifically, HiPHI combines 371.8 hours of diverse short-horizon whole-body motion and 245.7 hours of real-object interaction data, spanning 214 Frame–LU motion units across 22 frames. The interaction subset covers 40 distinct real-world objects from 12 categories. Each clip is indexed by a Frame–LU label and paired with a natural-language description. For human-object interaction sequences, synchronized object trajectories and meshes are captured together with human motion, making object state an integral part of the motion record. Capturing real objects is essential, as load, friction, inertia, resistance, and contact substantially shape control strategies in ways that pantomime motion cannot reflect.

We evaluate HiPHI in terms of motion-space coverage, data quality, interaction consistency, and humanoid policy learning performance. Motion coverage is measured using label-free kinematic embeddings; data quality is assessed through smoothness, ground contact, and interaction geometry; and downstream applications are tested through humanoid motion tracking and real-world deployment. Together, these analyses show that HiPHI is diverse, physically stable, interaction-grounded, and usable as executable reference data in humanoid learning, substantially surpassing existing datasets.

Our contributions are threefold: (i) a FrameNet-guided motion-space construction pipeline for systematic and scalable coverage of whole-body motion and interaction; (ii) a 617.5-hour, high-precision dataset including 245.7 hours of human–object interaction, synchronized object trajectories and meshes, Frame–LU indexing, and natural-language descriptions; and (iii) a comprehensive evaluation protocol for humanoid robot learning, including motion coverage, data quality, interaction consistency, motion tracking, and real-robot deployment.

Robot Teleoperation and Egocentric Vision Data. Embodied intelligence increasingly demands large and diverse data sources. One major category is real-robot data, which avoids the embodiment gap, including teleoperated robot data [30, 14, 40], manipulation benchmarks [12, 16, 28, 21], and low-cost interfaces such as UMI [4]. These data sources avoid gaps in robot embodiment or end-effector configuration, but are often limited by collection cost, data scale, behavioral diversity, and dependence on specific robots, hardware, sensors, and viewpoints. Recently, egocentric visual data have become increasingly popular, as they provide life-scale observations of long-tail human behaviors in the real world [6, 13, 10, 17, 44]. However, such data are typically limited to visual modalities, making it difficult to recover precise human motion and physical interaction states.

Human Motion Capture Data. Motion capture data can accurately record human motion, with widely used datasets including AMASS [27], LAFAN1 [9], BABEL/KIT/HumanML3D [35, 34, 7], Motion-X/Motion-X++ [20, 43], and the recent BONES-SEED [3]. These datasets were originally designed mainly for motion synthesis, annotation, generation, or animation, but have recently become widely used in humanoid learning. However, most were not systematically designed for embodied intelligence: few simultaneously provide large scale, high-precision motion, systematic motion-space coverage, and synchronized object states. In contrast, HiPHI is a large-scale dataset specifically designed for embodied intelligence and humanoid learning. It uses the well-established FrameNet theory to structure data collection and directly evaluates motion-space coverage.

Human-Object Interaction. Similarly, many HOI datasets have been introduced for motion synthesis, animation, and interaction understanding [37, 2, 11, 22, 18, 26, 23, 41]. However, these datasets are typically small in scale, often only a few hours long, and their contact-level accuracy and interaction diversity remain limited. In contrast, our dataset contains 245.7 hours of HOI data and explicitly emphasizes the physical accuracy of contact-rich interactions. This design supports efficient humanoid learning based on motion imitation methods [31, 33, 32, 39, 24, 45, 8, 19, 25, 42, 38].

3 FrameNet-Guided Motion-Space Construction

From FrameNet to HiPHI data construction. Motion-relevant frames and lexical units define motion seeds, which are expanded through controlled factors such as speed, direction, ampl... — Figure 2: From FrameNet to HiPHI data construction. Motion-relevant frames and lexical units define motion seeds, which are expanded through controlled factors such as speed, direction, amplitude, body-part involvement, and object/contact conditions.

3.1 Motion Coverage

The most suitable data for humanoid robot learning is not necessarily the full distribution of motions that appear in everyday human life. For example, we do not expect robots to over-learn behaviors such as sleeping, eating, or watching television, which robots do not need to perform. Instead, the data most relevant to humanoid learning should cover control-centric motion patterns, including locomotion, posture transitions, limb coordination, dynamic balance, and responses to external constraints. The goal is therefore not to reproduce the full catalog of human activities, but to cover the motion space that matters for humanoid learning.

On the other hand, the human behaviors or tasks in existing motion datasets are often derived from manually designed scripts. These scripts usually describe semantically meaningful daily actions, which makes the resulting motion clips natural and interpretable. However, because the tasks are manually designed at the semantic level, they can lead to substantial overlap in motion space. For example, “raising a hand to wipe sweat from the forehead” and “raising a hand to block sunlight” are two different scripts at the task level, but may produce nearly identical motions. Counting them as two different tasks does not give us two genuinely different motions. The reverse is also true: one simple action such as walking can produce many different whole-body motions when its direction, speed, stride length, turning pattern, or posture changes. A growing list of manually designed scripts therefore provides no clear way to tell which parts of the motion space have already been covered and which are still missing. For humanoid learning, we instead aim to enable robots to span the full reachable human motion space, so that learned policies can better adapt to diverse downstream tasks. This requires a structured set of motion units that can be systematically enumerated and expanded, rather than an open-ended list of task descriptions.

Finally, existing datasets often contain many human-object interaction motions without capturing the corresponding object states, leading to, for example, “a person sitting in the air”. In this example, the body trajectory is recorded, but the chair that provides the support surface and contact relation is missing, leaving the physical interaction incomplete. Data for object-constrained humanoid learning should therefore capture the relevant object states together with the human motion. For instance, pushing, carrying, dragging, or leaning can substantially change posture, foot placement, and center-of-mass motion through the object’s geometry, friction, load, and trajectory. These factors are essential for learning physically grounded humanoid behaviors. HiPHI records these objects together with the human body, preserving the complete interaction as physically executable motion reference.

3.2 A FrameNet-Guided Collection Pipeline

To systematically address the problem of motion-space coverage, we use FrameNet as a semantic foundation for motion construction. FrameNet organizes word meanings according to the events they describe. A frame represents a type of event, while a lexical unit (LU) represents a word used in one particular meaning within that frame. For example, walk, jog, and run under the Self_motion frame describe related forms of self-propelled movement. This distinction is important because the same word may describe completely different events: “run across the field” refers to body motion, whereas “run a company” does not. A Frame–LU pair identifies the first meaning precisely instead of treating every use of the term “run” as the same motion.

This follows the same organizing idea as ImageNet [5], where WordNet [29] provided a structured set of visual concepts to collect: FrameNet provides HiPHI with a structured set of motion meanings to collect. The goal is to avoid relying on manually designed task scripts and instead enumerate the motion space from a theoretically grounded and well-structured system. The action lexical units in FrameNet naturally serve this purpose, making them suitable seeds for generating a broad motion space. Figure 2 shows this complete pipeline from the FrameNet scaffold to the captured HiPHI motion space.

We select frames and lexical units that are most relevant to humanoid robots, including those related to body motion, posture change, directional movement, body-part motion, object actuation, and human-object interaction. Starting from these units, we further expand them along dimensions such as path, direction, speed, rhythm, amplitude, body posture, body-part involvement, support relation, and object/contact conditions. For example, a walking seed can be expanded with different routes, directions, speeds, stride lengths, turning patterns, and postures. A pushing seed can vary the object, load, contact point, pushing direction, and object trajectory. These variations change the actual human or object motion, rather than merely changing the story attached to it. These expanded scripts are then provided to motion-capture performers for execution. In this way, each Frame–LU pair maps to a family of capturable motions rather than a single clip.

Manual scripting versus FrameNet-guided construction. Manual collection grows by adding scripts one by one, whereas HiPHI organizes collection around Frame-LU motion seeds and expa... — Figure 3: Manual scripting versus FrameNet-guided construction. Manual collection grows by adding scripts one by one, whereas HiPHI organizes collection around Frame-LU motion seeds and expands them through shared factors. This turns dataset growth into a structured and scalable expansion of the motion space.

Figure 3 makes the difference from manual script design explicit. In a conventional workflow, scripts are typically written case by case, guided mainly by the designers’ experience and intuition, without an explicit taxonomy or coverage criterion. Dataset growth therefore becomes an ad hoc, trial-and-error process: there is no principled way to determine which motions are still missing, whether a new script genuinely expands the motion space, or whether it merely wraps an already collected motion in a different story. HiPHI instead starts from Frame-LU motion seeds and expands them through a shared set of factors, such as intensity, route, speed, and object conditions. This replaces the blind accumulation of disconnected scripts with a systematic and traceable expansion of motion units and their physical variations. To our knowledge, HiPHI is the first MoCap dataset for robot learning to prospectively adopt such a linguistic scaffold for data construction.

4 Dataset Composition and Statistics

Following this construction procedure, HiPHI is created as a LU-indexed motion dataset containing body-only and object-interaction sequences. This section summarizes its scale, relationship to existing datasets, Frame-LU composition, and object-interaction subset. The file organization and metadata are detailed in Appendix B.

4.1 Data Characteristics

HiPHI contains 617.5 hours of high-fidelity optical motion capture data in BVH format, corresponding to approximately 200.1 million frames. This is obtained by applying left-right mirroring to 308.7 hours of original motion captured with a high-precision optical MoCap system, following the convention used by BONES-SEED [3]. Human motion is captured at 90 Hz from 132 distinct performers, with detailed statistics in Appendix B. For object-interaction sequences, the data include synchronized human motion, object trajectories, and object meshes, making the object state an integral part of the motion record. Table 1 compares HiPHI with existing representative motion and human-object interaction datasets. Existing data often emphasize either large-scale body motion without object state or smaller-scale object-centric interaction. In contrast, HiPHI combines large-scale high-precision optical MoCap, Frame-LU indexing, synchronized object state, and a benchmark protocol designed for humanoid robot learning.

Table 1: Comparison with representative human motion and interaction datasets. HiPHI offers 617.5 hours of MoCap data, including 245.7 hours of object-state-aligned motion, that exceeds representative human motion and interaction datasets in scale and interaction coverage.

Dataset	Hours	Capture	Format	Motion index	Object motion
HiPHI	617.5	MoCap	BVH	Frame-LU	245.7 h
AMASS	$>$ 40	MoCap	SMPL	–	–
BONES-SEED	288.3	MoCap	SOMA/G1	NL segments	–
Motion-X++ (orig.)^†	40.4	Video	SMPL-X	text + pose	–
LAFAN1	4.6	MoCap	BVH	action themes	–
GRAB	3.8	MoCap	SMPL-X	intent labels	3.8 h
OMOMO	9.8	MoCap	SMPL-X	–	9.8 h
HIMO	9.4	MoCap	SMPL-X	text seg.	9.4 h

^† Original Motion-X++ data only, where its integrated third-party sources are excluded. NL indicates Natural Language.

4.2 Frame-LU Composition and Long-Tail Coverage

Each clip in HiPHI is paired with a Frame-LU label and a natural-language description. The Frame-LU labels also serve as indices for retrieval, sampling, and analysis. The 214 Frame-LU labels across 22 FrameNet frames, placing each motion meaning within its event context rather than relying on ambiguous motion names. As shown in Figure 4, the dataset spans both frequent motion units, such as locomotion, posture changes, and body-part movements, and a long tail of more dynamic, irregular, and constrained patterns. The top 50 Frame-LUs account for 53.7% of the released duration, leaving 46.3% distributed across the remaining long-tail motion units. Motion scripts generated from each LU are performed by approximately 24 different actors on average, and 154 Frame-LUs are performed by at least 10 actors, providing substantial performer variation within individual motion units.

Frame-LU composition and long-tail structure. The figure shows duration by FrameNet frame, the Frame-LU duration distribution, and cumulative duration share, illustrating that HiPH... — Figure 4: Frame-LU composition and long-tail structure. The figure shows duration by FrameNet frame, the Frame-LU duration distribution, and cumulative duration share, illustrating that HiPHI combines common motion units with a broad long tail of whole-body patterns.

4.3 Body-Only and Object-Interaction Motions

HiPHI includes 371.8 hours of body-only motion data and 245.7 hours of human-object interaction motion data. The body-only subset covers self-motion, posture changes, body-part movement, dynamic motion, and coupled whole-body coordination. The object-interaction subset records motions shaped by real geometry, contact, load, and object motion, such as sitting, leaning, supporting, pushing, pulling, and carrying. It contains 40 objects across 12 categories, ranging from furniture and containers to cleaning tools and sports equipment, with masses from 0.45 to 6.25 kg. The subset spans 90 Frame-LUs across 15 FrameNet frames. Together, HiPHI provide a large-scale reference base for physically grounded interaction analysis and object-constrained humanoid learning.

5 Experiments

In this section, we create the HiPHI benchmark and evaluate HiPHI along two axes: dataset-level coverage and quality (Secs. 5.1, 5.2), and downstream utility through humanoid tracking in simulation and sim-to-real deployment (Secs. 5.3, 5.4).

Full kinematic motion-space visualization. All datasets are encoded by the same unsupervised body-motion encoder and projected into one shared t-SNE space. (a) Global projection wi... — Figure 5: Full kinematic motion-space visualization. All datasets are encoded by the same unsupervised body-motion encoder and projected into one shared t-SNE space. (a) Global projection with balanced sampled points per dataset. (b) Support-envelope comparison between HiPHI and BONES-SEED, the closest large-scale baseline; HiPHI’s support nearly encloses BONES-SEED. (c) Grid-based local-coverage statistics on the same embedding using a $55{\times}55$ grid.

5.1 Motion-Space Diversity

We use a dataset-balanced protocol to compare kinematic motion-space coverage. All datasets are first mapped to a unified 23-keypoint, 30 FPS, root-aligned, and body-scale-normalized representation, and each sequence is divided into one-second windows. For shared-encoder training, we randomly sample 5,000 clips from each dataset and draw one window from each sampled clip. One exception is LAFAN1 that its full dataset contains fewer clips than 5,000, and thus we use its all available clips for comparison. We then train a shared temporal-convolutional autoencoder using only a reconstruction objective and project its 16-D latent codes into a common t-SNE space. The visualization and occupancy analysis likewise use at most 5,000 windows per dataset (see Appendix E). Figure 5 shows that HiPHI spans the broadest kinematic region among the compared datasets, covering most regions occupied by the other datasets while extending into additional parts of the motion space. We further discretize the embedding and report three statistics (defined in Appendix E): occupied cells (how wide the coverage is), effective occupancy (how uniform the coverage is across occupied cells), and long-tail share (the fraction of samples in globally rare cells). HiPHI covers more grid cells than the closest baseline BONES-SEED (1620 vs. 1438), with larger effective occupancy (1443 vs. 1114) and higher long-tail share (14.1% vs. 10.7%), indicating broader and more uniform coverage rather than denser sampling of common motions. In other words, HiPHI not only reaches more regions of the kinematic space, but also distributes its motions more broadly across those regions, under the same randomly sampled size. We further repeat the coverage analysis across multiple random seeds, t-SNE perplexities, and grid resolutions, and refer the detailed results to Appendix E. HiPHI retains a positive occupied-cell margin over the strongest baseline in every tested setting, confirming that its coverage advantage is consistent across projection and discretization choices.

5.2 Data Quality

Body-motion precision.

We measure five quantities (lower is better; aggregation defined in Appendix F) that capture common failure modes in humanoid imitation: jerk $\mathcal{J}_{\tau}$ (m/s³), acceleration $\mathcal{A}$ (m/s²), ground penetration $\delta_{\text{ground}}$ (mm), unsupported-floating share $\phi_{\text{float}}$ (%), and support-point drift $\nu_{\text{foot}}$ (mm/s). $\mathbb{Q}_{\tau}$ and $\mathbb{E}$ denote the upper-tail quantile and mean ( $\tau{=}0.95$ ); $\nabla_{t}$ is a finite-difference operator divided by the source-specific frame interval $\Delta t$ , so that $\nabla_{t}^{k}x$ has units of m/s^k; $\bar{x}_{j}$ and $x_{j}$ are smoothed and raw joint positions; $\mathcal{C}$ is the core-body joint set; $\mathcal{F}$ is the support-point set; $\mathcal{K}=\{(t,f):h_{f}(t)-g<\epsilon\}$ is the contact set of (frame, support-point) pairs within $\epsilon=30$ mm of the ground at height $g$ ; $h_{f}$ and $p_{f}$ are the height and position of support point $f$ ; $u(t)$ is the unsupported-floating indicator; and $\Pi_{xy}(\cdot)$ is an operator that projects the input onto the ground plane. Formally, we have

		$\displaystyle\mathcal{J}_{\tau}=\mathbb{Q}_{\tau}\!\left\{\left\\|\nabla_{t}^{3}\bar{x}_{j}(t)\right\\|_{2}:j\in\mathcal{C}\right\},\,\,\mathcal{A}=\mathbb{Q}_{1/2}\!\left\{\left\\|\nabla_{t}^{2}x_{j}(t)\right\\|_{2}:j\in\mathcal{C}\right\},\,\,\phi_{\text{float}}=\mathbb{E}[u(t)],$		(1)
		$\displaystyle\delta_{\text{ground}}=\mathbb{Q}_{\tau}\!\left\{\max\!\bigl(g-h_{f}(t),0\bigr):f\in\mathcal{F}\right\},\quad\nu_{\text{foot}}=\mathbb{E}\!\left[\left\\|\Pi_{xy}\left(\nabla_{t}p_{f}(t)\right)\right\\|_{2}:(t,f)\in\mathcal{K}\right].$		(1)

Floor-related metrics ( $\delta_{\text{ground}}$ , $\phi_{\text{float}}$ , $\nu_{\text{foot}}$ ) require a comparable absolute ground convention. Motion-X++ does not provide one and is therefore excluded from these three metrics in Table 2(a).

Table 2: Data quality comparison. (a) Body-motion smoothness and ground-contact metrics, lower is better: jerk

\mathcal{J}_{\tau}

, acceleration

\mathcal{A}

, below-ground depth

\delta_{\mathrm{ground}}

, unsupported-floating share

\phi_{\mathrm{float}}

, and support-point drift

\nu_{\mathrm{foot}}

. (b) Human-object geometric consistency, higher is better: non-conflict fraction

\eta_{\mathrm{nc}}

and near-surface grounding within 20 cm

\rho_{\mathrm{near}}

(a) Body motion precision

Dataset	Duration (h)	$\bm{\mathcal{J}}_{\tau}$ (m/s³)	$\bm{\mathcal{A}}$ (m/s²)	$\bm{\delta}_{\mathrm{ground}}$ (mm)	$\bm{\phi}_{\mathrm{float}}$ (%)	$\bm{\nu}_{\mathrm{foot}}$ (mm/s)
HiPHI	617.5	173.9	10.7	8	0.015	64
AMASS	62.9	529.0	14.9	111	0.76	87
BONES-SEED	288.3	294.2	11.3	18	1.02	86
Motion-X++^†	26.0	709.0	23.8	–	–	–
LAFAN1	4.6	383.1	23.4	29	7.22	157

(b) Object interaction geometry

Dataset	Duration (h)	$\bm{\eta}_{\mathrm{nc}}$ (%)	$\bm{\rho}_{\mathrm{near}}$ (%)
HiPHI	245.7	98.1	95.7
HIMO^∗	21.6	97.6	79.0
OMOMO	9.8	90.6	50.4
HUMOTO^‡	0.81	99.9	82.9

All metrics are aggregated over the full evaluated duration. ^∗HIMO contains 9.4 h of unique multi-object sequences. Since each object in a multi-object sequence is evaluated separately, the reported duration becomes 21.6 object-track hours. ^†Motion-X++ floor metrics require a comparable absolute floor convention and are therefore omitted. ^‡HUMOTO is evaluated on its public subset; the full licensed dataset is not directly downloadable as public data.

Among datasets sharing the same floor-plane convention, HiPHI achieves the best value on every reported body-motion metric in Table 2(a). These metrics correspond to important physical failure modes in humanoid imitation: $\mathcal{J}_{\tau}$ measures abrupt motion changes, $\mathcal{A}$ captures excessive acceleration, $\delta_{\text{ground}}$ indicates ground penetration, $\phi_{\text{float}}$ reflects unsupported floating, and $\nu_{\text{foot}}$ measures foot sliding during contact.

Object-interaction consistency.

Two quantities (higher is better) measure whether human motion and object geometry remain physically consistent in a shared frame: non-conflict fraction $\eta_{\text{nc}}$ (%) and near-surface grounding $\rho_{\text{near}}$ (%). Let $\mathcal{S}(t)$ be a set of points sampled along the human-skeleton segment centerlines (uniformly along each segment), $\Omega(t)$ the posed object as a closed 3D region, and $d(\mathcal{S},\partial\Omega)$ the minimum point-to-surface distance from $\mathcal{S}$ to the object boundary $\partial\Omega$ . Below, $\mathbf{1}[\cdot]$ is the indicator function (1 if the condition holds, 0 otherwise):

\eta_{\text{nc}}=\mathbb{E}\!\bigl[\mathbf{1}[\,\mathcal{S}(t)\cap\Omega(t)=\emptyset\,]\bigr],\qquad\rho_{\text{near}}=\mathbb{E}\!\bigl[\mathbf{1}[\,d(\mathcal{S}(t),\partial\Omega(t))<0.20\,\text{m}\,]\bigr].

(2)

Table 2(b) reports $\eta_{\text{nc}}$ and $\rho_{\text{near}}$ . HiPHI provides substantially longer object-interaction duration than existing full-body HOI datasets (245.7 h vs. 21.6 h for HIMO and 9.8 h for OMOMO) while maintaining strong geometric consistency. In particular, HiPHI achieves 98.1% non-conflict rate and 95.7% near-surface grounding, substantially improving interaction grounding over HIMO (79.0%) and OMOMO (50.4%).

5.3 Humanoid Learning

We evaluate the physical executability of HiPHI through physics-based humanoid tracking. For whole-body motion, we compare HiPHI with AMASS [27], LAFAN1 [9], Motion-X++ [43], and BONES-SEED [3]. All motions are retargeted to the Unitree G1 robot and trained with the same DeepMimic imitation learning pipeline [31]. For motion-with-object tracking, we compare with HUMOTO [23] and OMOMO [18], using Omni-Retarget [42] and BeyondMimic-style [19] to handle synchronized human-object demonstrations. This unified setup allows performance differences to mainly reflect the usability of the motion data. All experiments are conducted from unmirrored original motions.

5.3.1 Whole-Body Motion Tracking

Whole-body tracking tests how well each motion source can be physically imitated without the additional complexity of object interaction. Due to the limited usable duration of LAFAN1, we construct a balanced 3-hour subset for each data source and additionally evaluate a 20-hour setting for larger datasets, to ensure a fair comparison independent of HiPHI’s data-scale advantage. For each setting, subsets are sampled using the same random sampling procedure, and all policies are trained under the same optimization budget, with additional details provided in Appendix G. A rollout is considered successful only if its mean full-body position error remains below 0.5 m at every simulation step; otherwise, it is marked as failed.

As shown in Figure 6, HiPHI achieves the highest success rates and fastest convergence in both the 3-hour and 20-hour settings. The advantage remains consistent across five independent training runs, with HiPHI maintaining lower failure rates and smaller variance than the compared datasets. These results show that HiPHI provides motion references that are not only larger in scale, but also more directly executable for humanoid policy learning.

Whole-body humanoid tracking performance across different motion data sources. (a) Success rates under matched 3-hour and 20-hour data budgets. (b) Training failure-rate curves ave... — Figure 6: Whole-body humanoid tracking performance across different motion data sources. (a) Success rates under matched 3-hour and 20-hour data budgets. (b) Training failure-rate curves averaged over five independent runs, with shaded regions indicating variance. HiPHI achieves faster convergence and lower final failure rates across different data scales.

5.3.2 Data Scaling

Beyond the matched-budget comparison, we further investigate how the scale of HiPHI affects downstream humanoid tracking. We progressively increase the amount of unmirrored HiPHI training data from 3 to 300 hours and evaluate the resulting policies on AMASS, BONES-SEED, Motion-X++, and LAFAN1. Consistent with the experiments above, we randomly sample 20 hours from each dataset as the evaluation set, except for LAFAN1, for which 3-hour setting is used. All experiments are repeated 10 times, and we report the mean curves with variance bands. As shown in Figure 7, increasing the amount of training data consistently reduces cross-dataset MPJPE, demonstrating that the large scale of HiPHI translates into continuous downstream performance gains.

Scaling behavior of HiPHI for humanoid tracking. Increasing the amount of unmirrored HiPHI training data from 3 to 300 hours consistently reduces cross-dataset MPJPE on AMASS, BONE... — Figure 7: Scaling behavior of HiPHI for humanoid tracking. Increasing the amount of unmirrored HiPHI training data from 3 to 300 hours consistently reduces cross-dataset MPJPE on AMASS, BONES-SEED, Motion-X++, and LAFAN1, demonstrating continuous performance gains with increasing data scale.

5.3.3 Whole-Body Motion Tracking with Object

We further evaluate human-object interaction tracking on four categories: kick, carry, push, and lean. All datasets are processed using the same retargeting and physics-based tracking pipeline, allowing us to evaluate how effectively each dataset supports physically grounded human-object imitation. Some categories are unavailable in existing datasets: HUMOTO does not contain push motions, and OMOMO does not provide lean motions. These cases are marked as “N/A” in Table 3.

Table 3: Whole-body motion tracking with object comparison (a) Body tracking errors are measured by MPJPE in mm and Vel. in mm/frame. (b) Object tracking errors are measured by Obj-Pos. in mm and Obj-Ori. in degrees.

(a) Body tracking errors Dataset Kick Carry Push Lean MPJPE $\downarrow$ Vel. $\downarrow$ MPJPE $\downarrow$ Vel. $\downarrow$ MPJPE $\downarrow$ Vel. $\downarrow$ MPJPE $\downarrow$ Vel. $\downarrow$ HUMOTO [23] 47.77 4.75 55.37 11.67 N/A N/A 38.83 4.67 OMOMO [18] 85.31 7.05 75.57 6.83 66.09 11.28 N/A N/A HiPHI (ours) 26.31 3.68 50.33 4.14 99.20 6.26 30.76 2.12

(b) Object tracking errors Dataset Kick Carry Push Lean Obj-Pos. $\downarrow$ Obj-Ori. $\downarrow$ Obj-Pos. $\downarrow$ Obj-Ori. $\downarrow$ Obj-Pos. $\downarrow$ Obj-Ori. $\downarrow$ Obj-Pos. $\downarrow$ Obj-Ori. $\downarrow$ HUMOTO [23] 214.91 35.92 59.20 5.39 N/A N/A 136.71 4.81 OMOMO [18] 128.62 104.28 62.46 4.87 257.82 46.24 N/A N/A HiPHI (ours) 68.17 3.27 89.13 2.64 60.51 2.23 69.16 9.63

We report both body-level and object-level tracking errors. Mean per-joint position error (MPJPE) and velocity error (Vel.) measure humanoid pose and motion tracking quality, while object position error (Obj-Pos.) and object orientation error (Obj-Ori.) measure object trajectory and rotation accuracy. As shown in Table 3, HiPHI achieves the best overall body tracking performance across most categories. For object tracking, HiPHI obtains the best position and orientation accuracy on kick and push, and the best orientation accuracy on carry, indicating better interaction consistency between the humanoid and manipulated objects. Additional experimental details are provided in Appendix H.

5.4 Real-World Evaluation of HiPHI

To evaluate the practical usability of HiPHI beyond simulation, we deploy policies trained with HiPHI on Unitree G1. Despite actuation limits, sensing noise, and sim-to-real discrepancies, the robot successfully performs diverse behaviors, including locomotion, sitting, crawling, carrying, flipping, and pulling motions. As shown in Figure 8, these results demonstrate that HiPHI supports stable and physically plausible whole-body and interaction-rich control on real hardware.

Real-world deployment of policies trained with HiPHI. The humanoid successfully performs diverse whole-body and object-interaction behaviors, including (a) running, (b) sitting, (c... — Figure 8: Real-world deployment of policies trained with HiPHI. The humanoid successfully performs diverse whole-body and object-interaction behaviors, including (a) running, (b) sitting, (c) crawling, (d) carrying a box, (e) flipping, and (f) pulling a suitcase, showing the physical executability of our motion data.

6 Conclusion

We introduced HiPHI, a 617.5-hour high-precision optical MoCap dataset and benchmark for humanoid learning, including 245.7 hours of synchronized human-object interaction with object trajectories and meshes. Built upon a Frame-LU-guided motion-space construction pipeline, HiPHI provides a systematic and scalable approach for collecting broad whole-body motion and physically grounded interactions.

Through comprehensive evaluations, HiPHI demonstrates broader motion-space coverage, higher motion quality, stronger human-object geometric consistency, and improved downstream humanoid tracking performance. Policies trained with HiPHI achieve strong matched-budget tracking performance, continue to improve with increasing data scale, and successfully transfer to real humanoid hardware across diverse whole-body and object-interaction behaviors.

Together, these results establish HiPHI as a large-scale data foundation for learning physically grounded humanoid skills, bridging high-precision motion capture, object-aware interaction, and scalable policy learning.

7 Limitations

HiPHI currently captures only single-person motion. Multi-person interaction and human-human contact remain outside its scope and require complementary efforts. In addition, although HiPHI provides high-precision full-body motion and object trajectories, it focuses on kinematics rather than directly measuring contact forces or tactile signals. Finally, the dataset is collected in studio, and capturing accurate motion and object interaction with ego-vision in-the-wild could be a promising future data collection direction.

Acknowledgments

We sincerely thank Roch Nakajima for generously sharing his extensive experience in motion capture and for his invaluable support in interpreting FrameNet lexical units and assessing their suitability for motion capture. We thank Alex Martinez for developing a practical data visualization tool that greatly facilitated data inspection and quality control. We also thank Baoze Du for his valuable advice on standardizing the BVH skeleton specification. Finally, we thank the many other staff members at Noitom Robotics who operated and maintained the motion-capture facilities and equipment, coordinated data-collection logistics and infrastructure, and supported the performers throughout the capture process. We are also deeply grateful to all performers who contributed their time and effort to the creation of HiPHI.

References

[1] C. F. Baker, C. J. Fillmore, and J. B. Lowe (1998) The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL ’98/COLING ’98, USA, pp. 86–90. DOI, DOI
[2] B. L. Bhatnagar, X. Xie, I. A. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll (2022) Behave: dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15935–15946.
[3] Bones Studio (2026) BONES-SEED: Skeletal Everyday Embodiment Dataset. Note: Hugging Face dataset Dataset
[4] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song (2024) Universal manipulation interface: in-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329.
[5] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255.
[6] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. (2022) Ego4d: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18995–19012.
[7] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng (2022) Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5152–5161.
[8] L. Han, Q. Zhu, J. Sheng, C. Zhang, T. Li, Y. Zhang, H. Zhang, Y. Liu, C. Zhou, R. Zhao, et al. (2024) Lifelike agility and play in quadrupedal robots using reinforcement learning and generative pre-trained models. Nature Machine Intelligence 6 (7), pp. 787–798.
[9] F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal (2020) Robust motion in-betweening. ACM Transactions on Graphics (TOG) 39 (4), pp. 60–1.
[10] R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang (2025) Egodex: learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709.
[11] Y. Huang, O. Taheri, M. J. Black, and D. Tzionas (2024) InterCap: joint markerless 3d tracking of humans and objects in interaction from multi-view rgb-d images. International Journal of Computer Vision 132 (7), pp. 2551–2566.
[12] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison (2020) Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters 5 (2), pp. 3019–3026.
[13] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu (2025) Egomimic: scaling imitation learning via egocentric video. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 13226–13233.
[14] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. (2024) Droid: a large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945.
[15] C. M. Kim, B. Yi, H. Choi, Y. Ma, K. Goldberg, and A. Kanazawa (2025) Pyroki: a modular toolkit for robot kinematic optimization. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1312–1319.
[16] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. (2023) Behavior-1k: a benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp. 80–93.
[17] G. Li, Y. Lyu, Z. Liu, C. Hou, J. Zhang, and S. Zhang (2025) H2r: a human-to-robot data augmentation for robot pre-training from videos. arXiv preprint arXiv:2505.11920.
[18] J. Li, J. Wu, and C. K. Liu (2023) Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42 (6), pp. 1–11.
[19] Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu (2025) Beyondmimic: from motion tracking to versatile humanoid control via guided diffusion. arXiv preprint arXiv:2508.08241.
[20] J. Lin, A. Zeng, S. Lu, Y. Cai, R. Zhang, H. Wang, and L. Zhang (2023) Motion-x: a large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems 36, pp. 25268–25280.
[21] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, pp. 44776–44791.
[22] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi (2022) Hoi4d: a 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21013–21022.
[23] J. Lu, C. P. Huang, U. Bhattacharya, Q. Huang, and Y. Zhou (2025) HUMOTO: a 4d dataset of mocap human object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10886–10897.
[24] Z. Luo, J. Cao, K. Kitani, W. Xu, et al. (2023) Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10895–10904.
[25] Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castaneda, Z. Cao, J. Li, D. Minor, Q. Ben, et al. (2025) Sonic: supersizing motion tracking for natural humanoid whole-body control. arXiv preprint arXiv:2511.07820.
[26] X. Lv, L. Xu, Y. Yan, X. Jin, C. Xu, S. Wu, Y. Liu, L. Li, M. Bi, W. Zeng, et al. (2024) Himo: a new benchmark for full-body human interacting with multiple objects. In European Conference on Computer Vision, pp. 300–318.
[27] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black (2019) AMASS: archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5442–5451.
[28] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard (2022) Calvin: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters 7 (3), pp. 7327–7334.
[29] G. A. Miller (1994) WordNet: a lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, Paper
[30] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903.
[31] X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne (2018) Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37 (4), pp. 1–14.
[32] X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler (2022) Ase: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG) 41 (4), pp. 1–17.
[33] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa (2021) Amp: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG) 40 (4), pp. 1–20.
[34] M. Plappert, C. Mandery, and T. Asfour (2016) The kit motion-language dataset. Big data 4 (4), pp. 236–252.
[35] A. R. Punnakkal, A. Chandrasekaran, N. Athanasiou, A. Quiros-Ramirez, and M. J. Black (2021) BABEL: bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 722–731.
[36] J. Ruppenhofer, M. Ellsworth, M. Schwarzer-Petruck, C. R. Johnson, and J. Scheffczyk (2016) FrameNet ii: extended theory and practice. Technical report International Computer Science Institute.
[37] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas (2020) GRAB: a dataset of whole-body human grasping of objects. In European conference on computer vision, pp. 581–600.
[38] C. Tessler*, Y. Jiang*, X. B. Peng, E. Coumans, Y. Shi, H. Zhang, D. Rempe, G. Chechik†, and S. Fidler† (2025) ProtoMotions3: an open-source framework for humanoid simulation and control. GitHub. Note: https://github.com/NVLabs/ProtoMotions/
[39] N. Wagener, A. Kolobov, F. Vieira Frujeri, R. Loynd, C. Cheng, and M. Hausknecht (2022) Mocapact: a multi-task dataset for simulated humanoid control. Advances in Neural Information Processing Systems 35, pp. 35418–35431.
[40] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, et al. (2023) Bridgedata v2: a dataset for robot learning at scale. In Conference on Robot Learning, pp. 1723–1736.
[41] S. Xu, D. Li, Y. Zhang, X. Xu, Q. Long, Z. Wang, Y. Lu, S. Dong, H. Jiang, A. Gupta, et al. (2025) Interact: advancing large-scale versatile 3d human-object interaction generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7048–7060.
[42] L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi (2025) Omniretarget: interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction. arXiv preprint arXiv:2509.26633.
[43] Y. Zhang, J. Lin, A. Zeng, G. Wu, S. Lu, Y. Fu, Y. Cai, R. Zhang, H. Wang, and L. Zhang (2025) Motion-x++: a large-scale multimodal 3d whole-body human motion dataset. arXiv preprint arXiv:2501.05098.
[44] L. Y. Zhu, P. Kuppili, R. Punamiya, P. Aphiwetsa, D. Patel, S. Kareer, S. Ha, and D. Xu (2026) Emma: scaling mobile manipulation via egocentric human data. IEEE Robotics and Automation Letters.
[45] Q. Zhu, H. Zhang, M. Lan, and L. Han (2023) Neural categorical priors for physics-based character control. ACM Transactions on Graphics (TOG).

Appendix

The appendix provides additional details that complement the main text. Appendix A-B describe FrameNet-guided motion-unit construction and the release composition, metadata, and demographics. Appendix C states the release license and ethics. Appendix D shows representative sample sequences. Appendix E-F provide the unsupervised motion-space embedding setup and the full-run quality metrics. Appendix G-H document training details for whole-body and motion-with-object tracking.

Appendix A Additional Details for Dataset Construction

FrameNet-guided motion units.

HiPHI uses FrameNet as a semantic scaffold rather than a closed action taxonomy. We identify motion-relevant frames and lexical units (LUs), and use each Frame-LU pair as a seed for a motion unit that can be translated into performer-facing capture instructions. This keeps the public index compact while allowing each semantic seed to expand into multiple concrete realizations.

Table 4: Examples of FrameNet-guided motion-unit construction. Frame-LU seeds are expanded by controlled realization factors rather than treated as single fixed actions.

Motion family	Frame-LU examples	Realization factors
Locomotion and direction	Self_motion: walk, run, jog, stride	path, direction, speed, turning radius, step rhythm
Posture and support	Change_posture, Posture: crawl, kneel, lean	body height, support relation, amplitude, transition speed
Body-part motion	Body_movement: bend, toss, shake, clap	body part, range of motion, rhythm, unilateral/bilateral pattern
Object actuation and transfer	Cause_motion, Bringing: push, pull, lift, carry	object category, contact mode, load condition, object trajectory
Object-local dynamics	Cause_to_move_in_place: rotate, shake, swing	axis of motion, contact point, amplitude, repetition pattern

Performer-facing realization protocol.

Each selected Frame-LU seed is converted into short, repeatable capture instructions by varying factors that are directly observable during MoCap: direction, path, speed, intensity, amplitude, body height, body-part involvement, object category, contact mode, and load condition. For instance, a push seed is realized with different object categories, loads, and directions; a lean seed varies support surface and body height; and a crawl seed varies path shape, speed, and limb coordination. This protocol turns a compact semantic index into a broader set of controllable whole-body motion realizations.

Appendix B Additional Details for Dataset Composition and Metadata

Release duration convention.

All release-scale durations in the paper follow the public mirrored-release convention, where left-right mirrored sequences are included as normal released data. Under this convention, HiPHI contains 617.5 hours of motion, corresponding to approximately 200.1M frames at 90 Hz. The release contains 371.8 hours of body-only motion and 245.7 hours of strict object-interaction motion, where strict object-interaction denotes explicit physical interaction with a tracked object.

Table 5: Release-level composition used throughout the paper. Durations follow the mirrored-release convention.

Statistic	Value
Total release duration	617.5 h
Total frames	200.1M
Capture frame rate	90 Hz
Total performers	132
Body-only duration	371.8 h
Human object-interaction duration	245.7 h
Human object-interaction share	39.8%
FrameNet frames	22
Frame-LU labels	214
Median actors per Frame-LU	24
Frame-LUs with at least 10 actors	154
Object categories in strict object-interaction subset	12

Frame-LU composition.

The released semantic layer contains 22 FrameNet frames and 214 Frame-LU motion-unit labels. Its duration distribution follows a common-to-long-tail structure: the top 10 Frame-LUs account for 20.4% of duration, the top 20 for 31.8%, and the top 50 for 53.7%. Thus, nearly half of the release duration remains outside the top 50 Frame-LUs, supporting retrieval and analysis beyond a small set of frequent actions.

Performer demographics.

HiPHI contains 132 anonymized performer profiles, including 76 male and 56 female profiles, with heights from 155 to 185 cm and weights from 40 to 82 kg.Figure 9 summarizes the distribution.

Table 6: Metadata fields exposed by the HiPHI release. Fields are provided through the global motion manifest, performer metadata, and per-package metadata.json files.

Field	Scope	Meaning
Global motion metadata
motion_id	Global / package	Unique motion identifier. Mirrored motions use the corresponding ID with a __mirror suffix.
frame	Global / package	FrameNet frame label.
lu	Global / package	FrameNet lexical-unit label.
frame_lu	Global / package	Frame-LU motion-unit label.
duration_sec	Global / package	Temporal duration of the motion, in seconds.
frame_count	Global / package	Number of motion frames in the sequence.
actor_id	Global / package	Anonymized performer identifier in the release.
text_annotation	Global / package	Natural-language annotation aligned with the motion sequence.
is_hoi	Global / package	Whether the sequence contains tracked human-object interaction.
object_categories	Global	Object-category information associated with the sequence, when applicable.
mirrored	Global / package	Boolean indicating whether the motion is the left-right mirrored counterpart of an original sequence.
Performer metadata
actor_id	Actor metadata	Anonymized performer identifier used to link performer metadata to motion sequences.
height_cm	Actor metadata	Performer height in centimeters.
weight_kg	Actor metadata	Performer weight in kilograms.
gender	Actor metadata	Performer gender metadata.
Additional per-package metadata
dataset	Package only	Dataset identifier stored in the package metadata.
fps	Package only	Frame rate associated with the released motion package.
actor_metadata	Package only	Nested performer metadata associated with the sequence.
objects	Package only	List of tracked objects associated with the sequence; empty for body-only motion.
objects[].object_id	Package only	Identifier of a tracked object instance.
objects[].object_category	Package only	Semantic category of the tracked object.
objects[].trajectory_path	Package only	Package-relative path to the synchronized object trajectory.
objects[].mesh_id	Package only	Identifier of the corresponding object mesh.
objects[].mesh_path	Package only	Package-relative path to the corresponding object mesh.

Object-interaction metadata.

The HiPHI object-interaction subset contains 40 real-world objects from 12 categories, covering a mass range of 0.45-6.25 kg and spanning 15 FrameNet frames and 90 Frame-LU labels (Table 7). For strict object-interaction sequences, the release provides synchronized object trajectories and object meshes together with the human BVH motion.

Table 7: Object statistics of the HiPHI human-object interaction subset. The subset contains diverse real-world objects with synchronized human motion, object trajectories, and object meshes.

Statistic	Value
Object instances	40
Object categories	12
Mass range	0.45-6.25 kg
Covered FrameNet frames	15
Covered Frame-LU labels	90
Interaction duration	245.7 h
Object state modalities	Trajectory + Mesh

Appendix C Release Terms, License, and Ethics

License.

HiPHI will be released under the ModalityNet Open Research License v1.0, a custom non-commercial license for scientific research, education, and evaluation. The complete license terms will accompany the public release of HiPHI.

Release schedule.

The full dataset, including BVH motions, object trajectories, object meshes, and the Frame-LU index, will be released publicly soon. The supplementary material contains four representative sample sequences (three body-only and three object-interaction) that illustrate the release format and quality.

Ethics.

Data collection was reviewed and approved by an institutional ethics review board before capture. All performers were adults who provided written informed consent describing the capture protocol, the intended research use, and the planned public release of the motion data and anonymized demographic attributes. Performers were compensated for each capture session and may request withdrawal of their data at any time. During capture, RGB video and audio were recorded as references for production review; the public release of HiPHI contains only BVH motion, object trajectories, and object meshes, and does not include any RGB video, audio, facial imagery, or voice recordings. Performer identities are released only as numeric actor identifiers, with anonymized height, weight, and gender attributes. Specific institutional and protocol identifiers are withheld for double-blind review and will be provided in the camera-ready version.

Appendix D Representative Sample Sequences

We show six representative sequences from HiPHI. Per-sequence renderings are shown in Figure 10, and full motion playback is provided in the supplementary video.

Figure 10: Representative sample sequences from HiPHI. We show six representative sequences from HiPHI: (a)-(c) body-only motion sequences and (d)-(f) object-interaction motion sequences.

Appendix E Additional Details for Motion-Space Diversity

Unified motion representation.

For the motion-space analysis in Sec. 5.1, all datasets are first mapped to a common body representation with 23 keypoints in meters, resampled to 30 FPS in a $z$ -up coordinate frame. The keypoints are pelvis, spine, chest, neck, head, left/right hip, knee, ankle, foot, toe, shoulder, elbow, wrist, and hand. Each sequence is divided into 30-frame windows with a stride of 25 frames. The per-frame input contains heading-local, root-relative joint positions and root-relative joint velocities. We further apply body-scale normalization and subtract the per-window median pose, which reduces scale and skeleton-layout differences across datasets. The resulting feature dimension is $23\times 3\times 2=138$ per frame.

Dataset-balanced sampling.

To prevent dataset size from dominating either representation learning or coverage evaluation, we construct a balanced sample for each dataset. We randomly sample 5,000 motion clips from each dataset when available and draw one valid 30-frame window from each selected clip, yielding 5,000 training windows per dataset. LAFAN1 uses all available clips because it contains fewer than 5,000 samples. The t-SNE projection and grid-based occupancy analysis follow the same cap of at most 5,000 windows per dataset. Therefore, both the shared representation and the reported coverage statistics are dataset-balanced rather than proportional to raw dataset size.

Shared unsupervised encoder.

We train one temporal convolutional autoencoder jointly on the balanced samples from all datasets and use its encoder as the motion descriptor. The model takes a tensor of size $30\times 138$ , uses temporal Conv1D layers with residual dilated blocks, and outputs a 16-D latent code. It is trained only with reconstruction loss; dataset identity, text labels, Frame-LU labels, and object labels are not used. The default analysis uses the epoch-20 checkpoint. Before t-SNE, latent features are standardized and reduced to 5 dimensions with PCA. We then run t-SNE with Euclidean distance, perplexity 30, PCA initialization, automatic learning rate, 1000 iterations, and random seed 42.

Table 8: Embedding setup for the motion-space coverage analysis.

Item	Setting
Keypoints	23 body keypoints, meter scale, $z$ -up
Frame rate	30 FPS
Window / stride	30 frames / 25 frames
Feature type	Heading-local normalized joint dynamics and relative velocities
Feature dimension	138 per frame
Balanced sampling	Up to 5,000 clips and 5,000 windows per dataset
Encoder	Shared unsupervised temporal Conv1D autoencoder
Latent dimension	16
Objective	Reconstruction MSE
Checkpoint	Epoch 20
t-SNE	PCA dim. 5, perplexity 30, PCA init., auto learning rate, 1000 iterations, seed 42
Grid statistics	$55\times 55$ grid on the default 2-D embedding

Grid-based coverage statistics.

Let $\mathcal{G}$ be the $55\times 55$ grid in the shared t-SNE plane and $n_{D}(c)$ the number of sampled points from dataset $D$ in cell $c\in\mathcal{G}$ . Let $\mathcal{G}_{D}^{+}=\{c\in\mathcal{G}:n_{D}(c)>0\}$ be the cells occupied by $D$ . We report occupied grid cells, entropy-based effective grid cells, and long-tail share:

	$\displaystyle m_{D}$	$\displaystyle=\lvert\mathcal{G}_{D}^{+}\rvert,\qquad p_{D}(c)=\frac{n_{D}(c)}{\sum_{c^{\prime}\in\mathcal{G}}n_{D}(c^{\prime})},$		(3)
	$\displaystyle e_{D}$	$\displaystyle=\exp\!\Bigl(-\sum_{c\in\mathcal{G}_{D}^{+}}p_{D}(c)\log p_{D}(c)\Bigr),\qquad\ell_{D}=\sum_{c\in\mathcal{T}}p_{D}(c).$		(3)

Here, $m_{D}$ is the number of occupied cells, measuring how broadly a dataset spans the shared motion space. The effective occupancy $e_{D}$ measures how evenly its samples are distributed across the occupied cells. The long-tail share $\ell_{D}$ measures the fraction of samples located in globally rare regions. We use the convention $0\log 0=0$ . The set $\mathcal{T}\subseteq\mathcal{G}$ contains non-empty cells whose global occupancy is at or below the 25th percentile of all non-empty global cells; under the default $55\times 55$ setting, this threshold corresponds to 7 points per cell.

Coverage robustness analysis.

We further test whether the occupied-region advantage depends on a particular random seed, t-SNE perplexity, or grid resolution. Starting from the default configuration in Table 8, we repeat the analysis using random seeds 55, 73, and 112; perplexities 40, 55, and 65; and grid resolutions of $55\times 55$ , $88\times 88$ , and $111\times 111$ .

For every configuration, we report the occupied-cell margin

\Delta\mathrm{Occ}=m_{\mathrm{HiPHI}}-\max_{D\neq\mathrm{HiPHI}}m_{D},

(4)

where the second term is the occupied-cell count of the strongest baseline under the same configuration. A positive value means that HiPHI covers more cells than every compared dataset. Because occupied-cell counts naturally change with grid resolution, the margin is reported separately for each resolution.

As shown in Figure 11, HiPHI maintains a positive occupied-cell margin in every tested configuration. The coverage advantage therefore remains consistent across resampling, projection, and discretization choices.

Robustness of motion-space coverage. We repeat the coverage analysis using different random seeds, t-SNE perplexities, and grid resolutions. Each entry reports the occupied-cell ma... — Figure 11: Robustness of motion-space coverage. We repeat the coverage analysis using different random seeds, t-SNE perplexities, and grid resolutions. Each entry reports the occupied-cell margin between HiPHI and the strongest baseline under the corresponding setting. HiPHI maintains a positive margin across all tested configurations.

Appendix F Additional Details for Data Quality and Precision

Aggregation across sequences.

All quality statistics in Sec. 5.2 are aggregated over the full evaluated duration of each dataset, so that every recorded second contributes equally and dataset size does not bias the result. Let $r_{k}$ index any observation at the relevant level (frame, window, or sequence aggregate) with non-negative duration weight $w_{k}$ (i.e., $w_{k}=\Delta t$ of the source for per-frame observations, the window length for per-window observations, and the sequence duration $T_{i}$ for per-sequence aggregates). We report

\mathbb{E}[r]=\frac{\sum_{k}w_{k}\,r_{k}}{\sum_{k}w_{k}},\qquad\mathbb{Q}_{\tau}(r)=\inf\!\Bigl\{a:\frac{\sum_{k}w_{k}\,\mathbf{1}[r_{k}\leq a]}{\sum_{k}w_{k}}\geq\tau\Bigr\},

(5)

i.e., the time-weighted mean and quantile, with $\tau=0.95$ for upper-tail statistics. For ratio-type metrics ( $\eta_{\text{nc}}$ , $\rho_{\text{near}}$ , $\phi_{\text{float}}$ ), each $r_{i}$ is itself a frame-level ratio inside sequence $i$ , and weighting by $T_{i}$ recovers the frame-level aggregation across the dataset. For distribution-type metrics ( $\mathcal{J}_{\tau}$ , $\mathcal{A}$ , $\delta_{\text{ground}}$ , $\nu_{\text{foot}}$ ), $r_{k}$ is the per-frame (or per-pair) observation entering the corresponding equation in Sec. 5.2.

Body-motion metric sets and smoothing.

Equation 1 uses one joint set, one point set, and one pair set. $\mathcal{C}$ is the core-body joint set, excluding fingers and face joints when present. $\mathcal{F}$ is the support-point set: foot, toe, and ankle proxies when available. $\mathcal{K}=\{(t,f):h_{f}(t)-g<\epsilon\}$ is the contact set of (frame, support-point) pairs within $\epsilon=30$ mm of the ground at height $g$ ; speed is then used to measure drift over those pairs. The finite-difference operator $\nabla_{t}$ is divided by the source-specific frame interval $\Delta t$ , so $\nabla_{t}^{k}x$ has units of m/s^k. For jerk, trajectories are smoothed by a fixed physical window of $1/6$ s, implemented as an odd-length moving average with a minimum size of three frames; raw positions are used for acceleration. Floor-related metrics require a comparable absolute ground convention, which is why Motion-X++ is excluded from $\delta_{\text{ground}}$ , $\phi_{\text{float}}$ , and $\nu_{\text{foot}}$ in Table 2.

Object-interaction geometry.

Equation 2 uses a sparse skeleton-point proxy. The human body is represented by canonical skeleton segments covering hands, arms, legs, torso, pelvis, and head/neck regions. For each evaluated frame, segment centerlines are uniformly sampled (32 points per segment in the released code) to produce $\mathcal{S}(t)$ . The object is represented as a closed 3D region $\Omega(t)$ obtained by posing the released mesh; $\partial\Omega(t)$ is its boundary surface. Distances $d(\mathcal{S},\partial\Omega)$ are computed in the shared coordinate frame. HUMOTO is evaluated on its publicly released GLB subset; the full licensed dataset is not directly downloadable as public data.

Table 9: Summary of quality metrics used in Sec. 5.2.

Symbol	Meaning	Unit	Better
$\mathcal{J}_{\tau}$	Upper-tail fixed-window jerk over core-body joints	m/s³	lower
$\mathcal{A}$	Median core-body acceleration	m/s²	lower
$\delta_{\text{ground}}$	Upper-tail below-ground support-point depth	mm	lower
$\phi_{\text{float}}$	Unsupported-floating duration share	%	lower
$\nu_{\text{foot}}$	Support-point drift over contact frames	mm/s	lower
$\eta_{\text{nc}}$	Non-conflict fraction under skeleton-point / object-volume proxy	%	higher
$\rho_{\text{near}}$	Near-surface fraction within 20 cm of the object surface	%	higher

Applicability.

The body-motion smoothness metrics require only temporally aligned human joint trajectories and native timing. Ground-contact metrics additionally require a comparable up-axis and absolute ground convention. Object-interaction metrics require synchronized human motion, object pose, and object mesh in a shared coordinate frame. HUMOTO is evaluated on its publicly released GLB subset; the full licensed dataset is not directly downloadable as public data.

Appendix G Training Details for Whole-Body Motion Tracking

For whole-body motion tracking, we evaluate all datasets under the same physics-based imitation learning pipeline. All source motions are first retargeted to the Unitree G1 humanoid using PyRoki [15], and the resulting robot motions are then used as reference trajectories for policy learning. We use the same observation design, reward formulation, optimization budget, and evaluation protocol for all data sources so that performance differences mainly reflect the physical usability of the underlying motion data. Each tracking experiment is repeated five times with different random seeds. We report the mean curve and variance across runs. The detailed observation and reward are summarized in Tables 10 and 11.

Figure 12 shows qualitative whole-body tracking results of HiPHI on the Unitree G1 humanoid. The learned policy produces stable and physically plausible motions across diverse behaviors, demonstrating that the motions in HiPHI can serve as effective executable references for humanoid control.

Table 10: Observation terms for whole-body motion tracking.

State	Dim.
(a) Motion Command
Reference Body Position (heading frame)	$99$
Reference Body Position (relative)	$99$
Reference Body Orientation (heading frame)	$198$
Reference Body Orientation (relative)	$198$
Reference Body Linear Velocity (relative)	$99$
Reference Body Angular Velocity (relative)	$99$
(b) Proprioceptive State
Root Height (above terrain)	$1$
Body Position (non-root, heading frame)	$96$
Body Orientation (heading frame, tan-norm)	$198$
Body Linear Velocity (heading frame)	$99$
Body Angular Velocity (heading frame)	$99$
Last Action	$29$

Qualitative visualization of diverse humanoid motion skills trained with HiPHI. The motions cover diverse whole-body behaviors, including standing, walking, sitting, lying down, cr... — Figure 12: Qualitative visualization of diverse humanoid motion skills trained with HiPHI. The motions cover diverse whole-body behaviors, including standing, walking, sitting, lying down, crawling, kneeling, and recovery motions, demonstrating the coverage and diversity of the training data.

Table 11: Reward terms used for whole-body motion tracking.

Term	Expression	Weight	Remarks
Tracking Rewards
Global Translation (gt)	$\exp\!\left(c_{\mathrm{gt}}\,\frac{1}{n_{b}}\sum_{i}\\|p_{i}-\hat{p}_{i}\\|_{2}^{2}\right)$	$0.5$	$c_{\mathrm{gt}}=-25$
Global Rotation (gr)	$\exp\!\left(c_{\mathrm{gr}}\,\frac{1}{n_{b}}\sum_{i}d_{R}(q_{i},\hat{q}_{i})^{2}\right)$	$0.3$	$c_{\mathrm{gr}}=-5$
Global Linear Velocity (gv)	$\exp\!\left(c_{\mathrm{gv}}\,\frac{1}{n_{b}}\sum_{i}\\|v_{i}-\hat{v}_{i}\\|_{2}^{2}\right)$	$0.1$	$c_{\mathrm{gv}}=-0.5$
Global Angular Velocity (gav)	$\exp\!\left(c_{\mathrm{gav}}\,\frac{1}{n_{b}}\sum_{i}\\|\omega_{i}-\hat{\omega}_{i}\\|_{2}^{2}\right)$	$0.2$	$c_{\mathrm{gav}}=-0.1$
Root Height (rh)	$\exp\!\left(c_{\mathrm{rh}}\,(z_{\mathrm{root}}-\hat{z}_{\mathrm{root}})^{2}\right)$	$0.2$	$c_{\mathrm{rh}}=-100$
Regularization and Safety Penalties
Action Smoothness	$\\|\tilde{a}_{t}-\tilde{a}_{t-1}\\|_{2}$	$-0.02$	–
Power Consumption	$\sum_{j}\|\tau_{j}\,\dot{q}_{j}\|$	$-10^{-5}$	–

Here $d_{R}(\cdot,\cdot)$ denotes quaternion orientation error, $n_{b}$ is the number of tracked body links.

Appendix H Training Details for Motion-with-Object Tracking

For motion-with-object tracking, we first process synchronized human-object demonstrations using Omni-Retarget [42], which retargets human motion to the Unitree G1 humanoid while preserving the corresponding object trajectories. Because existing open-source human-object motion datasets usually do not specify object physical parameters, we use a unified object configuration across datasets for fair comparison. Unless otherwise specified, each object is modeled as a free-joint mesh object with a total mass of $2.0$ kg.

We then use a BeyondMimic-style tracking setup [19], following a similar observation design and reward formulation while extending the framework to explicitly model object interactions. Since the task requires the humanoid to reproduce both whole-body motion and the associated object trajectory, we additionally include the current object state in the policy observation. We also introduce object-tracking rewards by measuring the position and orientation errors between the simulated object and its reference trajectory. To improve policy robustness, we apply domain randomization to both the humanoid and the object, including object mass, friction, inertial properties, contact parameters, and external perturbations. The detailed observation, reward, and randomization settings are summarized in Tables 12, 13, and 14.

Table 12: Observation terms for motion-with-object tracking.

(a) Motion Command
State	Dim.	Policy	Critic
Reference Joint Pos	$29$	✓	✓
Reference Joint Vel	$29$	✓	✓
Reference Anchor Position	$3$	✓	✓
Reference Anchor Orientation	$6$	✓	✓
(b) Proprioceptive State
Base Linear Velocity	$3$	✗	✓
Base Angular Velocity	$3$	✓	✓
Joint Position	$29$	✓	✓
Joint Velocity	$29$	✓	✓
Last Action	$29$	✓	✓
(c) Object State
Object position in robot frame	$3$	✓	✓
Object orientation in robot frame	$6$	✓	✓
(d) Critic-Only Privileged State
Body position	$14\times 3$	✗	✓
Body orientation	$14\times 6$	✗	✓

Table 13: Reward terms used for motion-with-object tracking.

Term	Expression	Weight	Remarks
Tracking Rewards
Anchor position	$\exp\!\left(-\\|p_{a}-\hat{p}_{a}\\|_{2}^{2}/\sigma^{2}\right)$	$0.5$	$\sigma=0.3$
Anchor orientation	$\exp\!\left(-d_{R}(q_{a},\hat{q}_{a})^{2}/\sigma^{2}\right)$	$0.5$	$\sigma=0.4$
Body position	$\exp\!\left(-\frac{1}{n_{b}}\sum_{i}\\|p_{i}-\hat{p}_{i}\\|_{2}^{2}/\sigma^{2}\right)$	$1.0$	$\sigma=0.3$
Body orientation	$\exp\!\left(-\frac{1}{n_{b}}\sum_{i}d_{R}(q_{i},\hat{q}_{i})^{2}/\sigma^{2}\right)$	$1.0$	$\sigma=0.3$
Body linear velocity	$\exp\!\left(-\frac{1}{n_{b}}\sum_{i}\\|v_{i}-\hat{v}_{i}\\|_{2}^{2}/\sigma^{2}\right)$	$0.5$	$\sigma=1.0$
Body angular velocity	$\exp\!\left(-\frac{1}{n_{b}}\sum_{i}\\|\omega_{i}-\hat{\omega}_{i}\\|_{2}^{2}/\sigma^{2}\right)$	$0.5$	$\sigma=3.14$
Object position	$\exp\!\left(-\\|p_{o}-\hat{p}_{o}\\|_{2}^{2}/\sigma^{2}\right)$	$1.0$	$\sigma=0.3$
Object orientation	$\exp\!\left(-d_{R}(q_{o},\hat{q}_{o})^{2}/\sigma^{2}\right)$	$1.0$	$\sigma=0.3$
Regularization and Safety Penalties
Action rate	$\\|a_{t}-a_{t-1}\\|_{2}^{2}$	$-0.1$	Smooth action penalty
Joint limit	$\mathbf{1}(q\notin[q_{\min},q_{\max}])$	$-10.0$	Joint limit violation

Here $d_{R}(\cdot,\cdot)$ denotes quaternion orientation error, $n_{b}$ is the number of tracked body links, $c$ and $\hat{c}$ denote measured and reference contact states, and hatted variables denote reference motion states.

Table 14: Domain randomization and disturbance settings for motion-with-object tracking.

External Disturbances
Term	Value
Push robot	interval $=1$ – $3$ s, $v_{x},v_{y}\sim U[-0.5,0.5]$ m/s, $v_{z}\sim U[-0.2,0.2]$ m/s
Robot Dynamics Randomization
Torso COM offset	$x\sim U[-0.025,0.025]$ m, $y,z\sim U[-0.05,0.05]$ m
Encoder bias	$U[-0.01,0.01]$
Default joint position offset	$U[-0.01,0.01]$ rad
Rigid-body material	static friction $U[0.3,1.6]$ , dynamic friction $U[0.3,1.2]$ , restitution $U[0.0,0.5]$
Object Dynamics Randomization
Object mass	$U[0.3,2.0]\times$ default mass
Object COM offset	$U[-0.02,0.02]$ m

Figure 13 presents qualitative comparisons of motion-with-object tracking across three datasets: HiPHI, OMOMO, and HUMOTO. We visualize two representative interaction categories, including carry (first row) and kick interactions (bottom row). Existing datasets exhibit limited interaction coverage and tracking robustness in certain categories. For example, HUMOTO fails to reproduce stable object-aware behaviors in the carry scenario, while OMOMO suffers from reduced interaction fidelity. In contrast, HiPHI produces more stable whole-body motions and more consistent human-object interaction behaviors across diverse tasks.

Qualitative comparisons of motion-with-object tracking across different datasets. The top row shows carry interactions, while the bottom row presents kick interaction behaviors. Hi... — Figure 13: Qualitative comparisons of motion-with-object tracking across different datasets. The top row shows carry interactions, while the bottom row presents kick interaction behaviors. HiPHI demonstrates more stable whole-body tracking and better human-object interaction consistency across diverse tasks.