HiPHI Humanoid Motion Dataset Challenges the Video-First Path to Robot Learning

HiPHI has released a 617.5-hour humanoid motion dataset that challenges a central assumption in robot learning: more video does not automatically mean better physical training data. The release combines precise whole-body movement with tracked objects, semantic labels, and tests on a physical humanoid. Its bet is that robots need measured physical states, not just vast collections of visual observations.

That argument puts the HiPHI humanoid motion dataset between two established approaches. Internet video offers exceptional behavioral variety but lacks reliable joint, contact, and object states. Laboratory motion capture provides those states, yet many existing collections cover fewer behaviors or omit synchronized objects.

HiPHI tries to close that gap through structured collection rather than raw volume alone. Its researchers used FrameNet, a linguistic system for organizing event meanings, to define motion families and generate variations. They then trained reinforcement learning policies with the data and transferred selected behaviors to a Unitree G1 robot.

The HiPHI Humanoid Motion Dataset Expands the Training Record

HiPHI changes the available training base by combining scale, physical precision, and object-aware motion in one public research release.

The project was submitted to arXiv on August 17, 2026, and revised on September 8. The paper is accepted at the Conference on Robot Learning 2026, according to its research record.

The released dataset contains 617.5 hours of motion and about 200.1 million frames. Researchers captured the source material at 90 hertz from 132 performers using an optical motion-capture system. The team reports sub-millimeter marker-tracking accuracy.

However, the headline duration needs context. The release applies left-right mirroring to 308.7 hours of original captures, producing the reported 617.5 hours. Mirroring swaps appropriate left and right motion elements to create a valid counterpart, so it is useful augmentation rather than a second independent capture.

The total divides into 371.8 hours of body-only movement and 245.7 hours of human-object interaction. That interaction portion represents 39.8 percent of the release. It includes motions such as carrying, pushing, pulling, leaning, and sitting with tracked physical objects.

The human motion uses a standardized 55-joint BVH hierarchy. BVH is a common file format that stores a skeletal structure and its motion over time. Every interaction package also connects the body record to synchronized object trajectories and high-resolution object meshes.

The public dataset documentation identifies 40 canonical object meshes and their mirrored variants. Object tracks share timestamps with their corresponding body files, reducing the alignment work required before experimentation.

The data covers 40 real objects across 12 categories. Those objects range from furniture and containers to cleaning tools and sports equipment. Their reported masses run from 0.45 to 6.25 kilograms.

That mass range matters because carrying a light container and moving a heavy object require different balance strategies. Even when the visible arm path looks similar, the feet, torso, and center of mass can behave differently.

HiPHI also attaches natural-language descriptions and semantic identifiers to its clips. The resulting package supports motion retrieval, imitation learning, retargeting, motion generation, and object-aware control research.

This is more than a larger animation archive. The release is designed around a specific question: can a motion collection remain diverse while preserving enough physical structure for robot policies to execute it?

Why Internet Video Leaves a Physical Data Gap

Video captures what an action looks like, but humanoid control depends on physical states that pixels often reveal only indirectly.

Large video collections have an obvious appeal. They contain people performing countless tasks across homes, workplaces, streets, and unfamiliar environments. That diversity is difficult to reproduce inside a motion-capture studio.

Yet video normally records appearance rather than the complete physical state. A learning system must estimate depth, joint positions, occluded limbs, contacts, and object motion. Errors in those estimates can accumulate before any robot policy begins training.

Consider someone pulling a loaded suitcase. A camera records the person and suitcase, but it does not directly provide force, friction, exact contact geometry, or a clean skeletal trajectory. Clothing can hide joints, while perspective makes depth uncertain.

Motion capture addresses part of that problem by measuring body movement precisely. Traditional collections such as AMASS have become important resources for animation and humanoid control. However, many were assembled for motion synthesis, reconstruction, or graphics rather than object-constrained robot learning.

The missing object can make an otherwise realistic clip physically incomplete. A person appears to sit, but no chair state accompanies the body. A performer leans forward, but the support surface and contact relationship are absent.

A policy trained from that body trajectory can imitate the pose without understanding the constraint that made the movement stable. This is the difference between visual plausibility and executable control.

Real-robot demonstrations provide more direct supervision. They already reflect a machine’s joints, sensors, and limits. Their weakness is collection cost, and the resulting data often remains tied to one robot configuration.

HiPHI therefore occupies a middle position. Human performers supply behavioral range, while optical capture supplies accurate motion states. Synchronized object records preserve part of the interaction structure that body-only datasets discard.

The comparison is not video versus motion capture in every situation. Video remains valuable for semantic context, visual perception, and behavior in natural environments. Motion capture remains constrained by studios, markers, and planned sessions.

The real dispute concerns which data should serve as an executable motion prior. A motion prior is a learned model of how bodies tend to move. For humanoid control, its references must survive retargeting from a human body to a robot with different proportions and actuators.

HiPHI’s researchers argue that precision and grounding deserve greater weight at this stage. Their benchmark compares the data after converting multiple sources into a shared body representation and applying consistent evaluation procedures.

That approach pressures teams relying mainly on reconstructed internet video. Larger visual datasets can describe more situations, but their inferred physical states must become accurate enough to compete with measured trajectories.

It also pressures conventional motion-capture projects. High precision alone is insufficient when performers repeat a narrow set of familiar actions. The harder target is precise coverage across a deliberately designed motion space.

FrameNet Turns Language Into a Motion Collection Plan

HiPHI’s central mechanism is not its camera system but its use of language structure to decide which movements deserve capture.

Most motion datasets begin with lists of activities. Designers might request walking, running, sitting, waving, and lifting, then add scripts as new needs emerge. That process produces understandable clips, but it offers no dependable measure of what remains missing.

Different stories can generate nearly identical movement. Wiping sweat from a forehead and shielding the eyes from sunlight may use similar joint trajectories. Counting both as separate activities can exaggerate physical coverage.

The reverse problem also occurs. One word such as “walk” can generate many motions through speed, route, stride length, turning radius, posture, and direction. A single activity label can conceal significant kinematic variety.

HiPHI uses Berkeley FrameNet as a scaffold for handling this mismatch. FrameNet organizes language around events, situations, and the word senses that evoke them. A “frame” represents an event type, while a lexical unit connects a word’s specific meaning to that frame.

For example, “run” can describe human locomotion or managing a company. A Frame-LU pair separates those meanings. HiPHI selects pairs connected to physical movement and turns them into seeds for capture instructions.

The collection contains 22 FrameNet frames and 214 Frame-LU motion units. These units cover locomotion, posture changes, body-part movement, object actuation, transfer, and related interaction patterns.

Each semantic seed expands along observable physical dimensions. Performers vary direction, speed, amplitude, posture, rhythm, body-part involvement, contact position, load, and object trajectory.

A pushing seed can produce multiple objects, loads, contact points, and directions. A walking seed can vary route, pace, turning behavior, stride, and body height. These changes alter motion rather than merely changing its description.

This resembles the role WordNet played in structuring ImageNet. The taxonomy does not generate the data itself. It provides an organized map for deciding what to collect and where coverage remains thin.

The resulting distribution includes common behavior and a deliberate long tail. The 50 most frequent Frame-LU labels account for 53.7 percent of released duration. The remaining 46.3 percent sits outside those common units.

Performer variation also appears within the labels. HiPHI reports a median of 24 performers per Frame-LU, while 154 units include at least 10 performers. That reduces the chance that a motion category simply reflects one person’s style.

The HiPHI project benchmark measures coverage with a shared unsupervised motion encoder. Researchers normalized datasets into a common 23-keypoint representation, resampled them to 30 frames per second, and compared balanced samples.

Under that procedure, HiPHI occupied 1,620 cells in the projected motion space. BONES-SEED, its closest large-scale baseline, occupied 1,438. HiPHI also reported an effective occupancy of 1,443, compared with 1,114 for BONES-SEED.

Effective occupancy reflects how evenly samples fill covered regions. HiPHI’s reported advantage suggests that the collection did not achieve wider coverage by placing only a few outliers in distant regions.

Its long-tail share reached 14.1 percent, compared with 10.7 percent for BONES-SEED. The researchers repeated the analysis across several random seeds, projection settings, and grid resolutions, reporting a positive coverage margin in every tested configuration.

Those results still depend on the chosen representation and evaluation design. A two-dimensional projection cannot fully describe a complex motion manifold. However, balanced sampling makes the comparison more informative than raw hours alone.

Object Trajectories Make Interaction Data Physically Useful

A robot cannot learn carrying, pushing, or pulling from body motion alone because the object changes the movement it must execute.

HiPHI records object position and orientation alongside the performer’s skeleton. Each track has one entry for every body-motion frame, while the mesh supplies the object’s shape in a shared coordinate system.

This creates a connected representation of the person and object. The body does not merely mime holding a box. The dataset records how the box moves as the performer shifts posture, changes grip height, or transfers weight.

The distinction is important for whole-body control. Pushing a heavy object can require a forward lean, wider stance, and different foot placement. Pulling a suitcase can shift the torso and supporting leg as resistance changes.

Object meshes also clarify geometry. A chair’s surface determines where sitting contact should occur. A container’s dimensions influence hand placement, arm separation, and whether the torso must bend.

HiPHI evaluates this alignment with two metrics. The non-conflict rate checks whether the sampled human skeleton avoids penetrating the object. Near-surface grounding measures whether the person remains close enough to the object for an interaction to be plausible.

The dataset reports a 98.1 percent non-conflict rate and 95.7 percent near-surface grounding. HIMO reached 97.6 percent and 79.0 percent, respectively. OMOMO recorded 90.6 percent and 50.4 percent.

Those numbers favor HiPHI on the selected geometric tests, but they do not measure every aspect of contact. A hand can remain near an object without applying a realistic force. The metrics also do not verify friction, tactile response, or grasp stability.

Scale remains a notable difference. HiPHI reports 245.7 hours of interaction data. Its paper compares that with 21.6 object-track hours evaluated for HIMO and 9.8 hours for OMOMO.

The team also evaluated motion smoothness and common contact artifacts. Among datasets with comparable floor conventions, HiPHI reported the lowest values across five selected measures.

Its 95th-percentile ground penetration was 8 millimeters, compared with 18 for BONES-SEED and 111 for AMASS. Unsupported floating covered 0.015 percent of evaluated duration, while support-point drift measured 64 millimeters per second.

These are dataset-level measurements, not guarantees for every clip. Still, they target errors that matter during policy optimization. Sliding feet or inconsistent floor contact can teach a controller to chase references that physics will not permit.

The interaction benchmark later tests kick, carry, push, and lean sequences after retargeting. HiPHI achieved lower body-tracking errors in several comparisons, but not every one.

Push illustrates why the results require a careful reading. HiPHI recorded a body MPJPE of 99.20 millimeters for pushing, worse than OMOMO’s 66.09. MPJPE measures the average positional difference between corresponding joints.

At the same time, HiPHI produced substantially lower object position and orientation errors for that category. The result suggests that matching the object’s movement and matching the human pose can create competing demands.

Carry produced another mixed outcome. HiPHI’s body error was lower than both comparison results, but its object-position error was higher. These variations prevent a simple claim that one dataset wins every downstream task.

What HiPHI adds is a much larger test bed for those tradeoffs. Researchers can inspect when body tracking, object tracking, and physical stability align, and when optimizing one hurts another.

Reinforcement Learning Transfers HiPHI Motions to a Unitree G1

HiPHI matters because its researchers moved beyond static dataset statistics and tested whether trained policies could execute selected motions on real hardware.

The team retargeted human movements to a Unitree G1, a humanoid with different proportions and actuation limits. Retargeting converts a source skeleton’s motion into references that fit the target robot.

For body-only tests, the researchers trained policies with a DeepMimic-style reinforcement learning pipeline. DeepMimic research established a widely used approach for learning physics-based skills from reference motions.

The benchmark compares HiPHI with AMASS, LAFAN1, Motion-X++, and BONES-SEED under matched data budgets. Each source passed through the same retargeting and policy-training process.

HiPHI reportedly achieved the highest success rates and fastest convergence in both three-hour and 20-hour training settings. The comparison used five independent training runs and measured failure rates over training.

A separate scaling experiment increased unmirrored HiPHI training data from three to 300 hours. The resulting policies were evaluated on held-out motion from four other datasets.

Cross-dataset joint-position error consistently declined as the HiPHI training set grew, according to the paper. The experiment was repeated 10 times, with results reported as mean curves and variance bands.

That pattern supports the project’s core claim: additional structured motion continues improving generalization rather than merely repeating common actions. It also links dataset scale to an actual control metric.

The final step moved policies onto a physical G1. The robot performed running, sitting, crawling, carrying a box, flipping, and pulling a suitcase. These trials exposed the policies to actuator limits, sensing noise, and simulation-to-reality differences.

The demonstrations are meaningful because many motion datasets stop at reconstruction quality or simulation. A physical deployment provides stronger evidence that at least selected references can survive retargeting and hardware constraints.

Still, the phrase “real-world transfer” should not be interpreted as general-purpose autonomy. The reported trials cover chosen behaviors in controlled conditions. They do not show a robot independently perceiving unfamiliar objects, planning tasks, or adapting to an open-ended workplace.

A tracking policy also solves a narrower problem than a complete robot agent. It follows a reference motion while maintaining physical stability. It does not necessarily decide which action to perform or understand why a person performed it.

The G1 demonstrations therefore validate executability, not full task intelligence. That distinction matters as physical AI companies increasingly connect impressive motion videos with broader claims about useful labor.

HiPHI’s strongest contribution lies lower in the stack. It supplies reference data that can help policies learn coordination, balance, and object-conditioned movement. Planning and perception systems can then build on those capabilities.

The practical workflow is also demanding. Researchers must retarget BVH files, account for the robot’s joint limits, train in simulation, and validate safety before real deployment.

The project documentation explicitly warns that motions require retargeting and checks for the chosen embodiment. A reference that works on the G1 will not transfer unchanged to every humanoid.

For engineering teams, this makes dataset inspection and experiment tracking essential. A searchable engineering knowledge base can help connect motion identifiers, training configurations, failures, and hardware observations without changing the underlying robotics workflow.

What the HiPHI Results Still Do Not Measure

HiPHI narrows the motion-data gap, but it does not capture the complete physics, perception, or social context needed for general humanoid behavior.

The first limitation is environmental variety. All source motion was collected in a studio. That setting supports precise measurement, yet it removes the clutter, lighting changes, uneven surfaces, and unexpected obstacles found outside controlled facilities.

Internet video has the opposite profile. It provides messy environmental variation while sacrificing exact physical state. HiPHI strengthens one side of that tradeoff rather than eliminating it.

The second limitation involves force. The release records kinematics, meaning how bodies and objects move. It does not directly measure contact forces or tactile signals.

That omission matters for manipulation. Two clips can show similar trajectories while involving different grip pressure, friction, or resistance. A controller may need those hidden quantities to handle fragile, deformable, or slippery objects.

The third limitation is social interaction. HiPHI currently covers single-person motion. It excludes multi-person behavior and direct human-human contact.

Humanoids working around people must eventually model handovers, shared carrying, cooperative movement, and collision avoidance. Those tasks require data that represents two bodies and their changing intentions.

The fourth issue is augmentation. Nearly half of the reported release duration comes from left-right mirroring. This increases useful coverage, especially for unilateral behaviors, but does not add new performers or capture sessions.

Readers should therefore distinguish released hours from independently recorded hours. Both figures are valid for different purposes, but they answer different questions.

The fifth concern is benchmark independence. Most published comparisons and deployment evidence come from the dataset’s creators. CoRL acceptance adds peer review, yet broader confidence will require outside teams to reproduce the results.

Independent researchers should test whether HiPHI retains its advantage under different retargeters, policy architectures, robot bodies, and evaluation metrics. Results on smaller platforms or machines with different joint layouts would be particularly informative.

Licensing also affects practical adoption. The dataset uses a CC BY-NC 4.0 license, which allows research use with attribution but restricts commercial use. Companies must evaluate that restriction before incorporating the files into product training.

Privacy exposure appears limited in the public package. The release contains motion, trajectories, meshes, and anonymized performer attributes. It excludes captured RGB video, audio, facial imagery, and voice recordings.

However, motion itself can carry individual patterns. The authors use numeric actor identifiers and generalized demographic metadata, which supports analysis while reducing direct identification risk.

Finally, physically executable imitation is not equivalent to successful task completion. A robot can reproduce a carrying motion without recognizing the correct box, selecting a destination, or recovering when someone blocks its path.

HiPHI should be judged as motion infrastructure. It addresses how a humanoid can acquire varied, grounded movement references. It does not claim to solve perception, language-conditioned planning, safety certification, or open-world autonomy.

That narrower framing makes the work more credible. The open question is whether the dataset becomes a reusable foundation across research groups or remains closely coupled to its creators’ evaluation stack.

Three Signals Will Show Whether HiPHI Changes Humanoid Learning

The next test is adoption: independent reproduction, broader embodiment transfer, and richer physical sensing will determine HiPHI’s lasting value.

The first signal is independent benchmark reproduction. Research groups need to download the release, retrain comparable policies, and report results under documented conditions.

Reproduction would strengthen the claim that HiPHI’s coverage and precision drive performance. Large discrepancies would suggest that training choices, retargeting, or unpublished operational details contributed more than the dataset itself.

The second signal is transfer beyond the Unitree G1. Policies derived from HiPHI should be evaluated on humanoids with different proportions, joint ranges, actuator strengths, and control frequencies.

Successful multi-robot transfer would support the idea that the release captures reusable human-motion structure. Weak transfer would show that embodiment-specific adaptation remains the dominant bottleneck.

The third signal is complementary sensing. Future datasets or extensions should connect precise motion with force, touch, and first-person visual context.

Those modalities would address HiPHI’s clearest blind spots. Force data could distinguish light contact from weight-bearing support, while egocentric video could connect movement with what the performer saw.

The most promising path may combine rather than replace data sources. High-fidelity motion capture can supply executable physical references. Internet video can supply environmental and behavioral breadth. Robot demonstrations can anchor both to specific hardware.

HiPHI makes that hybrid direction easier to evaluate because it exposes a structured, searchable motion layer. Its Frame-LU system also provides semantic handles for sampling or linking related examples across sources.

Researchers should now ask concrete questions. Do policies trained on its long-tail motions recover better from disturbances? Do synchronized object records improve task success outside the capture studio? Can its semantic structure help models select motions from language commands?

The HiPHI humanoid motion dataset does not settle the debate between video scale and physical precision. It raises the standard for that debate by connecting collection design, measurable data quality, policy training, and real-robot execution.

The next useful step is direct experimentation. Download a subset, audit its mirrored and original sequences, retarget several interaction categories, and publish the failures alongside successful demonstrations. Those results will reveal whether structured human motion becomes shared infrastructure for physical AI or another impressive benchmark that robots struggle to carry into unfamiliar environments.