Physical AI training data

Egocentric video of people doing real physical work, captured to a brief and annotated for policy and world-model training.

Physical AI training data pairs what a system sensed with what was done, so a model can learn to act in the physical world. For robots it comes from four sources: teleoperated robots, simulation, video of people doing the task, and public datasets.

From a brief to a delivered file

  1. Capture

    A Capture Partner records against the brief.

    The consent receipt's SHA-256 is written into a hash-chained ledger.

    Recompute an example entry's hash

  2. Redact

    Faces the detector finds are blurred before any human review.

  3. Segment

    The Recording is cut into usable Clips in one pass.

    Each Clip id carries the first eight characters of its SHA-256.

  4. Annotate

    A model estimates 3D hand pose and confidence, both derived.

  5. Deliver

    The annotated data ships in the container the brief names.

    The package manifest lists each file's SHA-256.

    Recompute them with shasum -a 256.

A Recording and its Clips travel these five stages in order.

What physical AI is trained on

In NVIDIA's words, physical AI lets autonomous systems such as robots and self-driving cars “perceive, understand, reason, and perform or orchestrate complex actions in the physical world” (NVIDIA glossary). Its Cosmos paper splits what such a system needs in two: a policy model for the system itself, and a world model for its surroundings (arXiv:2501.03575).

GR00T N1, NVIDIA's open foundation model for humanoid robots, was trained on a mix of real-robot trajectories, human videos and synthetic data (arXiv:2503.14734). Human video is the part of that mix a brief commissions: egocentric video of hands doing the task, in the environment you name.

The four sources, side by side

Where physical AI training data comes from, with one published example of each
SourceWhat a model gets from itWhat limits itPublished example
Teleoperated robotsTrajectories with actions recorded in the robot's own jointsEvery hour needs a robot and an operator, and the actions belong to one robot bodyOpen X-Embodiment pooled data from 22 different robots across 21 institutions (arXiv:2310.08864)
SimulationAs many trajectories as compute allows, labelled as they are generatedSimulated physics and images differ from the real worldGR00T N1's synthetically generated datasets (arXiv:2503.14734)
Egocentric human videoReal tasks in real places, with the hands in viewNo robot actions: hand pose and action labels are derived from the videoEgoDex recorded its hand tracking on Apple Vision Pro at capture time (arXiv:2505.11709)
Public datasetsA free place to startFixed tasks, and licences that often rule out commercial trainingThe egocentric datasets and their licence terms

What the annotation layer carries

An annotated Clip is its video plus an FH-Ego v1 annotation file, and together they carry the signals below. The last column marks the recorded footage as measured, and everything software produced as derived.

Signals in an annotated Clip, with the provenance each one carries
SignalWhat it holdsProvenance
VideoThe Clip itself; faces the detector finds are blurred before anyone reviews itMeasured
Hand keypoints21 points per detected hand, from the MediaPipe hand landmark model, on sampled framesDerived
Wrist position and velocityWhere each detected hand's wrist sits in the image, with a depth value, and how fast it is moving. The values have no physical unitDerived
Per-hand confidenceThe hand detector's score for whether each detected hand is a left or a right handDerived
Action windowsWindows across the Clip, each with a verb-noun codeDerived
CaptionsOne text description per action window, and the file names the captioner that wrote themDerived

FH-Ego v1 documents every field, with one real Clip's values beside it.

Our reference clip shows the layer on one real Clip: 1,843 frames at 30 fps, hand keypoints on 185 sampled frames, 16 action windows, and a mean hand confidence of 0.87. The frame counts are measured; the windows and the confidence are derived.

Questions about physical AI data

How is physical AI training data different from ordinary video?
Each sample ties what was seen to what was done, so a model learns the next action rather than only what is in view. That is why our Clips carry hand keypoints and action windows beside the video, each marked as derived.
Should we use robot data or egocentric human video?
They are combined rather than chosen between: GR00T N1 trained on both, plus synthetic data. Egocentric video adds real tasks in real places, recorded by people rather than robots, and that is the part a Capture Brief commissions.
Which formats do you deliver?
LeRobot v2.1, HDF5 and FH-Ego v1 JSON as standard, with Zarr v2 and TFRecord on request. Every package's manifest lists each file's SHA-256.
How is consent recorded?
Each Recording carries a consent receipt the Capture Partner can withdraw under India's DPDP Act 2023. The receipt's SHA-256 sits in a hash-chained ledger, and you can look a receipt up yourself.
How does a project start?
You name the task, the environment and the delivery format. We scope a Capture Brief first, then price it.

Scope a physical AI brief

Tell us what your model has to learn and where that work happens. We scope a Capture Brief for the footage and the annotation tier it needs.

Scope a pilot Browse the catalogue labs@fourthuman.ai