FH-Ego v1, field by field

Read this before the tables

Tier 1 · Task / session

The clip

One object per delivered clip. Everything else hangs off it.

task_id — Internal identifier for the annotation run

title — Human-readable title for the clip

task_label — Taxonomy code for the task performed

task_name — Readable name of the task performed

details — Free-text description of the session

scene — Environment the capture was made in

usable — Whether the clip passed QA and may be delivered

start_frame — First frame of the clip

end_frame — The clip's frame COUNT, not the index of its last frame. Frames are numbered from zero, so the last one is end_frame minus one — on the reference clip this reads 1843 while the final action segment ends at 1842

fps — Frames per second of the source video

actions — The task-bounded action segments this clip was cut into

tracks — The parallel supervision layers

provenance — Where this clip came from — see the provenance fields below

Tier 1 · Task / session

Where the clip came from

Carried inside the annotation. The hash lets you verify the bytes offline by rehashing them; the consent receipt is the one handle that resolves against our API rather than in your hand.

clip_id — Canonical FH1 clip identifier; the tail is derived from the clip's SHA-256

scheme_version — Identifier scheme version

content_sha256 — SHA-256 of the capture. Rehash the file to verify this annotation describes the bytes you hold

cohort — Capture cohort

captured_at — When the capture was recorded (partner-declared)

consent_receipt_id — Consent ledger receipt covering this clip; verifiable at GET /api/v1/consent/receipt/{id}

qa_verdict — Automated QA outcome for this clip

qa_reject_reasons — Closed-vocabulary reject reasons, empty when accepted

derived_fields — Fields on this record that are model-generated, never measured (AGENTS.md 4.1)

caption_source — Which captioner produced the derived caption fields: a model id (e.g. claude-opus-5) or 'heuristic_stub' for development filler. CARRY C.6

Tier 2 · Action segment

One action segment

The clip is divided into fixed windows and each is given a verb-noun code. On the clip below that division is a template, not detection — content-aware segmentation is specified and not built.

start_frame — First frame index of action chunk

end_frame — Last frame index of action chunk

hand — Operative hand: 'left', 'right', or 'both'

scene_semantic — Interacting surface/workstation area

camera_motion — Camera motion classification

action — Standardized verb-noun action code (e.g. scoop_food)

description — Dense natural language sentence describing physical interaction

Tier 3 · Timeline track

The parallel tracks

Supervision layers that run the length of the clip alongside each other.

hand_control — Header describing the kinematics track

object_interactions — The noun half of each action code, de-duplicated. This is NOT object detection — no object detector runs anywhere in the pipeline, and these words are read off the action codes. Generated rather than observed, and listed in the clip's own derived-fields list

action_verbs — Distinct action codes appearing in the clip. Generated rather than observed, and listed in the clip's own derived-fields list. Where the clip records its caption source as a stub, these come from a fixed five-item pool rather than from a model

dense_captions — One caption per action segment. Generated rather than observed, and listed in the clip's own derived-fields list. Each clip records what produced them in its caption-source field; a stub value there means a template wrote them, not a model

kinematics_3d — The sampled wrist track, one entry per sampled frame per detected hand

Tier 3 · Timeline track

The kinematics track header

Describes the sampled track below it — including the sample interval, so the file states its own rate.

hand_dominance — Distinct operative-hand codes across this clip's ACTION SEGMENTS — 'left', 'right' or 'both'. Not a list of detected hands: 'both' is not a hand, and it is computed from the generated action track rather than from the tracker

total_frames_annotated — Frames in the source clip

kinematic_sample_frames — The number of ENTRIES in kinematics_3d, not a count of frames — one entry is written per detected hand per sampled frame, so a clip with two hands in view produces more entries than frames. On the reference clip it reads 294 across 185 distinct frames. The name is misleading and is kept only because it is already in use

kinematic_sample_every_n_frames — Every Nth frame was sampled into kinematics_3d.

kinematic_sample_rate_hz — Derived: clip fps ÷ kinematic_sample_every_n_frames.

Tier 3 · Timeline track

One kinematics sample

One entry per sampled frame per detected hand. This is the layer that carries the detector's output.

frame_idx — Index of the source video frame this sample was taken from

timestamp_sec — Time of this sample from the start of the clip

hand — Which hand this sample describes. Assigned by where the wrist sits in the frame, not by the detector's own left/right call — the leftmost hand is labelled left, and with a single hand in view, left of centre is left. The detector's handedness judgement is computed and then discarded

position_3d_m — Camera-relative 3D position of the wrist. NOT METRES despite the key name — x and y are image-plane fractions rescaled by hardcoded constants, z is an arbitrary offset plus a hand-relative depth. No camera intrinsics are published. The key keeps its name under V1

velocity_3d_mps — First difference of position_3d_m over the sample interval. Inherits position's unit problem: it is that dimensionless quantity per second, not m/s

confidence — Per-hand confidence — MediaPipe's HANDEDNESS classifier score — how sure the detector was about its own left/right call. We do not use that call: the hand field below is assigned by wrist position, so this score qualifies a judgement we discard rather than the label we deliver. It is not positional and not per-joint, and the 21 landmarks carry no confidence of their own

keypoints_21 — 21 MediaPipe hand landmarks, normalised image coordinates [x, y, z] per joint, in MediaPipe's canonical joint order (0 = wrist, 4 = thumb tip, 8 = index tip, … 20 = pinky tip)

What one real clip holds

Numbers below are counted from the published reference clip at build time, not typed.

The sample interval is recorded on every clip rather than assumed, so a file can always be checked against the rate it claims. A track sampled every tenth frame of thirty is a three-hertz track, and calling it thirty would be the easiest false number on this page to publish.

Open the clip these numbers come from · What arrives, and in what container · Every number we publish, and where it comes from

curl -sO https://fourthhuman.com/SHA256SUMS
curl -sO https://fourthhuman.com/reference-run.json
sha256sum -c SHA256SUMS        # or on a Mac: shasum -a 256 -c SHA256SUMS