Dataset · v1.0 · CC BY 4.0

HubEgo-100K

Summary

HubEgo-100K is an open dataset of egocentric video of skilled manual work for robot learning. Release v1.0 has 10,124 hours of 1080p video with synchronized 100 Hz IMU data from one head-mounted phone camera, recorded by 12,227 contributors in 15,548 environments. It is free under CC BY 4.0 on Hugging Face, and the program releases data in stages toward 100,000 hours.

Grid of first-person frames from HubEgo-100K showing skilled manual work across workshops, kitchens, construction sites and craft benches

Key facts (v1.0)

MetricValueSource
Hours of video (v1.0)10,124 hoursarXiv paper
Total video frames1,093,194,072 framesHF dataset card
Episodes72,754 episodesarXiv paper
Median episode length4.9 minHF dataset card
Maximum episode length30 minHF dataset card
Unique contributors12,227 contributorsarXiv paper
Unique environments15,548 environmentsarXiv paper
Setting types11 setting typesarXiv paper
Hours in workplace settings83%arXiv paper
Work categories30 work categoriesarXiv paper
Task types56 task typesarXiv paper
Action families35 action familiesarXiv paper
Unique task descriptions62,000+HF dataset card
Sampled frames with at least one hand94.7%arXiv paper
Sampled frames with both hands76.3%arXiv paper
On-task share (model estimate)96.9%HF dataset card
Hours annotated as hard34%HF dataset card
Hours annotated as medium47%HF dataset card
Video specification1920x1080, H.264, 30 fpsHF dataset card
IMU sample rate100 HzarXiv paper
Capture hardwareone head-mounted phone cameraarXiv paper
FormatLeRobot v3 + MCAPHF dataset card
LicenseCC BY 4.0arXiv paper
Access processshort form, automatic approvalarXiv paper
Program target (staged releases)toward 100,000 hoursarXiv paper

What the recordings show

Every episode in HubEgo-100K v1.0 is filmed by a Hub contributor performing their own unscripted manual work in their own workplace or home, using their own tools. Tasks span bricklaying, electrical wiring, plumbing, industrial sewing, lathe machining, watch repair and phone board microsoldering.

Across the release, 83% of hours take place in workplace settings and 18 work categories each hold more than 200 hours, with the largest category accounting for 13.1% of hours. No single action family exceeds 10% of hours.

Setting What it covers
Trades done in homes A trade or craft done in a home: a home sewing table or repair bench, or a tradesperson working in someone's home (plumbing, electrical, appliance and bike repair, jewelry, printing)
Workshop & repair bench A dedicated workshop or repair shop
Construction & renovation site Building sites and homes under renovation
Salon & personal care Hair, nail, beauty and pet grooming, in a salon or at home
Commercial kitchen Restaurant, bakery and food-production kitchens
Shop floor Shops and supermarkets
Warehouse & logistics Stock rooms, warehouses, packing
Service & front of house Offices, clinics, hotels, reception
Farm & garden centre Fields, greenhouses, nurseries
Factory & industrial Production lines, factory floors
Household Home DIY and gardening, and service tasks done at home (laundry, room cleaning)
Bar chart of the top work categories in HubEgo-100K v1.0 by hours
Distribution of hours across the top work categories in HubEgo-100K v1.0 (30 work categories total; 18 exceed 200 hours).
Bar chart of action families in HubEgo-100K v1.0 by hours
Hours across the 35 verb-level action families. No single action family accounts for more than 10% of hours.
Chart of task difficulty split in HubEgo-100K v1.0
Difficulty distribution across v1.0 hours: 34% hard and 47% medium, concentrated in technical trades and repair.

Annotations

Every episode has a row in meta/annotations.parquet, keyed by episode_index:

Field Content
description Short task description, for example "Machine-sew and overlock sleeveless top seams"
task_instruction One or two sentences describing the main goal, actions and objects in order
action_family Main manual action across 35 families (cut, sew, assemble, fasten, clean)
skill_group Skill area the task exercises across 18 groups (such as Electronics & Diagnostics)
difficulty Easy, medium or hard (34% hard, 47% medium by hours)
environment Setting type where the episode was filmed (11 setting types)
on_task_share Estimated share of episode time spent actively doing the task

meta/subtasks.parquet and meta/subtask_occurrences.parquet expose the step-level instructions in the LeRobot v3 subtask table format.

Data format and loading

All head-mounted monocular recordings live in the mono/ directory as a LeRobot v3 dataset, paired with native MCAP sensor archives and per-episode parquet metadata:

  • Video (mono/videos/observation.images.main/): one MP4 per episode (1920x1080, H.264, 30 fps). 84% of episodes are the phone's original stream; the rest were recorded at 60 fps and converted to 30 fps.
  • Frame data (mono/data/): LeRobot v3 parquet with one row per video frame. observation.state holds frame-aligned accelerometer (m/s2, gravity included) and gyroscope (rad/s) values.
  • Native sensors (mono/mcap/): one MCAP file per episode storing native accelerometer, gyroscope and device motion streams (plus magnetometer, heading and barometer where available), indexed by archive, member and byte offset in catalog.parquet.
  • Catalog (mono/catalog.parquet): task type, work category, pseudonymous contributor and recording ids, duration, device model, lens, field of view, hand visibility estimates and synchronization signals.

Read the catalog and annotation tables directly without downloading video:

import pandas as pd

cat = pd.read_parquet("hf://datasets/HubLabs/HubEgo-100K/mono/catalog.parquet")
ann = pd.read_parquet("hf://datasets/HubLabs/HubEgo-100K/mono/meta/annotations.parquet")

Load a small subset of episodes locally with LeRobot v3:

import glob
import pandas as pd
from huggingface_hub import hf_hub_download, snapshot_download
from lerobot.datasets.lerobot_dataset import LeRobotDataset

REPO, G = "HubLabs/HubEgo-100K", "mono"
EPISODES = [503, 203, 454]
local = snapshot_download(
    REPO,
    repo_type="dataset",
    local_dir="HubEgo-100K",
    allow_patterns=[f"{G}/meta/**", f"{G}/catalog.parquet"],
)
root = f"{local}/{G}"
meta = pd.concat(map(pd.read_parquet, glob.glob(f"{root}/meta/episodes/*/*.parquet"))).set_index("episode_index")
cat = pd.read_parquet(f"{root}/catalog.parquet").set_index("episode_index")
for e in EPISODES:
    c, f = meta.loc[e, "data/chunk_index"], meta.loc[e, "data/file_index"]
    hf_hub_download(REPO, f"{G}/data/chunk-{c:03d}/file-{f:03d}.parquet", repo_type="dataset", local_dir=local)
    hf_hub_download(REPO, f"{G}/{cat.loc[e].video_path}", repo_type="dataset", local_dir=local)
ds = LeRobotDataset(REPO, root=root, episodes=EPISODES)

Access and license

HubEgo-100K is released under CC BY 4.0. It is free to use, share and adapt for any purpose, including commercial use, with credit:

HubEgo-100K, Hub Labs (Hub Data, Inc.), https://huggingface.co/datasets/HubLabs/HubEgo-100K, CC BY 4.0

Access on Hugging Face uses short form, automatic approval. The catalog and annotation tables can be inspected without downloading video, and individual episodes can be fetched by index.

How the data was collected and curated

  1. Contributors recorded a named, unscripted task with one head-mounted phone camera during their own work, in their own workplace or home.
  2. A dense vision-language pass segmented every recording in time and retained stretches where the wearer's hands perform the task (96.9% on-task time).
  3. Human review decisions, where available, overrode automated segmentation; frame timing, repeated frames and frame-rate drift were logged per episode in catalog.parquet.
  4. Privacy filtering ran on every episode before release.
  5. Hours were rebalanced across the 56 task types so no single task dominates v1.0.

Privacy

Every contributor agreed that their recordings may be used to train and evaluate AI and robotics systems. Before release, Hub removed:

  • Stretches where a child, another person's face or private text (documents, screens, cards, license plates) was detected, cut with a 5-second margin, as well as recordings where faces fill a large share of the footage.
  • Episodes in which the video annotation detected a minor.
  • Care tasks performed on another person.
  • Audio, location, device identifiers and contributor identity (identifiers in the release are pseudonymous strings; phone make and model remain as capture metadata).

Limitations

  • Machine-generated annotations: Descriptions, instructions, action families, difficulty labels, hand visibility and on-task shares were generated by a vision-language model and have not been audited by human reviewers.
  • Monocular single-camera setup: v1.0 is recorded from one head-mounted phone camera with synchronized IMU. It does not include stereo depth, multi-camera views or ground-truth 3D hand poses.
  • Re-encoded subset: 84% of episodes are the phone's original stream; the rest were recorded at 60 fps and converted to 30 fps. Per-episode encoding and timing signals are stored in catalog.parquet.
  • Audio removed: Audio tracks are stripped from every episode as part of privacy filtering.
  • Staged release and benchmarks: v1.0 contains 10,124 hours, the first stage of a program releasing data toward 100,000 hours. Model benchmarks are planned for later versions of the paper.

Paper and authors

Authors (arXiv order): Javad Amirian, Armin Kiani, Loic Schaeffer, Tim Sprecher, Mohamed Taha, Karthik Yadati.

Cite HubEgo-100K

Amirian, J., Kiani, A., Schaeffer, L., Sprecher, T., Taha, M., Yadati, K. (2026). HubEgo-100K: In-the-Wild Egocentric Video of Skilled Work for Robot Learning. arXiv:2611.21938.

@misc{amirian2026hubego100kinthewildegocentricvideo,
  title         = {{HubEgo-100K}: In-the-Wild Egocentric Video of Skilled Work for Robot Learning},
  author        = {Javad Amirian and Armin Kiani and Loïc Schaeffer and Tim Sprecher and Mohamed Taha and Karthik Yadati},
  year          = {2026},
  eprint        = {2611.21938},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  doi           = {10.48550/arXiv.2611.21938},
  url           = {https://arxiv.org/abs/2611.21938}
}

@dataset{hubego100k2026,
  author    = {{Hub Labs (Hub Data, Inc.)}},
  title     = {{HubEgo-100K}},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/HubLabs/HubEgo-100K}
}

Changelog

  • v1.0: First release: 10,124 hours across 72,754 episodes, 12,227 contributors and 15,548 environments.

Resources and viewers