Robot Experience

The defining scientific resource of Physical AI — not raw robot data alone, but validated, physically grounded interaction.

Intuitively, robot experience is the physical history of a robot performing a task: what the robot and its surroundings were doing, how they interacted, what was observed about that interaction, and what outcome resulted. A useful recorded robot experience therefore captures not only the robot itself, but the evolution of the complete coupled physical process in which the robot participates — preserving the full perception–action–consequence loop, tied explicitly to embodiment, environment, task, and measurable outcome.

A Simplified Formal Picture

Formally, the Robot Data Factory represents an episode as the coupled state of the robot, the objects it interacts with, the environment, and any other agents involved — recorded over time as a trajectory. Because the complete physical state is never fully observable, every recorded episode is characterized by two combined qualities: how informative the physical interaction itself was (capability, performance, safety, robustness), and how faithfully the observation process captured it (completeness, accuracy, synchronization, ground truth). Both combine into a single experience-quality measure:

Q_ε = 𝒬(Q_obs, Q_phys)

Importantly, successful task execution is not a prerequisite for useful experience: a failed episode can constitute high-quality robot experience when the relevant trajectory is well observed and the cause and outcome of the failure are quantitatively characterized. Robot experience is therefore the fundamental scientific resource of Physical AI, and the primary object engineered by the Robot Data Factory.

How RDF Works

Every component of the Robot Data Factory is one piece of a single closed loop.

Physical training grounds (the chambers) generate missions through the agentic robot network and Mission-Control; the ubiquitous data pipeline synchronizes, validates, and scores the result into a structured robot dataset. Each mission is decomposed along the full hierarchy Mission → Task → Skill → Episode → Dataset → Benchmark → Capability, with Capability — not raw data volume — serving as RDF's ultimate measure of progress. Improved models are redeployed at increasing mission difficulty, and over time new chambers, robots, and missions extend the loop through global co-creation with other laboratories.

Each concept feeds directly into the next — a mission breaks down into tasks, tasks draw on reusable skills, every attempt is recorded and scored as an episode, scored episodes accumulate into a dataset, the benchmark ranks and tracks those scores over time, and that track record compounds into capability — defined precisely below:

Mission-Control drives this same loop day to day as a repeating cycle — Deploy, Measure, Learn, Repeat:

• Deploy — run the mission via the agentic robot network and Mission-Control, by policy, classical control, or teleoperation
• Measure — capture the episode through onboard sensors, external tracking, haptic devices, and Mission-Control logs
• Learn — convert validated episodes into datasets for world models, VLA models, and policies
• Repeat — redeploy improved models or operators while raising mission difficulty

The loop turns the Robot Data Factory from a one-off collection effort into a continuous learning system: each iteration is comparable to the last, so progress is measured by data quality, safety, and transfer performance — not just final task success.

System

Physical chambers, robot platforms, sensors, software, dashboards, and evaluation infrastructure operate as connected parts of a single robot-learning ecosystem.

Physical AI System

Every chamber follows the same layered hardware model. At the edge, robots and sensors generate data streams, processed locally by robot-side compute — an onboard embedded computer for mobile platforms, or a controller workstation for stationary ones. Each workstation uplinks over Ethernet and fibre to the data factory's shared switching fabric, which in turn connects it to compute, storage, monitoring, and time-synchronization infrastructure shared across all chambers — with wired, Wi‑Fi, or private 5G links chosen depending on whether a platform is stationary or mobile.

Two deployed chambers show the hardware model in practice, aligned with the scheme of our Physical AI System to ensure the connection:

Robot Nodes

Each robot is treated as a deployable agent node — the physical platform, local sensing, state interface, control interface, safety interface, and data-recording connection. Conceptually, every physical or digital component that participates in generating robot experience is represented the same way, as a node in the Robot Data Factory. New chambers, robots, and laboratories can join by implementing compatible node interfaces.

On top of the Physical AI System, the robot nodes are virtualized and connected via bridge in the Robot Nodes Dashboard, giving every chamber, robot, and peripheral a live entry showing its embodiment, connection state, and the number of active data topics relative to what's available — so operators can confirm at a glance that every node is configured and ready before a mission begins.

Each dashboard entry is also a hub: it links out to that robot's topic-stream view, the pit-stop application, dedicated pages in the R-QoS network and Data Pipeline dashboards, and its mission and demonstration gallery — making the Robot Nodes Dashboard the interface an operator moves through to reach every other component of the system. More than a user interface, it is the first gate of the data-quality, where a live dashboard reports robot readiness, connection, recording state, and safety status across the fleet.

Robot Network

Beyond the robots themselves, motion capture, external cameras, haptic devices, dashboards, digital twins, storage services, and benchmark evaluators are all connected over a shared network handling ROS2 bridges, time sync, telemetry, and safety status. That shared network is itself hybrid, combining wired Ethernet and fibre for fixed platforms and high-throughput links, Wi‑Fi for mobile robots, and an emerging private 5G layer for mission-critical, low-latency communication.

That network is made visible as a live topology graph: every robot appears as a node linked to its chamber, with link color and animation encoding latency, bandwidth, and connection state, so a healthy link stands out from a degraded or offline one at a glance.

Underneath the graph, the R-QoS framework continuously monitors link health, robot- and workstation-side network traffic, and clock synchronization across the fleet, preserving that network-quality evidence alongside the experiment data it describes.

Ubiquitous Data Pipeline

The Physical AI System, Robot Nodes, and Robot Network produce raw robot experience; the ubiquitous data pipeline is the infrastructure that converts that experience into learning-ready data, across five layers, from the sensors on each chamber robot up to the datasets.

• Data Production — raw streams from each robot's sensors and surrounding IoT systems.
• Data Communication & Control — networking, middleware such as ROS2 for control and data subscription, storage, and visualization.
• Data Package — episodes collected, curated, annotated, verified, and made retrievable.
• Data Consumption — feeds model training, benchmarking, and release.
• Data Governance — spans and governs the whole stack end-to-end.

Data Governance enforces this end-to-end: only episodes meeting criteria like stream completeness, timestamp alignment, and task completion enter the training dataset. A failed episode is kept only if the reason for its failure is recorded — otherwise it's dropped.

An operational instance of data pipeline — tracking sessions, packages, validation, datasets, and releases across the fleet — is provided by the Data Pipeline Dashboard. Recorded bytes stay in each chamber's own object store; the dashboard itself tracks only identity, catalog entries, checksums, and lifecycle state, making it a control-and-provenance layer over the data rather than only a transport path.

Digital Twins

Each chamber is mirrored by a 3D digital twin reproducing its geometry, objects, sensor placements, and dynamics. Twins are released alongside the chamber descriptions and support simulation, synthetic data generation, scenario replay, and sim-to-real / real-to-sim transfer — enabling risk-reduced experimentation and policy optimization before physical deployment.

Robot embodiments are twinned the same way: a growing catalog of controllable models — legged, wheeled, arm, humanoid, and underwater/surface platforms — each built with real kinematics, joint limits, and mass/inertia rather than visual geometry alone, so it simulates in physics engines rather than just rendering. Any robot in the catalog can be spawned inside a simulation environment to run the same digital-twin checks before a new embodiment ever enters the physical chamber.

Simulation here is not a substitute for real-world data: each twin must stay grounded in physical behavior, continuously validated against real robot execution, so synthetic experience remains trustworthy rather than drifting from what the physical chamber actually does.

Pitstop Leaderboard

The evaluation layer of the architecture: every validated mission is scored and ranked on the Pitstop Leaderboard, converting recorded episodes into a living, continuously updated measure of capability. Missions and scoring are detailed on the Missions & Benchmark page.

Training Grounds

Physical training environments that group related missions under common objects, dynamics, sensing modalities, and operational conditions.

No single physical environment can generate the diversity of robot experience that scalable Physical AI requires — domestic tasks demand manipulation and social interaction, marine and environmental tasks demand navigation under visual degradation and vehicle coordination, and energy tasks demand route planning and infrastructure inspection.

Researchers can contribute algorithms for evaluation on these shared physical platforms, enabling reproducible comparison under identical real-world conditions. The chambers below cover these complementary regimes, deliberately differing in physics, constraints, and sensing while sharing one data and evaluation framework.

↻ Hover over a chamber's photo to see its live 3D scan.

Home

Domestic training ground covering kitchen, dining, bedroom, and living room.

• Mobile manipulation & humanoid household tasks
• Serving, table setting, laundry sorting
• Shelf interaction & object handover

Environment

Marine and environmental ground with a pool, vegetation zone, ROVs, and USVs.

• ROV docking & underwater teleoperation
• Mangrove & vegetation health inspection
• Surface–underwater coordination

Energy

Inspection and monitoring ground with motors, pumps, solar panels, and electrical infrastructure.

• Route monitoring & viewpoint capture
• Anomaly documentation
• Solar panel inspection & cleaning

Space Coming Next Phase

Space-analog training ground for satellite servicing, inspection, and aerial–ground coordination.

• Satellite servicing & inspection
• Aerial & ground robot coordination
• Space-analog terrain navigation

Manipulation Coming Next Phase

Industrial training ground for manipulation, assembly, and human–robot collaborative production.

• Pick-and-place & assembly tasks
• Dexterous manipulation
• Human–robot collaborative production

Clinic Coming Next Phase

Envisioned extension into medical and surgical robotics, especially for remote and dexterous control.

• Remote surgery
• Remote diagnosis

Additional chambers are planned as the Robot Data Factory expands to a second phase.

Missions & Benchmark

Reproducible missions that generate interaction episodes, scored by a living benchmark.

A mission defines an objective, chamber, embodiment, and scoring rules — run by a human, teleoperator, shared-control system, autonomous policy, or a hybrid mode. Representative missions across the active chambers:

Table Setting

A mobile bimanual manipulator sets a dining table in the Home Chamber, placing each setting seat by seat.

Laundry Sorting

A humanoid robot locates, collects, and sorts scattered laundry into a basket in the Home Chamber.

Vegetation Sampling

A legged platform ascends a wet ramp to reach and grip a target plant specimen in the Environment Chamber.

ROV Docking

A teleoperated ROV aligns and docks beneath a USV in the Environment Chamber.

Inspection

A quadruped patrols energy infrastructure, framing gauges and documenting anomalies in the Energy Chamber.

Solar Panel Cleaning

A mobile manipulator executes a force-regulated cleaning pass across a solar panel array in the Energy Chamber.

Dishwashing

A dual-arm manipulator rinses and washes plates and mugs at the sink in the Home Chamber.

Cooking & Plating

A dual-arm manipulator stirs and finishes cooking on the stovetop, then ladles, plates, and garnishes the dish in the Home Chamber.

Difficulty rises along a skill curriculum — e.g. table setting from a clear table to clutter and occlusions, or docking from calm water to degraded visibility and moving targets. The mission library is actively growing — new missions, chambers, and difficulty tiers are continually developed and released.

Mission Score

Mission Score is a single number that quantifies how well one recorded mission attempt was executed, combining weighted sub-scores — for example, these six:

• Completion (20%) — was the task finished successfully
• Quality (20%) — accuracy, correctness, cleanliness
• Productivity (15%) — efficiency and speed
• Safety (15%) — collisions and unsafe interactions
• Data validity (15%) — synchronization, completeness, and usability of the recorded data
• Human similarity (15%) — behavioral closeness to expert execution

The weights above are an illustrative allocation, not a tuned result, and are expected to be refined per mission family once sufficient pilot data have been collected.

The resulting weighted sum is then multiplied by the difficulty multiplier below, so identical execution scores higher on a harder mission. Safety and data validity also act as gating factors that multiply the result directly, so unsafe execution or invalid data can't be offset by strong performance in other dimensions.

Difficulty Multiplier

The difficulty multiplier is based on properties of the physical mission itself — how demanding the embodiment, environment, and task are — combining three factors, each weighted per mission family:

• Embodiment demands
• Environmental conditions
• Task complexity

Sensor or modality richness is deliberately excluded: richer observation can improve data quality without making the underlying physical mission intrinsically harder.

The Pitstop Leaderboard

Every scored mission attempt feeds into the Pitstop Leaderboard, the living benchmark that ranks robots and policies over time. Results are separated into six tracks, so autonomy level is never hidden behind one score:

• Human reference
• Teleoperation
• Shared control
• Autonomous policy
• World-model prediction
• Data-factory quality

Separation prevents overclaiming: a policy that finishes a task but produces unusable or incomplete data doesn't pass as a full autonomy success, and a world model that predicts plausible trajectories without supporting reliable execution isn't the same as a working policy.

Dataset & Software

Validated, synchronized multimodal robot experience, and the software that produces it — reusable across embodiments and tasks.

Every episode is stored as a structured record — instruction, chamber configuration, robot embodiment and state, multimodal observations, actions, external ground truth, and metadata with an evaluation score — so that robot data remains scientifically useful rather than shipped as raw trajectories alone.

Alongside the dataset, the software that produces and evaluates it — data schemas and mission definitions, benchmark scoring scripts, and robot-node interface code — is also being prepared for release, so others can reproduce the pipeline rather than just consume its output.

Publishing Soon

• Mission-based dataset releases
• Data schemas & mission definitions
• Benchmark scoring scripts
• Robot-node interface code & repositories

A useful first release doesn't maximize size — it prioritizes synchronization quality, clear task definitions, reliable external ground truth, and transparent scoring.

The Robot Data Factory

Abstract

Physical AI requires more than increasingly large robot datasets. Unlike digital AI, intelligent robots acquire knowledge through continuous interaction with the physical world. The defining scientific resource of this paradigm is therefore not raw robot data alone, but validated robot experience: physically grounded interactions whose observations, actions, outcomes, and environmental context are synchronized into reusable scientific data. Existing robot datasets and benchmarks have advanced manipulation, navigation, teleoperation, and embodied policy learning, yet many remain constrained by single-platform collection, task-specific protocols, limited external ground truth, and static evaluation settings. These constraints make it difficult to generate reusable robot experience that supports cross-embodiment learning, world modeling, vision-language-action policies, and reproducible scientific comparison. They also limit the ability to study the full progression of robotic capability, from basic motion and locomotion to manipulation, mobile manipulation, loco-manipulation, and fine manipulation, where physical interaction, tactile feedback, embodiment, and task execution become increasingly coupled.

This paper presents the Robot Data Factory, a mission-driven training-ground framework for the continuous generation, validation, and engineering of robot experience. The framework elevates robot experience from a passive by-product of experimentation to the primary scientific resource of Physical AI. Reproducible missions generate interaction episodes, validated and synchronized multimodal observations become reusable robot experience, and this experience supports benchmark leaderboards, skill curricula, digital twins, world models, vision-language-action models, and embodied robot policies. Rather than serving as a repository of collected demonstrations, the Robot Data Factory operates as a continuously evolving scientific infrastructure that generates new robot experience, validates it with synchronized multimodal sensing and external ground truth, and benchmarks algorithms under reproducible physical conditions.

The Robot Data Factory is organized as a network of environment-specific training grounds, including domestic, environmental, and energy chambers. These chambers are connected through an agentic robot network, a mission-control dashboard, a ubiquitous data pipeline, and a benchmark framework. The resulting infrastructure supports progressive capability assessment across motion, locomotion, manipulation, mobile manipulation, loco-manipulation, and fine manipulation while preserving comparable records across diverse robot embodiments and task settings. Unlike static datasets, the system is designed as a living challenge in which missions, embodiments, difficulty levels, and benchmark leaderboards continuously evolve. Researchers can contribute algorithms for evaluation on shared physical platforms, enabling reproducible comparison under identical real-world conditions while continuously expanding the repository of validated robot experience. The proposed framework establishes robot experience as a continuously engineered scientific resource and positions the Robot Data Factory as a foundational infrastructure for scalable Physical AI.

Keywords

If you use the Robot Data Factory (paper, training-ground infrastructure, dataset, or software), please cite it as below.

@article{haddadin2026robotdatafactory,
  title   = {The Robot Data Factory},
  author  = {Haddadin, Sami and Laptev, Ivan and Reid, Ian and Song, Dezhen and Stefanini, Cesare
             and Swikir, Abdalla and Zuo, Xingxing and Saad Saoud, Lyes and Hamandi, Mahmoud
             and Heshmat, Mohamed and Doukhi, Oualid and Naceri, Abdeldjallil and Bashir, Attique
             and Mohamed, Abdelrahim and Tomi{\'c}, Teodor and Peng, Yue and Schneider, Samuel
             and Lee, Cheng-Chung and Guo, Janine and Zhang, Qinghao and Jeffery, Kim},
  year    = {2026},
  eprint  = {2609.16705},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}

We acknowledge the use of the Robot Data Factory (RDF; https://agentic-robotics-lab.github.io/robot-data-factory) — training-ground infrastructure, dataset, and software developed by the Agentic Robotics Lab at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI).

Contributors