Closing the Action Token Gap in VLA training

TL;DR

• The problem: VLA models understand the world but can't output precise actions. This missing "Action Token" is what blocks Embodied AI from commercial deployment.

• Real-world data: Six capture methods, from master-slave arms to VR teleoperation. UFACTORY arms work with GELLO, UMI, and Pika.

• Simulation: Isaac Sim, MuJoCo, and Genesis generate labeled data at near-zero cost — but the Sim2Real gap persists.

• The takeaway: Closing the gap takes both real and simulated data — and reliable hardware underneath.

Table of Contents

• Introduction: The Missing Link in VLA Model Training

• Part 1: Real-World Data Collection Methods

• Part 2: Virtual Simulation Data Collection

• Conclusion: The Long Road to Commercial Value

Introduction:
The Missing Link in VLA Model Training

Robotics is undergoing a major paradigm shift driven by Embodied AI, particularly through Vision-Language-Action (VLA) models. In VLA, "V" stands for Vision and "L" stands for Language. Today's AI models can learn a fundamental understanding of the physical world - often referred to as a "world model" - by pre-training neural networks on massive datasets of human videos.

However, this pre-training stage lacks the "A": the Action Token. Without precise action parameters, a robot cannot reliably execute what it "sees" and "understands." To bridge this gap, engineers generally follow two approaches:

Algorithmic Extrapolation: The robot attempts to infer the required torques, currents, and joint trajectories on the fly. This approach often struggles in complex tasks.

Data Fine-Tuning/Post-Training: High-quality physical or virtual demonstration data is collected to fine-tune the model. This is currently the most practical path toward high-precision manipulation.

To build an adaptable Embodied AI strategy, roboticists use data collection across both real-world environments and virtual simulations.

Part 1:
Real-World Data Collection Methods

Collecting high-fidelity action data directly from physical environments exposes models to real-world physics, but it also requires well-integrated hardware.

1. Master-Slave Arms (Kinematic Alignment & Scaling)

By mapping human operator movements through scaled-down or kinematically aligned mechanical tracking rigs, such as GELLO, ALOHA, and U-Arm, engineers can capture direct operational trajectories.

This approach has demonstrated strong performance when paired with industrial-grade hardware. For example, UFACTORY xArm 7 can be integrated with the GELLO setup for high-quality data collection.

Technical Benchmark: GELLO's joint-by-joint mapping to UFACTORY xArm 7 captures operator intent accurately, producing high-quality trajectory data suitable for precise manipulation tasks. (Dataset: 80 episodes, trained with ACT over 500,000 steps).

View Setup Tutorial on GitHub

2. Real Robot Mounted Grippers

Another common approach involves operators holding specialized hardware grippers, such as Stanford's UMI or Pika setups, to record manipulation demonstrations in real environments. While effective, this approach remains dependent on specific robot hardware parameters and operating environments.

All UFACTORY collaborative robot arm models can be integrated with UMI and Pika frameworks for rapid data acquisition:

Technical Benchmark: Pika works across a wide range of robot arms without per-robot customization, making it a convenient option for teams that switch environments frequently. (Dataset: 150 episodes, trained with ACT over 400,000 steps).

View Setup Tutorial on GitHub

A dual xArm6 setup uses UMI teleoperation to collect bimanual demonstrations and train a LeRobot ACT policy for coordinated manipulation. (Dataset: 50 UMI teleop episodes, trained with ACT over 200,000 steps).

View Setup Tutorial on GitHub

3. Portable Handheld Grippers Detached from Real Robots

To decouple data collection from fixed robot platforms, portable handheld collectors allow human operators to move freely through diverse spaces, recording hand trajectories and opening/closing motions. These trajectories are then mapped and retargeted to the robot through software algorithms.

4. Exoskeletons & Wearable Tactile Sensors

For higher-fidelity data collection, operators use mechanical exoskeletons, specialized Head-Mounted Displays (HMDs), and wearable tactile or force-control gloves embedded with flexible pressure sensors. The key advantage is richer data: beyond capturing structural joint angles (kinematics), these systems record detailed finger interaction forces (tactile data), allowing robots to learn how gently or firmly to interact with fragile objects.

5. VR/AR Spatial Computing Teleoperation

Using spatial computing headsets such as Apple Vision Pro or Meta Quest, operators can use hand tracking and VR controllers to teleoperate physical robots in real time.

One example is our partnership with Extend Robotics, a company specializing in VR-based robotic teleoperation.

Its technology provides a more flexible alternative to mechanically coupled master-slave systems, allowing operators to control robots without requiring a dedicated physical leader arm. Extend Robotics was also selected to participate in the 2026 Google DeepMind Accelerator: Robotics program, where it is gaining access to technical mentorship and Google's robotics AI technologies.

Companies such as Tesla have also used motion-capture equipment and VR headsets to collect human demonstration data for the Optimus humanoid robot. More broadly, VR-based teleoperation can support geographically distributed data collection, enabling an operator in one location to control a robot and record demonstrations at a remote site.

6. Learning from Video & World Action Models (WAM)

The latest trend is toward World Action Models (WAM). Here, algorithms analyze large collections of first-person (egocentric) or YouTube videos. Using diffusion models, similar to OpenAI's Sora technology, the system synthesizes highly realistic videos of a virtual robot performing tasks, such as wiping a kitchen counter. This synthetic video is then used to train physical robots. Companies such as NVIDIA, Google DeepMind, and 1X Technologies are rapidly lowering technical barriers in this domain.

Part 2:
Virtual Simulation Data Collection

To provide large-scale training data for VLA models, engineers often move from physical spaces to advanced simulators such as Isaac Sim, MuJoCo (DeepMind), SAPIEN (UCSD), Meta Habitat, Genesis, and Taoyuan 2.0.

The Core Advantages of Simulation Data

Low Cost & High Concurrency: Without purchasing expensive physical fleets, teams can run thousands of parallel cloud simulations at the same time, greatly increasing data throughput at relatively low compute cost.

Absolute Safety: When a robot falls, collides with walls, or breaks delicate objects during training, there is no hardware wear and no risk to operators. The cost of failure is essentially zero.

Instant Reset & Infinite Diversity: Complex scenes can be reset almost instantly. Algorithms can dynamically generate thousands of environment variations by changing object angles, material textures, lighting, and weather conditions.

Perfect Ground Truth Labels: From the simulator's ground-truth perspective, the system can natively generate pixel-perfect 3D coordinates, depth maps, and semantic segmentation data, eliminating much of the manual labeling effort.

The Major Pitfalls: The Sim2Real Gap

The Sim2Real Gap (Lack of Realism): Physics engines struggle to perfectly reproduce micro-level friction, soft-body deformation, and hardware latency. A model that performs perfectly in simulation may still fail when facing real-world variability.

High Asset Development Costs: Achieving high fidelity requires painstaking 3D reconstruction. Simulating long-tail anomalies, such as 10,000 uniquely wrinkled clothing variations or distinct textured couches, requires significant upfront asset creation and parametric tuning.

Overfitting to Ideal Environments: Simulation data is exceptionally clean. Because models become accustomed to zero-latency, noise-free environments, real-world deployment can fail due to minor camera glare or a small grease smudge.

Conclusion:
The Long Road to Commercial Value

Addressing the Action Token bottleneck is one of the key challenges for next-generation automation. Whether through precise robotic arms collecting operational benchmarks or humanoid systems learning through WAM and simulation pipelines, Embodied AI has significant transformative potential. However, the gap between data intake and reliable, autonomous factory execution shows that the path to commercial value remains long, requiring robust engineering and reliable hardware.