Physical AI Data Bottleneck: Brain Waves and Egocentric Video Solutions

The Physical AI Data Bottleneck and Brain Wave Solutions

In a warehouse in San Leandro, California, a Jenga game is pushing the boundaries of physical AI. Andrew Ceja, a robotic trainer at Encord, carefully pulls wooden blocks while wearing a headset that tracks both his vision and brain activity. This setup is part of a trial with Zander Labs, a German neuroscience startup, to measure mental states like error, intent, and surprise. The goal: create richer datasets for training humanoid and warehouse robots.

How Encord and Zander Labs Are Pioneering New Data Modalities

Encord, a company specializing in data tooling for AI models, is betting that brain wave-tagged data can improve robot performance. Lukas Gehrke, a Zander neuroscientist, explains that tracking brain activity during tasks helps model builders decide when to deploy high-effort models. Vineeth Velmurugan, Encord's head of robot learning, calls this the "bleeding edge" of solving the robotics data bottleneck.

The Role of Egocentric Video and Leader-Follower Rigs

Encord collects training data from two main sources: egocentric video from workers wearing cameras, and data from remotely operated robots. In their San Leandro facility, pilots use leader-follower rigs—paired robotic arms—to create data for tasks like pouring coffee and stacking poker chips. "Every humanoid company has asked us for these pieces," says Velmurugan.

Manufacturing Training Data for Humanoid Robots

Encord’s storage racks hold fake flowers, plastic vegetables, and kitty litter trays—props for training manipulators. At one station, pilot Sofia Infante maneuvers robotic arms to plug ethernet cables into servers, a task data centers want automated. But pincers lack human dexterity, highlighting why physical training data is critical.

From Jenga Towers to Data Center Tasks

Encord also tests muscle sensors strapped to forearms to detect electrical signals. Video alone often misses hand positions, but Velmurugan hopes to build 3D hand depictions using arm sensors. This creates a more robust understanding for models.

Muscle Sensors and Dense Annotations

Encord’s datasets include dense annotations like "right hand tightens bolt" to aid LLM-based models. Velmurugan estimates such annotations are worth 100 times more than "junky ego data" for specific tasks, though they cost 20 times more to produce.

The Economics of Physical AI Data vs. Internet Scraping

Unlike LLM builders who scrape text from the web, physical AI data must be manufactured. This changes the economics: generating physical training data is expensive. Velmurugan says it will take a dataset five times the size of YouTube's video corpus to break through, making data generation a business opportunity.

Why Data Generation Is a Business Opportunity

Encord sits between many robotics companies, spotting which data techniques gain traction industrywide. This vantage point helps them manufacture data efficiently. The dozen pilots at their facility, including Ceja and Infante, are part of a burgeoning workforce developing neural network building blocks.

Encord's Unique Industry Vantage Point

Ceja, who previously worked at Scale, enjoys the daily challenge of solving training tasks for robots. As the Jenga tower topples, he says, "It’s something new every day!"