# What Did ICRA 2026's REAL-I Challenge Reveal About VLA Training?

The first Real-world Embodied AI Learning (REAL-I) Challenge, held at ICRA 2026, produced a finding that should recalibrate how the humanoid industry benchmarks its AI stacks: **offline action-prediction metrics are poor proxies for closed-loop deployment success**. That single result — documented by teams from NUS-CLEAR, RCL-Lab, and Deeptouch.ai competing on a shared dual-arm humanoid platform — is worth more attention than most conference paper abstracts receive.

The challenge was structured around a fixed demonstration budget, forcing teams to maximize policy learning efficiency rather than scale their way to competence. Evaluations ran across simulation, real-robot trials, and an on-site final, all on the same dual-arm humanoid hardware. Three teams submitted competing approaches, each combining pretrained [vision-language-action models](https://humanoidintel.ai/glossary/vision-language-action-model) with task-specific [imitation learning](https://humanoidintel.ai/glossary/imitation-learning) pipelines. Their divergent strategies for data curation, staged adaptation, checkpoint selection, and action-space design provide a rare apples-to-apples comparison that the broader community has lacked.

The core research question — how can robot policies learn more effectively from a fixed demonstration budget — is one of the most commercially consequential problems in humanoid deployment today. Every operator facing a new task environment needs an answer.

---

## The Challenge Structure: Shared Hardware, Fixed Data Budget

The REAL-I Challenge's design deserves credit for methodological rigor. By constraining all competitors to the same dual-arm humanoid platform and the same demonstration budget, the organizers eliminated the hardware and data-scale variables that muddy most cross-team comparisons in the field.

Teams had to navigate three evaluation phases: simulation, real-robot evaluation, and an on-site final. This staged structure is significant. [Sim-to-real transfer](https://humanoidintel.ai/glossary/sim-to-real-transfer) performance and real-world closed-loop success are different problems, and a challenge that tests both — rather than optimizing for one — surfaces failure modes that single-environment benchmarks miss entirely.

The paper's authors — a group spanning NUS, and affiliated with the challenge's organizing structure — describe the challenge tasks, data interfaces, deployment interfaces, and competition results. The full technical comparison across the three competing teams' approaches appears in the paper itself (arXiv:2609.13679).

---

## Three Teams, Three Strategies

**NUS-CLEAR, RCL-Lab, and Deeptouch.ai** each arrived at the competition with systems built on a common architectural backbone — pretrained VLAs combined with task-specific imitation policies — but differentiated sharply on implementation choices.

The paper highlights four axes of differentiation:

1. **Data curation strategies** — how each team filtered and weighted demonstrations within the fixed budget
2. **Staged adaptation** — how pretrained VLA capabilities were preserved while adapting to the specific deployment environment
3. **Checkpoint selection** — which model snapshots were chosen for final evaluation and why
4. **Action-space design** — how each team structured the robot's output space, a choice with significant downstream effects on policy generalization

The team reports collectively surface three practical lessons that the broader humanoid community should internalize:

- **Adapting to the deployment environment while retaining prior capabilities** is the central tension in VLA fine-tuning. Teams that over-specialized on the training distribution paid a penalty in the on-site final; those that preserved the pretrained backbone's generality were better positioned for edge cases.
- **Temporal scale of demonstration quality matters.** The paper describes the importance of treating demonstration quality at "an appropriate temporal scale" — a finding that points toward the underappreciated role of temporal chunking and action consistency in [dexterous manipulation](https://humanoidintel.ai/glossary/dexterous-manipulation) tasks.
- **Suppressing errors in inactive robot components** emerged as a practical reliability concern on the dual-arm platform. When one arm is idle, uncontrolled drift or noise in that limb can degrade overall task performance — a problem that whole-body control researchers know well but that VLA training pipelines often ignore.

---

## The Offline Metrics Problem: A Direct Challenge to Industry Practice

The most consequential finding of the REAL-I Challenge is also the most inconvenient for current evaluation practice. As the paper states plainly, teams "expose the limitations of offline action-prediction metrics for forecasting closed-loop success."

This is not a novel suspicion — robotics researchers have long been skeptical of offline evals — but the REAL-I structure provides one of the cleanest empirical demonstrations to date. When you fix the hardware, fix the data budget, and run the same tasks in simulation and the real world, the gap between "predicts actions accurately offline" and "succeeds at tasks in closed loop" becomes measurable rather than anecdotal.

For commercial operators evaluating [Physical Intelligence](https://humanoidintel.ai/companies/physical-intelligence) or [Skild AI](https://humanoidintel.ai/companies/skild-ai) policy stacks, this finding has direct procurement implications. Offline benchmark scores — however convenient for vendor comparisons — need to be accompanied by closed-loop real-world evaluations before deployment decisions are made. The REAL-I results suggest the correlation between the two is weaker than the industry has assumed.

---

## Why Fixed-Data Learning Is the Right Problem to Benchmark

The challenge's framing around a fixed demonstration budget is more commercially realistic than the "more data solves everything" narrative that has dominated VLA discourse since large-scale pretraining became viable.

In practice, industrial operators deploying dual-arm humanoids on a new SKU, a new assembly variant, or a new logistics task cannot afford open-ended data collection. They need policies that generalize from tens or hundreds of demonstrations, not tens of thousands. The REAL-I Challenge's design constraint directly mirrors that operational reality.

The paper's authors propose integrating data, adaptation, evaluation, and deployment into a unified view of fixed-data robot learning. That framing is more actionable than treating each as a separate research problem — and it aligns with how the most pragmatic deployment teams at companies shipping humanoids into production are already thinking.

---

## Industry Trajectory Implications

The REAL-I Challenge is the first structured benchmark of this type on a shared dual-arm humanoid platform at a major venue. Its existence signals that the academic community is catching up to commercial deployment realities: simulation-only benchmarks are insufficient, and hardware-agnostic leaderboards obscure the variance that matters most.

Expect follow-on challenges at ICRA 2027 and CoRL 2026 to adopt similar designs — fixed hardware, fixed data, multi-phase evaluation. The teams that contributed to REAL-I (NUS-CLEAR, RCL-Lab, Deeptouch.ai) have set a methodological baseline that future benchmark organizers will need to meet or exceed.

For the VLA developer community, the paper is required reading not for its conclusions — which are directionally unsurprising — but for the empirical grounding those conclusions now have on real humanoid hardware under controlled conditions.

---

## Key Takeaways

- The first REAL-I Challenge at ICRA 2026 benchmarked VLA training on a **shared dual-arm humanoid platform** under a fixed demonstration budget
- Three teams competed: **NUS-CLEAR, RCL-Lab, and Deeptouch.ai**, each combining pretrained VLAs with task-specific imitation policies
- **Offline action-prediction metrics are unreliable predictors of closed-loop success** — the challenge's most commercially important finding
- Key training lessons: preserve prior capabilities during adaptation, attend to temporal scale of demonstration quality, suppress errors in inactive limbs
- The fixed-data framing mirrors real deployment constraints and should become standard for humanoid AI benchmarking

---

## Frequently Asked Questions

**What is the REAL-I Challenge?**
The Real-world Embodied AI Learning (REAL-I) Challenge is a competition held at ICRA 2026 that tested robot policy learning under a fixed demonstration budget. It used a shared dual-arm humanoid platform and evaluated teams across simulation, real-robot trials, and an on-site final. The first edition involved teams from NUS-CLEAR, RCL-Lab, and Deeptouch.ai.

**What is a vision-language-action model (VLA) and why does it matter for humanoids?**
A [vision-language-action model](https://humanoidintel.ai/glossary/vision-language-action-model) combines visual perception, language understanding, and motor action generation in a single neural architecture. For humanoid robots, VLAs are significant because they enable policies that can generalize across tasks described in natural language without task-specific reprogramming — critical for flexible deployment.

**Why are offline metrics insufficient for evaluating robot policies?**
Offline metrics measure how accurately a model predicts actions from held-out demonstration data, but they don't capture whether the robot succeeds when running in a closed loop with real sensor feedback. The REAL-I Challenge provided empirical evidence that the two measures are weakly correlated, validating long-standing researcher skepticism.

**What does "fixed demonstration budget" mean in robot learning?**
A fixed demonstration budget means teams cannot collect additional training data beyond a set amount. This constraint forces algorithms to maximize learning efficiency from limited examples — mirroring real-world deployment conditions where operators cannot run open-ended data collection for every new task.

**How does the REAL-I Challenge relate to sim-to-real transfer?**
The challenge's multi-phase structure — simulation evaluation followed by real-robot evaluation — directly tests [sim-to-real transfer](https://humanoidintel.ai/glossary/sim-to-real-transfer) quality. Performance gaps between simulation and real-world phases reveal where policies are overfitting to simulated dynamics rather than learning generalizable behaviors.