# Does Vision-Language Navigation Actually Work on Walking Robots?

A new benchmark from a team of eleven researchers says the answer is: sometimes, and the gap between simulation and the real world is measurable — but narrower than the field assumed. HumanoidVLN, posted to arXiv on August 14, 2026, presents the first physics-grounded VLN simulator built specifically for bipedal humanoids, demonstrating across four robot embodiments that the best-performing model — JanusVLN — achieves a mean success rate of 43.55% and an nDTW of 48.38. More critically, a 20-episode [sim-to-real transfer](https://humanoidintel.ai/glossary/sim-to-real-transfer) pilot on the [Unitree Robotics](https://humanoidintel.ai/companies/unitree-robotics) G1 produced a navigation error correlation of r=0.935 between simulated and physical runs, with a mean absolute position difference of 0.68m. For a field that has long questioned whether simulated VLN results carry any predictive value for legged platforms, that correlation coefficient is the single most important number in the paper.

The benchmark is built on NVIDIA Isaac Sim and tests four robots — the Unitree G1, Unitree H1, and two undisclosed internal platforms labeled Internal-A and Internal-B — spanning heights from 1.17m to 1.80m and 10 to 12 lower-body [degrees of freedom](https://humanoidintel.ai/glossary/degrees-of-freedom). It includes 933 collision-aware reference episodes, each paired with four instruction variants.

---

## Why Existing VLN Benchmarks Break on Bipedal Platforms

Every major VLN benchmark in the literature — R2R, VLN-CE, others — was designed around wheeled or abstracted agents. That design choice buries three problems that become critical the moment you deploy on a walking humanoid.

First, bipedal locomotion introduces continuous camera motion. A wheeled robot's egocentric camera is relatively stable; a biped's camera bobs and pitches with every step, distorting visual observations in ways that confuse models trained on static-platform data. Second, every humanoid platform has a different morphology: the Unitree G1 stands at 1.17m, while the H1 reaches 1.47m, changing the visual field and the physical clearances available for path planning. Third, the kinematic constraints of bipedal locomotion affect what paths are even traversable — a wheeled agent's footprint is a point; a biped's is not.

HumanoidVLN's architecture directly addresses all three. The hierarchical control stack pairs a reinforcement learning locomotion policy with interchangeable PD or MPC path trackers. The RL policy handles the stochastic physics of walking; the path tracker handles the higher-level navigation commands issued by whatever VLN model is under evaluation. This decoupling is the key architectural decision: it means the benchmark can test NaVILA, DualVLN, StreamVLN, or JanusVLN without retraining the locomotion layer for each.

---

## The Benchmark Construction: 933 Episodes, Multi-Agent Instruction Generation

The instruction dataset generation pipeline is worth examining closely because instruction quality is where navigation benchmarks typically cut corners. The authors used a dual generator-reviewer system plus a paraphraser, with human-in-the-loop verification, to produce the final episode set. Each of the 933 episodes ships with one fine-grained instruction and three coarse-grained stylistic variants — formal, natural, and casual. That four-way instruction set per episode allows researchers to directly measure how brittle VLN models are to phrasing variation, which is a genuinely underexplored failure mode in the field.

Environments come from two sources: artist-designed scenes and 3D Gaussian Splatting reconstructions. All scenes were filtered to ensure navigable areas exceed 100 square meters, preventing trivially short episodes from inflating success rates. The collision-aware path generation means reference trajectories respect the actual swept volume of each embodiment — a path valid for the compact G1 is not automatically valid for the taller, wider Internal-B platform.

---

## Performance Results: Model Rankings and Embodiment Sensitivity

Across the four tested VLN models and four robot embodiments, JanusVLN leads on both primary metrics — 43.55% mean success rate and 48.38 nDTW. The paper presents results across all model-embodiment pairings, and the interaction effects are the analytically interesting part: the same model does not rank identically across all embodiments, which confirms that VLN performance is not embodiment-agnostic. This finding has direct implications for companies evaluating which [vision-language-action model](https://humanoidintel.ai/glossary/vision-language-action-model) stack to license or develop — a leaderboard result on a wheeled platform or even on a single humanoid morphology does not transfer cleanly to a fleet of heterogeneous robots.

The 43.55% ceiling on the best model is also an honest signal about the state of the art. Navigation under physical bipedal constraints, with realistic egocentric video, is significantly harder than the clean-room conditions of prior benchmarks. The field should treat this number as a more realistic baseline than anything derived from wheeled-agent evaluations.

---

## The Sim-to-Real Pilot: What r=0.935 Actually Means

The 20-episode physical pilot using DualVLN on the Unitree G1 is the most commercially relevant section of the paper. The r=0.935 correlation between simulated and physical navigation errors means the simulator's ranking of episode difficulty closely tracks real-world difficulty — a property that is necessary (though not sufficient) for simulation to serve as a useful development proxy.

The mean absolute position difference of 0.68m and mean trajectory similarity of 0.782 nDTW (±0.188) give practitioners concrete numbers to work with when estimating how much physical validation is needed after simulation-based development. The standard deviation on nDTW (±0.188) is wide enough to matter — some episodes transfer cleanly, others do not, and the paper's analysis of that variance is where future work should focus.

The authors are candid that 20 physical episodes is a limited pilot. Skeptical readers should weight the correlation coefficient accordingly: it is encouraging, not conclusive, and the variance term deserves as much attention as the mean.

---

## Industry Trajectory: What This Means for Humanoid Navigation Development

The practical implication of HumanoidVLN is that humanoid navigation research now has a benchmark that actually penalizes you for ignoring bipedal physics. Every company building a service humanoid — whether for logistics, facility management, or consumer environments — needs a navigation stack that works on legs, not wheels. Platforms like [Unitree Robotics](https://humanoidintel.ai/companies/unitree-robotics)' G1 and H1 are already in the hands of research labs globally, which means the benchmark's hardware choices are immediately reproducible.

The extensibility design — new robots and VLN models integrate with "minimal effort," per the authors — is the right architectural choice for a benchmark that wants adoption. If HumanoidVLN becomes the standard eval for navigation on legged platforms, it will apply competitive pressure to VLN model developers to optimize for embodiment-aware performance rather than abstract graph-traversal scores.

The code, benchmark, and data are to be released upon acceptance. Until that release, independent replication is not possible — a standard academic caveat that applies here.

---

## Key Takeaways

- **JanusVLN achieves 43.55% mean success rate and 48.38 nDTW** across four humanoid embodiments — the highest of the four models tested.
- **Sim-to-real correlation of r=0.935** on 20 physical episodes with the Unitree G1 suggests HumanoidVLN's physics simulation is a credible development proxy, though pilot scale is limited.
- **933 episodes** with four instruction variants per episode; environments filtered to navigable areas exceeding 100 square meters.
- **Four robots tested**: Unitree G1, Unitree H1, and two internal platforms spanning 1.17m–1.80m and 10–12 lower-body DoF.
- **Hierarchical control stack** decouples RL locomotion from VLN model evaluation, enabling fair cross-model comparison.
- **Embodiment matters**: the same VLN model does not rank identically across all humanoid morphologies — a direct challenge to the practice of reporting single-platform results.
- Code and data to be released upon paper acceptance.

---

## Frequently Asked Questions

**What is HumanoidVLN and why does it matter?**
HumanoidVLN is a physics-grounded simulation benchmark for testing vision-language navigation models on bipedal humanoid robots, built on NVIDIA Isaac Sim. It matters because all major prior VLN benchmarks used wheeled or abstracted agents, which do not capture the locomotion-induced camera dynamics, varied morphologies, or physical path constraints that define real humanoid navigation.

**Which robots does HumanoidVLN support?**
The benchmark is demonstrated on four robots: the Unitree G1, Unitree H1, and two undisclosed internal platforms (Internal-A and Internal-B), spanning heights from 1.17m to 1.80m and 10 to 12 lower-body degrees of freedom. The architecture is designed to be extensible to additional humanoid platforms.

**What success rates do current VLN models achieve on HumanoidVLN?**
JanusVLN achieves the highest performance among the four tested models at a mean success rate of 43.55% and nDTW of 48.38. NaVILA, DualVLN, and StreamVLN were also evaluated; detailed per-model results are in the paper.

**How well does HumanoidVLN simulate real-world navigation performance?**
A 20-episode sim-to-real pilot using DualVLN on the Unitree G1 found a navigation error correlation of r=0.935 between simulation and physical execution, with a mean absolute position difference of 0.68m. This is a promising result but based on a limited pilot size.

**When will the HumanoidVLN code and dataset be publicly available?**
The authors state that code, benchmark, and data will be released upon paper acceptance. No specific date has been announced.