# Does Any Foundation Robot Policy Actually Work Alongside Humans?

**A new benchmark with 650+ evaluation episodes shows that GR00T, pi0.5, and ACT — three of the most capable foundation robot policies available today — all fail substantially at human-robot collaboration, even when they perform well on isolated manipulation tasks.** HRIBench, introduced by researchers Chang Liu, Jiawei Zhang, Tao Zhang, Ye Wang, Hongyu Zhou, and Qin Jin, is a diagnostic benchmark built around executable interaction scenarios that explicitly model three agent roles: Instructor, Collaborator, and Intruder. Across 13 role-conditioned tasks, the benchmark tests intent understanding, temporal synchronization, protocol compliance, and safety under human intervention — dimensions that existing [Vision-Language-Action Model](https://humanoidintel.ai/glossary/vision-language-action-model) benchmarks largely ignore. The most concrete result in the paper: fine-tuning on HRIBench data improved GR00T N1.5's physical-task success rate from 0.10 to 0.43 in a real-world adaptation study — a more than fourfold gain driven entirely by richer interaction-structured training signal.

This matters for the industry because humanoid deployment in warehouses, factories, and care settings is fundamentally a human-robot collaboration problem. Picking a bottle in isolation is not the same as picking a bottle while a human co-worker is reaching across the shared workspace with changing intent.

---

## What HRIBench Actually Measures

Existing VLA benchmarks — including those used to evaluate Nvidia's GR00T family and [Physical Intelligence (π)](https://humanoidintel.ai/companies/physical-intelligence)'s pi0 series — treat tasks as self-contained: the robot executes, success or failure is binary, and there is no other agent with dynamic, potentially conflicting goals.

HRIBench's central architectural decision is to represent collaborative tasks as **structured scenario scripts** that encode agent roles, temporal dependencies, coordination constraints, and distributions over human behavior. This lets evaluators generate diverse interaction trajectories and scene variations programmatically, which is how the benchmark produces its 650+ evaluation episodes across 13 tasks without requiring an equivalent number of costly real-world demonstrations.

The three roles are meaningfully distinct:

- **Instructor**: the human communicates intent and the robot must parse and act on it — testing intent communication and language grounding.
- **Collaborator**: the human and robot share a task simultaneously — testing temporal synchronization and joint coordination.
- **Intruder**: a human enters or disrupts the robot's workspace — testing robustness and safety under unplanned intervention.

This tripartite structure is analytically useful because it isolates *where* a policy breaks down. A model could be strong on Instructor tasks (intent parsing) and catastrophic on Intruder tasks (safety compliance), and current benchmarks would not surface that distinction.

Beyond binary pass/fail, HRIBench introduces interaction-centric metrics covering synchronization, responsiveness, protocol compliance, and safety. The paper does not detail the exact mathematical formulations of these metrics in the abstract, but the stated goal — interpretable, fine-grained scoring — directly addresses a known weakness in how the field currently reports VLA performance.

---

## The GR00T, pi0.5, and ACT Results

The paper evaluates adapted versions of three policies: Nvidia's GR00T, [Physical Intelligence (π)](https://humanoidintel.ai/companies/physical-intelligence)'s pi0.5, and ACT (Action Chunking with Transformers), under a unified evaluation protocol.

The headline finding is stark: **all three struggle substantially in collaborative settings despite strong manipulation ability.** The authors identify temporal coordination and intent-aware behavior as the primary failure modes. This is analytically coherent — these models were trained on demonstration data where a human operator was *absent from the scene*, so the policy has no learned representation of another agent's intent trajectory or the need to yield, wait, or synchronize.

The sim-to-real result is the most commercially significant number in the paper. Using simulation data generated by HRIBench's scenario-script infrastructure, fine-tuning GR00T N1.5 improved its physical-task success rate from 0.10 to 0.43. That is not a marginal improvement — it is the difference between a robot that almost never succeeds at a collaborative physical task and one that succeeds nearly half the time. The [Sim-to-Real Transfer](https://humanoidintel.ai/glossary/sim-to-real-transfer) pipeline here is doing real work, and the implication is that the *structure* of the training data — interaction-scripted, role-conditioned — matters as much as its volume.

**Skeptical note:** A 0.43 success rate on collaborative physical tasks is still low by any commercial deployment standard. The paper is correctly framed as demonstrating benchmark *value* rather than claiming a deployment-ready solution. The more interesting question the paper raises — but does not yet answer — is whether the performance ceiling for these architectures under interaction-centric evaluation is fundamentally limited by their training paradigm, or whether it is simply a data problem that more HRIBench-structured demonstrations can close.

---

## Why the Industry Should Care Now

The timing of HRIBench is not incidental. The humanoid field is at an inflection point where hardware capability is increasingly ahead of behavioral capability in collaborative settings. Robots from virtually every major humanoid developer can now physically perform many manipulation tasks in controlled conditions. What they cannot reliably do is perform those tasks *next to* or *with* a human whose behavior is dynamic and only partially observable.

This is the exact failure mode that enterprise customers — automotive OEMs running pilot programs, logistics operators, electronics manufacturers — encounter when they move from demo to deployment. A robot that scores well on isolated [Dexterous Manipulation](https://humanoidintel.ai/glossary/dexterous-manipulation) benchmarks but fails to coordinate temporally with a line worker is not a deployable robot; it is a safety liability.

HRIBench provides the field with something it has lacked: a principled, reproducible way to measure the collaboration gap and track progress against it. The 13-task, 650-episode structure is modest enough that it is accessible to academic labs, but the scenario-script abstraction is extensible, and the real-world GR00T N1.5 results establish that simulation-generated interaction data transfers.

For VLA developers, the practical implication is direct: if your evaluation suite does not include role-conditioned interaction tasks with temporal synchronization metrics, you are not measuring what enterprise deployment actually requires. The gap between manipulation benchmarks and HRIBench-style interaction benchmarks is not a gap in robot hardware — it is a gap in how the field has defined the problem.

---

## Key Takeaways

- **HRIBench** is a new diagnostic benchmark for human-robot collaboration built on structured scenario scripts with three agent roles: Instructor, Collaborator, and Intruder.
- The benchmark contains **13 role-conditioned tasks** and **over 650 evaluation episodes** generated from diverse interaction trajectories and scene variations.
- **GR00T, pi0.5, and ACT** all fail substantially on collaborative tasks despite strong manipulation performance, with temporal coordination and intent-aware behavior as the primary bottlenecks.
- Fine-tuning on HRIBench data improved **GR00T N1.5's physical-task success rate from 0.10 to 0.43** in a real-world adaptation study — a fourfold increase.
- HRIBench introduces interpretable metrics spanning synchronization, responsiveness, protocol compliance, and safety, moving beyond binary success/failure scoring.
- The result validates that **interaction-structured simulation data** meaningfully improves real-world collaborative performance via sim-to-real transfer.
- The broader implication: manipulation benchmarks are insufficient proxies for deployment readiness in any context involving human co-workers.

---

## Frequently Asked Questions

**What is HRIBench and what does it test?**
HRIBench is a diagnostic benchmark for human-robot collaboration introduced by researchers including Chang Liu and Jiawei Zhang. It evaluates robot policies on three interaction roles — Instructor (intent communication), Collaborator (joint coordination), and Intruder (safety under human intervention) — across 13 tasks and over 650 evaluation episodes, using metrics for synchronization, responsiveness, protocol compliance, and safety.

**Which robot AI models were evaluated in HRIBench?**
The benchmark evaluates adapted policies based on Nvidia's GR00T, Physical Intelligence's pi0.5, and ACT (Action Chunking with Transformers), all under a unified evaluation protocol. All three models showed substantial limitations in collaborative settings.

**How much did HRIBench improve GR00T's real-world performance?**
In a real-world adaptation study reported in the paper, simulation data generated by HRIBench improved GR00T N1.5's physical-task success rate from 0.10 to 0.43 after fine-tuning — more than a fourfold improvement.

**Why do current VLA models fail at human-robot collaboration?**
According to the HRIBench authors, the primary failure modes are temporal coordination and intent-aware behavior. Current foundation policies are trained on demonstrations without a co-present human agent, so they have no learned representation for synchronizing with another agent's dynamic intent.

**How is HRIBench different from existing manipulation benchmarks?**
Existing VLA benchmarks evaluate robots on isolated manipulation tasks with binary success/failure scoring and no modeled human agent. HRIBench explicitly encodes agent roles, temporal dependencies, coordination constraints, and human behavior distributions, and scores policies on interaction-centric dimensions rather than task completion alone.