A protocol for testing robot policies where they have to perform. Connect data quality, open benchmarks and controlled site trials to evidence a model team can inspect and an operator can act on.
Is the demonstration observable, synchronized and consistently labeled?
Can a frozen model complete a reproducible benchmark task?
Does it work on the tested robot, at the tested site, with assistance accounted for?
A benchmark becomes useful when the task, conditions and scoring rule stay fixed. This public design is the starting point for scoping an engagement, with thresholds and operating limits agreed for each robot and site.
Name the workflow, robot embodiment, model checkpoint and task population. Register success predicates, time limit, permitted assistance, stop rules and a trial budget before seeing outcomes.
Keep calibration and tuning outside the held-out test set. Group by site, recording session and object instance; do not scatter adjacent frames or near-duplicate trajectories across splits.
Record start states, resets, camera calibration, payload and control frequency. Randomize or counterbalance model order within matched site, task and shift blocks. Blind reviewers to model identity where practical.
Log every initiated attempt, including timeouts, interventions and aborts. Timestamp observations, actions, model latency and control ownership against synchronized video. Link each result to its episode manifest.
Apply the registered success predicate at the required final state. Report subgoal progress separately. Have human reviewers adjudicate ambiguous cases and audit model-judge outputs against a held-out human-labeled set.
Report per-task and per-condition denominators, uncertainty, failure categories and exclusions. Release the protocol version, checkpoint identifiers and permitted artifacts so another team can reproduce the comparison.
Work near live electrical equipment requires a separately authorized task and site-specific operating limits.
Correct tote reaches the marked bay within the time budget; payload remains intact and the route stays inside the authorized zone.
A useful training demonstration needs more than a video. It needs timing, action boundaries, arm identity, object attributes and an auditable relationship between the labels and what happened.
Two kinds of evidence appear in the supplied figure. Rubric percentages describe a score relative to its maximum; probabilities describe the judge’s belief.
Compute coverage from the union of labeled intervals over a declared eligible duration, per arm or per episode. Measure overlaps on the same track; simultaneous actions across arms can be valid. Check boundaries with a stated frame tolerance and validate required object attributes.
Audit a held-out subset with two human reviewers and an adjudication rule. Retain disagreements, missing data and rubric versions. Pin model-judge inputs and outputs, then test agreement and calibration against the human reference.
The example’s completeness score of 91% and its 98.5% temporal coverage measure different things. Coverage asks how much time is labeled; completeness is a rubric judgment about the annotation.
A model’s probability or reported confidence is not a measured robot success frequency. Using a judge for review does not replace controlled autonomous trials. JEV rubric type reference
The supplied evaluation view pairs task scores with an explicit evidence-quality label. Its motion estimate is sourced from a 2D vision-language model, with unknown scale and a sparse image path measured in frame axes. Those coordinates do not establish metric travel, joint motion or contact force. Keep estimated signals separate from calibrated measurements, and show missing evidence alongside the score.
A robot can finish a task while an operator does the hard part. Report autonomous completion, assistance, progress and operational burden as separate measures.
| Measure | Definition | Report |
|---|---|---|
| Unassisted task success | Attempts meeting the final-state predicate within the time budget, with no human takeover, divided by valid initiated attempts after pre-registered infrastructure exclusions. Publish all initiated and excluded counts. | k / N, with interval |
| Assisted completion | Attempts that finish after operator input or physical help. Record assistance type, count and duration; keep these separate from autonomous success. | k assisted / N |
| Subgoal completion | Completed required predicates divided by the registered predicate count. A partial score does not count as binary task success. | Per-episode fraction |
| Cycle time & latency | Start-to-terminal-state time, with timeouts retained. Show median and p95 for successful runs, the timeout fraction, and policy latency separately. | Seconds / milliseconds |
| Interventions & recovery | Share of attempts requiring a takeover; time under human control. Recovery is autonomous completion after a defined recoverable disturbance, on a separately declared subset. | Episodes, seconds, subset N |
| Operational incidents | Unexpected contacts, dropped items, boundary violations and stops. Define events and severity before trials; report exposure hours and affected attempts. | Count / exposure |
A policy-caused stop, timeout or human takeover remains a failed unassisted attempt. Assisted completion can be recorded separately. Infrastructure-invalid attempts may be excluded only under rules fixed before testing; preserve their logs and disclose counts, reasons and replacement trials. Include reset time and downtime when reporting tasks per wall-clock hour.
Separate zero-shot performance from performance after site adaptation. Disclose the adaptation budget and known training exposure; label unknown overlap as unknown. Do not average away a weak site or platform.
| Shift | Hold out | Report separately |
|---|---|---|
| Object | Held-out instances, geometry, texture and payload | Known category / new instance; new category separately |
| Environment | New site or layout; cameras, lighting and clutter | Within-site changes / held-out sites |
| Instruction | Paraphrases, new referents and task compositions | Same intent / new composition |
| Temporal | A later session, shift or deployment day | Session-disjoint and time-disjoint results |
| Embodiment | A different platform, gripper or action interface | Per-platform results; adaptation budget disclosed |
An 80% result from ten attempts and an 80% result from a hundred attempts carry different uncertainty. Publish the numerator, denominator and uncertainty alongside every success percentage.
95% Wilson interval: 71.1% to 86.7%
This is a local mathematical calculator, not a CosmicBrain result. Assumes independent binary trials. For repeated trials within shared sites or sessions, use an analysis that accounts for clustering.
For comparisons, use matched task blocks and an analysis that respects task, session and site structure. Report macro averages across tasks as well as pooled episode counts, declare weights, and retain slice-level results. Choose sample size and stopping rules before observing results.
Open benchmarks make experiments repeatable and expose distinct failure modes. These references inform the protocol; benchmark adapters and test scope are selected per engagement.
Distributed, double-blind comparisons across institutions and real environments on the DROID platform.
Task suites vary spatial relationships, objects, goals and longer-horizon behavior.
Kitchen tasks with controlled scene and object diversity, plus training and evaluation splits.
Goal-predicate completion and efficiency metrics distinguish progress from complete task execution.
Distinguishes success at any point in a rollout from success at its end.
Open policy evaluation tooling records outcomes and rollout videos for inspection.
Real-world demonstration data, policy training code and a reproducible robot setup.
Primary project and documentation sources reviewed October 2, 2026. Pin a release or commit, task IDs, initial-state distribution, controller, observation mode and episode horizon. Scores from different tasks, embodiments, simulators or time budgets are not directly comparable. Simulation and live-site results belong in separately labeled reports.
Give the model team enough detail to debug, and the site operator enough context to judge the tested operating envelope. A result should trace back to a task, a run and an outcome.
Templates are proposed specifications. Empty fields are left unmeasured. The example file contains values from the supplied figures, not raw episode records.
Scope a live-site evaluation around the workflow, platform and conditions that matter to you.