The Next Test for Robots Is Recovering From Failure

A useful robot needs to recognize when an action has gone wrong, work out what changed, and decide whether it can safely try again. Recent research is beginning to measure and train that ability directly.

Imagine asking a robot to put an empty mug on a shelf. The gripper catches the handle at an awkward angle, nudges the mug, and leaves it lying on its side. The original instruction still makes sense. The original plan no longer does. Before the robot can finish, it must understand the new position, find another grasp, and decide whether the recovery is within its capabilities.

That small interruption exposes a demanding part of physical intelligence. A system can know what a successful task should look like while lacking the skills to recover when its own actions change the situation. For anyone hoping to leave a robot working without constant supervision, the behavior after that first mistake deserves close attention.

Google's October 13 post on Gemini 4 Argon for robotics offers a timely illustration. Its robotic-arm example uses visual checks after each turn and explicitly directs the system to observe again after a failed grasp. This is a developer demonstration, rather than an independent reliability study, but its structure is revealing: observation and correction are part of the control process.

Several recent research projects examine that process more directly. Together, they suggest that recovery deserves its own training data, its own evaluation, and a clearer place in the definition of robot autonomy.

A successful attempt leaves an important question unanswered

A conventional task test might begin with the mug upright in a known area and ask whether the robot eventually places it on the shelf. Repeat that experiment across objects and positions, and the resulting success rate tells us something useful. It still leaves uncertainty about what happens once the robot creates an unfamiliar situation halfway through the job.

The September preprint LIBERO-RECOVER focuses on this distinction. Built around failures produced during simulated robot executions, it organizes recovery into four levels: retrying an action, adapting an action, restoring an object's state, and repairing an obstructed environment. Its scope is simulation, so the findings should not be read as a direct measure of performance on physical hardware.

The practical importance of those levels is easy to see. An unsuccessful grasp that leaves the mug untouched may permit another attempt. A mug that has rotated needs a different approach. A mug lying beneath another object may require an intermediate manipulation before the original task becomes possible again. Treating all three outcomes as the same error hides the work needed to continue.

An evaluator should therefore ask both how often a robot finishes from the intended starting conditions and how it behaves after a realistic disruption. These questions support different engineering decisions. Better initial grasping may reduce interruptions, while better recovery may reduce the amount of human attention required when an interruption still occurs. A deployment needs evidence about both.

A robot changes the conditions of its next decision

The difficulty begins with a property of action: each decision helps determine what the robot will observe next. A slightly misplaced movement can push the system into a state that its training examples rarely contain. The next movement then has to be selected from that altered state, and another mistake can compound the problem.

This issue predates today's foundation models. The DAgger research presented at AISTATS 2011 addressed imitation learning in settings where future observations depend on the learner's earlier actions. Its broader lesson remains relevant: training on expert behavior alone can leave a policy poorly prepared for the situations its own behavior creates.

For a robot learning from demonstrations, the distinction changes what counts as valuable experience. A smooth recording of someone lifting the mug teaches an efficient route through the task. A recording of a failed grasp followed by a careful correction can reveal which observations indicate trouble and which actions make progress possible again. The latter may look less polished, yet answer questions the clean demonstration never raises.

That does not make every failed attempt worth imitating. Repeating a bad movement can teach the wrong behavior. Useful recovery data need to identify the failure, preserve the state it produced, and show a defensible continuation. Otherwise, a larger dataset can contain more mistakes without teaching the model how to handle them.

Explaining a failure is only the beginning of fixing it

A vision-language model might correctly describe an empty gripper without knowing where the robot should move next. It also has to decide when to intervene. Stopping too late can make the situation worse; stopping a valid movement unnecessarily can turn a successful attempt into a failure.

VLA-FixBench, published at ICML 2026, examines diagnosis and recovery across simulated tasks and physical-robot experiments. The authors' results summary reports a particularly useful caution. In its real-robot comparison, the underlying policy achieved 30% task success, the best automated vision-language-model-assisted result reached 25%, and human-expert intervention reached 65%. The larger improvement came from the human upper bound. It was not achieved by an automated supervisor.

These results come from a specific experimental setup and should not be generalized to every model or robot. They nevertheless show why adding a model that watches another model needs careful testing. Recognizing a problem, selecting a recoverable point, and producing a precise correction are separate demands.

For a practical design, this suggests evaluating the supervisor's interventions as actions with their own costs. How often does it interrupt a trajectory that would have succeeded? When it recommends a correction, is the proposed movement grounded in the robot's actual coordinates? Can it request another view when the evidence is ambiguous? A persuasive explanation provides little assurance unless the subsequent behavior improves the outcome.

Retrying and repairing require different skills

In the mug example, repeatedly issuing the original grasp command may be sensible only while the relevant conditions remain unchanged. Once the mug has toppled, the robot may need to reposition it before proceeding. That preparatory action can look like a detour, even though it is essential to the goal.

The August FLARE paper develops a retry-and-reset framework. It augments demonstrations to help a policy handle execution deviations, then uses a separate pipeline to learn object-centered reset skills for more disruptive failures. A multimodal model monitors execution and selects between the task policy and the appropriate reset behavior. Here, reset means performing a learned corrective manipulation, rather than magically restoring the physical world.

This is a useful architectural proposal because it makes the missing capability explicit. A robot trained to insert an object may also need to learn how to retrieve it after a bad insertion, clear the obstruction, or return it to a graspable position. Those operations should not be assumed to emerge automatically from knowing the intended task.

The design also creates a question for testing: how does the system decide that a repair has worked? Resuming too early can recreate the original failure. Continuing a reset after the object is already usable can introduce another one. A recovery skill therefore needs an observable completion condition, followed by a fresh check that the main task can safely continue.

Failed episodes can become targeted training material

Simulation offers a powerful way to study these decisions. A researcher can restore a saved state and try several continuations from the same point. That makes it possible to ask which moments in a failed episode contain the most useful opportunities to learn recovery.

The September preprint Kintsugi-VLA takes this approach. It estimates how often a fixed expert can complete a task from restored simulator states and uses those estimates to select recovery examples. In one simulated Franka manipulation task, its targeted data achieved 38.4% recovery success under frame-budget matching, compared with 31.7% for uniform sampling within the same recovery window. Clean-task success also fell from 76.8% for the original policy to 74.7% for the targeted-recovery variant. These are study-specific results, not evidence of a universal trade-off.

The implication is that training-data selection matters alongside volume. Collecting another hundred clean demonstrations and collecting a hundred well-chosen recovery examples may teach different capabilities. Their value depends on the failures the deployed system encounters and on whether improving recovery preserves performance during ordinary operation.

There is a useful connection to Google's Argon demonstrations here. The same October post describes reconstructing robot scenes in MuJoCo and checking simulated contact behavior against recorded evidence. Such tools could make recovery experiments easier to construct. That possibility remains an inference: the demonstrations do not establish that an automatically reconstructed scene is accurate enough for every recovery decision.

For training, the simulator must preserve the details that make the failure difficult. A scene that looks convincing while getting friction or contact wrong can reward a correction that fails on hardware. Replaying a saved state is valuable only when the resulting experiment answers the physical question that matters.

Reliability needs to include the cost of getting unstuck

For a team deciding whether to deploy a robot, eventual completion is an incomplete operational measure. Two systems might finish the same share of tasks while requiring very different amounts of time and supervision. One could recover quickly from a missed grasp; the other could spend several minutes repeating movements before requesting assistance.

A useful evaluation should report clean-task success and post-failure recovery separately, then measure the time spent recovering, the frequency of human intervention, and any additional disruption caused by a repair. It should include successful executions as well as failures, so an overactive supervisor is penalized for interrupting healthy behavior. These are proposed deployment criteria, rather than a claim that the field has adopted a common standard.

Testing should also continue after the first recovery. A robot that handles one dropped object may become confused when a second problem appears later. Running longer sequences can reveal whether repairs preserve task state and whether the system still recognizes what remains unfinished.

The practical benefit is a clearer account of human workload. A robot that quietly fixes a common, low-risk mistake could require less attention even if its first-attempt success rate barely changes. Conversely, a robot that raises an alarm for every harmless deviation may move the work from manual manipulation to constant supervision. The relevant outcome is how reliably the whole process functions, including the people who support it.

Knowing when to stop is part of recovery

Some failures should end the attempt. A broken mug, an unstable load, or a person entering the working area changes the decision. The goal of putting something on a shelf does not justify improvising through every new condition.

A sensible recovery policy needs boundaries around what it can attempt, how long it can continue, and which observations require a safe stop or human help. Those boundaries should be enforced by the surrounding system as well as represented in the model's instructions. A research result about task completion provides no general safety guarantee.

The studies discussed here leave plenty unresolved, including transfer from simulation to hardware and the precision of automated interventions. Their contribution is to make the behavior after a mistake easier to inspect. They give researchers ways to separate a robot's ability to follow the expected path from its ability to regain control when that path breaks down.

The next time a robot demonstration looks impressive, watch what happens when the object moves unexpectedly or the first grasp fails. Does the system check the result? Does it change its approach? Does it recognize when help is needed? Those moments offer a more demanding view of whether the robot is ready to keep working after someone leaves the room.