DuoCook Benchmark: Leading AI Model Completes Just 26.2% of Orders

The DuoCook benchmark puts AI in charge of a virtual kitchen, where its strongest tested model completed an average of 26.2% of orders. That result challenges a practical assumption behind autonomous assistants: understanding a task should translate into organizing the work needed to finish it.

The October 22 preprint describes an Overcooked-inspired environment, rather than a direct test of the commercial game. A vision-language model, an AI system that processes images and text, controls both chefs. The central conflict is between selecting plausible individual actions and completing an interdependent workload.

Earlier systems, including MindAgent and Collab-Overcooked, already made virtual kitchens a testing ground for AI agent coordination. DuoCook adds another reason to examine execution closely. For developers and buyers, the useful question concerns how reliably an assistant turns its decisions into finished work.

What the DuoCook benchmark actually measures

The result concerns coordinated task execution under a particular interface, so its meaning depends on how the test works.

The researchers evaluated four models across 20 maps in five categories. GPT-6-Astra led the reported completion results; the other models averaged 8.8% to 11.2%. The DuoCook benchmark provides structured information and navigation assistance, with simulation paused during model inference.

Those details help define the claim. An AI cooking benchmark can test different combinations of perception, movement, scheduling, and cooperation. Changing that combination changes what a low score tells readers. A system struggling to recognize an ingredient presents a different engineering problem from one repeatedly assigning work in the wrong order.

Consider a hypothetical kitchen where a chef prepares vegetables while another manages a pot. Each assignment sounds reasonable in isolation. However, the vegetables might belong to an order that cannot finish before its deadline. Meanwhile, the pot might occupy equipment needed for an easier meal.

The important decision involves the relationship between those activities. A useful controller must consider what an action enables, what it delays, and whether its output remains useful. Recognizing the correct ingredient supplies only part of that decision.

This is why the completion figure deserves attention beyond its entertainment value. Finished orders provide an outcome that readers can understand without interpreting an elaborate explanation. The kitchen either receives the required sequence of work or it does not.

Still, readers should resist translating 26.2% into a universal AI success rate. The reported number summarizes performance on the evaluated suite. It does not describe every possible kitchen, every agent architecture, or every version of the tested models.

Nor does it establish how a human would perform under the same interface. A familiar-looking game can encourage an intuitive comparison that the experimental design does not necessarily support. Human experience with the commercial title does not automatically provide a matched baseline.

The useful reading is narrower and more concrete. Under the researchers’ setup, substantial room remains between issuing decisions and completing the assigned workload. That gap gives subsequent experiments something specific to investigate.

It also changes what an impressive demonstration should establish. A compelling explanation of the next move is insufficient evidence of dependable execution. The evaluation needs to follow the chain through its final outcome, including the consequences of earlier choices.

For readers watching autonomous software, this distinction is the story’s starting point. The kitchen gives a visible form to a broader evaluation problem: assessing whether locally sensible behavior produces the result a user requested.

Why unfinished orders put agent developers under pressure

The practical pressure falls on teams promising that an assistant can manage dependencies without constant human intervention.

An assistant can provide useful suggestions while leaving coordination to its user. Once it accepts responsibility for execution, the standard changes. It must preserve the user’s objective through intermediate decisions, interruptions, and competing demands.

Imagine an assistant preparing a project update. It needs to gather source material, identify changes, resolve conflicting statements, and produce a document. Each stage can appear successful while the overall assignment remains incomplete.

A collected document might be outdated. A correct summary might omit a decision that changes the recommendation. A polished draft might arrive after the meeting where someone needed it. These are hypothetical examples, but they clarify the difference between component quality and workflow quality.

The kitchen comparison becomes useful at that level. Its relevance comes from dependencies, rather than any claim that cooking and office work have identical difficulty. Both can require a controller to decide what must happen before another action becomes worthwhile.

For developers, this suggests a more demanding acceptance test. A task should remain under evaluation after the model produces a plausible instruction. The test should continue until the environment confirms that the requested outcome exists.

That approach also changes how teams diagnose failures. An incomplete task might reflect a mistaken observation, an invalid command, a poor priority, or a missed opportunity to recover. Treating all these cases as generic reasoning failure makes improvement harder.

Product teams face a related design choice about responsibility. They need to specify when an assistant acts independently, when it verifies progress, and when it asks for help. Those decisions shape whether an error remains contained or affects later work.

A visible checkpoint can make a workflow easier to supervise. For example, an assistant could confirm that required inputs exist before starting a dependent action. That proposal is an engineering response to the coordination problem, rather than a result established by this study.

Buyers should likewise ask what a demonstration leaves for a person to manage. Someone might quietly repair missing inputs, reset failed steps, or choose the next objective. Those interventions can be valuable, but they belong in the performance account.

The pressure is therefore immediate for evaluation design, even if the broader commercial implications remain uncertain. Teams can define completion criteria and record intervention requirements before making claims about general autonomy.

Knowledge workers have a simpler question to ask: what happens when the original plan stops matching reality? An assistant’s handling of that moment can matter more than the confidence of its initial answer.

AI agent coordination becomes a product requirement when users delegate outcomes. The relevant evidence must show how the system manages unfinished work, rather than merely how convincingly it describes the work ahead.

A convincing next action does not guarantee a finished task

The central reversal is that an action can be correct on its own and still undermine the overall objective.

Earlier research provides useful context for that distinction. The MindAgent framework explored language models as coordinators in CuisineWorld, a virtual cooking environment. Its authors reported scheduling capabilities and improvements from demonstrations, prompting, and feedback.

That work offers a constructive perspective on kitchen evaluations. A weak result does not require the conclusion that coordination is inaccessible to language models. It can instead motivate experiments with the instructions, feedback, and control structure surrounding them.

However, the possibility of improvement does not settle whether a particular system is dependable. A proposed method needs evaluation against completed tasks under clearly stated conditions. Better explanations and better outcomes remain separate claims.

The Collab-Overcooked study examines another part of this problem. Its authors reported strong goal interpretation alongside shortcomings in active collaboration and continuous adaptation. The benchmark uses process-oriented evaluation to examine behavior beyond the final result.

These findings help establish the main comparison: identifying what should happen versus sustaining the behavior that makes it happen. DuoCook belongs in that discussion, although its centralized controller differs from systems involving independently controlled teammates.

A hypothetical example makes the distinction concrete. Suppose a controller correctly identifies that soup needs another ingredient. Sending a chef to retrieve it appears helpful. Yet the trip may delay a different order whose remaining preparation is already complete.

The controller then faces a scheduling decision, meaning a choice about the timing and allocation of work. The best answer depends on the surrounding workload. Recipe knowledge alone cannot resolve it.

The same issue appears when an action changes another worker’s options. Reserving equipment can simplify one task while preventing progress elsewhere. Assigning both workers to the same objective can accelerate preparation while leaving delivery unattended.

These examples illustrate why evaluating isolated actions can miss the central difficulty. A decision’s value depends partly on the future actions it makes possible. That relationship becomes harder to assess when the environment changes during execution.

There is also a recovery problem. Once an earlier assumption fails, continuing the original plan can waste additional effort. A capable controller needs a way to recognize that its intended sequence no longer serves the objective.

For engineering teams, this suggests comparing alternative approaches under the same workload. A general model might be paired with explicit task tracking, resource reservations, or a separate scheduling component. Those are proposals to test, not fixes established by the paper.

The comparison should preserve the user-facing outcome. If an added component improves completion while requiring fewer interventions, that matters. If it only produces more detailed intermediate reasoning, its practical value remains unresolved.

This is where the research creates a useful challenge for agent builders. The goal is to connect judgment with execution across the entire task, including the points where reasonable choices conflict.

Added memory introduces another decision problem

Memory needs to improve the controller’s decisions; retaining more information does not establish that benefit by itself.

The DuoCook experiment also compares a shared notebook with a condition that retains recent action history. Reported score effects differ across models: Claude and Qwen improve on average, while GPT and Gemini decline. Mean model-call expenditure rises for every model.

That result complicates a familiar design instinct. When an assistant loses track of work, adding memory sounds like a direct response. Yet a memory system introduces decisions about what to record, what to retrieve, and when to revise it.

Consider a hypothetical note saying that a workstation is occupied. The observation might have been accurate when written. If the situation changes, the same note can become an obstacle unless the controller checks or updates it.

The problem involves the relationship between an observation and the current environment. A remembered statement needs enough context for the assistant to decide whether it still applies. Otherwise, persistence can preserve a mistake or extend the life of outdated information.

Developers should therefore distinguish storage from verified state. Storage holds information. Verified state represents what the system has checked about the present task. A design can contain extensive records while still lacking a reliable account of current progress.

This distinction matters for ordinary knowledge workflows too. A meeting note might capture a decision accurately, while a later message changes it. An assistant preparing a recommendation needs to recognize that relationship instead of treating both statements as equally current.

A searchable knowledge base can help people locate source material for review. However, retrieving relevant documents and autonomously coordinating a live workflow require different evidence. The kitchen experiment does not evaluate remio or establish conclusions about its features.

For agent designers, the research suggests a focused comparison. Test the same workflow with and without the proposed memory feature. Keep the other conditions as consistent as practical, then examine completed outcomes and the additional work required.

The evaluation should also inspect cases where memory changes the outcome. A summary average can reveal a difference without explaining it. Reading the sequence of observations and actions can help identify whether a remembered statement supported a useful decision.

Alternative explanations need attention as well. Additional context might change how the model interprets instructions or prioritizes information. A performance difference alone does not identify which mechanism produced it.

The appropriate conclusion is conditional: memory is a component whose value depends on its implementation and use. A notebook can help a controller maintain continuity, but its usefulness must survive comparison against a simpler alternative.

For buyers, that means asking what an advertised memory feature actually improves. Remembering a preference, retrieving an old document, and tracking a changing task are distinct capabilities. Each deserves an evaluation matched to its purpose.

Why the rankings remain preliminary

The findings justify further investigation, while the study design limits conclusions about stable rankings and workplace reliability.

Each model received one run per map and memory condition. The authors also say supporting episode logs and analysis scripts are absent from the current manuscript artifacts. Those limitations restrict repeatability assessment and detailed diagnosis of the reported failures.

A matched test can still provide useful observations. Giving systems comparable starting conditions helps make the reported differences interpretable. However, a single observation cannot reveal how consistently a model would behave when the same scenario runs again.

For a buyer choosing between products, consistency is part of the decision. A system that sometimes completes a task and sometimes stalls presents a different supervision burden from one with predictable limits. A leaderboard average does not fully describe that burden.

There is a separate question about what the environment measures. The OvercookedV2 research argues that failures in earlier coordination settings can reflect inadequate coverage of encountered situations during training. Its authors introduced changes intended to create more demanding coordination scenarios.

That is a methodological perspective, rather than a direct criticism of DuoCook. It reminds readers that a visible failure does not automatically reveal its underlying cause. Different experimental interventions can produce different explanations for behavior that initially looks similar.

For example, a controller might struggle because it misunderstands the interface. Another might understand the interface but choose an ineffective schedule. A third might need a familiar example before it can apply otherwise available knowledge.

These hypothetical explanations call for different remedies. They also show why a low completion number should begin an investigation rather than finish one. Strong claims about the cause need experiments designed to distinguish alternatives.

Generalization creates another boundary. Generalization means performing successfully beyond the situations used to develop or evaluate a system. A result from a fixed kitchen suite cannot, by itself, establish performance across unrelated work.

The Overcooked Generalisation Challenge specifically evaluates cooperation with unfamiliar partners in unfamiliar environments. Its authors report difficulties with that combined challenge. It therefore examines a dimension that differs from centralized control of both characters.

These research directions should complement one another without being collapsed into a single ranking. A controller managing shared resources and an agent adapting to an independent teammate face related but distinct demands. Their scores need interpretation within their respective setups.

The model names also require care. GPT-6-Astra, Claude, Gemini, and Qwen identify the evaluated systems in this paper’s comparison. The results do not establish a universal ordering of their capabilities across applications.

The strongest defensible conclusion remains specific: the tested setup exposes substantial unfinished work and raises questions worth reproducing. Broader claims about general intelligence, human replacement, or inevitable commercial failure would exceed the available evidence.

What would make the next results more convincing

The next useful developments are repeatable outcomes, clearer explanations of failures, and successful tests beyond the original kitchen suite.

Over the next few months, the first signal to watch is repeated evaluation under matched conditions. Further runs should show whether the leading model retains its advantage and how widely outcomes vary. Stable results would strengthen the current comparison; unstable results would weaken confidence in the ranking.

The same applies to the memory findings. If the reported benefits and declines recur, developers would have firmer grounds for investigating model-specific behavior. If they reverse frequently, the original averages would deserve a more tentative interpretation.

Repeated evaluation would also help readers distinguish an impressive episode from a dependable system. A successful demonstration is useful evidence that a behavior can occur. It supplies less information about whether a user can expect that behavior consistently.

The second signal is the availability of detailed execution records and reproducible analysis. A trajectory, the recorded sequence of observations and actions, would help researchers examine where progress stops. Supporting artifacts would also allow others to check how reported summaries were calculated.

Those records should make competing explanations testable. Did an incorrect observation precede the failure? Did a valid instruction arrive at the wrong time? Did a remembered statement conflict with the current situation?

Answers would help determine whether the next experiment should change perception, scheduling, feedback, or memory. They would also reduce the temptation to explain every failure through a broad label such as poor reasoning.

The third signal is improvement on unfamiliar environments and comparable practical workflows. A stronger result would preserve completion when layouts, dependencies, or resource constraints change. That would strengthen the case that an improvement extends beyond the original examples.

A gain limited to familiar maps would still be useful, but it would support a narrower claim. Developers should distinguish adapting a system to a known test from improving its ability to handle new situations.

Practical comparisons should follow the same logic. A workplace pilot needs its own task definition, completion criteria, and intervention record. The cooking result cannot supply a substitute for that evaluation, whether the pilot concerns research, document preparation, or software operations.

For users, the sensible response is to make the finish line explicit. Specify what must exist when a delegated task ends and which conditions require review. That gives both the assistant and its evaluator a clearer target.

When reviewing the next DuoCook benchmark result, follow the evidence from the headline number to the completed work. Check whether the comparison uses matched conditions, whether repeated runs support the ranking, and whether detailed records explain the improvement. Then ask how the system behaves when the setting changes. Apply those questions to an assistant demonstration before handing it a longer workflow. A convincing next action can be useful, but dependable delegation requires a sequence that reaches the requested outcome. What evidence would show that an assistant can finish your task, recover when its plan fails, and leave you with less coordination work?