AgentCIBench Shows Office Agents Still Leak Context When They Touch Real Work

AgentCIBench tested fifteen frontier computer-use agents on context integrity and recorded a 67.9 percent average leakage rate. The full study is available in the AgentCIBench research paper. As the authors summarize, “frontier agents exhibit systematic leakage when visual or semantic proximity overrides explicit access restrictions.”

The benchmark introduced three concrete failure types that occur when agents operate across documents, inboxes, browsers, and notes. Eleven of the fifteen models leaked restricted information in more than half of the test cases. The failures appeared in isolated prompts and in full end-to-end workflows alike.

AgentCIBench measures whether an agent respects context boundaries rather than raw task success. The results show that current capability gains do not automatically produce safe behavior around sensitive material.

Three Failure Modes Drive Most Leaks

AgentCIBench isolates three patterns that produce unintended disclosure.

Visual co-location occurs when an agent pulls a prohibited item that sits next to the task target on screen - for instance, when an approved budget screenshot sits beside an unapproved salary spreadsheet, causing the agent to attach the restricted file in a hypothetical scenario where the model is asked to email the budget to finance but includes the salary data because of screen proximity. Task ambiguity over-sharing appears when vague instructions cause the agent to include personal status or unrelated files, such as auto-forwarding a private medical leave note while summarizing team availability in a case where the prompt simply says “share team status” without clarifying boundaries. Recipient mismatch happens when content reaches the wrong person because the agent misreads address fields or thread context, for example routing a draft investor update to an internal all-hands thread instead of the finance lead when the agent confuses a similar-sounding internal alias.

Each mode was tested with controlled artifacts that mimic normal office documents and communications. The scenarios avoided contrived edge cases and instead used layouts and file groupings that workers encounter daily.

Benchmark Scope and Method

The evaluation covered agents from leading commercial AI providers and popular open-source models. All models received identical task prompts and the same set of simulated workspaces. Researchers recorded every file access, message send, and context expansion during execution.

Leakage counted as any transfer of information that the prompt had labeled restricted. The study tracked both direct copies and indirect references that revealed the same data.

Results held steady when tasks were expanded into multi-step sequences. Agents that avoided leaks in short prompts often failed once additional documents entered the workspace.

Capability Gains Do Not Close the Gap

Several high-performing agents on standard benchmarks still showed leakage above 60 percent. The data indicate that success on capability suites does not predict respect for context boundaries.

Teams that rely on agent output for reports or client communications therefore face an added review step. Every suggested attachment or forwarded note must be checked against original access rules.

Practical Stakes for Teams Using Office Agents

Knowledge workers already route meeting notes, project files, and client threads through agent pipelines. A single leak of pricing data, personnel records, or strategy documents can create compliance exposure.

The benchmark does not claim that all current agents are unusable. It shows that safe deployment requires explicit boundary checks rather than reliance on general performance scores.

remio Limits Exposure by Design

remio keeps context on the local device and only surfaces information that matches explicit user queries. The five-level memory system records meetings, files, and prior decisions without automatic cross-task transmission.

When a user asks remio to generate a report or extract action items, the agent draws only from sources the user has already granted access to within that session. No background forwarding or visual co-location scanning occurs.

What Teams Should Check Before Adoption

Organizations evaluating office agents need test suites that replicate their actual document layouts and permission structures. AgentCIBench provides one open reference, yet each company must run versions tuned to its own confidentiality tiers.

Key signals to watch in the next quarter include new agent releases that publish leakage rates on similar boundary tests and enterprise tooling that adds runtime permission gates before file access.

Teams that treat context integrity as a first-class requirement will favor systems that store and retrieve data locally while requiring explicit confirmation before any cross-recipient action.