SecRespond Finds 23 Frontier AI Models Miss Silent Intrusions

SecRespond reached Google News with a blunt result: none of 23 frontier models completed detection and remediation on any tested compromised host. The agents handled visible alerts much better than silent evidence, exposing a gap between AI-assisted triage and autonomous incident response.

Researchers from Alibaba Group submitted the SecRespond paper on July 29, 2026. They tested models from several major families through OpenCode, an agent harness that lets models inspect files and use command-line tools.

The test begins after an attacker has already succeeded. That detail creates the conflict at the center of the study. AI agents can follow an alert, but a security operations center needs investigators that also find threats nobody flagged.

SecRespond therefore challenges a common automation promise. A model that summarizes alerts can reduce analyst workload, yet that does not make it an independent incident responder. The benchmark found the difference inside disk artifacts, persistence mechanisms, incomplete cleanup steps, and unverified remediation plans.

What the SecRespond Benchmark Actually Changed

SecRespond moves the evaluation target from alert interpretation to investigation of an already compromised machine.

Many cybersecurity tests begin before compromise. They ask a model to identify a vulnerability, solve a capture-the-flag challenge, classify malware, or reason over selected security logs. Those tasks measure useful abilities, but they reduce the uncertainty that defines a real breach.

SecRespond starts later. Each agent receives a frozen forensic disk snapshot from a compromised cloud host. It also receives synthetic outputs resembling alerts, vulnerability scans, and security baseline checks from a host protection product.

A forensic disk snapshot is a preserved copy of a system’s files and artifacts at a specific moment. It can contain evidence that alerts never mentioned, including modified startup files, backdoor accounts, erased logs, scheduled jobs, or malicious binaries.

The agent must inspect that material and reconstruct what happened. It then produces reports covering intrusions, vulnerabilities, baseline risks, and remediation. The task also requires a progress file, creating a record of the investigation rather than accepting one polished final answer.

The benchmark contains 10 cyber ranges, meaning isolated environments built to reproduce security incidents. Those ranges cover four initial entry-point types, 21 techniques from the MITRE ATT&CK catalog, and five operating systems.

The researchers translated those environments into 52 capability items and 280 detailed checkpoints. The checkpoints ask whether an agent found concrete evidence, attributed it correctly, recommended suitable action, and covered necessary verification.

Detection and remediation planning receive separate scores. Detection has a maximum of three points per applicable checkpoint. Planning has a maximum of two, while checkpoints that do not apply to one dimension are excluded from that aggregate.

That separation matters because finding a malicious file does not answer what responders should do next. A safe response can require isolating a host, preserving evidence, terminating processes, removing persistence, rotating credentials, blocking infrastructure, restoring services, and verifying recovery.

The public SecRespond dataset includes the task prompts, evaluation materials, checklists, synthetic security outputs, and forensic archives. Its release makes the central claim testable by teams beyond the original authors.

SecRespond also defines a stricter boundary for AI incident response. An agent does not receive full credit because it probably checked something. Its report must state the finding and cite evidence that satisfies the relevant checklist.

That rule turns vague security language into measurable performance. “Investigate suspicious activity” does not equal identifying a specific process, file, account, endpoint, or persistence path. “Patch the server” does not equal a complete, sequenced, and verified recovery plan.

Google News coverage focused on the headline failure across 23 models. The deeper change is methodological. SecRespond asks whether an agent can pursue leads that were never handed to it, then connect those discoveries to a defensible cleanup process.

Why Google News Attention Matters for AI Security Buyers

The benchmark pressures vendors and security leaders to distinguish alert assistance from autonomous incident response.

AI already helps security operations centers summarize alerts, enrich indicators, search documentation, draft queries, and prepare case notes. These workflows remain valuable because analysts often confront fragmented evidence and repetitive administrative work.

However, SecRespond measures a higher level of independence. An autonomous responder must decide where to investigate, recognize missing evidence, test competing explanations, and continue after the most obvious alert has been resolved.

The benchmark’s central result shows why that distinction matters. Across all 23 evaluated models, no agent achieved complete detection and remediation on even one cyber range.

The best overall model in the reported experiment was Claude Opus 4.7. It reached a 79.0 percent average range-level checkpoint score for detection and 65.7 percent for planning.

The paper also reports a 72.4 percent average when combining those dimensions for the leading model. That performance still left malicious artifacts untouched and remediation incomplete, particularly in ranges with longer and broader attack chains.

Other leading results included Claude Opus 4.6 at 78.2 percent detection and 58.0 percent planning. GLM-5.1 reached 76.3 percent and 59.2 percent, while Qwen3.7 Plus reached 75.6 percent and 58.8 percent.

Those figures should not become a general ranking of the underlying models. They describe one agent harness, one benchmark version, one task design, and a specific evaluation process.

The results instead expose a shared failure pattern. Models found evidence connected to existing alerts more reliably than evidence requiring an unprompted search of the disk.

That pattern creates pressure for security vendors using broad labels such as “AI analyst” or “autonomous SOC.” Buyers need to ask which portions of the response lifecycle the system actually performs without a human-created lead.

A product might summarize an endpoint alert accurately yet overlook a second persistence mechanism. It might recommend deleting a malicious binary while failing to terminate its process, remove its loader, rotate exposed credentials, or verify service recovery.

Each omitted step changes the operational outcome. An attacker can return through an untouched account, scheduled task, webshell, service, registry entry, or shell hook. A technically correct first action can therefore create a false sense of containment.

Security leaders also need to separate investigation quality from report quality. Models often produce fluent explanations, but SecRespond scores whether those explanations contain the required evidence and remediation details.

This is a familiar problem for knowledge-intensive work. A confident narrative can hide incomplete retrieval. Teams building a searchable knowledge base face a related requirement: conclusions must remain traceable to source material.

The benchmark makes that traceability concrete for incident response. An agent must show which artifact supports each conclusion and which action addresses each identified condition.

Google News visibility can push this distinction beyond benchmark researchers. Procurement teams, CISOs, managed security providers, and internal audit groups now have a public example of why “handles alerts” and “handles incidents” are not equivalent claims.

The Real Blind Spot Is Unguided Investigation

The models’ weakest behavior appears when an incident leaves no obvious alert pointing toward the next artifact.

SecRespond groups performance into five capability areas. These cover intrusion entities, persistence mechanisms, baseline risks, vulnerability risks, and overall investigation and response quality.

An intrusion entity is a concrete malicious object, such as a process, file, network endpoint, or tampered artifact. Models performed best in this category because these objects often aligned with visible security signals.

Across models, average detection reached 75.4 percent for intrusion entities. Several leading systems performed substantially better, including Qwen3.7 Plus at 88.4 percent and Claude Opus 4.6 at 86.0 percent.

Persistence mechanisms produced a different result. Persistence refers to changes that allow attacker access to survive a reboot or an initial cleanup. Examples include scheduled tasks, services, shell startup hooks, webshells, account backdoors, and Windows Management Instrumentation subscriptions.

Average detection fell to 58.8 percent for persistence. The drop matters because persistence is precisely what responders must find before declaring a host clean.

The benchmark does not show that models lack all forensic reasoning. They can connect an alert with a relevant process or file and often describe the immediate threat correctly. The failure arrives when the investigation must expand beyond that starting point.

Consider a compromised web server. An alert might identify a malicious process or an outbound connection. Following that signal can reveal one executable, but a complete investigation must ask how the attacker entered, what credentials were exposed, and what survives termination.

The responder may also need to inspect startup scripts, service definitions, cron entries, user accounts, command histories, application directories, and altered logs. No single alert necessarily identifies those locations.

This creates a search problem with uncertain boundaries. The agent must decide which hypotheses deserve testing and how long to continue. It must recognize that the absence of one artifact does not eliminate other persistence routes.

Current language-model agents often optimize around the evidence already present in context. Alerts create high-salience anchors, so the agent can spend its budget explaining those anchors instead of looking for unmentioned evidence.

Longer attack chains amplify that weakness. Each additional technique introduces another branch, artifact type, timestamp, account, or service that the model must correlate.

The paper found that performance declined as attacks became longer and broader. That result fits the operational challenge: incident response is not one classification decision but a sequence of linked judgments under incomplete information.

A separate 2026 threat-hunting benchmark reported a related problem. Five frontier models searched raw Windows event logs from 26 attack campaigns, and the best model found only a small fraction of malicious events.

The two studies test different workflows, so their scores are not directly comparable. However, both suggest that unguided search remains harder than reasoning over preselected evidence.

This is the benchmark’s core reversal. Agents appear most capable where conventional security tools have already reduced uncertainty. They become less reliable where human investigators add the most value by questioning what the alert did not reveal.

A security team can still use AI productively within this boundary. The model can summarize evidence, propose hypotheses, draft queries, compare artifacts, and maintain an investigation timeline.

The unsafe leap is treating those capabilities as proof that the model has searched the whole incident. SecRespond shows that an articulate answer can coexist with undiscovered persistence and an incomplete account of attacker activity.

Detection Scores Hide a Larger Remediation Gap

Finding more evidence did not translate into equally complete cleanup plans, making remediation the benchmark’s second major failure.

Every evaluated model scored higher on detection than on planning. For GPT-5.5, the reported gap reached 34.7 percentage points.

The researchers attribute this pattern to agents applying an obvious first fix while omitting remaining actions. That behavior resembles checklist truncation: once the central malicious object receives a response, the model behaves as though the incident is resolved.

Real remediation rarely ends with one deletion or configuration change. A responder must consider dependencies, evidence preservation, business impact, service continuity, credential exposure, and the attacker’s alternative access paths.

SecRespond’s planning score asks whether an action is correct and complete. It also examines verification and side effects when the relevant checkpoint requires them.

Verification is not ceremonial. A plan to remove a scheduled task should confirm that the task no longer exists and that its payload cannot launch through another mechanism.

A plan to block an attacker’s address should address both inbound and outbound traffic where appropriate. It should also avoid implying that an address block removes malware, stolen credentials, or persistence already present on the host.

Standardized configuration problems proved easier. Claude Opus 4.7 reached 74.8 percent in planning for baseline risks and 72.6 percent for vulnerability risks.

Those tasks often map to familiar actions, such as strengthening a configuration or updating affected software. The agent can retrieve a recognizable remediation pattern and apply it to the finding.

Investigation and response quality remained much weaker. Average planning performance in that category reached only 31.8 percent.

This category covers work that depends on synthesis rather than one known fix. It includes attack-chain reconstruction, evidence quality, honesty about uncertainty, completeness, verification, and awareness of operational impact.

The strongest detection result in that category reached 75.5 percent. Almost every model remained below 50 percent on planning, according to the paper.

These results undermine a simple scale-up strategy. Giving a model more alerts does not automatically create a complete response plan. More visible findings can instead produce more disconnected recommendations.

A credible plan needs ordering. Teams may isolate a machine before changing it, preserve volatile evidence before terminating processes, and rotate credentials after determining the exposure scope.

They also need rollback and service considerations. Removing a compromised component without understanding dependencies can interrupt production or destroy evidence needed for attribution.

SecRespond evaluates written plans rather than live remediation on production systems. That limits what the benchmark can establish, but it also keeps the safety question visible.

If a model cannot consistently describe complete and verified remediation in a controlled environment, organizations have little basis for granting it unrestricted authority on a live host.

The benchmark therefore supports a narrower operating model. AI can suggest actions, organize evidence, and highlight missing fields, while human responders retain approval for containment and recovery steps.

This arrangement is not a rejection of SOC automation. It is a response to the specific asymmetry in the data. The systems were better at identifying known objects than ensuring every consequence received safe treatment.

Security teams should reflect that asymmetry in access controls. Read-only forensic access carries different risk from permission to kill processes, delete files, disable accounts, or change network policy.

An agent that misses a hidden artifact produces an incomplete report. An agent that acts on that incomplete report can disrupt recovery while leaving the attacker’s alternate route intact.

What the Numbers Do Not Prove

SecRespond is strong evidence of a shared limitation, but it is not a final verdict on every model or production SOC configuration.

The paper is an arXiv preprint rather than a completed peer-review outcome. Its authors include researchers from Tongyi Lab and Alibaba Cloud, and the benchmark evaluates models through one representative harness.

The choice of OpenCode helps standardize tool use across systems. It also means the results measure a model-and-harness combination, not an abstract model capability detached from prompting, tools, context management, and execution policy.

Different scaffolding can change performance. An incident-response agent could use a mandatory investigation checklist, specialized forensic utilities, retrieval over internal procedures, multiple cooperating agents, or deterministic validation scripts.

SecRespond remains useful because those improvements can be tested against the same ranges. However, the published numbers should not be treated as permanent limits for each model family.

The evaluation also uses an LLM-as-a-judge process, meaning language models grade generated reports against detailed checklists. Three proprietary judges were used independently to reduce reliance on one grader.

Those judges were Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4 Pro. Multiple judges reduce individual bias, but they do not remove every calibration problem.

A grader might interpret incomplete phrasing differently from a human forensic specialist. It might also reward explicit report language without fully resolving whether the underlying investigation process was sound.

The scoring instructions try to control that risk. Judges must cite evidence and award credit only for content explicitly present in the reports.

The benchmark’s 280 checkpoints provide additional structure. Yet every checklist encodes choices about which artifacts, response steps, and qualities deserve weight.

The 10 ranges are diverse enough to reveal repeated behavior. They do not cover every operating system, cloud architecture, identity platform, endpoint product, or attacker technique.

The environments are also controlled. Researchers provisioned and compromised the hosts for the benchmark, then sanitized credentials and personal data.

This design enables reproducibility and avoids exposing production information. It cannot fully reproduce the noise, incomplete telemetry, organizational constraints, and business dependencies of a live enterprise incident.

One result also illustrates how safety behavior can affect benchmark coverage. Claude Opus 4.7 refused the npm-worm task, so the paper omitted that model from the range’s detailed checkpoint table.

A refusal can lower operational usefulness during a legitimate defensive investigation. It can also reflect a provider’s effort to prevent dual-use assistance from drifting into harmful guidance.

SecRespond does not settle that policy tradeoff. It shows that safe deployment needs task definitions that distinguish authorized forensic work from offensive instructions.

The benchmark authors state that the released evidence comes from isolated environments and does not contain runnable exploit chains. The public materials are intended for defensive research.

That restriction matters when interpreting claims about “real-world” response. The ranges recreate end-to-end compromises over real network protocols, but the released package contains sanitized forensic evidence rather than active attack tooling.

There is also no independent field study showing how SecRespond scores translate into analyst time saved, incident severity reduced, or containment speed improved. Those outcomes require evaluations inside operational teams.

For buyers, the correct reading is therefore measured. The benchmark strongly challenges unsupported claims of autonomous incident response. It does not show that AI assistance has no value inside a human-led SOC.

It also does not establish that one named model will remain ahead across future versions. The reported Claude series improved across releases, while progress across other families was not universal.

The meaningful unit of evaluation is the deployed system. That includes the model, tools, prompts, permissions, retrieval sources, review gates, logging, and recovery procedures.

Three Signals to Watch After the SecRespond Google News Cycle

The next test is whether vendors improve unguided discovery, remediation verification, and reproducible production evaluation.

The first signal is independent reproduction. Researchers and security vendors can run the public benchmark repository with other harnesses, prompts, tools, and model versions.

Reproduction will show whether the silent-intrusion gap survives changes to scaffolding. If specialized forensic agents still miss unalerted persistence, the paper’s central judgment becomes stronger.

If deterministic search procedures produce large gains, the lesson changes slightly. The bottleneck would sit less in model knowledge and more in investigation design, tool routing, and enforced coverage.

That would still weaken claims about general-purpose autonomous agents. It would also provide a clearer engineering path for safer systems.

The second signal is whether vendors publish separate detection and remediation results. A single “incident response accuracy” number can hide the planning gap that SecRespond surfaced.

Useful evaluations should state what the system found, what it missed, which action it proposed, and how it verified completion. They should also report refusals, tool failures, and cases requiring human intervention.

Watch for testing on persistence mechanisms specifically. Improvements on alert-linked malware matter, but they do not address the benchmark’s main blind spot.

Also watch whether plans cover cleanup breadth. A stronger agent should handle processes, files, accounts, scheduled execution, network controls, credential rotation, service recovery, and post-remediation validation when applicable.

The third signal is evidence from supervised SOC deployments. Vendors need to show how their agents perform with real telemetry, internal procedures, access controls, and analyst approval gates.

The strongest operational evidence will not be a polished case study alone. It will include miss rates, escalation rates, unsupported claims, correction frequency, and the percentage of recommendations analysts approve unchanged.

A credible deployment should preserve an audit trail. Reviewers need to trace conclusions to artifacts and determine which searches the agent completed before it stopped.

Organizations should also test permission boundaries. Read-only investigation, recommended actions, and autonomous execution represent three different risk levels.

SecRespond supports adoption at the first two levels while placing a heavy burden of proof on the third. Its results do not justify handing broad containment powers to a model that has not demonstrated comprehensive discovery.

The Google News headline will fade, but the benchmark leaves security teams with a durable procurement question: what does the agent find when no alert tells it where to look?

Ask vendors to answer that question with reproducible evidence. Then ask how the system verifies every cleanup step and signals uncertainty to a human responder.

Those answers will reveal whether AI SOC products are becoming investigators or remaining fast assistants around existing detections. For now, SecRespond places all 23 tested models on the assistant side of that line.