Simon Willison Ships Invisible App Testing, Closing the Agent Feedback Loop

Simon Willison released Datasette Apps 0.2a0 with two new agent tools, including one that tests generated applications inside an invisible browser frame. The update gives Datasette Agent a feedback loop that text-only coding agents often lack. It can edit an app, load the result, run JavaScript, and inspect what actually appeared.

That distinction matters because generating valid code is not the same as producing a working interface. An agent can create flawless-looking HTML while missing a runtime error, an empty chart, or a button outside the viewport. The new app_debug() tool lets the agent investigate those failures without asking the user to become its test operator.

The approach puts Simon Willison on one side of a growing divide in agentic development. Products such as GitHub Copilot increasingly connect agents to visible browser automation through tools like Playwright. Datasette Apps instead embeds a narrowly scoped testing surface inside the product, giving its agent direct access to the application it just changed.

Datasette Apps 0.2a0 Gives the Agent Two New Tools

The release changes Datasette Agent from an application editor into an editor with limited browser-based verification.

Willison announced Datasette Apps 0.2a0 on August 1, 2026. The alpha release adds app_debug() and app_list(), two tools designed around creating and editing applications through Datasette Agent.

Datasette Apps allows custom HTML applications to run inside Datasette, an open source system for exploring and publishing structured data. Datasette Agent provides the conversational layer that can inspect data and call tools supplied by plugins.

The first new tool, app_list(), returns applications that the current user has permission to edit. This sounds administrative, but it solves an important discovery problem. An agent cannot safely modify an existing application unless it knows which applications exist and which ones fall within the user’s authorization.

That permission-aware inventory makes follow-up requests more practical. A user can ask the agent to revise an application without manually locating its internal identifier. The agent can retrieve the eligible list, identify the target, and continue through the same conversation.

The second tool, app_debug(), is the more consequential addition. It opens an application in an iframe, which is an HTML element that embeds one page inside another. Datasette applies zero opacity and disables pointer events, so the embedded page remains invisible and cannot receive ordinary user interaction.

The agent then supplies JavaScript for execution inside that sandboxed frame. That code can inspect the document, query elements, retrieve text, check browser state, and measure layout dimensions.

The application still loads as a browser would load it. Its scripts execute, its styles affect layout, and its document becomes available for inspection. The agent gains evidence about runtime behavior instead of reasoning solely from stored source code.

Willison describes the feature as suitable for smoke testing, which checks whether an application’s basic functions operate without obvious failure. The tool can also answer more precise questions, including whether an element exists or how much space it occupies.

Consider an agent that creates a dashboard from a SQLite table. The generated HTML might be syntactically valid, while a mismatched column name leaves the chart empty. Source inspection alone might not expose the final symptom clearly.

With app_debug(), the agent can load that dashboard and query the rendered document. It can check whether the chart container has children, inspect visible error text, and measure whether the container has a nonzero height.

That sequence does not guarantee a good dashboard. It does provide a factual signal that the interface rendered something usable. The gap between those two standards defines the release’s central tension.

Why Simon Willison Is Moving Verification Into the Product

Simon Willison is treating browser access as part of the application’s agent interface, not as an optional external accessory.

Datasette Agent first arrived as an extensible assistant for working with SQLite data. The project’s agent introduction describes a conversational interface that supports tool-calling models from multiple providers.

Its plugin model is central to that design. A model does not receive unrestricted access to the surrounding system. Plugins expose particular capabilities, allowing the application to define what the agent can inspect or change.

Datasette Apps extends that idea from data analysis into application creation. Once an agent can generate and revise an interface, however, it inherits a difficult verification problem. The visible result exists in a browser environment, while the agent’s reasoning often remains confined to code and tool responses.

Traditional software development handles this gap through several layers. Developers run unit tests, integration tests, browser tests, and manual reviews. Each layer catches defects that the earlier layers cannot reliably detect.

Conversational app generation compresses that process. A user asks for a result rather than specifying every implementation step. If the agent cannot inspect the rendered outcome, the user becomes responsible for reporting every broken control and awkward layout.

That creates a slow loop. The agent writes code, the user opens the app, the user describes a problem, and the agent guesses at a correction. Ambiguous descriptions can generate additional errors or consume several conversational turns.

Invisible inspection shortens the loop by letting the agent collect its own diagnostic evidence. It can ask the browser concrete questions before declaring the work complete.

The underlying mechanism comes from context.browser_task(), introduced in Datasette Agent 0.4a0. It gives plugin tools a controlled way to schedule browser-side JavaScript and return results to the agent.

This matters architecturally. The browser operation belongs to the application context, where Datasette can apply its own permissions and isolation rules. The model receives a callable tool rather than broad, unexplained control over the user’s browser.

The approach reflects a broader principle in agent design. Agents become more useful when their environment exposes narrow operations with structured results. They become harder to audit when they receive general authority without clear boundaries.

Datasette Agent’s open source foundation also makes the mechanism inspectable. Developers can review the tool implementation, the iframe attributes, the JavaScript bridge, and the permission checks. That does not remove risk, but it makes the trust boundary visible.

The project remains in alpha, as indicated by the release identifiers. Alpha software can change interfaces and behavior before a stable release. The practical significance lies less in immediate mass adoption than in the feedback-loop design being tested.

For developers, this is a recognizable form of agentic engineering. The agent receives a task, changes an artifact, observes the resulting system, and revises its work. Each stage uses tools defined by the host application.

For product teams, it suggests a narrower alternative to giving a coding agent a full desktop or unrestricted browser session. A product can expose exactly the runtime evidence needed for its own workflows.

That narrower approach also makes failures easier to interpret. If app_debug() reports a missing element, the agent can connect that finding directly to the application it edited. It does not need to infer which tab, environment, or deployment the user opened.

The Invisible Iframe Is the Release’s Real Mechanism

The clever part is not that an agent runs JavaScript, but that Datasette creates a controlled observation window around its own applications.

An iframe establishes a separate browsing context inside the parent page. Developers commonly use frames for embedded videos, payment forms, previews, and isolated third-party content.

Datasette Apps uses the same browser primitive for agent testing. The application appears inside the frame with opacity: 0, making it visually transparent. The pointer-events: none rule prevents the frame from intercepting ordinary mouse or touch activity.

These presentation rules keep the debugging session out of the user’s way. They do not, by themselves, create a security boundary. That responsibility belongs to the iframe sandbox configuration, application permissions, browser policies, and the JavaScript execution bridge.

The release’s usefulness comes from combining those pieces. Datasette knows which application the user can edit. The agent can identify that application through app_list(). It can then inspect the same target with app_debug().

This creates a coherent sequence:

1. The user requests a change to an existing application.

2. The agent lists applications available for editing.

3. The agent selects the permitted target.

4. The agent changes the application.

5. The agent opens the result in the hidden frame.

6. Agent-provided JavaScript checks the rendered state.

7. The agent revises the code when checks fail.

8. The user reviews the resulting application.

Every step narrows uncertainty. The agent no longer needs the user to supply an application identifier or translate a browser symptom into prose.

Layout measurement shows why runtime access adds information. HTML source can indicate that a panel exists, but it cannot reveal its final dimensions without accounting for styles, fonts, viewport rules, and neighboring elements.

JavaScript can retrieve a rendered element’s bounding rectangle. An agent could use that result to detect a zero-height chart, overlapping card, or control positioned beyond a known viewport.

The same method can check document text. If an application displays a runtime exception, the agent can search the page for its error container. It can confirm whether expected headings, rows, or status messages appear.

The tool can also inspect attributes and computed properties. It might determine whether a button is disabled or whether an element uses an unexpected display mode. Those observations give the model grounded facts for its next edit.

This is still smoke testing rather than comprehensive quality assurance. A smoke test asks whether essential behavior works at a basic level. It does not establish accessibility, visual consistency, security, or correctness across every input.

A dimension check also requires an expected result. Learning that a panel is 312 pixels wide means little without a design constraint or comparison point. The agent needs explicit acceptance criteria to convert measurements into decisions.

The same limitation applies to page content. Finding a heading proves that the heading rendered. It does not prove that the underlying data is complete, current, or interpreted correctly.

Still, the mechanism is more credible than an agent simply announcing success after writing code. It introduces an observation step that can contradict the agent’s earlier assumptions.

That contradiction is valuable. Coding models often produce confident completion summaries even when a runtime environment would reveal immediate problems. A browser-backed check gives the host application a chance to catch those errors before the user does.

This design also avoids making screenshots the only visual signal. Screenshot analysis can identify broad appearance problems, but structured JavaScript can return exact text, counts, states, and dimensions.

Screenshots and document queries serve different purposes. A screenshot helps assess visual hierarchy and clipping. DOM inspection, meaning examination of the browser’s document structure, provides precise values that support repeatable assertions.

Datasette Apps 0.2a0 currently emphasizes the second category. That choice fits an agent that needs compact, machine-readable evidence more than another visual artifact to interpret.

Browser Agents Already Exist, but Datasette Draws a Tighter Boundary

The main contest is between product-scoped verification and general browser automation, not between Datasette and one commercial coding assistant.

Browser-capable coding agents are no longer unusual. GitHub documents a workflow where Copilot uses a Playwright server to open local pages, interact with them, and run end-to-end tests.

The Playwright integration gives Copilot access to web pages through Model Context Protocol tools. GitHub says the default cloud setup restricts that browser access to resources inside the agent’s environment.

That model offers broad testing capabilities. Playwright can navigate pages, click controls, enter text, take screenshots, and assert conditions across complete workflows.

Datasette’s mechanism is smaller. It focuses on applications hosted within Datasette and on JavaScript executed through a sandboxed iframe. The tool exists because the host product understands the artifact being edited.

The difference resembles an external testing robot compared with an application-native diagnostic port. The robot handles many websites and workflows. The diagnostic port exposes a smaller set of signals with tighter product context.

Neither route is universally better. General browser automation supports complex interactions across pages and services. It can test login sequences, navigation, forms, and behavior that depends on real pointer events.

Product-scoped debugging can offer simpler authorization. Datasette already has an application permission model, so app_list() can reflect the same access decisions used elsewhere in the product.

It can also reduce setup. Developers do not need to install a separate browser automation server before the agent can inspect a Datasette application. The relevant capability ships with the application environment.

The tradeoff is coverage. An invisible frame with disabled pointer events cannot reproduce every human interaction. JavaScript can trigger some events programmatically, but that differs from a real pointer, keyboard, or assistive technology.

Browser automation frameworks also include established testing concepts. They support selectors, waiting behavior, screenshots, traces, network interception, and assertions. Datasette’s new tool is an early product feature, not a replacement for that mature testing ecosystem.

GitHub’s own testing guidance presents Playwright as one option alongside Selenium and Cypress. That comparison places browser testing within a wider engineering practice rather than treating agent access as a new testing category.

Datasette’s contribution is the integration pattern. The agent does not merely generate a Playwright file for someone else to run. It can invoke the verification mechanism during its own editing session.

That immediacy pressures other agent-enabled products to clarify their completion standard. Does the agent stop after saving code, after passing static checks, after running tests, or after inspecting the rendered result?

Products that stop at code generation transfer more validation work to users. Products that add browser observation accept more responsibility, but they also expand their security and reliability obligations.

The Datasette approach is particularly relevant for tools that generate dashboards, internal utilities, and data interfaces. These products often operate within one controlled host environment and already maintain user permissions.

They do not necessarily need a general-purpose browser agent. They need the model to inspect the exact interface it created and return a compact diagnostic result.

That pattern can extend beyond Datasette. A report builder could expose the dimensions and content of generated charts. A workflow editor could return validation errors from its canvas. A form builder could let an agent query missing labels and invalid field states.

The common principle is product-native observability. The application exposes structured evidence about generated output, while the agent uses that evidence before requesting human approval.

Teams designing similar systems will need careful documentation. A user should know which pages the agent can load, which scripts it can execute, what data returns to the model, and how long results persist.

Without that clarity, a narrow tool can feel indistinguishable from broad browser surveillance. Product scope must be visible in both the interface and the implementation.

Invisible Testing Still Leaves Security and Quality Gaps

A hidden browser is useful precisely because it runs real code, and that same property creates the release’s largest unresolved risks.

The word “invisible” describes presentation, not harmlessness. A transparent iframe still loads an application and executes its scripts. It can perform network requests, read permitted resources, and trigger application behavior within its assigned browser context.

Sandboxing can restrict those capabilities, but the exact guarantees depend on configuration. Browser isolation is not a single switch. Permissions, origins, content security policies, credentials, and messaging channels all affect the boundary.

Agent-provided JavaScript introduces another concern. The host must prevent that code from escaping its intended frame or accessing unrelated application state. It must also control what information returns through the tool response.

Permission-aware listing helps at the selection stage. It reduces the chance that an agent will edit an application outside the user’s authority. It does not prove that every subsequent browser operation preserves the same boundary.

The release notes describe app_list() as returning applications the user may edit. Developers evaluating the tool should examine whether those checks occur again when an application is opened or changed.

Repeated authorization matters because identifiers can be copied, modified, or supplied directly. A secure design should not assume that a valid list result guarantees every later request remains authorized.

Stored application code presents another threat surface. An application may contain malicious or unexpected JavaScript. Loading that code for debugging means the agent environment must treat the target as potentially hostile.

Prompt injection also deserves attention. An application could render instructions aimed at the model, such as text telling the agent to ignore its task or disclose information.

Structured DOM inspection does not automatically protect against that attack. If page content reaches the model, the system must distinguish untrusted application data from trusted instructions.

The iframe’s sandbox can limit direct browser actions, while tool design can limit returned content. Neither defense prevents a model from being influenced by hostile text that the tool deliberately reports.

Developers should therefore treat the debugger’s output as untrusted evidence. The agent can use it to diagnose an interface, but it should not follow instructions found inside the tested application.

Reliability remains a separate issue. A smoke test can pass while important workflows fail. An agent might confirm that a chart container exists without checking whether the chart represents the correct rows.

It might also optimize for its own tests. If the model writes both the application and the verification script, it can choose an easy assertion that misses the user’s real requirement.

Independent acceptance criteria reduce that risk. The user or product should define expected outcomes before the agent performs its final check.

For example, “build a dashboard” is too vague for strong verification. A better request identifies required metrics, date filters, accessible labels, and behavior when no records match.

The agent can then test those conditions rather than inventing a convenient definition of success. This is where good task context becomes as important as browser access.

Teams can preserve such requirements inside a searchable knowledge base. That context can help agents retrieve interface standards and acceptance rules before editing an application.

Human review remains essential. GitHub’s vibe coding guide recommends opening the finished application in a normal browser to verify a realistic user experience.

That advice applies equally to Datasette Apps. Invisible tests can reduce obvious defects, but users should still inspect important interfaces, especially those that expose sensitive data or drive operational decisions.

The proper claim is modest. Datasette Apps 0.2a0 gives its agent a better debugging instrument. It does not establish that agent-generated applications are correct, secure, accessible, or ready for unattended deployment.

That distinction should shape adoption. Developers can use the tool to shorten iteration while keeping conventional tests, security review, and human acceptance in place.

What Simon Willison’s Experiment Needs to Prove Next

The next test is whether invisible debugging produces measurably better applications without expanding the agent’s authority beyond understandable limits.

Three signals will determine whether this mechanism becomes a durable part of agent-assisted development.

The first signal is evidence of repeated repair loops. The project needs examples where Datasette Agent detects a runtime or layout defect through app_debug(), edits the application, and then confirms the correction.

A polished demo matters less than reproducible cases. Tests should include empty data, malformed values, missing elements, narrow viewports, and failures that appear only after scripts execute.

If those cases become routine, the release’s central argument grows stronger. Product-native browser feedback would show that it catches errors unavailable through source inspection alone.

If the agent mostly runs shallow checks after already-correct edits, the mechanism will remain an interesting convenience. Its value depends on changing outcomes rather than merely adding another completion step.

The second signal is a clearer security contract. Documentation should explain the iframe sandbox, origin behavior, authorization checks, data returned to the model, and defenses against hostile page content.

That contract matters before broader adoption. Developers need to evaluate risk without tracing every browser message and permission decision through the source.

Clear boundaries would strengthen the case for scoped verification over a general browser agent. Ambiguous boundaries would weaken the main advantage of embedding the tool inside Datasette.

The third signal is richer testing coverage. Future releases should reveal whether the mechanism stays focused on JavaScript inspection or grows toward screenshots, interaction, accessibility checks, and reusable assertions.

Expansion would make the tool more useful, but every additional capability changes its risk profile. Clicking, typing, navigating, and submitting forms can produce real side effects.

Datasette should preserve an understandable separation between observation and action. Read-only inspection deserves different permissions from interactions that alter data or invoke external services.

Version history will also show how stable the API becomes. Both Datasette Apps 0.2a0 and Datasette Agent 0.4a0 are alpha releases, so names and behavior remain subject to change.

Developers should experiment with them in controlled environments rather than assuming production stability. The useful question is not whether an alpha feature works perfectly today.

The useful question is whether its architecture points toward a better completion standard for coding agents. Simon Willison’s answer is that agents should inspect the runtime result before claiming they finished.

That standard is difficult to dispute. The open question concerns implementation: how much browser access is enough to find defects without turning every application agent into an opaque automation system?

For Datasette users, the immediate action is straightforward. Test the agent against applications with known rendering and runtime failures. Record what app_debug() catches, what it misses, and whether its repairs survive human review.

For teams building agentic products, examine where users currently act as the missing feedback loop. If people repeatedly describe visible errors that the product can already observe, a narrow diagnostic tool may remove unnecessary work.

The lesson extends beyond browser interfaces. Agents need access to consequences, not only instructions and source files. They become more dependable when a host application exposes those consequences through bounded, auditable tools.

Will Simon Willison’s invisible iframe become a model for that pattern, or remain a clever Datasette-specific experiment? The next several releases should answer that through repair evidence, explicit security boundaries, and stronger validation workflows.