Utah AI Healthcare Sandbox Expands, but Accountability Still Lags

Utah expanded its AI healthcare sandbox with three clinical pilots, despite unresolved questions about liability, federal oversight, and independent evidence. The new agreements cover acne treatment, routine prescription refills, and virtual physical therapy. Together, they move artificial intelligence closer to decisions that directly affect patient care.

The Utah AI healthcare sandbox is no longer a narrow experiment involving one prescription service. It is becoming a regulatory pathway for several healthcare companies and two major health systems. Utah also recruited six outside evaluators to examine participating systems as they operate.

That expansion creates the central conflict. Utah wants regulators to learn from real clinical use instead of waiting for perfect rules. Medical professionals argue that patients should not bear the risks of that learning process. The earlier Doctronic prescription pilot showed why those positions remain difficult to reconcile.

Utah’s Sandbox Is Moving Deeper Into Patient Care

Utah has expanded from testing one controversial service to building a broader pathway for AI-mediated healthcare.

On October 5, Utah’s Office of Artificial Intelligence Policy announced agreements involving August AI, Nolla Health, and Expect Fitness. The state also signed umbrella agreements with University of Utah Health and Intermountain Health.

August AI will test prescription renewals for routine, non-controlled medications used to manage chronic diseases. Its pilot is stage-gated, meaning the system must pass defined review stages before gaining more operational independence. Human physicians must review prescriptions before authorization during the approved phase.

Nolla Health will test visual assessments and treatment pathways for mild-to-moderate acne. Its system uses patient information and images to support an initial assessment. A specialist oversight process remains part of the pilot.

Expect Fitness will test an AI-enabled physical therapy pathway for postpartum recovery, urinary incontinence, and pelvic health concerns. Unlike prescription systems, this pilot centers on guided exercises and ongoing support. Its clinical risks are different, but incorrect advice or missed warning signs still carry consequences.

The state’s October agreements last between one and three years. Participants must disclose AI use, maintain specialized malpractice insurance, preserve patients’ existing legal remedies, and follow a 24-hour incident-response requirement.

University of Utah Health and Intermountain Health received a different type of agreement. Their umbrella arrangements create expedited routes for proposing future pilots. Each project still requires individual review, according to the state, but the institutional framework is already in place.

This distinction matters. Utah is not simply approving five standalone products. It is establishing infrastructure through which more healthcare AI projects can reach patients under negotiated rules.

A regulatory sandbox is a supervised program that temporarily adjusts how selected rules apply to a limited test. Utah says companies remain subject to other legal requirements. The state also says admission does not represent an endorsement or general product approval.

The model favors observation under controlled conditions. Regulators define safeguards, collect operating data, and use the findings to recommend permanent policy. That approach can expose problems that laboratory tests miss, including changes in user behavior and performance across patient populations.

However, healthcare is not a low-consequence testing ground. An inaccurate shopping recommendation is annoying. An incorrect medication renewal, missed contraindication, or unsuitable exercise can cause physical harm.

That is why the three new pilots cannot be evaluated only as software trials. Each one assigns some clinical work to an automated system. The important question is who remains responsible when that assignment produces the wrong result.

Utah has added safeguards since its first prescription experiment attracted criticism. Yet the expansion arrives before the state has completed its liability framework or secured a clear federal position. The sandbox is therefore scaling both its learning opportunity and its unresolved exposure.

The Earlier Prescription Pilot Still Defines the Debate

The Doctronic controversy turned an abstract policy experiment into a dispute about who can practice medicine.

Utah’s sandbox began attracting national attention after the state allowed Doctronic to handle routine prescription renewals. Patients could identify themselves, provide medical information, and request refills through an AI-driven process.

The system could check prescription history through pharmacy data and route eligible cases toward renewal. Requests presenting complications could be transferred to a clinician. Human doctors reviewed orders during the initial phase, while the company anticipated greater automation later.

The original eligible list covered 190 medications. Critics argued that “routine” renewal was an unsafe label because patient health can change between prescriptions. A previously appropriate medication can become dangerous after a new diagnosis, interaction, or change in symptoms.

The Utah Medical Licensing Board said it learned about the pilot through news coverage. Eleven board members later called for its immediate suspension, arguing that medical professionals should have reviewed the proposal before implementation.

Board chair Dr. Alan Smith highlighted the clinical information that can emerge during follow-up visits. A patient seeking a refill may have developed a condition that changes the medication’s risk. Automation must detect that change or reliably transfer the case.

The dispute forced concrete revisions. Utah removed Butalbital and Flecainide from the eligible list following scrutiny, according to state officials. Flecainide treats certain irregular heart rhythms, while Butalbital appears in some headache medications.

State AI policy director Zach Boyd described the reaction plainly: “We hit a nerve.” The remark captured more than professional resistance. The pilot crossed a boundary between supporting clinicians and assuming a task historically reserved for licensed prescribers.

The medical board has since created a working group that meets state regulators weekly. Boyd said the improved relationship followed six months of discussions and updates to the program. That is evidence of institutional adaptation, but it also shows that important clinical consultation came after launch.

The prescription debate has produced competing interpretations. Supporters see routine renewals as a constrained use case that can reduce delays and improve access. Critics see prescribing authority moving toward software before evidence and accountability have caught up.

Doctronic previously said its recommendations matched human prescription decisions in more than 99 percent of cases. That figure was based on company data and was not peer-reviewed.

A separate company-authored study examined records from 500 telehealth consultations. The reported diagnoses matched human doctors 80 percent of the time. That study also lacked independent review, according to reporting available when the expanded pilots were announced.

Those figures cannot be treated as interchangeable. Agreement on a selected refill decision differs from diagnostic performance across broader clinical encounters. Neither number, standing alone, establishes safety for continuous deployment.

The controversy also exposed the difficulty of testing a product that changes over time. Model updates, new patient groups, different medications, and altered workflow rules can all affect performance. A favorable result for one version does not automatically validate the next.

The new pilots inherit that history. Utah now requires more clinical participation and external evaluation, but its underlying strategy remains the same. The state is allowing bounded deployment to generate evidence that did not exist before deployment.

That sequence places pressure on medical boards, hospitals, and federal agencies. If they remain outside the process, state officials and vendors will define the operating rules. If they participate, they help legitimize experiments they may consider premature.

Utah AI Healthcare Sandbox Trades Speed for Unresolved Accountability

The sandbox can accelerate supervised testing, but it cannot negotiate away the question of who answers for patient harm.

Utah’s model gives participating companies regulatory relief in exchange for agreed limits, disclosures, monitoring, and data. That exchange works best when regulators can clearly define the rules being adjusted.

Liability does not fit neatly into that framework. Utah officials have concluded that the AI office lacks authority to rewrite liability law through sandbox agreements. If a patient is harmed, existing law still determines which people or organizations can be held responsible.

The immediate safeguards include specialized liability insurance and a prohibition against asking patients to waive legal claims. Those measures preserve a route for compensation. They do not settle whether responsibility belongs to the vendor, physician, health system, or another participant.

Boyd has warned that the default arrangement may place too much liability on physicians. That outcome would create a basic mismatch. A doctor could remain legally exposed for a decision generated by a system the doctor did not design, train, or fully inspect.

Requiring human approval does not automatically solve that problem. Human review can become ceremonial when clinicians face high volumes, incomplete explanations, or strong incentives to accept automated recommendations. The reviewer’s practical ability to identify an error matters more than the presence of an approval button.

The opposite arrangement also creates problems. Making a vendor directly responsible for every automated output could discourage useful experimentation. It could also obscure responsibility when a hospital configures the system, supplies incomplete data, or ignores escalation warnings.

Utah is working on legislation intended to start addressing this split. Boyd said officials hoped to introduce a liability bill during the state’s interim legislative period. He described the effort as the beginning of a multi-year process rather than a final solution.

That timeline reveals the tradeoff. The clinical pilots are operating now, while the durable accountability structure remains under development. Insurance serves as a temporary bridge, but Utah’s own regulator has questioned whether that bridge is sustainable.

The issue becomes harder as automation increases. August AI’s current pilot requires physician review before a prescription is authorized. Nolla Health includes specialist oversight. Future stages may reduce human involvement if early evidence appears favorable.

Each reduction in oversight changes the allocation of control. It should therefore trigger a new liability analysis, not merely a technical performance check.

This is also where the term “human-guided” needs precision. A person can guide system design, approve selected cases, monitor aggregate results, or intervene after an incident. Those roles provide different levels of protection.

The state says every pilot will remain under strict oversight. Yet oversight is not a single safeguard. It consists of access to decision logs, clinical review authority, incident detection, enforcement powers, and the ability to stop deployment.

Patients also need to know which entity made the consequential decision. A generic AI disclosure does not explain whether a physician reviewed the recommendation or which organization holds the relevant medical record.

The accountability problem extends beyond Utah. Vendors want products that can operate across many states. Medical practice laws, professional licensing requirements, and liability standards vary by jurisdiction.

Boyd has described the appeal of a system that could be vetted once and deployed broadly with oversight shared across jurisdictions. Such a model would reduce repetitive review and create more consistent expectations.

However, a successful national model requires an authority accepted by states, federal regulators, clinicians, and courts. Utah can test governance processes, but it cannot create that authority alone.

The state’s experiment therefore pressures lawmakers to define responsibility before automation becomes routine. It also pressures vendors to accept obligations that extend beyond technical accuracy claims.

For enterprise buyers, the lesson reaches beyond healthcare. Any organization deploying AI into consequential workflows needs a traceable record of evidence, decisions, and escalation rules. A structured knowledge workflow can help teams connect policies and source material, but documentation does not replace clinical accountability.

Independent Evaluation Is the Center of Utah’s Wager

Utah’s credibility now depends on whether independent evaluators can produce timely evidence that regulators and clinicians can inspect.

The state has recruited six organizations to assess sandbox participants: the Coalition for Health AI, Clarion AI Partners, Glacis Technologies, mpathic AI, Vega Health, and Stanford Clinical Excellence Research Center.

Their responsibilities include testing technical claims, auditing decision reporting, and evaluating clinical performance against medical standards. Future healthcare pilots must contract with an independent evaluator, according to Utah officials.

That requirement responds directly to criticism of vendor-reported results. A company has a commercial incentive to emphasize favorable outcomes. An external evaluator can challenge study design, inspect failures, and distinguish product claims from reproducible evidence.

Utah’s provisional evidence standards organize evaluation around three hypotheses. Participants should show that AI-assisted care is not inferior to traditional care, track bodily and psychological harm, and measure possible public-health benefits.

Non-inferiority is a demanding concept when applied properly. It requires a predefined comparison and an acceptable margin between the tested pathway and conventional care. A vague statement that users appeared satisfied would not meet that standard.

The safety requirement also needs more than a complaint count. Some patients may never connect a later health problem with an automated recommendation. Others may report symptoms to a physician rather than the company or state.

Public-health benefit is equally complex. Faster access can improve care, particularly where clinicians are scarce. Yet convenience metrics do not establish clinical quality. A system can complete requests quickly while missing changes that would have surfaced during a visit.

Utah expects pilot-specific evaluation plans. Mark Sendak, chief executive and co-founder of Vega Health, told Nextgov/FCW that the relevant data will depend on the application, population, outcomes, and system specifications.

That flexibility is reasonable because acne assessment, medication renewal, and pelvic health therapy involve different risks. A single benchmark would hide those differences.

Flexibility can also reduce comparability. If every company and evaluator chooses different measures, the public may struggle to determine which pilots performed well. Utah must publish enough methodological detail to make favorable conclusions meaningful.

Continuous evaluation is another essential feature. Clinical AI systems can drift when models, interfaces, patient populations, or operating rules change. Sendak expects evaluators to repeat assessments and provide updated information to the state.

That expectation is stronger than a one-time certification. It recognizes that a system’s safety profile belongs to a particular version and setting. A validated product can become unvalidated after a significant change.

Still, several practical questions remain. The public needs to know who pays the evaluators, how conflicts are disclosed, which data they can access, and whether companies can restrict publication.

Readers also need a clear distinction between an evaluator’s role and regulatory enforcement. An auditor can identify a problem. Only an authorized regulator or participating health system can impose consequences, restrict use, or suspend a pilot.

Utah has promised to share pilot data with lawmakers and the public. The value of that commitment will depend on timing and detail. Aggregate summaries released long after deployment will not support meaningful outside scrutiny.

Boyd acknowledged uncertainty when discussing the state’s expectations document. He said that “half” was probably wrong while also calling it among the most mature frameworks available.

The comment is useful because it reveals the sandbox’s premise. Utah is not claiming to possess a settled standard. It is treating the framework as a living hypothesis that deployment data will revise.

That candor does not eliminate risk. It increases the importance of transparent version histories, incident reports, and public explanations when safeguards change.

The evaluator network is thus the center of Utah’s regulatory wager. If it generates credible and comparable evidence, the sandbox can inform better policy. If it merely adds outside names to vendor-led studies, the expansion will reproduce the same evidence gap at greater scale.

Liability and Federal Oversight Remain Open Questions

State supervision cannot resolve the FDA’s role or establish a national standard for clinical AI.

Healthcare regulation in the United States divides authority between federal and state institutions. States regulate medical practice and license professionals. The Food and Drug Administration oversees many medical products, including qualifying software functions.

The boundary becomes unclear when software does more than provide administrative support. A system that assesses a condition, recommends treatment, or influences a prescription can resemble medical-device software. It can also resemble the practice of medicine traditionally regulated by states.

Doctronic’s leaders have treated their system as part of state-regulated medical practice. The company did not publicly clarify whether it sought FDA authorization, according to previous reporting.

The FDA said it had not authorized any AI chatbots when questioned about the earlier prescription program. The agency also said it supported medical innovation while keeping safety central to its decisions.

That statement did not determine whether Utah’s systems require premarket review. The federal agency had not publicly intervened in the Utah experiment when the new pilots were announced.

The silence matters because the sandbox may create operational precedent before the jurisdictional question is settled. Other states are considering their own approaches, while Texas and Wyoming have also used waivers for some AI activities.

Lawmakers in several states have considered legislation that would formally license AI medical services. This movement risks creating a patchwork in which similar systems face different evidence, disclosure, and oversight requirements.

A state-by-state path gives companies room to test narrow services. It may also encourage regulatory shopping, especially if vendors can choose jurisdictions with faster approvals or lighter requirements.

Federal involvement could bring consistent technical expectations. However, centralized premarket review can move slowly and may struggle with frequently updated models. Utah’s continuous-monitoring approach attempts to address that limitation.

The strongest framework would likely need both layers. Federal authorities could define product-level requirements, while states retain control over professional practice and local deployment. Shared data standards could reduce duplicate work without treating one pilot as universal proof.

Until that structure exists, hospitals and clinicians face uncertainty. A state agreement may permit a test, but it does not guarantee federal compliance. It also does not shield an organization from malpractice claims or reputational damage.

The Utah Medical Licensing Board’s earlier objection shows another governance risk. Subject-matter regulators can lose influence when a technology office negotiates relief around professional rules.

The board’s weekly working group is an improvement. Future pilots should involve relevant clinical authorities before agreements become operational. Consultation after public controversy is a costly way to discover missing expertise.

Security testing adds another unresolved layer. Researchers previously reported manipulating Doctronic’s public chatbot into producing harmful responses. That testing did not target the safeguarded system used in Utah’s pilot, so it does not establish that the clinical deployment had the same vulnerabilities.

However, the episode showed why environment-specific red teaming matters. A public demonstration and a protected clinical system can share model components while using different controls. Regulators must test the deployed configuration, not infer safety from either version.

The medical board dispute also challenged the idea that prescription renewals are simple administrative events. A refill can require new clinical judgment when a patient’s circumstances have changed.

That concern applies to the new pilots in different ways. An acne image may not capture an underlying condition. A physical therapy system may not recognize symptoms requiring in-person evaluation. A renewal tool may lack current records from another provider.

Escalation performance should therefore receive as much attention as automated accuracy. A cautious system that transfers uncertain cases can protect patients, although excessive transfers may erase its claimed access benefits.

Utah’s framework will be judged by how it handles these tensions, not by how many pilots it approves. The central measure is whether the system identifies limits before patients encounter them.

Three Signals Will Show Whether the Model Works

The next test is not another announcement. It is whether Utah publishes evidence, clarifies responsibility, and acts when safeguards fail.

The first signal is the proposed liability legislation. Utah officials expected to begin a multi-year legislative effort addressing responsibility for automated systems.

The initial bill should show whether lawmakers intend to hold vendors directly accountable, preserve clinician responsibility, or distribute liability according to operational control. Vague language would leave the sandbox dependent on temporary insurance requirements.

A credible proposal should also address changes between pilot stages. Liability should follow authority. If a vendor’s system acts with less physician review, the vendor’s obligations should increase accordingly.

The second signal is the quality of evaluator reports. The six external organizations need access to decision logs, incident data, system versions, and relevant clinical outcomes. Their public work should explain methods, comparison groups, limitations, and conflicts.

Utah’s reported framework calls for repeated assessment as models and patient groups change. The first published evaluations will show whether that promise creates practical transparency.

Reports that provide only broad safety conclusions would weaken Utah’s case. Detailed, versioned results with clear failure categories would strengthen it, even if they reveal problems.

The third signal is how pilot stages change after new evidence. August AI currently requires physician review before prescriptions are authorized. Nolla Health uses specialist oversight, while Expect Fitness operates within a defined physical therapy pathway.

If oversight decreases, Utah should explain what evidence justified the change. It should also publish escalation rates, significant incidents, and reasons for removing services or medications from scope.

The earlier decision to remove Butalbital and Flecainide from Doctronic’s list offers a useful precedent. It showed that scrutiny can alter a live program. The same discipline must apply across the expanded portfolio.

Health systems should watch these signals closely. University of Utah Health and Intermountain Health now have expedited pathways for future proposals. Their institutional reputation gives the sandbox more reach, but it also raises the cost of weak review.

Developers should watch the same evidence. Utah is beginning to define which logs, disclosures, safety measures, and evaluation processes clinical AI vendors need. Those requirements can influence product architecture long before a national standard emerges.

Enterprise buyers should focus on governance rather than headline accuracy claims. They need to know how systems handle uncertain inputs, incomplete records, population changes, model updates, and human escalation.

Patients should receive understandable answers to simpler questions. Was AI involved? Did a clinician review the result? What information did the system use? Who should they contact if something goes wrong?

The Utah AI healthcare sandbox deserves attention because it converts policy arguments into operating systems. Its pilots will create evidence that theoretical debates cannot. They will also expose patients to decisions made under rules that officials openly acknowledge are unfinished.

That does not make the experiment inherently reckless or responsible. The outcome depends on how quickly oversight reacts when evidence contradicts expectations.

For the next several months, ignore the number of companies joining the sandbox. Watch the liability bill, the first evaluator reports, and every change in human review. Those signals will reveal whether Utah is building accountable clinical testing or merely accelerating deployment.