Cognita Imaging FDA Contract Tests Whether LLM Juries Can Safely Grade Radiology AI

Cognita Imaging has received a $1.29 million FDA contract to test LLM juries across approximately one million patient exams. The 18-month project targets a difficult regulatory problem: evaluating AI systems that generate complete radiology reports without assigning every case to human reviewers.

The Cognita Imaging FDA contract does not authorize a medical device or endorse the company’s technology. Instead, it funds regulatory science research into whether several large language models can evaluate reports consistently. Radiologists will examine clinically important disagreements rather than manually reviewing the entire dataset.

That distinction carries the central tension. Automated evaluation makes million-case studies conceivable, but it introduces another fallible AI layer between the report generator and the human expert. The project must therefore evaluate both the reporting model and the models judging its work.

Cognita plans to deliver its software code, guidance for building LLM juries, validation-cohort comparisons, and radiologist-reviewed discrepancy studies. The FDA will also receive reports covering the project’s results and limitations.

The outcome matters beyond Cognita. Medical AI developers increasingly want to generate complete clinical text, while regulators must decide what evidence can support those broader systems. The contract will test whether automation can expand that evidence without weakening its clinical meaning.

What the Cognita Imaging FDA Contract Actually Funds

The contract funds an evaluation method, not permission to sell an autonomous radiology product.

The FDA awarded Cognita Imaging a firm-fixed-price contract valued at $1,294,042. Federal contract records identify it as award 75F40126C00035, covering AI-based virtual subject-matter-expert agents for radiology report assessment.

The contract took effect on June 22, 2026, and runs through December 21, 2027. Cognita announced the project publicly on September 16. It received the award through the FDA’s Broad Agency Announcement program for regulatory science research.

The project is called “Virtual Subject Matter Expert Agents for Radiology Report Evaluation.” Cognita will develop and validate a framework in which multiple LLMs assess human-written and AI-generated radiology reports.

An LLM jury is a group of language models that independently evaluate the same output. Their combined judgments can expose disagreements that a single automated evaluator might hide.

The method will then be applied to approximately one million patient exams from a large U.S. cohort. The research will examine performance across patient populations, care settings, imaging equipment, diseases, and rare findings.

That scale is central to the proposal. Conventional reader studies ask radiologists to review model outputs against reference cases. These studies remain essential, but expert time limits the number and variety of cases they can cover.

Complete reports also create a larger evaluation surface than narrowly focused detection systems. A triage model might identify a suspected blood clot or collapsed lung. Its output is constrained and can be compared with a defined reference.

A generated report can contain many findings, qualifications, comparisons, and recommendations. It can omit an abnormality, invent one, misstate its severity, or describe the correct finding ambiguously.

Cognita’s approach asks several LLMs to classify these differences at scale. Radiologists will step in when the jury identifies a clinically important disagreement.

According to the company’s contract announcement, reviewers will determine whether the problem originated in the generated report, the LLM jury, or the original radiologist’s report. That three-way adjudication is critical because a reference report is not automatically perfect.

The research will be led by Cognita co-founder Akshay Chaudhari. He is also an associate professor of radiology and biomedical data science at Stanford University. Cognita CEO and co-founder Louis Blankemeier will serve as co-investigator.

Cognita is a wholly owned subsidiary of Mosaic Clinical Technologies, which is owned by Radiology Partners. That relationship gives the research team access to clinical infrastructure and radiology expertise across varied practice environments.

Radiology Partners says its broader network includes more than 4,000 radiologists serving over 3,400 healthcare facilities. It reports interpreting more than 55 million imaging studies annually.

Those company figures describe potential operating context, not the contract’s validation results. The one-million-exam cohort still needs to be defined, tested, and reported with enough detail for others to assess its representativeness.

The contractual deliverables make the project more substantial than a private benchmark. Cognita must provide code, implementation guidance, large-versus-small cohort analyses, adjudicated discrepancy studies, and documented limitations.

However, the FDA has not promised to adopt the framework as a regulatory standard. The research can inform future policy without becoming a required pathway for every generative radiology product.

That boundary matters because contract funding and product authorization serve different purposes. Cognita’s separate chest X-ray model has received breakthrough device designation, but it has not received marketing authorization.

The original reporting also confirms that the research contract and breakthrough designation are distinct initiatives. Neither one substitutes for evidence supporting a future device submission.

Why Full Reports Break the Old Evaluation Model

Generative radiology systems turn evaluation from a narrow accuracy test into an open-ended clinical comparison problem.

Most authorized radiology AI has historically performed bounded tasks. A system might segment an organ, flag an urgent scan, measure a structure, or detect a specified abnormality.

These functions can still require complex testing. However, developers can usually define the expected output, reference standard, and target population before running a study.

Full-report generation is different. The model receives images and possibly clinical context, then produces a narrative covering everything it considers relevant.

One examination might contain several abnormalities with different clinical priorities. Another might be normal but include postoperative changes that must not be mistaken for disease.

A generated report can be factually accurate yet poorly prioritized. It can mention the correct finding but place it in language that changes its urgency. It can also produce a plausible sentence unsupported by the image.

Hallucinations describe information that appears credible but lacks support from the underlying evidence. Omissions occur when a report fails to include a finding that should be documented.

Both errors resist simple text matching. Two correct radiologists can describe the same image differently, while two similar reports can carry meaningfully different clinical implications.

Traditional language metrics often reward shared words or phrases. They do not reliably distinguish a harmless wording change from an error that changes patient management.

Cognita previously helped develop GREEN, short for Generative Radiology Report Evaluation and Error Notation. GREEN uses a language model to identify, categorize, and explain clinically significant differences between generated and reference reports.

The peer-reviewed GREEN paper reported stronger alignment with expert assessments than several conventional text metrics. It also produced explanations and error counts rather than only returning a similarity score.

The FDA contract expands the basic idea in two ways. It uses several model judges instead of relying on one, and it brings radiologists into cases where the automated judgments matter clinically.

A jury can reduce dependence on one model’s preferences, but only if its members contribute meaningfully different judgments. Several models can share training sources, reasoning failures, or sensitivity to the same phrasing.

Consensus is therefore not the same as correctness. Five models making the same mistake can create greater confidence without adding valid evidence.

The proposed discrepancy process addresses part of that risk. When judges disagree on a clinically significant issue, a radiologist can examine the reports and determine what failed.

Yet agreement also deserves testing. If every jury member overlooks the same omission, disagreement-based escalation will not send that case to a human reviewer.

The project therefore needs a separately sampled human-review set. That set can estimate errors within both unanimous and divided jury decisions.

Large and small cohort comparisons will also matter. The company must show what additional failure patterns become visible when evaluation expands beyond a few hundred selected cases.

A million examinations should include more rare conditions and unusual combinations. It can also reveal performance changes associated with scanners, institutions, patient populations, and clinical environments.

However, scale alone does not guarantee coverage. A large dataset can still underrepresent important populations, uncommon equipment, or settings outside its source network.

That is why cohort composition should remain part of the final analysis. Readers will need more than the headline number to judge whether the method reflects real U.S. practice.

The FDA Needs Evidence That Matches Generative AI

The contract arrives as the FDA is deciding how existing device rules should apply to systems with broader and less predictable outputs.

On August 18, the FDA released a discussion paper about generative AI-enabled medical devices. The agency requested feedback on risk assessment, premarket evaluation, postmarket monitoring, and lifecycle management.

The timing makes the Cognita Imaging FDA contract especially relevant. The agency is not only reviewing individual products. It is also examining whether its evaluation methods fit devices that create new content.

The FDA’s discussion paper raises questions about measuring performance when outputs are variable or open-ended. It also asks how developers should detect confabulations, assess human oversight, and monitor products after deployment.

An AI-generated radiology report touches each issue. Its output can vary between runs, contain several clinical claims, and influence decisions even when a radiologist signs the final report.

Premarket studies must establish safety and effectiveness before authorization. However, limited test sets can miss rare combinations that later appear in clinical use.

Postmarket monitoring offers broader evidence, but reviewing every deployed report would impose a large operational burden. Automated judges might help identify cases that deserve expert attention.

That creates a potential bridge between premarket validation and real-world surveillance. The same evaluation framework could test a model before launch and monitor selected outputs after deployment.

The framework would still need clear thresholds. A discrepancy that changes treatment deserves a different response from a stylistic disagreement or a harmless wording variation.

It also needs traceable records. Regulators and hospitals must be able to reconstruct why an automated judge assigned an error category and why a case reached a human reviewer.

Model updates add another challenge. If the report generator changes, its error distribution can change. If a jury member changes, the evaluation system’s judgments can also shift.

This creates two moving systems instead of one. A validated report generator could appear better or worse simply because its evaluators were updated.

Version control, locked test sets, and repeatable prompts will therefore be necessary. Developers must document both the product under review and every judge used to assess it.

The FDA already maintains an AI device list covering authorized products across medical specialties. Radiology continues to represent a large share of listed devices.

The agency notes that it wants to identify devices containing LLM-based functionality more clearly in future list updates. That signals a need for more precise regulatory visibility as generative features enter medical products.

The established radiology AI market also raises competitive pressure. Companies such as Aidoc, Viz.ai, Lunit, Annalise.ai, Rad AI, and DeepHealth have pursued different combinations of detection, triage, workflow, and reporting tools.

A scalable report-evaluation method would not automatically favor Cognita. Competitors could use similar evidence strategies if the FDA publishes broadly applicable guidance.

The pressure instead falls on every developer making broad performance claims from limited reader studies. Regulators and buyers can ask whether a few hundred curated examples reveal the model’s behavior across real clinical variation.

Hospitals may raise the same question during procurement. They need evidence that a model works across local scanners, protocols, patient groups, and reporting conventions.

If the contract produces a credible methodology, larger validation cohorts could become an expected part of the evidence package. That would increase testing demands while making broad generative claims harder to support with selective cases.

The LLM Jury Must Also Stand Trial

The core tradeoff is direct: LLM judges offer scale, but their own errors can distort the evidence regulators receive.

LLMs do not evaluate reports from a neutral position. Their judgments depend on training data, prompts, model versions, evaluation order, and the wording of each candidate report.

A judge might prefer longer answers even when additional text creates clinical risk. It might favor a familiar writing style or assign higher scores to outputs resembling its own responses.

A model can also produce inconsistent decisions across repeated evaluations. This creates a reproducibility problem when developers need stable measurements for regulatory submissions.

Healthcare raises the stakes because subtle differences matter. An evaluator that performs well on general medical questions might still mishandle uncertainty, laterality, temporal comparisons, or imaging-specific terminology.

A recent healthcare review of LLM judge systems describes automated evaluation as promising but emphasizes the need for validation, transparency, and human oversight.

The Cognita project is structured to investigate several of these concerns. Multiple judges can reveal instability, while radiologist adjudication can test whether disagreement points to clinically relevant uncertainty.

However, a jury architecture does not remove bias automatically. Its members must be selected and tested for complementary performance rather than assembled by model count.

The research should disclose whether the judges come from different model families. It should also explain whether any judge shares technology, data, or optimization methods with the report generator.

Self-preference is a particular concern. A model family can favor outputs that resemble its own structure, verbosity, or reasoning patterns.

That problem becomes more serious if a developer evaluates its generator using related models. Apparently strong scores might reflect stylistic affinity rather than clinical correctness.

Prompt sensitivity also needs examination. Small changes to instructions can alter how a judge weighs omissions, uncertainty, and clinical significance.

A regulatory framework cannot depend on a prompt that works only under one undocumented configuration. The prompt, temperature, model version, and decision rules should be reproducible.

The jury must also distinguish report quality from image truth. A text-only judge can compare two reports, but it cannot independently confirm which description matches the scan.

If the original report misses a finding, the generated report might be penalized for correctly adding it. The reverse can occur when both texts agree on a mistake.

Cognita’s proposed radiologist review can resolve some of these conflicts by returning to the images. The final methodology should state when image review is mandatory rather than optional.

A fully image-aware evaluator presents its own complications. Vision-language models can inspect images and text, but then their visual interpretation must also be validated.

This creates a layered assurance problem. Each added model expands capability while introducing another component that can fail silently.

Clinically important disagreement is also not a fixed category. A minor wording difference for one patient can become significant when it affects follow-up, urgency, or treatment.

The framework will need explicit error categories and severity definitions. Radiologists should show meaningful agreement when applying those rules.

Inter-reader disagreement should not be treated as noise to eliminate. It can reveal genuine ambiguity in the image, report, or clinical context.

The most useful outcome may therefore be calibrated uncertainty rather than a single universal score. A system that reliably identifies uncertain cases can support oversight without pretending to replace expert judgment.

Cost and latency will also influence adoption. Running several proprietary models across one million reports can require significant computing resources and repeated processing.

Those economics do not invalidate the approach, but they affect whether smaller developers can reproduce it. A regulatory method becomes less useful if only large organizations can afford the evaluation infrastructure.

Cognita’s code and implementation guidance could help. The final deliverables should still explain hardware requirements, model access, processing time, and sensitivity to closed model updates.

The contract’s success should therefore be judged by transparency and repeatability, not only headline agreement with radiologists. A credible negative result would also provide useful regulatory evidence.

If LLM juries fail on particular error classes, the FDA can restrict their role accordingly. The research does not need to prove universal reliability to improve evaluation practice.

What Success Across One Million Exams Would Mean

A successful result would make human review more targeted, not make radiologists optional.

The project’s central operating model is selective escalation. Automated judges process the broad dataset, while specialists focus on cases with meaningful uncertainty or disagreement.

That design can increase the reach of limited expert time. It can also concentrate reviewers on cases most likely to expose model weaknesses.

The benefit depends on triage sensitivity. The jury must send enough truly problematic reports to radiologists without overwhelming them with harmless differences.

False negatives present the greater safety concern. These occur when the jury accepts a clinically important reporting error and never requests human review.

False positives create a different burden. If too many safe reports are escalated, the promised efficiency disappears and the system approaches another full reader study.

The research should report both outcomes across different error severities. A single agreement percentage would hide the operational tradeoff.

Performance should also be divided by disease, modality, patient characteristics, facility type, and equipment. Aggregate results can conceal weak performance in smaller groups.

Rare findings deserve special attention. A model can achieve high overall accuracy while failing on uncommon but urgent conditions.

One million examinations make rare-event analysis more plausible, but the actual counts still matter. A rare subgroup with only a few examples cannot support a stable conclusion.

The comparison between small and large validation cohorts will be particularly informative. It can show whether conclusions drawn from conventional studies remain stable when the case mix expands.

If rankings or error rates change materially, developers and regulators will have evidence that smaller evaluations miss important behavior. That result could reshape study design even if the jury itself needs refinement.

Success would also influence postmarket monitoring. Hospitals could sample production reports, apply an approved evaluation method, and route high-risk discrepancies to quality teams.

Such monitoring would require local governance. Health systems would need clear responsibility for alerts, review deadlines, corrective actions, and communication with vendors.

Automated evaluation cannot become a substitute for reporting safety problems. It should generate evidence that supports existing quality and regulatory processes.

The method might also help compare product versions. Developers could evaluate whether an update fixes known errors without introducing new ones elsewhere.

That use requires a stable benchmark and frozen evaluator configuration. Otherwise, changes in the jury can be mistaken for changes in the product.

The broader competitive effect could be significant. Generative radiology developers increasingly claim that their systems can draft or produce comprehensive reports.

Those claims become more testable when evaluators can review large, diverse cohorts. Companies with narrow evidence may face pressure to publish subgroup results and adjudicated error rates.

Still, the FDA contract gives Cognita an early role in shaping the methodology. The company develops its own radiology AI, so conflict-management and external reproducibility will matter.

Independent teams should be able to test the framework on other report generators. They should also be able to reproduce its central findings without relying on Cognita’s private systems.

Open code can support that process, but code alone is insufficient. External validation requires accessible datasets, documented prompts, error definitions, and model-version records.

Patient privacy will limit how much source data can be released. The project can address this through controlled access, de-identified benchmarks, or independent validation environments.

The strongest outcome would combine scale with auditability. Regulators could then inspect how a result was produced rather than accepting an opaque score.

That standard would also benefit healthcare buyers. A hospital could compare claims using error categories tied to clinical consequences instead of vendor-defined accuracy figures.

Three Signals Will Show Whether LLM Juries Are Ready

The next evidence should reveal whether Cognita’s framework measures clinical safety or merely automates agreement between language models.

The first signal is a detailed validation protocol. It should identify the jury members, prompts, model versions, cohort composition, error taxonomy, and escalation rules.

Those details will determine whether the approach is reproducible. They will also reveal whether the jury contains meaningfully different evaluators or closely related models.

A strong protocol would specify how researchers sample unanimous decisions for human review. That test is essential because agreement can hide shared mistakes.

The second signal is the radiologist-adjudicated discrepancy analysis. Researchers should report where the generated report, original report, and LLM jury each failed.

This analysis should separate hallucinations, omissions, incorrect severity, laterality errors, unsupported recommendations, and less consequential wording differences.

It should also disclose subgroup performance. Stable overall results cannot compensate for weak evaluation in rare diseases, underrepresented populations, or particular care settings.

If the jury catches clinically important discrepancies while keeping the review workload manageable, the contract’s main thesis gains support. High miss rates or excessive escalation would weaken it.

The third signal is the FDA’s response after the research matures. The agency could reference the methodology in guidance, public workshops, device reviews, or postmarket-monitoring proposals.

The FDA does not need to mandate LLM juries for the project to matter. Even limited acceptance could establish expectations for large-cohort evaluation and human adjudication.

Silence would also be informative. It might indicate that the method needs additional validation or that regulators prefer product-specific evidence.

The public should not interpret the Cognita Imaging FDA contract as proof that AI can safely police itself. The project exists because that question remains unresolved.

Its more defensible premise is narrower. Automated judges might help radiologists inspect far more evidence if their own performance is measured with equal rigor.

Developers, clinicians, and healthcare buyers should watch whether the final work exposes failure modes instead of compressing them into one score. They should also ask whether independent teams can reproduce the findings.

The decisive question is not whether several LLMs can agree about a report. It is whether their agreement and disagreement reliably direct human attention toward the cases that matter most.