AWS Makes Model-agnostic PII Detection with LLMs a Contest Between Prompts and Fixed Schemas

AWS has released model-agnostic PII detection with LLMs after testing nine detectors on 49,365 records across five public datasets. The approach challenges a basic assumption behind conventional personally identifiable information detection. Instead of fixing supported entities during model training, AWS places those definitions inside a prompt.

That shift matters because enterprise data rarely follows a permanent privacy schema. A customer-support archive can contain account references, employee identifiers, wallet addresses, credentials, and organization-specific codes. A detector trained for names and phone numbers cannot automatically recognize every new category.

AWS says its design can add an entity through instructions, without labeling another training set or retraining a tagger. It can also switch among Amazon Bedrock models or use a custom self-hosted backend. The contest is therefore not simply AWS against another vendor. It is prompt-configured detection against the fixed schemas that have defined this software category.

The benchmark gives that contest substance, but not a decisive verdict. AWS reports that Mistral Large 3 reached 83.1 percent Core F1, compared with 80.7 percent for OpenAI PrivacyFilter. Yet the results come from AWS and its accompanying sample repository, not an independent evaluation.

The most important result concerns uncommon entities, rather than that narrow headline margin. AWS reports that changing the prompt raised extended-entity F1 several-fold across every tested model. If that result holds on real enterprise text, privacy teams gain a faster way to adapt detection policies.

What AWS Actually Released

AWS released a configurable detection pipeline, not a newly trained privacy model.

The project appeared on September 10, 2026, through the AWS Machine Learning Blog. Authors Christophe Dupuy and Rahul Gupta described an instruction-driven detector and published its implementation in an open sample repository.

The detector accepts free-form text and asks an LLM to identify sensitive spans. A span is the exact portion of text containing an entity, such as a name or account number. The model returns each value with a category inside a structured JSON list.

A post-processing layer then finds every returned value in the original text. It assigns exact character offsets and removes duplicate detections. This choice avoids asking the LLM to calculate positions, a task AWS says models do not perform reliably.

The default prompt defines 15 categories. These include private and public names, full and partial addresses, contact information, financial data, identification numbers, credentials, dates, digital identifiers, and URLs.

The prompt also includes exclusions, short definitions, and optional examples. That text acts as the detector’s operating schema. A team can add a category by modifying the instructions instead of modifying model weights.

AWS separates this schema from the inference backend. An interface called Inferencer takes messages and returns the model’s response. The supplied adapter calls the Amazon Bedrock Converse API, while developers can implement the same interface for another environment.

The published code supports inputs that exceed a model’s context limit by splitting text at word boundaries. It also retries transient service errors. According to the repository, Boto3 is its only runtime dependency.

A recovery layer handles another characteristic LLM failure. Models can return plausible but unsupported labels, such as DATE when the prompt expects DATES. The software maps recognized variants back to its defined vocabulary and marks unresolved labels as unknown.

That mechanism makes the output more usable, but it also reveals why an LLM cannot serve as the entire privacy control. The model’s response requires validation, offset recovery, deduplication, and label normalization before another system should act on it.

AWS presents the resulting spans as inputs for a later redaction stage. The detector itself does not complete the full data-cleaning process. Teams still need policies for masking, deletion, review, retention, and exception handling.

The immediate target is training data preparation. Free-form documents can contain personal information that a model later memorizes or reproduces. Customer conversations, HR records, financial documents, and chat histories create particularly difficult detection conditions.

However, the same design can sit before search indexing, analytics, support automation, or an internal AI assistant. Any workflow that moves unstructured text between systems needs a reliable boundary for sensitive information.

That creates a natural connection to how engineering teams manage a searchable knowledge base. Retrieval quality matters, but privacy controls must determine what enters the index first.

Why Fixed-Schema Detectors Are Under Pressure

The AWS proposal pressures detectors whose supported entity list can change only through another training cycle.

Many established PII systems combine regular expressions, dictionaries, rules, and named-entity recognition models. Named-entity recognition labels spans according to categories learned from annotated examples. That structure can be accurate and predictable within its intended scope.

The weakness appears when an organization’s definition of sensitive data moves beyond the training labels. A hospital might need internal patient codes. A manufacturer might protect equipment identifiers linked to customers. A financial platform might treat wallet addresses as sensitive.

A conventional tagger does not infer these policy choices automatically. Teams often need new examples, annotation guidance, model training, evaluation, and deployment. Each change becomes a small machine-learning project.

Model-agnostic PII detection with LLMs moves much of that work into an instruction layer. The team defines the entity, provides examples, and sends the revised prompt to the selected model. The application interface can remain unchanged.

This separation also reduces dependence on one model family. AWS demonstrates managed models through Amazon Bedrock and open models hosted on Amazon EC2. The same detection class can use a custom backend if it follows the expected message interface.

Amazon describes its Converse API as a consistent interface for supported conversational models. That consistency gives the detector a common integration point, although model behavior still differs.

The model choice controls accuracy, response time, operating cost, context length, and deployment location. The prompt controls which entities the detector should find. Keeping those decisions separate is the project’s central architectural claim.

That claim challenges both fixed taggers and single-model LLM tools. A fixed tagger limits the schema. A detector coupled to one foundation model limits procurement and deployment choices.

Open-source systems such as Presidio take a broader orchestration approach. They can combine recognizers and support custom logic. Such systems remain relevant because enterprises often value deterministic patterns and auditable rules.

The emerging contest is therefore not LLMs versus regular expressions in every situation. Credit-card formats, phone patterns, and known identifiers can still benefit from deterministic recognition. The pressure falls on systems that cannot adapt their semantic categories quickly.

Context is where LLMs offer a distinct advantage. The same number can represent an age, an account reference, a date, or harmless text. A language model can interpret surrounding words instead of relying only on surface structure.

Yet context can also make outcomes less predictable. Prompt wording, model updates, sampling settings, and input formatting can affect a response. A fixed rule may be narrow, but teams can usually explain exactly why it fired.

Enterprise buyers must compare two kinds of maintenance. Conventional systems require code, labels, or model updates when the schema changes. Instruction-driven systems require prompt governance, regression testing, and ongoing model evaluation.

AWS reduces the cost of changing the declared schema. It does not remove the need to prove that the revised detector works. That distinction separates convenient configuration from dependable privacy enforcement.

The project arrives as organizations send more private text into generative AI pipelines. Data moves from archives into embeddings, fine-tuning corpora, agent memory, and retrieval systems. Each additional copy expands the consequences of a missed entity.

The NIST Privacy Framework treats privacy as an enterprise risk-management problem, not simply a classification problem. Detection supports that program, but governance still determines acceptable collection, use, storage, and disclosure.

Model-agnostic PII Detection with LLMs Wins Its Strongest Case on New Entities

The benchmark’s most consequential result is the prompt’s ability to recover uncommon categories, not one model’s narrow F1 lead.

AWS evaluated the approach using five public PII datasets hosted on Hugging Face. The sample contained 49,365 records and 222,114 ground-truth core spans across eight languages.

Those languages were German, English, Spanish, French, Hindi, Italian, Dutch, and Telugu. The datasets included synthetic profiles, HR documents, financial text, and customer-service material.

AWS compared nine LLM-based detectors. Three ran through Amazon Bedrock, while six used models hosted on Amazon EC2. OpenAI PrivacyFilter appeared as one of the self-hosted comparison systems.

The evaluation mapped inconsistent dataset labels into 12 canonical entities. These covered names, addresses, contact details, dates, ages, national identifiers, financial data, network addresses, URLs, usernames, credentials, and identification numbers.

A prediction counted as correct only when its start position, end position, and label exactly matched the ground truth. AWS reported precision, recall, and F1, which balances the first two measures.

Mistral Large 3 delivered the highest base Core F1 at 83.1 percent. OSS-GPT 20B on EC2 followed at 81.6 percent. PrivacyFilter reached 80.7 percent, while the lowest listed score was Nova Lite 2 at 74.9 percent.

These results do not show that every Bedrock model beats an off-the-shelf detector. One managed model led PrivacyFilter, while another trailed it by 5.8 percentage points. Model selection clearly remains material.

Latency varied even more. Gemma-4-E4B-it had an estimated per-detection time of 0.43 seconds, while Qwen3.5-9B took 15.31 seconds. Mistral Large 3 required 1.16 seconds in AWS’s estimate.

AWS warns that those figures are extrapolated from parallel batch execution. They should not be read as guaranteed single-request latency. Infrastructure, batching, record length, and service conditions can all change production performance.

The OSS-GPT 20B comparison supports the backend portability claim. AWS reports Core F1 of 81.6 percent on EC2 and 81.3 percent through Bedrock. A 0.3-point difference suggests similar accuracy in those tested configurations.

The stronger evidence appears in the extended-entity experiment. Several datasets included categories outside the canonical core, such as occupations, company names, wallet addresses, vehicle identifiers, and user-agent strings.

The base detector had little reason to flag those categories because its prompt did not define them. AWS then added definitions and examples through an Extended configuration. No underlying model received additional training.

For Qwen3.6-35B-A3B, extended-entity F1 reportedly rose from 9.4 percent to 80.5 percent. OSS-GPT 20B rose from 12.1 percent to 73.3 percent. Mistral Large 3 rose from 17.3 percent to 72.7 percent.

Core performance did not collapse after the schema expanded. AWS reports that Mistral Large 3 increased from 83.1 percent to 89.1 percent Core F1. OSS-GPT 20B increased from 81.6 percent to 83.1 percent.

Those gains should be treated as AWS-reported benchmark outcomes. The full methodology and mappings appear in the project’s benchmark documentation. Independent replication remains necessary.

Still, the experiment directly tests the project’s main argument. New entity definitions produced useful detections without a new labeled training run. That is a meaningful operational difference from a frozen tagger.

It also changes who can modify the detector. Privacy specialists and domain owners can help write entity definitions and examples. Machine-learning engineers remain necessary for evaluation, infrastructure, and failure analysis, but not every schema revision.

The result favors a policy-as-prompt workflow. A team can version instructions beside application code, test each change, and route identical samples through several models. That makes model replacement easier than rebuilding the surrounding pipeline.

However, prompt portability does not guarantee behavioral equivalence. Two models can follow the same entity definitions and return different spans. Model-agnostic architecture means the backend is replaceable, not that every replacement performs alike.

The Benchmark Leaves a Production-Sized Verification Gap

AWS shows a credible prototype and a broad internal benchmark, but neither establishes safe performance on an organization’s private data.

The first limitation is source independence. AWS designed the detector, selected the evaluation method, ran the infrastructure, and published the interpretation. The released code improves transparency, but external replication would strengthen the findings.

The second limitation is dataset realism. Public corpora make repeatable comparison possible, yet several contain synthetic or standardized examples. Production text includes misspellings, formatting damage, code switching, copied signatures, unusual abbreviations, and organization-specific shorthand.

The third limitation is the remaining error rate. An F1 score near 83 percent can be useful for triage, but privacy failures are asymmetric. One missed national identifier can matter more than several false alarms.

Aggregate F1 can also hide category-level weakness. AWS reports that OSS-GPT 20B exceeded 95 percent for SSN, financial, and identification categories on one dataset. The same analysis placed date detection near 50 percent.

Dates illustrate a genuine policy problem. A date can identify a birth, appointment, transaction, publication, or public event. The correct label often depends on both context and an organization’s privacy rules.

Exact-match scoring is demanding because a partially correct boundary receives no credit. That rigor is useful for redaction, where leaving part of a value exposed can defeat the control. It can also magnify minor annotation differences.

Multilingual averages create another risk. AWS reports an 83 to 90 percent Core F1 band across eight languages for one model and dataset. That evidence does not establish performance across dialects, mixed-language documents, or every writing system.

Prompt injection deserves special attention. The detector places untrusted text near instructions sent to a general-purpose model. A document might contain language that attempts to override the task or manipulate the output.

The published prompt structure and parsing layer can reduce malformed responses. They cannot guarantee that every model will ignore adversarial content. Teams need tests containing instruction-like text, encoded values, fragmented identifiers, and deliberate evasion.

Hallucinated labels create a related concern. AWS’s recovery layer maps familiar variants to approved categories and exposes unrecognized labels as unknown. That is safer than silently inventing a category, but it still requires monitoring.

Offset recovery introduces edge cases too. The model returns values rather than positions, and the software searches for those values in the source. Repeated strings, normalized punctuation, Unicode variants, or altered whitespace can complicate matching.

The project’s reliance on model output also creates change-management obligations. A provider can update a managed model without changing the calling code. Teams should detect whether those updates change recall, false-positive rates, or formatting behavior.

A production deployment therefore needs a versioned test suite built from representative internal samples. The suite should include rare entities, multilingual text, adversarial passages, empty records, long documents, and known hard negatives.

Human review remains appropriate for uncertain cases and high-impact workflows. A detector can prioritize records, mark spans, or block automatic ingestion. It should not automatically delete source data without a recoverable process and explicit policy.

Teams should also avoid sending raw sensitive text to an unintended region or service. Bedrock access, identity permissions, network paths, logging, encryption, and data residency require separate review. Model portability helps only when the deployment is configured correctly.

Cost and throughput need measurement on actual records. Per-detection latency can become substantial across millions of documents. Batching and concurrency can improve throughput, while longer prompts and repeated examples consume more tokens.

A hybrid architecture may prove more practical than an LLM-only design. Deterministic recognizers can catch structured patterns quickly. An LLM can handle ambiguous context and organization-specific entities, with review reserved for uncertain results.

This is where the fixed-schema versus prompt-schema framing becomes less absolute. Enterprises seldom need one universal detector. They need a layered control whose components fail differently and expose enough evidence for investigation.

The Real Decision Is About Control, Not the Highest Score

Model choice determines performance, but operating control determines whether the detector fits a regulated workflow.

Amazon Bedrock offers a managed path with one conversational interface across supported models. That can reduce infrastructure work and simplify controlled comparisons. Teams can change a model identifier without changing the detector’s public call pattern.

Self-hosting offers a different form of control. An organization can keep inference within infrastructure it manages and select hardware, network boundaries, and update schedules. It also assumes responsibility for capacity, patches, availability, and monitoring.

AWS’s architecture supports both routes because the Inferencer interface is small. Any compatible adapter can receive messages and return the assistant’s text. That abstraction is valuable even for teams that never use Bedrock.

It makes model evaluation an ongoing procurement decision instead of a one-time application commitment. A team can compare accuracy, latency, and operational constraints using the same prompt and test corpus.

However, backend portability can encourage false confidence. A one-line configuration change is technically simple, but a model replacement should trigger regression testing. Equivalent interfaces do not imply equivalent privacy outcomes.

Different models may interpret exclusions differently. They can disagree about public names, business addresses, partial identifiers, and contextual dates. They can also vary in JSON reliability and their resistance to instruction conflicts.

Prompt changes require the same discipline. A new category can overlap with an existing exclusion or broaden detection unexpectedly. AWS notes that its Extended configuration removes exclusions that conflict with newly protected categories.

For example, an organization might begin treating employer names as sensitive. It must then reconsider any instruction that exempts public company information. That is a policy decision, not just a prompt-editing convenience.

Version control can make those choices visible. Teams can review each definition, example, exclusion, model identifier, and evaluation result. Deployment approvals can reference a specific configuration instead of an informal prompt.

A mature workflow should also store detection evidence without creating another privacy problem. Logs need enough context to diagnose errors, but copying full sensitive values into observability systems increases exposure.

One option is to record category, position, confidence proxy, configuration version, and a protected document reference. Reviewers can retrieve the source through authorized systems when investigation becomes necessary.

Confidence itself remains difficult. Generative models do not automatically return calibrated probabilities for each span. Teams may need agreement across models, repeated evaluation, or separate validation rules to prioritize uncertain cases.

The detector’s ability to swap models makes ensemble testing feasible. A team could compare outputs from a fast model and a more accurate model. Yet that doubles complexity and can increase data movement.

The best initial deployment is likely a bounded gate before a downstream AI workflow. The detector can flag or redact candidate spans, while existing policy controls handle approval and exceptions.

Training-data preparation fits that pattern because cleaning already happens before model development. Search ingestion and agent memory are also strong candidates because teams can intercept text before it spreads.

Real-time customer interactions create harder constraints. Latency, false positives, and availability matter immediately. A detector that takes several seconds per record may require asynchronous handling or a faster first-stage filter.

AWS has made experimentation easier by releasing the implementation. It has not eliminated the engineering required for production privacy. The value lies in reducing schema friction while preserving deployment choice.

Three Signals Will Decide Whether the Prompt-Based Route Holds Up

Independent replication, adversarial testing, and sustained production use will determine whether this design becomes a dependable privacy layer.

The first signal is an independent reproduction of the benchmark. Researchers or enterprise teams should run the published detector against the same datasets and report complete category-level results.

Replication should verify the prompt, model versions, sampling settings, mappings, infrastructure, and scoring code. A close match would strengthen AWS’s central claims. Large differences would expose hidden sensitivity in the evaluation.

The second signal is evidence from adversarial and organization-specific data. Tests should include prompt injection, obfuscated identifiers, multilingual code switching, repeated values, Unicode substitutions, and uncommon internal codes.

Success would mean the prompt-defined schema survives messy text without unacceptable recall loss. Failure would suggest that easier customization shifts too much risk into prompt behavior and post-processing.

The third signal is measurable production adoption with published operating practices. Useful reports would describe throughput, review rates, false-positive costs, model updates, residency controls, and incident handling.

Adoption alone would not validate accuracy. Documented safeguards would show whether teams can govern the detector through configuration changes and backend replacements. Silent deployments with no evaluation detail would provide much weaker evidence.

In the near term, developers should treat the project as a testable reference architecture. It offers code, mappings, examples, and benchmark results rather than only a product claim.

Privacy teams should start with a corpus they understand. Define the entities that matter, including internal identifiers that generic tools miss. Then measure recall by category and inspect every serious false negative.

Platform teams should compare at least two suitable backends. They should record latency distributions, malformed responses, unknown labels, offset failures, and total throughput. Average F1 alone cannot select a production model.

Security reviewers should attack the full pipeline. Their test plan should include hostile document instructions, logging exposure, excessive permissions, regional routing, and unsafe downstream deletion.

The broader bet behind model-agnostic PII detection with LLMs is straightforward. Privacy definitions change faster than traditional model-training cycles, so those definitions should become configurable runtime policy.

AWS has supplied meaningful evidence for that bet, especially on extended entities. It has also exposed the work that remains, including model variance, post-processing, adversarial resilience, and independent validation.

The right next action is not to replace every existing detector. Select one bounded dataset, preserve the current control as a baseline, and run both systems against reviewed ground truth.

Which result should determine the decision? Track missed high-risk entities first, then review burden, latency, and operating control. If the prompt-based detector adapts faster without weakening those measures, AWS’s architectural argument becomes much harder to dismiss.