Tomáš Bruckner’s LLM Behavioral Fingerprinting Can Catch API Model Swaps With Random Numbers

Tomáš Bruckner tested 165 AI models and found that LLM behavioral fingerprinting can identify suspicious API substitutions using about 120 single-token responses.

The method starts with an almost childish request: name a random number between 1 and 100. Yet models do not answer randomly. GPT-4o favored 42, 37, and 57, while Claude Sonnet 5 concentrated heavily on 47. Qwen3-Max returned 42 in every one of 30 trials.

Those preferences become useful when an API provider promises one model but delivers responses from an opaque serving chain. Bruckner’s experiment suggests that repeated, low-cost prompts can expose a behavioral mismatch without weights, logits, or cooperation from the provider.

That does not turn one number into definitive proof of fraud. The important signal is the distribution across many answers, prompts, and languages. It offers buyers an independent check on a market where a model name remains little more than an unverified label.

One-token answers produced distinct model fingerprints

The experiment converts predictable model bias into a practical identity signal.

Bruckner, a researcher at the Prague University of Economics and Business, published the initial version of his fingerprinting study on July 11, 2026. He examined models served through OpenRouter, an aggregator connecting customers with numerous models and upstream inference providers.

The study began with 342 catalog entries captured on July 6. Documented rules removed specialized systems, unstable aliases, duplicate checkpoints, non-chat models, and endpoints with mandatory hidden reasoning. That process left 165 models spanning 19 documented families and 53 serving providers.

Each model received ten simple tasks in English, Russian, Chinese, and Arabic. The resulting 40 task-language combinations included random numbers, letters, colors, animals, cities, words, and coin flips.

The researcher requested 30 responses from each combination under the main sampling configuration. Each prompt asked for a one-word answer, with optional reasoning disabled where the endpoint supported that setting.

Across the full census, the collection contained 326,047 responses. It represented 23.3 million input tokens and 1.16 million output tokens. The experiment’s design kept each visible answer extremely short.

The responses were valid in 97.6 percent of cases. The median model produced valid outputs 99.6 percent of the time, although some systems ignored the requested format more frequently.

The random-number results provide the clearest illustration. GPT-4o distributed many of its answers among 42, 37, and 57. Claude Sonnet 5 concentrated on 47, while Llama 3.3 preferred 53. Qwen3-Max chose 42 in all 30 English trials shown in the paper.

These choices reflect familiar language-model behavior. A model does not call a secure random-number generator merely because a user includes the word “random.” It predicts a likely textual continuation from patterns shaped by its training and alignment.

Human culture also influences those predictions. The number 42 carries a famous literary association, while numbers ending in seven often feel subjectively random. Prior controlled sampling research found that language models struggle to reproduce target distributions reliably.

Bruckner’s contribution is not simply another demonstration that models are bad dice. He treats each recurring bias as evidence about the system that generated it.

One answer remains almost meaningless. Thirty identical or similarly distributed attempts start revealing a pattern. Combining many distributions produces a larger behavioral profile that can be compared with a trusted reference.

The study calls each task-language pairing a probe cell. A fingerprint consists of the empirical answer distributions across those cells, rather than any single favorite response.

That distinction matters for the headline claim. A random-number prompt supplies the most memorable example, but one isolated number cannot reliably identify a model. The usable method depends on repeated sampling and a broader battery.

Why LLM behavioral fingerprinting pressures API resellers

Model buyers often receive a model name and an answer, but no direct evidence connecting the two.

First-party developers can expose models through their own APIs. Aggregators may then route requests through several upstream providers, each using different hardware, serving software, or model variants.

Additional resellers can sit between the aggregator and the final application. Every layer makes purchasing easier, but it also reduces the customer’s visibility into the serving stack.

A dishonest operator has several possible substitutions. It can route traffic to a smaller checkpoint, use an older version, or serve an aggressively quantized model. Quantization reduces numerical precision to lower resource requirements, sometimes changing output behavior.

The operator can preserve the advertised model identifier in its response metadata. A customer examining ordinary API logs might see nothing unusual, even when the underlying model has changed.

This creates a sharp commercial conflict between provider claims and observable service. Buyers pay for a named model’s expected behavior, yet they usually cannot inspect its weights or deployment configuration.

Model substitution is not solely a theoretical threat. An earlier peer-reviewed model equality audit found that 11 of 31 commercial endpoints advertising Llama models differed statistically from reference deployments.

That result did not establish malicious intent in every case. Configuration differences, quantization, version drift, and serving changes can all produce deviations. However, it showed that the advertised label does not always describe a reference-equivalent service.

The new approach lowers the operational barrier to detecting those deviations. Existing tests may require full generated passages, access to token probabilities, or specially engineered prompts.

Bruckner’s probes ask for ordinary, tiny answers. They can work against text-only APIs and do not require the model owner to embed a watermark beforehand.

This matters for developers managing many endpoints. A team could enroll a fingerprint from a trusted first-party deployment, then periodically compare an intermediary’s responses against that baseline.

The test can also run beside production monitoring. Instead of waiting for users to report vague declines in quality, an operator could detect statistical drift soon after a routing change.

For enterprise buyers, the issue reaches beyond benchmark scores. A substituted model can alter tool selection, formatting consistency, safety refusals, or instruction following. Those changes may disrupt workflows even when casual chat responses look acceptable.

They can also undermine internal evaluation. A team might approve one checkpoint, then unknowingly run another during deployment. Its original security and reliability tests would no longer describe the production system.

LLM behavioral fingerprinting therefore pressures resellers to make serving details more transparent. A provider that rotates deployments may need to disclose those variations or publish independent verification results.

It also pressures model developers. Customers need stable reference endpoints and clear version identifiers before any external auditing system can distinguish substitution from an authorized update.

The method does not eliminate trust. It shifts part of that trust from a provider’s label toward repeatable, customer-controlled measurement.

The fingerprint lives in a distribution, not a magic number

The core mechanism measures how often each answer appears, then compares that pattern with a trusted model’s pattern.

Language models assign probabilities to possible next tokens. Training data, tokenization, preference tuning, and decoding rules all influence those probabilities.

A request for a random color might favor “blue” in one model and “red” in another. A random-city prompt might repeatedly produce Paris, Tokyo, or London at different rates.

These biases are not unique secrets deliberately inserted by developers. They are behavioral residue from the model-building process. Bruckner’s technique aggregates that residue until it becomes measurable.

The paper uses Jensen-Shannon divergence to compare answer distributions. This bounded statistical measure expresses how far two probability distributions differ while treating both sides symmetrically.

Lower divergence indicates more similar response patterns. Higher divergence suggests that the two sets of answers arose from different behavioral distributions.

Two separate halves of the same model’s samples had a median cell-level divergence of 0.075. Samples from different models produced a median of 0.489. The gap was large enough to support identity verification across the full battery.

The distributions were sharply nonuniform. Across 6,572 usable cells, the median entropy was 1.00 bit, and the most common answer captured a median 71 percent share.

A truly uniform selection from 100 numbers would have an entropy near 6.64 bits. The models’ much lower result shows how heavily their answers clustered around a few preferred choices.

This concentration makes the measurement economical. If every model sampled uniformly, the answers would reveal little about identity without a vast number of requests.

The model-specific peaks create the signal. Repetition estimates where those peaks sit and how dominant they are.

The broader battery also prevents the method from depending entirely on 42 or another cultural reference. Ten tasks across four languages probe different slices of training and post-training behavior.

Language variation may expose different tokenization and corpus effects. The Chinese version of a task can produce a distribution unlike its English version, even when both ask for the same concept.

The fingerprint therefore includes 40 related views of a model. A substitute must match more than one famous answer to reproduce the complete pattern.

The study also tested model-family attribution. A nearest-neighbor classifier placed an unseen model into its documented family with 59.5 percent accuracy, compared with an 18.4 percent frequency-weighted chance rate.

Performance varied across families. GPT models achieved 90 percent recall, while GLM models reached 83 percent. Claude recall was 58 percent, and Gemini recall was 55 percent.

Those numbers are useful but limited. Verification asks whether an endpoint matches a specific enrolled reference. Family attribution asks the harder question of where an unknown model belongs.

The first task has immediate value for API auditing. The second offers investigative clues but does not deliver reliable identification across every lineage.

Other research follows a different path. LLMmap uses engineered queries to distinguish model versions and reported strong results across 42 models. Its USENIX paper shows how carefully selected prompts can extract highly discriminative behavior.

Bruckner trades single-query distinctiveness for inconspicuous repetition. Random colors, numbers, and cities resemble ordinary traffic more closely than a fixed set of unusual audit strings.

That tradeoff defines the method’s appeal. It is less accurate than a perfect identity oracle, but easier to deploy against providers that expose only conventional chat endpoints.

About 120 queries reached a 10.6 percent error rate

The study’s most practical result is a measured relationship between query volume and verification error.

The evaluation treated verification like a biometric check. One set of samples enrolled a reference identity, while another set tested whether the claimed endpoint matched it.

Genuine trials compared separate halves of the same model’s responses. Impostor trials compared halves belonging to different models.

The threshold can reject a genuine endpoint or accept an impostor. The equal error rate, or EER, reports the point where those two error rates are equal.

Using one randomly selected probe cell produced a 23.3 percent EER. Four cells reduced it to 13.2 percent, while eight cells reached 10.6 percent.

Each cell used 15 repetitions in the split-sample evaluation. Eight cells therefore required about 120 single-token queries.

Sixteen cells lowered the EER to 9.5 percent. Using 32 cells produced 8.4 percent, while the complete 40-cell battery reached 7.3 percent.

The improvement flattened after roughly 16 to 24 cells. That leaves operators with a practical choice between frequent lightweight checks and less frequent, broader audits.

A 10.6 percent EER is not sufficient for an automatic fraud verdict. Roughly one error in ten remains meaningful, especially when account suspension or contract enforcement could follow.

It can still work as an alert. Teams regularly use noisy signals to trigger deeper investigation, provided they do not treat every alert as final proof.

An endpoint that repeatedly fails several independent audits deserves closer inspection. The customer can rerun the battery, compare another trusted deployment, and examine quality or latency changes.

The method may become stronger when combined with other evidence. Token usage, latency distributions, benchmark tasks, provider metadata, and contractual deployment records can support or challenge the fingerprint result.

Bruckner’s ecosystem findings show why that layered approach matters. Among 34 pairs serving the same named model through different providers, ten diverged beyond the study’s specified impostor boundary.

The most extreme pair involved Llama 3.2 3B Instruct served through Cloudflare and Parasail. Its cross-provider distance reached 0.716, placing the pair deep in impostor territory.

Even GPT-4 differed between Azure and OpenAI’s first-party deployment, with a reported distance of 0.392. Different serving stacks can produce substantial variation without proving that either provider deliberately substituted a model.

Across all comparable pairs, the median cross-provider distance was 0.227. That was above the single-deployment noise floor but below the median impostor distance of 0.463.

In other words, provider choice usually moved the fingerprint less than changing models. Yet the exceptions were large enough to complicate automated enforcement.

The paper also identified a particularly striking anomaly. Writer’s Palmyra X5 endpoint produced a fingerprint only 0.141 away from Qwen3 235B A22B 2507.

That distance matched the median found between genuine samples of the same model. The paper described the two served distributions as statistically indistinguishable under its measurement.

This is not evidence that Writer intentionally misrepresented its model. Similar training ancestry, authorized technology use, post-training choices, or an undocumented serving arrangement could produce related behavior.

The author explicitly frames such findings as statistical deviations, not accusations of fraud. Raw data and serving metadata were released so affected organizations can contest or clarify the results.

That caution should guide every operational use. LLM behavioral fingerprinting can identify a reason to investigate. It cannot independently establish intent, contractual breach, or the exact weights running on a server.

Model updates and adaptive providers remain major limits

A behavioral fingerprint is a moving statistical reference, not a permanent serial number.

Models change. Vendors update safety layers, system prompts, tokenizers, inference settings, and routing policies. Even a legitimate update can move the answer distribution away from an older baseline.

Reference enrollment must therefore be refreshed. The paper measures short-term stability but does not establish how fingerprints behave across several months.

Auditors also need a trusted deployment of the claimed model. Without that reference, they can compare an endpoint with a public database, but they cannot guarantee equivalent configurations.

This limitation becomes especially important for closed models. Customers cannot inspect the weights, and family labels depend on vendor documentation.

Mandatory reasoning creates another gap. Some current models conduct a hidden or visible reasoning process before returning the requested word.

The study excluded endpoints when that reasoning could not be disabled. Their final answer comes from a different generation process than the direct one-token sample defined by the method.

Some endpoints also ignored requests to disable reasoning. The experiment excluded 2,486 affected responses across 14 model-provider combinations, representing 0.76 percent of the census.

Other flagship endpoints reportedly consumed 40 to 60 completion tokens while showing only one word. The hidden computation made it impossible to confirm that the visible answer represented direct sampling.

A provider could also recognize published audit prompts. It might route those requests to the genuine model while substituting another system for normal traffic.

The paper addresses this problem with paraphrases, languages, and ordinary semantic tasks. An auditor can vary “name a random color” in many ways and mix probes with real traffic.

However, the study leaves a dedicated paraphrase-invariance experiment for future work. It has not yet shown how far prompts can change before a reference fingerprint loses comparability.

A sophisticated provider might learn likely probe categories instead of exact wording. It could special-case random numbers, colors, cities, and coin flips as a group.

Expanding the battery would raise that provider’s burden. Yet it would also require new reference data and more careful calibration.

Serving-layer response caching can cause another apparent fingerprint. Replaying the same stored answer would artificially collapse the output distribution.

The study checked latency and usage metadata for evidence of response-level caching. It found no convincing signature across the census, although prompt-prefix caching appeared in some traffic.

Heavy post-training can erase family clues as well. Nvidia’s Llama 3.3 Nemotron derivative sat closer to Qwen models than to any tested Llama checkpoint.

That result does not invalidate specific-model verification. It does show that behavioral lineage and documented ancestry can diverge after extensive additional training.

The research also covers only one aggregator. The protocol does not depend on OpenRouter, but its ecosystem findings may not describe other marketplaces or direct resellers.

Finally, the paper remains a July 2026 arXiv preprint. Its released artifacts support reproduction, but independent teams still need to test the protocol across time, vendors, and changing model generations.

These limitations do not make the technique useless. They define the difference between a monitoring instrument and a cryptographic guarantee.

Hardware-backed attestation could provide stronger evidence about deployed code and weights. A separate substitution analysis argues for such infrastructure because ordinary output checks remain vulnerable to nondeterminism and adaptive behavior.

Behavioral checks occupy another layer. They can be deployed immediately against ordinary APIs, while stronger attestation requires cooperation from providers and infrastructure operators.

The most credible future system will likely combine both. Attestation can verify authorized deployments, while behavioral monitoring can detect drift, routing mistakes, or unexplained changes in actual responses.

What API customers should watch next

Three signals will determine whether this research becomes a useful audit standard or remains an intriguing laboratory result.

The first signal is independent replication. Other researchers need to enroll trusted first-party models, audit separate providers, and reproduce the reported error curve.

Replication should include endpoints outside OpenRouter. It should also measure fingerprints over several months, across vendor updates and changing inference infrastructure.

If the same model remains measurably closer to itself than to substitutes, confidence in routine deployment will rise. Frequent unexplained drift would weaken the method’s operational value.

The second signal is provider response. Aggregators and resellers can publish deployment identifiers, quantization details, upstream routing records, and version-change notices.

They can also offer customer-visible attestations or reference fingerprints. Such disclosures would help distinguish approved serving variation from silent substitution.

A lack of transparency will make third-party measurements more important. Conversely, clear routing records could resolve many anomalies without statistical inference.

The third signal is resistance to adaptive evasion. Future tests should randomize task wording, languages, timing, and probe categories while preserving comparable distributions.

Researchers should measure whether models keep recognizable fingerprints under paraphrases they never saw during enrollment. They should also test providers that actively attempt to detect audits.

Success would strengthen the paper’s central claim. Failure would show that published behavioral checks work primarily against careless substitution.

Teams do not need to wait for a formal standard before improving their own monitoring. They can retain model-version metadata, archive representative outputs, and rerun evaluations after unexplained behavior changes.

Knowledge-intensive workflows especially need that history. Model drift can change summaries, project recommendations, research synthesis, and the retrieval of prior decisions without producing an obvious outage.

A structured AI knowledge base can preserve prompts, source documents, outputs, and review notes around important model-assisted work. That record helps teams investigate whether an endpoint change affected real decisions.

The central lesson is simple but narrower than the viral framing suggests. One random number does not reveal an AI model’s identity. A repeated distribution of tiny answers can provide a useful, inexpensive warning when an API behaves unlike its claimed reference.

For developers and enterprise buyers, the next step is not to accuse a provider after seeing 42. It is to establish trusted baselines, run recurring audits, and investigate persistent deviations with several forms of evidence.