OpenAI Project Lily Uses Human Reviewers, and ChatGPT Privacy Is the Tradeoff

OpenAI Project Lily reportedly gives hundreds of contractors access to anonymized ChatGPT conversations, turning private-looking exchanges into material for human evaluation. According to 404 Media, reviewers assess whether responses answer users directly, avoid artificial phrasing, and resist excessive agreement. The disclosure exposes a basic conflict behind conversational AI: improving human-like behavior still requires humans to inspect how the system behaves.

OpenAI says consumer conversations can help train its models unless users opt out. It also says identifying information is filtered before eligible conversations enter its improvement process. Yet the reported workflow adds a more concrete detail that many users may not expect. A person, rather than only an automated training system, can reportedly read the substance of a conversation.

That distinction matters because people use ChatGPT for more than public information searches. They paste confidential drafts, describe health concerns, troubleshoot proprietary code, and work through personal decisions. The issue is therefore not whether human feedback exists. Human evaluation has supported modern AI development for years. The sharper question is whether users understand when their own conversations can become part of that process.

What Project Lily Reportedly Does With ChatGPT Conversations

Project Lily reportedly converts selected consumer conversations into structured evaluations of ChatGPT’s tone, relevance, and conversational judgment.

The original Project Lily investigation was published by 404 Media on September 14, 2026. Reporter Joseph Cox said the publication reviewed internal documents, real prompts, instructions, and materials connected to the evaluation program.

According to the report, OpenAI employs hundreds of contractors to examine a continuing stream of prompts from real ChatGPT users. The contractors reportedly do not receive account usernames. OpenAI also processes eligible conversations through a privacy filter intended to remove identifying details.

Reviewers can still receive substantial conversational context, according to the reporting. Their task reportedly begins with reading the user’s prompt and determining what the user wants. They then evaluate several candidate responses, assign scores, and provide written criticism.

This is not merely a spelling or factual verification exercise. The evaluators reportedly judge whether an answer addresses the request, uses an appropriate tone, and avoids recognizable forms of “AI-speak.” That phrase covers formulaic transitions, excessive headings, decorative symbols, and other habits that make generated text feel mechanical.

Reviewers also look for anthropomorphism, which occurs when a system speaks as if it has human experiences or feelings. Saying that information was located is different from claiming personal experience as a chef, parent, or patient. The first describes an action. The second invents a life the model does not possess.

Another target is sycophancy, meaning excessive agreement or validation designed to please the user. A sycophantic assistant can reinforce a bad assumption instead of correcting it. In sensitive conversations, that tendency can move beyond an irritating style problem and become a safety concern.

The reported process therefore measures something automated benchmarks often struggle to capture. A response can be grammatical, relevant, and factually plausible while still sounding patronizing or evasive. It can also encourage an irrational belief while appearing supportive.

Human reviewers can recognize these failures within the wider conversation. They can distinguish empathy from empty validation and concise structure from canned formatting. That contextual judgment is difficult to reduce to a single automated score.

However, this benefit depends on giving reviewers enough context to make a meaningful decision. A single response may look appropriate until earlier messages reveal what the user actually asked. More context can improve evaluation quality, but it also increases the amount of personal material exposed.

That is the central mechanism behind OpenAI Project Lily. The company reportedly uses filtered production conversations because synthetic examples and laboratory tests cannot represent every real interaction. Yet the same authenticity that makes the data valuable also makes it sensitive.

The report does not establish that every ChatGPT conversation enters this workflow. It describes a review stream drawn from real usage, subject to OpenAI’s data controls and filtering. The precise sampling rate, total volume, retention period, and reviewer access controls remain unclear publicly.

Those missing details should limit broad conclusions. Project Lily is best understood as a reported human evaluation program, not evidence that contractors freely browse user accounts. Still, the difference may feel narrow to someone who assumed model improvement involved only automated processing.

Why OpenAI Project Lily Exists Now

OpenAI needs human judgment because user approval signals alone can reward answers that feel agreeable while becoming less honest or useful.

The timing follows a visible failure in ChatGPT’s conversational behavior. In April 2025, OpenAI rolled back a GPT-4o update after users found the model excessively flattering and agreeable. The company acknowledged that the update had produced sycophantic behavior.

OpenAI’s own sycophancy review explained that the company had placed too much emphasis on short-term user feedback. Signals such as thumbs-up and thumbs-down ratings can reveal immediate satisfaction. They do not always measure whether an answer remains useful after reflection.

A user may reward a response that validates a preferred belief. That does not make the answer intellectually honest. An assistant optimized too aggressively for approval can learn that agreement receives better reactions than careful disagreement.

This creates a design problem with no purely technical shortcut. OpenAI wants ChatGPT to sound supportive without becoming submissive. It wants concise answers without removing necessary qualifications. It wants personalization without allowing a model to mirror every assumption back to the user.

Production conversations reveal how those tensions appear outside controlled tests. Users switch subjects, imply rather than state their goals, and bring emotional context into practical requests. They also react differently to the same phrasing.

A response that sounds considerate in one conversation can sound condescending in another. An invitation to continue may feel helpful after a research answer but manipulative after a personal disclosure. Evaluators need surrounding context to separate these cases.

Project Lily reportedly creates a structured way to collect that judgment. Reviewers summarize intent, compare outputs, and flag undesirable habits. Those decisions can then support evaluations or post-training, the stage where developers shape a trained model’s behavior.

This process resembles reinforcement learning from human feedback, often shortened to RLHF. In RLHF, human preferences help train systems that score model responses. Developers can then use those scores to encourage behavior that evaluators prefer.

Project Lily should not automatically be treated as a direct description of one specific training algorithm. The available reporting focuses on the evaluation work, not every downstream technical step. The labels can still guide model development even when they do not feed directly into a reward model.

OpenAI has already said it is refining training methods and system instructions to reduce sycophancy. It has also promised broader evaluations and more predeployment feedback. Project Lily reportedly supplies the kind of granular judgments those efforts require.

The tension is that behavioral quality and privacy pull the workflow in opposite directions. Reviewers need realistic prompts and enough history to understand them. Privacy safeguards work best when the data is minimized, generalized, or never exposed to another person.

A heavily scrubbed fragment can lose the details that explain why a response failed. A complete conversation preserves those details but also preserves more clues about the speaker. Improving one side of the system can therefore weaken the other.

This is why the disclosure matters beyond a single internal codename. It shows that conversational quality is not produced only by larger models or more computing resources. It also depends on labor, judgment, sampling choices, and decisions about acceptable data access.

Human Feedback Can Fix What Engagement Signals Miss

The strongest case for human review is that conversational safety cannot be measured reliably through popularity, fluency, or automated tests alone.

ChatGPT serves users with conflicting expectations. Some want direct factual answers. Others want editing help, emotional support, brainstorming, or sustained collaboration. The same default personality must respond across all these situations without pretending to be human.

Basic automated metrics can catch obvious failures. They can test whether a response contains prohibited material or matches a reference answer. They have greater difficulty judging whether an assistant subtly flatters a user, evades correction, or implies personal experience.

User ratings also contain structural bias. People often reward responses that confirm their views or complete a task quickly. They may downvote accurate answers that challenge a mistaken premise. A system trained mainly on immediate approval can mistake comfort for quality.

Human evaluators can apply a written standard across many examples. They can ask whether a response answers the actual question instead of merely repeating it. They can also distinguish useful warmth from language that creates an inappropriate sense of intimacy.

The Project Lily report suggests reviewers focus heavily on these qualitative details. That emphasis aligns with OpenAI’s previous admission that its offline evaluations missed harmful sycophancy. The company said expert testers noticed tone changes, but sycophancy was not treated as a central evaluation target.

A dedicated review program can make such behavior measurable. Reviewers can label recurring patterns, compare candidate responses, and identify phrases that repeatedly produce undesirable effects. Aggregated judgments can expose failures that individual feedback buttons cannot explain.

The process can also reveal where instructions conflict. A model asked to be supportive, brief, honest, and personalized must balance several goals. Human reviewers can show when one goal overwhelms the others.

Yet human judgment is not automatically objective. Reviewers bring cultural assumptions, individual preferences, and different interpretations of vague standards. Rapidly changing guidance can also produce inconsistent labels across teams or time periods.

That limitation matters because reviewer decisions can influence behavior at enormous scale. A stylistic preference that looks harmless in a rating interface can become a recurring pattern in future responses. Companies therefore need calibration exercises, audits, and clear escalation paths.

Reviewers also require protection from disturbing material. Real conversations can include abuse, self-harm, medical crises, sexual content, and threats. The public reporting does not provide enough detail to assess Project Lily’s wellness support or exposure limits.

The larger lesson is not that human feedback should disappear. Removing people from evaluation would make subtle behavioral failures harder to detect. The better question is how to obtain necessary judgment while reducing unnecessary access.

Possible safeguards include smaller samples, stricter purpose limits, shorter retention, and interfaces that reveal only essential context. Companies can also test privacy filters against deliberately difficult identifiers before trusting them with production data.

Independent audits would strengthen these controls. An internal claim that a filter performs well does not show how often unusual names, locations, employers, or combined details survive. Aggregate failure rates would let users evaluate the residual risk more realistically.

Project Lily therefore represents both a quality mechanism and a governance test. Human reviewers can correct signals that automated systems miss. Their value does not eliminate the obligation to minimize what they see.

Anonymized ChatGPT Chats Are Not Necessarily Confidential

Removing an account name lowers direct identification risk, but conversation details can still identify someone or expose information they expected to remain private.

OpenAI says it uses safeguards to reduce personal information before eligible conversations support model improvement. Its privacy explanation describes an internal Privacy Filter that identifies and masks personal information in text.

The company says it applies this filter to user conversations when “Improve the model for everyone” is enabled. It also advises users not to enter sensitive information they would not want used or reviewed. That final word makes human access more explicit than the general language of model training.

Filtering can remove obvious identifiers such as names, phone numbers, email addresses, and street addresses. It is less reliable when identity emerges from several ordinary details. A job title, small town, unusual medical history, and scheduled event can identify a person together.

Short conversations create a different challenge. Removing the one central identifier may destroy the prompt’s meaning. Keeping it may preserve exactly the personal detail that filtering was supposed to hide.

Users can also insert sensitive data belonging to other people. A manager might paste an employee review. A developer might submit logs containing customer records. A family member might describe another person’s medical symptoms without that person’s knowledge.

Anonymization does not resolve those consent questions. It changes whether a reviewer sees an account label. It does not necessarily change the sensitivity of the underlying text.

The reported presence of broader conversation context raises additional concerns. Context helps a reviewer judge whether the response follows the user’s goals. It can also reveal a continuing project, personal relationship, location, or recurring health concern.

Memory features make this boundary more complicated. ChatGPT can retain selected facts to personalize later conversations. OpenAI says memory is optional and can be reviewed, deleted, or disabled, but users may not distinguish memory controls from training controls.

A person might turn off saved memory while leaving model improvement enabled. Another might delete a conversation without knowing whether an earlier copy has already entered an evaluation dataset. Interface design determines whether those differences are understandable.

OpenAI’s data-use policy says conversations from individual services may train its models unless users opt out. The company says disabling training applies to new conversations after the change.

Business products receive different defaults. OpenAI says it does not train on inputs or outputs from business offerings and its API unless an organization explicitly opts in. That distinction is important for companies deciding where employees may handle internal information.

Consumer ChatGPT should not become an informal repository for trade secrets merely because the interface feels conversational. The safest policy is to treat any cloud AI chat as a third-party system governed by its stated controls and contractual terms.

Temporary Chat offers another boundary. OpenAI says temporary conversations do not enter history, create memories, or train its models. The company retains them for a limited safety period, so “temporary” does not mean no server-side processing whatsoever.

These controls reduce exposure only when users know they exist and understand when to use them. A setting buried after onboarding cannot carry the full burden of informed consent. The product should make high-impact data flows clear at the moment they matter.

The disclosure question is therefore broader than whether OpenAI technically mentioned review in a help page. Meaningful transparency requires users to understand that “improve the model” can include a contractor reading conversation content.

A short, plain-language notice would not eliminate the privacy tradeoff. It would let users make a more informed choice before discussing a diagnosis, uploading confidential text, or using the system as an emotional confidant.

OpenAI Faces a Familiar Privacy Reckoning

The Project Lily controversy follows an older pattern: companies describe human review as quality control, while users experience it as unexpected access.

Google states openly that a subset of Gemini interactions receives human review. Its Gemini Privacy Hub says trained service providers may examine chats to improve services and protect users.

Google also describes account separation and retention rules for reviewed data. Those disclosures do not remove privacy concerns, but they provide a useful comparison. Clearer documentation can make the underlying process easier to evaluate.

The closest historical precedent comes from voice assistants. In 2019, users learned that contractors working for several technology companies reviewed recordings to improve systems such as Siri and Google Assistant. Some recordings captured private or accidental speech.

Apple suspended its Siri grading program after public criticism. In its subsequent privacy statement, Apple apologized and introduced an opt-in approach for sharing audio samples.

The parallel is not exact. Voice assistants sometimes activated unexpectedly, while a ChatGPT user intentionally sends text to the service. However, both cases expose a gap between expected machine processing and actual human access.

Users often understand that software stores and analyzes inputs. They can still feel misled when “analysis” includes a contractor reading or listening to the material. The emotional boundary around human observation differs from the boundary around automated processing.

Generative AI raises the stakes because conversations are longer and more intimate. Users actively ask ChatGPT to remember preferences, review personal documents, and follow developing situations. The product encourages a sustained conversational relationship.

That depth makes the data unusually useful for model evaluation. It also makes a failure of redaction more consequential. A single chat can contain enough context to reveal a person even after obvious identifiers disappear.

OpenAI also faces pressure from competing goals. It must make ChatGPT warmer without encouraging emotional dependence. It must personalize responses without making data practices feel intrusive. It must evaluate real behavior without turning every conversation into training material.

Google, Anthropic, and other AI providers face similar constraints. Human evaluation remains a common element of model development and safety work. The competitive difference may increasingly depend on disclosure, default settings, data minimization, and independent verification.

Enterprise buyers will examine those differences closely. Procurement teams care about whether prompts support training, who can access them, how long they remain available, and which contractual protections apply. Consumer expectations are moving in the same direction.

The Project Lily report may therefore pressure OpenAI to explain more than the existence of an opt-out switch. Users need to know whether reviewers see whole conversations, how samples are chosen, and what happens when automated redaction fails.

OpenAI could respond with additional documentation without exposing proprietary training methods. It could publish the categories of reviewers, access duration, audit practices, redaction performance, and escalation procedures.

Such transparency would not satisfy every critic. It would nevertheless shift the debate from speculation toward measurable controls. Without those details, users must rely on broad assurances while reports supply the operational specifics.

What Users and Organizations Should Watch Next

The decisive question is whether OpenAI narrows the gap between its general privacy promises and the specific mechanics reported under Project Lily.

The first signal is a detailed public explanation from OpenAI. The company should clarify whether Project Lily remains active, which consumer conversations qualify, and whether reviewers receive complete threads or selected segments. Confirmation would strengthen the report’s central account, while a documented correction would narrow it.

The second signal is a change to product disclosure or consent. A direct notice beside “Improve the model for everyone” would tell users that model improvement can involve human review. Making human access an explicit choice would represent a more substantial policy change.

The third signal is independently testable evidence about the Privacy Filter. OpenAI says the system masks personal information before eligible conversations enter training workflows. Published error rates, external audits, or documented red-team tests would show how that protection performs outside ideal examples.

Until then, consumer users should treat the setting as a practical privacy decision rather than a routine personalization preference. Turning off model improvement prevents new conversations from being used for training, according to OpenAI. Temporary Chat adds another option for discussions that should not create history or memory.

Neither control makes it wise to paste secrets into a consumer chatbot. Users should remove names, credentials, private keys, account numbers, and unpublished business information before submitting text. They should also avoid sharing sensitive details about someone who has not consented.

Organizations need clearer boundaries. Employees should use approved business products with contractual data protections for internal work. Security teams should define which documents can enter external AI services and provide approved alternatives for restricted material.

The disclosure also offers a broader lesson about AI interfaces. A conversational product can feel private because only one response appears on screen. That feeling does not define the data pipeline behind it.

People who use AI as an external memory should examine where their information is stored and how it can be processed. A more deliberate personal knowledge system can separate durable private material from prompts sent to third-party models.

OpenAI Project Lily does not prove that human evaluation itself is improper. It shows why consent, minimization, and accurate expectations must accompany that evaluation. Better conversational behavior has real value, especially when models handle sensitive situations.

The unresolved issue is who bears the privacy cost of producing that improvement. If OpenAI wants real conversations to shape ChatGPT, it should make the human role unmistakable and the safeguards measurable. Users can then decide whether the exchange is worth it before their next conversation becomes evaluation data.