OpenAI Project Lily Puts Human Review Behind ChatGPT’s Private-Looking Conversations

OpenAI Project Lily reportedly employs hundreds of contractors to review real ChatGPT conversations, despite the private atmosphere created by a one-on-one chatbot interface. The reviewers help evaluate model responses, but some conversations reportedly contain personal information that automated filtering failed to remove.

The program’s existence was detailed in a September 14 investigation based on internal guides, workplace discussions, rating materials, and real prompts. According to the original investigation, reviewers can receive complete conversations rather than isolated, context-free questions.

The central conflict is not whether humans contribute to artificial intelligence. Human feedback has shaped leading language models for years. The sharper issue is whether consumer disclosures adequately prepare people for a contractor reading an exchange that feels confidential.

Google already states that trained service providers review a subset of Gemini conversations. Anthropic also confirmed that it uses human review for eligible conversations, according to the Project Lily reporting. The practice appears broader than one company, but OpenAI now faces direct questions about notice, data minimization, and whether its privacy filter provides enough protection.

What OpenAI Project Lily Reviewers Reportedly See

Project Lily turns selected consumer conversations into structured evaluation tasks for human reviewers.

The reported workflow begins with an actual ChatGPT prompt. A contractor reads the user’s request, summarizes the apparent intent, and evaluates several model-generated answers. The reviewers reportedly receive four candidate responses and score them on a scale from one to seven.

That process is more involved than selecting a thumbs-up icon. Reviewers must identify strengths and weaknesses within an answer, explain their reasoning, and judge whether the response meets OpenAI’s preferred behavior.

The rating instructions reportedly address both substance and tone. Reviewers look for accuracy, relevance, appropriate formatting, and compliance with the user’s request. They also penalize habits that can make generated writing feel mechanical or manipulative.

Those habits reportedly include unnecessary checkmark symbols, cluttered formatting, and endings designed mainly to prolong engagement. Reviewers are also instructed to flag claims of human experience, such as a chatbot presenting itself as someone with a personal history.

Another target is sycophancy, the tendency of a model to excessively agree with a user or reinforce the user’s framing. A sycophantic assistant can sound supportive while failing to question an irrational, mistaken, or dangerous premise.

Human judgment matters here because these faults are highly contextual. A rigid automated metric cannot always distinguish helpful empathy from irresponsible validation. It also struggles to determine when a friendly answer becomes ingratiating or deceptive.

This explains why model developers still require people after spending heavily on computing infrastructure and automated evaluation. A response can be grammatically sound and factually plausible while still feeling evasive, patronizing, or disturbingly agreeable.

However, the same context that helps reviewers judge a response also increases the privacy stakes. An isolated prompt might reveal little. A complete exchange can expose relationships, health concerns, workplace disputes, financial problems, or other identifying details.

Some reviewer tasks reportedly display a user memory summary above the conversation. ChatGPT memory can preserve useful details from earlier interactions so later responses feel more personalized. That summary can also provide extra clues about a person’s occupation, location, interests, or circumstances.

The reviewers reportedly do not receive ChatGPT usernames. OpenAI also says conversations pass through a system called Privacy Filter before reaching them. That automated model is intended to identify and remove personal information.

Yet removing an account name does not necessarily make a detailed conversation anonymous. A distinctive job, unusual medical history, named community, and specific family event can collectively identify someone. Each fragment might appear harmless when inspected separately.

The reported ChatGPT human review process therefore depends on two imperfect safeguards. The first is automated redaction. The second is organizational control over contractors who still need enough context to perform meaningful evaluations.

Secondary coverage describes hundreds of contractors participating in the project. Reports also connect the work with Mercor and Crossing Hurdles, though OpenAI has not published a full operational account.

The investigation does not establish how many total conversations enter Project Lily, how they are sampled, or how long reviewers can access them. It also does not identify which specific model versions receive the resulting feedback.

Those gaps matter. “Humans read ChatGPT chats” suggests universal surveillance, which the available evidence does not support. “Anonymized samples improve models” can sound reassuring, but it understates the reported visibility of whole conversations and memory context.

The defensible conclusion sits between those claims. Selected conversations reportedly enter a human evaluation pipeline, and OpenAI applies privacy protections before review. Those protections do not guarantee that every sensitive detail disappears.

Why ChatGPT Human Review Creates a Consent Problem

The strongest privacy concern is the gap between formal data controls and the expectations created by a conversational product.

Most people understand that public posts can be read by strangers. A chatbot presents a different psychological setting. It offers an empty text box, immediate replies, and no visible human audience.

That interface encourages disclosure. People use ChatGPT to revise confidential emails, interpret medical language, discuss relationships, debug proprietary code, and think through workplace problems. Others treat it as a journal or emotional sounding board.

A user can therefore understand that data helps improve a service without imagining that a contractor might inspect the text. Model training sounds computational. Human evaluation sounds interpersonal.

OpenAI does disclose that consumer content can help improve its models. Its published explanation says the company retains some interaction data while taking steps to reduce personal information before training. It also distinguishes consumer products from business offerings in its model improvement policy.

Project Lily changes how readers interpret those disclosures. “Improve our models” reportedly includes people reading, summarizing, criticizing, and scoring selected conversations. That mechanism deserves clearer attention because it affects what informed consent means.

OpenAI told the investigating publication that sensitive details can pass through its filter. Automated redaction systems face difficult language problems, especially when identifying information appears indirectly.

A filter can recognize an email address more easily than a rare occupation paired with a small town. It can remove a conventional phone number while missing an identifier written in an unusual format. Ambiguous references are harder still.

Context also creates a basic technical tradeoff. Stronger redaction protects privacy but can damage the material reviewers need. Weak redaction preserves evaluation value but increases exposure.

Suppose a user asks ChatGPT to draft a response to a manager following a workplace dispute. Removing names may preserve the basic task. Removing the industry, role, allegation, and prior exchanges might make the response impossible to assess properly.

The same tension applies to health and relationship conversations. Reviewers need context to judge whether an answer is appropriately cautious or dangerously validating. That context may contain the most sensitive parts of the exchange.

Memory intensifies the problem because information can accumulate across sessions. A single detail might not identify someone, while a structured summary of recurring details narrows the possibilities.

This is why privacy cannot depend entirely on conventional personally identifiable information. Effective de-identification must consider combinations, inference, and linkability. A name-free conversation can remain deeply personal.

The report also quotes a contractor who doubted that users imagine someone analyzing their exchanges. That observation is not a scientific measure of user awareness. It does capture the product-design problem, however.

OpenAI can point to a valid setting and an accurate privacy policy. Users can still develop a materially different expectation from the interface. Both statements can be true.

The issue is especially important for vulnerable users. Someone asking a chatbot to keep a secret may interpret its conversational reply as a confidentiality promise. The model cannot change OpenAI’s backend data practices through its answer.

Chatbots are not therapists, lawyers, doctors, or privileged professional advisers. Yet their interfaces increasingly imitate patient, personalized conversation. That resemblance raises the cost of weak or poorly surfaced privacy explanations.

The Federal Trade Commission has warned AI providers to honor their privacy and confidentiality commitments. Its AI privacy guidance specifically highlights sensitive information submitted to model services and the importance of clear notice.

The Project Lily report does not establish that OpenAI violated any law. It does create a test for whether present disclosures are sufficiently clear about human access and contractor review.

A buried policy can describe data processing accurately without correcting the expectation built by the product. The pressure on OpenAI is therefore larger than one leaked project name.

The company must reconcile two messages. ChatGPT is becoming more personal and context-aware, while conversations can also become training material reviewed by people.

Better Responses Depend on Context, and Context Raises the Risk

Project Lily exposes a difficult tradeoff: the most useful material for improving ChatGPT can also be the hardest material to anonymize.

Model developers need examples that resemble real use. Synthetic prompts can cover known scenarios, but they rarely reproduce the disorder, ambiguity, and emotional texture of millions of genuine conversations.

Real users change direction, omit context, contradict themselves, and mix several tasks. They also respond to the model’s earlier wording. Reviewing a complete exchange helps evaluators see whether ChatGPT caused confusion or reinforced a bad premise.

That feedback can address problems that benchmark scores miss. A model might answer a factual test correctly while frustrating users through repetitive disclaimers. Another might sound compassionate while endorsing a harmful belief.

Human reviewers can identify these distinctions. Their ratings can support post-training, the process used to shape a pretrained model’s behavior after it learns broad language patterns.

The Project Lily instructions reportedly focus partly on making ChatGPT less sycophantic and less prone to human-like self-presentation. Those goals directly concern safety and user trust, not merely writing style.

Removing every personal clue before review can weaken that assessment. An evaluator judging sensitive advice needs to know what the user disclosed and how the conversation developed.

This creates an uncomfortable reversal. Personal context improves the reviewer’s ability to identify unsafe behavior, but the same context makes human exposure more consequential.

The tension becomes sharper as AI assistants gain memory. A memory system can help ChatGPT avoid repeated questions and tailor later replies. It also converts scattered disclosures into a persistent profile.

That profile can contain ordinary preferences alongside highly sensitive material. Even when no direct identifier appears, the combination may reveal a user’s approximate location, profession, family structure, or ongoing problems.

This is not an argument that memory or human evaluation must disappear. It is an argument for treating contextual data as sensitive by design.

The safest model improvement pipeline would minimize what each reviewer sees, separate identity from content, restrict access, log every interaction, and enforce short retention periods. It would also test whether redaction performs consistently across languages and unusual identifiers.

OpenAI has not publicly supplied enough Project Lily details to evaluate those controls. The report says usernames are withheld and a Privacy Filter processes conversations. It does not provide an independent security audit or measured failure rate.

The absence of those details should not be converted into evidence of failure. It should be treated as an unresolved verification gap.

Contractor arrangements deserve particular scrutiny because they expand the number of organizations and workers involved. Every additional access point creates operational questions about training, supervision, device security, copying, retention, and incident response.

A contractor can follow strict rules while still seeing sensitive material required for the task. Privacy risk does not require misconduct. Authorized access itself can conflict with a user’s expectations.

This distinction often disappears in consumer privacy debates. Security asks whether unauthorized people obtained data. Privacy also asks whether authorized use was proportionate, understandable, and consistent with the context of collection.

OpenAI Project Lily appears designed to improve model quality through legitimate evaluation work. The controversy concerns whether the data pipeline matches the intimacy of the product experience.

For individuals, this is also a knowledge-management problem. Drafts, research notes, and workplace details can reveal more when combined than when stored separately. A private AI knowledge base needs explicit boundaries around where its source material travels.

Users should not assume that deleting names makes a prompt safe. Proprietary code can reveal a company. A contract excerpt can identify parties through dates and obligations. A medical narrative can expose someone through rare circumstances.

The better rule is data minimization before submission. Provide only the details necessary for the task, replace identifiers locally, and avoid pasting secrets into consumer tools without appropriate controls.

That approach does not excuse unclear platform disclosures. It acknowledges that no automated privacy filter offers a complete substitute for careful product design and user judgment.

OpenAI’s Privacy Controls Matter, but They Have Boundaries

Users can prevent new consumer conversations from being used for model improvement, although that choice does not eliminate every form of review or retention.

OpenAI’s current ChatGPT privacy controls provide a direct opt-out. Signed-in users can open Settings, choose Data Controls, and disable “Improve the model for everyone.”

According to the official data controls guide, conversations remain visible in chat history after the setting is disabled. New conversations are not used to train ChatGPT.

The setting applies across devices for the same account. OpenAI also says prior opt-out requests submitted through support or its privacy form remain honored.

This distinction between new and existing data is important. Turning off model improvement changes how future conversations are handled. Users should not assume the switch retroactively removes material already processed or incorporated into a training workflow.

Temporary Chat offers another boundary. OpenAI says an unsaved temporary conversation does not appear in history, does not create new memories, and is not used for model improvement.

The company may retain a copy for up to 30 days for safety purposes. OpenAI also says temporary conversations may be reviewed to monitor abuse.

That means temporary does not equal invisible, instantly deleted, or exempt from every human-access pathway. It means the conversation follows a more restrictive product-improvement and memory policy.

A temporary conversation can also use existing memories when personalization remains enabled. Users can choose a non-personalized temporary session when they do not want prior memories or custom instructions applied.

If a user saves a Temporary Chat, it becomes a regular conversation. It then follows the account’s model-improvement and personalization preferences.

These details make ChatGPT privacy controls useful but easy to misunderstand. Three separate concepts are involved:

• Model improvement determines whether eligible content supports training and evaluation.

• Memory determines whether information can personalize later conversations.

• Safety monitoring permits limited retention or review for abuse detection.

Changing one does not necessarily disable the others. A user who turns off memory has not automatically turned off model improvement. A user who chooses Temporary Chat has not received an absolute guarantee against safety review.

Business products follow a different default. OpenAI says it does not train on inputs or outputs from ChatGPT Business, Enterprise, Edu, healthcare offerings, teachers’ products, or its API platform unless an organization explicitly opts in.

That distinction matters for companies deciding whether staff can place internal documents into an AI system. A consumer account and an approved business workspace do not carry identical data commitments.

Even business protections do not remove the need for governance. Organizations must still control access, connected applications, retention, and employee handling of customer information.

Individual users should also separate convenience from confidentiality. A chatbot can help rewrite a sensitive message without receiving the original names, employer, addresses, or account numbers.

Developers can use fabricated values while debugging. Researchers can summarize a case before asking for structural feedback. Employees can follow approved workplace tools and avoid transferring restricted material into personal accounts.

For recurring private work, a local-first second brain can reduce unnecessary copying between systems. The key question is not simply where information is stored, but which services receive it during processing.

Project Lily also reveals why interface-level warnings matter. Users should not need to infer human access from a broad phrase about model improvement.

A clear disclosure could explain that selected conversations may be reviewed by trained personnel or service providers after automated privacy processing. It could appear near the model-improvement toggle rather than only inside policy documents.

The interface could also remind users before they enable memory or upload sensitive files. More granular controls could separate automated training, human evaluation, and safety review where operationally possible.

None of these controls would make a cloud chatbot equivalent to legally privileged communication. They would make the boundaries more visible and easier to understand.

Human Review Is an Industry Practice, Not an OpenAI Exception

OpenAI faces the current spotlight, but human evaluation is embedded across consumer AI development.

Google’s Gemini documentation offers the clearest public comparison. Google says trained service providers review a subset of chats to identify low-quality, inaccurate, or harmful responses.

Google also says reviewed chats are disconnected from user accounts before reaching service providers. Its Gemini privacy hub warns users not to enter confidential information they would not want a reviewer to see.

That warning makes the human role unusually explicit. Google says reviewed conversations and associated information can be retained for up to three years, even after a user deletes related activity.

Users can turn off the Keep Activity setting to stop future chats from being used to improve Google’s services. Temporary chats are also excluded from model improvement, although limited safety processing can continue.

Anthropic told 404 Media that it also uses human review for conversations made eligible through its improvement setting. According to the reporting, Anthropic removes account identifiers, including email addresses, before review.

These comparisons do not establish that every company collects the same material or uses identical controls. They show that human evaluation is not a temporary workaround unique to OpenAI.

Language models depend on judgments that automated systems cannot fully supply. People evaluate whether answers are helpful, accurate, safe, appropriately cautious, and aligned with complicated instructions.

The competitive pressure is obvious. Companies want assistants that feel natural without pretending to be human. They want emotional intelligence without reflexive agreement. They want personalization without unacceptable privacy exposure.

Each goal pushes against another. More context can improve relevance. More retained context can increase sensitivity. More human judgment can improve nuanced safety. More human access can weaken user confidence.

The industry’s preferred answer has been de-identification, access controls, contractor agreements, and opt-out settings. Project Lily tests whether those measures remain persuasive as assistants become more intimate.

Competitive pressure can create another problem. Model quality depends partly on diverse, current examples of real use. A provider that sharply limits review data might lose valuable evidence about emerging failures.

That does not mean privacy and performance are mutually exclusive. It means privacy-preserving alternatives need serious investment and measurable validation.

Companies can improve synthetic evaluation, use on-device classification, limit reviewers to narrow excerpts, or employ controlled internal teams for the most sensitive tasks. They can also publish aggregate information about sampling and redaction performance.

Independent audits would help distinguish strong protections from reassuring language. Useful disclosures could include reviewer locations, access duration, redaction testing, incident rates, and whether memory summaries are necessary for each task.

OpenAI might also explain whether conversations are sampled randomly, selected by automated systems, or chosen because users provided feedback. Each mechanism creates different expectations and potential biases.

Sampling design affects model quality too. Conversations selected by user feedback may overrepresent failures and unusually engaged users. Random samples might capture ordinary behavior but expose material from people who never actively submitted feedback.

The Project Lily reporting does not resolve those questions. It identifies the people and mechanisms behind a category previously described mostly as “model improvement.”

That visibility is valuable even if human review remains necessary. Consumers can make better choices when the labor behind an AI service is not hidden behind technical language.

The same applies to contractor working conditions. Evaluators can encounter distressing, sexual, violent, or deeply personal content. Responsible programs need mental-health protections, clear escalation procedures, and realistic productivity expectations.

Those labor questions should support the privacy analysis, not replace it. The central opponent remains OpenAI’s promise of controlled model improvement versus the reality of contextual human access.

What Project Lily Still Has to Prove

The next phase should be judged through disclosure, verifiable privacy performance, and changes to the reviewer interface.

The first signal is whether OpenAI publishes a direct explanation of Project Lily. A meaningful statement would describe which conversations qualify, how consent settings apply, and what contractors can view.

Confirmation of the reported workflow would strengthen the conclusion that human review is a routine component of consumer model improvement. A narrow denial or substantial correction would weaken parts of the current account.

Silence would leave the evidence dependent on leaked materials and anonymous worker testimony. That material is significant, but it cannot answer every operational question.

The second signal is whether OpenAI provides measurable information about Privacy Filter. Users need more than a statement that personal information is reduced before training or review.

Useful evidence would include error testing across languages, uncommon identifiers, indirect references, and long conversations. Independent evaluation would carry more weight than a general description from the company.

A low demonstrated failure rate would support OpenAI’s claim that it meaningfully limits reviewer exposure. Evidence of inconsistent redaction would strengthen demands for narrower sampling and stronger controls.

The third signal is whether the reviewer interface changes. Memory summaries deserve particular attention because they can combine details across conversations and make re-identification easier.

OpenAI could limit those summaries to tasks that genuinely require personalization context. It could provide reviewers with only the minimum relevant memory or replace personal details with abstract categories.

A visible notice near ChatGPT’s model-improvement setting would also matter. It should plainly state that selected conversations can receive human review after privacy processing.

Regulators will watch the relationship between those disclosures and actual practice. The FTC has already made clear that AI companies must honor privacy promises and avoid concealing secondary data uses.

For users, the immediate response is simpler. Review ChatGPT privacy controls, disable model improvement when appropriate, and use Temporary Chat with a clear understanding of its safety exception.

Do not submit passwords, financial credentials, private keys, unredacted medical records, or confidential employer data. Replace names and identifying details before requesting assistance.

Organizations should verify that workers use approved business products rather than personal consumer accounts. They should also document which information categories can enter AI systems and which must remain excluded.

OpenAI Project Lily does not prove that every ChatGPT conversation reaches a contractor. It does show why a private-looking interface should never be mistaken for a private room.

Human review can improve accuracy, tone, and safety. It can also expose the personal context that makes those judgments possible. The industry must explain that exchange in language ordinary users can understand.

Check your current settings before your next sensitive conversation. Then ask a harder question: would the prompt remain acceptable if a trained reviewer saw it without your username? If the answer is no, remove the identifying context, choose a product with stronger data commitments, or keep the material outside the chatbot.