As use of artificial intelligence increases, AI assistants(nouvelle fenêtre) are becoming trusted companions. Millions of people turn to chatbots like ChatGPT, Claude, Gemini, DeepSeek, and others to ask questions and solve problems. Their queries can be deeply sensitive, asking about health concerns, sharing client data, or seeking emotional support and creative inspiration.
For many, these chatbots feel human, responsive, and confidential. But recent developments (A legal order has compelled OpenAI to preserve ChatGPT output data(nouvelle fenêtre) indefinitely, and Sam Altman, the founder of OpenAI, says ChatGPT does not offer “legal confidentiality(nouvelle fenêtre)”) shine a light on the risk that every word you type could be logged, stored, analyzed, and possibly misused or leaked to third parties.
Here’s what that could mean, and why it should concern us all.
The stakes are higher with AI
The rise of AI assistants feels like a new frontier, but in many ways, we’ve been here before. The concerns surrounding AI chat logs echo those that surfaced in the early days of search engines, many of which proved well-founded. Search logs have been used to build lucrative ad-targeting profiles, have been exposed in data breaches, and have been subpoenaed in court.
With AI, the stakes are higher because the data is more personal and detailed. Increasing numbers of people are turning from search engines to AI chatbots for their answers, with predictions that search engine usage will drop by 25% by 2026(nouvelle fenêtre). Unlike a search bar, which typically captures short, fragmented queries, questions to chatbots are framed as if the askers are talking to another person, revealing much more about themselves in the process. A single conversation might contain personal thoughts(nouvelle fenêtre), emotional struggles, health concerns(nouvelle fenêtre), financial fears, or even full names and addresses.
In short, chat logs offer an incredibly rich and intimate map of your identity, more so than search engine logs — and with far greater implications for your privacy. Big Tech could gain access to your chat logs and use them to build a psychological profile on you that can be used for targeted advertising or even disclosed to the government.
Your ‘anonymous’ chats won’t always stay that way
A common argument against worrying about chat log privacy is that the sheer volume of data makes individual exploitation unlikely, especially when users aren’t logged in. But recent history tells a different story.
In July, reporters found that more than 100,000 conversations in ChatGPT(nouvelle fenêtre) were indexed by Google and made searchable. Users clicking the “share” button to send a conversation to friends or colleagues almost certainly didn’t realize their private conversations would be visible to everyone on the internet.
This example is a warning: So-called “anonymous” data is rarely anonymous. Paired with IP addresses, timestamps, device fingerprints, and session and cookie IDs, these logs become far more identifiable.
How chats can expose our inner lives
The popular AI platforms ChatGPT(nouvelle fenêtre), Gemini(nouvelle fenêtre), and DeepSeek(nouvelle fenêtre) all state that user chat logs are used to help improve model performance and are not used to “sell” your data for advertising or user profiling. But policies can change overnight, as OpenAI’s legal battle shows.
For example, Anthropic updated its privacy policy(nouvelle fenêtre) without warning in August 2025 to extend the data retention period. With logs stored indefinitely, the personal data that once seemed off limits could suddenly become a lucrative source of profits.
It wouldn’t be the first time that happened. When Google launched in 1998, co-founders Sergey Brin and Larry Page warned(nouvelle fenêtre) that advertising-based search engines would inevitably be “inherently biased towards the advertisers and away from the needs of the consumers”. A decade later, Google began using search histories to target ads(nouvelle fenêtre).
With direct access to people’s inner lives, AI logs can push ad targeting to its limits. With increasing numbers of users turning to chatbots for emotional support(nouvelle fenêtre), someone may confide:
- “I’m feeling overwhelmed and alone lately.”
- “Do you think I have depression?”
- “What can I do if I can’t pay rent next month?”
These statements aren’t just keywords or a collection of interactions, they’re clear emotional and situational signifiers. If stored and analyzed, they could be used to build psychological profiles and manipulate user decisions.
It’s not alarmist to suggest that such intimate, sensitive information could be used for exploitative purposes — because the Big Tech companies have done exactly that. In 2017, Facebook demonstrated that no interaction is off-limits by exploring ways to target emotionally vulnerable teenagers(nouvelle fenêtre) with ads during moments of insecurity.
Chatbot logs are a window into your psyche. And that level of access raises significant privacy and ethical concerns in a sector that does not yet have clear guidelines and regulations on data usage and application.
The legal risks for users
Search history has long played a role in criminal investigations and court cases. There are multiple instances in which online queries contributed to guilty verdicts, notably Moira Akers vs State in the US.
After experiencing a stillbirth in 2018, Akers was charged with murder and in 2022 was sentenced to 30 years in prison. A key part of the prosecutor’s case was Akers’ internet searches about abortion, conducted during her early stages of pregnancy. According to the prosecution, these searches served as evidence of Akers’ intent — despite the fact that she carried the pregnancy to full term. The Supreme Court of Maryland overturned her conviction in 2025, ruling the search query evidence inadmissible(nouvelle fenêtre). But Akers had already spent almost three years in prison. The case highlights how stored logs can be used to tell a story to fit an unfavorable narrative, even if there is evidence that contradicts it.
The stakes could be even higher with AI stored chats that may be subject to legal evidence. Unlike search queries, these interactions are more conversational and even speculative, sometimes as a way to prompt a certain type of response from the LLM, or large language model. A private, innocent exchange with an AI could later be interpreted as incriminating in a legal setting, regardless of intent at the time.
A powerful tool for manipulation
Those who can see chatbot logs can gain access to detailed behavioral blueprints: personal habits, emotional triggers, relationship dynamics, and even professional vulnerabilities. In the wrong hands, this data is a powerful tool for social engineering, which is a common hacking tactic involving psychological manipulation.
This risk is far from hypothetical. In January 2025, the AI platform DeepSeek left a massive database exposed online(nouvelle fenêtre). The breach included over one million AI chat logs and API keys, all publicly accessible to anyone who knew where to look. Among the exposed data were plain-text conversations that may have contained discussions about family, finances, and confidential projects — information that could easily be exploited for fraud, blackmail, or identity theft.
Once leaked, this data can be weaponized to impersonate you or manipulate you. By referencing personal details drawn from past conversations, attackers can gain your trust and exploit it to create serious financial or security vulnerabilities.
The specter of political surveillance
In authoritarian regimes, the consequences of stored digital interactions can be far more severe. For example, several countries have banned China-based AI platform DeepSeek from use on government devices(nouvelle fenêtre) over security concerns about how users’ personal information is handled by the operator. South Korea, Australia, and Taiwan raised concerns that this data could be accessed by Chinese authorities and potentially used for surveillance.
For dissidents, activists, or journalists, this isn’t just a privacy issue, it’s a matter of safety. A seemingly casual chat about protest rights, media laws, or asylum routes could be interpreted as incriminating evidence in jurisdictions with limited protections for free speech and civil liberties.
A threat to intellectual property
In 2023, Samsung engineers unintentionally uploaded proprietary source code(nouvelle fenêtre) to ChatGPT while trying to debug an issue, under the mistaken belief that their queries were private. Once submitted, that data became part of OpenAI’s training system, sparking concerns about the security and ownership of intellectual property shared with AI tools.
A related concern emerged in the high-profile copyright case brought by The New York Times against OpenAI(nouvelle fenêtre) mentioned at the start of this article. The case prompted a court order requiring OpenAI to preserve all ChatGPT output data indefinitely after allegations that the company used millions of the newspaper’s articles to train its models without permission. Though the case centers on OpenAI’s web crawlers scraping news content, it raises broader questions for creatives — writers, designers, musicians, and others — who contribute original work to AI platforms.
If user-submitted content is retained and reused by the model, there’s a real risk that proprietary or creative material could be redistributed to other users without attribution or consent. This blurs the line between user contribution and AI training data, and it raises urgent questions about authorship and ownership in the age of generative AI.
What needs to change and how to protect yourself
Most people want and expect privacy in their real lives. We believe the same expectations should extend to people’s digital lives. We built a suite of encrypted services that make online privacy accessible for anyone, whether you’re emailing your friends, streaming your favorite shows(nouvelle fenêtre), or backing up photos. It’s not too late to correct the early mistakes of the internet, and with AI in its early days there’s still time to chart a new course for this technology that respects your privacy.
We built our AI assistant(nouvelle fenêtre) Lumo in a way that upholds key protections by design:
- No data logging: Users should be able to engage with chatbots without concern that their submissions will be stored, used for profit, or used against them.
- No model training: Private chats should never be exploited as training data, because it runs the risk of reappearing in the model’s outputs later.
- Transparency in data handling: It should be clear how stored user data is used, who can gain access to it, where it is stored, how long it is retained, and how it can be deleted.
- Zero-access encryption by default: Conversations should be secured by using the user’s encryption keys, so even the AI provider can’t gain access.
While the dangers of chatbots that quietly log our data are real and pressing, we are not powerless. By choosing privacy-first AI tools and demanding transparency and accountability from their creators, we can shift the balance of power back into the hands of users, affirming the belief that privacy is a right, not a privilege.
