An AI Data Privacy Cautionary Tale: Court-Ordered Data Retention Meets Privacy
“Data systems are designed to remember data, not to forget data.” - Debbie Reynolds “The Data Diva”
The rapid rise of generative artificial intelligence technologies and their increasing usage have brought both breakthrough innovation and mounting data risks. I have spent years telling individuals and organizations in my keynotes and writings not to submit personal, sensitive, confidential, or proprietary data into generative AI systems. Why? Once data enters these systems, organizations and users often lose control over what happens to it. As I have said time and again, “Data systems are designed to remember data, not to forget data.” That reality has now been confirmed by a federal court order in the case of New York Times v. OpenAI.
In a significant development with far-reaching implications, the United States District Court for the Southern District of New York recently ordered OpenAI, the company behind ChatGPT, to preserve all user interactions across its AI products. This includes chat logs, input prompts, and generated outputs from both individual users and businesses accessing the system via API. The order also suspends all deletion mechanisms previously in place, including those that allowed users to request the removal of their data. OpenAI has publicly acknowledged the implications, calling the order a “privacy nightmare” (Ars Technica article). The full preservation order, dated May 13, 2025, is publicly available (court order PDF).
This court decision serves as a cautionary tale, illustrating what can happen when privacy, litigation, and artificial intelligence systems intersect. It confirms that the protections many users believe they have when interacting with AI platforms can evaporate the moment those platforms become entangled in legal proceedings.
When “Delete” Does Not Mean “Delete”
Many users assume that deleting data from a platform actually removes it. In reality, deletion often removes only the visible or user accessible layer, while other parts, such as system backups, logs, caches, and training artifacts, remain intact. In the context of generative AI systems, this becomes even more complicated. These models may absorb patterns from user inputs and retain them in ways that cannot be undone, even if the user-facing chat log is cleared.
Now, with the court’s order in effect, OpenAI is legally required to suspend deletion capabilities altogether. All user data, past and future, is to be preserved. This includes information that users may have intended to be temporary or confidential in nature. The decision affects every user of ChatGPT and API-based integrations, regardless of location or purpose.
This development is a stark reminder that the legal system can override user expectations of privacy. It also demonstrates that generative AI platforms store data in a manner that is inherently more persistent than other types of data systems. Once information enters these systems, users may not be able to retrieve, control, or erase it in the conventional sense.
API Business Users Are Not Exempt
Many businesses mistakenly believe that using the API version of ChatGPT (or other AI tools) offers greater protection separate from the general user data pool. Some assume their enterprise integration insulates them from broader data retention or usage. That belief has now been proven incorrect with this court order.
This data preservation order applies to all ChatGPT interactions, including those submitted through API calls and those generated by the API. This means any business using ChatGPT for internal tools, customer interactions, or employee-facing applications must now assume that every prompt and response is being logged, stored, and held under court authority.
Even if your company has internal policies forbidding the sharing of proprietary or sensitive data, any accidental or uninformed submission is now preserved indefinitely. This includes trade secrets, legal inquiries, financial data, health-related information, and other protected categories. These submissions are no longer just internal issues. They may become part of a litigation archive beyond your control.
This presents serious challenges for legal compliance, data governance, and risk management. Companies must now proactively manage the use of AI across their workforce and supply chain. They should also review vendor contracts, privacy policies, and technical safeguards with renewed attention and diligence.
When Privacy Meets Litigation
Data privacy laws, such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), grant users significant rights. These include the right to access, correct, and delete their data. But once data becomes relevant to litigation, courts can impose preservation obligations that override those privacy rights. This has always been the case; however, with AI in the picture, companies are sharing even more of their data with AI systems, which increases their data risks.
That is what has occurred here. The court has directed OpenAI to preserve all data that might be relevant to the case brought by The New York Times. In doing so, the order effectively renders OpenAI unable to comply with user deletion requests. OpenAI itself has voiced concern that this raises the likelihood of privacy violations and increases the risk of data breaches.
This scenario presents a troubling conflict between the principles of legal due process and data protection. When data collected by AI tools is drawn into litigation, users lose the ability to manage, control, or know where their information resides. This is especially concerning in cases where individuals are not parties to the lawsuit but are nonetheless swept up in the preservation mandate.
Businesses Must Reassess Their Risk
Companies that use generative AI tools, especially through API access, are now on notice. This court order shows that simply using an API does not shield an organization’s data from external retention or legal exposure. Any prompt submitted, even through automated or backend systems, can be preserved under court order without the company’s awareness or consent.
The consequences are severe. A business might find itself in violation of internal privacy policies, client contracts, or regulatory obligations simply because it submitted data into a third-party AI tool that is now frozen under litigation. Even organizations that believed they were minimizing risk could now face unintended legal or reputational fallout.
To respond, businesses must:
Governance must transition from theory to practice. That includes building safeguards into development environments, sales tools, HR systems, and customer engagement platforms that rely on AI.
No One Is Immune
The preservation order in New York Times v. OpenAI sends a powerful message. Whether you are a casual user, a developer, or a multinational business, your data is not beyond reach. Once entered into a generative AI system, your input can become part of a legal process that is completely outside your control.
Courts will not stop to ask whether you expected your data to be deleted. They will only ask whether the data might be relevant to a case. If the answer is yes, the platform is obligated to retain it, even if that creates direct conflicts with privacy regulations or customer promises.
This moment highlights the fundamental problem with relying on deletion as a means of safeguarding privacy. I have long said that AI systems are designed to remember, not to forget. That is not a flaw in the code, it is a core function of how these systems operate. And now, courts are enforcing that persistence in ways that users never anticipated.
Prioritizing Internal Controls and Data Discipline
Rather than waiting for regulators or courts to define boundaries, companies should adopt disciplined internal data practices now. This includes curating what information is allowed into AI systems in the first place. Businesses must treat every prompt or data entry as a potential record that could be retained, exposed, or discoverable.
Good data hygiene starts with defining clear policies. Who is authorized to use generative AI? What types of data are permitted? What business functions are off-limits? These questions should not be left to informal guidance or after-the-fact corrections. They should be codified into enforceable internal policies, with training, technical enforcement, and routine audits.
Organizations should also require AI vendors to offer usage transparency. What is being stored? For how long? Is it retrievable or exportable? Can organizations review and purge data associated with their account?
Just as businesses learned over time to restrict sensitive data from being emailed, stored in the cloud, or shared across insecure channels, they now need to build new muscle memory for AI usage. This is not a hypothetical compliance exercise. It is a material protection of intellectual property, trade secrets, customer trust, and legal exposure.
Moving Forward
For users, the lesson is clear. Never assume your interaction with an AI system is private or temporary. Do not share personal, medical, financial, or business-critical information unless you are willing for it to be stored indefinitely. For organizations, the stakes are even higher. You must now govern AI use with the same rigor you apply to legal, financial, and compliance systems.
The court’s preservation order is a cautionary tale. It confirms what I have warned about for years on stages around the world. The permanence of AI data is not just a technical design issue; it is also a legal and operational risk. If we fail to act by implementing strong internal controls, educating users, and being intentional about how we engage with AI, we may find ourselves exposed in ways that cannot be reversed. Taking the right next steps will help your organization make data privacy a business advantage.
Need a Keynote Speaker on "Data Privacy", Data Protection, and Technology issues? View our keynote speaker page for popular talks and topics. Ready to speak to "The Data Diva" about your speaking event? Fill out our speaker request form and Schedule a call now.
Debbie Reynolds "The Data Diva" Keynote Addresses
I'm thrilled to extend my heartfelt thanks to TikTok, US Senate, Volkswagen Credit, USDA, Ally Financial, National Grid, Lawrence Livermore National Laboratory, Northwestern Mutual, PayPal, Coca-Cola, FRTIB, Hewlett Packard Enterprises, WestRock, Capital Group, Johnson & Johnson, Uber, S&P Global, FDIC, DHL Supply Chain, The Erikson Institute, and Rubrik for the privilege of being your Keynote Speaker. Your commitment to innovation and excellence is inspiring, and I'm honored to have contributed to your events.
👉 Ready to elevate your Data Privacy and Emerging Tech game? Contact me today for a 15-minute call to book your team's virtual or in-person customized keynote or workshop session! Reply to this post or Direct Message me for details. Here is the #1 most requested Data Privacy Keynote and Workshop for 2025:
"Data Privacy and the Three Pillars of Human-Centric Data Use"
The Pact Data Privacy Trust Framework
Debbie Reynolds, "The Data Diva", launched the PACT "Data Privacy" Trust Framework & Scorecard. This Framework can evaluate regulatory and business risk and the Trust of individuals around "Data Privacy". It is a gut check for organizations of all sizes to rate and triage their "Data Privacy" challenges. This Framework addresses Purpose, Alignment, Context, and Transparency. Watch this video to learn the basics as Debbie Reynolds explains the PACT Data Privacy Trust Framework & Scorecard in 6 minutes.
Visit our website to learn more about the PACT Data Privacy Trust Framework & Scorecard.
🌍 Debbie Reynolds and "The Data Diva" Talks Privacy podcast has reached a major milestone - 741,177+ downloads as of June 2025! 🎧
"If anybody knows where to go for the types of conversations that they need to have on Data Privacy, you are that person. You are a rock star in this space." – Dr. Nicol Turner Lee
🙏 I want to thank our amazing listeners from over 128+ countries and 2899+ cities worldwide. Your support and enthusiasm have been truly extraordinary! Additionally, I would like to acknowledge the 14,700+ subscribers to The Data Privacy Advantage Newsletter, who faithfully read, comment on, and share our work. 🌎🌍🌏
Here are more of our accolades:
Watch a video short of our podcast The Data Diva E240 - June 10, 2025 - Ian Glazer , Vice President of Product Strategy, SGNL, Product Executive and Digital Identity Expert. Here is a sneak preview of our Data Diva Podcast guests:
Listen and subscribe to the new weekly episodes of "The Data Diva" Talks Privacy Podcast.
Introducing Debbie Reynolds “The Data Diva” Power Play Series: Volume #6 –"Data Guardians: Ethics, Inference, and the Future of Data Privacy"
💃🏽 Introducing Debbie Reynolds “The Data Diva” Power Play Series: Volume #6 –"Data Guardians: Ethics, Inference, and the Future of Data Privacy""The Data Diva Power Play", a new monthly YouTube series where we dive deep into our award-winning “The Data Diva” Talks Privacy Podcast vault. This series highlights essential conversations with global experts that business leaders need to know about Data Privacy and Emerging Technology topics that matter most RIGHT NOW.🔍 The Data Diva Power Play features four high-demand, timely podcast episodes showcasing actionable insights and groundbreaking discussions. Playlist Length: 3 hours, 10 minutes, 19 secondsIn Volume #6: "Data Guardians: Ethics, Inference, and the Future of Data Privacy", We explore:
🎯Jeff Jockisch (Data Brokers & Data Privacy)Debbie Reynolds and Jeff Jockisch dissect the opaque world of data brokers, exploring how unregulated data aggregation harms individuals and businesses. They advocate for purpose-driven data collection laws and discuss Jeff’s groundbreaking dataset tracking brokers’ unethical practices.
🎯David Kruger (Cybersecurity & Data Control)Debbie and David Kruger go into self-defending data technologies, emphasizing encryption and cryptographic controls to secure information even when breached. They critique outdated “castle-and-moat” security models and highlight the urgency of global standards for third-party data transfers.
🎯Jennifer Pierce, PhD (Ethical AI & Human-Centered Tech)Debbie and Jennifer Pierce tackle the ethical pitfalls of AI, arguing for decentralized, human-centered innovation to combat bias and corporate monopolies. They stress the need for psychological safety in tech workplaces and data sovereignty as a fundamental human right.
🎯Kurt Cagle (Semantic Web & Data Bias)Debbie and Kurt Cagle unpack the Semantic Web’s power to structure data for smarter AI, while warning of biases in rigid classification systems. They debate the ethics of inference-driven profiling and call for transparency in how algorithms shape societal narratives.📺 Join the ConversationWatch or listen to all episodes of Volume #6: "Data Guardians: Ethics, Inference, and the Future of Data Privacy"on our YouTube playlist, and stay tuned for more impactful discussions designed to help companies thrive.🎥 Playlist Link: https://lnkd.in/dVKYysjx
Empower your organization to master the complexities of Privacy and Emerging Technologies! Gain a real business advantage with our tailored solutions. Reach out today to discover how we can help you stay ahead of the curve. 📈✨
The Data Diva Talks Privacy Podcast offers podcast sponsorships. Each level reflects a different degree of involvement and support for the podcast, catering to a wide range of sponsors from different sectors of the privacy community. If your organization is interested in exploring podcast sponsorship, please contact us!
In addition, and by popular demand, we have expanded our Influencer offerings to include:
Do you need a Data Diva Exclusive? Courtesy of Data Diva Media and "The Data Diva", in cooperation with our podcast's generous supporters, I am happy to share some valuable exclusives with our newsletter subscribers.
Do you need and AI Literacy Course? This course, offered by AI & Partners and led by instructor Dr. Alexandra Belias, provides an in-depth exploration of the EU AI Act, offering insights into its regulatory framework and implications for businesses and organizations. Participants will gain a clear understanding of compliance requirements and risk management strategies related to artificial intelligence. Sign up here to access the course: EU AI Act Course or visit the full link: https://course.ai-and-partners.com/courses/EUAIAct?ref=3b38e9.
Technics Publications has graciously offered a Data Diva Promotion. Anyone who uses the coupon code TheDataDiva receives 20% off. The Promotional code is good for all books on the website, except DMBOK books. Visit the Technics Publications website now to take advantage of this offer.
Need a publication discount on Data Privacy books and digital products? Purchase any products (including Data Privacy books) from the Manning Publications website, and you can use The Data Diva's permanent 35% discount code (good for all our products in all formats) using the following code at checkout: poddatadiva22
Need a VPN, Internet Controls, and Virus Protection? Data Diva Podcast alumni guest for episode 60, Brad Hawkins, CEO of SaferNet, has a special offer! SaferNet provides a very easy-to-use 3-in-1 device-level Cyber Safety protection solution, including an award-winning VPN, Internet Controls, and Virus Protection. SaferNet is ideal for individuals and small to medium-sized businesses who want reliable data protection. "The Data Diva" herself loves the product! Go to https://www.safernet.com/ and buy an annual SaferNet plan for 25% off, which can be paid monthly or annually using the case-sensitive code: datadiva
See our recently featured five-minute videos on Data Privacy from The Data Diva:
Do you want to see more original video content on emerging Data Privacy topics? Subscribe to our YouTube channel to get notified about each week's new video.
🚨 Join me on Wednesday, June 18, 2025, at 12 PM ET / 9 AM PT for a critical webinar hosted by Piwik PRO on personal health data privacy: Navigating Personal Health Data – HIPAA, Consumer Privacy Laws, and Emerging Challenges. Health data now extends far beyond hospitals, flowing through apps, wearables, online activity, and AI tools, raising urgent questions about regulation, trust, and risk. I will be joined by Mateusz Krempa , CCXO at Piwik PRO, as we explore why HIPAA is no longer enough, how consumer privacy laws are evolving, and what organizations can do to stay ahead. If you care about data, tech, health, or compliance, this conversation is for you. 🔗 Register here: https://campaign.piwik.pro/webinar-navigating-personal-health-data/ — I look forward to your questions and your presence. 🧬📱
Join Debbie Reynolds at MyData 2025 – Helsinki, Finland Debbie Reynolds, “The Data Diva,” will deliver a keynote address at the MyData 2025 Conference, taking place September 24–26, 2025, with pre-conference events on September 23. This globally renowned event explores human-centric approaches to personal data and digital trust. Join Debbie and fellow leaders in Helsinki, Finland, for three days of high-impact discussions on privacy, data governance, and emerging tech. Learn more and register: https://conference.mydata.org/
🎤 We’re thrilled to welcome Debbie Reynolds – The Data Diva – to the keynote stage at hashtag#MyData2025!A globally recognised voice in hashtag#DataPrivacy and emerging technology, Debbie brings over two decades of insight to the table. She’s advised major organisations like TikTok, Coca-Cola, PayPal, and Uber, and her thought leadership has been featured in The New York Times, Forbes, and Wired. 💡 Debbie also hosts the award-winning podcast “The Data Diva Talks Privacy”, with listeners in 120+ countries. We’re honoured to have her join us this September in Finland to help shape the next decade of hashtag#ethical, human-centric data and hashtag#AI.🗓️ Explore the programme and grab your Early bird ticket:👉 https://shorturl.at/rh2wD
Join “The Data Diva” in France this fall! I will be delivering a Keynote Address at the AI ON US Conference on October 15 and 16, 2025, in stunning Biarritz, France, hosted at the iconic 5‑star Hôtel du Palais & Casino on the Bay of Biscay. AI ON US is more than a summit; it’s a high-impact, intimate gathering built for visionaries. We’ll convene 100 C-level leaders, along with over 20 global experts in AI law, policy, ethics, and innovation. This isn’t a typical AI conference. Attendance is deliberately small and selective for a reason; only 100 executive seats are available. To claim your spot, book a 15-minute “program fit” call with the AI ON US team: 👉 https://www.ai-on-us.com/ When you book, please mention Debbie Reynolds, “The Data Diva” and I look forward to meeting you all in person.
Many thanks to the press organizations and reporters who seek my commentary on important events around Data Privacy. Also, here are links to some of my other media collaborations. Here is a collection of a few of my 2025 media mentions and collaborations:
Please see our website's media mention section for a full list of media mentions.
🚨 Introducing the “Deep Dive - The Data Privacy Advantage Podcast! 🚨Curated by Debbie Reynolds, “The Data Diva”I am happy to introduce a brand new Data Diva Media podcast series exclusively for YouTube and LinkedIn:🚨This AI podcast, created using the experimental Google Notebook LM audio overview feature, offers curated deep dives into some of my most popular Data Privacy Advantage Newsletter essays. Each episode is a dual-host, short, entertaining, and insightful exploration of the practical and actionable guidance that matters most in data privacy and emerging technology. It provides a new lens for “The Data Diva” insights—perfect for busy professionals on the go!
🆕Episode 005 - Beyond Regulation Risks: Data Privacy Strategies for Organizations🎧22 minutes
📅 Watch Episode Now on Linkedin or via YouTube: https://youtu.be/losXprhyjJI
📅Subscribe the the entire podcast on YouTube here to be alerted to new episodes: https://lnkd.in/gHR7EhKd
We're excited to support Pamela Isom's podcast, "AI or Not," produced by Data Diva Media!
"AI or Not" is the podcast where digital transformation meets real-world wisdom. Hosted by Pamela Isom, a seasoned leader with over 25 years of experience in guiding businesses through digital disruption and transformation, this show explores the intersection of artificial intelligence, innovation, cybersecurity, ethics, and technology. With awards recognizing her as a change agent and digital disruptor, Pamela brings a wealth of knowledge and insight to the table.
The show demystifies the complexities of AI and emerging technologies, shedding light on their impact on business strategies, governance, product innovations, humanity, and societal well-being with esteemed guests from around the globe. Whether you're a professional seeking sustainable growth, a leader navigating digital ethics, or an innovator striving for meaningful impact, "AI or Not" offers insights, experiences, and discussions to illuminate your path in the digital age.
Data Diva Media is a media production operation providing world-class video and podcast editing services.
Our Media Services include:
Ready to start your media project with "Data Diva" Media? Visit our Data Diva Media Website Page for more details and to schedule a meeting with the "Data Diva" Talks Privacy Podcast
Our LinkTree
Good stuff.
Thanks (as usual!) for the summarization Debbie
Treat AI like any other tech tool. Put governance parameters around it. Give folks guidance for how it should be used, for what specific business purpose, in what capacity. There's guidance out there on how to set this up. This AI kool-aid gulping is the equivalent of "let's drive the surf board to the grocery store"... just lots of it is completely inappropriate & doesn't make sense. Stop putting identifiable information into AI
Debbie Reynolds, thoughtful and timely information, including the concept of "data discipline."
I posted about this too. It’s such an important case because it shows how U.S. litigation norms often override digital privacy expectations. Our legal system is adversarial, which means each side has broad rights to collect evidence. That’s very different from places like the EU, where judges control what can be accessed and discovery is much more limited. In the U.S., once data enters a system, it can easily become part of a legal record—even if the user expected it to be private or temporary. This isn’t a new issue, but this case makes the conflict between privacy and legal hold much harder to ignore.