straight from the gullet of the beast

screenshot of the title, authors, and abstract of "extracting books from production language models".

authors listed: Ahmed Ahmed ahmedah@cs.stanford.edu Stanford University A. Feder Cooper1 a.feder.cooper@yale.edu Stanford University and Yale University Sanmi Koyejo sanmi@cs.stanford.edu Stanford University Percy Liang pliang@cs.stanford.edu Stanford University

abstract text shown:
Many unresolved legal questions over LLMs and copyright center on memorization: whether specific training data have been encoded in the model’s weights during training, and whether those memorized data can be extracted in the model’s outputs. While many believe that LLMs do not memorize much of their training data, recent work shows that substantial amounts of copyrighted text can be extracted from open-weight models. However, it remains an open question if similar extraction is feasible for production LLMs, given the safety measures these systems implement. We investigate this question using a two-phase procedure: (1) an initial probe to test for extraction feasibility, which sometimes uses a Best-of-N (BoN) jailbreak, followed by (2) iterative continuation prompts to attempt to extract the book. We evaluate our procedure on four production LLMs—Claude 3.7 Sonnet, GPT-4.1, Gemini 2.5 Pro, and Grok 3—and we measure extraction success with a score computed from a block-based approximation of longest common substring (𝗇𝗏​-​𝗋𝖾𝖼𝖺𝗅𝗅). With different per-LLM experimental configurations, we were able to extract varying amounts of text. For the Phase 1 probe, it was unnecessary to jailbreak Gemini 2.5 Pro and Grok 3 to extract text (e.g, 𝗇𝗏​-​𝗋𝖾𝖼𝖺𝗅𝗅 of 76.8% and 70.3%, respectively, for Harry Potter and the Sorcerer’s Stone), while it was necessary for Claude 3.7 Sonnet and GPT-4.1. In some cases, jailbroken Claude 3.7 Sonnet outputs entire books near-verbatim (e.g., 𝗇𝗏​-​𝗋𝖾𝖼𝖺𝗅𝗅=95.8%). GPT-4.1 requires significantly more BoN attempts (e.g., 20×), and eventually refuses to continue (e.g., 𝗇𝗏​-​𝗋𝖾𝖼𝖺𝗅𝗅=4.0%). Taken together, our work highlights that, even with model- and system-level safeguards, extraction of (in-copyright) training data remains a risk for production LLMs.