When Scott Aaronson was recruited by OpenAI, chief scientist Ilya Sutskever asked him to consider “a mathematical definition of what it would mean for AI to love humanity.” Aaronson almost laughs when he tells the anecdote. He comes from another world.
A computer scientist by training, he has long been a professor at MIT and is now a professor at the University of Texas at Austin. He is one of the leading experts in the theory of computational complexity and quantum computation. A winner of the ACM Prize in Computing in 2020, he was elected in 2026 to the National Academy of Sciences.
Since August 2, the AI Act requires providers operating in the EU to make detectable the content produced by their models as having been generated or manipulated specifically by an artificial intelligence. In the wake of this, Anthropic announced the introduction into Claude’s outputs of an invisible watermark rekindling older debates: is such a marking desirable? Can it be introduced without degrading the quality of responses? How robust is it against the development of techniques designed to erase it? Doesn’t it primarily serve labs to recognise texts generated by LLMs and thus train only on human writings, for fear that training a model on AI-generated content would lead to a decline in reliability and quality? Will it ultimately raise questions of intellectual property?
We asked Scott Aaronson, who has been working on these questions even before the launch of ChatGPT, to start from the beginning. He told us how he designed the watermarking and revealed the reasons for his departure from Sam Altman, while sharing his concerns and questions about what he sees as the end of an era of mathematical research “in the form we knew it.”
To receive our interviews and analyses directly in your inbox, try a Grand Continent subscription from 8 euros per month
When you began working for OpenAI in 2022, you developed the watermarking. Your approach today informs most major AI labs. What is the general principle, and how did you come to work on this?
The idea of textual watermarking is to tweak the operation of a language model very slightly, without degrading its quality. Users receive responses that look exactly like the ones they’re used to from Claude, ChatGPT, or other models, but a statistical signal is embedded in the randomness (the cryptographic pseudo-random function) used to select the next word. It then becomes much easier to establish not only that a text was generated by AI, but also that it came from a specific model.
Many people have thought about this idea. To my knowledge, I was the first to think specifically about watermarking LLMs. It was shortly after I joined OpenAI, in the summer of 2022. I had been recruited by Ilya Sutskever and Jan Leike, who were then leading alignment research. I imagine they were readers of my blog. That’s how they found me and invited me to join them. It was just before the launch of ChatGPT, on November 30, 2022.
What vision did you have for AI development back then?
I was aware of what was happening. I had tested GPT-3. It had left a strong impression on me around 2020 or 2021, and I wondered why more people weren’t paying closer attention.
But at the time, OpenAI was still saying: “Look, we’re just a small non-profit organization. We’re trying to do things the right way for humanity and in a safe manner.”
“I learned to be extraordinarily cautious about what I read online and never assume that a message truly came from the person it claimed to be.”
Scott Aaronson
They offered me a year where I could remain mostly in Texas with my students and family, traveling to San Francisco only monthly, to think about how theoretical computer science could contribute to AI safety.
Where did the research on these questions stand? Was making AI safe a priority?
At that moment, there was already a group working on these questions. Much of it had formed around Paul Christiano, a former student I had at MIT who, when I arrived, had just left OpenAI to create his own organization, the Alignment Research Center (ARC).
Paul had shown that the key to making AI safe could lie in complexity theory. That was partly what drew them to me. But I already saw a major obstacle: progress in controlling AI, assigning goals, and ensuring reliable adherence to them was almost entirely empirical.
In what sense was that a problem?
It wasn’t enough: we needed to explicitly articulate the values and principles we wanted AI to respect, in a fashion akin to Isaac Asimov’s stories. For that, we devised what we now call “constitutions.”
From these principles, we could then empirically evaluate the effects of different approaches and publish results showing, for instance, that a given intervention reduced cheating behaviors by 20%, or that it reduced a model’s propensity to assist a user in manufacturing chemical weapons by 30%. We also explored other avenues, such as reinforcement learning.
You thus moved from your primary field of expertise, computer science, to what could be called experimental psychology…
All of this without truly understanding what was happening. In the summer of 2022, I wondered what theory could contribute to AI safety. I asked whether it was possible to do something with a real, tangible product impact. That summer, I faced a particularly violent harassment campaign on my blog, with people posing as colleagues or other interlocutors. I therefore learned to be extraordinarily wary of what I read online, and to never assume a message came from the person it claimed to be.
How did that event shape your research?
I explored multiple avenues. One would have been to ask the major labs to keep a record of all content generated by their models, so that one could later present a text and ask: “Have you produced this content before?” Such a solution would raise obvious privacy concerns. And even if it could be implemented without compromising user privacy, it would be very hard to persuade the public.
Another idea was to treat this as just another AI problem, namely to train a neural network to distinguish as well as possible between human-generated text and AI-generated text. This has already been done to a large extent. A company, GPTZero, was focused on that. A tool with a lot of visibility today is Pangram, and many professors use it to try to determine whether their students used AI.
But it was already clear, four years ago, that all these tools would inevitably produce false positives. And even with a relatively low rate of a few percent, the problem remains substantial: when a university accuses a student of cheating, it must be able to do so with an extremely high level of certainty. Therefore the false-positive rate must be brought down to about one in a thousand, or even less.
That is what led me to the idea of watermarking.
From what feature of LLMs’ functioning did you develop watermarking?
The operation of LLMs is inherently probabilistic. Anyone can verify that: you can send exactly the same prompt multiple times and get a different answer each time. Why? Because the transformer neural network does not compute the next word directly. It computes a probability distribution over the various possible words or tokens that could follow.
Then, in normal operation, it simply samples a token from that distribution. This yields a new input, which is fed back into the LLM. It then generates a distribution for the next token, samples one, and so on autoregressively.
What is the link between the length of a response, its entropy, and the statistical confidence required to determine that it came from a particular model?
Text generation by an LLM inherently involves entropy. It is even possible to measure how much. The effective uncertainty at the moment of choosing the next token is often around one bit per token. In other words, the model has a small degree of freedom to choose among several plausible continuations.
However, if you ask ChatGPT to list the first hundred numbers, it can do it, but there will be virtually no entropy, unless it plays with spaces or line breaks. That is not content you can reasonably watermark. The same goes for, say, the text of the Declaration of Independence.
But in general, when we ask something of an AI, even though there is in a sense a uniquely correct answer, there are exponentially many ways to express it—the longer the text, the more room there is to embed a signal.
When you look to detect the watermark, you do not have access to the probabilities. You do not see the prompt that led to the text. You only see the text itself. Based on it alone, you should be able to determine whether there is a watermark. Even better, you should be able to determine which parts of the text contain a watermark and which do not.
I designed a concrete method to do all of this. There were statistical questions: given the entropy in this sequence of tokens, what should be done to maximize the statistical power of the watermark per token? There is a trade-off between the probability of a false positive and that of a false negative. But we want to minimize the error probability and do so with as few tokens as possible.
I developed part of the underlying theory by establishing a rule for choosing the next token using a bit of analysis and statistics that I had to dust off from my years of study. Machine-learning researchers later told me I had rediscovered a rule already known in their field: the Gumbel-Softmax trick. I thus called my method the Gumbel-Softmax scheme. It had never been used for this purpose before.
Many people believe there must be a trade-off between watermarking and the quality of the response. Why is this intuition misleading?
Immediately, some people raised this objection about a trade-off between watermarking and the quality of text, assuming the probabilities would be biased in a certain direction that would degrade the response quality.
I realized that this trade-off did not exist. It’s enough to replace the entropy used to select the next token with a particular type of pseudo-randomness that favors certain unusual combinations of tokens over others. Some n-grams—for example, a sequence of five consecutive tokens—would be favored, while others would be disfavored.
“At OpenAI, they told me: ‘Our mission is to prevent AI from destroying humanity. Are we really that concerned about students cheating on their homework?’”
Scott Aaronson
But since that choice is based on a pseudo-random function, it remains imperceptible to an ordinary reader. One could even say something stronger: the response could be made cryptographically indistinguishable from a normal LLM response. I had not fully proven it, but researchers such as Sam Gunn, Miranda Christ, or Or Zamir, many of whom were at Berkeley, subsequently extended my work to show that true cryptographic indistinguishability could be achieved.
There remained interesting mathematics to do to determine exactly how to choose the next token in a pseudo-random way so as to preserve the same apparent distribution while embedding a signal that could be detected later.
How did the major labs approach the watermark question?
OpenAI assigned me an engineer, Hendrik Kirchner. He developed an implementation of my method, and we tested it. It appeared to work exactly as theory predicted. On my side, I gave public talks to raise awareness about the topic.
It was precisely at that period that ChatGPT was released. The question then became whether OpenAI would actually deploy this technology. It never did. For the rest of my time at OpenAI, I kept revisiting the issue. I even brought the topic directly to Sam Altman.
Why did you advocate deploying the watermark?
First, my academic colleagues urged me to do it. By 2023, they were already seeing a massive influx of AI-generated student work, and the situation has only worsened since then.
I had in mind all sorts of possible misuses of LLMs. Academic cheating was perhaps the most obvious, but there was also identity theft or phishing.
Things aren’t always so binary. An argument that repeatedly came up at OpenAI, which I hadn’t considered: “There are all these people whose native language is not English and who use GPT to make their English more fluent. If all responses contained a watermark, we could even consider it discriminatory to impose that on them.”
So there was indeed a trade-off of this kind. But I felt that, for the vast majority of my academic colleagues, once the issue was explained, the answer was straightforward: of course we should do it.
“I brought the watermarking topic directly to Sam Altman. OpenAI never deployed it.”
Scott Aaronson
OpenAI’s sales teams conducted user surveys, and a non-negligible share outright rejected the idea of watermarking. Deploying it risked pushing users toward a competitor’s LLM. I think that argument ultimately tipped the balance at OpenAI.
The safety and alignment teams were never particularly enthusiastic, because their line of thinking was: “Our mission is to prevent AI from escaping control and destroying humanity. Are we really that worried about students cheating on their homework? Maybe the homework should change, or even disappear.”
Many people had the intuition that the battle was lost from the start. Why even try? But I felt the social cost was very low and that if we could persuade AI companies to coordinate on this, it would send a positive signal.
After Anthropic’s announcement, a GitHub repository titled “Watermarks Remover for AI Content” reached about 11,000 stars. Are we doomed to a cat-and-mouse game of applying and removing watermarks?
The other question we asked was: is this really useful? As soon as we implement it, people will adopt countermeasures. They will find ways to remove the watermark. I did not have a very convincing answer, because it’s true that with minimal effort, they can be removed.
The simplest attack would be to tell ChatGPT: “Write my homework, but in Spanish or French,” or in any other language. Then run the result through Google Translate. You would obtain a document entirely different from the one in which the watermark had been embedded.
Another method is to ask ChatGPT: “Write my homework, but insert the word ‘pineapple’ between every word.” Then simply delete all the ‘pineapple’s. That can slightly degrade quality, but overall it works.
For any watermarking that relies solely on the sequence of n-grams, you end up with an entirely different n-gram sequence. And one can easily imagine people creating web tools that make this manipulation even easier. There are already all sorts of tools to help circumvent AI detectors.
Since this will happen in a predictable way, why bother? There are many things we do simply to introduce friction. We lock doors with a key. Can someone pick the lock? Yes, but it adds friction to the process. One could also ask why Google doesn’t simply give up its long-running battle against SEOs. It’s simply about making their task harder. So I told myself that perhaps it was worth trying.
You presented semantic watermarking as a promising research direction: this would encode provenance at the level of meaning rather than at the level of token-by-token choice. What could semantic watermarking concretely look like?
I realized that if we wanted a watermark capable of resisting attacks like Google Translate or the “pineapple” attack, we would need to embed a signal at the level of the underlying concepts rather than at the token level.
I didn’t know how to do that. Yet there are ideas and things one could try, because inside LLMs concepts are represented as vectors in high-dimensional spaces. So one could try to perturb these vectors, nudging them slightly to embed a watermark.
Unlike my previous method, we would have no theoretical guarantee that it works. But we can test it empirically.
Progress has been made in this domain: some semantic watermarking methods do seem to work and show some degree of robustness. Still, in my view, it remains a very active area of research. It isn’t what Anthropic deployed. But perhaps they will need to pursue it in the future if their current approach becomes too easy to circumvent.
The European Union insists on watermark interoperability. How do you see the coordination challenge among the different providers?
By the end of my time at OpenAI, I realized I could not solve the coordination problem alone. I did discuss watermarking with people at DeepMind and Anthropic. Anthropic didn’t seem particularly interested at the time, in 2023. I am of course glad to see that they have now actually deployed watermarking.
What I proposed led Google to develop SynthID. SynthID was very close to my proposal with one key difference: Google never made publicly accessible a tool to detect watermarking. You have to request it. Some academics I knew who had asked for access never received a reply.
“Clearly, we are living in the last year of research in mathematics and theoretical computer science in the form we knew it.”
Scott Aaronson
That was another topic of discussion at OpenAI. If we use watermarking, do we allow anyone to detect the watermark? Or should access be restricted to professors, journalists? If so, one would need to build an entire infrastructure to decide who gets access.
I concluded that I did not have the influence or political skill to persuade OpenAI to do that. Their reasoning was: “Why should we if Anthropic or DeepMind won’t?” Perhaps the three should do it together, but that would become a coordination problem.
Is enforcing it by law more appropriate?
I’ve already discussed with California lawmakers who wanted to make watermarking mandatory. California has indeed passed a law mandating watermarking, but only for audiovisual content, exempting textual content.
But outside California, who else would want such a law? The U.S. federal government does not plan to impose such a requirement. The other obvious candidate was the European Union, which included watermarking in the AI Act, but the details do not seem clear.
Why?
Part of the difficulty is legal: if watermarking becomes mandatory, which exact technique must be mandated? And what happens if that technique is subsequently circumvented? You want to be able to say you must use best efforts or the best approach available at a given time. But how do you translate such an evolving obligation into the law?
You recently wrote that it was “Quite clearly the last year of research in mathematics and theoretical computer science in the form we know it.” What changes should we expect?
Last week, I returned to OpenAI for the first time since 2024. They hosted a workshop on the future of mathematics. We witnessed something remarkable this summer: more and more significant mathematical problems are being solved with the help of AI models, or entirely by them.
They still don’t know how to do everything. But if you compare the situation to three years ago, when they struggled with basic mathematics and made obvious errors, today they are doing high-level mathematics research.
If you project this trajectory forward a few years, it will lead to a complete transformation in the nature of the work of mathematicians or theoretical computer scientists. In the best case, we will continue to contribute our scientific expertise, determine which major research directions are interesting or not, understand results, and explain them.
What reflections does this raise for you?
I think about it a lot. I think about it with respect to my students. What should I train them for? What skills will be useful in the world we are heading toward?
This is a question that virtually every mathematical researcher is already considering or should consider.
How should one prepare for this transformation?
One of the important ongoing discussions concerns publication norms for AI-generated work. We have not yet determined them.
There was that famous case of the Jacobian conjecture, refuted by someone who simply posted on Twitter: “Hello. I was watching the World Cup, and my friend Fable found the following counterexample to this hundred-year-old conjecture.” It was hilarious, but I suspect this will not be the standard way to announce such results in the future.
Should we publish them as research articles? But there will be so many AI-generated papers that there won’t be enough people to read them all. Will we end up with AI-written papers, evaluated by AI? Will we deposit them on preprint servers?
“It’s a bit like in The Lord of the Rings, when the inhabitants fortify their cities before the invasion of the Orcs.”
Scott Aaronson
Let us decide that, to be published in a journal, a result may have been discovered by an AI, but a human must have internalized the idea and be able to explain it to other humans. Or should a human assume responsibility if the result is false? We are still determining what those norms should be.
I’ve spoken with people who sit on program committees of major theoretical computer science conferences. It’s a bit like in The Lord of the Rings, when the inhabitants fortify their cities before the Orc invasion. They anticipate thousands of AI-generated submissions in the next conference cycle, and they have no idea how they will manage this, because there simply won’t be enough people to review them.
They will therefore be forced to rely in part on AI.
Do you think it is important to preserve a human role in research, or will humans merely slow down the pace of discovery?
I think it is extremely important to continue thinking about the world we want to live in and the role we want to play in it. In the end, this question goes well beyond the careers of mathematicians or theoretical computer scientists. For our small world, it is obviously important, but these are questions humanity as a whole will face.
Do we really want robots to perform all jobs if they can do them better than us? And what world would that leave us with? A utopia where all our needs are met and we simply enjoy ourselves? Or a dystopia where robots eventually wonder: “In the end, why do we still need humans for anything?”
As long as we keep asking these questions, or as long as we cannot guarantee AI safety, then yes, absolutely: we want to retain humans who have enough skills to understand, as much as possible, what AI is doing.
And if AI starts to devise strategies against us or, say, to provide mathematical proofs that appear correct but are subtly false because they pursue another goal, we must be able to detect that.