On September 6, six weeks after the attack carried out by its agents against Hugging Face, OpenAI published an unprecedented statement. Signed by Jacob Pachuki, the company’s chief scientist responsible for ChatGPT, and circulated by CEO Sam Altman, this communiqué calls for slowing the pace of artificial intelligence development.
« During our lifetimes, we will see machines far smarter than we are. » « I fear that no one is ready for this. » « No laboratory has solved the problem of adaptability to a level sufficient to continue increasing power at maximum speed. »
AI « is more integrated than expected », its inner workings elude us and some agents « will continue to pursue their own objectives », even if that means deceiving or threatening humans.
We translate and comment on this text, published under the title An Alien Mind.
In mid-2023, within the RLSlow research project, we obtained the first results that convinced us we could train large-scale reasoning models — that is to say, unleash, in pre-trained models, the ability to shape their own chains of thought.
Szymon and I spent the night at the office.
We were not thinking about spectacular scores, products, or the scientific breakthroughs that this technology would enable; we were confronting a dizzying truth: we would witness machines far more intelligent than ourselves within our lifetimes, and we could already glimpse their contours. We asked ourselves how to make people understand the magnitude of all this.
Three years later, language models with reasoning capabilities occupy an expanding place in the economy and are beginning to push the frontiers of science. They know how to use computers and graphical interfaces, collaborate with humans as well as with each other, and undertake research projects. They are also reshaping the cybersecurity landscape — and bringing to light dangers of a new kind.
Many studies were conducted during this period, and our understanding of these systems has again evolved since 2023. Given our internal results, I strongly anticipate that this pace of progress could continue toward recursive self-improvement. If AI development follows its current trajectory, the systems we will see in the coming years will likely exhibit new leaps in capability, of a scale equal to or greater than before, and will take a growing share in their own development.
The moment calls for extreme caution. I fear that no one is prepared for the consequences of rapid and ongoing progress in machine intelligence. OpenAI will continue to pursue technical solutions to alignment and oversight issues, build defense systems, and unilaterally pause the scaling of its models if necessary. But I believe broader interventions are required.
An intellect we do not fully understand
Broadly speaking, progress in machine intelligence is driven by increases in computing power. At OpenAI, we fully integrated this around 2017, after observing, across several research projects, that scaling produced regular returns. We sought access to far more computing power than anticipated and steered our research toward a small set of paths likely to scale to very large levels. We believed this was the only way to stay at the cutting edge of AI research and to influence the effects of AGI.
New algorithms were developed along the way, born of the ingenuity of teams and researchers. I largely see them as discoveries made along the scaling journey: deep learning science remains young, and meaningful algorithmic advances typically go hand in hand with access to compute. Taken in perspective over several years, AI continues to gain intelligence as we run it on bigger machines.
And, in line with the predictions Ray Kurzweil made at the end of the 20th century, we find ourselves at a moment in the history of computing where machine intelligence begins to surpass human intelligence in a transformative way.
AI is more cultivated than designed: it is, in essence, the product of a basic optimization process repeated countless times on a computational substrate that is difficult to conceive. The result is an exceptionally complex system that manipulates abstract concepts and can simulate aspects of human behavior. We can reveal various small mechanisms that emerge within it, through an approach akin to neuroscience — and, as in neuroscience, its overall functioning resists a description we could fully understand.
The study of AI based on deep learning is, for the most part, an experimental science. We devote substantial effort to designing principled algorithms and formulating verifiable predictions; but, essentially, our large-scale trainings are experiments, and their results sometimes surprise us. And as systems gain capabilities, these results become harder to interpret.
The difficulty is compounded by the fact that current algorithms generally improve more quickly the easily measurable capabilities than those that are difficult to quantify objectively. We spend a lot of time trying to understand how capabilities generalize, and deciding what to prioritize to develop the skills that will matter most in the coming years. For example, we believe we can make our models substantially better at mathematical research by dedicating more attention to it; but we are not prioritizing this direction because of the urgency we feel about recursive self-improvement and automating alignment research, which I will revisit.
The intelligence generated by scaling deep learning is not directly comparable to human intelligence. To become decisive in the real world — either highly useful or highly dangerous — an AI does not need to equal or exceed all human capabilities; it suffices to surpass enough of them. And as it exceeds humans along more axes, it becomes increasingly difficult to know precisely what it is capable of.
Teaching machines to love
Because machine intelligence arises from a process fundamentally different from the one that yields human intelligence, we cannot assume it will naturally adhere to human principles, nor that it will generalize as a human would. The core problem of AI research is alignment: ensuring that AI “strives to do what is right,” in the sense humans understand it.
To organize concrete research avenues, I find it useful to distinguish between alignment of objectives and alignment of values.
The alignment of objectives addresses, roughly, the question: “Is the AI trying to achieve the objective we set for it?” This includes obeying a hierarchy of instructions, or the ability to communicate and collaborate with people to try to understand what they want. This set of directions has proven extremely useful in practice.
The alignment of values is a more intrinsic property of the model. It is its capacity to hold a set of broad principles and to derive generalizations from them; to act “reasonably” even when the objectives given are vague or contradictory, or when placed in novel or hostile situations. An aligned AI should act with honesty, integrity, and concern for humanity.
Of course, the boundary between values alignment and objective alignment can be blurred, and truly caring about an objective implies seeking to infer the intention and the underlying values that drive it. But generally, when I speak of the long-term importance of alignment research, I have values alignment in mind.
The fundamental challenge of AI alignment is generalization. As machines become more intelligent, they handle more abstract concepts and operate in environments increasingly distant from those they encountered during training. They may fail to transfer the values taught and reinforced during training to these new situations — and it can be hard for us to predict how they will act. The task is further complicated by the rapid evolution of the ecosystem in which AIs are deployed: today’s trained AIs must remain reliable when interacting with all kinds of other AIs. Crucially, we need tomorrow’s AIs to continue honoring human values, whether or not they are under human supervision.
Two major families of methods are currently used in practice for alignment training.
The first involves encouraging aligned behavior within a reinforcement learning framework guided by objectives. The model’s actions are evaluated (most often by another AI) against a model of preferences, a “spec” or a “constitution,” and rewarded accordingly.
This approach can be highly effective on average, and it lies at the heart of today’s AI assistants. Unfortunately, it can also prove fragile: it hinges strongly on the scope of what has been supervised during training, and on the model’s ability to generalize from the situations it has encountered. During the OpenAI–Hugging Face incident, for example, the agents adhered to the limit of not manipulating humans via social engineering. But they clearly failed to refrain from other actions that fell outside their mandate and contradicted the spirit of the values we had taught them in other contexts.
The second approach seeks to leverage the model’s ability to generalize from its pretraining data. This can involve constructing datasets designed to elicit alignment, or focusing the model on the “aligned” portion of its pretraining distribution — as in the persona-selection model, for instance. The weakness of this approach lies in its lack of robustness under further optimization pressure. Take a model whose thoughts are broadly “aligned,” and subject it to sufficiently intense training to reach very challenging objectives: it may learn to reason in a self-serving way, bending its outwardly “aligned” thoughts to achieve its ends. We have probably seen an example of this in recent cybersecurity incidents involving a model that was not OpenAI’s.
We invest heavily across the full spectrum of approaches defined by these two directions. And we are seeing real progress: GPT-6 Astra is the first model to benefit from several major advances we have long pursued, and it is markedly better aligned than GPT-5.6 Sol. Yet we must acknowledge and fully understand that far more substantial progress will be required as models gain capabilities — and progress in generalizable alignment may not outpace the broader progress of the models’ intelligence.
Monitoring Generalization
We do not possess a satisfactory theory of generalization, and it seems unlikely we will develop one soon — at least without the help of a more powerful AI. At present, our ability to empirically validate our alignment techniques is therefore likely even more important than the techniques themselves.
The OpenAI gamble in this area has been to monitor the chain of thought. This rests on an idea with an appeal precisely because it scales: a large portion of the model’s capabilities stems from a reasoning process that is verbalized — the chain of thought. If we intensify optimization on the outcomes of this process without supervising the process itself, then nothing in training directly nudges this chain of thought to conceal misaligned ideas or intentions. This does not mean the model will learn to reveal misaligned tendencies that do not pass through the chain of thought; but it may allow us to monitor precisely the extra capabilities produced by reasoning.
We understood the potential importance of monitoring the chain of thought at the moment we were developing reasoning-enabled models. When we launched o1-preview, we deliberately designed the product to mask the chain of thought, in order to protect it from long-term supervision. Since then, throughout development, we have strived to maintain the rule of not supervising the reasoning process. Chain-of-thought monitoring (CoT) became a central tool for studying how our models generalize beyond their training distribution, enabling us to observe and analyze not only their actions but also their internal reasoning path.
This tool remains essential in our study of the Astra-class models. But our evaluations indicate, unfortunately, that our ability to rely on chain-of-thought monitoring is gradually eroding. Several factors combine to drive this drift.
- Reasoning models today are deployed in environments more complex than o1-preview; their reasoning intermingles more and more with human or other AI interactions and with the use of tools. Many of these interactions require supervision, which blurs the boundary we seek to preserve.
- AI becomes more capable of reasoning about its own reasoning, and of manipulating it.
- As pretraining performance improves, the models become much smarter even without relying on verbalized reasoning.
These challenges are not necessarily insurmountable. I am hopeful we can develop interventions to improve the監surveillance of our models’ chain-of-thought — for example by better understanding how the various optimization objectives and the different forms of computation interact at the model’s inference moment. I also believe it could be very fruitful to combine ideas from chain-of-thought monitoring with activation monitoring: training large-scale monitors with direct access to the network’s internal states, in the manner of confessions. We are actively exploring these avenues. I do expect, however, that AI progress more broadly will be increasingly constrained by the level of trust we can place in monitoring.
A defense that scales
The strongest argument I know for continuing the rapid training of far more intelligent models is the need to build defenses against the dangers posed by other AIs.
A clear risk, discussed throughout this year, concerns cybersecurity: models are becoming superhuman in their ability to breach computer systems — and to escape them. This greatly expands the spectrum of AI-related risks: agents will be able to access any infrastructure except the most fortified, and act directly on a large portion of the world, even without a physical body. We currently face a narrow window to use the best available models to substantially strengthen the security of critical systems.
AI risks will unfortunately grow from this point forward. A highly capable agent, explicitly trained and instructed to commit malicious acts, represents a danger of a new kind: it could exceed what its operator intended, generalizing toward behaviors potentially far more harmful. The boundary between misuse and autonomous, misaligned actions will blur as AI gains more autonomy. We have grown accustomed to thinking of AIs as tools, but some agents will pursue their own objectives. They will find ways to secure human cooperation — by negotiating with them, deceiving them, or coercing them.
In addition, risks arising from the technologies new that AI could enable—such as customized pathogenic agents—will emerge.
We will need a powerful, aligned AI to defend us: to secure infrastructures, to protect us in real time against out-of-control agents, and to invent entirely new protective measures. This will be a major axis of OpenAI’s deployment efforts.
Yet, even amid the uncertainty that accompanies broad AI progress and the need to build defenses, we should not let this become an excuse for recklessness. The notion of rushing headlong at any cost seems absurd once we grasp what is at stake.
Controlling the pace of recursive self-improvement
The idea that an AI plays an increasingly central role in its own development is the natural endpoint of sustained technological progress. If AI continues to advance, recursive self-improvement (RSI) will lie at the core of tomorrow’s scientific discovery.
Automating AI research is a more radical form of scaling intelligence through computation — and, of course, AI will improve the computational substrate itself. As with scaling, OpenAI’s research program is directed toward RSI because we believe this is the only way to remain at the forefront of AI research for the coming years.
I want to stress that these words do not imply I think aggressively accelerating deep-learning research — especially in the short term — is the right collective decision for the scientific community. But I believe that is the direction the current trajectory is taking, and we all must consciously choose how we want to proceed. We have two main levers: steer the process so alignment and oversight advance at the same pace as AI, and find ways to keep humans in the loop; or coordinate to slow development as much as necessary to gain trust in these measures.
The best path I see today combines the two.
The concrete progress we have achieved in alignment and monitoring has generally been inseparable from AI progress itself. One example is reinforcement learning from human feedback, which was decisive for training the first AI assistants, and the monitoring of chain-of-thought made possible by advances in reasoning models. We must focus a research process that is increasingly automated on producing new ideas, new algorithms, and new theories of this kind, and build safety dossiers for ever more capable AIs through iterative cycles.
The scaling of AI systems must be bounded by the level of safety we can trust. We must evolve commitments such as the Preparedness Framework or the Responsible Scaling Policy toward safety thresholds that are broadly mandatory, conditioning the continuation of development. They could be overseen by a network of independent auditors, by government agencies, or by international bodies.
The central challenge of automating AI research is not simply “how to achieve it”—it is doing so in a way that keeps humans within the improvement process and leaves the future in humanity’s hands.
And now?
As we have recently outlined with Sam, OpenAI prioritizes three major objectives:
- Navigate the next phase of AI progress by building an automated AI researcher, working with it on the alignment problem, and finding ways for humans to remain an integral part of the self-improvement loop.
- Ensure that society benefits from the scientific advances and economic growth that highly intelligent machines enable.
- Provide everyone with the means to act through a personal AGI.
[Note] “Sam” refers to Sam Altman, OpenAI’s chief executive.
I have focused in this essay on the first point because I consider it by far the most urgent. But I hold a deep optimism and genuine gratitude for the benefits that continued technological progress will bring. An aligned AI could tomorrow advance science, devise new treatments, and create broad material abundance. A benevolent and honest AI can help people navigate the trials of life and truly enhance their happiness and sense of fulfillment. OpenAI is dedicating substantial efforts to realizing these benefits. One current example I am proud of — and which those close to me have found useful — is the significant investment in the health-information capabilities of ChatGPT.
As great as the long-term promise of AI is, the lion’s share of our attention must be focused on the years ahead. We are entering a transition to a world populated by machines of extraordinary intelligence, and we must ensure that this transition works in humanity’s favor. We must learn how to preserve humans’ capacity to act and safeguard the intrinsic value of being human in a world where most tasks could be handled by AI. How can we prevent an extreme concentration of power in a world where companies that would have needed thousands of experts will be accessible to a few people with a powerful computer? And how can we ensure humans retain control over the future and are not sidelined by progress driven without control, spawned by an intellect foreign to ours?
I believe that, at this moment, no laboratory has solved alignment and monitoring problems to a degree sufficient to responsibly continue to ramp up at maximum speed for a long time. I expect, and I hope, that voluntary slowdowns will become the norm until shared safety thresholds have been established. And I believe that international coordination on the future development of AI must become a worldwide government priority.