OpenAI chief scientist Jakub Pachocki has proposed that AI labs use a single technical measure, called perplexity, as a shared benchmark to track how capable their models are becoming, and to agree on limits before crossing certain thresholds, TechCrunch reported. OpenAI chief executive Sam Altman publicly praised the post as unusually significant for the company.
Pachocki laid out the proposal in a blog post on OpenAI’s website on November 22. TechCrunch reported that Altman endorsed the post on X, writing it was “the most important thing OpenAI has published in a while.”
Perplexity measures how well a language model predicts the next word in a sequence of text, with a lower score meaning better predictions and, by extension, a more capable underlying model. Pachocki argued in the post that purpose-built benchmarks testing specific skills go stale quickly as models improve, while perplexity has stayed a consistently informative signal of overall progress across generations of models.
The numbers behind the proposal
TechCrunch reported that Pachocki’s post put GPT-4’s perplexity at about 0.91, a level he said helped motivate the development of OpenAI’s later reasoning models. The post says that once model perplexity fell below 0.8, OpenAI began seeing transformative capabilities emerge in domains such as cybersecurity and mathematics.
| Model era | Approximate perplexity | What OpenAI says followed |
|---|---|---|
| GPT-4 | ~0.91 | Motivated work on reasoning models, per Pachocki’s post |
| Later models | Below 0.8 | Transformative capabilities in cybersecurity and mathematics, per the same post |
The post includes a chart tracking OpenAI model perplexity since GPT-3.5, which TechCrunch’s report describes as the visual backbone of Pachocki’s argument that the metric has tracked real capability jumps consistently, unlike narrower benchmarks.
The report notes that task-specific benchmarks, the kind used to claim a model is better at coding or exam questions, tend to saturate once models start scoring near the maximum, leaving little room to show further progress. Pachocki’s argument, as described by TechCrunch, is that perplexity does not have that ceiling problem in the same way, which is why he wants labs to anchor safety commitments to it rather than to any single downstream test.
What the proposal actually asks for
Pachocki’s proposal is that AI developers could use perplexity as a common yardstick and commit, collectively, not to push past an agreed threshold until researchers and regulators have a better handle on how to understand and monitor what more capable models can do, according to the report.
The idea echoes a plan Anthropic chief executive Dario Amodei published in September, described in the report as an effort to pace frontier AI development through common safety standards and limits on the rate of unchecked AI progress. TechCrunch reported that Amodei’s plan had drawn apparent support from both Altman and Tesla and X owner Elon Musk.
Why labs are talking about slowing down
TechCrunch’s report links Pachocki’s post to a wider recent pattern of senior AI figures calling publicly for more caution. The report cites a former researcher who had worked at both OpenAI and Anthropic describing the two companies as taking reckless risks with people’s safety by racing ahead without adequate safeguards.
Anthropic’s own filing for its initial public offering warned investors that advanced AI systems could pose catastrophic or existential risks to society, the report noted, underscoring that the caution extends beyond any single company’s messaging.
None of this amounts to a binding agreement between the major AI labs. TechCrunch’s report frames Pachocki’s post as a proposal meant to start a conversation about what a shared, durable measure of progress could look like, not as a commitment OpenAI, Anthropic or any other developer has formally signed on to.
The proposal lands weeks after Anthropic’s own stock market debut, which Praman News covered earlier this month, as investor and public attention on how fast frontier AI labs are moving, and on what guardrails they are willing to accept, continues to grow across both the US and Indian technology markets.
Frequently asked questions
What is perplexity, in simple terms? It is a measure of how well a language model predicts the next word in a passage of text. A lower number means the model is predicting better, which researchers treat as a proxy for overall capability.
What is Pachocki proposing exactly? That AI labs adopt perplexity as a shared benchmark and use it to set, and respect, agreed limits on how far model capability advances before oversight tools catch up.
Is this an official OpenAI policy? It is a proposal from OpenAI’s chief scientist, publicly endorsed by chief executive Sam Altman, but TechCrunch’s report does not describe it as a binding company commitment or an industry agreement.
How does this relate to Anthropic’s position? Dario Amodei’s September plan to pace frontier AI development proposed similar common safety standards, and TechCrunch reported Altman had expressed apparent support for that plan as well.



Leave a Reply