With ChatGPT approaching its second anniversary, we have been waiting for some sort of major announcement from OpenAI. Reports said a few weeks ago that the GPT-5 upgrade we’ve been waiting for could be rolled out in December.
When said ChatGPT upgrade does roll out, it might not even be called GPT-5, given the somewhat negative connotations around the name. Instead, GPT-5 might be called something else. Orion is the internal name associated with OpenAI’s next-gen AI model currently in development.
Orion should be more powerful than GPT-4o and o1, the models OpenAI has launched this year. The latter is still in beta as it is. But Orion (GPT-5) might not be available to regular ChatGPT users like you and me initially. That’s what The Verge claimed a few weeks ago. Reportedly 100 times more powerful than GPT-4, Orion would target enterprise users first.
But fast-forward to mid-December, and we have several reports saying the same thing. The big AI firms, including OpenAI, Google, and Anthropic, are struggling to train the next-gen, more powerful AI models that should succeed the current models.
This implies that a GPT-5 delay might be in order, especially if OpenAI isn’t happy with Orion’s performance compared to GPT-4o and o1. While that might seem like a step back for the industry and a potential indication that the AI bubble might eventually burst, it’s not bad news for the AI landscape overall.
We should not expect innovation at break-neck speeds, especially given the nature of this particular type of research.
A GPT-5 delay isn’t bad
Some of the people involved in AI researchers and developers out there worry about the dangers of unsafe AI. The faster these AI innovations drop, the more risky it might be, as some companies might be pressured to ignore safety over profitability. Any delays should actually be beneficial in the grander scheme of things.
The big AI firms should also work on making AI more reliable while they develop the next big thing. As amazing as ChatGPT and Gemini might be, they still hallucinate. That’s a pressing genAI problem that needs fixing. Moreover, companies can focus on new AI features that might be available from current models, like the ability to control computer apps.
These development delays might also benefit newcomers, and I’m thinking of Apple. The company’s Apple Intelligence will eventually catch up to ChatGPT and Gemini. It will probably happen faster than we expect it, with Apple likely to become a leader in the industry, particularly considering the privacy-first approach it’s going for.
This is all speculation based on a series of reports from various sources that have all heard the same thing. Reuters said a few days ago that AI companies face delays in training new large language models, identifying a key problem.
Lack of human data
The AI companies are running out of human-made data to feed to future AI models during their training phase. That’s a problem most people saw coming.
On one hand, OpenAI, Google, and others have gobbled most of the publicly available data to train ChatGPT, Gemini, and other AI chatbots. On the other, I’m not contributing any data to training ChatGPT, and I suspect other people aren’t, even though we all understand the AI’s need for more data. I also want privacy from AI products.
Reuters did not mention OpenAI’s plans to release GPT-5 but did say that “major AI labs have been running into delays and disappointing outcomes in the race to release a large language model that outperforms OpenAI’s GPT-4 model.” OpenAI qualifies as a major AI lab interested in being at the forefront of AI innovation.
The report notes that training runs for large languages can cost tens of millions of dollars and require hundreds of chips to run simultaneously. Other risks include hardware failure and power outages.
AI researchers have developed a new training method to overcome the challenges:
To overcome these challenges, researchers are exploring “test-time compute,” a technique that enhances existing AI models during the so-called “inference” phase, or when the model is being used. For example, instead of immediately choosing a single answer, a model could generate and evaluate multiple possibilities in real-time, ultimately choosing the best path forward.
The method could be helpful in more complex tasks, including math and coding, that require human-like reasoning and decision-making.
OpenAI is apparently using inference for the o1 model. Reuters also says that the “secret sauce” of o1 is another set of training on top of the base models like GPT-4. OpenAI wants to apply this technique to its bigger models. While the report doesn’t name GPT-5, one would assume that’s the goal.
The report also suggests that OpenAI might be leading the pack. Rival companies, including Anthropic, Google, and XAI, are developing versions of this ChatGPT training method.
Interestingly, Ilya Sutskever, the key OpenAI cofounder and mind behind ChatGPT, is also working on new AI training methods. He told Reuters as much without disclosing what tech his new startup, Safe Superintelligence (SSI), uses for the pre-training phase.
All big AI firms are struggling with GPT-5 rivals
A few days after Reuters, Bloomberg came out with a similar report that details the same struggles the big AI firms are experiencing.
Among other things, Bloomberg reports that OpenAI has put Orion through a months-long post-training process that incorporates human feedback to improve responses and the chatbot’s tone. However, things aren’t going according to plan. A release delay for GPT-5 is, thus, in order:
But Orion is still not at the level OpenAI would want in order to release it to users, and the company is unlikely to roll out the system until early next year, one person said.
Again, OpenAI has not committed to rolling GPT-5 in late 2024. But that was the expectation, according to reports from late October. These new stories all point to the same problem AI labs are facing.
Training more sophisticated models than currently available is increasingly more difficult and costly. It’s not just OpenAI. Anthropic also had to delay the planned release of Claude 3.5 Opus as the model failed to deliver a big upgrade over its predecessor in line with expectations:
After training it, Anthropic found 3.5 Opus performed better on evaluations than the older version but not by as much as it should, given the size of the model and how costly it was to build and run, one of the people said.
Google is facing a similar issue. Bloomberg reports that a future version of Gemini is not living up to internal expectations.
Anthropic Chief Executive Officer Dario Amodei detailed in a podcast the report referenced the massive efforts that go into training “bleeding-edge” AI models. The process might cost $100 million this year and up to $100 billion in the coming years. That’s not to be seen as a deterrent to developing better AI chatbots while building towards developing AGI, or AI that can tackle any issue, much like a human.
Back to the Orion upgrade for ChatGPT, Bloomberg also pointed to a Reddit Ask-Me-Anything session that OpenAI Sam Altman hosted on Reddit recently. The CEO acknowledged in a response the challenges of developing AI models and features.
“All of these models have gotten quite complex and we can’t ship as many things in parallel as we’d like to,” Altman said, adding the company faces “a lot of limitations and hard decisions” about how to best use its computing power.
Altman also said that OpenAI will have “very good releases” later this year, but GPT-5 won’t be one of them. Still, the CEO teased that “better and better models” are coming. Before that, OpenAI might focus on AI agents. “I think the thing that will feel like the next giant breakthrough will be agents,” he said.
Anthropic has started demoing Claude’s capabilities that let the chatbot control the computer. Google is doing it, too, with Gemini. Apple Intelligence will give us a smarter Siri with agentic abilities. Siri will be able to control some iPhone and Mac apps on your behalf. It only makes sense to see ChatGPT get similar powers, regardless of when GPT-5 drops.
Before that AMA session, Altman teased ChatGPT’s second anniversary and a potential birthday present for that. It’s unclear whether OpenAI will release any new ChatGPT features soon. ChatGPT Search did drop a few days after the tweet above, but it can’t qualify as a surprise.
