About a month ago, Ilya Sutskever announced his departure from OpenAI. Considering what had happened in November when OpenAI CEO Sam Altman was fired and then promptly rehired following all the outrage, this wasn’t a surprise.
Sutskever was a cofounder and chief scientist at OpenAI. He was also the man who was supposed to make ChatGPT and other AI products safe for us, and was instrumental in both Altman’s firing and the CEO’s return. That’s what the public was told, at least.
Sutskever chose to depart OpenAI at a time when the newest ChatGPT developments were turning lots of heads. I’m talking about the GPT-4o model OpenAI unveiled a day before Google’s big I/O 2024 event, which was focused almost entirely on AI.
I said at the time I couldn’t wait to see what Sutskever would be working on next, as he teased “a project that is very personally meaningful to me about which I will share details in due time.” Fast-forward to mid-June, we now have the name and purpose of said project.
The new AI company is called Safe Superintelligence Inc. (SSI), and we probably won’t hear any specifics about its products anytime soon.
Safe Superintelligence has two other cofounders: Former Apple AI lead Daniel Gross, and former OpenAI engineer Daniel Levy. Sutskever announced the new venture on X, where he outlined the purpose of Safe Superintelligence. Here’s a part of the announcement:
SSI is our mission, our name, and our entire product roadmap, because it is our sole focus. Our team, investors, and business model are all aligned to achieve SSI.
We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.
This way, we can scale in peace.
Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
That sounds great, at least in theory. A company that develops safe superintelligence is the dream we, humanity, didn’t know we needed until recently.
We want superintelligence to be safe. We wouldn’t want AI to take over the world and eventually destroy humanity. That’s the worry out there, and it’s why it was so important that Sutskever was working on ensuring the safety of ChatGPT. It’s also why the massive changes at OpenAI’s superalignment team were so disturbing.
I can’t help but wonder whether or not what Sutskever wants to deliver is possible. In an interview with Bloomberg, Sutskever said that the company won’t launch any products anytime soon. “This company is special in that its first product will be the safe superintelligence, and it will not do anything else up until then,” he said.
Sutskever didn’t reveal the names of his backers or the amount of money he raised for his Safe Superintelligence startup. Those backers will want some sort of return, no matter how aligned they might be with Sutskever’s interests. That’s harder to do without a commercial product.
Safe superintelligence, the product, should be artificial intelligence that’s far beyond what humans can do. The next step in AI should be AGI (artificial general intelligence), or AI that can reason as well as humans. But that’s not necessarily the path we’ll take. We could always reach superintelligence before AGI.
Safe Superintelligence will not be the only player in the industry trying to achieve superintelligence. It’s very likely that many other tech giants and startups, including OpenAI, are also pursuing that goal.
Also, reputation aside, Sutskever doesn’t have a monopoly on AI safety. It’s not like OpenAI, Google, Microsoft, Meta, and Apple want to develop unsafe AI. But all these companies will be able to test their AI innovations via commercial products, whereas Safe Superintelligence will not launch anything anytime soon.
I’ll also say that Sutskever’s work on safe AI doesn’t guarantee that his AI or another AI won’t go rogue. Anyone with the right resources can train AI models. The theoretical risk remains that someone will reach AGI or superintelligence without properly aligning AI with humanity’s interests.
