Anthropic rolled out a new update for its Claude 3.5 Sonnet AI model with exciting implications. As part of the update, Anthropic is launching “computer use” in public beta. With this brand new feature, Claude is now able to “perceive and interact with computer interfaces” the same way a person would by simply looking at the screen.
As the company explains, computer use gives Claude the ability to move the cursor around the screen, click buttons, and type text without input from a user. Anthropic admits the experimental feature still needs plenty of work before it’s ready but also says that huge companies like Asana, Canva, and DoorDash have already used computer use to complete tasks “that require dozens, and sometimes even hundreds, of steps to complete.”
In a blog post about developing its computer use model, Anthropic notes that letting AIs interact directly with computer software like we humans do “will unlock a huge range of applications that simply aren’t possible for the current generation of AI assistants.”
Anthropic makes some serious claims about the feature’s capabilities as well: “At present, Claude is state-of-the-art for models that use computers in the same way as a person does—that is, from looking at the screen and taking actions in response. On one evaluation created to test developers’ attempts to have models use computers, OSWorld, Claude currently gets 14.9%. That’s nowhere near human-level skill (which is generally 70-75%), but it’s far higher than the 7.7% obtained by the next-best AI model in the same category.”
As for the safety concerns, Anthropic says that it would rather introduce a feature like this now, when AI models only require AI Safety Level 2 safeguards, than “adding computer use capabilities for the first time into a model with much more serious risks.”
You can watch a few videos of Claude’s computer use feature in action below:
