Anthropic says latest AI model can control users’ computers

Unlock the Editor’s Digest for free

Roula Khalaf, Editor of the FT, selects her favourite stories in this weekly newsletter.

Anthropic has enabled its artificial intelligence to control actions on a computer, including searching the internet, clicking buttons and inputting text, as companies increasingly seek to leverage the new technology to build virtual agents.

The “Computer Use” feature, unveiled for developers on Tuesday, grants access to Claude, its AI model, to conduct actions on users’ behalf, with their consent, “like a human collaborator”, the company said — controlling the mouse and keyboard to browse the internet, schedule calendar appointments and fill in forms.

“That sort of repetitive thing that people absolutely hate, I call it automating the drudgery of life,” Mike Krieger, chief product officer at Anthropic, told the Financial Times.

“At the moment, Claude can help you do things that might have taken an hour in two minutes . . . [We want to help] people focus on the creative part of [tasks] that is fun and human, and let Claude take the stuff that is repetitive and less exciting.”

It is the latest step by the San Francisco start-up, which is backed by Amazon and Google, to build more agentic systems seen as the next frontier for AI technology, alongside similar moves from rivals OpenAI, Microsoft and Meta. 

On Monday, Microsoft announced the ability to create autonomous agents in its 365 product suite, while OpenAI recently launched developer access to its voice chat feature to build more agentic systems.

Creating AI agents that can conduct mundane tasks, such as filing expenses or booking travel, is considered the future for AI companies and a way to generate revenue from their powerful but costly models.

Anthropic gave one example of a user prompt: “My friend is coming to San Francisco and I want to watch the sunrise with him tomorrow morning. I will be coming from [the] Pacific Heights [neighbourhood]. Could you find us a great viewing spot, check the drive time and sunrise time and then set up a calendar event that gives us enough time to get there?”

Claude then searched online for locations and sunrise time and used the maps application to determine the route and driving time. It later created a calendar appointment with these details.

The software uses screenshots of the computer to interpret content and can click buttons or enter text into the system, whether on a Mac or Microsoft PC. It uses real-time access to the internet and will be available to developers to build specific applications.

The company said it was still in its “early stages” for experimentation. Anthropic acknowledged that there was a possibility that the model could use unreliable information from the internet. It is also exploring how to bring these capabilities to other devices such as mobile phones.

The company is building a consumer-facing product using the technology and hopes to build user trust in its abilities. Krieger compared the technology to self-driving cars, as people generally do not yet trust the vehicles with complete control.

“I would say that’s more of the metaphor right now than I would trust it with absolute full autonomy over long periods,” Krieger added. “It will be an evolution where at first it might be in more constrained environments . . . a virtual machine with limited access, doing a particular task repeatedly.”

“I see part of our mission [as] being a helpful guide to the future of AI in a human-centric way,” he added. “And so we want to do that in our products as well with computer use.”