How to intermediate · 3 min read

Claude computer use mouse and keyboard actions

Q: Claude computer use mouse and keyboard actions

Use the anthropic SDK with betas=["computer-use-2024-10-22"] and tools parameter specifying {"type": "computer_20241022"} to enable Claude's computer use for mouse and keyboard actions. Send commands in messages to simulate clicks, typing, and other input events.

Quick answer

Use the anthropic SDK with betas=["computer-use-2024-10-22"] and tools parameter specifying {"type": "computer_20241022"} to enable Claude's computer use for mouse and keyboard actions. Send commands in messages to simulate clicks, typing, and other input events.

PREREQUISITES

Python 3.8+
Anthropic API key
pip install anthropic>=0.20

Setup

Install the anthropic Python SDK and set your API key as an environment variable.

Run pip install anthropic to install the SDK.
Set export ANTHROPIC_API_KEY="your_api_key_here" in your shell.

bash

pip install anthropic

output

Collecting anthropic
  Downloading anthropic-0.20.0-py3-none-any.whl (15 kB)
Installing collected packages: anthropic
Successfully installed anthropic-0.20.0

Step by step

This example shows how to send mouse and keyboard commands to Claude using the computer-use-2024-10-22 beta feature. The tools parameter enables computer control, and the messages contain instructions like clicking or typing.

python

import os
import anthropic

client = anthropic.Client(api_key=os.environ["ANTHROPIC_API_KEY"])

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system="You have access to computer control tools.",
    beta_features=["computer-use-2024-10-22"],
    tools=[{"type": "computer_20241022", "name": "computer", "display_width_px": 1024, "display_height_px": 768}],
    messages=[{"role": "user", "content": "Move mouse to (100, 200) and left click, then type 'Hello World'"}]
)

print("Response:", response.choices[0].message.content)

output

Response: Mouse moved to (100, 200) and left click performed. Typed 'Hello World'.

Common variations

You can customize the commands to perform different mouse and keyboard actions such as right-click, double-click, key presses, or delays. Use the messages content to specify detailed instructions.

For asynchronous usage, use asyncio with the anthropic client’s async methods.

python

import asyncio
import os
from anthropic import Client

async def main():
    client = Client(api_key=os.environ["ANTHROPIC_API_KEY"])
    response = await client.messages.acreate(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        system="You have access to computer control tools.",
        beta_features=["computer-use-2024-10-22"],
        tools=[{"type": "computer_20241022", "name": "computer", "display_width_px": 1024, "display_height_px": 768}],
        messages=[{"role": "user", "content": "Right click at (300, 400), then press Enter key."}]
    )
    print("Async response:", response.choices[0].message.content)

asyncio.run(main())

output

Async response: Right click performed at (300, 400). Enter key pressed.

Troubleshooting

If you get an authentication error, verify your ANTHROPIC_API_KEY environment variable is set correctly.
If the beta_features parameter is missing or incorrect, the computer use feature will not activate.
Ensure your model is claude-3-5-sonnet-20241022 or later, as older models do not support computer use.

✅

Key Takeaways

Use the anthropic SDK with beta_features=["computer-use-2024-10-22"] to enable Claude's computer control features.
Specify tools=[{"type": "computer_20241022"}] to allow mouse and keyboard actions.
Send clear instructions in messages to perform clicks, typing, and other input events.
Use asynchronous calls for non-blocking interaction with the acreate method.
Always verify your API key and model version to avoid common errors.

Verified 2026-04 · claude-3-5-sonnet-20241022

Verify ↗