Recipes
View as Markdown Open in ChatGPT Open in Claude

Computer Use Agent

This recipe creates a full virtual desktop sandbox and connects it to Claude's Computer Use capability — letting the agent see the screen, click, type, and launch applications.

Prerequisites

  • Caged CLI installed (brew install caged-dev/tap/caged or curl -fsSL https://get.caged.dev | sh)
  • Caged account with API key (caged login)
  • An ANTHROPIC_API_KEY for Claude Computer Use

Quick Start

CLI

# Create a desktop sandbox
caged up --template desktop --budget 5

# Get sandbox ID
SANDBOX_ID=$(caged list --format json | jq -r '.[0].id')

# Take a screenshot via API
curl "https://api.caged.dev/v1/sandboxes/$SANDBOX_ID/screen/screenshot?format=base64" \
  -H "Authorization: Bearer $CAGED_API_KEY"

Python SDK

import caged

# Create desktop sandbox
sandbox = caged.create(template="desktop", budget=5.0)

# Take a screenshot
result = sandbox.mcp.call("screen_screenshot", {})
print(f"Screen: {result['width']}x{result['height']}")

# Launch browser
sandbox.mcp.call("screen_launch", {"command": "chromium-browser https://example.com"})
sandbox.mcp.call("screen_wait", {"seconds": 3})

# Click on something
sandbox.mcp.call("screen_click", {"x": 640, "y": 300})

# Type text
sandbox.mcp.call("screen_type", {"text": "Hello from Caged!"})
sandbox.mcp.call("screen_key", {"key": "Return"})

Full Computer Use Agent

This example creates a complete agent loop that connects Claude's Computer Use to a Caged desktop sandbox:

import anthropic
import caged
import base64
import json
from typing import Any

def run_computer_use_agent(task: str, budget: float = 10.0):
    """Run a Computer Use agent in an isolated Caged sandbox."""
    
    # Create desktop sandbox
    sandbox = caged.create(
        template="desktop",
        budget=budget,
        resources={"memory": 4096, "cpu": 2}
    )
    print(f"Sandbox created: {sandbox.id}")
    
    # Initialize Claude
    client = anthropic.Anthropic()
    
    # Computer Use tool definition
    computer_tool = {
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 1280,
        "display_height_px": 800,
        "display_number": 0
    }
    
    def execute_computer_action(action: str, **kwargs) -> dict:
        """Execute a computer action in the sandbox."""
        
        if action == "screenshot":
            result = sandbox.mcp.call("screen_screenshot", {})
            return {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": result["image_base64"]
                }
            }
        
        elif action == "mouse_move":
            x, y = kwargs["coordinate"]
            sandbox.mcp.call("screen_click", {"x": x, "y": y})
            return {"success": True}
        
        elif action == "left_click":
            sandbox.mcp.call("screen_click", {"x": kwargs.get("x", 0), "y": kwargs.get("y", 0)})
            return {"success": True}
        
        elif action == "right_click":
            sandbox.mcp.call("screen_click", {"x": kwargs.get("x", 0), "y": kwargs.get("y", 0), "button": "right"})
            return {"success": True}
        
        elif action == "double_click":
            sandbox.mcp.call("screen_click", {"x": kwargs.get("x", 0), "y": kwargs.get("y", 0), "double": True})
            return {"success": True}
        
        elif action == "type":
            sandbox.mcp.call("screen_type", {"text": kwargs["text"]})
            return {"success": True}
        
        elif action == "key":
            sandbox.mcp.call("screen_key", {"key": kwargs["key"]})
            return {"success": True}
        
        elif action == "scroll":
            direction = "down" if kwargs.get("coordinate", [0, 1])[1] > 0 else "up"
            info = sandbox.mcp.call("screen_info", {})
            sandbox.mcp.call("screen_scroll", {
                "x": info.get("mouse_x", 640),
                "y": info.get("mouse_y", 400),
                "direction": direction,
                "amount": abs(kwargs.get("coordinate", [0, 3])[1])
            })
            return {"success": True}
        
        else:
            return {"error": f"Unknown action: {action}"}
    
    # Start agent loop
    messages = [{"role": "user", "content": task}]
    
    while True:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=4096,
            tools=[computer_tool],
            messages=messages
        )
        
        # Check for end conditions
        if response.stop_reason == "end_turn":
            # Extract final text response
            for block in response.content:
                if hasattr(block, "text"):
                    print(f"\nAgent completed: {block.text}")
            break
        
        # Process tool calls
        tool_results = []
        for block in response.content:
            if block.type == "tool_use" and block.name == "computer":
                action = block.input.get("action")
                print(f"Action: {action}")
                
                result = execute_computer_action(action, **block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": [result] if action == "screenshot" else json.dumps(result)
                })
        
        # Continue conversation
        messages.append({"role": "assistant", "content": response.content})
        messages.append({"role": "user", "content": tool_results})
        
        # Check budget
        if sandbox.cost > budget * 0.9:
            print(f"Warning: Approaching budget limit (${sandbox.cost:.2f}/${budget})")
    
    # Cleanup
    sandbox.destroy()
    print(f"Total cost: ${sandbox.cost:.2f}")

# Run the agent
run_computer_use_agent(
    task="Open Chrome, go to news.ycombinator.com, and summarize the top 3 stories",
    budget=5.0
)

Config File

For reproducible setups, use .caged.yaml:

# .caged.yaml
template: desktop

resources:
  memory: 4096  # 4GB recommended for desktop
  cpu: 2
  disk: 10240   # 10GB for browser cache, downloads

timeout: 7200   # 2 hours max
budget: 10.00   # $10 hard limit

env:
  DISPLAY: ":99"
  SCREEN_WIDTH: "1280"
  SCREEN_HEIGHT: "800"

Then run:

caged up

Available Screen Tools

Tool Description
screen_screenshot Capture PNG screenshot (base64)
screen_click Click at coordinates (left/right/double)
screen_type Type text with optional delay
screen_key Press keys (Return, Tab, ctrl+c, etc.)
screen_scroll Scroll up/down at coordinates
screen_drag Drag from start to end coordinates
screen_launch Launch an application
screen_info Get screen size, mouse position, active window
screen_wait Wait for animations/page loads (max 30s)

WebSocket Streaming

For real-time screen updates, use the WebSocket endpoint:

const ws = new WebSocket(
  `wss://api.caged.dev/v1/sandboxes/${sandboxId}/screen`,
  { headers: { Authorization: `Bearer ${apiKey}` } }
);

// Request continuous screenshots
setInterval(() => {
  ws.send(JSON.stringify({ type: "screenshot" }));
}, 500);

// Handle responses
ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === "screenshot") {
    updateScreenDisplay(msg.image_base64);
  }
};

// Send interactions
function click(x, y) {
  ws.send(JSON.stringify({ type: "click", x, y }));
}

Observability

All screen actions are captured in session replay:

# View session replay
caged sessions list
caged replay sess_xxxxx

# Or via API
curl "https://api.caged.dev/v1/sessions/sess_xxxxx/replay" \
  -H "Authorization: Bearer $CAGED_API_KEY"

The replay includes:

  • Screenshots before and after each action
  • Click coordinates and button states
  • Typed text (sensitive data redacted)
  • Application launches
  • Trust score changes

Trust Scoring

Desktop actions affect the session trust score:

Action Score Impact
Taking screenshots +1 (observation before action)
Launching browser +1 (normal application use)
Typing credential-like text -10 (potential credential exposure)
Running suspicious commands -15 (potentially malicious)

Set a minimum trust threshold in your policy:

# Policy requiring high trust
policy:
  trust:
    min_score: 70
    action: pause  # pause for approval if score drops

Best Practices

Always take a screenshot before interacting to understand screen state

Use screen_wait after launching apps or loading pages

Desktop sandboxes use more resources — always set a budget limit

For real-time control, use WebSocket instead of REST

Limitations

  • Resolution: Default 1280×800 (configurable via env vars)
  • GPU: Software rendering only (no hardware acceleration)
  • Audio: Not supported
  • Clipboard: Use screen_type and screen_key for copy/paste
  • Multiple monitors: Single display only
Was this page helpful?