Devops Monk

Claude’s computer use capability — the ability to see your screen and interact with applications via mouse clicks, keyboard input, and scrolling — is one of the most discussed features in the AI space. It is also one of the most misapplied.

The correct mental model is not “Claude can now automate everything on my screen.” It is “Claude now has a flexible fallback layer for tasks that do not have a structured API integration.”

How Computer Use Actually Works

sequenceDiagram
    participant You as Developer
    participant Claude as Claude
    participant Screen as Your Screen

    You->>Claude: Check the Vercel deploy and send me the URL if green
    Claude->>Screen: Take screenshot
    Screen-->>Claude: Current screen state
    Claude->>Screen: Navigate to Vercel dashboard
    Screen-->>Claude: Dashboard screenshot
    Claude->>Screen: Read deploy status
    Claude-->>You: Deploy is green, URL is https://app.example.com

Claude receives screenshots of your screen and sends back coordinates to click, text to type, or scroll instructions. It operates like a remote control for your desktop.

This is fundamentally different from an API integration: it is slower, less reliable when UI layouts change, and has inherent overhead from screenshot processing. But it works on anything with a UI — including tools that have no API, legacy internal dashboards, and one-off tasks that do not justify building a dedicated integration.

The Two-Layer Tool Model

The right architecture for AI-powered DevOps workflows uses two layers:

flowchart TD
    Task[Automation task] --> Q1{Does a reliable API\nor MCP server exist?}
    Q1 -->|Yes| API[Use API or MCP\nFast, reliable, structured]
    Q1 -->|No| Q2{Is this task\nrepeated daily?}
    Q2 -->|Yes| Build[Build an API integration\nWorth the investment]
    Q2 -->|No - one-off or occasional| CU[Use computer use\nFlexible fallback]

Layer 1 — Structured integrations (APIs and MCP servers): GitHub, PagerDuty, AWS CloudWatch, Linear, Sentry. These are fast, deterministic, and the right choice for anything you do every day.

Layer 2 — Computer use (UI fallback): The legacy internal tool that has no API. The vendor dashboard you use twice a month. The one-off task that would take longer to write a script for than to just do with computer use.

Practical DevOps Use Cases

Verify a deployment across multiple dashboards

You want to confirm a deploy is healthy: the CI job passed, the deployment is running, the error rate did not spike. Three different tools, none of which share data directly.

After this deploy completes, check:
1. GitHub Actions — confirm the deploy workflow shows success
2. DataDog — confirm error rate is below 0.1% for the payments service
3. Our internal status page — confirm all services show green

Message me on Slack with a summary of what you find.

Without computer use, this is three manual checks across three tabs. With computer use, it is a single delegated task.

Interact with a legacy internal tool

Many engineering organisations have internal tools built years ago with no REST API. Computer use makes Claude an interface to those tools without requiring a rewrite or a custom integration.

Go to the internal deployment portal at deploy.internal.company.com,
find the batch job that failed at 14:30 UTC, and restart it.

Screenshot-based design review

When a designer shares a Figma link and you want Claude’s analysis without building a Figma MCP integration:

Open this Figma URL, screenshot the checkout flow components,
and tell me if the spacing is consistent with our design system guidelines.

The Dispatch Pattern

The most productive way to use computer use is asynchronously — you send the task from your phone or another device while Claude works on your desktop in the background.

sequenceDiagram
    participant Phone as Your Phone
    participant Claude as Claude (on desktop)
    participant Desktop as Desktop Tools

    Phone->>Claude: Check if the staging deployment is healthy and Slack me
    Note over Claude,Desktop: You go into a meeting
    Claude->>Desktop: Opens CI dashboard
    Claude->>Desktop: Opens DataDog
    Claude->>Desktop: Checks Sentry for new errors
    Claude->>Desktop: Posts summary to Slack
    Note over Phone: You receive Slack message during meeting

The task is done before you return. This works particularly well for:

• Pre-meeting checks you would otherwise rush through
• Deployment verification that does not need your full attention
• Status gathering across multiple tools before a standup

When Not to Use Computer Use

For daily repeated tasks — If you are checking the same three dashboards every morning, build a proper integration or use MCP. Computer use has enough overhead and fragility that repeating it daily will eventually break.

For write operations on critical systems — Computer use makes mistakes on complex multi-step workflows more often than API-based approaches. Do not use it to apply Terraform or run database migrations.

When a UI changes frequently — Computer use instructions reference UI elements by position and appearance. If the vendor redesigns their dashboard, the instructions break. API integrations are immune to visual redesigns.

For anything requiring speed — API calls take milliseconds. Computer use takes seconds per step. For latency-sensitive automation, API is the only option.

Getting Started

Computer use in Claude Code is available in the desktop app. To enable it for a task:

Use computer use to: [your task description]

Be specific about what success looks like and what tools to use. Vague instructions produce slow, wandering behaviour. Clear instructions with specific tool names and expected outcomes produce fast, accurate results.

Start with low-risk, read-only tasks: check a status, read a dashboard, screenshot a UI. Build confidence with those before delegating write operations.

Summary

Computer use is a powerful capability best understood as a flexible fallback rather than a primary automation strategy. The decision framework is straightforward:

• Use APIs and MCP servers for structured, daily, reliable workflows
• Use computer use for UI-only tools, one-off tasks, and asynchronous delegation
• Do not use computer use for anything that needs to run reliably at scale

The teams getting the most value from it are those who have already set up their core integrations properly and are using computer use to fill the gaps — not those trying to replace structured automation with screenshot-based interactions.