R

Roamkiln

Platform

Solutions

Resources

How it works

Docs

Log in

Sign up

Get a demo

Platform

Solutions

Resources

How it works

Docs

Log in

Sign up

Get a demo

Platform

Solutions

Resources

How it works

Docs

Log in

Sign up

Get a demo

Rethinking computer use with code

Mara Velde & Owen Hale

Engineering

7 min read

TL;DR: Letting a model click through every screen is flexible but slow. Pairing it with code for the predictable parts gives agents the speed of a script and the judgment of a model.

Computer use agents look at a screen and decide what to do next, one action at a time. That works well on unfamiliar pages. On pages an agent visits every day, it spends most of its time rediscovering things it already knows.

Where screenshots fall short

Every step that goes through a screenshot costs time, and every decision is a chance for a small mistake. For a form the agent fills in every day, those costs add up without adding much value.

  • Use code for steps where the page and the goal are known.

  • Hand control to the model when the page looks different from what was expected.

  • Record which path was taken so the run is easy to review.

Mixing code and model steps

In Roamkiln, a session can run a few lines of code and then call session.act() for the part that needs judgment. Both share the same browser, so cookies and page state carry over.

Over time, steps the model handles the same way on every run are good candidates to move into code. The model stays in charge of the parts that change.

What we want to learn

We are still learning where the line between code and model steps belongs for different kinds of work. If you are building agents this way, we would like to hear what you have found.

Talk to us about Roamkiln for your team

Get started with sessions for your agents

Get in touch

Roamkiln

Primitives

Browsers

Web APIs

Runtime

Identity

Model Gateway

Observability

Tinder SDK

MCP

Industries

AI

Healthcare

GTM

Tax

Legal

Developers

Docs

APIs and SDKs

Changelog

Status

Github

Company

Careers

Customers

Partner with us

Trust and Security

Privacy Policy

Terms of Service

Demo site: legal, social and company items are labels, not links.

Primitives

Browsers

Web APIs

Runtime

Identity

Model Gateway

Observability

Tinder SDK

MCP

Industries

AI

Healthcare

GTM

Tax

Legal

Developers

Docs

APIs and SDKs

Changelog

Status

Github

Company

Careers

Customers

Partner with us

Trust and Security

Community

X

LinkedIn

YouTube

Privacy Policy

Terms of Service

Demo site: legal, social and company items are labels, not links.

Roamkiln

Roamkiln

Primitives

Browsers

Web APIs

Runtime

Identity

Model Gateway

Observability

Tinder SDK

MCP

Industries

AI

Healthcare

GTM

Tax

Legal

Developers

Docs

APIs and SDKs

Changelog

Status

Github

Company

Careers

Customers

Partner with us

Trust and Security

Privacy Policy

Terms of Service

Demo site: legal, social and company items are labels, not links.

Create a free website with Framer, the website builder loved by startups, designers and agencies.