Rethinking computer use with code

Mara Velde & Owen Hale
Engineering
7 min read

TL;DR: Letting a model click through every screen is flexible but slow. Pairing it with code for the predictable parts gives agents the speed of a script and the judgment of a model.
Computer use agents look at a screen and decide what to do next, one action at a time. That works well on unfamiliar pages. On pages an agent visits every day, it spends most of its time rediscovering things it already knows.
Where screenshots fall short
Every step that goes through a screenshot costs time, and every decision is a chance for a small mistake. For a form the agent fills in every day, those costs add up without adding much value.
Use code for steps where the page and the goal are known.
Hand control to the model when the page looks different from what was expected.
Record which path was taken so the run is easy to review.
Mixing code and model steps
In Roamkiln, a session can run a few lines of code and then call session.act() for the part that needs judgment. Both share the same browser, so cookies and page state carry over.
Over time, steps the model handles the same way on every run are good candidates to move into code. The model stays in charge of the parts that change.
What we want to learn
We are still learning where the line between code and model steps belongs for different kinds of work. If you are building agents this way, we would like to hear what you have found.
Talk to us about Roamkiln for your team
Get started with sessions for your agents
Get in touch
Keep reading

Introducing Vaults in Roamkiln Functions
Mara Velde & Jonah Price
Engineering

What is Agent Passport?
Ilse Moreau
Engineering

How to design long running browser tasks with Roamkiln
Theo Brandt, Nia Okafor & Lena Sato
Engineering

Roamkiln and Keystone: shared sign in for browser agents across every app your team uses
Roamkiln Team
Company
View all blog posts
