typesafe-computer-use

A computer-use agent for macOS, written in Python, built so that a step costs about $0.0002 where a frontier model reading the same screenshot costs $0.032. It reads the screen deterministically — Vision OCR over a crop of the frontmost window, plus the labelled controls from the accessibility tree — has code add the facts a model would otherwise have to work out, and sends one TypeSafe request that answers three Choices: which kind of action, which item, which site. A writing model is called for free text and for the answer at the end, and never to pick an action. Thirteen action keys cover clicking, pressing hidden controls, typing, browser navigation, the keyboard, scrolling, waiting and stopping. Every step writes a run folder holding the capture, the payload sent and every probability the classifier returned. There is an experimental Windows adapter over UI Automation and Windows.Media.Ocr, an opt-in browser backend that reads the DOM over the Chrome DevTools Protocol, and an OSWorld agent that runs one benchmark task beside OSWorld’s own GPT agent. MIT licensed, 1,102 stars.

Who built itA software engineer who published the repository under his own name in September 2026 and wrote 109 of its 116 commits; the other seven came from seven different people, one commit each. The history carries 112 co-author trailers, 105 of them naming a Claude model — 86 “Claude Fable 5.1”, 13 “Claude Opus 5.5” and six marked with 1M context.

1. 01

   A fiftieth of a cent a step, and what the number rests on

   The price is in the README’s first sentence — a Mac driven toward a goal typed in plain English “for about a fiftieth of a cent per step” — and the table under it is the measurement behind the claim: the same screenshot, the same goal, one decision each. The input side is matched on purpose, 4,882 tokens for jev against 4,785 for Claude Opus 5 reading a bare screenshot, in a row that labels the two “same”. With input held equal the whole gap sits on the output side, and the README says why: TypeSafe answers a Choice over up to 255 options with a full probability distribution and a calibrated confidence, in a few hundred milliseconds, with free output tokens. So $0.0002 a decision against $0.032, 155 times cheaper; $0.003 against $0.40 to $0.90 for a twelve-step task; and about 1.5 seconds end to end with capture and OCR against about 5.5. The arithmetic itself is not in the material. There is no per-token rate, no multiplication and no invoice anywhere in the repository, so how 4,882 input tokens and free output tokens become $0.0002 is never shown. The figure is a measurement whose inputs are visible and whose formula is absent, and the scale is corroborated from another direction: the benchmark rows committed under benchmarks/osworld/ carry a cost per task, and the six-task batch recorded in PR 53 runs from $0.0013 to $0.0054.

2. 02

   Reading the screen instead of looking at it

   The first commit is named “Screen OCR to TypeSafe Choice to mouse click prototype”. The README’s case: a frontier model ships a screenshot every step and waits seconds for a plan, while most steps need one choice from a short list, quickly and cheaply, with a confidence number to gate on. The cost is stated in the same document. The big model read the event dates off the pixels and compared them unaided, and the classifier needed the date parsing in dates.py, because “every piece of reasoning the frontier model does for free has to be rebuilt here as deterministic state”. Perception is two-source because neither is sufficient. On ten apps on one Mac the accessibility tree labelled 100 percent of Finder’s on-screen controls and 88 percent of Chrome’s, but none of Spotify’s, whose CEF shell exposes three window buttons, so the tree is “a bonus source, never a replacement”. OCR is the expensive half, about two thirds of a step, and it charges by the amount of text rather than the number of pixels, so reading less is the only real saving: a crop of the frontmost window plus the menu bar strip over the same columns, and a cache that keeps the OCR lines of tiles that did not change between captures. That difference is measured rather than asserted: the opt-in browser backend reads the DOM instead of pixels, and against it OCR recovered only 7 of 13, 7 of 88 and 3 of 85 of the DOM’s labels.

3. 03

   Three Choices, and a rule against two options that mean one thing

   One request answers three Choices — which kind of action, which item, which site — plus a fourth when there are hidden controls to offer. The kind list runs from clicking and pressing an off-screen control through typing, browser navigation, the keyboard and scrolling to waiting, done and stop: thirteen keys. The design doc explains the split: “Every stall found while building this came from two options that meant the same thing. Confidence measures concentration, so overlapping options always read as doubt. Keep the action set mutually exclusive.” The rule is the first ground rule in CONTRIBUTING.md, and the clearest case is a reader’s report against it. In System Settings the window carries its own labelled Go Back button, so clicking it and choosing the keyboard Back are the same move: with a focus of going back, the kind question split 0.44 for the click against 0.40 for Back. Dropping the keyboard Back whenever the items include a real accessibility button labelled “Back” or “Go Back” — not just text that happens to read that way off the screen — moved the next decision to a 0.71 click. Two contributors wrote a fix independently, and the browser loop carries the same instinct differently: an option the loop cannot execute is filtered out before the classifier sees it, since a guaranteed stall reads as model doubt.

4. 04

   How a wrong click is caught, and the two places it still is not

   A click is not a mouse click if it does not have to be: an item that came from the accessibility tree is pressed through the tree, “so the press lands on the control rather than on whatever covers it”, and a mouse click at the centre of the box is the fallback, after closing whatever popup is in front. The bug behind that rule is written up. On one benchmark task, in all three trials, Chrome’s “Restore pages?” bubble sat over the bookmark manager’s top-right corner, the classifier picked the page’s Organise button, and the click closed the bubble instead; the run spent two steps and about thirteen seconds finding out. The fix treats a popup as a window beside the browser window and closes it with its own close button, or Escape when it has none, never with its other buttons, because “Restore” reopens the last session. The wording was tuned on a replay of 81 captured requests from screens with a popup, where calling an item “covered by” the popup drew the classifier toward Escape and lowered its confidence, while “under” left both as they were. The coarser net is the stall rule: each step keeps a signature of the screen, and the run stops after three actions in a row that left it unchanged, or after two already taken on that screen earlier. Those rules lean the other way on purpose — when more changes than that every step, a run getting nowhere reaches its step limit, because they “err toward running on, never toward stopping a run that is making progress”. A run also stops on Ctrl-C or by slamming the mouse into the top-left corner, checked before every click, key and scroll and between typed characters, so it stops mid-word. Two gaps are reported and still open: a typed field whose value does not read back falls back to keystrokes aimed at whatever holds focus by then, and the browser loop ends a run at a satisfaction score of 0.5, a coin flip that one report says overrides a confident click.

5. 05

   Thirteen days, one beta release, and a README cut from 725 lines

   The repository was created on 2026-09-16 and the last commit before this record landed on 2026-09-29: 116 commits in thirteen days, 109 of them by the author. There is exactly one release, published on 2026-09-22 and titled “v0.2.0 (beta)” with the prerelease flag set, and two tags, v0.1.0 and v0.2.0; the project file says 0.2.0. The beta notice is direct: “This is under heavy development. … It drives your real mouse and keyboard, so start with a dry run.” The documentation was reorganised once, with the numbers recorded: the README went from 725 lines to 146, keeping only what a newcomer needs in the first two minutes, and everything else moved into docs/, one page per topic. Because the tool drives a real computer — usually the maintainer’s own Mac while they are using it — the rules written for agents working on it say an agent must ask first, every time, before anything that would take over the screen, the input, the apps or the cloud machine, and one approval covers that command, not that kind of command from then on. Tests are kept off the machine by a guard in tests/conftest.py that makes every call which would reach it refuse, and a new call must join the guard in the same change. A task the loop cannot do becomes a scenario on a simulated computer in tests/world.py, driven by the real step loop with a policy standing in for the classifier, and left failing until the loop can do it, because “every stop rule in the loop was found or fixed that way”. The limits are written down rather than left to be discovered: only the main display is captured, an icon-only button in a terminal, a canvas or Spotify reaches neither source, and using the machine during a run fights it for focus and the cursor.

6. 06

   The benchmark, a failure kept on the record, and the outside contributors

   jev runs in OSWorld as an agent beside OSWorld’s own GPT agent on the same task, with the same fifty steps and the same two-second pause after each action. On the first real Chrome task, after four runs and a list of fixes, jev finished with a score of 1.0 in six clicks, while the comparison agent never ran because the account’s access to the machine ended mid-session. The change the author held open on purpose is the revealing one. OSWorld waits a fixed two seconds after every action, which charges per step and penalises an agent that takes more, cheaper steps, so the wait went to zero — but jev’s own “wait” action pointed at the same constant, so at zero a wait waited nothing. At zero seconds the run solved one task of six against six of six at the baseline, and the causes are recorded rather than summarised: the screenshot is taken before Chrome has drawn the page; jev’s tree is fetched after that screenshot, so the two disagree; and a refused action sends nothing, so no new observation arrives. The pull request says it is “held open by the maintainer’s request: speed and accuracy get weighed across this and the next two changes before anything merges”, and the failing rows were committed in the meantime. A file kept for the comparisons that do not flatter the design records that jev sees only its last eight actions as text and cannot tell whether an action worked, and that the classifier picks the same item for the same request only about 95 percent of the time. The outside contribution is thin: seven people other than the author, one commit each, and the thirty most recent issues and pull requests, numbered 31 to 60, are largely theirs. One reader filed six numbered bugs; another posted seven macOS bugs from three small tasks — a multiplication in Calculator, a new file in a text editor, a link click in a browser — with 740 passing tests. The seventh only a real machine produces: a shortcut held Command down after the press, so every letter typed afterwards arrived as a Command shortcut, nothing reached the document, and the next click landed as a Command-click.