K
Kuldeep Singh
Guest
Testing my Chrome extension meant opening its side panel and reloading it after every change, by hand. I wanted a coding agent to run that loop. Browser-automation tools couldn't, because they drive the page and the toolbar and side panel belong to the browser. Codex's computer use showed me an agent could do it on a Mac, so I built an open one.
That small problem led me into the part of macOS that screen readers use, and it turned out to be a good interface for agents too. This is what I learned building YouxPC on it, including three things I got wrong.
macOS exposes every window as a tree of elements. Each has a role (button, text field, link), usually a name, and sometimes a value. This is the accessibility tree, the thing VoiceOver reads.
For an agent, that's a description of the screen with no pixels and no vision model. It also supports actions: a button can be asked to press itself. No mouse event is involved, so the pointer doesn't move, and the app doesn't need to be in front.
That's the whole idea. The rest is the work of making it dependable.
One short line per useful element, with a number on the ones you can act on:
An element costs about 38 characters in this format. The JSON element list in the project I forked cost 144 for the same element. The agent acts with
My first version numbered elements by position. It worked in every demo and failed on the first real table.
Take six rows, each with an Approve and a Reject button. Remove one row and every later button moves up one ID. An ID the agent read a moment ago now presses a different row's button. Nothing errors, and the wrong person gets approved.
The fix is that an ID belongs to one element for the session. If its element is gone, the ID is retired and not handed to a neighbor. I checked it on a six-row table with identical buttons. After approving one row, the IDs I'd read earlier for two other rows still pressed those rows' buttons, and they held across a modal dialog opening and closing.
There's a related rule. When a click names something several controls share, like "Approve" with six matches, the tool refuses and lists each candidate's ID and row. Guessing the first match was the other way to approve the wrong person.
My early code activated the target app and then sent synthetic key events. Activation is asynchronous, and keystrokes go to whatever is frontmost when they arrive. In one test run, the activation was late, and my test input went into the chat window of the app hosting the agent.
The design was wrong, and bad luck had little to do with it. Every keystroke is now posted to a specific process, never to "whatever is frontmost." The one exception is Cmd and Option shortcuts. Apps ignored those when I posted them to their process, so they go through the frontmost app after focus is confirmed.
I also tried the obvious way to click without moving the pointer, which is posting mouse events straight to a background app's process. Chrome ignored them in every variant I tried. That's why clicks go through accessibility.
In an early format, an element read
The format you show the model is part of the product. Whatever it can confuse, it eventually will.
Access is also all-or-nothing at the operating-system level, so the narrowing has to happen in the tool. YouxPC blocks the app hosting the agent, password managers, System Settings, and terminals by default.
I benchmarked four free desktop tools on the same 17 tasks with the same model. YouxPC passed 50 of 51 runs in about half the time of the one tool that passed one more. I wrote the tasks myself and built YouxPC alongside them, and it's one model on one Mac, so read that as a first measurement. The full write-up, with the failed run shown, is on Hashnode (https://ar5en1c.hashnode.dev/desktop-control-benchmark) and in the repository.
If you try it, the most useful thing you can tell me is an app where the tree is empty or wrong.
That small problem led me into the part of macOS that screen readers use, and it turned out to be a good interface for agents too. This is what I learned building YouxPC on it, including three things I got wrong.
The screen already has a text version.
macOS exposes every window as a tree of elements. Each has a role (button, text field, link), usually a name, and sometimes a value. This is the accessibility tree, the thing VoiceOver reads.
For an agent, that's a description of the screen with no pixels and no vision model. It also supports actions: a button can be asked to press itself. No mouse event is involved, so the pointer doesn't move, and the app doesn't need to be in front.
That's the whole idea. The rest is the work of making it dependable.
What a window looks like to the agent.
One short line per useful element, with a number on the ones you can act on:
Code:
Brave Browser — "Button panel - Brave"
web "Button panel"
# "Button panel"
[e14] button "Alpha"
[e15] button "Bravo"
[e17] button "Delta"
"Nothing pressed yet."An element costs about 38 characters in this format. The JSON element list in the project I forked cost 144 for the same element. The agent acts with
pc_click index:"e17", and the answer says what changed on screen, so it rarely needs to look again.Mistake One: IDs that were positions
My first version numbered elements by position. It worked in every demo and failed on the first real table.
Take six rows, each with an Approve and a Reject button. Remove one row and every later button moves up one ID. An ID the agent read a moment ago now presses a different row's button. Nothing errors, and the wrong person gets approved.
The fix is that an ID belongs to one element for the session. If its element is gone, the ID is retired and not handed to a neighbor. I checked it on a six-row table with identical buttons. After approving one row, the IDs I'd read earlier for two other rows still pressed those rows' buttons, and they held across a modal dialog opening and closing.
There's a related rule. When a click names something several controls share, like "Approve" with six matches, the tool refuses and lists each candidate's ID and row. Guessing the first match was the other way to approve the wrong person.
Mistake Two: A keystroke that landed in the wrong app.
My early code activated the target app and then sent synthetic key events. Activation is asynchronous, and keystrokes go to whatever is frontmost when they arrive. In one test run, the activation was late, and my test input went into the chat window of the app hosting the agent.
The design was wrong, and bad luck had little to do with it. Every keystroke is now posted to a specific process, never to "whatever is frontmost." The one exception is Cmd and Option shortcuts. Apps ignored those when I posted them to their process, so they go through the frontmost app after focus is confirmed.
I also tried the obvious way to click without moving the pointer, which is posting mouse events straight to a background app's process. Chrome ignored them in every variant I tried. That's why clicks go through accessibility.
Mistake Three: A model that couldn't tell a key from an ID.
In an early format, an element read
[7] button "9". On Calculator, with keys named "7", "8", "9", a model was asked for 47 × 19 and pressed 13 × 79. IDs are now written e7, so a number can't be mistaken for a key. If the model uses a label as an ID anyway, the answer says which ID that button actually has.The format you show the model is part of the product. Whatever it can confuse, it eventually will.
What accessibility isn't good at.
- Canvas apps, games, and custom-drawn interfaces have nothing to list.
- VS Code exposes almost nothing, so it's close to unusable here.
- Chromium only builds a page's accessibility tree once a client asks for it, so the first snapshot of a browser takes about a second.
- Nothing asks for confirmation before sending, deleting, or paying. A never-press list helps with accidents, but you have to set it, and it isn't a security boundary.
Access is also all-or-nothing at the operating-system level, so the narrowing has to happen in the tool. YouxPC blocks the app hosting the agent, password managers, System Settings, and terminals by default.
Was it worth it?
I benchmarked four free desktop tools on the same 17 tasks with the same model. YouxPC passed 50 of 51 runs in about half the time of the one tool that passed one more. I wrote the tasks myself and built YouxPC alongside them, and it's one model on one Mac, so read that as a first measurement. The full write-up, with the failed run shown, is on Hashnode (https://ar5en1c.hashnode.dev/desktop-control-benchmark) and in the repository.
If you try it, the most useful thing you can tell me is an app where the tree is empty or wrong.
- Code and install: https://github.com/Ar5en1c/youxpc
- Replay of real runs and charts: https://ar5en1c.github.io/youxpc/