We measured the accessibility tree against screen capture. Both lose.

If you are building software that reads what is on someone's Mac, you pick one of two primitives. You capture the screen and run OCR on the pixels, or you read the accessibility tree that macOS already exposes to screen readers.

This gets discussed as a preference. Pixels are universal but expensive. AX is cheap but spotty. Pick your tradeoff.

We have both paths compiled in the same binary, so we measured them against each other on the same windows on the same machine in the same session. The tradeoff framing is wrong. On the single window where we could get a clean head-to-head, both approaches failed to return the content anyone would actually want, in different directions, and one of them failed while reporting success.

That last part is the finding worth your time.

The number#

Chrome. One window, 1710 by 985 points, 3420 by 1970 at 2x. Ten runs of each path, median reported.

Capture plus OCR is 6.6 times slower and burns more than forty times the CPU, and it still could not read the page. The accessibility tree returned twenty-five times more tokens, and none of them were the document either. Zero AXWebArea elements. Walking to depth 12 got us to 165 nodes, all of it more chrome.

Someone reading an article in Chrome, asking a question about it: neither primitive has the article.

Machine was an M2 on macOS 26.5.1, arm64. mac-ax 1.2.5, mac-vision 1.0.1. Screen Recording was already granted on this machine. These Vision numbers are real. They are not the shipped product default, which leaves capture compiled and off.

Failures that report success#

The Chrome result is the honest kind of failure. You ask for content, you get chrome, the node count is small, and nothing pretends otherwise.

The interesting failures are the ones that come back looking rich.

Three apps in our sample returned an empty kAXWindows array. When that happens, the walker climbs to the application element instead of failing, and what it hands back is enormous:

Terminal produced 11,630 tokens. It is File, Edit, View, and every submenu underneath them. Around 96% of those nodes are hidden. The call returns ok: true.

To be precise about what this does and does not show: an app with no open window returning no window content is correct behavior, not a bug. Calendar, Messages, and Terminal were running with zero visible windows. What matters is the shape of the response. From the calling side you get a successful read, a four-figure node count, and a tree that contains a menu bar and nothing else. If your code checks ok && nodeCount > 0 before deciding an app is readable, all three of these pass, and a model planning against that tree will plan against File and Edit.

Our own source has a comment admitting this path once turned a Mail read into a 5,000-node cap hit. Same behavior, wearing a green check.

Electron gives you two useless answers#

Cursor, measured across ten runs: 52ms, 9 nodes. Two window elements and seven groups. Zero buttons. Zero text. The editor is not in the tree.

In the same session, when the window elements disappeared, the same call returned 207 nodes, 206 of them menu items.

Both answers are wrong and neither reports an error. Any capability probe that asks "does this app have AX" gets yes in both cases.

The depth trap#

Finder was the best result we got. A real window, 92 nodes at the default walk depth of 6, including 61 file icon names. That is genuinely usable.

Then we asked for depth 12 and the tree went to 448 nodes.

So the default walk was showing us a folder with most of the folder missing, and there was nothing in the response indicating that. Depth is a silent truncation. Pick 6 and you get an answer. It is just not the whole answer, and you will not be told.

Mail is the polite version of the same problem. A real 1488 by 928 window, 46 nodes, two scroll areas, ten text nodes. It looks like Mail. It is not your mailbox, because the message list and body are inside scroll views that have not been materialized. Depth 12 does not help.

Vision will read the wrong window and not mention it#

This one is worth publishing on its own.

Our capture path takes a pid filter. When the filter fails to match, it falls back to the frontmost window. We passed pids for Mail, Finder, Messages, Terminal, and Cursor. Every one of them returned Chrome's 1710 by 985 window.

We only caught it because we were checking the returned point size against a known AX window frame. Without that check, this entire measurement run would have been seven OCR passes over one browser screenshot, each one labeled with a different app name, and every number in the table would have looked plausible.

If you are building on screen capture, the failure mode is not "the OCR was wrong." It is that you captured a different application than the one you asked for, with full confidence, and nothing in the response tells you.

What we did not measure#

Character-level accuracy against rendered text. That would mean publishing the contents of a real inbox, so we did not take it.

Placeholder and accessibility-label text versus what is actually rendered on screen. Known to diverge. Not measured here.

Overlap scoring between the two paths. The window mismatch above means any similarity number from this run would be comparing two different applications. Do not cite one.

Figma, Miro, and the rest of the canvas apps. Not installed on this machine. We have a denylist for them in our own code and this run confirmed nothing about it either way.

The trade we took#

Accessibility: 52 to 306ms depending on the app, CPU that does not register on a 10ms clock, one permission that macOS calls Accessibility, and it can drive the UI as well as read it.

Screen capture: 509ms and 425ms of CPU on the one window we could match, the Screen Recording permission, a recording indicator in the menu bar, and every pixel on the display including windows you are not driving.

We ship the accessibility path and keep the capture path compiled but off by default. The speed matters less than the fact that the fast path is the one that can act, and 17 lines of OCR from a full browser window is not a product.

That is still a bet, and this data is the argument against it as much as for it. Four applications on one desk returned a tree with nothing usable in it. Two of them returned a large tree with nothing usable in it, which is worse. Canvas apps are hopeless, Electron apps are a coin flip, and any app whose window list is momentarily empty will hand you a menu bar and call it a success.

If you are building on AX, the thing to take from this is not a latency number. It is that ok: true and a healthy node count tell you nothing, and you need a content check before you let a model plan against a tree.

A note on how Melvin actually works#

Because the table above invites the wrong conclusion: the accessibility tree is not how we read Mail, Calendar, Notes, Chrome, Slack, or Finder. Those go through purpose-built paths. Calendar reads through EventKit. Mail reads the local Envelope database. Notes reads NoteStore. Chrome page content comes from the DevTools protocol against a live tab, which is why zero AXWebArea elements is expected rather than alarming. Slack is the Web API. Finder operations are filesystem operations.

AX is the long-tail lane, for the apps that have no API and no database to read. That is where the coverage table above is the real constraint, and it is why we measured it.