Technical accessibility project
ScreenRecognition: VoiceOver support for apps that don't have any.
Some professional software is completely invisible to VoiceOver. I worked out and tested a Mac app that finds controls on screen, reads their text with OCR, and puts an accessible layer on top.
What this project shows
I needed to use software that wasn't accessible, so I designed and tested a way in.
ScreenRecognition uses a YOLO model to find UI elements on screen, Vision framework OCR to read their text, and a transparent overlay of NSAccessibilityElements so VoiceOver can navigate and interact with them.
My contribution
- Defined the user problem, product behavior, and accessible interaction model
- Directed implementation across the macOS accessibility, OCR, and object-detection layers
- Tested each iteration with VoiceOver and identified incorrect roles, labels, focus, and input behavior
- Set acceptance criteria based on whether previously invisible controls became operable
- Made technical tradeoffs using daily VoiceOver experience
How it works
It works in three steps.
Detection
A YOLO object detection model trained to recognize five common UI control types: buttons, disclosure triangles, images, links, and text areas. The model runs against a screen capture of the target application.
Recognition
Apple's Vision framework performs OCR on detected elements and their surrounding context to extract labels, placeholder text, and nearby descriptive content. A label association algorithm matches text to the correct control using spatial proximity and layout heuristics.
Overlay
A transparent window sits over the target application. Each detected element becomes an interactive NSAccessibilityElement with the correct role, label, and click action. VoiceOver sees and navigates these elements as if the app had built-in accessibility support.
The result
Apps that were blank to VoiceOver become partially navigable. Buttons can be clicked, text fields identified, links followed. It's not as good as real accessibility support in the app, but it gets you in.
Technical decisions
Label association
YOLO often classifies nearby text labels as buttons. Rather than treating this as a model error, the label association algorithm identifies when a detected "button" is actually a label for an adjacent text field, using tight row-alignment tolerances to avoid picking up section headers.
Proximity fallback
When the ML model misses a label entirely, a fallback system uses OCR results and spatial proximity to find the nearest text that could plausibly describe a control. This catches cases the model was not trained on.
Text input handling
Text fields use NSTextView rather than NSTextField to avoid a VoiceOver bug where the first visit to a text field announces it as static text instead of an editable field. I found this because I use VoiceOver every day.
My role
What my part was
I worked out what it should do, reviewed the code as it was built, and tested every version with VoiceOver.
Implementation uses
- Swift and Objective-C
- YOLO v8 object detection (trained on custom UI element dataset)
- Apple Vision framework for OCR
- NSAccessibility protocol for accessible element creation
- Core Graphics for screen capture and coordinate mapping
Want to see more of my work?
The portfolio includes web and mobile accessibility audits alongside these technical projects.