Technical accessibility project
ScreenRecognition: VoiceOver support for apps that don't have any.
Some professional software is completely invisible to VoiceOver. I defined and tested a macOS concept that detects visual controls, reads their text with OCR, and creates an accessible overlay.
What this project shows
I needed to use software that wasn't accessible, so I designed and tested a way in.
ScreenRecognition uses a YOLO model to find UI elements on screen, Vision framework OCR to read their text, and a transparent overlay of NSAccessibilityElements so VoiceOver can navigate and interact with them.
My contribution
- Defined the user problem, product behavior, and accessible interaction model
- Directed implementation across the macOS accessibility, OCR, and object-detection layers
- Tested each iteration with VoiceOver and identified incorrect roles, labels, focus, and input behavior
- Set acceptance criteria based on whether previously invisible controls became operable
- Made technical tradeoffs using daily VoiceOver experience
How it works
Three layers working together to turn a visual-only interface into something VoiceOver can navigate.
Detection
A YOLO object detection model trained to recognize five common UI control types: buttons, disclosure triangles, images, links, and text areas. The model runs against a screen capture of the target application.
Recognition
Apple's Vision framework performs OCR on detected elements and their surrounding context to extract labels, placeholder text, and nearby descriptive content. A label association algorithm matches text to the correct control using spatial proximity and layout heuristics.
Overlay
A transparent window sits over the target application. Each detected element becomes an interactive NSAccessibilityElement with the correct role, label, and click action. VoiceOver sees and navigates these elements as if the app had built-in accessibility support.
The result
Apps that were blank to VoiceOver become partially navigable. Buttons can be clicked, text fields identified, links followed. It's not the same as native accessibility, but it makes inaccessible software usable.
Technical decisions
Design choices shaped by actually using VoiceOver every day.
Label association
YOLO often classifies nearby text labels as buttons. Rather than treating this as a model error, the label association algorithm identifies when a detected "button" is actually a label for an adjacent text field, using tight row-alignment tolerances to avoid picking up section headers.
Proximity fallback
When the ML model misses a label entirely, a fallback system uses OCR results and spatial proximity to find the nearest text that could plausibly describe a control. This catches cases the model was not trained on.
Text input handling
Text fields use NSTextView rather than NSTextField to avoid a VoiceOver bug where the first visit to a text field announces it as static text instead of an editable field. I found this because I use VoiceOver every day.
Why this matters
This project goes beyond identifying the barrier.
It turns an accessibility requirement into a working, testable prototype. My role is the accessibility direction, product judgment, implementation review, iteration, and VoiceOver validation.
Implementation uses
- Swift and Objective-C
- YOLO v8 object detection (trained on custom UI element dataset)
- Apple Vision framework for OCR
- NSAccessibility protocol for accessible element creation
- Core Graphics for screen capture and coordinate mapping
Want to see more of my work?
The portfolio includes web and mobile accessibility audits alongside these technical projects.