Accessibility engineering project
ScreenRecognition: making inaccessible macOS apps usable with VoiceOver.
Most accessibility work starts with an app that almost works and needs fixes. This project starts with apps that have no accessibility support at all and builds it from scratch using machine learning and macOS Accessibility APIs.
What this project shows
When an app has zero accessibility, sometimes the answer is to build the accessibility layer yourself.
ScreenRecognition is a macOS application I designed and built to solve a problem I face daily: professional software that is completely invisible to VoiceOver. Instead of waiting for vendors to fix their apps, I built a tool that detects UI elements on screen using machine learning, reads their text with OCR, and creates a transparent overlay of interactive accessible controls that VoiceOver can navigate.
Skills demonstrated
- macOS accessibility API implementation (NSAccessibility protocol)
- Machine learning model training and integration (YOLO object detection)
- Accessible UI architecture from first principles
- OCR-based label extraction and intelligent label association
- Real-world problem solving driven by lived screen reader experience
How it works
Three layers working together to turn a visual-only interface into something VoiceOver can navigate.
Detection
A YOLO object detection model trained to recognize five common UI control types: buttons, disclosure triangles, images, links, and text areas. The model runs against a screen capture of the target application.
Recognition
Apple's Vision framework performs OCR on detected elements and their surrounding context to extract labels, placeholder text, and nearby descriptive content. A label association algorithm matches text to the correct control using spatial proximity and layout heuristics.
Overlay
A transparent window sits over the target application. Each detected element becomes an interactive NSAccessibilityElement with the correct role, label, and click action. VoiceOver sees and navigates these elements as if the app had built-in accessibility support.
The result
Applications that were completely invisible to VoiceOver become navigable. Buttons can be clicked. Text fields can be identified. Links can be followed. It is not perfect, and it is not the same as native accessibility support, but it turns a completely blocked experience into a usable one.
Technical decisions
Design choices shaped by actually using VoiceOver every day.
Label association
YOLO often classifies nearby text labels as buttons. Rather than treating this as a model error, the label association algorithm identifies when a detected "button" is actually a label for an adjacent text field, using tight row-alignment tolerances to avoid picking up section headers.
Proximity fallback
When the ML model misses a label entirely, a fallback system uses OCR results and spatial proximity to find the nearest text that could plausibly describe a control. This catches cases the model was not trained on.
Text input handling
Text fields use NSTextView rather than NSTextField to avoid a VoiceOver bug where the first visit to a text field announces it as static text instead of an editable field. This is the kind of detail you only catch by being a daily screen reader user.
Why this matters
This is what accessibility engineering looks like when the problem is bigger than missing labels.
Most accessibility work involves auditing existing apps and recommending fixes. ScreenRecognition demonstrates a different capability: designing and building an entire accessibility system from scratch when the application itself provides nothing to work with. It combines ML, platform APIs, and deep assistive technology knowledge into a working tool I use every day.
Built with
- Swift and Objective-C
- YOLO v8 object detection (trained on custom UI element dataset)
- Apple Vision framework for OCR
- NSAccessibility protocol for accessible element creation
- Core Graphics for screen capture and coordinate mapping
Want to see more of my work?
The portfolio includes web and mobile accessibility audits alongside these engineering projects.