Apple adds AI smarts to Voice Control, VoiceOver and Magnifier ahead of Accessibility Day

Apple has previewed a new batch of accessibility features coming later this year, with Apple Intelligence being used to improve Voice Control, VoiceOver, Magnifier and generated subtitles across its devices.

The announcement came ahead of Global Accessibility Awareness Day, which falls today as we publish this article, on Thursday, May 21, and is the annual moment when technology companies often set out new work on digital access and inclusion.

The most interesting change for anyone who relies on hands-free access is an update to Voice Control. Apple says users will be able to describe onscreen controls in more natural language, rather than having to remember exact labels, overlays, or rigid commands.

Examples given by Apple include phrases such as “tap the guide about best restaurants” or “tap the purple folder.” The company also says the feature could help when app controls are not labelled properly for accessibility.

That may sound like a small change, but for disabled people who use voice as their main way of operating an iPhone or iPad, it could make a real difference. Voice Control is already one of Apple’s most important accessibility tools, but it can still be brittle. If the wording does not match what the system expects, the command can fail.

A more flexible “say what you see” approach could make voice navigation feel less like issuing machine instructions and more like asking for what you want.

Apple says Voice Control powered by Apple Intelligence will be available in English in the UK, US, Canada and Australia later this year. However, Apple’s announcement specifically describes the new natural language navigation as helping people navigate iPhone and iPad by voice, with no clear mention of Mac support for this particular Voice Control update.

That absence is important. For many people who rely on Voice Control, the Mac is not a secondary device. It is where longer writing, work, email and publishing happen. If natural language Voice Control launches first on iPhone and iPad only, Mac users may still be left waiting for the AI-assisted voice access that would help most with daily work.

VoiceOver, Magnifier and generated subtitles get Apple Intelligence treatment

Apple is also using Apple Intelligence to improve visual description tools.

VoiceOver’s Image Explorer will provide more detailed descriptions of images, including photos, scanned documents and other visual content. Apple also says users will be able to ask follow-up questions about what appears in the iPhone camera viewfinder.

Magnifier will gain similar AI-powered description features, along with spoken controls such as “zoom in” and “turn on flashlight.”

There is a new generated subtitles feature for videos that do not already include captions. Apple says this will use on-device speech recognition and work across iPhone, iPad, Mac, Apple TV, and Apple Vision Pro.

For deaf and hard-of-hearing people, that could be useful. It may also help anyone dealing with personal videos, shared clips or online content where captions are missing. However, generated subtitles will initially be limited to English in the US and Canada.

Vision Pro moves into wheelchair control

One of the more striking announcements is a new Apple Vision Pro feature that will allow compatible power wheelchair drive systems to be controlled with eye tracking.

Apple says the feature will support Tolt and LUCI alternative drive systems in the United States, using Bluetooth or a wired connection.

For some powered wheelchair users who cannot operate a joystick, that could be valuable. Wheelchair control is not a niche issue for the people affected by it. It is about independence, safety and the basic ability to move through the world.

But there are obvious practical questions here, starting with Vision Pro itself.

As a full-time electric wheelchair user, I would not be seen dead driving down my high street wearing an Apple Vision Pro headset. It is bulky, heavy and visually conspicuous. More seriously, I would not want to see severely disabled people expected to wear one for long periods to control a wheelchair, especially when many already deal with fatigue, posture problems, respiratory weakness or limited head and neck strength.

Cost is another barrier. Many disabled people live in poverty, and the Apple Vision Pro’s UK starting price of £3,499 (the Stateside starting price is slightly lower at $3,499 ) would put it out of reach for many. That would come on top of the cost of any compatible wheelchair drive system, support, setup, and maintenance. A feature can be technically impressive and still remain impractical if the hardware required is far too expensive.

That does not make the announcement unimportant. It may be most interesting as a sign of where the technology could go next. I would look at this very differently if the same kind of eye-control system eventually arrived on more traditional Apple smart glasses: lightweight, socially acceptable and practical to wear for long periods.

That is where the idea could become more useful for people who struggle to use a wheelchair joystick. Vision Pro may be the early test bed, but lightweight glasses could be the form factor that makes this kind of wheelchair control usable.

For now, this looks like an early and specialist step. Wheelchair control is safety-critical, so it will need careful testing, strong safeguards and real-world feedback from disabled people before anyone can judge its value properly.

I am glad Apple is looking at the issue. The current implementation may not be practical for many people, but the underlying idea deserves attention.

Apple is moving in the direction some of us asked for

Calls for a smarter Voice Control are not new. In 2023, I wrote for The Register that Apple needed to bring more AI into Voice Control, especially to improve dictation accuracy and support people with non-standard speech.

At the time, I argued that Personal Voice showed Apple already had some of the underlying technology to understand an individual voice more deeply. The obvious question was whether that intelligence could be applied to recognition as well as voice generation.

Apple now appears to be taking a step in that direction, but with navigation rather than dictation.

That is still useful. Voice Control needs to become less rigid if it is to serve people who depend on it every day. But it leaves a larger issue unresolved.

Apple still has a dictation gap to close

The wider voice-accessibility picture is now complicated.

Apps such as Aqua Voice have shown how good AI-powered dictation can be. For many people, these newer tools are far more accurate and natural than traditional built-in dictation systems. They are especially strong at turning spoken thoughts into clean text without the user having to micromanage every comma and correction.

But dictation is only half the problem

Apple’s Voice Control is still one of the few mainstream tools that can control the operating system itself by voice. It can open apps, tap buttons, select menus, scroll pages and move around the interface. Third-party AI dictation apps may be better at writing, but they do not have the same deep system access.

That leaves disabled people in an odd place. The best dictation experience may come from one app, while the best hands-free control still comes from the operating system.

For people who cannot easily touch a screen, keyboard or mouse, the ideal future is not choosing between accurate dictation and reliable control. It is having both work together.

That is why this Voice Control update is worth watching. It suggests Apple is starting to apply newer AI methods to one of its most important accessibility tools. But the next step should be more ambitious: a system-level way for advanced dictation and accessibility controls to work together.

Whether Apple builds this itself or opens up deeper accessibility APIs for trusted apps, the goal should be the same. Users should be able to dictate accurately, correct text, move around apps, press buttons, send messages and control the operating system without switching between separate voice tools.

Call it Universal Accessibility Control, or simply the next generation of Voice Control. The name matters less than the result: one joined-up voice experience that combines accurate dictation, command recognition and hands-free navigation.

For now, Apple appears to be improving navigation before it tackles the harder dictation problem.

Reliability will decide it

Apple’s announcement also includes larger text support on tvOS, expanded Name Recognition, new FaceTime APIs for sign language interpreter apps, Vehicle Motion Cues for Vision Pro, and wider support for adaptive gaming controllers.

But the Voice Control update is likely to attract the most attention from people who rely on hands-free access.

Apple has not announced a major Siri accessibility overhaul here. Nor has it announced major changes to Personal Voice, Vocal Shortcuts or atypical speech recognition in this particular update. The company also has not said whether it plans to make Apple Watch more accessible to disabled people with severe upper limb disabilities.

Natural language Voice Control could be valuable if it works reliably. For disabled people, accessibility features are not just nice additions. They are often the difference between using a device independently and not using it at all.

The announcement is encouraging, coinciding with Global Accessibility Awareness Day. But Apple should not stop at making Voice Control more conversational. The larger task is to treat dictation, correction and navigation as parts of the same workflow.

The test comes later this year, when disabled people can try these features in daily life. The longer term question is whether Apple can turn this first AI step into a fuller model of hands-free computing. ®