Face Tracking and Eye Gaze for the iPhone and iPad: Access and AAC

By: Kathryn Helland For the AAC Community

At long last, apps are available that take advantage of Apple’s True Depth camera to provide Head Tracking on the latest iPhones or iPad Pro.  These devices actually have two cameras; your usual selfie camera and an infrared camera. An infrared emitter projects 30,000 points of light.  These are ‘photographed’ to allow your device to recognize the topography of your individual face. This allows your device to unlock without a passcode.

Head tracking for Apple devices has been a long wished for accessibility feature. An eye gaze bar that hooks up to a computer costs around $2000.  A dedicated communication device with eye gaze can cost upwards of $10,000. Access to face tracking on the iPad is a big deal!

News of this new technology came out at Apple’s WWDC in June of 2018. Apple’s ARKit 2.0 introduced an eye tracking feature.  Folks quickly realized that this tool was not just for advertisers, but could really benefit people with disabilities, such as ALS and other motor neuron diseases. It took a few months for the first apps to follow.

I will mention some of the apps that are making use of this feature. The list will hopefully continuing to grow.

Hawkeye Access allows you to browse the internet using your eyes. When the app is opened, it takes the user through a nine-point calibration process.  There is a brief tutorial that teaches you how to navigate: i.e., scroll, go back, or return to the home page.

This app does rely on the user being able to speak.  Dictation is used to perform a search or provide input to text fields (such as your Twitter login). This poses a challenge for people who need AAC, or whose speech is not clear.

You can use Hey Siri to open the app, so the user is not dependent on outside help. The app is customizable, so you can adjust the sensitivity, pick your selection mode, and configure your timing. Pretty cool.

There are now some options out there for those who need AAC.  One app is called I Have Voice. It is is set up for the basic communication of physical needs and does not provide access to generative language.

The app is aimed at adults with ALS. You can use your eyes to select between two pages of “Actions” (messages). It is possible to edit and create new actions.  There is a limited set of icons that you can choose to go with your written text.

You can choose between using dwell or a blink to activate the icons. There is no calibration procedure, so it takes a while to get the feel for how to make small motions to control the cursor on the screen. It is possible to change the sensitivity of  the cursor, your blink, or your dwell time.

Some other apps that offer similar functionality are Pigio and SpeakProse.

For those who need text to speech, another app is called Jabberwocky. This app uses face tracking to access a keyboard with predictive text. Calibration is as simple as looking at one point on the screen.

The iPhone camera translates your head motion into movement of the cursor. The app allows you to speed up your input by using “Draw” mode.  The cool thing with Type mode is that you see how your head movements track across the screen.

The app saves your recent, favorite, and frequent utterances in your history. There is a pre-set ”I use this computer to help me talk” sentence to allow the user to ask for more time. While the app used to cost $250, it is now FREE!  That is a the kind of app update I like.

Jabberwocky also offers a web browser. You can use head tracking and speech to navigate the internet.  This will also work for those who use AAC with a bit of tweaking.  You would need to browse the web on a different device than you use for speech. You would also want to set up a folder of frequently used phrases and sentences for web browsing on your AAC device.

Predictable AAC has also added head tracking.  This is a text to speech app that has symbol-supported folders for topical communication. There are some updates coming soon.  You will soon be able to see your “Recents” and other frequently used messages at the top of the screen.

Another option is PRC Saltillo’s TouchChat HD with WordPower. Head tracking works with the app on both the iPad Pro (2018) and the iPhone X or beyond.  I have tried it out and it works great!  You simply turn on head tracking within the settings in the TouchChat app. As long as you have the True Depth camera on your device, it is immediately available without any additional in-app costs. How awesome is that?

TouchChat also allows you to set your cursor shape, tracking speed, and off-screen indicator.  A great feature is that you can choose between 7 different Trigger Actions, such as raised eyebrows, smile, or tongue out. This allows you to further customize head tracking based on the individual’s abilities.

Sesame Enable: Sesame had been offering head tracking on Android-based devices and Windows computers. Their business model became difficult to sustain and the company closed in the fall of 2019.  Instead of ending access to their software, they have made it free! On the iPad, you can download the Sesame Talking Keyboard. It works with the True Depth camera to allow head tracking for typing. It does not offer the ability to save frequently used messages, but it is great for on the spot, hands free communication.

Eye Gaze for the iPad: A new device is now being offered by Inclusive TLC that pairs the iPad with an eye gaze bar.  It is called the Skyle and runs for around $3000.  This is much cheaper than other dedicated communication devices with eye gaze. Skyle offers a choice of AAC apps, access to social media, email, and environmental control (using the Environ app to control Pretorian smart home devices). The protective case comes with a built in mounting plate for wheelchair users. It works with the Skyle eye tracker utility app.

The above technology has the potential to truly make a difference for people with complex bodies.  By making alternate access available on consumer electronics, it brings the price point way down. My hope continues to be that Apple will add head tracking to its accessibility settings in the future. Let’s give more people the power to access the entire device!