Pressure-sensing smartphones: Software lets mobile devices feel force

What if you could dial 911 by squeezing your smartphone in a certain pattern in your palm? A different pattern might turn the music on, or flip a page on the screen.

New software developed by CSE engineers and inspired, in part, by a Batman movie, could give any smartphone the capacity to sense force or pressure on its screen or body. “ForcePhone” offers new ways for people to command their mobile devices.

The software could also enable users to push a bit harder on a screen button to unlock a menu of additional options, similar to right-clicking with a mouse. The developers envision these and many other uses for their technology, which could offer the masses a coveted feature of the latest generation of smartphones.

The iPhone 6s has a force-sensing screen but the less expensive iPhone SE doesn’t. No commercially available device has a pressure-sensitive body.

“You don’t need a special screen or built-in sensors to do this. Now this functionality can be realized on any phone,” said Kang G. Shin, the Kevin and Nancy O’Connor Professor of Computer Science. “We’ve augmented the user interface without requiring any special built-in sensors. ForcePhone increases the vocabulary between the phone and the user.”

Shin created the system with Yu-Chih Tung, a doctoral student in CSE.

“I think we’re offering a natural interface,” Tung said, “like how you turn a knob. It’s the next step forward from a basic touch interface and it can complement other gestured communication channels and voice.”

ForcePhone works by borrowing two of a phone’s fundamental attributes – its microphone and speaker. The software sets the speaker to emit an inaudible tone at a frequency higher than 18 kHz, which is outside the range of human hearing. But the phone’s mic can still pick up the vibration caused by the sound.

When a user presses on the screen or squeezes the phone’s body, that force changes the tone. The phone’s mic can detect that, and the software translates any tone tweaks into commands.

“Having expensive and bulky sensors installed into smartphones can solve every problem we have solved, but the added cost and laborious installation prevent phone manufacturers from doing it,” Tung said. “Our sound-based solution can fill this gap, providing the functionality without making any hardware modification. Everything is just software.”

The idea of harnessing the phone’s microphone and speaker for other purposes is an approach Tung initially picked up from a Batman movie. In The Dark Knight, from 2008, Batman turns all the smartphones in Gotham City into a sonar system. High-frequency audio signals bounce off the city’s infrastructure. He uses them to track the Joker.

“I thought it was a interesting idea to turn smartphones into a sonar-based system and felt this could lead to new applications to address challenges faced by smartphone users,” Tung said.

ForcePhone is the third piece of software based on this idea. Previously, Tung and Shin created BumpAlert, an Android application designed to warn distracted pedestrians of objects in their path, and EchoTag, which let phones tag and remember specific indoor locations and associate those places with certain modes or tasks.

BumpAlert uses the phone’s speaker and microphone to create an acoustic detector; its accelerometer to estimate motion and speed; and its rear camera to identify dangerous objects in the path. An algorithm combines the data and warns walkers by vibrating. EchoTag works by sending out inaudible tones and picking up how they’re reflected. It can do things like turn on an alarm clock or enable silent mode when you set a phone on a nightstand at the end of the day, for example.

While BumpAlert and EchoTag used sonar to create new applications for smartphones, ForcePhone builds more fundamentally on core interactions a user can have with a phone.

Shin is a leader in developing applications for mobile devices. Software he developed called E-MiLi, which stands for Energy-Minimizing Idle Listening, could extend smartphone battery life by as much as 54 percent. Another system called GapSense helps reduce dropped calls and wireless device interference.

The researchers will present their newest software, ForcePhone on June 27-29 in Singapore at MobiSys 2016, the Association for Computing Machinery’s international mobile systems and applications conference.