Tech firms are racing to come up with new “gesture-swipe” keyboards that let you wave your hands and around in the air to control your keyboard, type messages and manipulate data without sitting down and typing in one letter at a time. Based in part on sensor technology built for the Microsoft Kinect games, researchers say these ideas will help doctors use a computer while doing surgery, for example, or just drive a video game race car with an imaginary steering wheel.
Microsoft Research has already come up with a prototype keyboard “sensing rich and expressive motion gestures performed both on and directly above the device,” according to a research paper presented at the Computer Human Interaction 2014 conference this week in Toronto.
It lets you pinch and swipe just like a smartphone screen a few inches above the keyboard, using built-in infrared sensors to detect movement.
Last year, rival Apple patented a new iPhone user interface that will allow you to swipe individual keys instead of typing in letters.
But others want to go further. Danish researchers have come up with a software called Vulture that allows the user to actually “draw” words on a virtual keyboard floating in front of them, rather than resting on a desk.
“You can see in front of you what you are actually doing,” said Anders Markussen, a doctoral student at the University of Copehnagen, who also is presenting his work at CHI2014. "If you want to write the word test, you move your hand to the location where the 'T' is, then draw the shape that crosses the remaining letters 'E'-'S'-'T', then release your pinch. The shape would be recognized as the word test."
Markussen said volunteers in his experiments using Vulture got close to 30 words per minute, which is slower than typing but pretty good for mid-air gesturing. See the video below.
“We need to explore more what the advantages are and how fast it can get,” Markussen said.
Another team has developed a system to hold up fingers for counting numbers. It’s something that seems logical, but hasn’t been done before, according ot Joseph LaViola, associate professor of electrical engineering and computer science at the University of South Florida.
“It’s very fast and you don’t have to move a cursor around,” said LaViola, who presented his work at CHI2014.
The idea is to use your fingers as numbers for various menu options for gaming, educational presentations in museums, or even shopping.
“It’s more natural,” LaViola said.
LaViola says that the advances in software are being pushed by more accurate and smaller cameras built by Intel and others that better capture gestures made by hands or body parts.
“These (3-D imaging camera) devices are starting to come out everywhere,” LaViola said. “By the end of the year they will be embedded in laptops and tablets. You will easily be able to track someone’s fingers in 3-D space and it easily lends itself to these applications.”