Controlling our electronic devices has always been something of a challenge. There is always some sort of mythical way to interact and control our devices that seems just out of reach. We started out with physical switches and buttons, then moved onto touch-sensitive interaction with trackpads and now touchscreens. Throughout it all, there has also been work done to create voice-interaction with our electronics, which has come a long way, but is still not perfect, and can't cover all of the needs we have. Now, we have a number of different companies trying to push forward with touch-free gesture controls.
At Mobile World Congress this past week, there was a company, Noalia, showing off another iteration of touchless gesture controls. This time, the controls were not only touchless, but didn't require the use of a camera either, as many options do. The Aramis technology shown off by Noalia detects the electrostatic energy around your hand, and translates that into on-screen gestures. This means that you can control a phone or tablet from a distance of about 10cm without touching the device.
Back in December, we saw an Israeli startup, XTR3D, that was showing off Kinect-like gesture-based controls for electronics and mobile devices, and every once in a while we've seen companies coming along and trying to make it seem as though the future is in touchless gesture control, but frankly we just can't see why.
The case against touchless control
We can certainly understand the idea behind Kinect, which is Microsoft's gesture control system first introduced as part of the XBox 360 gaming system, and has now officially made its way to PCs as well. The logic is understandable, and it can be seen at the beginning of the video showing off XTR3D's technology. On larger screens, like TVs or desktops, where there is no inherent touch-control system and the device itself is sometimes out of reach, a gesture-based system makes some sense, although it doesn't really add any benefits over traditional controls with remotes, mice or keyboards. The trouble is that companies seem to think it is a good idea to convert this technology to mobile devices as well. Of course, once you apply touchless gesture technology to a device designed for touch interaction, you suddenly run into a number of troubles.
First, what is the need for touchless interaction on a device that's designed to be in your hand at all times? The use case for touchless gestures on a mobile device are incredibly few and far between. Essentially, the only reasons you would ever have to use touchless gestures on your smartphone or tablets fall into just one category: touch-free means smudge-free. If your hands are dirty or if you really hate smudges, touch-free controls are a benefit, but beyond that, there is little reason to need touchless gesture controls. Touch-free screens would also make using touchscreen devices in the winter much easier when you're wearing gloves, but again, touchscreens have already evolved to the point where many can be used even through gloves (depending on thickness) without the need for special tips on the fingers. There is a possibility for using touchless gestures as a way to easily "fling" information from one screen to another, but that seems more like a gimmick used in "futuristic" technology concepts. The same thing can be done right now with either NFC, or wireless connections and traditional buttons. There is no benefit to moving that function to touchless control.
If you're in a car, gesture controls don't help, because you're still taking your hands off the wheel, so voice-control is still the far better option. Some may say that touchless gestures will be good for touchscreen gaming, because your thumbs will finally get out of the way and allow you to see more of the screen. But, most gamers will also tell you that gaming without direct feedback isn't the same experience. It may not change what happens in the game, but being able to hit the screen a bit harder for an important movement is just flat out more gratifying than waving your hand in the air.
Design for humans
And, that's the real issue behind this that no one bothers thinking about: hands are designed to touch and manipulate things. A wave doesn't covey the same thing as a handshake, just like flipping the bird doesn't convey the same thing as a slap in the face. We are designed to feel, touch, and interact physically, so why would we want to design products that go against that natural design? The logical extension of a touchpad is a touchscreen, but the logical extension of a touchscreen is not a touchless screen. Hands are designed to feel things and understand kinetic feedback, but touchless gestures feature none of that. A far better idea is one that we saw early in December last year, where a company called Senseg has designed a screen that can generate kinetic feedback in order to make it feel as though the screen has various different textures on it. This sort of technology has far more uses than any touch-free gesture system.
With kinetic feedback, you can create touchscreen devices that can be used by blind people. For more casual users, games could be made where users can feel different textures. Imagine pulling back an Angry Bird in its slingshot and being able to feel the feathers of the bird, and because the technology works by creating electrostatic friction, you may even feel the tension of the slingshot band as you pull back. Similarly, if you have an app of a zen sand garden, you could actually feel the sand and stones as you move things around. Doctors could use screens with kinetic feedback to train, or musicians could practice the piano on the go and actually feel the keys on their tablet. And, there is no telling what innovations (not to mention absolutely disgusting websites) that could come to the Internet if suddenly websites could have textures and kinetic feedback.
It all comes back to the fact that we design products to be extensions of ourselves, because essentially they are just tools for us to better interact with each other, the world and possibly most importantly to interact with our own ideas. Last year, Bret Victor wrote one of the best posts on the subject that we've ever seen, and it fits into what we're driving at here, although his post focuses mostly on the fact that we only use our fingertips to interact with devices rather than our whole hands, arms, etc. The idea is still the same though: we design products in part to fit a need, but also to maximize our own natural abilities. Right now, we have designed touchscreen devices to the point that we can very easily interact with computers in a traditional way, but it is time to push forward, and we don't see that next step of evolution being to take away the touch interaction.
Other options
More likely, the devices that are the next step in evolution for mobile are concepts like what we've seen from Nokia. Nokia has shown off two different concepts that we would love to see as a reality far more than any touch-free gesture system. The Nokia Gem was an idea for a smartphone that is 100% touch-capable, meaning we could use our entire hand for interactions. Even more impressive was the HumanForm concept which not only included full touch capability, but a flexible design for real physical interaction by bending the device. The teardrop shape may not be the best for viewing pictures or video, but obviously that's the least important thing here.
We've also seen flexible displays multiple times, and Samsung swears that they are coming "within a year", but this only gets us part of the way there. The first couple generations of flexible screens may not see much use beyond being unbreakable touchscreens for devices that are no different from what we already have. We still need flexible components like batteries in order to be able to really build fully flexible devices.
Either way, we need more ideas and we need to remember the core philosophy of design, which is to build things that make sense, not just things that look cool. A Kinect may make for some cool commercials, but so far it hasn't added much as far as useful interactions. Sure, it looks cool to swipe the air in order to navigate a menu, but it is in no way more efficient or faster than using a remote control. The same idea goes for touch-free gesture control. Sure, it looks cool and futuristic, but it is also completely pointless unless you are in a situation where touching the device is not an option at all. Voice control currently fits into the same category. It has come quite a long way, and can be extremely useful, but only if you're in a relatively quiet place and have an acceptable accent.
Looking one step beyond
The best options are still out there, but are being overshadowed by these technologies which have niche uses and make for cool demos, but really don't push forward devices all that much. The transition from physical buttons to touchscreens lead to a huge leap forward in functionality because of the options available with the new method of interaction. The next step for our mobile devices will create a similar leap in functionality, because it should make just as much design sense. Touchscreens allowed for more than just pressing buttons, they opened the door to gestures and multitouch interactions. Moving to touch-free doesn't add anything, it only takes away. What we really need is to add more senses like our sense of touch with kinetic feedback.
And that is where touch-free controls can really find a revolutionary application. Once we can combine touch-free controls with tactile/kinetic feedback, that's where we have something really out of the future. We never think about it, but all of the "futuristic" interfaces that we see in movies like Minority Report, Iron Man, or District 9, where people are interacting with holograms would be extremely difficult to use without some sort of tactile feedback. Without that feedback, you would have to be looking at what your hands are doing at all times in order to be sure you're doing what you intend.
Remember, we saw versions of laser keyboards which would project a standard QWERTY keyboard onto a tabletop and would detect when you hit a "key". The problem was that if you can't feel the keys under your fingers, you can't touch type, which drops your efficiency a huge amount. Smartphones have been able to make touch typing without tactile feedback work because you are essentially looking at the keyboard all the time anyway. If you are interacting with a holographic object and you can't feel it, it requires too much of your attention. However, when we can combine the technology shown off by Senseg and that shown off by Noalia to create touch-free gestures with tactile feedback, that's when we'll really be seeing the future technology we all want.
