Touchscreen Next?

[Antoine Wright]

Since the introduction of the iPhone, the mainstream mobile user has had a new revelation about the power and abilities of touchscreens. I encounter people who are regularly in awe about what any device with a touchscreen can do.

Granted, part of this is because of the terrible user interfaces that mobile device users have been subjected to over the years. In many ways, one can say that touchscreens are a revelation mainly because we've been forced to learn something new when we already had digits to point and select with that we already knew how to use.

That being said, touchscreens as we know them are still probably not truly paradigm changing until they do more. They need to get out and touch our worlds with enhanced abilities beyond just point and show.

With the iPhone, Instinct, and several other tablet-style mobiles here and coming soon, what could be next in terms of touchscreens and interactive mobile displays? Some things are just a matter of changing the design a bit, but others are going to be a bit of un-learning on our parts. And it might open up some incredible possibilities.

Gestures

Gestures, or the act of using some flowing finger or hand motion to initiate an action, is something that we've seen since the early years of computing. With devices like the iPhone and Instinct however, gestures have become  an artful way to add usability to mobile devices without adding additional buttons.

Gestures can take either of the following forms:

• physically moving the device from a stationary point in a uniform motion
• physically moving a finger or fingers across the screen in a uniform motion

The recognition of that motion by the software and hardware delivers an effect such as flipping a page or scrolling a website.

What is starting to happen with gestures is that developers are seeing the fun that people are having with the Nintendo Wii, and integrating some of that functionality into mobile games and other applications.

For example, a few racing games in development for the iPhone and Nokia S60 devices allow the use of the mobile device as a steering wheel. The screen shows where the user is going, and either pressing a button or tilting the device causes the virtual car move forward or stop.

Another example is the navigation software on the T-Mobile G1. This will allow you to view a panoramic image of a remote location, and then scroll around on the image by holding the device up and moving it around.   

Software such as NokMote and RockNScroll by KEYnetik are some other early options in this area.

Projection with Gestures

Taking the idea of gestures a bit further, I can see the idea of using projectors built into mobile devices working along with gestures to create new mobile interfaces.

Currently, projection technology is being developed that will fit a projector into a mobile device that would allow it to use any a large, flat surface as a display. Taking that and adding the idea of gestures -- with either moving the device itself or the user moving their hands in front of the mobile device -- can create other ways of interacting with content. We already have a basic semblance of this in the wireless laser keyboard.

An idea that I see here is being able to take a picture of an unfamiliar area, and then project the image onto the ground. Then, using the built-in GPS of the device along with gesturing, a person would be able to not only find out where they are, but also search the image's context for online references for points of interest -- parks, ATMs, restaurants, etc. The idea being to take the physical environment and meld it with a virtual one with the mobile device as the "controller."

We'll Have to Wait and See

What is to come is not really known at this point. New mobile interfaces come onto the scene every 18 to 24 months, and it usually takes something major for them to become paradigm shifting. We saw this with the QWERTY keyboards RIM and Palm made famous, and then with the capacitive and multi-touch screens that Apple has brought onto the scene in the past year. 

What ever new mobile-friendly input mechanism will have to answer to the challenge of little space and high efficiency. No matter the input element, if a person cannot find information quickly and easily, the technology will quickly fail the test of what works best for the next generations of mobile device users.

 

[berylrb]

Interesting thouhts AW ... I was surprised you didn't mention much about the actual inputting of data.

much has been said about Apple's virtual keyboard, what would the future hold for this functionality?

I personally was very skeptical moving from keys to Graffiti, and then from Graffiti to iPhone virtual, and each step I was amazed at how the software smoothed out or handled user error as in missed keystrokes. I recall my youthful arrogance/fear, for the lack of a better phrase saying, "There is no way I'm giving up my keypad for Graffiti!" Then after being wooed by Palm, "There's no way I'm giving up Graffiti/stylus for virtual!" I wonder what the next iteration of "No way!" will be

I'm pretty much sold now, bring on the new! The key to change is to let go of fear!

[questionfear]

For me, it's the lack of response that stops me being comfortable with virtual entry methods such as the iPhone's keyboard. When I input with Graffiti, I can feel that I am forming the letters correctly, even without looking at what I'm doing. Without a haptic feedback system on devices - to let you 'feel' a virtual keypress or 'touch' a virtual mid-air menu - users will find it hard to adapt.

It's not awful to type on an iPhone once you are used to it. For short bursts it is fine, and as long as you are looking directly at the screen it is no problem. Typing one handed is very tough, and you can guess but not accurately at what you hit if you aren't looking at the screen. I am about 50/50 on accuracy if I am not looking.

Personally, I miss buttons. A lot. And while I like the touchscreen and I like how Apple has implemented it, I don't feel like it's the end-game for mobile devices, or that every device now has to have a touchscreen. Touchscreens are great but they are best if you're sitting or standing in one place and using your device. Typing and walking, or even dialing and walking, is tougher because you have to look directly at your device to do it.

[berylrb]

good tips QF, but I actually find it easy to type one-handed in portrait, landscape I don't even try or do you mean holding the phone with one hand and typing with the other being using two-hands?

I hear what you're saying Nine re tactile ... oh ... haptic feedback, but you'd think if Wii could do it so could the cellmakers, eh?

[questionfear]

good tips QF, but I actually find it easy to type one-handed in portrait, landscape I don't even try or do you mean holding the phone with one hand and typing with the other being using two-hands?

I hear what you're saying Nine re tactile ... oh ... haptic feedback, but you'd think if Wii could do it so could the cellmakers, eh?

I mean typing and holding the phone with one hand, while the other hand holds a bag, scratches your nose, holds your keys, etc. It is tougher (not impossible: just tougher) IMHO on an iPhone than on a tactile keyboard, mostly because you have to coordinate holding it a certain way to hit each key, and (this might just be me) I've occasionally triggered letters by brushing across the phone with my thumb or the palm of my hand when I try to do it one-handed.

As for haptics, the Wii does do a great job with feedback, but they have the advantage of just timing noises and vibrations right; your tv screen does not actually vibrate (how awesome would that be if it did???) I think the bigger issue is building the vibration feedback into a touchscreen that feels natural and not forced from the sheer amount of electronics shoved underneath the LCD.

Then again, the Instinct and the the upcoming Thunder (Storm?) both have it, so clearly the tech behind it, like everything else, is humming along at an alarmingly fast speed...

[Antoine Wright]

Interesting thouhts AW ... I was surprised you didn't mention much about the actual inputting of data.

much has been said about Apple's virtual keyboard, what would the future hold for this functionality...

...I'm pretty much sold now, bring on the new! The key to change is to let go of fear!

There will be a point where I'll never mention inputting data, because the data you want will already exist and your mobile can respond to it without you needing to input anything

That viewpoint I hold back... for now

There's nothing new about Apple's keyboard. As I highlighted in the article, its just predictive text. Its different because of perception, not because of function.

That being said, haptics and gestures, gestures and haptics. Just because you can touch the screen doesn't mean you are interacting with the data. Give it a few months and this will make more sense.

As for letting go of fear, I wish more folks had that view, but if they did then I'd not have a job

Personally, I miss buttons. A lot. And while I like the touchscreen and I like how Apple has implemented it, I don't feel like it's the end-game for mobile devices, or that every device now has to have a touchscreen. Touchscreens are great but they are best if you're sitting or standing in one place and using your device. Typing and walking, or even dialing and walking, is tougher because you have to look directly at your device to do it.

QF, this is only because of the implementation, not necessarly the technology. Nokia demoed a type of haptic feedback that allowed muscle memory to be used.

Take that, add buttons that can morph to the needed context, then no matter the application you are in, the touchscreen works, without you looking at it.

Give it a few months, this should either be on the market or a definite concept to release. That much I am very certain of.

[berylrb]

I mean typing and holding the phone with one hand, while the other hand holds a bag, scratches your nose, holds your keys, etc. It is tougher (not impossible: just tougher) IMHO on an iPhone than on a tactile keyboard, mostly because you have to coordinate holding it a certain way to hit each key, and (this might just be me) I've occasionally triggered letters by brushing across the phone with my thumb or the palm of my hand when I try to do it one-handed.

As for haptics, the Wii does do a great job with feedback, but they have the advantage of just timing noises and vibrations right; your tv screen does not actually vibrate (how awesome would that be if it did???) I think the bigger issue is building the vibration feedback into a touchscreen that feels natural and not forced from the sheer amount of electronics shoved underneath the LCD.

Then again, the Instinct and the the upcoming Thunder (Storm?) both have it, so clearly the tech behind it, like everything else, is humming along at an alarmingly fast speed...

right on, I was trying to say that, but I guess I'm having a bad "inputting" day!

I see a lot of BB users who use two thumbs to input, whereas on the iPhone i use one hand and hold it with the other, much faster than my grafitti days.

There will be a point where I'll never mention inputting data, because the data you want will already exist and your mobile can respond to it without you needing to input anything

That viewpoint I hold back... for now

There's nothing new about Apple's keyboard. As I highlighted in the article, its just predictive text. Its different because of perception, not because of function.

That being said, haptics and gestures, gestures and haptics. Just because you can touch the screen doesn't mean you are interacting with the data. Give it a few months and this will make more sense.

As for letting go of fear, I wish more folks had that view, but if they did then I'd not have a job
... snipped cool stuff on the horizon ...

come to think of it Palm had a similar virtual keyboard

Yeah, you are right we are inputting not interacting, but how about when the icons dance and one can move them around, that's interaction eh?

[Antoine Wright]

Well, it's slightly different. I believe if you are inputting a word on the iPhone keyboard, the software will scan available words and letter combinations based on the start of the word you've typed and automatically make the area to be sensed slightly larger around relevant letters only - making it less likely that you'll miskey.

Its predictive text. The core tech isn't different, just how you perceive how its being done

[Antoine Wright]

The nature of what it is doing, yes - it is just predictive text. But it's predictive text plus - making the method of inputting the text being predicted less likely to fail by minimising the scope for error via maximising available buttons.

You are describing user experience.
This article is about user interface.
The former is a super-set of the latter.

[Antoine Wright]

Here are some ideas from a recent concept show in Japan:
http://www.akihabaranews.com/en/news...UJITSU_001.jpg

http://www.akihabaranews.com/en/news...UJITSU_004.jpg

Taking the screen past something that's just view/input, to something that interfaces with your contextual environment.