Interface with Kinect

Microsoft Kinect is an intereseting interface, taking the motion and gesture capbility found first in the Wii to the extreme. What is this extreme? No buttons, no hand controls. This frees up the play from the interface and the computer system entirely.

Charades on steroids? Some people have found it a bit intimidating to get up in front of their friends / family / date / cat and strut their stuff, but some more practical problems have come up for developers – How do you make a menu system with no buttons?

Ars Technica summarizes a talk given at the Montreal International Game Summit by the developers at Harmonix for their game Dance Central.

Their design process resulted in two key take-aways:

Everyone has a different interpretation of a given gesture, and that it is easier to teach people to use an interface that to bulletproof the interface for all possible users.

No Air-Click.

The obvious first thing to try with an interface is a mapping of what everyone already knows: pushing or clicking. Unfortunately in a gesture interface there is too much ambiguity in what a ‘click’ is, since depth and ranging information is still the least accurate axis in which a camera-based system can track.

A way to measure if a control scheme is going to result in confusion is called ‘coupling’. Here’s an example of a control scheme that is 100% coupled and hence has total ambiguity: Blink once for yes, once for no. Useless? Yes. How about: blink once for yes, twice for no? Well we have a version of that: double clicking, and there is still coupling – the two controls are coupled in that they both look the same at first (a single click) and a problem like that is even worse in gesture control where the of clicking isn’t there, and snapping fingers is a non-starter. So what else have we got?

Solution? More Decoupling.

One thing a Kinect is very good at is detecting motion orthogonal to the camera system; it knows left-right and up-down very well. Starting with this, the menu system can be arranged so that left-right is a toggle function, and up down is item selection. The two actions are decoupled as much as is possible in gesture land, which still allows fine positional selection.

In fact, this is not new news. Look at gesture-based communication – it is designed to be clear and non ambiguous. Semaphores, sign language, even radio communication. Decoupling reduces ambiguity.

In the end, new interfaces take new interface languages. All of these were a bit revolutionary on their introduction, even the keyboard:

           Mouse     Touchscreen     Gesture    Keyboard
Rollover     X                          X
Click        X            X                        X
Swipe        X            X             X
Depth                                   X

Now there is some ambiguity in interfaces themselves – what is the difference between an interface and an interface gesture? You could argue that a keyboard is a gesture based interface with a click confirmation, and that the rollover events are ‘tactile feedback’. But that is the point, really – there are no clear delineations, just strengths to be built on and ambiguities to be avoided. The potential of a full-body, controller-free interface is amazing, but you would not use it to implement an ‘air keyboard’ or even an ‘air touchscreen’.

Feedback Loops

Harmonix’ key lesson was that people can learn to use a good interface quickly as long as there is minimal ambiguity. The designer’s job is to provide an interface which may be new, novel, and even alien, but that provides clean feedback to the user, allowing the user to learn quickly. People love to learn something new, as long as it is not confusing, and ‘hacking’ the user is as important as programming the hardware.

And, if you are a developer, there is always the Open Source drivers for the Kinect device which are becoming available.

(image from wikipedia)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s