For decades now, the mouse, keyboard, and monitor layout has been the primary interaction with a computer, and although it has worked well for us in the past, I think it's time we start seeking out alternative interaction schemes. People have experimented with replacing the ol' mouse and keyboard ranging from multi finger touch screens to a wave of your hand. Let's take a quick look at a few of these UI schemes and take some pros and cons.
I've had my eye on Con10uum for awhile. If you watch the video, they explain how they combine the ergonomic design of the mouse with the functionality of a multi-touch touch pad that's 1:1 to the size of your monitor and incorporate that into the OS. With the innovative approach of losing the 2nd dimension in window management and having the global/local context strips outside of the touch zone for the screen, it will take awhile to get used to, but I'm sure it will enable us to be more efficient with our time. This touchpad replaces the mouse, but not the keyboard, which the video displays as part of the touchpad.
This is Reac Table. It's been around for a few years and this video is strictly of it in use as an instrument. It works by having a camera watch the bottom side of a piece of glass, this glass being the surface that the tiles are on. This camera watches for the amoeba shaped images on the undersides of each piece or for a finger pressed on the surface. These amoeba shaped images are unique from each other and are designed so that they do not have any sort of rotational symmetry. Without the rotational symmetry, the camera can tell to what degree each tile is rotated. The camera is connected to a computer that is running the application and sending video output to a projector that is projecting back to the underside of the same glass surface that the camera is watching. For their musical implementation, you can see that different tiles relate to each other by proximity, such that if a tile lies between another tile and the center, it modifies the sound of the tile that's farther out. There is plenty of diversity among the tiles, ranging from reverb to a square wave, even a beat box (yes, bad pun, I know). In the last scenario in the video, if you keep up with what each individual tile is doing, you'll notice that only one tile is actually generating sound. All the other tiles are modulating or reverberating that sound. The idea of physical objects being tied to virtual objects opens up numerous possibilities for interactions, especially when you factor in proximity, rotation and whatever factors you let the touch screen modify.
Microsoft's Kinect. Although specifically released for the Xbox 360, the Kinect brings together new hardware to remove the interaction with a physical controller through direct manipulation... I.E. A camera watches your body and YOU are the controller. Ok, I admit at first it sounds gimmicky, but hear me out, there's potential. Let's break down the important hardware that makes this possible. First a camera, just your everyday RBG color camera. It takes color pictures, nothing too special by itself. There's an Infrared (IR) projector or emitter that works along with an IR camera. This camera is for depth. Being IR, it won't pick up any colors or complications, the closer an object is, the more IR it bounces back to the camera, the more intense the IR returned to the camera, the closer it must have been and you get a nice depth image. This depth image then goes through hand tracking and skeleton tracking and whatever other tracking it can think to do. This tracking is all done in software, the device does not hand you pretty little 3D figures of all the players... if only it did. What if's aside, thanks to OpenKinect you can write software on many different platforms that use the Kinect device. It's just a USB device (that needs external power) and they have all the drivers. I'm holding out that someday they'll include the Kinect and all the body tracking as part of the XNA, but who knows if that will happen anytime soon. Either way, the Kinect offers a great user experience in that all the user has to do is stand in front of it and wave around. Of course there's all sorts of exercise applications that can be made, but imagine if Minority Report became reality.
I think it's time we re-evaluate how we interact with computers. Not simply altering how we manage windows, but how do we interact with files, streams, and other virtual objects in addition to windows (if we even keep windows around). As the Con10uum video describes, we're reduced to the bandwidth of a simple X,Y location and two different variations of clicking. We could open this bandwidth of input into our ten fingers, or even our entire body.
Development
con10uum, kinect, reactable, reac, table, ui, user, input, gui, mouse, keyboard