How Microsoft Kinect Works


In June 2010, two players demonstrate how the Kinect Sports game uses the controller-free motion-detection in Microsoft Kinect. See more video game system pictures.
Michal Czerwonka/Getty Images

Microsoft Kinect is poised to shake up the video game console experience. Announced and demonstrated as Project Natal in June 2009, Kinect seems almost magical the way it can "see" every movement of your body and reproduce it within the video game you're playing. Plus, it recognizes your face and voice so it can pick you out in the room and know who you are, even if you're playing with a group of friends. November 2010 marks Kinect's historic and anticipated release as a new addition to Microsoft's Xbox 360 product line.

As it turns out, Kinect isn't magical at all. It's a highly innovative combination of cameras, microphones and software that turns your body into the video game controller. The name Kinect is inspired by the words "kinetic," which means to be in motion, and "connect," which means it "connects you to the friends and entertainment you love" [source: Rule]. It's not just the games that get you moving, either: Kinect turns your Xbox 360 into a voice-activated console with video capturing and facial recognition, applicable for everything from selecting a TV show to creating digital artwork.

Microsoft has also designed Kinect as an enhancement to the Xbox Live experience. Xbox Live Video Chat makes use of Kinect's cameras and microphones for a webcam-like live chat with multiple friends at once [source: Stevens]. Plus, Microsoft teamed with ESPN to create a live interactive sports experience for Xbox Live Gold users. Not only can you watch sporting events in HD from ESPN, but you can also "join" other fans in rooting for your favorite team and answering sports trivia questions [source: Microsoft].

This article recalls the earliest buzz around Project Natal, describes the hardware, software and development behind the project, and explores how Microsoft Kinect could change the video game console market compared to the Nintendo Wii and the PlayStation Move.

 

Project Natal Challenges the Wii

The 3DV Systems motion-sensitive ZCam camera was on display at CES 2008, before the company was acquired by Microsoft. The device was likely a predecessor of Project Natal and Kinect.
The 3DV Systems motion-sensitive ZCam camera was on display at CES 2008, before the company was acquired by Microsoft. The device was likely a predecessor of Project Natal and Kinect.
AP Photo/Paul Sakuma

On June 1, 2009, Microsoft's annual E3 press conference announced the development of a motion-sensitive gaming technology code-named Project Natal. The promotional video on YouTube, posted the same day as the press release, portrayed Project Natal as a controller-less way to play games, watch movies and keep in touch with friends. The video featured game control by just moving your body, plus facial and voice recognition for each player. At the end of the video, Microsoft gave viewers a peek at the hardware: a sleek, black, low-profile box sitting on a small platform, looking at you with two camera-like eyes in the middle and a green light offset to the left.

At the heart of the project was a 3-D detection digital camera. Reports suggested that the technology likely originated with Israeli start-up 3DV, a company Microsoft had acquired just two months before. Press images of the 3DV camera from months before were significantly different from the Project Natal form factor: a small upright box with a series of purple lights and a protruding camera lens in the center.

The natural response from video game console enthusiasts was to compare the technology to the Nintendo Wii. Nintendo had revolutionized console gaming when it released the Wii in 2006. The Wii meant that, for the first time, the game controller was more than just a series of buttons: It also detected the movement of the controller and allowed the player to interact with the game based on that movement. For three years, the Wii was unmatched for bringing motion detection to home video game consoles.

The central part of the Wii experience was the controller. The movement of the controller was the means of transmitting the player's movement to the console. In fact, Nintendo and other vendors created enhancements so the player could hold an object similar to that in the game (like a guitar or a tennis racket), or so the player could get motion feedback (a "rumble" effect). The Wii Balance Board controller added even more interactive options for players, detecting pressure from standing on it and shifting your weight.

Project Natal promised a dramatic leap forward from the Wii as it seemed to have overcome the need for any physical controller. Even better: Those with existing Xbox 360 consoles could use it without buying a completely new console. For the year following its original announcement, Microsoft gave celebrities and talk show hosts a chance to try it out. The result was a series of TV segments and YouTube videos buzzing about the good, the bad, and the creepy of the Project Natal experience. By the next E3 event, Project Natal, officially branded Microsoft Kinect, was one of the most anticipated Microsoft products in years, with a line of Kinect-based games to appeal to the same diverse audience as the Wii.

The Kinect Sensor

The Microsoft Kinect sensor as shown in a press briefing in June 2010
The Microsoft Kinect sensor as shown in a press briefing in June 2010
Michal Czerwonka/Getty Images

The innovative technology behind Kinect is a combination of hardware and software contained within the Kinect sensor accessory that can be added to any existing Xbox 360. The Kinect sensor is a flat black box that sits on a small platform, placed on a table or shelf near the television you're using with your Xbox 360. Newer Xbox 360s have a Kinect port from which the device can draw power, but the Kinect sensor comes with a power supply at no additional charge for users of older Xbox 360 models. For a video game to use the features of the hardware, it must also use the proprietary layer of Kinect software that enables body and voice recognition from the Kinect sensor [source: Rule].

There's a trio of hardware innovations working together within the Kinect sensor:

  • Color VGA video camera - This video camera aids in facial recognition and other detection features by detecting three color components: red, green and blue. Microsoft calls this an "RGB camera" referring to the color components it detects.
  • Depth sensor - An infrared projector and a monochrome CMOS (complimentary metal-oxide semiconductor) sensor work together to "see" the room in 3-D regardless of the lighting conditions.
  • Multi-array microphone - This is an array of four microphones that can isolate the voices of the players from the noise in the room. This allows the player to be a few feet away from the microphone and still use voice controls.

A further look at the technical specifications for Kinect reveal that both the video and depth sensor cameras have a 640 x 480-pixel resolution and run at 30 FPS (frames per second). The specifications also suggest that you should allow about 6 feet (1.8 meters) of play space between you and the Kinect sensor, though this could vary depending on where you put the sensor [source: Microsoft Store].

The Kinect hardware, though, would be nothing without the breakthrough software that makes use of the data it gathers. Leap forward to the next page to read about the "brain" behind the camera lens.

Kinect Software Learns from "Experience"

Kinect's software layer is the essential component to add meaning to what the hardware detects. When you first start up Kinect, it reads the layout of your room and configures the play space you'll be moving in. Then, Kinect detects and tracks 48 points on each player's body, mapping them to a digital reproduction of that player's body shape and skeletal structure, including facial details [source: Rule].

In an interview with Scientific American, Alex Kipman, Microsoft's Director of Incubation for Xbox 360, explains Project Natal's approach to developing the Kinect software. Kipman explains, "Every single motion of the body is an input," which creates seemingly endless combinations of actions [source: Kuchinskas]. Knowing this, developers decided not to program that seemingly endless combination into pre-established actions and reactions in the software. Instead, it would "teach" the system how to react based on how humans learn: by classifying the gestures of people in the real world.

To start the teaching process, Kinect developers gathered massive amounts of data from motion-capture in real-life scenarios. Then, they processed that data using a machine-learning algorithm by Jamie Shotton, a researcher at Microsoft Research Cambridge in England. Ultimately, the developers were able to map the data to models representing people of different ages, body types, genders and clothing. With select data, developers were able to teach the system to classify the skeletal movements of each model, emphasizing the joints and distances between those joints. An article in Popular Science describes the four steps Kinect's "brain" goes through 30 times per second to read and respond to your movements [source: Duffy].

The Kinect software goes a step further than just detecting and reacting to what it can "see." Kinect can also distinguish players and their movements even if they're partially hidden. Kinect extrapolates what the rest of your body is doing as long as it can detect some parts of it. This allows players to jump in front of each other during a game or to stand behind pieces of furniture in the room.

A Game Changer for the Game Console Market?

The sleek Xbox 360 console with the Kinect sensor (left) and a hand-held controller (right) scheduled for release in November 2010
The sleek Xbox 360 console with the Kinect sensor (left) and a hand-held controller (right) scheduled for release in November 2010
Michal Czerwonka/Getty Images

Just as the gasoline engine freed vehicles from horses to pull them, Kinect frees the video game player from needing a controller to use the system. While Kinect may not change the world on the scale of the gasoline engine, its potential impact on the video game console market is no less profound. Plus, we have yet to discover what the technology could mean for other industries.

As previously mentioned in this article, Microsoft Kinect provides an interactive video game experience through motion detection like the Nintendo Wii, but without the need for physical controllers. YouTube videos of people trying out Kinect at E3 show breathless, happy players jumping, kicking and dancing more vigorously than they could with Nintendo Wii controllers in hand or under feet. As of this writing, there's no word on how Nintendo will respond with its own product line.

From a price perspective alone, the Wii is the better bargain until you have three or more people playing at one time. At full retail price, the Wii console with one controller (including Nunchuk) is $199, and each additional controller plus Nunchuk could set you back $60. The Kinect sensor retails at $149 and includes a copy of "Kinect Adventures." The device must be attached to an Xbox 360 console, and if you don't have one yet, the Xbox 360 with 4 GB of internal flash memory costs $199. You can get the same unit bundled with a Kinect and "Kinect Adventures" for $299 [source: Microsoft Store]. This doesn't count purchasing additional games, batteries (for the Wii controllers), or a subscription to Xbox Live to connect and play over the Internet (no extra cost for Wii owners to play online games).

Sony, the other video game console giant, is also making strides in this race. At the same E3 event where Microsoft announced Project Natal, Sony demonstrated its own controller-free, motion-detecting technology it called the Interactive Communication Unit (ICU). ICU was created in collaboration with the Swiss firm Atracsys, which already sells hands-free controls for medical professionals in sterile environments [source: Atracsys]. However, Sony's March 2010 announcement of the motion-detecting PlayStation Move, revealed that it was not based on ICU, but rather on light-emitting hand-held controllers similar in size to the Wii controllers [source: Patel].

For players who prefer the tactile feedback of props, the Nintendo Wii and PlayStation Move will have the advantage. For example, holding a video game gun that lets you aim and pull the trigger at the screen might give you a better first-person-shooter experience than merely mimicking holding and firing a gun. To capture this audience, Microsoft might respond with games that use, or even require, props.

Will Kinect's innovation go beyond the entertainment industry? At the time of this writing, it was too soon to tell. However, there was already buzz about how it could be used in healthcare and education. Imagine how Kinect could be used for tracking your progress in physical therapy, driving your car when you can't use a steering wheel, or simulating a day in the life of a soldier during the American Revolution.

The game is on for Kinect. Start your own adventure with some of the links on the next page.

Related HowStuffWorks Articles

More Great Links

Sources

  • Atracsys. "ICU - Interactive Communication Unit." (July 5, 2010)http://www.atracsys.com/_products/icu.php
  • Duffy, Jill. "Exclusive: Inside Project Natal's Brain." Popular Science. Jan. 7, 2010. (July 4, 2010)http://www.popsci.com/gadgets/article/2010-01/exclusive-inside-microsofts-project-natal
  • Kuchinskas, Susan. "Binary Body Double: Microsoft Reveals the Science Behind Project Natal for Xbox 360." Scientific American. Jan. 7, 2010. (July 4, 2010)http://www.scientificamerican.com/article.cfm?id=microsoft-project-natal
  • Microsoft. "E3 2009: Project Natal Milo demo." YouTube. June 1, 2009. (July 4, 2010)http://www.youtube.com/watch?v=CPIbGnBQcJY
  • Microsoft. "Project Natal." YouTube. June 1, 2009. (July 4, 2010)http://www.youtube.com/watch?v=g_txF7iETX0
  • Microsoft. "Xbox 360 - ESPN - Xbox E3 Media Briefing." YouTube. June 19, 2010. (July 5, 2010)http://www.youtube.com/user/xbox#p/search/18/wmEi27Sr3b0
  • Microsoft Store. "Kinect for Xbox 360." (July 5, 2010)http://store.microsoft.com/microsoft/Kinect-for-Xbox-360/product/C737B081
  • Patel, Nilay. "PlayStation Move motion controller launched at GDC, starter kit to be under $100 with game." Engadget. March 10, 2010. (July 5, 2010)http://www.engadget.com/2010/03/10/playstation-move-motion-controller-launched-at-gdc/
  • Rule, Trisha. Microsoft spokesperson. Personal interview. Conducted on June 29, 2009, via e-mail.
  • Stevens, Tim. "Microsoft Kinect gets official, Video Chat announced." Engadget. June 13, 2010. (July 5, 2010)http://www.engadget.com/2010/06/13/microsoft-kinect-gets-official/
  • Terdiman, Daniel. "Microsoft's Project Natal: What does it mean for game industry?" CNET. June 1, 2009. (July 4, 2010)http://news.cnet.com/8301-10797_3-10253892-235.html
  • Terdiman, Daniel. "New game controller: Your hands." CNET. Dec. 10, 2007. (July 4, 2010)http://news.cnet.com/New-game-controller-Your-hands/2100-1043_3-6222024.html
  • Terdiman, Daniel. "Report: Microsoft taking on Wii with motion-sensing camera." CNET. May 12, 2009. (July 4, 2010)http://news.cnet.com/8301-10797_3-10239304-235.html