The Natural User Interface (NUI) depends upon a sensor array contained within the Kinect sensor, system services, tools, and the API that your app uses to implement its user interface.
There are two broad classes of NUI data available from the NUI platform: vision data and speech data.
Vision data
The NUI system services deliver vision data at a number of levels of abstraction, from basic image and depth data to more sophisticated information such as body position, hand state, and even players’ expressions. Your app uses this data to implement its user interface based on players’ movements, expressions and gestures.
Speech data
The NUI system services support user interfaces based on speech recognition. Your app can define a set of voice commands and recognize when users say those commands.
The NUI platform also includes a set of tools that facilitate the development and testing of NUI apps.
The Kinect sensor is connected to your Xbox One console. The Kinect sensor provides an active infrared (Active IR) image sensor and a depth sensor with a broader field of view and 3D depth information with higher fidelity than that provided by Xbox 360 Kinect Sensor. The Kinect sensor also provides the color camera and audio input.
The NUI platform is organized differently than the comparable components on Xbox 360. On Xbox 360, titles link NUI libraries into the title executable, and NUI code runs in the title’s resource reservation. While the Xbox 360 libraries are organized on a “pay for what you use” basis, it is still true that using NUI features in an Xbox 360 title imposes overhead on the title, and consumes resources that would otherwise be available for the title to use.
The bulk of the support for NUI on Xbox One is implemented in system services that operate in the system resource reservation. This means that games are free to use NUI features while still using the entire game resource reservation for game features, and that game developers do not have to be concerned about their titles absorbing the burden of overhead from the NUI system processing. (It also means that the system can use NUI inputs for system-related purposes while a game is running; this is one reason why the system reservation exists even when a game is running.) For more information about NUI, see the The NUI Runtime for Xbox One white paper. The full list of white papers is available at Xbox Game Developer: White Papers.
There are a number of different data streams to support your NUI implementation. All of these streams return a sequence of data frames, one at a time. Each frame is a snapshot of input data values at a given time, just as the display frames that your app renders to the screen are displayed one at a time, sequentially, to animate your app’s output. Some of these data streams deliver image frames directly from the color, infrared, and depth sensors. Other data streams perform processing on one or more image frames, and deliver more abstract data. For example, the NUI system combines information from the image sensors to recognize human bodies, and provides two streams with body data: Frames of the Body Index stream indicate which body is closest to the Kinect sensor at each pixel of the corresponding color image frame. Frames of the Body Stream are not images, but rather a list of joint locations and orientations that describe each of up to six bodies recognized in the field of view.
The NUI Gesture API uses machine learning on data gathered with Visual Gesture Builder (VGB), a tool that recognizes gestures by finding statistical correlations between examples of the intended gesture. For more information about VGB, see VGB - Visual Gesture Builder.
The Body data stream also includes expression information, so that your code can identify when a player appears to be smiling or engaged, and use that information within the game. Even humans will disagree as to whether a person in a single picture is necessarily smiling or not. So the algorithms used to return expression data don’t return a single state, but rather a list of the possible states, with confidence values for each state. And your code should use those confidence values (possibly in combination, or over a number of frames, or in conjunction with other data) to determine whether to react to an expression.
The Identity system manages the mapping from bodies in the sensor array’s field of view to users who are signed on to the console. Bodies visible to the sensor array are tracked as controllers, and each controller can be paired to a user object representing a signed-in user. When properly configured using the Biometric Enrollment Tool, the Identity system uses biometric data to recognize that a particular body corresponds to a particular user, and maps them automatically.
The NUI system implements speech recognition of what players say, by using the microphone array in the Kinect sensor. This enables users to interact with games or the system by using voice commands.
For more information about NUI Speech, see Speech Overview.
The following tools help you to develop your title to use NUI:
NuiView is a tool that provides visualizations of NUI data streams. It allows you to record and play back Xbox raw files (XRFs), to record and play back Xbox event files (XEFs), and to identify users who are already enrolled on the console.
For more information, see NUIView.
Xbox Studio Clip Conversion tool is a command-line tool that converts raw recordings into processed recordings and upgrades previously processed recordings.
For the command-line syntax and usage examples, see KSConvert - Kinect Studio Clip Conversion Tool.
Visual Gesture Builder is a tool that enables you to generate data for gesture detection at run time for the apps using the NuiGesture API. For more information, see VGB - Visual Gesture Builder.
For a quick start on building a gesture detector, see Tutorial: Building a Gesture Detector.
Xbox Studio is a new version of Xbox Studio that has been updated for Xbox One and the Kinect sensor. In this release, you can use Xbox Studio to play and record vision data, to view saved depth and active IR data from pre-recorded XEFs, and to explore the augmented field-of-view (FOV) and the enhanced resolution produced by the Kinect sensor. For more information, see Kinect Studio.
SpeechLab is a tool suite for authoring, validating and tuning grammars (phrases) used in your experience. For more information, see SpeechLab.