Windows Runtime (WinRT), which was first introduced in Windows 8, has become the framework for exposing APIs that cover a wide range of development languages and platforms. WinRT for Xbox One defines and implements WinRT APIs for Kinect that are used to collect data from the Kinect sensor for Xbox One. This includes video and audio data.
Xbox One supports one sensor, which is called the default sensor. The KinectSensor Class has static members to help configure the Kinect sensor and access sensor data.
The IsOpen property allows you to check to see if other applications have the sensor open.
Key APIs are:
Once a Kinect sensor is chosen, the sensor must be opened and a mode specified—shared or exclusive.
The IsOpen property allows you to check to see if other applications have the sensor open.
Use the sensor to access data from several sources including: color, depth, body, body index, infrared, and audio. Functionally, each source is controlled by the following three things:
Source type
Inspect or configure a source and open a source reader.
Source reader type
Access to the source’s frames using eventing or polling.
Frame type
Access the data from a particular frame from the source.
For examples of how these are represented in the API, see the following topics:
Once a developer has an open sensor they can use that instance to gain access to the individual streams: color, depth, body, body index, and infrared. Each stream is broken up into 3 basic pieces:
Stream type
lets you inspect or set the stream settings and allows you to open a stream reader.
Stream reader type
Gives you access to the stream’s frames via eventing or polling.
Frame type
Gives you access to the data from a particular frame off of the stream.
The stream type’s job is to give you access to the stream and allow you to configure it (if in exclusive mode) and open a stream reader. Each stream has its own type, but they all share the same basic functionality.
| API element | Description |
|---|---|
| BodyIndexFrameSource.OpenReader Method | Opens a reader for this stream. This will fail if conflicting streams are already open. |
| BodyIndexFrameSource.IsActive Property | Indicates whether there are any active stream readers for this stream. |
| BodyIndexFrameSource.FrameDescription Property | Gives you the information you need to know about the resulting frames so that you can pre-allocate a correctly sized buffer to store frame data. Does not apply to BodyFrameStream |
| BodyIndexFrameSource.KinectSensor Property | A back pointer to the sensor that this stream is exposed from |
Each stream has its own type of stream reader, but they all share the same basic functionality.
| API element | Description |
|---|---|
| BodyFrameReader.AcquireLatestFrame Method | Get the latest frame. |
| BodyFrameReader.FrameArrived Event | Subscribe to this event to be notified when new frames are available. |
| BodyFrameReader.IsPaused Property | Controls whether or not frames will be delivered to the users of this particular stream reader instance. This will have no effect on the stream itself or other open stream readers. |
| BodyFrameReader.Close Method | Closes the stream reader. When the last open stream reader for a stream is closed, the stream itself will be disabled. |
| BodyFrameReader.BodyFrameSource Property | A back pointer to the stream this reader was opened from. |
A frame contains the data delivered from the sensor. Use the frame objects to copy frame data into the applications buffer or access the underlying system buffer. Each frame stores frame data temporarily to avoid memory allocation. An application should get the data out of each frame and close/dispose it as quickly as possible to free up the underlying handle and make sure that the system does not need to keep allocating new items to store incoming frame data.
Each stream has its own frame type, but they all share the same basic functionality. While each frame does have many similarities, these are the types that are most different between each individual stream.
| API element | Description |
|---|---|
| DepthFrame.CopyFrameDataToArray Method | Copies the frame’s pixel data to an app provided array based buffer. |
| DepthFrame.CopyFrameDataToBuffer Method | Copies the frame’s pixel data to an app-provided IBuffer. |
| DepthFrame.LockImageBuffer Method | Gives application access to the underlying buffer used by the system to store this frame’s data. Keeping this buffer active for long periods of time will result in the system continuously allocating new buffers for incoming streams. Attempting to write to this will fail on Xbox One. |
| DepthFrame.Close Method | Releases the handle to the underlying system buffer. After this is called, all other operations will fail. |
| DepthFrame.DepthFrameSource Property | A back pointer to the stream reader that provided this frame. |
| DepthFrame.FrameDescription Property | Provides the information the app needs to know about how to allocate a buffer for this frame. Does not apply to BodyFrame. |
| DepthFrame.RelativeTime Property | Provides a TimeSpanUInt64 representing when this frame was created. This value does not allow you to determine the absolute, real-world time of creation, but it does allow you to know when frames were delivered in relation to each other, both within and between frames from a given Kinect sensor. Units are in milliseconds. |
Video data from the Kinect sensor includes low-level data, such as infrared and color, as well as processed data, like depth and body (commonly referred to as skeleton). These APIs have been designed to follow standard WinRT design guidelines which support all WinRT-compatible languages and UI frameworks.
The infrared image frame is the most basic of all of the stream types; it follows exactly the pattern described earlier. The pixel type is UInt16.
The DepthFrameSource Class is very similar but has a few additional members. The pixel type is UInt16.
| API element | Description |
|---|---|
| DepthFrameSource.DepthMinReliableDistance Property, DepthFrameSource.DepthMaxReliableDistance Property | The high confidence range in the depth field. This is dependent on DepthRange DepthFrameSource members. |
Color frame source extends the pattern defined in depth stream, because there are a few options to be set on the stream. Use this source to get a copy of the image frame data converted into the desired color image format (such as RGB or YUV). Regardless of the format, the pixel type will be a UInt8 (byte). There are CopyRaw and LockRaw accessors for the different formats, as well as new CopyConverted accessors, which convert the raw image to the specified format in the provided buffer.
| API element | Description |
|---|---|
| ColorFrameSource.CreateFrameDescription Method | Creates a FrameDescription object that describes a color frame with the provided format and resolution. This replaces the FrameDescription property found on other image streams. |
| ColorFrameSource.OpenReader Method | Opens a new stream reader. This reader must be disposed. |
| ColorFrame.CopyConvertedFrameDataToArray Method | |
| ColorFrame.CopyConvertedFrameDataToBuffer Method | Fills provided buffers with a version of the image converted into the specified format. |
| ColorFrame.CopyRawFrameDataToArray Method | |
| ColorFrame.CopyRawFrameDataToBuffer Method | |
| ColorFrame.RawColorImageFormat Property | Returns color format of raw pixels. |
Audio data is collected from a microphone array and processed into beams that emphasize the sound from a certain direction. A beam can be set to automatically track a sound source or to be aimed in a specific direction. The sound data for a beam is divided up into frames that roughly correspond to each video frame. The sound frames are sub divided into subframes that are 16ms long and contain mono 32-bit IEEE floating point PCM data, sampled at 16 kHz, with typical PCM values that range from -1 to +1. While a vector of audio beams is available to you, only one audio beam is currently supported.
The audio APIs provide a way to correlate audio data with tracked bodies. The SubFrames property of the AudioBeamFrame class provides a vector of at most 8 AudioBeamSubFrame objects, which are the 16ms-long frames of audio data. To correlate an AudioBeamSubFrame with a body, use the AudioBodyCorrelations property, which returns a vector of at most 6 AudioBodyCorrelation objects, sorted by their angular distance from the audio beam angle. This vector will be empty if there are no tracked bodies.
To begin collecting audio data, first acquire the Kinect sensor as described above. Then, use the AudioSource property to get the audio source for the sensor. Once you have the audio source, you have three options for processing audio data:
The following sections will discuss each of those methods.
If you are confident that your application’s performance will not suffer by responding to audio data nearly continuously, you can respond to FrameArrived events to process audio data in real time.
To respond to FrameArrived events, register for the FrameArrived event of the AudioBeamFrameReader class. When the event is raised, you can then inspect the data that is contained in the AudioBeamFrameArrivedEventArgs object that is passed with the event. This object contains a vector of AudioBeamFrame objects. As noted above, there is currently only one audio beam in this vector.
By using the polling method of acquiring audio data, you can control when your application spends resources retrieving audio data, perhaps improving your application’s performance.
An AudioSource has an OpenReader method that returns an AudioBeamFrameReader. An AudioBeamFrameReader, in turn, has the AcquireLatestBeamFrames method, which returns a vector that contains the most recent audio data. In the current release, this vector contains exactly one AudioBeamFrame instance.
In addition to supporting the frame-based polling and event methods for data acquisition, WinRT for Xbox One allows you to stream audio data by using an IInputStream. The OpenInputStream method of the AudioBeam class will give you an input stream. Note that the stream contains only audio data, and does not include confidence information or beam angle information, for example. However, the beam angle, beam angle confidence, relative time, and beam mode information are accessed directly off of the AudioBeam class from which you obtained the stream reader. These properties are continuously updated and represent the real-time characteristics of the audio beam.
An app can overwrite the default automatic beam mode, which tracks the loudest sound source. If the sound source location is known in advance relative to the Kinect Sensor, the app can fix the beam to an angle that points to the sound source. To do this, get the AudioBeam and first set the AudioBeamMode property to manual. Then set the BeamAngle property to the desired beam angles in radians, in the range [-pi, +pi].