The SimpleNuiInteractions sample demonstrates how to use the NUI interactions API to implement gesture interactions for enabled targeting with a hand cursor, pressing a button, and scrolling.
Use this table to look up the user action that is required to give you the desired interaction with the SimpleNuiInteractions sample.
| Desired Interaction | User Action |
|---|---|
| Become the engaged user (which will show a hand cursor). | While no other user is engaged, hold hand in front of body, palm facing the sensor. |
| Select a colored square. | While the hand cursor is over a square, press hand towards the screen to full hand reach extent, and then retract the hand. |
| Scroll items around screen. | While the hand cursor is visible, close hand until gripped cursor shows, then move the cursor to scroll the items on the screen. Open hand to stop scrolling. |
| Exit the sample. | As always, Xbox button will take you to Home. View button will quit the app (for development ease only). |
The Simple NUI Interactions sample illustrates the key concepts for using NUI Interactions with the Xbox One XDK. The sample creates 4 buttons and 1 scrollviewer and creates the associated GestureRecognizers for those 5 controls. To download the sample, visit Samples, search on that page for “Simple NUI Interactions”, and click the link that you find.
In game.cpp, the game registers handlers for KinectInteractionQuery and PointerMoved events on ICoreWindow.
ICoreWindow.PointerMoved is raised to update the location of each tracked hand in the scene; the hand’s position is represented by PointerEventArgs.PointerPoint. A title should determine which hands should be represented by Cursors.
When PointerPoint.Properties.IsPrimary is true, that represents the hand that the system has deemed to be engaged with the Xbox. A title can also use PointerPoint.Properties.BodyTrackingId and PointerPoint.Properties.HandType to choose which PointerPoints to use.
Replace the logic in Game::IsEngagedHandPointer if your game wants to control who is engaged.
Using Game::_scrollCaptureElements and Game::_pressCaptureElements, the Game class keeps a map from pointerId to UIElement. Each entry in those maps represents a UIElement which has scroll or press capture for a specific pointerId.
In the SimpleNuiInteractions sample, UIElement is the base class for Button, ScrollViewer, and Cursor. UIElement keeps track of the GestureRecognizer the button or scrollviewer.
In the OnInteractionQuery handler, the title should use the same filter as mentioned above in “Choosing Which PointerPoints to Use”. For those hand pointers that will be used to represent hand cursors on the screen, one should return a gesture recognizer via the eventArgs.KinectInteractionQuery.ProvideResult() method. Scroll capture should be preferred over press capture, when determining which gesture recognizer to return.
The NUI Interaction system uses this information to ensure that the PointerPoints provided in the PointerMoved event have the correct coordinates. You’ll notice as you press a button, that the hand pointer is attracted to the center point of the buttons. This is done by the press detection gesture recognizer when pressing a button. PointerPoint.RawPosition and .Position give you the unchanged and the changed x and y values. Cursor rendering should use .Position.
In OnPointerMoved, one can see the critical logic required by the UI Framework of your title to appropriate route PointerPoints to the right GestureRecognizers.
This routine calls GetElementsAtPoint to understand which elements are under the point. HitTestResult includes 2 elements – the first Kinect aware element and the first Kinect aware manipulation (scrolling) element.
Using a combination of which elements are captured, which elements were returned in the HitTestResult, the title needs to route the PointerPoints to the appropriate GestureRecognizers so that they can recognize when a “Press” of a button happens and when a “Scroll” of a scroll viewer happens.