Silver Tongue: The Xbox One Speech Cheat Sheet

This topic cover the most important things to know about grammar construction using NUI Speech, designing a game experience with speech, and potential pitfalls with speech technology. We recommend that all game designers read this guide as they begin a new speech-enabled project.

Grammar Construction

When you create your grammar state, the following guidelines will help you avoid common issues as you incorporate speech into the title.

Grammar construction Impact Comments
More phrases in a grammar state mean more testing and more data collection Having fewer active phrases leads to better chances for recognition accuracy and clearer constraints for the user. Change context based upon game state, players past speech, gestures, and so on. Titles can enable and disable individual rules dynamically.
Typically avoid the garbage rule Doing so decreases false acceptances, but it increases false rejections and CPU utilization. It is better to understand and tune for confidence thresholds and to use selective engagement models.
Use phrases with two or more syllables Longer phrases lead to more robust recognition and reduce false acceptance of something outside of the grammar. Balance length with ease of utterance. Avoid unnaturally long phrases.
Provide custom pronunciations where appropriate Increase recognition for uncommon or invented words and phrases. Nui Speech makes an educated guess, but to maximize success, consider providing pronunciations or alternate spellings for words that do not recognize well .
Consider localization during design Use of semantic tags rather than specific words can reduce code changes, because the recognition code path remains consistent across languages. Leverage SRGS tags for branching rules. However, be aware of potential differences in phrase construction across languages, and when localizing, use spoken (versus written) constructions.

Note Use custom pronunciations with extreme care. If done incorrectly, adding custom pronunciations will result in lower accuracy.

Game design

These guidelines are important to remember whenever you create any game design that uses speech recognition.

Game design Comments
Target a 90 percent rate of user-perceived successful recognition This is often accomplished as much through game design as grammar construction: use context-sensitive recognition, act intelligently for false rejections or low-confidence values, incorporate feedback from usability and play testing.
User education and training Tutorials, training, and visual/auditory feedback that help you understand when the title is actively listening for the player to say something.
Engagement model There are many options for engagement and interaction models. For menu navigation scenarios, we suggest See It Say It as the primary engagement model.
Provide appropriate direction For constrained grammars, consider supporting synonyms. For broader grammars, keep commands intuitive and consider ways to communicate options to the player. Remember that the larger the grammar, the more testing and tuning is required to deliver a good experience.
Provide feedback to the player Communicate both successes and failures in intuitive ways. Consider using context to maximize perceived successes. Consider confirmations in critical situations or when confidence is low.
Don’t compete with the player In scenarios where speech is expected, consider reducing game-generated sounds (particularly in the speech frequencies).
Combine speech with gesture Give the player alternate avenues for performing actions when speech or gesture is not recognized.

Potential Pitfalls

Look out for the following common problem areas when you design speech experiences in games:

Potential pitfalls Examples Mitigations
Very short phrases Phrases comprised of individual digits, letters, or other single-syllable words (pause, start, and so on). Consider longer phrases.
Broad or large active rules The alphabet; multiple synonyms for multiple enabled actions. Look for ways to reduce active scope based on context.
Ambiguous or confusable phrases Phrases that differ by few sounds, or are wholly or partially contained within each other. Evaluate available words, across all localizations, in any given game context.
Keyword spotting Surrounding interested action words with the garbage rule. Rather than depending on key words that could be missed, train players to speak longer commands.
Speech as an afterthought “Speech-ifying” menus without evaluating their grammars, flow, or error handling. Consider and thoughtfully plan functionality by using these best practices. Speech is an input and needs to be part of the design from the beginning.

Relevant Documents

Please see the following white papers and XDK documents to find out more about speech design and implementation guidance on Xbox One.