Steven Hammond, Advanced Technology Group
Updated April 4th, 2017
This white paper provides insight into the Xbox One certification process to minimize the chance of failing your first submission. Since the launch of Xbox One, the percentage of games that fail their first certification submission has steadily declined. This paper is a collection of the knowledge we’ve gathered based on the experience of previous game submissions. You can avoid unforeseen delays in your game launch by incorporating this knowledge into your own test and submission process.
After you’ve created a build that’s ready for certification, you still need to go through a few steps before it’s ready to be released:
After testing is complete, a report will be generated with any issues found during testing. The report may contain minor bugs and/or Conditions for Resubmission (CFRs). We strongly encourage you to address all of the issues reported, but only CFRs are required to be fixed before a game can be published.
Note: In general, it is necessary to meet all of the Xbox Requirements (XRs) listed in the certification documentation. If a game is unable to meet an XR, either for technical reasons or because of particularly innovative or unprecedented design, you can request an exception before entering certification. Testing against any XR that you have been granted an exception for will not occur. If you feel a CFR was wrongfully reported during certification, you can submit a challenge against a CFR. For more information about exceptions and challenges, see the Xbox One Certification Process Guide.
The rate at which games make it through certification on the first submission has been increasing since the launch of Xbox One. This has been due in part to a simplification of the Xbox Requirements, improvements in the Xbox One APIs, and better educational materials for preparing for submission. More than anything, though, it has been due to developers making higher quality games and being more familiar with what needs to be tested. Being familiar with the XRs that cause the most CFRs will help you get through certification on the first try.
The most common reason for a holdup during certification is that the game or provided materials are not in a state in which they can even be tested. XR-003 is a comprehensive requirement that states that your game must be in a state in which it’s ready for global release when it is submitted for certification.
Make sure the following are included:
Certification testing will be as comprehensive as is practical within their time and resource constraints. This includes every combination of supported input and game options available. Be sure to include the use of all available settings in your game, and have testers rotate their input devices to include all types of controllers. If appropriate for your game, include:
Some specific problem areas to keep in mind with your own test matrices include:
The majority of the CFRs under this requirement are found while testing other XRs; requirements include:
To maximize your chances of catching stability issues, use a combination of black box and white box testing. Keep in mind the following characteristics of white box testing:
Black box testing is also necessary for a well-rounded test matrix. The characteristics of black box testing include:
Some specific scenarios to keep in mind when setting up tests:
This XR requires integration of joinable game sessions with the Xbox One Friends application. If your game supports joinable single player or multiplayer gameplay, players must appear as joinable in the Friends application. If a player is in a non-joinable game session, they must not appear as joinable in the Friends application.
Be sure to include these common CFRs when designing your test matrix:
Note: A failure to join a game session may occur if a player attempts to join their friend in a game session that’s already full or that fills up in the time taken to launch the game and attempt to join the session. Failures to join a session must be properly messaged to the player from within the game UI.
The behavior under this XR is fairly straightforward, but there are some corner cases that must be kept in mind when implementing it. The core of this requirement is that your game must allow players to send invitations from within the game if joinable multiplayer gameplay is supported.
A set of APIs is available to make sending invitations fairly straightforward. Games developed with an XDK released before 2015 would have used the ShowSendInvitesUIAsync and/or InviteToPartyAsync functions. With the newer 2015 APIs, these would be replaced with ShowSendGameInvitesAsync and MultiplayerService::SendInvitesAsync** respectively.
Common CFRs encountered include:
Note: How can you accidentally join the wrong session? Title code should create sessions and tear them down. If your game shuts down unexpectedly, such as crashing or getting shut down by a debugger, the session will remain alive on the Xbox Live servers and be associated with the player it was originally created for. Avoid having friends join these lingering sessions by querying for existing sessions for the current user in the initial interactive state (IIS) of your game. Any existing sessions during the IIS can be assumed to be stale and should be torn down.
Handling an inactive profile is another area to keep in mind when handling invitations. Consider the following scenario:
It is common to see that only profile B is added to the game session. In this scenario, make sure that inactive profiles (in this case, profile C) are also added to the game session.
Lastly, not all invitations can be sent or accepted. In these instances, make sure it is messaged to the player why an invitation can’t be accepted. For example, if a player sends an invitation to two of their friends to play a one-on-one game, only one of those invitations can be accepted. It is alright to send multiple invitations, but if both friends accept the invitation, only the first one to accept it will be able to join. When the second friend accepts the invitation, they must be alerted that they weren’t able to join and shown a message as to why they weren’t able to. In this case, they weren’t able to join because the joinable session has no more open slots.
Ensure that:
This XR simply requires that your game handle a loss of connectivity gracefully by handling failures returned from any calls made to Xbox Live services. Common failures include games hanging or crashing when connectivity is lost. Make sure you include the following in your test cases:
Following are a few methods of testing this XR:
Note If you’re having trouble with this XR, consider adding the RequireXboxLive=”True” attribute to your manifest if your game has no offline gameplay. This should only be used for games that require a connection to Xbox Live services in all gameplay modes.
Resources:
Service Interruption Resiliency for Titles white paper
This XR was introduced to reduce costly user support on the part of Microsoft and publishers. High quality and accessible support information can improve the user experience at a time when the user is most likely unpleased with the console or game.
To meet this XR, make sure that all of your files are ready for final certification testing. It is not enough to have those components scheduled for release at your title’s street date. This includes external websites that the Help file links to. However, it is admissible to point users to an in-game Help component as long as it is clearly messaged in the Help file how to navigate to that in-game experience.
If your game supports multiple languages and locales, your Help file must also support all of those languages. At the time of certification, it is admissible to have your game point to an English version of your Help file as long as all supported languages are ready to go at the time your game is released.
Resources:
This is a simple requirement that’s easy to overlook. Your game must indicate who the active user is and allow the opportunity to change who the active user is before taking any profile-specific actions. This include writing stats, saving or reading save data, or modifying preferences.
When your game is first launched or resumes from a suspended state, the following must be done:
Note: There are multiple ways to determine who the active user is. The CurrentUser property can be queried to see who launched the game. If only one user is signed in, it can be assumed they are the active user. If only one controller is connected, it can be assumed it is being used and queried for a user paired with it.
Make sure your game is able to gracefully handle a change in the active user via the account picker. Changing the active user often causes a crash, or not all game engine components are updated with the new active user.
Resources:
This requirement states that anytime the player’s gamertag or gamerpic are displayed in the UI, the correct data must be used and it must be displayed properly. The most common causes of CFRs for this requirement include:
The Xbox One SDK includes the GameDisplayName, Gamertag, and GameDisplayPic functions, as well as RESTful and Xbox Live APIs, to get the latest profile information. Rely on these functions to get the most up-to-date information at all times instead of relying on previously cached data. It doesn’t need to be queried every frame, but it should be updated every time the data is required or during PLM events.
Your game must be able to render at least 16 characters when displaying the gamer’s name. If the display name is longer, it is admissible to display an ellipsis. The display name may contain Unicode characters that you do not have to support. Unsupported characters must be replaced by a rectangle character. They cannot be replaced by other characters, such as question marks or apostrophes.
Note: A special gamerpic is available on development consoles to test this requirement. When signed in to your profile, open the Social or Friends app and select My profile. From there, open the drop-down menu from Customize profile and select Change gamerpic. The first gamerpic in the collection should be an Xbox logo with a black background. When it is selected, the display name and gamertag of your test profile will change to a long test string that contains various special characters. Use this gamerpic to test odd gamer display names.
The core of this XR is knowing which user should be associated with progress for stats, achievements, and so forth, as well as which user is associated with saved data. There is the chance that any user may be signed in or out at any time. This includes the current active user.
Consider the following scenarios:
Make sure you include the following tests:
The XbUser command-line utility can be used to test users being signed in or out at will. These commands are especially useful with this utility in an Xbox One command prompt:
To test process lifetime management (PLM) state transitions, these commands are useful in an Xbox One command prompt:
Note: If you prefer working with a GUI instead of the command prompt, consider using Xbox One Manager. It is a desktop application that allows you to manage users and test PLM events on your console as well as other common tasks.
Resources:
The goal of certification is to ensure a consistent experience for users and a quality bar for the Xbox One platform that publishers can count on. Having a good understanding of the certification process will improve the likelihood that you’ll be able to get through it without any holds. Remember the following when preparing your game for certification testing: