By: Miguel Guerrero, Andrew Farrier, Simon Cooke, Barry Bond, Brian Spanton, Tad Swift, Jonathan Morrison, Shawn Farkas, Jacob Haynes, Julia Guo, Charles Sanglimsuwan
Xbox Platform Team
Updated: September 20, 2016
Game consoles have historically been static hardware devices that have reliable behavior for the life of a given console generation. This model is changing, however, with cross-platform compatibility and mid-generation hardware revisions becoming the industry norm. These changes are making consoles act more like PCs in their feature sets. Games could inadvertently take dependencies on empirically derived platform behaviors that the platform does not always guarantee. This current model is a problem for creating games that work well across platforms and revisions. Worse, when these problems happen, they are often very hard to fix.
Platform abstraction and optimal performance can sometimes be conflicting goals. This is a problem that PC developers work with on a daily basis, but it is becoming a problem for console developers as well. This paper aims to give ideas for how to handle this conflict when platform abstraction is a priority. Going forward, operating system (OS) updates will continue to be released frequently. Hardware improvements in and beyond the current console generation will mean that games need to be prepared to run on a variety of platforms with different performance characteristics.
This paper will be updated periodically upon discovery of new issues.
Various consoles have different CPU speeds. A title must take these speed variations into account within its code in some form or another. This section covers the most common issues that arise from these differences, and recommends some solutions.
Time is one of the most difficult and confusing parts of a system because many programmers think linearly and locally with regard to a single thread of execution. However, time is a system-wide resource that is non-local and non-linear. Concretely, the following method is often used to measure the execution time of a thread.
LARGE_INTEGER StartTime;
LARGE_INTEGER StopTime;
LARGE_INTEGER ElapsedTime;
QueryPerformanceCounter(&StartTime); // Get the start time
// Code that is being measured here.
// Execute the code
QueryPerformanceCounter(&StopTime); // Get the stop time
// Calculate the elapsed time
ElapsedTime.QuadPart = StopTime.QuadPart – StartTime.QuadPart;
Most programmers assume that ElapsedTime will measure only the time taken to execute the measured code; however, this is not true. The time that the code takes to execute will be a portion of the ElapsedTime value; however, other higher priority system work that occurred (such as interrupts, DPCs, APCs, or context switches) will be included in the measurement as well. The bottom line is that in the previous example, the time measured and stored in ElapsedTime could be in the range of 1 to infinity. Synchronization activities should never use the value of ElapsedTime for reliable, dependable, and repeatable measurements.
CPU throughput is defined as how many instructions per second title threads retire. This can be affected by physical CPU architecture including ALU and FPU performance, pipelining, clock speed, power stepping, bus architecture, cache semantics, cache layout, CPU virtualization, and sharing with other processes.
Constrained mode is when the Xbox One console reduces the system resources available to the running title. This can mean losing exclusive access to CPUs, which in turn means that the title will have less CPU throughput, as well as changed relative timings. Even worse, the resource reduction may not be uniform per core.
Constrained mode can cause the following issues in certain titles:
A multiplayer host operating at half-speed, causing all the session peers to observe a similar drop in performance.
Starvation of video, causing drastic frame rate reduction, up to 50%.
Starvation of audio, causing extreme stutter issues.
A title should be prepared for this. It will receive OS notifications when being transitioned in and out of constrained mode. When in constrained mode, common design tactics might be to pause the title or reduce game fidelity.
Perhaps the most important consideration is how the title’s multiplayer sessions behave when a host or peer is constrained. The title must deliberately ensure that its constrained mode behavior does not become a tool for malicious players. If the title can’t maintain multiplayer fidelity when constrained, consider how to ensure that the constrained peer is the only affected party.
Create test cases that frequently switch between constrained and unconstrained modes in each game mode. On Xbox One, this can easily be done manually by pressing the Home button, but can also programmatically enter and exit constrained. For more information, see the white paper, Getting PLM Right the Second Time, and the sample, Simple PLM, on the Xbox One Samples page.
Titles may have access to many measurements that may be useful for game timing, such as QueryPerformanceCounter, __rdtscp, GetTickCount, GetThreadTimes, timeGetTime, and vblank (vertical blanking interrupt). All of these sources measure something, but have different guarantees about accuracy, precision, and consistency. Some may skip or freeze. Some continue to run when the title is paused in the debugger—others do not. In reality, these measurements and what the title requires may not actually be “time” in the classic sense.
For example, some titles have coupled world simulation to empirical vblank interrupts, without configuring vblank to a fixed and reliable interval.
You can set up events that will be signaled when vblanks or line-interrupts occur by using the DXGIXSetVLineNotification function.
Note This link goes to a specific release of the XDK (mar16.aspx). To get the latest information for this function when the XDK is updated, replace the current date in the URL with the new XDK release date.
Vblank configured to 30 or 60 Hz will reliably average that frequency over time, but instantaneous intervals may vary. QueryPerformanceCounter attempts to use the CPU Time Stamp Counter (TSC), which synchronizes across all of the cores and has very high resolution. If it can’t synchronize across all of the cores, it will resort to a lower resolution source for time. Using QueryPerformanceFrequency will provide the accuracy of the counter. GetTickCount reports milliseconds since the OS booted up, but its value is not continuously updated every millisecond and can wrap in a reasonably short amount of time.
For more information, see Acquiring high-resolution time stamps.
There was a real world case where a title was using empirical measurements and made an assumption for timing to lock the title to 30 frames per second. When the title was running on a newer generation of hardware, it ran faster, which broke the assumption. The effect was that the title ended up running between 30 and 45 frames per second, creating a much choppier feel. The title should have used a hardware-based timer such as vblank to lock to 30 frames per second.
Generally good advice:
Never assume that any clock has a predictable starting value.
Never assume that between any two clock measurements the title was running the entire time.
No matter how “close” in code two clock measurements are, theoretically any amount of real time may have passed.
Do not hard code the clock frequency. Use QueryPerformanceFrequency and cache the result.
Add code to compensate for clock changes to all system event handlers.
Read the documentation carefully for any time source.
Be careful when using two different sources of time in different parts of the title, especially when using one for video and one for audio and you expect both to remain synchronized. Be careful also when transitioning to/from constrained mode, and even more so in 7-core ERAs. Recent XDK changes have improved GetTickCount accuracy when in constrained mode; however, the accuracy can still change when the title is constrained.
Newer hardware platforms often have more cores per CPU socket and more CPU sockets per console. While it can be advantageous to carefully place threads on specific cores, the best practice is to split work into as many logical threads as can be reasonably synchronized. For work requiring no synchronization, a thread pool setup based on the number of cores detected can be very helpful. Generally, be prepared to lay out threads across a variable number of cores to best utilize them for future hardware platforms.
For information about detecting core layout for Win32, see Getting Hardware Information.
For information about the Universal Windows Platform (UWP), see CPUSets for game development.
Title ports from PC to Xbox One provide real world cases. Titles were configured to run on either two or four cores. This worked well when running on a PC due to the fact that the average PC has fewer but faster processor cores than a console, which tends to have more but slower processor cores. When the title moved to Xbox One, it did not expand to fill all of the cores and took a serious hit in performance. The solution is to make sure that the task system can expand to all of the cores and reliably keep them all working.
Titles should test against a variable number of processor cores. The best way to do this is to test the title on all available console generations with different core counts. A second method is by using SetProcessAffinityMask to restrict the title to a reduced set of cores. If a PC build of the title is available, testing against multiple machine configurations can be very valuable. Currently consumer level PCs are available with up to eight processor cores.
Interlocked operations can help prevent race conditions between CPU cores, but can have drastic differences between CPUs. The overuse of interlocked operations, such as thread-safe reference counting, can have performance impacts on a title. Cases have been seen of up to 1 millisecond total cost per frame.
On some systems, the use of an interlock can stall the GPU because the interlock requires locking the entire bus. Interlocked operations on some CPU architectures also require flushing any pending writes flushed to DRAM before they complete, again stalling the system.
Many titles tend to take dependencies on the precise results of a floating point calculation. However, floating point comparisons should use a small epsilon to allow for minor variations in final floating point bit patterns as hardware changes. Alternatively, use fixed-point math where necessary to ensure accuracy (for example, for the player world location sent over the network).
As an example, a title was making a binary decision based on a value read out of a texture. On one piece of hardware, a value of 128 was sampled as 0.50196840 (hex 0x3f008100), so the code tests whether the sampled value is <= that number stored in a constant. On a newer generation of hardware, the sample value is 128 as 0.50196844 (0x3f008081), which inverts the logic.
These floating-point calculation differences apply to cross-platform multiplayer environments as well. The same calculation may produce different results depending on the CPU model, GPU model, in-family revisions of those over time, and in-model revisions over time.
Memory throughput will always vary in a shared system such as a game console. Therefore, it is a bad idea to design around memory access times. Some of the things that can impact the throughput of memory include:
Contention from other components in the system, such as:
Other CPU cores
The GPU
Other devices
Other processes
Temperature
Cache design
Disk fragmentation – page faults
The speed of light
The ether
A lot of issues show up when a title runs on hardware with different timings within their threading models. The issues run from dead locks to intermittent stalls to crashes. The primary cause is usually improper synchronization between threads and an assumption on timings between actions.
Important There is no guarantee on the order of execution between threads unless the title code specifically enforces it.
Future hardware may have different timings between its CPU cores. The difference could be time slicing with other virtual machines or just other threads sharing the cores.
Take for example the following piece of code seen in a real world title.
void MainThreadProc()
{
WorkerData *data;
CreateThread(nullptr, 0, &WorkerThreadProc, data, 0, nullptr);
data.workType = work1;
}
DWORD WorkerThreadProc(void *param)
{
WorkerData *data = (WorkerData *)param;
switch (data.workType)
{
case work1: DoWork1();
case work2: DoWork2();
default: Crash();
}
return 0;
}
The title is modifying a data structure passed to another thread after the call to CreateThread. In this case, the title was relying on empirical data that the new thread would not use the data before the first thread had a chance to initialize it, which is an incorrect assumption. The solution in this case is to initialize the data before the call to CreateThread. Rigorous code review and using static analysis tools such as those built into Visual Studio can be very helpful in catching the issues before they make it into the source code.
A common problem seen in titles is a lock convoy where small grained work requires sequential execution. The issue shows up when each of these pieces of work is running on different threads that block waiting for the completion of dependent pieces of work. If one of the tasks happens to take longer than normal, the entire system stalls waiting for a lock.
Take the following piece of code seen in a real world title.
void TaskA()
{
DoWorkA();
SetEvent(WorkADone);
}
void TaskB()
{
WaitForSingleObject(WorkADone, INFINITE);
DoWorkB();
SetEvent(WorkBDone);
}
void TaskC()
{
WaitForSingleObject(WorkBDone, INFINITE);
DoWorkC();
SetEvent(WorkCDone);
}
In this case, task C was waiting on task B, which was waiting on task A. Task A ends up taking slightly longer to execute, which causes task B to suspend until A is finally done. This delay then cascades down to task C. In this case, what should have taken only several microseconds to execute ends up taking over a millisecond due to the threads constantly suspending and resuming waiting for the previous task to complete.
The solution to the problem is to avoid fine-grained lock convoys on tasks. When tasks depend on one another, moving the lock to a larger, higher level task often lowers the locking overhead and results in smaller impact on the overall execution time.
Another solution is to implement time outs when waiting for dependent tasks to execute. If the system times out waiting for a task to complete, move on to other work and check again next frame. This only works for tasks the current frame does not need, for example, loading of objects off the disk.
The cost for using an OS locking primitive is not always cheap if the code is required to transition to the kernel. In this case the cost is a minimum of 10 microseconds. The main issue is that this cost could change in future generations of hardware and operating systems.
Take the following piece of code seen in a real world title.
void JobCreator()
{
for(;;)
{
InsertJobIntoQueue(newJob);
DoWorkFor4Microseconds();
}
}
void JobConsumer()
{
for (;;)
{
newJob = WaitForJobInQueue();
newJob->PerformWorkFor5Microseconds();
}
}
In this case, the time to execute each task is about 5 microseconds, and the time between adding new tasks to the job queue is 4 microseconds. Based on empirical evidence, the job thread never needs to suspend because the job queue receives new tasks faster than they can execute. However, when moving to new hardware, the time to execute a task reduces to 3 microseconds. In this case the job thread is constantly suspending waiting on work to perform. This increases the cost for each task from 3 microseconds to a minimum of 13 microseconds on what should be faster hardware.
The solution to the problem is to create larger tasks for the job system or to insert tasks faster into the job system. Creating jobs that take several hundred microseconds to execute will give enough room for many future generations of hardware. However, a better solution is to create a batch of jobs and insert them all at once into the job system. In the second case any possible stall is going to happen between batches and have a much smaller effect on the overall system.
Most titles need some kind of mutual exclusion protection within various areas of their code. This is to protect against several threads trying to access the same pieces of data at the same time and ending up with some form of corruption. The problem comes into play when these are used too heavily, resulting in thread stalls. A change in timing for just one piece of code ends up affecting all other threads that are using the same locking primitive.
One area that has shown up many times is in memory allocation. Many memory allocation schemes provide multi-thread protection by using a CRITICAL_SECTION around all allocations and deallocations of memory. This can cause heavy contention on that one CRITICAL_SECTION, and during times of heavy heap usage the entire title effectively becomes single-threaded.
Another area that this has shown up in frequently is within physics systems. Physics tends to access at least the location of objects within the world frequently. When there is only one shared locking primitive used by all of the objects, this causes a large amount of contention and frequent stalls between threads.
In both of these cases, as the number of threads increases based on the detected number of available processor cores, the amount of contention increases. This can frequently cause the title to run slower even though it should be executing faster as threads are constantly suspending waiting to acquire the locking primitive.
There are several solutions to the problem. The overall goal is to significantly reduce the amount of locking and contention, not remove it entirely.
Make sure that the use of locks is protecting the minimum amount of code possible. Hold the lock for the minimum amount of time to avoid contention.
Switch to lockless implementations whenever possible. For example, in the memory allocation case, there may be a small cache per thread that does not require protection. Taking the locking primitive is only required when the cache needs to be refreshed from the master shared pool.
Proper use of reader/writer locks. In the physics case, most of the access is only to read the data for the object; only one thread needs to actually update the data. Only the update needs protection. This is the prime usage case for a reader/writer lock, which will drastically reduce the amount of contention.
Consider using algorithms that treat input data as read-only, writing results to a secondary region of memory. This write-only region then becomes the input for the next round of simulation. All threads can read the data simultaneously without needing the lock—apportioning the input data appropriately between rounds allows this.
A common theme seen in a lot of titles is the use of thread synchronization objects created in house: the “not invented here” train of thought. There seems to be two prime motivators for this train of thought. The first is the attempt to create a faster implementation, and the second is to create code that multiple platforms easily share.
The major problem with hand-rolled synchronization methods is that it’s very difficult to implement them correctly and fairly. There are a large number of empirically discovered, subtle race conditions that might only show up once a month in production. Any change in the underlying timing at either the OS or hardware level can cause these issues to show up, but only in the retail environment. The cost to communicate between processor cores and processor modules can easily introduce other issues.
The moral of the story is to use the operating system locking primitives provided. There are several optimized for maximum performance, for example a CRITICAL_SECTION with a proper spin count. The platform primitives receive a lot of testing through the entire Windows ecosystem, easily billions and billions of hours.
The GPUs in various consoles have different speeds as well as a different number of compute units. A title must take these variations into account within its code in some form or another. This section covers the most common issues that arise from these differences, and recommends some solutions.
Implementing dynamic resolution in your titles is the primary tool for scaling across GPU hardware.
PC titles have long supported multiple resolutions for their main display, expecting the user to adjust the display settings based on the wide variety of PC hardware. More recently, console titles have implemented a technique called dynamic resolution. Dynamic resolution attempts to present the best resolution available, and to preserve frame rate, the title will downgrade or upgrade the title's resolution automatically as the title becomes more or less GPU bound.
A prerequisite for both these approaches is to remove assumptions about resolution throughout the graphics pipeline. This includes:
Removing hardcoded resolution magic numbers in the title's C++ and HLSL code.
Not making assumptions about size or divisibility in shaders; that is, fixed width kernels.
Furthermore, for dynamic resolution:
Don't assume that the previous frame buffer was the same size. Use normalized texture coordinates for sampling, or if using Load(), keep track of the previous resolution.
Set a minimum and maximum resolution, create buffers at maximum resolution, and alias smaller buffers into that memory space.
On newer, more powerful console hardware, a carefully designed dynamic resolution system would allow the developer to simply add a set of higher resolutions available to the title to display.
For more information about implementing dynamic resolution, see the Xfest presentation Dynamic Resolution and Interlaced Rendering (pptx) and video.
Like the CPU, the graphics hardware is a set of parallel processing units. The number of these units can change some expectations and behavior for titles.
In one case, a title was making an assumption on the number of available processing units. It was performing an overlapped memmove operation and using a constant as the offset for the move operation between threads. On a newer generation of hardware, more threads ended up running due to more processing units. This caused corruption during the move operation due to the fixed offset being used, causing memory overwrites. The intention was to improve performance, but it came at the cost of compatibility.
There are several classes of data hazard with Compute:
A sequence of Dispatch calls with data hazards between calls. Stock D3D11 detects the hazards for you and inserts synchronization between calls. D3D11.X allows the title to opt out and do its own hazard detection. Stock D3D12 and D3D12.X both handle synchronization for you, but it's the title's responsibility to insert ResourceBarrier calls to signal the hazards. The debug layer may detect failure to insert barriers.
Multiple Dispatch calls running concurrently on parallel engines (graphics and async compute). D3D12[.X] expects the title to signal resource barriers here, and the debug layer may detect conflicts as in case 1. D3D11.X leaves the responsibility to the title.
Different resources aliasing the same memory. D3D12[.X] has resource barriers for this purpose. Stock D3D11 doesn't allow aliasing. D3D11.X leaves synchronization to the title.
A single Dispatch call with data hazards between threads. Neither stock D3D nor Xbox D3D can help you here. It's the title's responsibility to avoid conflicts, either by removing the hazards or by correct use of HLSL synchronization primitives and atomics.
A few things to watch out for:
Assume any Dispatches can run concurrently, even if they are far apart in execution order, unless you insert a barrier.
Don't assume that waves have a particular number of threads (exception: we will need to support such an assumption on the console because important optimizations depend on it).
Don't assume any limit on the number of compute units (CUs) on the machine.
Assume that different engines (graphics and async compute and dma) can run in arbitrary overlapping order, unless you manually synchronize them.
Run with the debug layer (also known as the validated driver) on PC and Xbox, and make sure you run clean, or else understand the errors and warnings you disable.
By calling Present on the swap-chain with a value of 1 or greater for the SyncInterval parameter, the frame-buffer flipping can use the vblank interval to synchronize, eliminating tearing.
On Xbox consoles, always use a value of least 1 unless the rendering is highly over-budget and this would add extra latency to frame rendering. This will prevent screen tearing when the title is executing on faster generations of hardware.
A better solution is to tailor the value for the SyncInterval parameter based on the time needed to prepare the frame. If the current frame is running over budget, use a value of 0 to present the frame immediately. If the current frame is under budget, use a value of at least 1 to wait until the next vblank.
For more information, see the Xfest 2014 talk Frame Buffer Afterlife and the white paper Presentation Queue and Display Planes on Xbox One.
Latency in GPU is the time it takes an operation to complete or retrieve a result. In particular, a title was relying on a visibility query result to return in time for the physics engine to update. There is no guarantee on the order or when the compute work will execute. This creates extra latency depending on the exact piece of hardware in use and the resulting camera jitter. The solution is to avoid tight coupling between GPU and CPU work that latency can affect.
Avoid making empirical conclusions about memory and GPU timing, especially if it leads to a belief that explicit synchronization is not needed. There is no guarantee on the speed of memory access. Even different compute units could have different L1 caches that will affect latency of tasks. Always use the correct synchronization primitives to determine when the GPU completes work. For more information, see the Xfest 2015 talk Demystifying Xbox One GPU Synchronization.
How a title loads their data can have a drastic effect on how the user perceives the title. If the user has to wait several minutes before they can start playing, they are less likely to have a positive experience. Depending on how your title performs I/O, there can be significant changes in these timings across multiple console generations.
The performance differences between rotational hard drives, optical media, and SSDs is well known. What is not as well-known is exactly what type of media your title may be running on for the user. Maybe the title must run directly off the optical media or maybe the user has replaced their rotational disk with an SSD.
An issue commonly seen in real world titles is to reuse the loading code used on previous generations of hardware that was optimized for reads from optical media. It would break up a read into specific block sizes and make assumptions on timing tolerance between reads to minimize latency. When this code moved to a rotational hard drive, the timing tolerance between reads became much smaller. This meant that the hard drive head was slightly out of position and the title had to wait for a full rotation of the hard drive platter for every single read. This increased loading time by an order of magnitude on what should have been much faster hardware.
The solution is to increase the block size to the maximum needed to be read at that time. This will reduce the number of read requests and thus reduce the overall latency cost because it is per read request.
By its nature I/O performance is slower than main memory performance in all cases. This means if I/O operations performed synchronously within a frame, the frame has a chance to stall. The issue may not be noticed on a developer console if an SSD is being used for development. However, in retail, the user may be using a rotational disk with drastically longer I/O times.
A real work example came during video playback in a specific title. The title was loading and processing the data for a frame using an asynchronous double buffered system. This was so only a full frame of video was rendered. This worked fine in most cases during cut-scenes; however, it failed during level loading.
The title assumed I/O performance that allowed the render thread to also process level load I/O requests. During video playback, it was mostly idle and had enough free time for this. Differences in I/O performance on different generations of hardware caused the level loading to take longer and the resulting video playback to stutter. The level loads turned the asynchronous video playback into a synchronous operation as the render thread stalled waiting for data.
The solution is to treat all I/O requests as asynchronous operations and to not bind them to any frame dependent work.
There are two ways to perform I/O operations:
Blocking – Performs one read operation at a time, waiting for each read to finish before requesting another read. This is very common in cross-platform code that uses the standard C library functions such as fopen, fread, fclose.
Overlapped – Performs multiple read requests at the same time, allowing the underlying OS and hardware to reorder the requests for maximum performance.
Many titles are written to be able to handle asynchronous read requests. However, due to the title calling blocking APIs, this capability is not utilized.
The title pushes requests into a pending queue that sorts based on location within the title’s package. The OS receives these requests one at a time as opposed to as many as possible at the same time. It’s making an assumption based on the physical layout of data on the storage media, the performance characteristics of the storage media, and the time needed to access each piece of data.
Each of the assumptions made will be wrong on even a slightly different piece of hardware. There may be fragmentation on the hard drive, or the rotational speed of the hard drive could be different. The only system that knows the actual location of data in relation to the read head is the hardware itself. Using OVERLAPPED I/O with the maximum number of requests possible allows the hardware to reorder the requests to match the underlying layout. This creates the minimum amount of time needed to load data across all generations of hardware.
For information about the relative performance costs between I/O operations, see the white paper Maximizing File Performance.
The title is the only system creating various title specific files during execution. This means it’s possible to cache data about what files are available.
One specific case seen in a real world title was the enumeration of saved games for the user. Each time the user requested a saved game, the title would enumerate all of the current save game files to create a new unique name. Differences in file systems between platforms caused this time to jump from 200 milliseconds to 2000 milliseconds. Because this was happening every time a save happened, the user experienced a stall.
The solution was to cache this data once during title startup. Because the title is the system saving these types of files, they couldn’t change during title execution. When creating a new save, that name could be added to the cache.
All of the I/O functions return some form of value for the actual amount of data read. If the data is not available, the title may not get back the amount of data requested during an operation.
Assumptions on the way asynchronous operations work has been seen in retail titles. In one specific case, a title was making an assumption that at least four bytes were available in a buffer for the first chunk of an asynchronous operation. The title was ignoring the return value on the read request and ended up with corruption when an OS change broke this assumption.
Though not directly related to I/O performance, cases have been seen with titles mixing binary and text mode. This resulted in the unfortunate case where binary data was changed by the text parser, for example seeing 0x1A, which maps to /r. The binary data was supposed to be the actual size of the save data. The title was not checking the return value from a read request, which resulted in random corruption and reading off the end of a buffer.
Activation and connection time outs can vary drastically in the retail environment, and will likely change in the future. Be sure to test all Xbox Live API calls under varying conditions, and be loose in your requirements for time outs.
Avoid testing solely on internal LAN networks, to make sure your title can handle different network topologies, bandwidths and latencies. Also try adding varying levels of packet-loss and delays to your network. The preferred method is through hardware; for example, some routers support simulating varying network conditions. Alternatively, adding code to the title’s network layer for simulation works as well.
Perform testing with a realistic number of friends associated with all test accounts - at least 50 for each account that is connected to a multiplayer session. Many cases have been seen that only show up in the retail environment due to this lack of testing. The title will stall or stutter for multiple frames as it processes all of the friends a player may have.
The minimum supported network environment for Xbox consoles that support multiplayer is:
192 Kbps download bandwidth
192 Kbps upload bandwidth
1% packet loss
75 ms round-trip latency
… but a title may experience worse in the real world.
In 2016, fine grained rate limiting was enabled for all new titles connecting to Xbox Live. All service calls to Xbox Live are subject to this policy, and is applied on a per title, per user model. Rate limited service calls receive an HTTP 429 response with a JSON body that includes how much time the title needs to wait until it can retry. The Xbox Live SDK includes a Fiddler plug-in called the Xbox Live Resiliency Tool, which titles can use to help manually inject accurate Xbox Live error responses back to their game clients. We encourage you to use this tool to test against a wide range of micro-outage scenarios and rate-limited service calls.
The Xbox Live SDK also includes the Live Trace Analyzer tool, which analyzes network captures made from xbtrace.exe or Fiddler to identify suboptimal calling patterns to Xbox Live. Run Live Trace Analyzer prior to submitting to certification, to increase service resiliency and improve network performance of your titles.
For more information, see the white paper Service Interruption Resiliency for Titles.
Contact your account manager for details on running an Alpha or Beta of the title, which can help discover these issues before releasing the title to retail.
Various consoles use different hardware to create audio, and in some cases audio can also be driven by the CPU. A title must take these variations into account within its code in some form or another. This section covers the most common issues that arise from these differences, and recommends some solutions.
The minimum size for audio packets accepted by APIs may be larger or smaller on different system generations. Attempts to synchronize gameplay to audio by making assumptions on the size of those packets can result in audio stutter. The OS may be queuing them up behind the scenes if the device requires larger packets. If the device requires smaller packets, the OS may split the packet and send it on in pieces.
Ideally, the title should query packet sizes from the audio APIs where possible. It can then adjust the packets sent to the audio hardware to avoid extra latency caused by the OS having to convert the packets to match the hardware.
Audio and video latency can vary from system to system, and by specific output device configuration and settings in the users’ home.
Future audio devices may have differing values for latency (from API call to audio out of the speaker) than current devices. Various video devices also have differing values for latency (from API call to actual display on the TV). Do not create hard-coded assumptions based on empirical evidence seen in testing.
If the title relies on tight correlation between user actions and audio/video output, it should consider advising the user to put their TV in game mode because this reduces end-to-end lag between rendering and actual display on the TV.
If the title is a rhythm game, it may need to determine the latency in the system through a calibration step in the title. Most music/rhythm games already provide a user-guided calibration system because this lag varies based on output device (TV/audio receiver). A calibration system should be able to take into account any changes in latency/effects of packet-size differences in future systems as well – provided that the title can support a broad enough window.
Update packets from controllers may have varying degrees of latency – up to roughly 1 frame, depending on the technology used and underlying infrastructure. Always try to poll the latest data from the controller in a single batch. If the title relies on exceptionally tight timing, provide a calibration step.
In games using mechanics like quick-time events, it may be useful to provide the user with an option to reduce the difficulty by increasing the duration of those events. This also helps older users of the title, who have slower reflexes.
When using APIs, it’s important to double check the parameters that the API uses regularly.
Many issues arise in apps and titles that made an assumption about what are legal parameters, and what aren’t.
This includes:
Allowed characters in file names
File name lengths
Whether or not spaces are valid in strings
Use of exotic characters in string parameters
Maximum and minimum allowed lengths of strings
Be sure to test with a variety of worst-case configurations, for example max GameDisplayName, ApplicationDisplayName, and gamertag length including foreign names.
Read the API documentation.
Observe proper max length on buffers to OS functions.
Verify proper use of MBCS, Unicode-16, Unicode-32, etc. Check which functions take which and what buffer size means on the API context.
The worst problems seen occur when people step outside of the published APIs. Always follow these rules.
Do not assume that opaque OS-structure layouts will remain the same.
Clear structures to 0 before use if the API says to do so, and set up any parameters (such as the length of the structure).
Do not use undocumented sources of API information; if it’s not on MSDN or in the XDK documentation, it’s not supported and may change in the future. Don’t use it.
Do compile your title using static code analysis. Microsoft headers are marked up using SAL annotations for better results from static code analysis. This will save a large amount of time and effort locating possible bugs. We also recommend the use of SAL annotations for title code if possible; the payoff in reduced bug fix time can be large.
After a title ships or updates anything, everything used to create the title or update should be archived. Game life cycles are often long and hard to predict; there are still versions of Galaga and Pac Man shipping today.
You must be ready both for simple debugging and bug fixes. You should also be ready to update on an existing platform or port your code to a new platform. Cases have been seen where widespread title issues, such as crashes in retail, could not be debugged because studios did not even archive symbols. They may have also only properly archived the RTM/Gold Master and not subsequent updates. Properly archiving everything needed to rebuild the title and get it back up and running protects investments. It also allows the ability to easily ship the title again on future devices.
To archive a title, at a minimum, save copies of the source code, assets, generated symbols, layout information, and all tools used to create a build.
To robustly archive your title, save the following:
ISOs for the OS you were using for development, and keys associated with them.
The build environment, including any IDE/compilers used, build tools, and instructions on use.
SDK and XDK versions used to build the project.
All middleware used to create the title.
All artist and design tools along with license information.
Supporting documentation and specifications for features.
An up-to-date image of the entire source tree.
A copy of the full build output as an image.
The revision control system, including all history.
Symbol files for all released builds.
The full bug database.
All art, audio, animation, and video assets in both uncooked and final cooked forms.
Source art and animation files.
At least two devkits for the platform released on, including controllers and power-supplies.
A working build PC, which is set up to create a releasable version of the title from beginning to end.
An image of the build PC.
All server source and assets that are used to run title services used.
Architectural documentation for all title services.
Copies of all license keys for third-party tools used.
If necessary, IP licenses, rights-assignments, contracts, and other legal documentation for tools, assets, code, and other resources shipped in your title. This will make it possible to sell or transfer any IP if desired.
When archiving files or transmitting files, ensure that at least MD5 hashes are generated along with time stamps for all files. This allows verification that they are not corrupt. Free tools are available online to help you with this. When access is needed to any archived files, verify the MD5 hashes first.
Ideally, keep three or more copies of everything in different physical locations. Finally, a digital archive should be created every time a new release is produced.
This looks onerous at first glance. However, cases have been seen where a title did not even archive symbol information for released builds. This had a drastic effect on the ability to fix bugs that only existed in the retail environment. In some cases, the title decided to not fix the bug due to the effort in locating the issue without matching symbols.
Automated test suites are incredibly valuable to the game development process. Automated test suites playing the game 24/7 is a valuable way to flush out hard-to-repro bugs, give an indication on breaking changes, and give a better idea of overall stability and performance. Having automation test targeting specific areas and features in a repeatable manner can help drop bug fixes from weeks to days or hours and prevent new, hard-to-fix bugs in the first place. Logging performance and creating ways to compare that performance over time and across all of your platforms allows the ability to gain a very good understanding of the overall health of the title and can give a better focus for optimizations.
Spending a man month early in the project to set up automated testing can easily save several man years near the end of the project trying to track down issues.
Testing with a variety of platforms or in a variety of conditions may add additional workload up front but pays off in the long run by exposing bugs earlier when they are cheaper and easier to fix. Setting a higher quality bar throughout development leads to more robust code that is easier to maintain and port in the future.
On Xbox One, tools and techniques are already in place to help you test more efficiently:
Running with xbstress.exe. See the Tools section.
Testing in constrained mode, with snapped apps, and while game-streaming to a PC.
Running from an external hard drive and SSDs.
Running with debug XDK components such as DirectX.
Adding perturbations to title code to expose timing issues:
Adding extra dummy load threads to various processors to change relative timings.
Adding random or fixed sleeps and spin waits in debug or test builds to ensure your title behaves when things take longer than expected.
Read about the cpufuzz tool in the Tools section.
Microsoft provides several tools to help you test your titles’ resiliency.
xbstress.exe—Stresses a variety of components. Test scenarios include:
Increased CPU load from the system
Network slowdowns or failures
Heavy disk uses
Heavy cache uses
System UI events
For usage details, see Stress (xbstress.exe) in the XDK documentation.
xbconfig.exe—Changes or retrieves settings on the development kit
cpufuzz.exe—Helps to detect race conditions by adjusting the timing between the title threads. This tool and its full source are available in the System section of the Samples page on Game Developer Network (GDN).
Hopefully the topics and examples discussed in this paper will create a better understanding about how to make titles more compatible with a variety of platforms both now and in the future. Keeping platform variance in mind will make games more robust even on a single platform. It’s also worth keeping in mind when writing middleware, engines, or internally reusable components.