Improving Iteration Times

By: Andrew Farrier, Xbox Advanced Technology Group

Published: August 23, 2017

Update: June 5, 2018

This document is confidential and provided to you under a Microsoft non-disclosure agreement. While we have tried to ensure the accuracy of this document, we provide no express warranties or guarantees regarding the information. The information is subject to change. Microsoft may have intellectual property rights in the subject matter of this paper. This document doesn’t grant you a license to those rights—it’s for informational purposes only.

Abstract

When you’ve changed only a couple of files, lengthy build times can seriously impact your productivity during development. This paper covers the various ways to improve iteration time—how long you wait between changing a file and being able to debug that change—by choosing different build options.

Introduction

The standard business application is typically made up of many smaller projects. Each of these smaller projects creates a DLL. Any change to the code requires an update to only that DLL; no other feature area needs to be touched by the build system.

In contrast, the typical game project is a single, large, monolithic solution: each individual project builds a library that’s statically linked into the final executable. Because of this, any change to a source file has the potential to cause a ripple effect across the entire code base. This is even more true with link-time code generation, where a large amount of optimization is pushed to the linker, which then processes optimizations across all the static libraries.

This paper covers a wide range of options that can affect build times, especially during daily iteration work. In particular, individual settings that are appropriate and necessary for a full build may have a negative impact on the more frequent, iterative builds that developers must run to see their changes to a small number of source files. Try out the various recommendations presented here to determine which work the best for you and your development scenarios.

How to use these recommendations

We present the recommendations here in several sections: Project setup, Compiler settings, Linker settings, Deployment, Debugging and Miscellaneous. Each section provides a range of options, ordered from the most impact on developer iteration times to the least.

Development machine hardware

It’s well known that the more powerful a developer’s machine is, the faster build times will be. A machine with a limited amount of memory constantly thrashes the hard drive as pages are swapped in and out. A machine with a limited number of available processor cores can’t handle as many source files simultaneously as a machine with more cores.

Minimum recommended machine specifications

Recommended machine specifications

(Note that as the number of cores increases, the increase in performance per core drops due to interdependencies.)

Project configurations

We recommend that you consider at least four different build configurations:

Recommended settings

Here are the recommended configuration settings to optimize turnaround time for iteration builds in a developer’s daily work. Individual sections in this paper provide more details about why these recommendations were chosen.

Project setup

One of the primary, compiler-related reasons for slow builds is the constant processing of header files for each source file within the project. It’s common to see one small source file (10-20 KB) expand into a huge, final preprocessed file (10-15 MB). If your build times are I/O bound, this is the primary reason. The options in this section all attempt to drastically reduce the time to preprocess headers.

Precompiled headers (PCH files)

Precompiled headers are the preprocessed versions of header files. A large number of headers are loaded and processed by the preprocessor, and the result is saved to a single file. This allows the compiler to block-load the entire data set into memory. This may create a time hit up front for creating the file, but it dramatically reduces the time to process a source file that uses the PCH file. If the development machine has enough memory, it’s quite conceivable the PCH file will sit in the cache. This means the compiler can load this data via memory copy.

The major limiting factor to using precompiled headers is that if any header file contained in the PCH file changes, the entire PCH file must be recreated. This means that each dependent source file must be rebuilt in this situation. Even though the source may not be dependent on that header file that changed, it is dependent on the PCH file itself.

The cost of rebuilding PCH files can prompt a conservative approach as to which files are preprocessed. One rule may be to include only system files—for example, XDK/SDK headers. However, following this rule drastically reduces, and in some cases nearly eliminates, the benefit gained by PCH files.

We recommend an aggressive approach. A good rule of thumb is to include any header file that changes less frequently than once a week or is used in at least 50% of the source files for the project. You can use the /showIncludes option with the compiler to see the full include tree for each source file. Consider a script that creates a score for each include file—for example, the number of references to a header file times the number of files it includes. Place all headers that score over a certain threshold into the PCH file. Test build performance with various thresholds for your project.

For daily iteration work, the compiler would not need to regenerate the PCH file or process any header file included in it. This means that the time to compile changed source files drops significantly. The PCH file would need to be regenerated only during a clean build.

Even if a developer was actively working on a header file that is included in the PCH the aggressive rule would still apply. A majority of the source files would need to be rebuilt whether the header in question was in the PCH file or not.

The math behind the improvements caused by PCH files is straight forward. Because the PCH is block loaded into memory the time to generate the PCH is almost exactly subtracted from the time to build each source file using the PCH.

Given:

N = Number of source files

T1 = Time to build PCH

T2 = Time to compile source file with PCH

T3 = Time to compile source file without PCH

In almost all cases T3≈T1+T2

This means the following formulas hold true

Time with PCH

Total Time = T1 + T2 * N

Time without PCH

Total Time = (T1 + T2) * N

As you can see as long as the same PCH is used for at least two source files the overall build time is improved. The dominating factor is entirely the number of source files using the same PCH.

Proper use of PCH files can easily create gains that are close to those delivered by several of the other ideas presented in this paper, yet with no side effects and very few negatives. The entire process is fully supported in the Visual Studio build environment. We recommend that you start your build-time improvement effort with proper PCH file creation, because this strategy also works with all the other approaches covered here. In many cases proper PCH creation can cut in half the time required to compile a title.

#pragma once

It’s essential to avoid including a header file multiple times within one source file compile. The primary reason for this is that multiple definitions for the same object result in a compile-time build failure, due to violations of the “One Definition Rule” in the C++ standard.

The traditional way to protect against multiple include files is to implement an include guard by using the #ifndef/#define/#endif idiom around the entire file. The first time the file is included, a unique define is created. Later includes of the same protected file are not processed. This code has the benefit of being portable across all compilers. However, it has the drawback of still requiring the file to be opened and processed each time the compiler sees a request for it.

The C++ standard has the #pragma keyword, which enables compilers to implement their own extensions. Visual Studio uses this to implement the keyword #pragma once for better include-file protection. When Visual Studio sees #pragma once, it adds the name of the header file to an internal list and ignores any future requests for the same file. This removes the need to open and process the file multiple times in one source file compilation.

This ability reduces build times so well that most compilers also support the #pragma once keyword. We recommend that all header files use the #pragma once keyword to speed up builds. You can also use #pragma once along with the #ifndef/#define/#endif guard if you want to.

Single Compilation Unit

There is no limit to the types of files that can be handled by using the #include keyword. This allows the creation of Single Compilation Unit builds: multiple source files are included within one source file that is then processed by the compiler as a single unit. The primary gain in compilation speed comes from reducing the processing of header files involved in compiling each source file individually. This has been shown to decrease build times significantly.

The optimizer can also perform optimizations across multiple source files at once. This can alleviate a large amount of the need for link-time code generation, where cross-file optimizations are performed by the linker (which can see all the source files). With Single Compilation Unit builds, the compiler sees the multiple source files as one, which enables it to perform many of the same optimizations.

There are several drawbacks to this approach:

These drawbacks can be addressed, but we recommend that you focus on creating proper precompiled header files before attempting Single Compilation Unit builds. PCH files are fully supported by the compiler and can give you similar performance gains if properly implemented.

Multi-processor project builds

Visual Studio supports building multiple projects in parallel. It starts multiple copies of msbuild, one for each project file. It does perform dependency analysis to guarantee that projects that depend on other projects are built in the correct order. Lowering the number of dependencies between projects will decrease build times because more projects can be built in parallel.

We recommend that you enable this feature to maximize the use of the development PC’s resources. For some very good advice on fine tuning, see Tuning C++ build parallelism in The Visual Studio Blog. (Even though the advice is from Visual Studio 2010, it’s all still valid.) This advice, along with using the Multi-Processor source builds option, will give you large performance gains in relation to iteration times.

IncrediBuild

IncrediBuild is a third-party product that distributes the build across multiple processor cores on the local machine and/or multiple machines in the network. Visual Studio includes a license for IncrediBuild when you use it in the local PC mode. This is a replacement for the Multi-Processor project builds and Multi-Processor source builds options provided by Visual Studio. In many cases, IncrediBuild provides faster iteration times than you get by using the native components of Visual Studio. All the options recommended in this paper will work with IncrediBuild. For more info, see Improving your build times with IncrediBuild.

We recommend that you test the included local agent option of IncrediBuild for daily iteration work to see how it affects your build times. However, also consider the recommendations in Distributed build systems when using IncrediBuild across multiple PCs.

Exceptions

The use of C++ exceptions has generated a lot of conversation over the years about their performance as compared with error codes. It’s important to understand the implementation in the x64 environment and their effect on performance. For detailed info, see Exception Handling (x64), and also the Code Generation for Xbox One Best Practices white paper.

Two factors affect run-time performance in relation to code execution speed: the general cost for any overhead with exceptions enabled, and the cost when an exception is fired. The x64 exception model moves all the exception cost to when the exception is fired by using frame data—descriptions of functions with data needed for stack unwinding.

We recommend keeping the default options for exceptions enabled (that is, /EHsc). This won’t affect build speeds; the only overhead is the addition of the frame data to the total executable size. If exceptions are not enabled, the frame data is missing, and that keeps the exception handler from performing proper stack unwinding. This means there will be no useful call stacks in any minidumps created.

DLL-based builds

One option for decreasing iteration time is to convert static libraries to DLLs. This can influence linker performance because the linker needs to consider only smaller import libraries. The biggest gain comes from not having to merge the PDB files from each individual library, but there are some drawbacks for title performance at run time.

The primary gain from using /DEBUG:FASTLINK (described later in the paper) is the removal of the final step in merging PDB files from all the static libraries. Each DLL has its own, unique PDB file that is fully supported by 100% of the development tool chain. This has been shown to reduce link times by up to 50% and is a good option if your tool chain does not support /DEBUG:FASTLINK.

There are some drawbacks to using DLL-based builds:

We recommend that you try converting static libraries to DLLs for daily developer iteration builds, maybe just for the code the developer is working on. The gains to the performance of the linker can outweigh the negatives mentioned. It’s important, though, to avoid the DLLs for a final optimized build, when the negatives can have a significant impact on final title performance.

Compiler settings

Compilation is one of the major steps during the build process and can be a significant source of iteration-time problems. Any time spent optimizing this step can deliver cascading performance improvements through the build process. Giving less code to the compiler and making more resources available can drastically improve the performance in the compilation step.

We recommend that you start with generating proper PCH files. This greatly reduces the amount of code the compiler needs to process for each individual source.

Multi-processor source builds

Each instance of the compiler is given a set of source files to build sequentially. However, the compiler supports the ability to build each of these source files in parallel across multiple processor cores. This ability is enabled by using the /MP option This causes the compiler to spawn multiple copies of itself, one for each source file up to the maximum number enabled on the command line.

You should test with different numbers for the number of parallel sources files that can be built simultaneously. With a large number there can be issues with resource contention on the computer which can slow down the entire build process. For example VirtualAlloc is required to always allocate space in the backing page file. With enough requests in flight it is possible for them to backup and eventually cause VirtualAlloc to timeout. The requests spend too long waiting for the underlying file system to process the request to grow the backing page file.

We recommend that you enable this feature along with multi-processor project builds, or that you use IncrediBuild for local builds. This allows the compiler to use the maximum resources of the development PC and can significantly reduce iteration times. For example, a four-core machine could take a quarter of the normal compilation time by working across all four cores simultaneously. For some very good advice on fine tuning, see Tuning C++ build parallelism in The Visual Studio Blog. (Even though the advice is from Visual Studio 2010, it’s all still valid.)

You should test with your particular title and hardware to find the sweet spot on the number of parallel builds. In general we recommend to use faster cores as opposed to more cores. This can reduce the overall contention on other parts of the machine such as the file system.

__forceinline

__forceinline is used as a hint to the optimizer that the heuristics for choosing when to inline a function should be ignored. For example, the heuristics may determine that inlining a function will make the code slower due to bloat. However, if the function is marked __forceinline it will still be inlined and the code could run slower. There are still certain cases where a function can’t be inlined—for example, when the function is called recursively or through a function pointer.

The problem is that indiscriminate use of __forceinline can lead to both dramatically longer build times and slower code execution. The problem lays in how Visual Studio treats the __forceinline directive. It is always honered if possible and does not affect choices around any other inline function expansion. The original source code is also placed directly at each call site. This increases options for the optimizer, however it also means a lot of work needs to be performaned multiple times.

One case, we saw indiscriminate use of __forceinline increase a function from 200 lines to 20,000 lines. This caused an increase in both the function source and the complexity of the code to be handled by the optimizer. Above a certain level of complexity, the optimizer must scale back the amount of work it performs on analysis. It can start to produce lower-quality optimizations, which leads to a lower-performing final executable. You can test if this is happening in your title with the /we4883 compiler option, this will output a warning whenever the optimizer starts to scale back due to function size. You can force the optimizer to optimize these huge functions with the /d2OptimizeHugeFunctions compiler option.

For another example of how indiscrimite usage can affect code execution time. Consider a function that is __forceincline several times within one function. This requires multiple copies of the code in each section. During execution each of those copies needs to be loaded into the instruction cache with the possibility of evicting other code. If the function was not inlined then it would be loaded into the instruction cache once and evict less code. The optimizer will create the minimum amount of code needed for the function call, even skipping register saving if it knows the called function doesn’t modify them. Since the jump is to a constant address the branch predictor will be 100% accurate helping keep the instruction pipeline from stalling.

We recommended that you avoid the use of __forceinline except in carefully controlled and profiled locations. The problems just mentioned can cause both serious expense at build time and poorer performance at run time. These performance problems increase the more often the inlined function is called. Allow the optimizer to perform its analysis and determine when to inline and when not to inline.

Streamline PDB Generation

A new feature was added in the 15.3 update of VS2017 that streamlines PDB generation. Previously there was a lot of traffic between the compiler and the PDB service that used a significant amount of time. A large amount of work was performed to streamline the traffic which has been shown to reduce full build times on average by 25%.

It’s recommened that you enable this option with the /Zf flag for the compiler. This works in conjunction with the other debug related flags that require PDB creation during compile time. However, it does not have any effect when the /Z7 flag is being used which adds the symbol information directly to the object files.

Starting in the 15.7 update of VS2017 this option has been enabled by default.

Minimal rebuild

The compiler offers the Enable Minimal Rebuild (/Gm) option to increase compiler performance. The option works by keeping a database of all the code affected by any class definition. The compiler rebuilds only code that is dependent on a class where the definition changes. This can provide a significant performance boost with builds that use PCH files. Even if the PCH file needs to be updated, only the code that is directly affected will be recompiled.

The minimal rebuild option does have one significant drawback: it doesn’t work with the multi-processor source build options. If many source files need to be updated, they can be built only one at a time and sequentially. The build system can still use multi-processor project builds, though, to enable multiple projects to be built in parallel.

We recommend that at least you profile whether this option helps with iteration times. In most scenarios, though, the multi-processor source build or IncrediBuild options will give better performance on a modern development PC.

Linker settings

When you make minor changes to source files, the time taken by the linker will be the dominating factor. The primary factor in linker performance is the amount of data it needs to process. Five settings have the most effect on this: /DEBUG:FASTLINK, /MAP, Incremental Linking, Link Time Code Generation, and Function level linking.[]{#_Function_level_linking .anchor}

/DEBUG:FASTLINK

Debug information is stored within a Program Database (PDB) file. The linker creates one PDB file for each library used by the title. At the end of the link step, each of the individual library PDB files is merged into one final title PDB file. This merging step can account for a large amount of the link time, in some cases up to 50%.

With the /DEBUG:FASTLINK option, the merge step now creates a final title PDB that has references only to the individual library and object files. The debugger and other Microsoft tools understand these references and can load information out of the subfiles. This works great for daily iteration work during development because it greatly reduces build time and still allows you to debug your changes. If your tools are built using the Debug Interface Access (DIA) SDK then they will work with FASTLINK-enabled symbols as well.

However, this option should not be used in nightly builds that are used in play testing and content production. The problem is that all the build artifacts must be present to resolve symbols, because the master PDB file contains references back to the individual object files.

Problems have been seen when using FASTLINK-enabled builds and some distributed build systems. The problem is that the final object files must be present on the developers machine while debugging. Some distributed build systems do not copy these object files back to the original machine. You should test with your particular distributed build setup.

Previously issues have been seen when using FASTLINK-enabled symbols and VS2015. The primary problem being increased memory usage that led to instablility, slow performance, and correctness issues. These issues have been addressed with recent versions of VS2017. Memory usage during debugging has been brought back down to the same level as full PDBs which has led to increased performance and stability. The correctness issues have also been addressed.

If you need for a final, single, monolithic PDB file for a FASTLINK-enabled build, it’s possible to create one from either the command line or the Visual Studio 2017 IDE. From the command line, use the mspdbcmf.exe executable. From the Visual Studio 2017 IDE there is the Build full program database file option in the build menu.

/MAP

The /MAP option creates a text file that contains a full list of symbols, along with their locations within the final executable and memory. For example, some titles use this file to look up the name of a function, given a specific address, to aid in debugging. However, creating this file is very expensive, in many cases accounting for up to 50% of the overall link time.

We recommend removing this option from daily iteration builds run by developers. The debugger will already have all the necessary information from the PDB files. Enable the /MAP option only in builds where this information is strictly necessary.

Work was performed in the VS2017 15.6 update to help with map file creation. Previously map files were created sequentially with the rest of the link step. In the 15.6 update this was changed to build the map in parallel with the rest of the link step. However even with this change there is still the possibility of map file creation being a bottleneck when implementing the other changes recommended in this paper.

Build-speed improvements that involve excluding MAP file output will affect some in-game symbol-resolution solutions that depend upon this data to resolve stack data on the console. xbSymbolProxy.exe can aid in off-console symbol resolution, which directly consumes PDBs on the developer machine or server and thus avoids the need to transport symbol information to the console. In-game code can directly consume these APIs in the Title OS without additional work or file transport. For more information about xbSymbolProxy.exe, see the XDK documentation.

Incremental linking

Incremental linking is the ability of the linker and compiler to update only the changed functions within a build. The linker doesn’t need to touch any unchanged functions. This reduces the amount of fix-up work the linker needs to perform and thus helps reduce iteration times.

The only drawback to incremental linking is that all function calls go through a jump table. In theory, this could affect title performance. However, the branch predictor will be 100% correct since the destination of the jump is always the same. The CPU can still prefetch and keep the instruction pipe filled.

We recommend that you enable incremental linking to improve daily iteration times.

Link-time code generation

Compilers have a front end that converts the title source code to an intermediate representation. A back end then converts this intermediate representation to the final machine code. Each phase can perform its own optimization passes. Link-time code generation moves the back end from the compiler to the linker, which allows for a wider range of optimizations. For example:

The major drawback to link-time code generation is that it can increase build times, in some cases by a significant amount. However, it can also increase title performance significantly.

There is the option to use incremental link-time code generation. The optimizer recompiles only he affected functions from the changes since the last build. It doesn’t attempt to reoptimize the entire title. This can create a significant performance boost during the link phase. For details, see Speeding up the Incremental Developer Build Scenario in the Visual C++ Team Blog.

Each of the updates to VS2017 have increased the performance of link time code generation and it’s ability to multi-thread. This has be done through creating finer grained locks between threads. It’s possible to control the number of threads link time code generation uses with the /cgthreads[#] option. The default is to use 4 threads, however it can be set to use up to 32 threads.

We recommend enabling full link-time code generation only for nightly builds used by content generation and play tests. Take a look at using incremental link time code generation for daily iteration work by developers. It will add some time to builds, however it’s possible to perform profiling with incremental link-time code generation. This can provide minimal daily iteration time cost when working on final optimization work by getting closer to the execution speed of a fully optimized build. We also recommend trying different values for /cgthreads[#] to find the best value for your particular title.

Function-level linking

Function-level linking is the removal from the final executable of all functions that are not called. This option is accessed primarily via the -Gy flag and the /OPT:REF flag. Removing unreferenced functions can lead to a smaller final executable size, a reduction in the amount of code the linker needs to process, and a reduction in the memory footprint of your title.

The primary usage for this option is to locate “cruft” in the code base. This is extra code that has collected over time as the engine changed over the years. This is code that is still using up time being compiled but then thrown away during the link step. If this code was removed from the codebase the overall build times would improve.

The use of the /verbose option to the linker will output a list of functions and data that was discarded. You should review this list and determine if these can be deleted from the codebase without affecting any of the build configurations.

Even though the linker has to perform extra work to remove the unreferenced function we recommend leaving the option enabled. Use the output from the /verbose option to locate possible code and data that can be removed from the project to help overall build times.

COMDAT folding

COMDAT folding is the ability for the optimizer to merge duplicate code blocks into one block. It is enabled by using the /OPT:ICF option to the linker. The most common candidates for merging are template functions that end up generating the same code. For example, take the function std::vector<>::push_back. This function for std::vector<classA *> and std::vector<classB *> will generate the same code. They each append a 64-bit pointer to the end of the std::vector instance. Their generated code will be identical between the two versions. Without COMDAT folding there would be two versions of the function, amounting to code bloat.

There are are several trade-offs that need to be considered on whether to enable or disable this option with your code base. Enabling can result in a decreased executable size by removing code bloat. However this can lead to problems when debugging around folded functions, only one version of the function exists in the executable and this is the only function where a breakpoint can be set.

The use of the /verbose option to the linker will output a list of folded functions. The selected version to remain and the versions that were replaced. You can use this information if needed when debugging around the functions that were affected by comdat folding.

The linker must do extra work to determine which functions are duplicates of other functions. We recommended choosing whether to enable or disable the option based on how it will affect the developers current iteration work.

Test numbers

Some numbers were collected from an internal title to show the effect of the various options on overall link times. You can clearly see the advantage to using the recommended options, which achieve a significant reduction in build times. Numbers are included for both the 15.5 update and the 15.6 update, this is due to the change in the 15.6 update to build map files in parallel with the other link steps.

VS2017 15.5

Options Time in seconds
/DEBUG:FULL /OPT:ICF /MAP:<path> 118
/DEBUG:FULL /OPT:NOICF /MAP:<path> 107
/DEBUG:FASTLINK /OPT:ICF /MAP:<path> 68
/DEBUG:FULL /OPT:ICF 87
/DEBUG:FASTLINK /OPT:NOICF 32

VS2017 15.6

Options Time in seconds
/DEBUG:FULL /OPT:ICF /MAP:<path> 91
/DEBUG:FULL /OPT:NOICF /MAP:<path> 80
/DEBUG:FASTLINK /OPT:ICF /MAP:<path> 47
/DEBUG:FULL /OPT:ICF 91
/DEBUG:FASTLINK /OPT:NOICF 34

Deployment

After the title has finished building, it must be deployed to the Xbox development hardware for testing. There are several options, and each one has an overall effect on daily iteration times.

Push

Push deployment was the original deployment scheme used since Xbox One launch. In this case, all the files for a title are copied to the console before it can be executed. The system was smart enough to copy over only the changed files. However, because all changed files had to be copied, the system could copy over files that were not accessed in daily iteration—for example, content that changed, but that the developer is not loading.

We recommend using Run from PC. It delivers better performance on average for daily iteration work because it copies only the files that are being accessed by the title.

Pull

Pull deployment is an older system in which the console would request files from the PC as they were accessed. It worked by means of a special driver on the PC. This system had several major issues; it has since been replaced by Run from PC and is now deprecated. No new work is being done on it and some of the issues are still present. We recommend not using this system, because any existing bugs in it will not be fixed and it may be totally removed from future XDK releases.

Run from PC

The recommended deployment system is Run from PC or RfPC. This system directly mounts a folder that is shared on the PC, and executes the title from that share. It provides for the minimal amount of transfer between the PC and the console. Only the files that have changed, and that are directly accessed by the console, are sent between the console and the PC. We recommend Run from PC for all deployment scenarios for daily iteration. For details about how to set it up, see Xbox One Deployment.

Xbox Transfer Device

The release of Xbox One X introduces a new piece of development hardware called the Xbox Transfer Device (XTD). This is a box that sits between the PC and the console, and uses a USB 3.0 connection to enable a copy speed of up to 350 MB/s between the two. The XTD can also work in parallel with the standard network connection and can provide combined speeds up to 450 MB/s. The limiting factor becomes the speed of the two hard drives—the one on the PC and the one on the console. We highly recommend that both be SSDs, because rotational drives don’t have this level of throughput.

All deployment methods and PIX will automatically use this device if it’s present. If needed, the xbconnect /data command can tell you the connection speed being used between the PC and the Xbox. We recommend that you always use the XTD because it will greatly reduce iteration times. The XTD is also supported by the entire Xbox One Family of consoles.

Locating build hot-spots

Even when using the previously mentioned options there may still be hot spots that are affecting build performance. There are several tools available to narrow down to which parts of your build process are having the most impact on performance.

Logging

Visual Studio supports calculating build times for each phase of the build. This can be enabled from Options dialog > Projects and Solutions > VC++ Project Settings > Build Timing. At the end of each project’s build step, a complete list is displayed of the time taken for each phase is displayed—for example, how many milliseconds linking required.

If Build Logging is also enabled (it’s on the same page as Build Timing), all the times are saved to the logs. This allows analysis to determine where slowdowns happen during the build phase and where to focus the efforts on improving them.

We recommend enabling build logging to help in tracking down performance issues in the build process. For example, if the link step takes longer than expected, see Linker settings for possible ways to increase performance.

msbuild

If you are using msbuild for you builds we recommend to enable at least the following set of flags in the file logger.

Compiler/Linker

Both the compiler and linker support several options that can help locate expensive files as well as an option that can help in the creation of PCH files. The main difference between many of these options is whether link time code generation is being used. If link time code generation is being used the option needs to be given to the linker as opposed to the compiler. This is because the back end (c2.dll) is called by the linker in this case as opposed to being called by the compiler.

Timing

The /Bt+ compiler flag will cause the compiler to emit build times for each source file for both the front end (c1xx.dll) and the back end (c2.dll). Investigation of these times can point to particularly expensive files to build that may be suitable for further investigation.

The /time+ linker flag will cause the linker to emit times for various phases of the link process. For example the time taken to perform comdat folding. This can help you make the trade off of leaving a particular option enabled in the linker or not when improving build times.

Includes

It’s possible to get a complete list of all the files included by a single source file along with the include hierarchy. This is enabled with the /showIncludes option sent to the compiler. This will output the name of each file included by the source file along with what files each of those files include. This continues down the entire include hierarchy.

It’s possible to analyze the include hierarchy for all of the source files in your library to determine a useful set of headers to place within a PCH file. Pick a reasonable threshold for the number of times a header is included across the entire library for which header to add to the PCH file. Adjust the threshold as needed to achieve the optimial build times for your particular setup.

Note: If the source file is already using a PCH file the output will not show all of the includes brought in by the PCH file. It will only show the top level header that represents the PCH file. However it will show all of the later files included by the source file.

Code Generation Summary

This option is used by the backend and needs to be enabled differently based on if link time code generation is enabled. If link time code generation is not being used the /d2cgsummary flag should be passed to the compiler. If link time code generation is enabled then the flag /d2:”-cgsummary” should be passed to the linker. The reason is that link time code generation calls the back end (c2.dll) as opposed to the compiler.

This flag will output the summary of code generation for each source file. The total number of functions in the source file, the average time to build each function, and which functions had an anomalistic compile time. The anomalistic functions are a good candidate for investigation on why they took longer than normal to build. Possible reasons are through the heavy use of __forceinline or heavy template usage.

Inline Functions

This option is used by the backend and needs to be enabled differently based on if link time code generation is enabled. If link time code generation is not being used the /d2inlinelog flag should be passed to the compiler. If link time code generation is enabled then the flag /d2:”-inlinelog” should be passed to the linker. The reason is that link time code generation calls the back end (c2.dll) as opposed to the compiler.

This flag will output the inline function hierarchy for each function within the source file. Even if a function produces not code it will still show up in the inline function hierarchy. Using this information with information gathered from the code generation you can make a determination if excessive build times are being caused by extra usage of __forceinline.

Miscellaneous

Several other options are available to improve daily iteration times. These span a range of features available on the console, development PC, and Visual Studio.

FIM

Fast Iteration Mode (FIM) keeps the Title OS from being torn down and then created again on each run of the title. The removal of the tear-down and startup time can save between 5 and 10 seconds for each contiguous execution of the title. The title can even change between each run, so code changes can be applied, deployed, and tested without paying the recreation cost of the Title OS.

We recommend enabling Fast Iteration Mode, which you can do in the settings section in Xbox One Manager. You can also use xbconfig FastIterationMode=on from the Xbox One XDK command prompt.

Edit and Continue

Visual Studio includes a very powerful feature called Edit and Continue, which allows the developer to make code changes during the debug session and have them instantly applied to the title. The debugging session is not interrupted. There is no need to create a new build, deploy, and navigate to the section of interest in the title being debugged.

Edit and Continue does have some limitations. The main one is the need to effectively disable the optimizer. The problem is that many optimizer settings can create changes that cascade throughout the code base. For example, a function that was previously not inlined might become inlined. How would the debugger handle the session if the developer is currently stopped inside that function? For a list of code changes that are supported and unsupported by Edit and Continue, see Supported Code Changes.

We recommend that you try out Edit and Continue. If your daily iteration scenarios can work within the supported code-change requirements, the increase in productivity can be dramatic. It can reduce the time to test a change from several minutes to several seconds.

Distributed build systems

Most distributed build systems are designed to mimic a larger number of processor cores, to enable more source files to be built in parallel. They do this by means of a cluster of build agents running on multiple PCs. These systems give a significant performance boost when performing clean builds in which every source file must be touched.

There is a trade-off between network performance to copy the relevant files to the remote PC and the local performance from a lower number of available processor cores. For example, depending on the system you use, the PCH file may have to be regenerated on each build machine or copied over from the source machine. This file can be significant in size, requiring tens to hundreds of megabytes to be transferred over the network. This transfer overhead from building in parallel across several machines can be greater than the cost to build several files sequentially on the local machine.

We recommend that you profile whether a distributed build system should be enabled for daily iteration builds. We’ve seen cases where the cost of the build can double with a distributed build system, when only several files have changed. If the number of changed files is lower than the number of available processor cores, it’s usually better to disable the distributed build system.

As mentioned earlier in this paper /DEBUG:FASTLINK provides significant build performance improvements, however it requires the availability of the individual obj files on the developers machine. Some distributed build systems may not copy these files back to the developers machine. You should determine if /DEBUG:FASTLINK is compatible with your distributed build system and copies the individual obj files back to the developers machine.

Antivirus

Most antivirus software works by scanning all updates to a file on disk to detect possible malware infection. This can drastically increase build times due to the overhead from the file scan. With the multiple gigabytes used by a typical title, this can have a significant impact on iteration times.

We recommend disabling any antivirus software for at least the directories used for any build artifacts—that is, those files constantly being updated by the build. If possible, also disable scanning of the locations of all source files. If Windows Defender is being used on the development PC, see Add an exclusion to Windows Defender Antivirus for steps to exclude those locations.

However do not disable it from any machines used in a build farm or the machines that are used to build your final title for submission. We have seen infected titles submitted by studios during the certification process.

Debugging

Here are some recommendations for speeding up the debugging portion of an iteration.

Stepping through code

A common scenario during daily iteration is stepping through code in the debugger. Several features in the debugger can cause slow performance when single-stepping.

Both systems require extra communication between the console and development PC, iterating over all threads and decoding the resulting data. This cost increases linearly with the number of threads. This can cause a slowdown in performance between each step command. In some cases, it’s been seen to take several seconds for a single step.

We recommend that you not select either of these options unless necessary. For more options on improving debugging performance, see Make Debugging Faster with Visual Studio on the Microsoft DevOps Blog.

Symbol server setup

The Visual Studio build system supports the concept of a symbol server. This is a location on a shared network drive where a debugger can locate the required PDB files for any build. Symbols do not need to be copied and stored alongside each build; they need to be stored only once on the symbol server. All Microsoft-provided tools support the use of a symbol server as well as any tools built using the Debug Interface Access SDK (DIA).

The use of a symbol server won’t affect daily iteration times. However, it can drastically affect the ease of debugging across the studio. Any issue that shows up during play tests, on a content creator’s machine, producer’s machine, etc. can be debugged locally and the correct matching symbols will be downloaded. A special build can also easily be shared across the team and guarantee that local debugging can be performed anywhere that has access to the symbol server.

We recommend always setting up a symbol server for all builds. Microsoft provides symbols for all its products at http://msdl.microsoft.com/download/symbols. If you want, you can remove all direct source file information for security by using the /PDBSTRIPPED linker option.

Source server setup

The Visual Studio build system supports the concept of a source server. During the build process, extra information is added that stores the exact version of a source file that was used for that build. The debugger is then able to retrieve the correct version for the source from the source control system to use during debugging. All Microsoft-provided tools support the use of a source server.

We recommend that you set up a source server for the same reasons that we recommend a symbol server. It can greatly increase the ability to debug any build across the studio that has access to the server.

Resources

Recommendations

Start at the top of each section in this paper and measure the performance of each recommendation for your build target, to see which options increase performance for your daily developer iteration scenario. The recommendations are listed in order of their maximum expected performance gain.

Summary

The cost to perform a build after minor changes to a title’s code base has a big impact on developer productivity. Following the recommendations in the paper can greatly reduce this iteration time, delivering an increase in developer productivity and a greater chance of hitting schedule targets.