PGOLite is a lightweight alternative to the full PGO process on Xbox One. It is similar in concept to PGOLite for Xbox 360.
Full PGO requires a feedback loop of building an instrumented executable, gathering training data, and applying it to get an optimized result. The process can be time consuming, but the improvement can be large (above 10%, sometimes above 20%). Full PGO can leverage a variety of techniques to get improvements, one of which is altering function layout in memory to avoid I-cache thrashing.
PGOLite uses PIX capture files to generate a set of link order files, which is used to relink the code with commonly called functions in close proximity in memory, which reduces I-cache thrash. PGOLite’s process is much lighter than full PGO, but improvements are smaller (single digit %) since it leverages only one technique to make an improvement.
The linker is unable to re-order functions unless the code is compiled with /Gy, which specifies function level linking. If you are already using the linker option /OPT options to reduce size and fold identical code, it is likely you are already building with /Gy. See the Linker Warnings section below for more information.
PGOLite is a command line tool (PGOLite.exe from the Durango XDK bin dir). Its command line takes the form:
PGOLite input0.pix3....inputn.pix3 [ignoreNTSymbolPath] [-o outputdir] [PDBDir0....PDBDirn]
The tool takes in one or more PIX3 capture files, and processes them to generate a set of link order files, one per module in the application. If your application is split into an EXE and multiple DLLs, then clearly the linker is not able to reorder functions from one DLL or EXE into another.
PGOLite only accepts PIX3 file containing function summary captures, call graph captures or instruction traces. These are the only types that contain callgraph information that can be used to determine a good order for functions. Other PIX3 file types generate a warning message but PGOLite continues.
The output files will be named after the module it is for, with .linkorder.txt appended. For example, RetailGame.exe would generate a link order file called RetailGame.linkorder.txt.
The –ignoreNTSymbolPath option causes PGOLite not to use the _NT_SYMBOL_PATH environment variable to try to find symbols for the modules being referenced. See the Symbols section below for more details.
The –o option allows specification of the directory to put the link order files in. The default is the current directory.
Remaining entries on the command line are additional directories that will be searched for symbols. See the Symbols section below for more information.
Open your project solution in Visual Studio. For each of the projects that build an EXE or DLL in your solution go through and:
Now relink (not rebuild, just relink) all of the EXE and DLL projects in your solution. Your game should be ready to go, hopefully with improved performance.
If you link from the command line, the additional option is /ORDER:@GameRetail.linkorder.txt (or whatever your generated link order file is).
The link order files are just lists of function names, in plain text. They are relatively readable, with the exception that the function names are decorated. You can use the undname utility that comes with Visual Studio if you need to view undecorated names; often you can figure it out easily as the class name and function name are unchanged in amongst the various type and parameter information encodings in the decorated name.
PGOLite will flag an error and stop if:
The last part deserves some explanation. If you build an EXE or DLL with /INCREMENTAL, the code generated very likely contains thunks which allow the linker to replace functions incrementally, without a full relink. These thunks are identifiable and PGOLite can see them. These thunks are also not ideal for performance, especially I-cache locality since function calls will jump. It is very likely your full release or retail build configuration does not use /INCREMENTAL. In any case, PGOLite cannot relocate incremental linker thunks; but in any PGOLite is going to struggle to make I-cache improvements up against an incrementally linked binary. In this scenario, PGOLite exits with an error message.
PGOLite works by looking at all of the call graphs in the capture files provided, and building a map of which functions call which other functions. From there, it is able to decide which functions should be located close to each other, and that gives rise to a function order.
PGOLite can only reorder functions inside a module, it cannot move them across modules. So function A in module N calling function B in module M many times is not going to result in B being moved next to A in memory.
Clearly, PGOLite can only reorder functions based on the callgraphs it can see in the capture files. If function F calls function B 750 times a frame but never function C in the capture you give PGOLite, but calls function C 18,000 times a frame at some other points in the game, PGOLite cannot know that. It will move B near A, not C. Take care to capture situations representative of your heaviest loading.
PGOLite needs symbol information to determine function names and locations. In the case of function summary and callgraph capture, that symbol information is embedded in the PIX3 file. Currently PGOLite only needs PDBs to resolve function information in instruction traces. In later revisions it will not require PDBs at all.
By default PGOLite will look in the following places for PDB files, based on the capture files supplied:
PGOLite can take in several input PIX3 files. This gives an opportunity to feed PGOLite captures from different heavy loading situations and let it crunch the call graphs and decide a good order.
Not all title CPU usage is consistent frame to frame (for example, some systems update more slowly that every frame, some systems run across more than one frame, some systems vary heavily in behavior over short periods of time, like physics and collision systems or effects).
Both instruction trace and call graph captures typically have a very short duration – perhaps a single frame at the very most. Function summary on the other hand is able to run over a period of many seconds.
PGOLite decides which functions call which others based on total call count. Obviously, in a call graph capture or instruction trace lasting a frame or less, the count of calls to a function will be much smaller than in a function summary lasting several seconds. If you combine function summary data with instruction trace and call graph capture data, the function summary data will tend to outweigh the other shorter inputs. However this is likely appropriate as the function summary data is a better representation of overall call count and call patterns.
PGOLite will make little to no improvement if:
Once you add a link order file to your linker settings, you may see some warnings.
LNK 4065: The function could not be re-ordered, as it is in a translation unit not built with /Gy. The linker just ignores this function, no critical problem is caused. Of course, that is a function that is not close to the location PGOLite thinks it should be in, so a performance improvement may be lost. Make sure the project this function is from is built with /Gy.
LNK 4037: The function specified does not exist. This means somehow the function specified by the warning is named in the PIX3 file, but not in the object files being linked; this can just mean an old PIX3 file compared to the source tree, or perhaps a PIX3 file from a different configuration of the solution (especially if, for example, the layout of modules is different in different configurations). This is also not a critical error, though it may indicate to you that your link order files are a little old; it might be worth regenerating them.
In both cases, only a few of these warnings is not a problem. Large numbers mean the link order file wholesale does not match the build.
In particular, single functions that are not often called failing to be reordered is no big issue. The I-cache improvements from relocation functions tend to be important only on very heavy call paths.
Link order files should not break anything in your build or affect correctness. They are after all only moving around the COMDATs that are the linker’s unit of relocation. However, if the link order files mismatch to a great degree (see Linker Warnings above) you may adversely affect I-cache performance.
As a guide you should consider regenerating the link order files when: