Overlay Folders

NOTE: Generally the Tools SKU enables you to upgrade the Tools from the XDK without having to upgrade other XDK components (e.g., headers, libraries, game OS). The Overlay Folders feature is an exception in that it depends on support from the game OS, which is released as part of the XDK but not the Tools SDK. As a result support for Overlay Folders functionality in XDK tools is only available when you use May 2018 XDK or later (ideally, June 2018 XDK or later).

Canonical uses of Overlay Folders

Overlay Folders are well-suited to scenarios where a game developer wants to locally update a single file, or a handful of files, within a title but leave the underlying storage for the original game files intact. For example:

Abril is testing a packaged version of her title on her ERA devkit. She encounters a bug in the title, knows the root cause and quickly fixes the offending code locally on her development PC. On her devkit she creates an overlay folder associated with her title’s package. She copies the executable containing her fix into the overlay folder, launches the title and is immediately able to verify that the fix corrected the bug. This dramatically increases her iteration time as previously she had to repackage the whole title and redeploy the package to the console before she could test this kind of fix.
Xin maintains a playtest lab where devkits are configured to run a variety of games on demand, over his studio’s network, from loose file builds stored on a centralized build server (using Run from PC deployment model). One of the playtesters discovers a severe bug. A developer provides a private .dll to Xin who configures it in an overlay folder on one of the lab machines. With the private .dll overlayed, the game now runs without issue. Xin was able to test the fix locally on one devkit without having to modify the network-shared version of the game, lowering risk of disrupting other playtesters.

Overlay Folders are designed to enable quick, iterative changes to a title even when the original package used to install the title is unavailable, or cannot be modified.

Overlay Folder Concepts

Using an Overlay Folder a developer can “overlay” or replace the version of a file in a game with an alternate version that is stored elsewhere including loose on a local drive or somewhere on the network. The core concepts you need to understand to use Overlay Folders successfully are:

Set - Associate the package full name for your game with the path to a folder where your overlay files will be stored. When set, the package full name and path essentially form a key-value pair that persists across reboots of the devkit.
Get - Retrieves the overlay folder path currently associated with a specific package full name.
List - Lists all PFN / Overlay Folder Path pairs registered on the Dev Kit.
Clear - You can clear (a.k.a., reset) the Overlay Folder for a specific package full name, or for all package full names. This simply removes any Overlay Folder path previously associated with the package full name.

Let’s use an example to illustrate. Neil is testing a game called, unimaginatively, MyGame. The game’s main executable is MyGame.exe and it is installed as part of the game’s launch chunk. Other parts of the game can be loaded while the game is running, so these are in chunks streamed to the console after the launch chunk has been installed. installlazily.dll is one binary that may be streamed to the console after the game is launched. A developer, Nadia, fixed a couple of bugs in last night’s packaged build of MyGame and would like for Neil to test the fixes. Neil can set up to testing the fixes easily, even without asking Nadia to repackage MyGame, using an Overlay Folder as follows:

Neil uses the Xbox One Neighborhood feature to create a new folder on the System Scratch drive of his Xbox One dev kit.
```
@REM Create the overlay folder on the devkit’s system scratch drive
mkdir \\MyDevKit\SystemScratch\MyGameOverlay\
```

He then copies the fixed binaries into the newly created overlay folder.

@REM Copy two files to the overlay folder on the console
copy c:\Development\MyGame\game.exe \\MyDevKit\SystemScratch\MyGameOverlay\
copy c:\Development\MyGame\installlazily.dll \\MyDevKit\SystemScratch\MyGameOverlay\

Next, Neil sets his folder as the overlay folder for his game.

@REM Register the overlay folder for the package.  Since it's in the scratch drive it is relative to D:\
xbapp overlayfolder MyGame_1.0.0.0_neutral__cw5n1h2txyewy D:\MyGameOverlay

Finally, he installs the launch chunk from last night’s package build

@REM Install the package’s launch chunk and register the game
xbapp install /l c:\Development\MyGame\Packages\MyGame_1_0_0_0.xvc

Now he’s ready to test. He launches his game as usual.

When the game is launched, game.exe from the overlay folder is loaded (i.e., not the game.exe copied to the console as part of the package’s launch chunk). Once streaming install progresses to the point that the chunk containing installlazily.dll has been installed, the game is notified that the chunk s available and it loads installlazily.dll. The version of installlazily.dll from the overlay folder will be loaded. As a result, the fixed binaries from the overlay folder enable the unmodified package to run properly and Neil reports that the issues are resolved.

Design Implications

The dev kit is doing a lot of work transparently on behalf of the developer to make Overlay Folders easy to use. Although it’s transparent, there are aspects that you should understand in order to maximize the benefits you get from Overlay Folders:

Flexible Workflow

In the example above, Neil decided to set up the Overlay Folder first and then install last nights’s packaged build. He could just have easily decided to install the package first. As long as the folder containing the overlay files exists and is reachable, the overlay folder has been set to associate the game’s package full name with the overlay folder path, and the game is installed/registered at the time the developer attempts to launch it, the files in the overlay folder will be used to run the game. The order of those operations is up to the developer.

The game does not have to be installed/registered at the time the overlay folder is set.
The directory referenced in the overlay folder path does not need to exist or be reachable when the overlay folder is set.
New files and directories added to the overlay folder, including during game execution, are utilized in subsequent FileOpen calls as the game runs.
There is no side-effect caused by uninstalling / reinstalling the game after the overlay is set.

Choice of Development Tools

Above, Neil created a directory on the Xbox One devkit and copied files into it using the Windows command prompt (i.e., using mkdir and copy). Instead he could have chosen to use other tools:

XDK remote tools (e.g., xbdir and xbcp)
Windows graphical tools (e.g., File Explorer)
Bash shell commands
3rd party tools

Similarly, Neil set his overlay folder using xbapp on the command line. Overlay Folder configuration is also supported via:

Xbox One Manager
Xbox Device Portal
Dev Home
Xbox Tools Framework API

Transparent Operation and Diagnosis

Visual Indicator

An Overlay Folder is easy to “Set and Forget”. Once an Overlay Folder is Set, files and directories within it can be added/updated/removed without changing the overlay folder setting. As a result, we provide visual cues in the graphical tools to proactively remind developers that they have set an overlay folder for a title.

Debug Output

When a game is activated and an Overlay Folder is set for its package full name, debug output will be written to reflect that an overlay was found, and files that are loaded from the overlay folder are output as well.

Configuring Overlay Folders

Configuring an Overlay Folder from the Command Line

As illustrated in the prior section of this document, an Overlay Folder for a given package full name can be retrieved, set, updated, and reset using the xbApp overlayfolder command.

Configuring an Overlay Folder from XBOX One Manager (XBOM)

XBOX One Manager provides the ability to view, add, and delete the overlay folder path for a package full name. For more information, see Managing Applications and Files (Xbox One Manager)

Configuring an Overlay Folder from Xbox Device Portal

Xbox Device Portal provides the ability to view, add, and delete the overlay folder path for a package full name. For more information, see Managing Overlay Folders (Xbox Device Portal) for more information.

Configuring an Overlay Folder from Dev Home

Dev Home provides the ability to view, add, and delete the overlay folder path for a package full name. See Managing Overlay Folders (Dev Home) for more information.

Configuring an Overlay Folder using the Xbox Tools Framework (XTF) API

XTF provides the ability to query, add, and clear the overlay folder path for a package full name. See Additional Xtf APIs for more information.

Security Considerations

Accessing Overlay Folders via XTD Cable

When an overlay folder is configured on the developer PC with the intention of retrieving overlay files over an XTD cable, it is necessary to explicitly grant the console permission to access the folder on the PC. The XRFS service running on the developement PC will only allow the the console to retrieve files from local folders that have been added to its access list. An overlay folder is added to the XRFS access list using a command line as follows:

@REM On the development PC, add the local overlay folder to the XRFS Service's "allow list" 
C:\>xrfssvc.exe /paths:add=C:\overlayfolders\MyGame1

Accessing Overlay Folders on Network Shares

When accessing an overlay folder that resides on a Network Share, it is necessary to ensure that user credentials with permission to access the network share are registered on the devkit. This can be achieved using the xbCred command, or equivalent capabilities in Xbox One Manager and Xbox Device Portal.