Xbox One Family Memory System

Advanced Technology Group

Updated June 13th 2017

In this topic

Memory Clients

System coherency

The CPU

The GPU (Xbox One and Xbox One S only)

The GPU (Xbox One X)

Direct memory access (DMA) engines

Summary

Memory clients

The central elements of the Xbox One memory system are the North Bridge and the GPU memory controller. The GPU memory controller supports multiple clients (for example, the CPU and the GPU), coherent and non-coherent memory access, and two types of memory (DRAM and ESRAM). Xbox One X contains only DRAM and no ESRAM.

Figure 1 illustrates the Xbox One and Xbox One S memory clients with the maximum available bandwidth in every path. Figure 2 illustrates the memory clients for Xbox One X.

Note: The bandwidths shown are theoretical-peak figures. Keep in mind that, in practice, it can be difficult to attain theoretical-peak performance with any hardware.

As you can see on the right side of the diagram, the Xbox One and Xbox One S console have:

Xbox One X changes the memory from DDR3 to GDDR5 and increases the clock speed of the North Bridge. These changes result in a drastic increase in memory performance.

Figure 2: Peak bandwidths of Xbox One X memory clients

The change from DDR3 to GDDR5 results in an increase from 68 GB/s to 326 GB/s bandwidth to main memory. This cascades through the rest of the components, also increasing their available bandwidth over Xbox One and Xbox One S.

DRAM

The maximum combined read and write bandwidth to DRAM on Xbox One and Xbox One S is 68 GB/s (gigabytes per second). In other words, the sum of read and write bandwidth to DRAM cannot exceed 68 GB/s. You can realistically expect that about 80–85% of that bandwidth—that is, approximately 55 GB/s—will be achievable in practice.

On Xbox One X, the same rules hold true for maximum available bandwidth. The only difference is that the bandwidth increases to 326 GB/s. You can realistically expect to achieve about 80-85% of that bandwidth, or approximately 275 GB/s.

DRAM bandwidth is shared among the following components:

ESRAM (Xbox One and Xbox One S only)

The maximum theoretical combined ESRAM read and write bandwidth is 192 GB/s. However, this level is not sustainable, and it is not even achievable in practice except in very short bursts. Most rendering scenarios should plan for peak ESRAM bandwidth of 109/117 GB/s. It is true that in some color blending and read-modify-write compute shader scenarios, sustained 110–133 GB/s is achievable in practice. Having such high bandwidth and lower latency makes ESRAM a very valuable memory resource for the GPU.

ESRAM bandwidth is shared among the following components:

System coherency

There are two types of coherency in the Xbox One Family memory system:

The two CPU modules are fully coherent. The term fully coherent means that the CPUs do not need to explicitly flush caches in order for the latest copy of modified data to be available, (except when using Write Combined or another type of uncached memory access). This is because the bus snoops CPU caches on every coherent memory access.

The rest of the Xbox One infrastructure, which includes the GPU and I/O devices such as audio and the Xbox One sensor, is I/O coherent. The term I/O coherent means that coherent memory access from those clients can snoop CPU caches, but the clients’ own caches aren’t snooped by the system.

In the case of coherent memory access, when the CPU produces data, other system clients can choose to consume that data without any extra synchronization work from the CPU.

Note: The total coherent bandwidth through the North Bridge is limited to about 30 GB/s (Xbox One and Xbox One S) and 40 GB/s (Xbox One X).

The CPU requests do not probe any other non-CPU clients, even if the clients have caches. (For example, the GPU has its own cache hierarchy, but the GPU is not probed by the CPU requests.) Therefore, I/O coherent clients must explicitly flush modified data from their caches for any latest-modified copy to become visible to the CPUs and to the other I/O coherent clients.

The GPU can perform both coherent and non-coherent memory access. Coherent read-bandwidth of the GPU is limited to 30 GB/s when there is a CPU cache miss, and it’s limited to 10–15 GB/s when there is a cache hit on Xbox One and Xbox One S. Xbox One X increases these numbers to 40 GB/s for a cache miss and 20 GB/s when there is a cache hit. A GPU memory page attribute determines the coherency of memory access. The GPU section of this document describes this memory access in more detail.

The CPU

The Xbox One Family has two CPU modules, and each CPU module has its own 2 MB L2 cache. Each module has four cores, and each of the four cores in each module also has its own 32 KB L1D cache and 32KB L1I cache.

The Xbox One Family CPU uses the MOESI protocol to keep the caches coherent. The farther each level of cache is from the requestor, the more costly it becomes to access that cache. It can be up to 40 times slower for a cache miss as opposed to a cache hit.

Table 1. Typical latencies for local and remote cache hits.

Cache hit type Latency
Local L1 hit Three cycles for 64-bit values
Five cycles for 128-bit values
Local L2 hit Approximately 30 cycles
Remote L2 hit Approximately 100 cycles
Remote L1 hit Approximately 120 cycles
DRAM hit 140-160 cycles.

Each of the two CPU modules connects to the North Bridge by a bus that can carry up to 20.8 GB/s in each direction on Xbox One and Xbox One S. Xbox One X increases this bandwidth to 27 GB/s per module. The two modules have their own lanes to the North Bridge and do not steal bandwidth from each other.

Access to the L1 and L2 caches is controlled directly by the CPU clock speed. This means the cost to access the CPU cache on all Xbox One Family consoles is the same number of cycles.

From a program standpoint, normal x86 ordering applies to both reads and writes. Stores are strongly ordered (becoming visible in program order with no explicit memory barriers), and reads are out of order.

Keep in mind that if the CPU uses Write Combined memory writes, then a memory synchronization instruction (SFENCE) must follow to ensure that the writes are visible to the other client devices.

The GPU (Xbox One and Xbox One S only)

The GPU can read at 175/185 GB/s and can write at 109/117 GB/s through multiple combinations of its clients. Examples of GPU clients are the color and depth blocks and the GPU L2 cache.

The GPU has a direct non-coherent connection to the DRAM memory controller and to ESRAM. It also has a coherent read/write path to the CPU’s L2 caches and to DRAM.

For each read and write request from the GPU, the request uses one path depending on whether the accessed resource is located in “coherent” or “non-coherent” memory.

Note: It is possible to specify memory attributes like GPU-coherency and CPU-cacheable by using XMemAlloc or D3DGraphicsAllocateMemory APIs.

Some GPU functions share a lower-bandwidth (54.6 (Xbox One)/58.4 (Xbox One S) GB/s) bidirectional read/write path through the hub. Those GPU functions include:

Internally, hub clients are connected to one of the two 27.3/29.2 GB/s internal busses. If two hub clients are not sharing a single bus, they each can use up to 27.3/29.2 GB/s of bandwidth each. If they share a bus, they share the 27.3/29.2 GB/s bandwidth.

Because the GPU is I/O coherent, data in the GPU caches must be flushed before that data is visible to other components of the system.

1 The peak for the instantaneous read bandwidth of the GPU is 218/234 GB/s, but this speed cannot be sustained. The maximum read bandwidth from DRAM and ESRAM combined can be only 170 GB/s. For this reason, the GPU has a peak read bandwidth of 170 GB/s. ESRAM bandwidth of 133 GB/s was attained by alpha-blending an FP16 × 4 render target.

The available bandwidth and requirements of other memory clients limit the total read and write bandwidth of the GPU. The following table shows an example of the maximum memory bandwidths that the GPU can attain with different types of memory transfers.

Table 2. Maximum GPU memory bandwidths (Xbox One/Xbox One S) associated with different types of memory transfers.

Source memory Destination memory Maximum read bandwidth (GB/s) Maximum write bandwidth (GB/s) Maximum total bandwidth (GB/s)
ESRAM ESRAM 54.6/58.4 54.6/58.4 109/117
ESRAM DRAM 68.2 68.2 136.4
DRAM ESRAM 68.2 68.2 136.4
DRAM DRAM 34.1 34.1 68.2

Note: ESRAM-to-DRAM and DRAM-to-ESRAM scenarios are symmetrical.

The GPU (Xbox One X)

The GPU can read and write at 450 GB/s through multiple combinations of its clients. Examples of GPU clients are the color and depth blocks and the GPU L2 cache.

The GPU has a direct, non-coherent connection to the DRAM memory controller. It also has a coherent, read/write path to the CPU’s L2 caches and to DRAM.

For each read and write request from the GPU, the request uses one path depending on whether the accessed resource is in “coherent” or “non-coherent” memory.

Note: It is possible to specify memory attributes like GPU-coherency and CPU-cacheable by using XMemAlloc or D3DGraphicsAllocateMemory APIs.

2 Although ESRAM has 109/117 GB/s of bandwidth available, in a transfer case, the DRAM bandwidth limits the speed of the transfer.

Some GPU functions share a lower-bandwidth (75 GB/s) bidirectional read/write path through the hub. Those GPU functions include:

Internally, hub clients are connected to one of the two 37.5 GB/s internal busses. If two hub clients are not sharing a single bus, they each can use up to 37.5 GB/s of bandwidth. If they share a bus, they share the 37.5 GB/s bandwidth.

Because the GPU is I/O coherent, data in the GPU caches must be flushed before that data is visible to other components of the system.

The available bandwidth and requirements of other memory clients limit the total read and write bandwidth of the GPU.

Direct memory access (DMA) engines

The Xbox One and Xbox One S console have 27.3/29.2 GB/s of read and 27.3/29.2 GB/s of write bandwidth, and Xbox One X has 37.5 GB/s of read and write bandwidth. The bandwidth is shared between:

You may wonder what happens when the GPU is busy reading or writing data and a DMA engine is told to copy data from one type of memory to another. In this situation, the memory system of the GPU shares bandwidth fairly between source and destination clients. You can calculate the maximum bandwidth by using the peak-bandwidth diagram shown in Figure 1.

DMA engine 0 is reserved by the system, so to maximize your title’s DMA bandwidth you may want to use DMA engine 1 and 3 together because they don’t share the same memory bus.

DMA engine example (Xbox One and Xbox One S)

The design of this DMA engine example, which is illustrated in Table 3, assumes the following conditions:

Note: This example is an idealized situation.

Table 3. DMA engine example.

ESRAM read clients ESRAM write clients DRAM read clients DRAM write clients  
DMA engine GPU GPU GPU DMA engine

Keep in mind that the combined read and write bandwidth for DRAM cannot exceed 68 GB/s for all read and write clients. This example has one read client and two write clients.

With the bandwidth available, each client gets 22.6 GB/s, which means that:

Note: The DMA engine’s write bandwidth (22.6 GB/s) does not exceed its write bandwidth limit of 27.3/29.2 GB/s total.

For all read and write clients, the combined read and write ESRAM bandwidth cannot usually exceed 109/117 GB/s. This example has one read client (the DMA engine) and one write client (the GPU), so each client should get half of the bandwidth.

However, the DMA engine’s read bandwidth is limited to 27.3/29.2 GB/s. Moreover, because the DMA engine is performing a copy, its read bandwidth should be equal to its write bandwidth. So, in this case, the read bandwidth of the DMA engine is 22.6 GB/s. And in this situation, the GPU gets the rest of the ESRAM bandwidth, which is 80.4 GB/s.

The GPU writes to DRAM at 22.6 GB/s and to ESRAM at 85/91 GB/s, which doesn’t exceed the maximum write bandwidth for the GPU (109/117 GB/s). So, the GPU’s read bandwidth is never a bottleneck.

Table 4. The final result for the maximum bandwidth in the example.

  DRAM ESRAM
DMA engine read 22.6 GB/s  
DMA engine write   24/26 GB/s
GPU read 22.6 GB/s  
GPU write 22.6 GB/s 85/91 GB/s
Total read and write 67.8 GB/s 109/117 GB/s

DMA engine example (Xbox One X)

Xbox One X is similar to the Xbox One. However, there’s no ESRAM, and this drastically lowers the combinations of memory access. The only real combination is between the DMA engines and the GPU accessing memory at the same time. In this case, the DMA gets its full bandwidth to DRAM, the GPU loses the same amount of bandwidth on a one-to-one ratio. One DMA engine writing to memory uses 37.5 GB/s, which reduces the available GPU bandwidth to DRAM to 288.5 GB/s. The limiting factor is the 326 GB/s bandwidth of DRAM.

Summary

The Xbox One and Xbox One S memory system can be complicated, but Xbox One X simplifies this with the removal of ESRAM. There are still multiple memory busses, complex cache hierarchies on the CPU, the GPU, and the DMA engines. Understanding the Xbox One Family memory system is crucial to optimal placement of game resources.