Emerging Applications for CXL

Introduction

In the previous article, Introduction to the Compute Express Link (CXL) Standard, we described the fundamentals of CXL and the initial target applications. This article highlights some of the emerging applications that the industry is discovering for CXL, and how they are being enabled by the rapidly evolving CXL specification.

Consider that at the time of this article, membership in the CXL consortium has grown to over 120 members and is now clearly the largest of the new high-speed interconnect/coherency standard consortiums, eclipsing membership in Cache Coherent Interconnect for Accelerators (CCIX) consortium with about 50 members, Gen-Z with approximately 70 members, and OpenCAPI with around 38 members. This is really astonishing, considering that the CXL Consortium was formed about 18 months ago compared to nearly four years ago for the other two. So, CXL is gaining traction in the industry and targeting many exciting new emerging applications.

Figure 1 shows a snapshot of the CXL specification development from the initial announcement of the CXL consortium and 1.0 specification last March to the present day. Looking at the Figure you can get a sense of how quickly things are changing.  

New Memory Applications with CXL

The introduction of switching to CXL will mean that multiple storage devices can be connected to a single Host and allow the Host to access the memory coherently as part of its memory space. This was not possible in CXL 1.1, since it allowed only a single Host-to-Device connection (no fanout). With the next generation of CXL, memory that is attached to multiple downstream devices may be able to be shared across multiple Hosts, and the memory can be split among those Hosts as needed for a particular application.

Emerging memories, often transactional, may have non-deterministic and asymmetric Read/Write timing. This makes them poorly suited for sharing the DDR bus with DRAM. Essentially, only homogeneous DIMMs can share the DDR bus, typically requiring:

• Same generation of DDR
• Same speed grades and timing to maintain bus efficiency
• Same device geometry to enable interleaving across all channels
• Same power and thermal envelopes

This is a significant limitation with the current homogeneous DIMM approach to memory expansion. CXL provides an exciting alternative for memory expansion that doesn’t rely on the DDR bus or adding more DIMMs, allowing a true heterogeneous memory attach solution. As shown in Figure 2, future memory devices can be connected to system-on-chips (SoCs) using CXL, which creates a standard, media-independent connection to virtually any memory types, including DDR, LPDDR, persistent memory, etc. Since each CXL link can connect to an optimized controller for a particular memory, performance characteristics like persistence, endurance, bandwidth, and latency, can be optimized and matched to each particular memory type.

As SoCs with multiple CPUs need to access more memory and different types of memory, this CXL-based memory approach can be ideal, as additional memory using traditional DDR interfaces can be augmented with CXL links. In addition to allowing the inclusion of varying memory types, the CXL memory expansion also uses a reduced number of pins compared to adding DDR interfaces. A CXL x4 interface with 16GB/s bandwidth in each direction would only require 16 pins. If the CXL-based memory module form factor supports a x8 CXL link, each CXL link on the SoC would still only require 32 pins, and the bandwidth would go up to 32GB/s in each direction.

While the CXL links on the SoC in the right side of Figure 2 could each be connected to a separate CPU, if the SoC design includes an embedded CXL switch (taking advantage of the next generation of CXL switching capability), then the various Host CPUs could access multiple memory devices on different CXL links, opening even more possibilities for the CXL memory expansion approach.

CXS: A New Interface for CXL

While there is no MOU with CCIX, there are rumblings in the industry of cooperative efforts that are being considered, and that includes one I will discuss in the next section, referred to as CCIX over CXL. It is important to understand the new application-side interface that has been recently enabled by Arm and Synopsys¹.

The standard application-side interface for CXL comprises of three separate interfaces—one for each of the CXL protocols, CXL.io, CXL.cache and CXL.mem, as shown in Figure 3a. With Arm’s recent announcement supporting CXL, it now becomes possible to consider a CXL controller that includes a CXS interface.

Arm defined a new streaming interface, called CXS, to bridge from a CCIX controller to their Coherent Mesh Network (CMN-600), as shown in Figure 3b. CCIX controllers, such as the DesignWare® CCIX IP, implementing Arm’s CXS interface allow direct plug into the Arm CMN-600. Recently, Arm collaborated with Synopsys to update the CXS interface definition and enable Arm’s next-generation Coherent Mesh Network to connect directly into a new version of a CXL controller with a CXS interface. To make this work some of the CXL link layer functionality is moved out of the CXL controller and is taken over by the logic in the CMN, or other applications. In its place, a CXS interface block is added in lieu of the CXL.cache and CXL.mem interfaces, and the result is a design, shown in Figure 3c, that looks very similar to the CCIX controller in 3b.