CXL 2.0 and 3.0 for Storage and Memory Applications

PCI Express (PCIe) implementations are expandable and hierarchical with embedded switches or switch chips allowing one root port to interface with multiple endpoints, such as multiple storage devices (as well as other endpoints like Ethernet cards and display drivers). However, limitations of these implementations are seen in large systems with isolated memory pools that require heterogeneous computing where the processor and accelerator share the same data and memory space in a single 64-bit address space. The lack of a cache coherency mechanism  makes memory performance for these applications inefficient and latency less than acceptable when compared to alternative implementations using CXL. 

While PCIe typically transfers a large block of data through a direct memory access (DMA) mechanism using load-store semantics (load = read, store = write), CXL uses a dedicated CXL.mem protocol for short data exchanges and extremely low latency.

While the introduction of PCIe 6.0.1 at 64GT/s helps increase the bandwidth available for storage applications with minimal or no increase in latency, the lack of coherency still limits PCIe applications like traditional SSDs, which are block storage devices. For these storage applications, NVMe, which uses PCIe as the transport interface, has become the dominant SSD technology. Next-generation SSDs, with CXL interfaces instead of PCIe, are currently being developed.

Table 2 shows a summary of some of the important characteristics for storage applications for PCIe versus CXL. This article highlights CXL’s main characteristics for high-performance computing storage applications. 

Table 2: Characteristics of PCIe versus CXL for storage applications

Memory pooling introduced by CXL 2.0 has been calculated to theoretically enable support for CXL attached memory of at least 1.28 petabyte (PB), and with multi-level switching and other features introduced in CXL 3.0, it could be even higher. This opens the door for new approaches to solving large computation problems, in which multiple hosts can work on massive problems while accessing the entire dataset simultaneously. As an example, with access to a petabyte of memory, whole new models can be created and coded to work on complex problems, like modeling climate change, with an assumption that the system can handle the entire problem all at once, rather than breaking the problem down into smaller pieces. 

Advanced fabric capabilities introduced in CXL 3.0 are a shift from previous generations and their traditional tree-based architectures. The new fabric supports up to 4,096 nodes, each able to communicate with one another via a port-based routing (PBR) mechanism. A node can encompass a CPU host, a CXL accelerator (memory included or not), a PCIe device, or a Global Fabric Attached Memory (GFAM) device. A GFAM device is a Type 3 device that effectively acts as a shared pool of memory whose I/O space is owned either by a single Host or a Fabric Manager. After configuration other Hosts and devices on the CXL Fabric can directly access the pooled memory of the GFAM device. The GFAM device allows an array of new possibilities in building systems made up of compute and memory elements that are arranged to satisfy the needs of specific workloads. For example, with access to a terabyte or a petabyte of memory, it’s possible to create whole new models to tackle complex challenges such as mapping the human genome. 

Table 3 shows some of the key feature that are driving the adoption of CXL for memory and storage applications.

Table 3: Key CXL characteristics for storage applications 

Traditionally, there have been only a couple of options to attach additional memory to accelerators or other SoCs. The most common method is to add additional DDR memory channels supporting more standard DDR memory modules. The only other viable option was to integrate memory together with the SoC within the same package. With CXL it becomes possible to put memory onto something that is very much like the PCIe bus (CXL uses PCIe PHYs and electricals). This enables a system to support more memory modules using a card with a standard CXL interface without the need for additional DDR channels. Figure 1 shows an example of how this can vastly increase the memory accessible to the SoC, both in amount (GB) and it type (RAM or persistent memory). Using this approach, memory begins to look like a pooled resource and can be accessible to multiple hosts via switching, which was first introduced in CXL 2.0 and vastly expanded in CXL 3.0.

CXL is rapidly becoming the interface of choice for managing and sharing large amounts of memory coherently among multiple Devices and Hosts. It is enabling a true heterogeneous composable and disaggregated architecture supporting more than just memory. The CXL 3.0 spec expands on previous versions of CXL, doubling the per-lane bandwidth to 64 GT/s with no added latency, while adding multi-level switching, efficient peer-to-peer communications, and memory sharing.

To ease and accelerate adoption of the latest CXL protocol, Synopsys offers a complete CXL IP solution, encompassing controller with IDE Security Module, PHY, and verification IP to deliver secure, low-latency, high-bandwidth interconnection for AI, machine learning, and high-performance computing, including storage, applications. 

Built on silicon-proven Synopsys PCIe IP, our CXL IP solution lowers integration risks for device and host applications and helps designers achieve the benefits that CXL 3.0 brings to SoCs for data-intensive applications. As an early CXL contributor, Synopsys had early access to the latest specification, enabling our engineers to deliver a more mature solution. Already, Synopsys has delivered CXL 2.0 and 3.0 solutions with IDE support to several customers, including for next generation SSD and advanced memory applications with proven silicon in customer products and successful third-party interoperability demonstrated in hardware.