Scaling Bandwidth for AI/HPC Multi-Die Designs with Ethernet, PCIe & UCIe IP

One of the most significant challenges in constructing an AI infrastructure lies in interconnecting tens of thousands of servers spread across multiple data centers to form a vast network capable of handling AI workloads. AI data center’s complexity features multiple CPUs and accelerators, various switches, numerous NICs, and a host of other devices. Connecting these components seamlessly requires an efficient network. This is where scaling up and scaling out technologies become key. IO disaggregation provides an opportunity to address the scale up and scale out strategies. In a scaling-up scenario, PCIe & UCIe, leveraging UCIe IP for die-to-die connectivity, can act as the internal network fabric. Meanwhile, in a scaling-out scenario, Ethernet & UCIe IP can be used to enable high-speed, low-latency links between servers. 

Scaling up, or vertical scaling, involves boosting the resources of a single server by adding more CPUs, expanding memory, or enhancing storage capacity. This approach simplifies architecture and reduces latency, as all resources are contained within one machine. Central to scaling up is PCIe technology, which acts as the internal network fabric. The latest iteration of this technology, PCIe 7.0, would be used to connect CPUs, GPUs, NICs, storage drives, and other peripherals with low latency and high bandwidth, ensuring efficient communication within the server. 

On the other hand, scaling out, or horizontal scaling, distributes the workload across multiple servers, creating a network of machines that work in tandem. This approach is cost-effective, provides redundancy, and offers flexibility for handling growing workloads. However, it introduces complexity in network configuration and management, as communication between multiple machines can add latency. Here, Ethernet technology and the upcoming Ultra Ethernet standard, become vital, providing the high-speed, low-latency communication needed to link servers across a data center. Emerging standards are being discussed to support high-speed links between AI accelerators and switches, ensuring efficient data transfer and coordination. 

As shown in Figure 1, there are many opportunities for multi-die designs to enable scaling up and out. Multi-die designs with PCIe, Ethernet and UCIe IP are essential to address the time to market, costs, and risk reduction challenges, while offering full architectural flexibility. Let’s dive into the main types of IO chiplets for multi-die designs, including very large AI training chips, switch SoCs and retimers. 

1. Very Large AI Training Chips 

AI chips must become significantly more efficient at both computation and data management to handle the massive data models of today. Specialized AI training chips are designed to meet these immense computational and data processing demands, integrating multiple processing units, memory, and interconnects on a single silicon die to deliver unparalleled performance and efficiency. This is where multi-die designs, integrating 40G UCIe and 224G Ethernet, step in to enable AI training efficiently. Instead of relying on thousands of huge GPUs, data centers could run their AI training with significantly less beachfront in SoCs while achieving unprecedented bandwidth, extended reach with least latency and power overhead.

224G Ethernet PHY IP provides a robust and customizable interface. With CEI-224G in development, achieving 224Gbps per lane while maintaining ecosystem interoperability and optimizing power is critical for AI training operations. Additionally, UCIe IP can deliver up to 40Gbps of high-speed, low-latency, energy-efficient data transfers across multiple dies, significantly enhancing the scalability and modularity of these chips.

To scale bandwidth for multi-die designs with the integration of high-speed interfaces like PCIe and Ethernet, along with UCIe IP and link health monitoring features. Synopsys provides high-quality and complete IP solutions for UCIe up to 40Gbps with signal integrity monitors and testability features, 224G Ethernet, and PCIe 7.0, enabling maximum bandwidth, low latency and scalability. Synopsys IP Solutions for multi-die designs are compliant with the evolving standards, achieving interoperability with ecosystem products and silicon successes across multiple technologies, making it a low-risk solution to enabling the next-generation of data center AI chips. 

Synopsys’ comprehensive and scalable multi-die solution, encompassing EDA and IP products, enables early architecture exploration, fast software development and validation, efficient die/package co-design, robust die-to-die connectivity, and improved manufacturing and reliability.