Data Movement Depends on PCIe

Article By : Gary Hilson

The Peripheral Component Interconnect Express (PCIe) bus standard has a lot riding on it. Or perhaps more accurately, needs to accommodate a lot of data flowing through it.

The Peripheral Component Interconnect Express (PCIe) bus standard has a lot riding on it. Or perhaps more accurately, needs to accommodate a lot of data flowing through it.

Both the relatively mature Non-Volatile Memory Express (NVMe) protocol as well as the fledgling yet rapidly evolving Compute Express Link (CXL) are leveraging the ubiquity of PCIe, with 6.0 expected to be widely released by the end of 2021.

Mark Orthodoxou, director of Microchip Technology’s data center business unit said, the value of PCIe is its ubiquity in that it’s interoperable across CPUs, and it openness has allowed for an ecosystem to be built around it. He said the drawbacks of PCIe stem from it becoming quite complex over time, but those challenges are surmountable because there’s lots of licensable IP that can be drawn upon.

The PCIe I/O bus specification is expected to continue an update cadence of every three years as new opportunities present themselves in mobile and automotive, and NVMe and CXL take advantage of it improve storage and memory performance access.  (Click on the image for a larger view. Courtesy PCI-SIG)

One of the drawbacks is that some features and functions that exist today weren’t even conceived in the earlier days of PCIe. As new applications are mapped, said Orthodoxou, unanticipated problems arise. One example is having a device removed from a server in form of hot plugging NVMe SSDs. “It wasn’t designed as a hot plug interface, so there was work required to go and deal with things like that. Over time, PCIe has become more complex as people have found different ways to use it.” Even as new uses arise that leverage PCIe, such as CXL, there’s nothing inherently limiting innovation, he said. “Ultimately it’s a protocol that just lives over the PCIe electricals.”

While it’s possible that CXL could leverage PCIe 6.0 sooner than other devices, Orthodoxou said it’s likely that the CXL consortium needs as much time as possible to sort out what CXL needs to look like to support the use cases that will require the latest and greatest PCIe. “It’s not like if we had some different electrical interface available that CXL would be moving faster.”

CXL uses a flexible processor port that can auto-negotiate to either the standard PCIe transaction protocol, or the alternate CXL transaction protocols. (Source: CXL)

Microchip has many products across different business units that leverage PCIe, including industrial, automotive and data center markets, including PCIe switches and NVMe controllers. “PCIe has created a great deal of market opportunity for us,” he said. Aside from its CXL Retimer, however, the company hasn’t announced any CXL products, but it’s active within the consortium.

In the meantime, there’s been nearly a decade of NVMe adoption, noted Jeff Janukowicz, IDC research vice president for solid state storage and enabling technologies, but last year marked an inflection point where more than half of enterprise spending on flash SSDs was NVMe PCIe. “It’s taken some time to make that transition, but we are seeing a lot of the benefits around PCIe starting to manifest.” He said the release of NVMe 2.0 is going to allow for further advancements around PCIe, as well as innovation around form factors that weren’t possible with hard disk drives. “We’re starting to move away from some of the limitations from a hardware perspective.”

While PCIe has become a foundational technology, said Janukowicz, the advent of protocols such as CXL reflect the growing diversity of workloads. “There is a lot of innovation trying to satisfy different environments, and where you start to see some of the interest ais round things like CXL.” While NVMe was built with solid state storage in mind, he said CXL is being built around memory rather than storage to drive higher performance and lower complexity, and includes memory-attached devices, memory expansion and accelerators.

Like PCIe, said Janukowicz, a big part of the value proposition is that it’s an open standard, and anyone can participate to create solutions that address the direction workloads are taking. “We’re seeing a lot more interest in expanding memory pools to support some of the next generation workloads, whether it’s in-memory types of applications and in-memory databases, but also emerging applications like artificial intelligence (AI) and machine learning.”

Steve Woo

But not everyone sees the open standards approach of PCIe as being the only the option. Nvidia opted to develop its own PCIe alternative, NVLink. First introduced in 2014, the wire-based communications protocol for near-range semiconductor communications can be used for data and control code transfers in processor systems between CPUs and GPUs and solely between GPUs.

While PCIe may be ubiquitous, said Paresh Kharya, senior director for product management for Nvidia, the speed is far less than what Nvidia can provision in its own servers for workloads such as AI and high-performance computing. “There’s a constantly increasing need for computing capabilities. One of the ways you scale up compute is by having many GPUs work together as one.” NVLink provides a fast and scalable interconnect so Nvidia GPUs can work together as a giant accelerator. When it was first introduced, NVLink offered five times as much bandwidth as PCIe 3.0, he said, and now provides 600 gigabytes per second of bi-directional bandwidth, which is nearly 10 times that of the dominant PCIe 4.0.

Although NVLink is Nvidia’s proprietary technology, its GPUs still support PCIe. “That’s how we connect to the CPU today,” said Kharya. While other vendors such as IBM have worked with Nvidia to incorporate NVLink into their processors, by and large the interconnect is primarily used to accelerate workloads using Nvidia GPUs, he said. Nvidia provides baseboards and motherboards for servers that already have NVLink capabilities to accelerate adoption by the broader ecosystem, and some of the fastest supercomputers around the world use NVLink.

Kharya said NVLink enables Nvidia to rapidly innovate and keep on improving the performance for its customers. “We expect to continue to evolve NLink link as fast as we can.” But it’s also actively working with the CXL community and participating in the advancement of PCIe standards, he said, even though Nvidia views NVLink as the clear winner in terms of the bandwidth it offers today. “We really want the PCIe standards to evolve as quickly as it can.”

Janukowicz said customized approaches do have their merits and optimize around certain metrics such as performance latency, cost or power for a specific environment, but having an open standard provides a lot of benefits just in terms of bringing some economies of scale to the market. “Open standards historically have really been necessary to drive the market.” He said customers always hesitant to be locked in with a specific vendor, especially given today’s supply chain challenges caused by the pandemic. “It highlights the necessity around some of that flexibility that you tend to get with more of an open environment where, you can certainly dual source or have multiple options out there.”

In the meantime, updates to PCIe are continuing apace as CXL gathers steam. Rambus fellow Steve Woo said CXL is an example of how data is driving architectural evolution, while standards for High Bandwidth Memory (HBM) and Graphics DDR (GDDR) are evolving faster than they have historically in response to the importance of data and these architectures. “An important driver like CXL is going to continue to push the industry on PCIe speeds,” he said. “It’s reliable, people understand it, it’s been in the industry for a while, and it’s trusted.”

Al Yanes

Data movement has started to become the bottleneck, not memory or compute, said Woo. It used to be the graphics market that drove PCIe to be faster as graphics became higher resolution and frame rates improved. But those data rates are nowhere near what the cadence is today, he said. The amount of data is doubling every couple of years, which means the performance of the pipe must keep up.

PCIe 6.0 promises to do just that. In a webinar update in late May by the PCI Express Special Interest Group (PCI-SIG), the organization that defines PCIe I/O bus specifications and related form factors, the latest iteration was described as the most significant jump since the move from PCIe 2.0 to PCIe 3.0. The upcoming update doubles its data transfer rate and delivers higher signaling rate and tighter signal integrity requirements than its predecessor. The expectation is that it will be officially released by the end of the year, although the PCI Express draft 0.71 specification had been released for member review in early July. Draft 0.9 is expected to follow and undergo a two-month review for any final issues.

PCIe 6.0 implements PAM4 signaling, allowing it to pack more bits into the same amount of time on a serial channel Includes low-latency Forward Error Correction (FEC) with additional mechanisms to improve bandwidth efficiency and improve reliability. It also provides Integrity and Data Encryption (IDE) of TLPs by applying AES-GCM for encryption and authenticated protection of the entire TLP. A single PCIe 6.0 x16 can support 800G Ethernet. This latest iteration targets high-bandwidth applications such as cloud, AI and machine learning, and edge computing.

Nvidia opted seven years ago to develop its own PCIe alternative. In an NVIDIA A100, NVLink doubles inter-GPU communication bandwidth to nearly 10x the PCIe Gen4 Bandwidth. (Courtesy Nvidia)

Its completion will come just as PCIe 5.0 devices are expected to beginning hitting the enterprise market. That specification was released in May 2019 and doubled the bandwidth of its predecessor while maintaining backward compatibility with all PCIe generations. PCIe 5.0 also featured electrical changes to improve signal integrity and mechanical performance of connectors.

PCI-SIG president and board chair Al Yanes said the group’s membership continues to “blossom” with nearly 900 members worldwide and the focus is across the full spectrum of devices with PCIe uptake in mobile spaces and lots of interest from the automotive industry. “We formed a work group and got a lot of participation, so that’s our new target industry.”

He expects the cadence of an update every three years or less with a continued focus on compliance and backward compatibility, both of which have been essential ingredients of the bus standard’s recipe for success.

This article was originally published on EE Times.

Gary Hilson is a general contributing editor with a focus on memory and flash technologies for EE Times.


Subscribe to Newsletter

Leave a comment