Understanding the Impact of RAM Latency on Modern CPU Design

RAM latency has been a significant bottleneck in processor design for quite some time. With the advent of RAM clock speeds surpassing 3GHz, the issue of high latency becomes all the more critical. In this article, we delve into why Intel has not prioritized reducing RAM latency more, and how on-chip memory controllers have evolved to address this gap.

The Early Days of Memory Controllers

In the early 2000s, the Intel processors did not even have on-chip memory controllers. The CPU had to communicate with the 'Northbridge' chip, which then communicated with the DRAM. This process was inherently slow. It was only when AMD introduced on-chip memory controllers with its Athlon series that the competitive landscape shifted. Intel eventually responded by integrating three on-chip memory controllers with the Nehalem architecture, significantly reducing latency.

Modern Approaches to Reducing Latency

Today's Skylake EP processors and their ilk boast as many as six on-chip memory controllers, dedicated to making these systems as fast as possible. Despite these advancements, the actual programs that are limited by memory latency are few and far between. The on-chip caches are substantial and sophisticated. Hardware prefetchers, at least three of them, anticipate the next moves based on program behavior, effectively working in parallel. The out-of-order execution engines can process over 100 instructions in the future, further compensating for memory latency.

However, while bandwidth can always be improved through multiple memory controllers, high bandwidth memory (HBM) modules, and future DDR5 technology, RAM latency remains a critical factor. Factors of hardware prefetching, cache management, and out-of-order execution can only go so far in mitigating the impact of high RAM latency.

The Role of In-Order vs. Out-of-Order Designs

There is an underlying truth to the observation that in-order designs are simpler. For instance, the Intel Knights Landing, with up to 72 Atom-derived cores, can handle arithmetic operations efficiently. However, the market's preference for a large number of cheap CPUs with cached memory remains strong, reflecting economic efficiency gains.

It's also worth noting that Seymour Cray once quipped that Apple's Macintosh computers were designed with a Cray while he designed his Cray machines using a Mac. This amusing statement reflects the evolving landscape of computing design, where simplicity and cost-effectiveness are paramount.

Conclusion

The evolution of on-chip memory controllers has been instrumental in reducing RAM latency in modern processors. While Intel continues to integrate more memory controllers and improve bandwidth, the reality is that the benefits of reducing RAM latency are not as universally applicable as might be expected. Nevertheless, continued exploration in this area remains crucial for future advancements in CPU design.