AMD Instinct™ MI300 Product Analysis

1. The advantage of AMD MI300A compared with GH200 is larger memory, but the computing power has not been tested. It may be used as a substitute for GH200 in the future.

2. The advantage of AMD MI300X is also larger memory, but it can reduce the number of GPUs and the power loss caused by data migration during training.

3. At present, although AMD has tried to have a breakthrough in software, the public believes that its impact on NVIDIA's existing software services is not obvious.

AMD has officially launched its Instinct MI300 APU, which combines 24 Zen 4 CPU cores and a CDNA 3 GPU core along with up to 153 billion transistors and 128 GB of HBM3 memory.

Because its CPU and GPU share a single cache memory, the MI300A is able to significantly reduce data movement between the CPU and GPU. This is important because data movement typically consumes more energy than the actual computation, which results in increased latency and reduced performance and energy efficiency. By minimizing data movement, the MI300 is able to deliver impressive performance while keeping power consumption to a minimum.

The Instinct MI300A also uses a chip to achieve such a CPU+GPU, but NVIDIA's GH200 is integrated at the module level, and the Instinct MI300A is implemented with a smaller chip, which is an advantage.

Karl Freund, founder and principal analyst at Cambrian AI Research LLC, said the MI300X could become an alternative to Nvidia's GH200 Grace Hopper superchip. Companies like OpenAI and Microsoft need to have such alternatives, and while he doubts AMD will make these companies an offer they can't refuse, AMD won't take much market share away from Nvidia.

With the large memory of the AMD Instinct MI300X, customers can now install large language models, such as Falcon-40, 40B parametric models, on a single MI300X accelerator. AMD also introduced the AMD Instinct™ platform, which combines eight MI300X accelerators into one industry-standard design to provide the ultimate solution for AI reasoning and training. The MI300X began delivering samples to key customers in the third quarter.

The AMD Instinct MI300X has 192GB HBM3, 5.2TB/s memory bandwidth and 896GB/s Infinity Fabric bandwidth. This is a 153B transistor unit that is directly benchmarked to NIVIDIA's H100.

The advantage of having a lot of onboard memory is that AMD requires fewer Gpus to run models in memory and can run larger models in memory without having to connect to other Gpus or CPU links via NVLink. There is a huge opportunity in the market to run large AI inference models and more GPU memory, with larger and more accurate models that can run entirely in memory without the power consumption and hardware costs of spanning multiple Gpus.

Although AMD's new chip has aroused great interest from all sides of the market, compared with Nvidia's H100 chip, the MI300X faces some challenges, mainly in the following four areas:

First, Nvidia's H100 starts full shipping today; By far, Nvidia still has the largest ecosystem of software and researchers in the AI industry.

Second, while the MI300X chip offers 192GB of RAM, Nvidia will be catching up very quickly at this point, and possibly even in the same time frame, so it's not a big advantage. And the MI300X price will be very high, and there will be no significant cost advantage over Nvidia's H100.

The third is the real key: the MI300 doesn't have a Transformer Engine like the H100 (a library for accelerating Transformer models on Nvidia Gpus), which can triple the performance of large language models (LLM). If it takes a year to train a new model with a few thousand (Nvidia's) Gpus, it may take another 2-3 years to train with AMD hardware, or three times as many Gpus to solve the problem.

Finally, AMD has yet to disclose any benchmarks. But the performance when training and running LLM depends on the system design and GPU, so look forward to seeing some comparisons with competitors in the industry later this year.

In the latest performance comparison, AMD demonstrated that the Instinct Mi300 has an 8-fold improvement in AI performance (TFLOPs) and a 5-fold improvement in AI performance per watt (TFLOPs/watt) compared to the Instinct MI250X. AMD's Instinct MI300A APU accelerator is out now, and the MI300X will be out in the third quarter of 2023. Both products are also expected to go into production in the fourth quarter of 2023.

In fact, Nvidia's leadership comes not only from its chips, but also from the software tools it has provided AI researchers for more than a decade. Anshel Sag, an analyst at Moor Insights & Strategy, said: "Even though AMD is competitive in terms of hardware performance, people still don't believe its software solutions can compete with Nvidia.