Mr. Musk's xAI, 'Colossus,' is reportedly planning to expand to 10 times its current size, equipped with 1 million NVIDIA GPUs - FT report.
The Financial Times (FT) reported on the 5th local time that the AI startup xAI led by Elon Musk is planning to expand the existing supercomputer 'Colossus' to ten times its current size.
According to the plan, the expanded system is expected to have 1 million NVIDIA GPUs installed. The expansion project has already started in Memphis, Tennessee, and NVIDIA, Dell Technologies $DELL, Super Micro Computer $SMCI are planning to establish a local base to support the project.
The current Colossus is equipped with over 0.1 million NVIDIA GPUs and was built in just 3 months earlier this year. Jensen Huang, CEO of NVIDIA $NVDA, commented to the newspaper, 'Building a datacenter of this scale typically takes 3 years.'
FT estimates that hundreds of billions of dollars in investment will be needed for new GPU procurement. xAI has raised about $11 billion this year and is valued at $45 billion. In terms of competitors, OpenAI has formed a partnership with Microsoft for about $14 billion, while Anthropic recently received an $8 billion investment from Amazon to secure access to over 100,000 AI chips.
On the other hand, concerns have been raised about the validity of regulatory approval procedures and the impact on the regional electrical grid regarding the same project. In response, xAI's Brent Mayo explains that they will ensure the stability of the electrical grid using Megapack technology, as reported by FT.
📍 The significant investment by xAI raises questions about 'scale' and 'efficiency'.
In terms of improving AI model performance, the 'scaling law' is unavoidable. The three elements of model size, dataset, and computational resources are closely related, and the fact that increasing computational resources boosts performance is well known.
However, there are limitations to this approach. As demonstrated in DeepMind's 'Chinchilla' paper, increasing model size does not necessarily lead to efficient performance improvements. Rather, there is a possibility of inefficiency due to 'overparameterization' lacking balance with data volume. The massive investment by xAI may not just be about pursuing scale, but also about achieving this 'optimal balance' as an experimental ground.
Efficient utilization of computational resources requires optimization of model architecture. As evidenced by the evolution of Transformers, improving computational efficiency is closely related to algorithm and architecture innovation. Additionally, the efficiency of distributed learning systems and improvement in data quality also play important roles.
From these perspectives, xAI's efforts may aim to achieve a new level of efficiency beyond just 'scaling up'. In particular, this investment seems to embody the character of a project exploring next-generation AI development methods, rather than just a simple enhancement of computational resources.
The huge investment by xAI may bring changes to the entire AI industry ecosystem. Competitors like OpenAI and Anthropic are also securing vast computational resources through partnerships with NVIDIA, Microsoft, and Amazon. This competition holds the potential to shape a new order in the AI industry.
Noteworthy is the strengthening of the competitive structure of the "resource concentration" by such efforts. As companies with vast computational resources hold market dominance, how small and medium players demonstrate their presence will be an important theme in the future. Moreover, the impact this will have on the overall AI industry will become clear within a few years.
For investors, what is important is not only the success or failure of such large-scale projects. It is crucial to identify the fundamental question of "scale expansion versus efficiency" underlying these endeavors. Regardless of whether the bet on AI is successful or not, this attempt will provide new insights into the evolution of the entire AI industry. We need to maximize the insights obtained through this process.
According to the plan, the expanded system is expected to have 1 million NVIDIA GPUs installed. The expansion project has already started in Memphis, Tennessee, and NVIDIA, Dell Technologies $DELL, Super Micro Computer $SMCI are planning to establish a local base to support the project.
The current Colossus is equipped with over 0.1 million NVIDIA GPUs and was built in just 3 months earlier this year. Jensen Huang, CEO of NVIDIA $NVDA, commented to the newspaper, 'Building a datacenter of this scale typically takes 3 years.'
FT estimates that hundreds of billions of dollars in investment will be needed for new GPU procurement. xAI has raised about $11 billion this year and is valued at $45 billion. In terms of competitors, OpenAI has formed a partnership with Microsoft for about $14 billion, while Anthropic recently received an $8 billion investment from Amazon to secure access to over 100,000 AI chips.
On the other hand, concerns have been raised about the validity of regulatory approval procedures and the impact on the regional electrical grid regarding the same project. In response, xAI's Brent Mayo explains that they will ensure the stability of the electrical grid using Megapack technology, as reported by FT.
📍 The significant investment by xAI raises questions about 'scale' and 'efficiency'.
In terms of improving AI model performance, the 'scaling law' is unavoidable. The three elements of model size, dataset, and computational resources are closely related, and the fact that increasing computational resources boosts performance is well known.
However, there are limitations to this approach. As demonstrated in DeepMind's 'Chinchilla' paper, increasing model size does not necessarily lead to efficient performance improvements. Rather, there is a possibility of inefficiency due to 'overparameterization' lacking balance with data volume. The massive investment by xAI may not just be about pursuing scale, but also about achieving this 'optimal balance' as an experimental ground.
Efficient utilization of computational resources requires optimization of model architecture. As evidenced by the evolution of Transformers, improving computational efficiency is closely related to algorithm and architecture innovation. Additionally, the efficiency of distributed learning systems and improvement in data quality also play important roles.
From these perspectives, xAI's efforts may aim to achieve a new level of efficiency beyond just 'scaling up'. In particular, this investment seems to embody the character of a project exploring next-generation AI development methods, rather than just a simple enhancement of computational resources.
The huge investment by xAI may bring changes to the entire AI industry ecosystem. Competitors like OpenAI and Anthropic are also securing vast computational resources through partnerships with NVIDIA, Microsoft, and Amazon. This competition holds the potential to shape a new order in the AI industry.
Noteworthy is the strengthening of the competitive structure of the "resource concentration" by such efforts. As companies with vast computational resources hold market dominance, how small and medium players demonstrate their presence will be an important theme in the future. Moreover, the impact this will have on the overall AI industry will become clear within a few years.
For investors, what is important is not only the success or failure of such large-scale projects. It is crucial to identify the fundamental question of "scale expansion versus efficiency" underlying these endeavors. Regardless of whether the bet on AI is successful or not, this attempt will provide new insights into the evolution of the entire AI industry. We need to maximize the insights obtained through this process.
参考
Disclaimer: Community is offered by Moomoo Technologies Inc. and is for educational purposes only.
Read more
Comment
Sign in to post a comment
Magnificent7 : It's an endless battle of AI's peak
What about japanese companies?
Kimihiko OP Magnificent7 : softbank group co
Jamaica no problem : I feel a mix of end-of-the-century and futuristic vibes. Through nvidia, it seems like incredible items and weapons are about to appear.
Magnificent7 Kimihiko OP : It's the only choice, right?
Magnificent7 Jamaica no problem : I feel like watching a Matrix movie after a long time
Jamaica no problem Magnificent7 :![[undefined]](https://static.moomoo.com/nnq/emoji/static/image/default/default-black.png?imageMogr2/thumbnail/36x36 "undefined")
NOIR(のあ) Magnificent7 : I also support SBG. It's been quite a long relationship. In japan, SBG also has ARM under its umbrella, and is making efforts with AX Integrated. The business model is changing rapidly and becoming specialized in AI.