Our rough estimate of targets includes traditional HPC (based upon past trends over the past 20 years) systems at the 10–20 FP64 exaflops level and beyond in the 2025+ timeframe and 100+ FP64 exaflops and beyond in the 2030+ timeframe through hardware and software acceleration mechanisms.
This is roughly 8 times more than 2022 systems in 2026 and 64 times more in 2030. For AI applications, we would be interested in BF16/FP16 performance projections, based on current architectures, and would expect additional factors of 8 to 16 times or beyond the FP64 rates for lower precision.
If you are cynical like that – and sometimes you have to be when facing down the slowing of Moore’s Law – then you can already get 2X performance with today’s technology just by scaling the power. Frontier weights in at 29 megawatts when fully burdened, so the first 4 exaflops is easy. Just double the size of the system and count on software engineers to figure out how to scale the code.
Money is a real issue as far as we are concerned. The rest is just engineering around the money. Let’s take the inflation adjusted budgets for the machines at Oak Ridge for the past decade against their peak FP64 performance:
The 1.75 petaflops “Jaguar” system cost $82 million per petaflops
The 17.6 petaflops “Titan” system cost $6.5 million per petaflops
The 200 petaflops “Summit” machine cost a little more than $1 million per petaflops
The new 2 exaflops “Frontier” machine cost $400,000 per petaflops.
$500 million for a 10 exaflops machine, based on the accelerated architectures outlined above, in 2027 and $250 million for one in 2032. That also implies $2.5 billion for a 100 exaflops machine in 2032. And that implies a GPU accelerated architecture.
You can see why everyone wants to figure out how to use AI to stretch physics simulations and to use AI’s mixed precision engines to goose HPC performance, and you can see why the big machines are based on accelerators for the most part using some sort of accelerator.
A lot of high-end machines have been tested using the HPL-AI benchmark and its iterative solvers, which have been refined since that time. On the June HPL-AI list, the refined solvers are able to get a 9.5X speedup on Summit, making it behave like a 1.41 exaflops machine.
On Frontier, which is a completely different architecture, a dominate slice of the machine is rated at 1.6 exaflops peak, 1.1 exaflops on HPL, and 6.86 exaflops on HPL-AI, and there is no reason to believe the effective flops can be boosted by around 10X, which has happened on a machines with very different architectures.
Either way, can’t just be thinking of the actual and effective FP64 rates on these machines. The mic drop in that paragraph from the RFI above is the need to train AI models with 100 trillion parameters.
Right now, the GPT-3 model with 3 billion parameters is not particularly useful, and more and more AI shops are training the GPT-3 model with 175 billion parameters.
The hyperscalers are trying to crack 1 trillion parameters, and 100X more than that sounds absolutely as insane as it is probably necessary given what HPC centers are trying to do for the next decade.This is going to be a lot harder than slapping together 100,000 GPU accelerators with 100 million SMs, with 1.6 Tb/sec Ethernet and InfiniBand interconnects, with CXL 5.0 links linking together CPU hosts and maybe a dozen GPUs into a node using 1 TB/sec PCI-Express 8.0 x16 links. But that’s probably where we would start and work our way backwards from there.
You must be logged in to post a comment.