There are reports that the Meta H100 is experiencing malfunctions every 3 hours.

$NVIDIA (NVDA.US)$During the training of the Llama 3 large-scale language model, it became apparent that Meta was plagued by frequent failures of the H100 GPU. With 16,384 H100 80GB GPUs in training, unexpected component failures were occurring at an average rate of once every 3 hours. More than half of these astonishingly frequent failures were attributable to the GPU or memory.