NVIDIA unveils NVFP4 4-bit pretraining methodology for efficient AI

NVIDIA has successfully pretrained a 12-billion-parameter Mamba-Transformer model on 10 trillion tokens using a new 4-bit precision method. This method, called NVFP4, achieved accuracy nearly identical to its 8-bit counterpart. The development could reshape how organizations approach large language model development by making advanced AI more attainable.

Training large language models traditionally demands immense computational resources and high precision. However, Near-equivalent performance can be achieved with significantly reduced precision and cost using NVIDIA's NVFP4. This challenges a core assumption in AI training, suggesting that high precision is not always necessary for competitive LLM performance at scale.

The industry is likely to see a rapid acceleration in the development and deployment of even larger and more complex AI models. This makes advanced AI more accessible and cost-effective for a broader range of applications, opening doors beyond a handful of hyper-scale players.

How NVIDIA Achieved 4-bit Precision with High Accuracy

• NVFP4 uses a two-level scaling strategy, employing a fine-grained E4M3 FP8 scaling factor for each 16-value micro-block. A second-level FP32 scalar is applied per tensor, according to Developer Nvidia.
• The NVFP4 format has a structure of 1 sign bit, 2 exponent bits, and 1 mantissa bit, supporting a value range of approximately -6 to 6.

This two-level scaling and specific bit allocation are crucial for NVFP4 to maintain numerical stability and precision. It enables efficient operation within a 4-bit format, overcoming traditional low-precision arithmetic challenges.

Setting New Benchmarks for Large-Scale 4-bit Pretraining

NVIDIA validated NVFP4 by pretraining a 12-billion-parameter hybrid Mamba-Transformer on 10 trillion tokens. The longest publicly documented training run in 4-bit precision to date was achieved by NVIDIA's NVFP4, according to MarkTechPost. The resulting NVFP4-trained model achieved 62.58% on MMLU-Pro 5-shot, closely matching the 62.62% of the FP8 baseline.