The latest MLPerf Inference 2.1 results demonstrate NVIDIA’s hardware-software co-design delivering unprecedented performance:
- 4.5x speed boost over A100 in data center workloads
- New FP8 precision (E4M3/E5M2) enables 99.9% FP32 accuracy with 2x throughput
- Breakthrough Hopper features:
- Asynchronous transaction barriers for latency reduction
- Tensor Memory Accelerator for efficient data transfers
- Thread block clusters enhancing GPC-level efficiency
Edge AI Advancements with Jetson AGX Orin
- 50% better perf-per-watt vs previous submission
- 17% faster BERT throughput using TensorRT 8.5 optimizations
- Power-saving innovations:
- MaxN power mode frequency boosts
- 64K page size reduces TLB misses
- cuDLA integration for DLA engine improvements
- BERT Inference
- FP8 quantization maintains accuracy without retraining
- Fused multi-head attention (2x speedup)
- Padding removal for compute efficiency
- RetinaNet Object Detection
- Handles 264-class Open Images dataset
- TensorRT-accelerated post-processing with EfficientNMS
- Group convolution optimization for ResNeXt backbone
- 3D U-Net Medical Imaging
5% end-to-end gain via INT8 Linear format plugin
2.7x faster initial convolution layer processing
- Hopper Architecture’s 4th-gen Tensor Cores
- TensorRT 8.5 with DLA-native execution
- L4T image optimizations for edge deployments
- CUDA-X AI software stack enhancements
These results validate NVIDIA’s platform approach – from data center H100 deployments to energy-constrained edge systems using Jetson AGX Orin. The MLPerf 2.1 submission underscores continuous performance scaling through architectural innovation and deep software optimization.
Read more such articles from our Newsletter here.
