Small Llm Inference 5 1

The MLCommons organization released the Small LLM Inference 5.1 benchmark, the latest version in the MLPerf Inference series focused on small language models (1-3B parameters). This benchmark aims to evaluate SLM performance in real-world inference scenarios, including tasks such as chatbots and text generation, helping developers select efficient hardware and optimization frameworks.

Benchmark Overview and Updates

The Small LLM Inference benchmark simulates Chatbot Arena-style interactions and supports multiple SLMs such as TinyLlama, Gemma 2 2B, Llama 3.2 1B, and Phi-3.5-mini-instruct. Key updates in version 5.1 include:

• Added Llama 3.2 1B model, supporting extension to vision-language tasks.
• Optimized test scenarios: Prefill and Decode phases evaluated separately, with total tokens reaching 2048.
• Introduced inference engines such as SGLang and vLLM, supporting dynamic batching.
• Expanded hardware scope to data center GPUs (e.g., H100, H200), edge devices (e.g., Jetson Orin), and CPUs.

Tests follow MLPerf rules, divided into closed and open tracks, ensuring fair comparison.

Key Performance Results

In data center scenarios, the closed track's 90th percentile response time and system throughput are key metrics.

Closed Track Leaders

• NVIDIA H200: Llama 3.2 1B model, achieving peak throughput of 1200+ queries/s, a 25% improvement over the previous generation.
• AMD Instinct MI300X: Closely following, with outstanding cost-performance.
• Intel Gaudi3: Submitted for the first time, demonstrating competitiveness.

Open Track Highlights

The SGLang framework leads in decoding efficiency, combined with FlashAttention-2, achieving 1.5x speedup. vLLM performs stably under high concurrency.

Among edge devices, the NVIDIA Jetson AGX Orin excels in real-time response, suitable for mobile AI applications.

Leaderboard and Analysis

For the complete leaderboard, see the official page. These results highlight the importance of quantization (INT4/FP8) and continuous batching, driving the migration of SLMs from cloud to edge. In the future, the benchmark will support more multimodal SLMs.

This release provides valuable reference for AI hardware selection, facilitating efficient SLM deployment.