Gemma 4 Technical Report

Google DeepMind’s Gemma Team presents Gemma 4, an open-weight, natively multimodal model family spanning dense and Mixture-of-Experts variants from 2.3B to 31B parameters, plus a 26B MoE with 3.8B active parameters. The report’s main novelty is a unified multimodal design that combines stronger vision and audio encoders with a 12B encoder-free model that ingests raw 40ms audio chunks and image patches directly into the language backbone. Gemma 4 also adds a thinking mode for explicit reasoning traces, speculative decoding via an autoregressive multi-token prediction drafter, quantization-aware training for low-memory deployment, and long-context optimizations including a 5:1 local-to-global attention pattern and KV-cache reuse that cuts global cache footprint by up to 37.5%. On evaluation, Gemma 4 31B reaches 85.2 on MMLU Pro, 89.2 on AIME 2026 without tools, 80.0 on LiveCodeBench v6, and 2150 Elo on Codeforces, while also leading the dense open category on Chatbot Arena with 1451 Elo. The 12B model shows competitive multimodal gains without separate encoders, and the audio stack achieves a 78% reduction in encoder footprint from 390 MB to 87 MB, alongside relative improvements of 12% and 10% on CoVoST translation and 17% and 12% on FLEURS ASR for E2B and E4B. Across long-context tasks, Gemma 4 scales to 128k and even ~256k token evaluations, with RULER accuracy of 96.4 at 128k for 31B and 89.8 for 26B-A4B, demonstrating that Google DeepMind has pushed an efficient open multimodal system into frontier-adjacent territory.