Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU

Llamas on the Web introduces LlamaWeb, a WebGPU backend for llama.cpp that targets browser-based LLM inference with static memory planning, performance-portable kernels, and broad quantization support. The key novelty is a design that eliminates dynamic GPU allocation by preallocating all buffers, including a fixed-slot parameter arena and intermediate memory for FlashAttention and FlashDecoding, while streaming GGUF weights asynchronously from the browser’s Origin Private File System without materializing them in the WebAssembly heap. Across 16 devices from 8 vendors and 10 models, LlamaWeb reduces peak browser memory by 29–33 percent relative to WebLLM and Transformers.js, and avoids the memory growth and tab crashes observed in those systems. For performance, it uses a templated shader library that specializes kernels by device features such as subgroups and subgroup matrix operations, and by workload parameters such as tile size, with tuning derived from an exhaustive sweep of thousands of configurations. On four GPUs, this portability layer improves decode throughput by 45–69 percent over WebLLM and Transformers.js, and on some systems it matches or beats native backends: on Apple M4 Pro it is up to 59 percent faster than Vulkan during prefill, and on Intel Arc B580 it outperforms native SYCL by 23 percent in prefill with WebGPU checks disabled. The paper also extends llama.cpp’s quantization ecosystem by implementing templated dequantization for 23 weight formats, including legacy q4_0 and q8_0, K-quants such as q4_k_m, I-quants, and the new q1_0 format, enabling one kernel family to support many compressed models without format-specific rewrites.