RNG: Flat Datacenter Networks at Scale

“RNG: Flat Datacenter Networks at Scale,” from Amazon Web Services, presents the first production deployment of a flat expander-based datacenter fabric built from quasi-random graphs. The paper solves three long-standing blockers for expander topologies: scalable routing, practical cabling, and predictable design. Its Spraypoint protocol is a fully distributed, demand-oblivious routing scheme that uses ECMP plus controlled waypoint “spraying” to create close to the maximum number of edge-disjoint paths; in evaluation it finds over 50 edge-disjoint paths for almost all endpoint pairs and over 60 for half of them in a 64-degree fabric, versus a median of 5 for 8-shortest-path routing and 35 for 64-shortest-path routing. For physical deployment, RNG introduces ShuffleBox, a passive optical device that internally shuffles fiber connections so cabling complexity stays comparable to fat trees while preserving expander connectivity. The authors also derive analytic models for path length, oversubscription, and cost, showing that RNG can be 9–45 percent cheaper than equivalent fat trees and, at the same oversubscription ratio, often delivers higher throughput across clique, hub, and matching traffic patterns. In production benchmarks on Amazon’s server mesh and edge mesh, RNG matches fat-tree performance for multipath transport and storage workloads, despite variable path lengths. The key novelty is that Amazon is not just proposing an expander network; it has made one operational at scale and defaulted it for most workloads.