Why I Rewrote Our API Gateway in Rust

Eighteen months ago, our API gateway — the thing standing between every client request and our internal services — was written in Node.js. It worked fine for a long time. Then our traffic tripled, and it stopped working fine.

The problem

At peak load, we were seeing p99 latencies creep past 800ms on a gateway that should have been adding single-digit milliseconds of overhead. CPU usage on our Node instances was pegged at 90%+ even though the actual work — routing, auth checks, rate limiting — was conceptually simple.

The root issue was the event loop. Under heavy concurrent load, a handful of synchronous operations (JSON parsing on large payloads, some regex-heavy routing logic) were blocking the loop just long enough to create a pile-up effect. We tried worker threads, we tried offloading parsing, we tried tuning GC — and clawed back maybe 15% improvement.

Why Rust

We picked Rust for three reasons: predictable performance without a garbage collector, fearless concurrency via tokio, and a type system that catches entire categories of bugs before they hit production. The tradeoff was obvious too — slower initial development, and a steeper learning curve for the team.

async fn route_handler(req: Request) -> Result {
    let route = router.match_path(req.uri())?;
    let auth = authenticate(&req).await?;
    proxy_to_upstream(route, req, auth).await
}

The rewrite took about ten weeks with two engineers, running in shadow mode against production traffic before we ever cut over. That shadow period mattered more than I expected — it caught three edge cases in header handling that would have caused real outages.

The results

Post-migration, p99 latency dropped to under 40ms. Memory usage fell by roughly 70%, and we were able to run the entire gateway tier on a third of the instances we needed before. CPU usage under the same load sits comfortably under 30%.

Was it worth ten weeks of engineering time? For us, yes — this was a critical path service where latency directly affected customer experience. I wouldn't recommend rewriting every Node service in Rust reflexively; but for anything CPU-bound and latency-sensitive sitting on your hot path, it's worth seriously considering.