Rate Limiting & Throttling

Rate limiting protects a service from one client's bad day becoming everyone's. The algorithms are simple; the interesting decisions are what to key on, where to enforce it, and how to tell clients what happened.

The algorithms#

The fixed-window flaw is worth spelling out because it catches people. With a limit of 100 per minute, a client can send 100 requests at 10:00:59 and another 100 at 10:01:00 — 200 requests in one second, entirely within the rules. If your limit exists to protect a downstream system, that burst is exactly what it was supposed to prevent.

Distributed limiting with Redis#

A limiter must share state across instances, and the check-and-increment must be atomic. Lua gives you both:

@Component
public class SlidingWindowRateLimiter {
 
    // Runs atomically inside Redis: trim, count, decide, record.
    private static final String LUA = """
        local key    = KEYS[1]
        local now    = tonumber(ARGV[1])
        local window = tonumber(ARGV[2])
        local limit  = tonumber(ARGV[3])
 
        redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
        local count = redis.call('ZCARD', key)
 
        if count >= limit then
          local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
          return { 0, limit - count, oldest[2] }
        end
 
        redis.call('ZADD', key, now, now .. '-' .. math.random())
        redis.call('PEXPIRE', key, window)
        return { 1, limit - count - 1, 0 }
        """;
 
    private final RedisScript<List> script = RedisScript.of(LUA, List.class);
    private final StringRedisTemplate redis;
 
    public Decision check(String clientId, int limit, Duration window) {
        long now = System.currentTimeMillis();
        try {
            @SuppressWarnings("unchecked")
            List<Long> result = redis.execute(script, List.of("rl:" + clientId),
                    String.valueOf(now), String.valueOf(window.toMillis()), String.valueOf(limit));
 
            return new Decision(result.get(0) == 1L, limit, result.get(1).intValue(),
                                Instant.ofEpochMilli(now).plus(window));
        } catch (RedisConnectionFailureException ex) {
            // Fail OPEN: a monitoring dependency must not take the API down.
            // Fail closed only when the limit protects something you cannot
            // afford to over-serve, such as a metered third-party API.
            log.warn("rate limiter unavailable, allowing request", ex);
            return Decision.allowed(limit, limit, Instant.now().plus(window));
        }
    }
}

That catch block is a genuine design decision, not an oversight. Failing open means a Redis outage removes your protection; failing closed means it removes your service. For a public API, open is almost always right; for a limiter guarding a per-call cost you pay in cash, closed may be.

Bucket4j for token buckets#

@Component
public class RateLimitFilter extends OncePerRequestFilter {
 
    private final ProxyManager<String> buckets;   // Redis-backed
    private final TierResolver tiers;
 
    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
 
        String clientId = clientIdOf(request);
        Tier tier = tiers.resolve(clientId);
 
        Bucket bucket = buckets.builder().build(clientId, () -> BucketConfiguration.builder()
                .addLimit(limit -> limit
                        .capacity(tier.requestsPerMinute())
                        .refillGreedy(tier.requestsPerMinute(), Duration.ofMinutes(1)))
                // A second, tighter limit stops a client spending its whole
                // minute allowance in one second.
                .addLimit(limit -> limit
                        .capacity(Math.max(10, tier.requestsPerMinute() / 10))
                        .refillGreedy(Math.max(10, tier.requestsPerMinute() / 10), Duration.ofSeconds(1)))
                .build());
 
        ConsumptionProbe probe = bucket.tryConsumeAndReturnRemaining(1);
 
        response.setHeader("X-RateLimit-Limit", String.valueOf(tier.requestsPerMinute()));
        response.setHeader("X-RateLimit-Remaining", String.valueOf(probe.getRemainingTokens()));
 
        if (probe.isConsumed()) {
            chain.doFilter(request, response);
            return;
        }
 
        long retryAfterSeconds = Math.max(1, probe.getNanosToWaitForRefill() / 1_000_000_000);
        response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
        response.setHeader("Retry-After", String.valueOf(retryAfterSeconds));
        response.setContentType("application/problem+json");
        response.getWriter().write("""
            {"type":"https://api.acme.com/errors/rate-limited",
             "title":"Too Many Requests","status":429,
             "detail":"Rate limit exceeded. Retry after %d seconds."}
            """.formatted(retryAfterSeconds));
    }
}

The two-tier limit is a pattern worth copying. A single per-minute limit lets a client fire its entire quota in the first second, which produces exactly the load spike the limit was meant to smooth.

What to key on#

Order the filter so authentication runs first where possible — keying on client id is far more precise than keying on IP. For unauthenticated endpoints such as login, IP-based limiting is the only option, and it is exactly where you need it most.

Per-operation cost#

Not all requests are equal. A search that fans out to Elasticsearch costs more than a key lookup. Weighted consumption expresses that:

int cost = switch (operation) {
    case POINT_LOOKUP -> 1;
    case SEARCH       -> 5;
    case EXPORT       -> 50;
};
if (!bucket.tryConsume(cost)) throw new RateLimitExceededException();

This is the difference between a limit that models request count and one that models load.

Tiers and quotas#

Rate limits and quotas are different controls that often get conflated. A rate limit bounds requests per unit of time and exists to protect capacity — it is about smoothing load. A quota bounds total usage over a billing period and exists to enforce a commercial agreement. A client can be well within its rate limit and out of quota, and the two need different responses: 429 with a Retry-After for the first, 403 with a link to upgrade for the second.

Publishing tiers makes both legible. A typical shape is a free tier measured in requests per minute, a paid tier an order of magnitude higher, and an enterprise tier negotiated per contract. Resolve the tier from the authenticated credential rather than from anything the client sends, and cache the lookup — a database read on every request to decide the limit defeats the purpose.

Where the limit protects a downstream you pay for per call, the limit should be derived from that budget rather than chosen by intuition. If your model provider allows 10,000 calls a minute across the account and you have four services sharing it, the sum of your service limits must be under that number, with headroom.

Testing that the limiter works#

Rate limiters are unusually easy to get subtly wrong and unusually hard to notice when you have, because the failure is silent: the limit is simply never enforced. Three tests catch most of it.

Assert that the nth request is rejected — fire limit+1 requests and expect the last to return 429 with a Retry-After. Run this against a real Redis via Testcontainers, not a mock, because the atomicity of the script is exactly what you are testing.

Assert that the limit refills. Advance the clock or wait the window, then confirm requests succeed again. A limiter that rejects forever after the first breach is worse than none.

Assert the concurrency behaviour. Fire the limit's worth of requests in parallel from several threads and confirm the total allowed does not exceed the limit. This is where a non-atomic check-then-increment fails, and it will pass every sequential test you write.

Telling clients how to behave#

Send the limit headers on every response. A well-written client reads X-RateLimit-Remaining and slows down before it gets rejected; one that only ever sees headers on a 429 cannot pace itself at all.

Document the limits per tier, state whether they are per minute or per second, and say explicitly what a client should do on a 429 — honour Retry-After, back off exponentially with jitter, and never retry immediately. Most clients that hammer a rate-limited API do so because nobody told them what the contract was.

What to take away#

Put a coarse limit at the gateway and specific limits on expensive operations. Use a token bucket with a burst sub-limit unless you need strict window semantics. Make the Redis path atomic with Lua, decide the failure mode consciously, and always send the headers a client needs to pace itself.