Rate Limiting in AI Gateways: Complete Guide

If I had to boil this down to one point, it’s this: I use rate limits to control traffic, token load, and daily spend before AI usage turns into slow responses, 429 errors, or a big bill.

AI gateway limits are not just about request count. In this article, I’d focus on four checks:

• Requests per second to protect uptime
• Requests per day to split access across users and teams
• Tokens per day to limit heavy text usage
• $ per day to cap pay-as-you-go cost

A few numbers matter right away:

• NanoGPT uses a default global limit of 25 requests per second
• Daily quotas usually reset at 12:00 AM UTC
• A common traffic rule is to start queueing at 90% of provider capacity
• Teams often alert when p99 latency passes 8 seconds or errors go above 2%

Here’s the simple way I’d think about it:

• Use request limits when the problem is traffic volume
• Use token limits when the problem is LLM compute load
• Use spend limits when the problem is billing risk
• Use per-key and per-tenant caps so one user or script does not take over shared capacity
• Use jitter on retries so clients do not hit the gateway again at the same moment

For enforcement, the article covers the main models: fixed window, sliding window, token bucket, and leaky bucket. In most cases, I would not use only one. A mixed setup often works better, such as:

• token bucket for per-second traffic
• fixed window for daily spend caps
• shared counters across nodes, often through Redis, for multi-node gateways

Text and image traffic also need different rules. Text fits token-based limits. Image jobs fit job, resolution, concurrency, or dollar-based limits. And for streaming, retry storms, and multi-tenant traffic, queueing often works better than failing every request at once.

Stop GenAI Rate Limits: Model Routing & Token Throttling with WSO2 AI Gateway

sbb-itb-903b5f2

Quick comparison

Limit or Model	What it controls	Best use
Requests per second	Short-term traffic pressure	Protecting gateway uptime
Requests per day	Daily access	Splitting usage across keys or tenants
Tokens per day	Text model compute	Heavy prompt and response traffic
$ per day	Daily cost	Blocking runaway spend
Fixed window	Simple quota checks	Daily caps
Sliding window	Smoother rolling checks	Reducing window-edge spikes
Token bucket	Burst traffic	Spiky prompts and retries
Leaky bucket	Steady outbound flow	Smoother backend pressure

If you want the short answer, it’s this: I’d match the limit to the workload, watch 429s, latency, queue depth, retries, and daily spend, then tune from traffic data instead of guesswork.

Core Rate Limiting Models and Metrics

Once the gateway knows what to limit, the next step is deciding how to enforce those limits day to day. Most AI gateways don't rely on just one cap. They usually combine request limits, token limits, and spend limits to keep the system steady, control usage, and avoid surprise costs.

Identity-Aware Limits for Users, API Keys, and Tenants

Per-key limits help split traffic across users, teams, and environments, so one traffic spike doesn't eat up shared capacity. A simple setup is to issue separate keys for each context. Production, staging, and development each get their own key and quota.

That way, if a development job goes off the rails, it hits its own limit without affecting production. That's the whole point: contain the blast radius.

When a limit is crossed, the gateway returns an OpenAI-compatible 429 Too Many Requests error with specific codes like daily_rpd_limit_exceeded or daily_usd_limit_exceeded, which makes it much easier for developers to see exactly what happened.

After identity-based controls, the next question is short-term traffic pressure. How much of a spike can the gateway take before things start to slow down?

Burst Capacity vs. Sustained Throughput

Sustained throughput is the steady request rate a gateway can support without stress. Burst capacity is the extra headroom it can use for short spikes. Put simply: sustained limits protect the backend, while burst limits help soak up brief traffic jumps.

A common rule of thumb is to aim for a sustained request rate at about 80% of the provider's hard limit, then begin queueing requests at 90% before the ceiling is reached. This gives the system room to breathe. Instead of failing requests right away, the queue adds backpressure and helps smooth out short bursts.

The Metrics Teams Actually Watch

These controls only work if teams keep an eye on the right signals.

• A rising 429 rate often means limits are too low or traffic is spiking.
• p50 and p99 latency show whether queueing is adding delay.
• Queue depth can warn you before latency starts to slip.
• Retry volume shows whether clients are backing off or just hammering the gateway again and again.

Teams should also track daily USD consumption per key in real time. That's one of the best ways to stop a runaway job before it burns through its cap.

Rate Limiting Algorithms and Their Trade-Offs

Once you know what to limit, the next job is deciding how to enforce it. Get that choice wrong, and you can end up blocking normal users while runaway jobs still sneak through.

Fixed Window and Sliding Window Counters

A fixed window counter tracks requests or tokens inside a set time period, then resets. It’s simple, which is why teams often start here. The catch is the spike at the window edge.

That edge case matters for AI systems. Traffic can jump fast during launches, retries, and chat bursts. A user might pack usage into the end of one window and the start of the next, briefly going past the intended rate without breaking the written rule.

A sliding window counter checks usage over a rolling period instead. That makes enforcement smoother and gets rid of those boundary spikes.

Token Bucket and Leaky Bucket

The token bucket algorithm adds tokens at a steady rate until it hits a maximum capacity. Each request spends tokens. If tokens are still available, short bursts can pass through. That makes it a good fit for bursty prompts, retries, and streaming jobs.

The leaky bucket pushes outbound traffic at a fixed rate. It’s less forgiving with bursts, but it gives you steadier flow, which helps protect serving infrastructure.

Algorithm	Accuracy	Burst Handling	Implementation Complexity	Backend Stability
Fixed window	Low to medium near window boundaries	Weak	Low	Medium
Sliding window	Medium to high	Moderate	Medium	Medium to high
Token bucket	High for burst control	Strong	Medium	High
Leaky bucket	High for traffic smoothing	Limited but predictable	Medium	High

Hybrid Designs for Distributed Gateways

No single algorithm handles every case. Most production AI gateways mix methods instead of betting on just one.

A common setup uses token bucket for per-second throughput and a fixed window for daily USD spend caps. That pairing lets the system absorb short bursts while still enforcing hard cost ceilings on high-cost workloads.

The harder part is keeping limits in sync across nodes. In a horizontally scaled gateway, shared counters - often managed via Redis - enforce limits across multiple instances. When many gateway nodes follow the same policy, mixed algorithms let each node deal with local burst pressure while global counters keep the overall cap in place.

The best fit depends on the limit type, the traffic pattern, and how much burstiness you can live with. Those trade-offs affect latency, cost, and stability at scale.

How Rate Limits Affect Performance, Scalability, and Cost

Those algorithms shape three things fast: latency, scale, and cost.

Latency, Retries, and User Experience

When limits are too tight, users usually notice it as shaky latency first. Requests get throttled, clients retry, and the system can get hit twice for the same work.

The problem gets worse when retry logic fires at the same moment across many clients. Without random jitter, those retries land in a tight cluster and create a retry storm. That second wave can make the original issue worse. Add jitter, and the retries spread out instead of piling on all at once.

Streaming responses are even more fragile. If a rate limit hits in the middle of a stream, the stream may stop outright or return error frames. For users, that can show up as broken or inconsistent UI behavior. And once traffic climbs, this stops being just a UX issue. It becomes a scaling issue too.

Scaling Multi-Tenant AI Traffic Safely

As the number of tenants grows, per-key quotas stop being enough on their own. At that point, queueing often works better than hard fails because it helps keep throughput steady under load.

Priority queuing matters too. Paid users with high-priority requests should move to the front, while batch jobs can wait their turn. That keeps the system from treating every request the same when they clearly don't have the same business value.

"The queue provides backpressure without degrading the experience for your most valuable users." - DeepReviewAI

Text and image workloads also shouldn't sit under the exact same limits. Different models can carry very different costs and throughput profiles, so they need separate controls. That same backpressure isn't just about traffic flow. It also helps keep spend in check.

Using Limits to Control Pay-As-You-Go Spend

With pay-as-you-go billing, cost control becomes part of rate limiting. On platforms like NanoGPT, usage is billed per request with no subscription. So if an API key has no cap, charges can climb fast.

For that setup, rate limits should do more than slow traffic. They should also include a per-key $/day cap and separate caps for costly jobs like image generation.

Backpressure, caching, and routing help on the cost side too - but only when they cut more API calls than they add in infra cost.

Building a Practical Rate Limiting Strategy

A practical rate limiting setup starts with a simple idea: match the limit to the kind of work you're handling, then tune it as traffic shifts.

Text and image traffic don't behave the same way, so they shouldn't share the same guardrails. And budget control needs its own check too. The job here is pretty direct: pick limits by workload first, then refine them using live traffic instead of hunches.

Setting Limits for Text and Image Workloads

Text workloads usually map best to token limits. Image workloads are different, so they fit better with limits based on jobs, resolution, or concurrency.

On top of those workload-based caps, add identity-aware limits. That extra layer matters because it keeps one user, team, or key from eating up more than its share. On NanoGPT, a USD per day spend cap per key is especially useful because it catches runaway costs across both text and image traffic.

There’s also a practical privacy angle here. Identity-aware limits can be enforced with API key metadata and counters alone, without storing full prompt histories centrally, keeping usage tracking functional while remaining privacy-conscious.

Monitoring and Adjusting Limits Over Time

After the limits are in place, the next step is to watch for strain before users notice it.

At a minimum, the review loop should track HTTP 429 rate, peak traffic windows, token burn rate, and retry volume. Those signals tell you where the pressure is building. If 429s stay rare and latency remains flat, you likely have headroom. If 429s jump during certain windows, that's usually a burst problem, which means you may need to raise the burst ceiling or start queuing sooner.

A simple operating rule helps here:

• Queue at 90%, and absorb overflow at 100%.
• Alert when p99 latency goes above 8 seconds or error rates climb past 2%.
• Watch queue depth closely, since a rising queue is often the first sign that traffic is building faster than the system can handle.

The key point is to recalibrate from traffic data, not guesswork. Limits should move with usage, not sit there until something fails. If a key keeps hitting 90% of its daily token budget, it may need a higher quota or a move to a higher tier. If image concurrency limits are almost never touched, they're probably too loose and can be tightened to free capacity for other tenants.

Conclusion: The Simplest Rules That Work

Use the metric that fits the job: tokens for text, jobs or resolution for images, and dollars for spend control. Keep rate limiting tied to workload, cost, and fairness as traffic changes.

FAQs

Which rate limit should I use first?

Start with global throughput limits to cap immediate request volume and keep the system from getting overloaded. That gives you a stable baseline first.

Once traffic settles, add optional per-key daily limits, like requests per day or USD per day, to control how each key is used and how much it can spend.

Using both gives you two layers of control: one for overall performance, and one for usage and cost. If you get close to these limits, handle 429 errors with exponential backoff and jitter.

When should I queue instead of return 429s?

Use queuing instead of returning 429s when you need backpressure without making the user pay for it. A 429 pushes the retry problem to the client. A queue lets you accept the request, then process traffic at a pace your system can handle.

This works well for high-value requests where the user experience matters more than immediate rejection. Instead of saying “try again later,” you keep the request, place it in line, and smooth out the load behind the scenes.

Queuing also helps when some work matters more than other work. For example, you can move high-priority customer requests ahead of lower-priority batch jobs. That keeps urgent work from getting stuck behind background tasks.

The result is steadier traffic and fewer spike-and-crash cycles. Think of it like a metered on-ramp for a busy highway: cars still get on, just not all at once.

How do I set fair limits for different tenants?

Use per-API-key daily limits in the NanoGPT dashboard. Give each tenant a separate API key so you can set different caps for Requests per Day (RPD) and USD per Day.

These counters reset each day at 12:00 AM UTC. If a tenant goes over its limit, the API returns a 429 error. That gives you a simple way to protect system stability and keep resource use fair across tenants.