LLM gateway · 2026
Switchyard
A provider-agnostic reverse proxy exposing one stable OpenAI-compatible API on the front and routing to heterogeneous providers on the back, with policy-based routing, circuit breakers, cross-provider fallback, per-tenant rate limiting and a semantic cache.
- Cache hit rate
- 35.2%
- Cost reduction
- 34.7%
- p50 latency reduction
- 32.0%
- Cache precision
- 91-97%
at 40% duplicate rate
at 0.90 threshold
The problem
Anything built on a single LLM provider inherits that provider's outages, rate limits and pricing. Swapping providers usually means touching every call site. A gateway fixes that, but only if it stays out of the way. A proxy that adds latency or silently returns the wrong cached answer is worse than no proxy.
Approach
01One API, many backends
An OpenAI-compatible surface with four routing policies (priority, weighted, latency-aware, cost-aware) defined in a YAML control plane, fronting Groq, Gemini, OpenRouter and Ollama. Per-provider circuit breakers trip on failure and requests fall through to the next backend in the alias, so a provider outage degrades rather than breaks. Two-dimensional per-tenant rate limiting (request and token rates via Redis) keeps tenants from starving each other. SSE streaming passes through intact.
02Semantic caching with a tuned threshold
Requests are embedded with a local MiniLM model and matched against a FAISS index, so near-duplicate prompts hit cache instead of the provider. The similarity threshold is the whole ballgame: at 0.85 the embedding collides genuinely distinct prompts and precision falls to roughly 45%, so it ships at 0.90.
03Benchmarked against a validated simulator
A harness sweeps duplicate rate and threshold to measure hit rate, cost and latency percentiles. The simulator was checked against live runs, reproducing hit rate and cost exactly and p95 within about 0.2%, which made it usable for tuning without burning API spend.
Results
At a 40% duplicate rate the semantic cache served 35.2% of requests, cutting cost 34.7% and p50 latency 32.0%. At a more conservative 20% duplicate rate it still returned 17.5% hit rate and 18.2% cost savings. Holding the threshold at 0.90 kept precision in the 91-97% band at roughly 89% recall, the point where the cache is trustworthy rather than merely effective. The whole stack runs at zero cost on a laptop against provider free tiers.
| Duplicate rate | Hit rate | Cost | p50 | p95 |
|---|---|---|---|---|
| 20% | 17.5% | −18.2% | −9.4% | −3.7% |
| 40% | 35.2% | −34.7% | −32.0% | −15.7% |
What it doesn't do
- Cache precision depends entirely on the embedding model. MiniLM is fast and free but coarse. A stronger embedder would likely allow a lower threshold and a higher hit rate.
- Benchmarks come from a synthetic workload with a controlled duplicate rate. Real traffic distributions are messier and hit rates would differ.
- Tested against four providers. Adding one is a YAML entry plus an adapter, but each has its own quirks around streaming and error semantics.
Next project
Tearsheet