QLoRA fine-tune + serving benchmark · 2026
Featherweight
A QLoRA fine-tune turning Llama-3.1-8B into a function-calling specialist that beats GPT-4o on the Berkeley Function-Calling Leaderboard, then a serving benchmark measuring what that specialization actually costs.
- BFCL V4 non-live AST
- 43.15%89.44%
- Malformed calls
- 9.35%0.40%
- Cost / 1K calls
- $0.083
- Trainable params
- 41.9M
GPT-4o: 87.50%
23× reduction
GPT-4o: $1.347
0.52% of base
The problem
Small open models are cheap but unreliable at tool use: they pick the wrong function, or emit JSON that won't parse. The interesting question isn't whether a fine-tune can close that gap on a benchmark; it's whether the resulting model is actually cheaper to run once you account for serving throughput, and what you give up in exchange.
Approach
01Train narrow, on purpose
QLoRA via Unsloth on 67K verified examples, deliberately mixing tool-use cases with irrelevance cases so the model learns when not to call a function. The adapter is 41.9M parameters, 0.52% of the base model, which is what makes the whole thing runnable on a single T4.
02Benchmark against a real leaderboard
Evaluated on BFCL V4 non-live AST across 1,240 gold cases, scoring the base model, the fine-tune and GPT-4o under the same harness rather than reporting a single number in isolation.
03Then measure the serving cost
Served 4-bit on one T4 under vLLM and ran 200 prompts through both AWQ and bitsandbytes quantization to get cost per 1K calls, throughput and p95 latency, the numbers that decide whether a cheaper model is actually cheaper.
Results
Non-live AST accuracy went from 43.15% to 89.44%, past GPT-4o's 87.50%, and malformed output collapsed from 9.35% to 0.40%, matching GPT-4o exactly and a 23× improvement. Held-out refusal accuracy went from 0.00 to 0.89. On serving, AWQ 4-bit came in at $0.083 per 1K calls against GPT-4o's $1.347, about 16× cheaper, at 1.76× the p95 latency. The quantization choice turned out to matter as much as the fine-tune: bitsandbytes was 5.9× slower and 5.8× more expensive than AWQ for the same weights.
| Model | Accuracy | Malformed | Irrelevance detection |
|---|---|---|---|
| Llama-3.1-8B base | 43.15% | 9.35% | 90.42% |
| GPT-4o | 87.50% | 0.40% | 85.00% |
| Featherweight | 89.44% | 0.40% | 76.25% |
| Configuration | Cost / 1K calls | Throughput | p95 |
|---|---|---|---|
| GPT-4o API | $1.347 | n/a | 3,070 ms |
| vLLM + bitsandbytes 4-bit | $0.4775 | 0.20 req/s | n/a |
| vLLM + AWQ 4-bit | $0.083 | 1.17 req/s | 5,417 ms |
What it doesn't do
- Irrelevance detection regressed from 90.42% to 76.25%, below both the base model and GPT-4o. Training a model to call tools well made it more eager to call them, a real cost of specialization, not a bug to hide.
- p95 latency is 1.76× GPT-4o's. The 16× cost advantage is real but it buys you a slower model.
- Serving figures are marked [FT-equiv] in the repo: they were measured on an equivalent-architecture 4-bit deployment, not the exact adapter build.
Next project
Switchyard