Galahad vs. vLLM on an NVIDIA A40
A disk-backed KV cache cut median revisit time from 3.29 s to 0.91 s on an NVIDIA A40. Here’s how I measured latency, answer checks, and GPU energy.
Applied AI & Full-Stack Engineer
What happens when an LLM sees the same long prompt again after its key-value cache has been pushed out of GPU memory? I tested whether Galahad’s disk-backed cache could make those repeat requests faster than plain vLLM.
On this workload, Galahad brought the median revisit time from 3.29 seconds to 0.91 seconds, a 3.6× speedup. In a separate pair of runs using the same workload, measured GPU energy fell from 34.81 Wh to 28.03 Wh, a 19.5% reduction. Both configurations passed all 36 answer checks.
How I designed the test
I ran Qwen3-4B-AWQ on an NVIDIA A40 with local NVMe-backed storage. Both configurations used vLLM 0.31.0, a 32,768-token maximum context, and 0.85 GPU memory utilization. The comparison was plain vLLM against the same setup with GalahadConnector (galahad-kv 1.32.0) and a 30 GB disk cache budget.
vLLM reported room for 245,728 tokens in its GPU KV cache. I wanted the benchmark to put that cache under pressure, so I generated 36 distinct prompts totaling 738,909 tokens, roughly three times its capacity. Each prompt was about 20,500 tokens long and contained an eight-character code for the model to find. I used the model’s tokenizer to size the workload.
For each run, I sent all 36 prompts with four requests in flight, then resent the first 12 prompts one at a time. I repeated that sequence three times for each configuration and compared the median of the three run medians.
The timer covered the request through receipt of the full response. For the answer check, the harness looked for the expected code in the returned text. So 36/36 means 12 revisits across three runs passed that check.
Results
| Measurement | Plain vLLM | vLLM + Galahad |
|---|---|---|
| Median revisit time | 3.29 s | 0.91 s |
| Revisit answer checks passed | 36/36 | 36/36 |
| GPU energy across the full benchmark | 34.81 Wh | 28.03 Wh |
| Average GPU power | 280.8 W | 263.3 W |
| Duration of the GPU energy run | 446.2 s | 383.2 s |
I sampled nvidia-smi once per second during the dedicated energy runs, covering the fill and all three revisit runs. Average GPU power was lower and the session finished sooner, so both contributed to the reduction in measured GPU energy.
Why cache size matters
The early prompts were the ones I wanted to revisit after filling the cache. With plain vLLM, their revisit timings were consistent with recomputing the prompt. With Galahad, disk-backed KV data gave the server a way to reuse work after it had been evicted from GPU memory.
The 30 GB disk budget is part of the result. Long prompts consume substantial cache space, so the useful sizing question is how much of the revisited data can survive until the next request. A benchmark that changes prompt length, revisit order, or disk budget can produce a different hit rate and a different latency result.
What this benchmark tells me
For this model, GPU, storage setup, and repeat-heavy workload, disk-backed KV caching made earlier prompts faster to revisit while preserving the measured answer checks. The benchmark measures full-response latency for sequential revisits and GPU energy. It leaves open how much the same setup would help first-time requests, concurrent revisit traffic, or total system energy including storage and CPU.
The repository has the harness, per-prompt timings, server logs, and GPU samples. To reproduce the comparison, start with the same model and serving settings, then vary one parameter at a time. Disk budget and revisit frequency are the first two I’d test on a different workload.