vLLM vs llama.cpp on an RTX 3090: vLLM 2.5× faster at 32 requests, close at one

· in self-hosted AI

Tested on Fedora 44 with Docker 29.9.2 and an RTX 3090 (24 GB), comparing vLLM 0.33.0 (Qwen3.5 9B AWQ 4-bit) with llama.cpp v1.0.0, build 12359 (Qwen3.5 9B Q4_K_M GGUF), on 16 November 2026.

Results card, vLLM vs llama.cpp v1.0 on an RTX 3090 with Qwen3.5 9B: speed with one request is close, 115.3 tokens/s for vLLM (first word after 0.06 s) and 121.6 for llama.cpp with its defaults (first word after 0.14 s); with 32 requests at once, vLLM produced 1,730.0 tokens/s in total running up to 32 at once, llama.cpp 679.4 with 32 slots and 322.9 with its default 4 slots; the longest wait for the first word with 32 requests was 0.6 s on vLLM, 2.7 s on llama.cpp with 32 slots and 22.5 s with 4 slots; GPU memory 12.7 GiB for vLLM with a 60% memory limit (92% by default) and 10.1 to 15.1 GiB for llama.cpp with 16 to 32 slots, 13.6 with its defaults; ready 72 s after docker run for vLLM with a warm compile cache and 2 to 5 s for llama.cpp with any slot setting. Verdict: llama.cpp for one user or a few clients, vLLM when many requests arrive at once.

This test ran vLLM and llama.cpp v1.0 on one RTX 3090 with Qwen3.5 9B, sending 1 to 32 requests at once.

  • One request at a time: close. llama.cpp wrote 121.6 tokens per second and vLLM 115.3, 5% apart.
  • 32 requests at once: vLLM produced 1,730.0 tokens per second in total, 2.5× llama.cpp with 32 slots, and started every answer within 0.6 s.
  • llama.cpp's defaults: only four requests run together, so the last of 32 waited 22.5 s. Setting -np (short for --parallel, the number of requests llama.cpp works on at once, called slots) fixes most of that.
  • Start-up: llama.cpp was ready in 2 to 5 s, vLLM in 72 s.

vLLM and llama.cpp in short

llama.cpp is a C/C++ library for running language models, and llama-server is its HTTP server with an OpenAI-compatible API. It reads GGUF files and runs on NVIDIA, AMD and Intel GPUs, Apple silicon, Vulkan and plain CPUs. Ollama is built on it: since version 0.30, Ollama runs llama.cpp's server underneath, which the Ollama vs llama.cpp comparison measured on this same machine.

vLLM is an inference server built to serve one model to many users at once. Its README names its two main techniques: PagedAttention, which stores each request's working memory in small blocks, and continuous batching, which adds new requests to the batch the GPU is already working on. It loads models in the Hugging Face format, and for 4-bit weights it uses formats such as AWQ and GPTQ.

llama-server can also work on several requests at once. Its README lists "parallel decoding with multi-user support" and continuous batching, which is on by default, and the -np option sets how many requests it works on together. This test measured llama.cpp with its defaults and with 16 and 32 slots.

The short answer

Speeds are in tokens per second (tok/s), and higher is faster. Both servers got the same prompts and wrote the same number of tokens per request (see how the test was run). The marker shows which tool came out ahead.

vLLMllama.cpp
Speed, one request (short prompt)115.3 tok/s; first word after 0.06 s121.6 tok/s with its defaults; first word after 0.14 s ▲ 5%
Total output, 32 requests at once1,730.0 tok/s ▲ 2.5×679.4 tok/s with -np 32, 322.9 with its default 4 slots
Longest wait for the first word, 32 requests0.6 s ▲ 2.1 s sooner2.7 s with -np 32, 22.5 s with its default 4 slots
GPU memory12.7 GiB with a 60% limit; reserves 92% of the card by default10.1 to 15.1 GiB, depending on -np and -c similar
Ready after docker run72 s with its compile cache in place2 to 5 s ▲ about a minute sooner

Verdict: for most home servers, run llama.cpp: one user or a few clients get answers at least as fast as from vLLM, and it starts in seconds. Run vLLM when many requests really do arrive together, and set -np on llama.cpp if several clients overlap.

Is vLLM faster than llama.cpp?

It depends on how many requests arrive at once. For one request at a time, no: llama.cpp generated text a little faster in every single-request test here. vLLM was faster from four requests up, and the gap grew with every step. At 32 requests at once, vLLM's total output was 2.5 times that of the best llama.cpp setting, and each request in the batch still ran at 61.1 tokens per second, against 26.7 on llama.cpp.

The figures are for a 9B model in 4-bit on one consumer card. Red Hat's tests on a data-center H200 found a larger gap at 64 users, with vLLM's output about 44 times llama.cpp's. Their earlier article lists the llama.cpp options it set: threads and GPU layers, not the number of slots.

How the test was run

  • Machine: Fedora 44, Docker 29.9.2, an NVIDIA RTX 3090 with 24 GB (driver 615.78.08), an AMD Ryzen 9 5900X and 62 GB of RAM. The desktop used about 2.5 GiB of the GPU's memory during the test. GPU memory on this page is in GiB, as nvidia-smi and vLLM report it.
  • Servers: vLLM 0.33.0 from the vllm/vllm-openai:v0.33.0 image, and llama.cpp v1.0.0 (build 12359, released on 13 November 2026) from the ghcr.io/ggml-org/llama.cpp:server-cuda-v1.0.0-b12359-b1b770d image. The llama.cpp v1.0 article measures what changed from the build before it.
  • Models: Qwen3.5 9B in each server's usual 4-bit format. llama.cpp loaded Qwen3.5-9B-Q4_K_M.gguf (5.7 GB). vLLM loaded cyankiwi/Qwen3.5-9B-AWQ-4bit (9.1 GB). They are different files, so this compares two ways of running the same model at 4 bits. Answer quality was not compared.
  • vLLM settings: --max-model-len 32768 --gpu-memory-utilization 0.6 --max-num-seqs 32. The default memory setting would not start with the desktop's memory in use; the section on memory explains why.
  • Requests: a short prompt of about 70 tokens and a long prompt of about 4,120 tokens. One request at a time generated 512 tokens on the short prompt and 256 on the long one (256 on the short prompt for the -np 16 run). With several requests, each generated 256 tokens. Both servers got the same prompts and the same output limit (max_tokens), and every answer on both ran to that limit, so both wrote exactly the same number of tokens per request. The servers counted 64 to 75 prompt tokens for the short prompt and 4,116 to 4,125 for the long one, on both tools; the small spread comes from the random ID. Every prompt started with a random ID, so neither server could reuse a cached prompt. Thinking was off and the temperature was 0.
  • Method: the same Python client sent the same requests to both servers' /v1/chat/completions endpoint and timed them on the client side. Both servers ran one after the other in one session on 16 November, with every other container on the machine stopped; a log taken after each test shows nothing else on the GPU. Before each round, the client waited until the GPU had cooled to 50 °C. Single-request results are the median of five rounds, and results with several requests are the median of three.

The test scripts and raw results are on GitHub. Facts about the tools link to the page they come from. Figures without a link were measured in this test.

One request at a time: close, with llama.cpp slightly ahead

MeasurevLLMllama.cpp defaultsllama.cpp -np 32
Generation, short prompt (tok/s)115.3121.6 ▲ 5%121.3 ▲ 5%
Generation, long prompt (tok/s)112.5119.1 ▲ 6%119.3 ▲ 6%
First word, short prompt0.06 s ▲ 0.08 s sooner0.14 s0.14 s
First word, long prompt (4,120 tokens)1.00 s ▲ 0.04 s sooner1.04 s1.05 s

For one person chatting with a model, the two servers were close. llama.cpp generated 5 to 6% more tokens per second, with either slot setting. vLLM started its answer slightly sooner, but on a short prompt both started within 0.14 s.

Many requests at once: vLLM produced 2.5 times as much at 32

A home server often has more than one client: a chat window, an agent and a script, all sending requests to one model. In this test, 4 to 32 requests started at the same moment, and the table shows the total output per second until the last one finished. The marker compares vLLM with the best llama.cpp setting in the same row.

Requests at oncevLLMllama.cpp defaults (4 slots)llama.cpp -np 16llama.cpp -np 32
1113.9117.7 close117.3117.8
4397.6 ▲ 1.2×332.1332.6330.3
8771.1 ▲ 1.5×329.5498.0497.5
161,282.5 ▲ 2.0×327.2635.6630.5
321,730.0 ▲ 2.5×322.9617.3679.4
Bar chart of total output in tokens per second at 1, 4, 8, 16 and 32 requests at once. vLLM: 113.9, 397.6, 771.1, 1282.5, 1730.0. llama.cpp defaults: 117.7, 332.1, 329.5, 327.2, 322.9. llama.cpp -np 32: 117.8, 330.3, 497.5, 630.5, 679.4.
Total output as more requests arrive at once. vLLM's total kept rising up to 32 requests, while llama.cpp with its default four slots stayed near 330 tokens per second.

The single-request row is lower than the generation speeds above because it includes the time to read the prompt.

Why: with its defaults, llama-server picked four slots by itself (n_slots = 4 in its log), so four requests ran together and the rest waited in a queue. That is why its total did not grow from 4 requests to 32.

Setting -np raises the number of slots. With 16 or 32 slots, llama.cpp roughly doubled its total at 16 requests, and with 32 slots it reached 679.4 tokens per second at 32 requests.

vLLM's lead came from how much faster each request ran inside a big batch. At 32 requests, each vLLM request generated about 61 tokens per second, while each llama.cpp request generated about 27. vLLM was built for this case. llama.cpp handles it, but less efficiently.

How to set llama.cpp's slots

Three llama-server options decide how it shares the GPU between requests. All three are in the llama-server README:

  • -np (--parallel) sets the number of slots. The default of -1 lets the server choose; it chose 4 here.
  • -c (--ctx-size) sets the context size. With a unified cache, this is one pool that all slots share.
  • --kv-unified makes all slots share that one pool instead of splitting it evenly. It is on by default only when the number of slots is automatic, so add it when you set -np yourself.

The tested settings were -np 32 -c 262144 --kv-unified and -np 16 -c 131072 --kv-unified.

Tip: pick a slot count close to the number of requests you expect at the same moment. With 16 slots and 32 requests, half of them waited, and the last one waited 7.8 s for its first word.

Waiting time: vLLM started every request within 0.6 s

Total output tells you how much work the server gets done. Waiting time tells you how the server feels to the person behind the last request in the queue. The table shows the longest time any request waited for its first word, and the marker compares vLLM with llama.cpp -np 32.

Requests at oncevLLMllama.cpp defaults (4 slots)llama.cpp -np 32
80.2 s ▲ 0.4 s sooner3.4 s0.6 s
160.3 s ▲ 1.0 s sooner9.7 s1.4 s
320.6 s ▲ 2.1 s sooner22.5 s2.7 s
Bar chart of the longest wait for the first word at 8, 16 and 32 requests at once. vLLM: 0.2, 0.3 and 0.6 seconds. llama.cpp defaults: 3.4, 9.7 and 22.5 seconds. llama.cpp -np 32: 0.6, 1.4 and 2.7 seconds.
The longest wait for the first word of an answer. With four slots, llama.cpp made the last of 32 requests wait 22.5 s.

With long prompts the waits grow on both servers, because each prompt has to be read before its answer starts. With 8 requests of about 4,120 tokens each, the last request waited 7.9 s on vLLM and 8.8 s on llama.cpp with 32 slots. Totals were 191.0 and 162.8 tokens per second.

GPU memory and start-up time: similar memory, llama.cpp starts in seconds

SetupGPU memoryReady after docker run
vLLM, --gpu-memory-utilization 0.612.7 GiB72 s
llama.cpp defaults13.6 GiB4.6 s
llama.cpp -np 16 -c 13107210.1 GiB4.8 s
llama.cpp -np 32 -c 26214415.1 GiB2.3 s
Bar chart of GPU memory used: vLLM with a 0.6 limit 12.7 GiB, llama.cpp defaults 13.6 GiB, llama.cpp -np 16 10.1 GiB, llama.cpp -np 32 15.1 GiB, against the card's 24 GiB.
GPU memory held by each server with the 9B model loaded. The vLLM figure is with its memory limit lowered to 60%.

vLLM decides how much memory to take before it loads anything. By default it reserves 92% of the card and fills what the model doesn't need with cache. On the first day of testing, the desktop and a browser held about 2 GiB, and vLLM refused to start:

ValueError: Free memory on device cuda:0 (21.58/23.61 GiB) on startup is less than
desired GPU memory utilization (0.92, 21.72 GiB). Decrease GPU memory utilization or
reduce GPU memory used by other processes.

Tip: on a home server that also drives a screen or runs a second model, lower the limit with --gpu-memory-utilization. In an earlier run on the same machine, 0.6 alone made vLLM 0.33.0 refuse again on this hybrid model, because its default of 256 sequences needed more cache blocks than fit. Adding --max-num-seqs 32 fixed that.

In this setting vLLM held 12.7 GiB, and its speed matched an earlier run with the full reservation: about 115 tokens per second for one request and about 1,730 for 32.

llama.cpp works the other way: it loads the model and sizes the cache from -c and -np, and its --fit option (on by default) adjusts unset values to fit the card. It never asked for more than 15.1 GiB here.

Start-up differs even more. llama-server answered requests 2 to 5 s after the container started. vLLM compiles the model and records CUDA graphs before it serves anything, which took 72 s with its compile cache already on disk. In an earlier run, the first start without that cache took 173 s, and changing a vLLM argument triggered a fresh compile.

Model formats: GGUF for llama.cpp, AWQ for vLLM

The two servers read different files. llama.cpp uses GGUF, a single file with the weights and the chat template. vLLM uses the Hugging Face safetensors format, so a 4-bit model for vLLM is a separate AWQ or GPTQ upload. vLLM can load GGUF, but its docs call that support highly experimental and under-optimized, and it has moved to a separate plugin. This test did not try it.

A 4-bit file is not always the same size on both sides. The AWQ file here is 9.1 GB against 5.7 GB for the Q4_K_M GGUF, because its recipe leaves the embeddings, the output layer and the linear-attention layers unquantized. That recipe uses the AWQ method, and the weights are stored in the compressed-tensors format (its config.json says "quant_method": "compressed-tensors"); vLLM recognised it without a --quantization flag. The model was downloaded once from Hugging Face, at revision e4d2d39, into the local cache that the container mounts. Check the file size, not only "4-bit", when you size a model for your card.

Commands used in this test

Both run in Docker with an NVIDIA GPU once the NVIDIA Container Toolkit is installed. Bind the ports to 127.0.0.1, as below, so other machines on your network can't reach them.

llama.cpp with 32 slots, serving a GGUF file from ~/models. The tag pins v1.0.0, the build tested here. The plain server-cuda tag is rebuilt every day from the newest build, so pin a tag if you want the results above.

docker run -d --name llama --gpus all -p 127.0.0.1:8080:8080 \
  -v ~/models:/models:ro ghcr.io/ggml-org/llama.cpp:server-cuda-v1.0.0-b12359-b1b770d \
  -m /models/Qwen3.5-9B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 \
  -np 32 -c 262144 --kv-unified

vLLM with the lower memory limit. The first volume shares your Hugging Face download cache, and the second keeps the compile cache between restarts:

docker run -d --name vllm --gpus all --ipc=host -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/vllm-cache:/root/.cache/vllm \
  vllm/vllm-openai:v0.33.0 --model cyankiwi/Qwen3.5-9B-AWQ-4bit \
  --max-model-len 32768 --gpu-memory-utilization 0.6 --max-num-seqs 32

Both then answer on /v1/chat/completions. Qwen3.5 thinks before it answers unless you turn it off. Both servers accept "chat_template_kwargs": {"enable_thinking": false} in the request; vLLM documents it, and llama-server passed it to the model's template in this test.

Neither server asked for a password here. Set --api-key on llama-server or vLLM if anything else on your network can reach the port, and keep the port closed on your router; the port forwarding guide explains what an open port exposes.

Day-to-day differences

vLLMllama.cpp
Best atMany requests at onceOne to a few requests, fast start-up
Model filesHugging Face safetensors; AWQ or GPTQ for 4-bitGGUF
HardwareNVIDIA, AMD and Intel GPUs, and x86, ARM and Apple silicon CPUsNVIDIA, AMD and Intel GPUs, Apple silicon (Metal), Vulkan, and CPUs
GPU memoryReserves 92% of the card unless told otherwiseTakes what the model, -c and -np need
Start-up in this test72 s with a warm compile cache2 to 5 s
LicenseApache-2.0MIT

Which one to run on a home server

For most home servers, llama.cpp is the better fit. One person, or a few clients that rarely send requests at the same moment, will get answers at least as fast as from vLLM. llama.cpp starts in seconds and takes only the memory it needs, so it shares the card with a desktop or a second program. If several clients do overlap, set -np to about the number of requests you expect at once and add --kv-unified. If you want model switching and downloads by name on top, Ollama runs the same engine.

vLLM is worth its start-up time and memory planning when many requests really do arrive together: a family or small team sharing one model, an agent that sends dozens of calls in parallel, or batch jobs. At 32 requests at once it did 2.5 times the work of llama.cpp and kept every user's wait under a second. Give it a memory limit if anything else uses the GPU.

Frequently asked questions

Is vLLM faster than llama.cpp?

For one request at a time, no: on an RTX 3090 with Qwen3.5 9B, llama.cpp v1.0 generated about 122 tokens per second and vLLM about 115. With 32 requests at once, vLLM produced about 1,730 tokens per second in total, 2.5 times llama.cpp with 32 slots.

Can llama.cpp serve several users at once?

Yes. llama-server has parallel slots and continuous batching, and the -np option sets the number of slots. Its default chose four slots in this test, so set -np and --kv-unified if more requests arrive together.

Why does vLLM use so much GPU memory?

vLLM reserves 92% of the card by default and fills the space the model doesn't need with cache for requests. Lower it with --gpu-memory-utilization; at 0.6, with --max-num-seqs 32, it used 12.7 GiB in this test.

Can vLLM run GGUF files?

vLLM's docs call its GGUF support highly experimental and under-optimized, and it has moved to a separate plugin. For GGUF files, llama.cpp is the better fit; for vLLM, use an AWQ or GPTQ upload of the model.

Which one starts faster?

llama.cpp. llama-server was ready 2 to 5 s after its container started, while vLLM took 72 s even with its compile cache already on disk.

Does Ollama use llama.cpp?

Yes. Since version 0.30, Ollama runs llama.cpp's server underneath and reads GGUF files.

Sources

  1. llama.cpp README at v1.0.0 (backends, license)
  2. llama.cpp: llama-server README at v1.0.0
  3. llama.cpp v1.0.0 release notes
  4. llama.cpp license
  5. llama.cpp Docker build workflow (daily image rebuild)
  6. vLLM on GitHub (README and license)
  7. vLLM 0.33.0 release notes
  8. vLLM 0.33.0 source: default GPU memory utilization
  9. vLLM docs: GGUF
  10. vLLM docs: installation
  11. vLLM docs: reasoning outputs
  12. vLLM docs: OpenAI-compatible server
  13. Ollama 0.30.0 release notes
  14. NVIDIA Container Toolkit install guide
  15. Qwen3.5-9B GGUF files (unsloth)
  16. Qwen3.5-9B AWQ 4-bit (cyankiwi)
  17. Red Hat Developers: llama.cpp vs. vLLM (2026)
  18. Red Hat Developers: vLLM or llama.cpp (2025)