Ollama vs llama.cpp vs LM Studio vs vLLM: Qwen3.8-27B on one RTX 3090

· in self-hosted AI

Tested on Fedora 44 with Docker 29.9.2, an RTX 3090 (24 GB) and a Ryzen 9 5900X, comparing Ollama 0.40.15, llama.cpp v1.0.0 (build 12359), LM Studio's llmster server with CUDA runtime 2.79.0 and vLLM 0.33.0, with Qwen3.8-27B (Q4_K_M GGUF; Ollama's own Q4_K_M file for its MTP download; AWQ 4-bit for vLLM), on 20 November 2026.

Results card, Ollama vs llama.cpp vs LM Studio vs vLLM with Qwen3.8-27B on one RTX 3090. One request at a 32k context, 70-token prompt and 512-token answer: Ollama 43.5 tokens/s, and 57.1 with its MTP download; llama.cpp 43.7, and 71.1 with MTP; LM Studio 65.6 with MTP on by default; vLLM 43.3, because MTP did not fit. Largest context on the GPU for one request: Ollama 98,304 tokens, and 61,440 with its MTP download; llama.cpp 106,496, the largest, and 65,536 with MTP; LM Studio 65,536 with MTP on by default; vLLM 32,768, the smallest. 32 requests at once, total output, 70-token prompts and 256-token answers: Ollama 96.3 tokens/s working on 4 requests at a time with 8k each, and 38.1 with its default of 1 at a time, the lowest; llama.cpp 295.4 working on 32 at a time sharing a 32k context, and 117.9 with its default of 4 at a time; LM Studio 138.9 with its default of 4 at a time sharing 8k; vLLM 345.8 working on up to 32 at a time with at most 8k per request, the highest. Eight long prompts at once, 4,121-token prompts and 256-token answers: Ollama 41.3 tokens/s with 4 at a time and 8k each, all answered; llama.cpp 48.3 with 4 at a time sharing 97.8k, but it failed with 32 at a time; LM Studio failed, because its 4 requests at a time share an 8k context; vLLM 47.5 with at most 8k per request, all answered. Ready after docker run: Ollama 88 s until the first answer, the slowest; llama.cpp 4.0 s, the fastest; LM Studio 8.3 s with the model loaded; vLLM 46 s, and 155 s on the first start. Verdict: one user: LM Studio, or llama.cpp with MTP. Many users at once: vLLM, or llama.cpp at -np 32.

This test ran Ollama, llama.cpp, LM Studio and vLLM one after another on the same RTX 3090, with Qwen3.8-27B in each, and sent each the same requests: one at a time, then up to 32 at once.

  • One request at a time: with a 70-token prompt and without speculative decoding, Ollama, llama.cpp and vLLM wrote 43.3 to 43.7 tokens per second with the same 32k context. The context size did not change the speed: llama.cpp wrote 43.7 at 32k, 96k and 106,496 tokens.
  • Speculative decoding with MTP: LM Studio turned it on by itself and wrote 65.8 tokens per second with its default 8k context. llama.cpp v1.0 with LM Studio's settings and MTP wrote 70.9, also at 8k, and Ollama with its separate MTP download 57.1 at 32k.
  • 32 requests at once: with 70-token prompts and 256-token answers, vLLM produced the most, 345.8 tokens per second in total, working on up to 32 at a time with at most 8k of context per request. llama.cpp working on 32 at a time with a shared 32k context produced 295.4, LM Studio working on 4 at a time with a shared 8k context 138.9, and Ollama at most 96.3, working on 4 at a time with 8k each.
  • 24 GB is the limit: the largest context that fit was 106,496 tokens on llama.cpp, 65,536 with MTP, and 32,768 on vLLM. Three setups whose shared context was smaller than eight long prompts need failed when those prompts arrived together.
  • Start-up: llama.cpp was ready in 4 s and LM Studio in about 8 s. vLLM took 46 s, and Ollama answered its first request after 88 s.

The four servers in short

Three of the four run the same engine. llama.cpp is a C/C++ program, and its llama-server has an OpenAI-compatible API. Since version 0.30, Ollama runs llama.cpp's server underneath, and the Ollama log in this test showed it starting llama-server with the model file. LM Studio downloads llama.cpp as a separate runtime; the CUDA runtime used here, version 2.79.0, is llama.cpp build 12521. All three read GGUF files, so this test loaded one GGUF file into all three.

vLLM is a separate engine, built to serve one model to many users at once. It loads models in the Hugging Face format, so for 4-bit weights it needs an AWQ or GPTQ upload instead of a GGUF file.

Ollamallama.cppLM StudiovLLM
Enginellama.cppllama.cppllama.cppits own
Model file in this testGGUF (imported)GGUFGGUF (linked)AWQ 4-bit
Version tested0.40.15v1.0.0 (build 12359)llmster, runtime 2.79.00.33.0
LicenseMITMITProprietary, free for personal and internal business useApache 2.0

The licenses are from Ollama's, llama.cpp's and vLLM's license files and from LM Studio's app terms. Earlier tests on this site compared the servers in pairs with Qwen3.5: Ollama vs llama.cpp, LM Studio vs Ollama, vLLM vs Ollama and vLLM vs llama.cpp.

The short answer

Speeds are in tokens per second (tok/s), and higher is faster. Each cell gives the server's defaults first, then the best setting tested where it differs.

Ollamallama.cppLM StudiovLLMBest
Speed, one request (70-token prompt, 512-token answer)43.5 tok/s with a 32k context
57.1 tok/s with the MTP download and a 32k context
43.8 tok/s with a 97.8k context
70.9 tok/s with MTP and an 8k context
65.8 tok/s with MTP (on by default) and an 8k context43.5 tok/s with an 8k context; MTP did not fitllama.cpp with MTP, then LM Studio
First word (70-token prompt)0.39 s0.27 s0.27 s0.12 svLLM
Total output, 32 requests at once (70-token prompts, 256-token answers)38.1 tok/s with 1 request at a time and a 32k context
96.3 tok/s with 4 requests at a time and an 8k context each
117.9 tok/s with 4 requests at a time sharing a 97.8k context
295.4 tok/s with 32 requests at a time sharing a 32k context (-np 32)
138.9 tok/s with 4 requests at a time sharing an 8k context345.8 tok/s with up to 32 requests at a time and at most 8k of context per requestvLLM; llama.cpp with -np 32 15% behind
Eight long prompts at once (4,121-token prompts, 256-token answers)All answered with 1 request at a time and a 32k context, and with 4 requests at a time and an 8k context eachAll answered with 4 requests at a time sharing a 97.8k context
Failed with 32 requests at a time sharing a 32k context (-np 32)
Failed with 4 requests at a time sharing an 8k contextAll answered with up to 32 requests at a time and at most 8k of context per requestOllama, llama.cpp defaults, vLLM
GPU memory17.0 GiB with a 32k context21.5 GiB with a 97.8k context18.2 GiB with an 8k context22.0 GiB with an 8k contextOllama
Ready after docker run88 s to the first answer4.0 s8.3 s46 s; 155 s on the first startllama.cpp

Verdict: for one person chatting with Qwen3.8-27B, run LM Studio, or llama.cpp with MTP turned on: on the short prompt both wrote 50 to 63% faster at an 8k context than Ollama with its defaults or vLLM, and 15 to 24% faster than Ollama's MTP download. For many requests at once, run vLLM, or llama.cpp with -np 32, which came within 15% of it at 32 requests and starts in seconds.

How to read the table: Ollama, llama.cpp and LM Studio loaded the same GGUF file, so their numbers compare the servers alone. vLLM loaded a different file with the same model in 4 bits (AWQ), so its numbers compare vLLM with an AWQ file against the others with a GGUF file. "With MTP" means speculative decoding with the prediction layer that is built into Qwen3.8. "4 at a time" means the server works on 4 requests together, and a fifth request waits until one of them finishes. "Shared" means all the requests in progress draw from one context of that size, so one request can use all of it, but the requests together can't use more. "Per request" or "each" means every request has its own limit of that size, however many are running. In the results tables, green and red arrows compare a row with the same server's defaults row, or in the MTP table with the row that has the same settings without MTP: green is better, red is worse. LM Studio turns it on by default. llama.cpp ran it only when it was turned on, Ollama only with its separate MTP download, which uses Ollama's own GGUF file, and vLLM ran without it.

Why Qwen3.8-27B, and how big is it?

The earlier comparisons on this site used Qwen3.5 9B, a model that leaves plenty of a 24 GB card free. Qwen3.8-27B is a larger and newer model, and it fills most of the card, so it shows where each server runs out of memory. Its model card lists 27B parameters and a native context of 262,144 tokens. It mixes two kinds of layers: of every four, three are Gated DeltaNet layers and one is a Gated Attention layer. Gated DeltaNet is a form of linear attention. It was also trained with multi-token prediction (MTP), and it reads images and video as well as text. The license is Apache 2.0.

In 4 bits, the files used here were:

On the GPU, the servers used 16.0 to 22.0 GiB with the model loaded, depending on how much context they kept (see GPU memory). The 4-bit model runs on a 24 GB card, but a long context or many requests at once take most of what is left.

How the test was run

  • Machine: Fedora 44, Docker 29.9.2, an NVIDIA RTX 3090 with 24 GB (driver 615.78.08), an AMD Ryzen 9 5900X and 62 GB of RAM. The desktop held about 1.2 GiB of the GPU's memory. GPU memory on this page is in GiB, as nvidia-smi reports it.
  • Servers: each ran in Docker from its official image: ollama/ollama:0.40.15, ghcr.io/ggml-org/llama.cpp:server-cuda-v1.0.0-b12359-b1b770d, lmstudio/llmster-preview:cpu with LM Studio's CUDA 12 runtime 2.79.0, and vllm/vllm-openai:v0.33.0.
  • Same file: Ollama imported the GGUF with a one-line Modelfile (FROM the file), and LM Studio linked it with lms import. Ollama's copy had the same SHA-256 as the original file. The one exception is Ollama's MTP download, qwen3.8:27b-mtp-q4_K_M, which comes with Ollama's own 16.8 GB Q4_K_M file.
  • Settings: each server ran with its defaults, plus the settings named on each row. vLLM ran with --max-model-len 8192 --gpu-memory-utilization 0.93 --max-num-seqs 32 --max-cudagraph-capture-size 32 --language-model-only, because 32k did not fit when the test started; the section on GPU memory explains why.
  • Requests: a short prompt of 70 tokens and a long prompt of 4,121 tokens; the servers counted 70 to 73 and 4,120 to 4,122 tokens, depending on their chat template. One request at a time generated 512 tokens on the short prompt and 256 on the long one. With several requests, each generated 256 tokens. Every prompt started with a random ID, so no server could reuse a cached prompt. Thinking was off and the temperature was 0.
  • Method: one Python client sent the requests and timed them, through each server's own API for Ollama and LM Studio and through /v1/chat/completions for llama.cpp and vLLM. The servers ran one at a time on 20 November, and the runs at 32k, 96k and the largest context on 20 and 21 November, with no other server container running. Before each round, the client waited until the GPU had cooled to 50 °C. Single-request results are the median of five rounds, and results with several requests are the median of three.

Answer quality was not compared. The test scripts and raw results are on GitHub. Facts about the tools come from the pages listed under Sources, and every other figure was measured in this test.

One request at a time: the context size did not change the speed, MTP did

The context is the most tokens one request can hold, prompt and answer together. The first table runs each server with its defaults, so the contexts differ. The next three set every server to the same context: 32k, 96k, and the largest that fit with the whole model on the GPU. Each server kept its default number of requests at a time, and only one request was sent.

Each server's defaults

SetupContextSpeed, 70-token prompt, 512-token answerSpeed, 4,121-token prompt, 256-token answerFirst word, 70-token promptFirst word, 4,121-token promptGPU memory
Ollama32,76843.5 tok/s42.8 tok/s0.39 s3.26 s17.0 GiB
Ollama, MTP download32,76857.1 tok/s (▲ 31%)65.0 tok/s (▲ 52%)0.43 s (▲ 10%)3.63 s (▲ 11%)19.5 GiB
llama.cpp97,79243.8 tok/s43.0 tok/s0.27 s3.02 s21.5 GiB
LM Studio (MTP on)8,19265.8 tok/s76.6 tok/s0.27 s3.79 s18.2 GiB
vLLM (context set in this test)8,19243.5 tok/s42.7 tok/s0.12 s3.35 s22.0 GiB

32k context (32,768 tokens) on every server

SetupSpeed, 70-token prompt, 512-token answerSpeed, 4,121-token prompt, 256-token answerFirst word, 70-token promptFirst word, 4,121-token promptGPU memory
Ollama (its default)43.5 tok/s42.8 tok/s0.39 s3.26 s17.0 GiB
Ollama, MTP download (its default)57.1 tok/s (▲ 31%)65.0 tok/s (▲ 52%)0.43 s (▲ 10%)3.63 s (▲ 11%)19.5 GiB
llama.cpp43.7 tok/s43.0 tok/s0.24 s2.99 s17.5 GiB
llama.cpp with MTP71.1 tok/s (▲ 63%)79.7 tok/s (▲ 85%)0.25 s (▲ 4%)3.09 s (▲ 3%)19.9 GiB
LM Studio (MTP on)65.6 tok/s76.9 tok/s0.27 s3.77 s19.9 GiB
vLLM43.3 tok/s42.4 tok/s0.11 s3.36 s22.0 GiB
Bar chart of writing speed for one request with a 32,768-token context on every server, in tokens per second. 70-token prompt, 512-token answer: Ollama 43.5, Ollama MTP download 57.1, llama.cpp 43.7, llama.cpp with MTP 71.1, LM Studio 65.6, vLLM 43.3. 4,121-token prompt, 256-token answer: Ollama 42.8, Ollama MTP download 65.0, llama.cpp 43.0, llama.cpp with MTP 79.7, LM Studio 76.9, vLLM 42.4.
Writing speed for one request, with the same 32k context on every server. Without MTP, Ollama, llama.cpp and vLLM were within 0.6 tokens per second of each other.

96k context (98,304 tokens) on every server

SetupSpeed, 70-token prompt, 512-token answerSpeed, 4,121-token prompt, 256-token answerFirst word, 70-token promptFirst word, 4,121-token promptGPU memory
Ollama43.4 tok/s42.7 tok/s0.37 s3.23 s21.1 GiB
Ollama, MTP downloadDid not fit: Ollama moved 19% of the model to the CPU
llama.cpp43.7 tok/s42.9 tok/s0.24 s2.99 s21.5 GiB
llama.cpp with MTPDid not fit
LM Studio (MTP on)Did not fit
vLLMDid not fit: it needed 6.32 GiB for this context and had 0.97 GiB

The largest context that fit on the GPU

SetupLargest contextSpeed, 70-token prompt, 512-token answerSpeed, 4,121-token prompt, 256-token answerFirst word, 70-token promptFirst word, 4,121-token promptGPU memory
Ollama98,30443.4 tok/s42.7 tok/s0.37 s3.24 s21.1 GiB
Ollama, MTP download61,44057.6 tok/s64.4 tok/s0.41 s3.60 s21.4 GiB
llama.cpp106,49643.7 tok/s42.9 tok/s0.25 s3.00 s22.1 GiB
llama.cpp with MTP65,53670.3 tok/s80.6 tok/s0.25 s3.10 s22.0 GiB
LM Studio (MTP on)65,53665.3 tok/s76.0 tok/s0.27 s3.78 s22.0 GiB
vLLM32,76843.3 tok/s42.4 tok/s0.11 s3.36 s22.0 GiB

The largest context was searched in steps of 4,096 tokens, and a setup counted only when the whole model stayed on the GPU. The desktop held 0.9 to 1.4 GiB of the card during these runs, so on a card without a desktop each server would fit somewhat more.

Without MTP, the context size made no difference to writing speed. llama.cpp wrote 43.7 tokens per second on the short prompt at 32k, 96k and 106,496 tokens, and Ollama 43.4 to 43.5 at 32k and 96k. A larger context only took more GPU memory: llama.cpp held 17.5 GiB at 32k and 22.1 GiB at 106,496 tokens.

MTP was the one setting that changed the speed, and it costs context. llama.cpp with MTP wrote 63% faster than without it at the same 32k, but its largest context fell from 106,496 to 65,536 tokens. LM Studio, which turns MTP on by default, also stopped at 65,536.

The wait for the first word was close. On the short prompt, vLLM started its answer after 0.11 to 0.12 s, and Ollama after 0.37 to 0.43 s. On the long prompt, all four needed about 3 to 4 s to read the 4,121 tokens before the first word.

Speculative decoding with MTP: why LM Studio is the fastest out of the box

Speculative decoding lets the model guess several tokens ahead and then check the guesses in one step. Qwen3.8 has a small prediction layer for this built in, called MTP. The llama.cpp v1.0 test explains how it works and measured it on Qwen3.5-27B.

Why: LM Studio turned MTP on without being asked. Its llama-server ran with --spec-type draft-mtp --spec-draft-n-max 3, and its log showed 44 to 47% of the guessed tokens accepted. MTP explains most of the gap between LM Studio's 65.8 tokens per second at 8k and the 43.8 of llama.cpp with its defaults at 97.8k; the table below compares the two at the same 8k.

To check this, llama.cpp v1.0 ran with LM Studio's exact settings (8k context, 4 requests at a time, --kv-unified, the same batch sizes), once with MTP and once without:

Setup (all: 8,192 tokens of context shared, 4 at a time)One request, 70-token prompt, 512-token answerOne request, 4,121-token prompt, 256-token answer4 requests at once, total, 70-token prompts, 256-token answers32 requests at once, total, same prompts
llama.cpp, LM Studio's settings, no MTP43.7 tok/s43.0 tok/s122.5 tok/s118.2 tok/s
llama.cpp, LM Studio's settings, MTP70.9 tok/s (▲ 62%)78.1 tok/s (▲ 82%)164.7 tok/s (▲ 34%)160.1 tok/s (▲ 35%)
LM Studio, defaults65.8 tok/s76.6 tok/s143.2 tok/s138.9 tok/s

The results:

  • MTP helped most with one request: llama.cpp wrote 62% faster on the short prompt and 82% faster on the long one, probably because the long prompt asks for a summary, which repeats words from that prompt.
  • It helped less with many requests: 34 to 35% more total output at 4 and 32 requests, because the GPU is already busy with several requests at once.
  • llama.cpp v1.0 was slightly ahead of LM Studio: 8% on the short prompt and 2% on the long one, even though LM Studio ran the same flags on a newer llama.cpp build (12521 against 12359).

The other two servers:

  • vLLM: its MTP setting is --speculative-config '{"method":"mtp","num_speculative_tokens":3}', and the AWQ file includes the MTP layer. vLLM ran out of GPU memory while building the MTP layer: it tried to take another 2.37 GiB with about 0.8 GiB free, in both tries. On a 24 GB card, vLLM ran this model without MTP.
  • Ollama: the GGUF imported with a Modelfile ran without MTP. Ollama's library has a separate qwen3.8:27b-mtp-q4_K_M download, and with it Ollama 0.40.15 started its llama-server with --spec-type draft-mtp --spec-draft-n-max 4 and 1 request at a time. It wrote 57.1 tokens per second on the short prompt and 65.0 on the long one, 31% and 52% faster than the imported file without MTP. Its log counted 41% of the guessed tokens accepted over the whole run. This download uses Ollama's own 16.8 GB Q4_K_M file, not the unsloth file, so the gap is not from MTP alone, and this test did not find out why it stayed behind llama.cpp and LM Studio with MTP.

llama.cpp v1.0 with MTP had one problem. After the 32-request test, the next request stopped the server with failed to remove sequence 0. It happened again when the test was repeated. LM Studio's newer build did not fail in the same test.

Many requests at once: vLLM produced the most, llama.cpp running 32 at a time close behind

A home server often has more than one client: a chat window, an agent and a script, all sending requests to one model. In this test, 4 to 32 requests, each with a 70-token prompt and a 256-token answer, started at the same moment, and the table shows the total output per second until the last one finished.

SetupContext4 at once8 at once16 at once32 at once
Ollama, defaults (1 at a time)32,76841.0 tok/s40.6 tok/s39.7 tok/s38.1 tok/s
Ollama, MTP download (1 at a time)32,76853.2 tok/s (▲ 30%)51.9 tok/s (▲ 28%)51.3 tok/s (▲ 29%)49.3 tok/s (▲ 29%)
Ollama, 4 at a time (OLLAMA_NUM_PARALLEL=4)4 × 8,192100.6 tok/s (▲ 145%)100.3 tok/s (▲ 147%)98.4 tok/s (▲ 148%)96.3 tok/s (▲ 153%)
llama.cpp, defaults (4 at a time)97,792 shared122.1 tok/s121.4 tok/s119.8 tok/s117.9 tok/s
llama.cpp, 32 at a time (-np 32)32,768 shared121.4 tok/s (▼ 1%)194.8 tok/s (▲ 60%)264.2 tok/s (▲ 121%)295.4 tok/s (▲ 151%)
llama.cpp with MTP (4 at a time)8,192 shared164.7 tok/s (▲ 35%)164.4 tok/s (▲ 35%)162.2 tok/s (▲ 35%)160.1 tok/s (▲ 36%)
LM Studio, defaults (4 at a time, MTP)8,192 shared143.2 tok/s143.0 tok/s142.5 tok/s138.9 tok/s
vLLM, 32 at a time8,192 per request150.6 tok/s287.9 tok/s282.8 tok/s345.8 tok/s
Bar chart of total output in tokens per second, 70-token prompts and 256-token answers. 8 requests at once: Ollama defaults 40.6, Ollama MTP download 51.9, Ollama 4 at a time 100.3, llama.cpp defaults 121.4, llama.cpp 32 at a time 194.8, llama.cpp with MTP 164.4, LM Studio 143.0, vLLM 287.9. 32 requests at once: Ollama defaults 38.1, Ollama MTP download 49.3, Ollama 4 at a time 96.3, llama.cpp defaults 117.9, llama.cpp 32 at a time 295.4, llama.cpp with MTP 160.1, LM Studio 138.9, vLLM 345.8.
Total output with 8 and 32 requests at once. Only vLLM and llama.cpp running 32 at a time kept growing past four requests.

A server's total grows only while it can add requests to the batch. Ollama with its defaults runs one request at a time, so its total stayed near its single-request speed; with the MTP download, it stayed at 49.3 to 53.2 tokens per second. Ollama set to 4 at a time, and llama.cpp and LM Studio with their default of 4 at a time, all stopped growing at four requests, and the other requests waited in a queue. With four requests, Ollama's default of 1 at a time made the last one wait 19.1 s for its first word.

vLLM and llama.cpp with -np 32 could run all 32 together. At 32 requests, vLLM produced 17% more than llama.cpp. Once a request had started writing, it wrote 33.2 tokens per second on vLLM and 12.1 on llama.cpp; the total is lower than 32 times that, because requests started at different times. vLLM's total did not grow from 8 to 16 requests. In the 16-request test, the median request waited 0.9 s for its first word and the longest 8.7 s, so vLLM probably held some requests back until others finished.

Setting -np is the change that matters on llama.cpp: -np 32 raised its total at 32 requests from 117.9 to 295.4 tokens per second. On Ollama, OLLAMA_NUM_PARALLEL does the same job.

Waiting for the first word

Total output shows how much work gets done. The wait shows how the server feels to the person behind the last request in the queue.

Setup, 32 requests at once (70-token prompts, 256-token answers)ContextMedian waitLongest wait
Ollama, defaults (1 at a time)32,768100.6 s209.0 s
Ollama, MTP download (1 at a time)32,76879.4 s (▼ 21%)162.2 s (▼ 22%)
Ollama, 4 at a time4 × 8,19237.6 s (▼ 63%)75.9 s (▼ 64%)
llama.cpp, defaults (4 at a time)97,792 shared30.8 s61.6 s
llama.cpp, 32 at a time (-np 32)32,768 shared6.6 s (▼ 79%)6.9 s (▼ 89%)
llama.cpp with MTP (4 at a time)8,192 shared19.8 s (▼ 36%)45.8 s (▼ 26%)
LM Studio, defaults (4 at a time, MTP)8,192 shared26.4 s53.5 s
vLLM, 32 at a time8,192 per request9.3 s17.4 s
Bar chart of the median wait for the first word with 32 requests at once, 70-token prompts and 256-token answers: Ollama defaults 100.6 s, Ollama MTP download 79.4 s, Ollama 4 at a time 37.6 s, llama.cpp defaults 30.8 s, llama.cpp 32 at a time 6.6 s, llama.cpp with MTP 19.8 s, LM Studio 26.4 s, vLLM 9.3 s.
Median wait for the first word with 32 requests at once. llama.cpp running 32 at a time started every request within 6.9 s.

llama.cpp running 32 at a time had the shortest waits: every request started within 6.9 s. vLLM's median was 9.3 s and its longest 17.4 s. Working on 1 at a time, Ollama made the last of 32 requests wait 209.0 s.

Eight long prompts at once: three setups failed

Eight requests with the 4,120-token prompt and a 256-token answer need about 35,000 tokens of context in total. This test shows how each server handles more context than it has.

SetupContextResultTotal output, 8 × 4,121-token prompts, 256-token answers
Ollama, defaults32,768, one request at a timeAll answered26.6 tok/s
Ollama, MTP download32,768, one request at a timeAll answered31.6 tok/s (▲ 19%)
Ollama, 4 at a time4 × 8,192All answered41.3 tok/s (▲ 55%)
llama.cpp, defaults97,792 shared, 4 at a timeAll answered48.3 tok/s
llama.cpp, 32 at a time (-np 32)32,768 sharedFailednone
llama.cpp, LM Studio's settings8,192 shared, 4 at a timeFailednone
LM Studio, defaults8,192 shared, 4 at a timeFailednone
vLLM8,192 per request, 30,840 in totalAll answered47.5 tok/s

Key point: a server that shares one context between all the requests it works on at a time fails when the requests together need more than that context. llama.cpp answered "Context size has been exceeded", and no request in the batch got an answer, in any of the three rounds. Ollama and vLLM queued the extra requests instead.

LM Studio's default 8,192-token context holds one long prompt, so one long request at a time worked. With --kv-unified, size llama.cpp's -c for the requests you expect together: eight of these requests need about 35,000 tokens, more than the 32,768-token context that fit on this card with 32 at a time.

GPU memory: what fits in 24 GB

SetupContextGPU memory
Ollama, defaults32,768, 1 at a time17.0 GiB
Ollama, MTP download32,768, 1 at a time19.5 GiB
Ollama, 4 at a time4 × 8,19217.5 GiB
llama.cpp, defaults97,792 shared, 4 at a time21.5 GiB
llama.cpp, 32 at a time (-np 32)32,768 shared21.7 GiB
llama.cpp, LM Studio's settings, no MTP8,192 shared, 4 at a time16.0 GiB
llama.cpp, LM Studio's settings, MTP8,192 shared, 4 at a time18.2 GiB
LM Studio, defaults (MTP)8,192 shared, 4 at a time18.2 GiB
vLLM8,192 per request, 0.93 memory limit22.0 GiB
Bar chart of GPU memory held by each server, against the card's 24 GiB: Ollama defaults 17.0 GiB, Ollama MTP download 19.5, Ollama 4 at a time 17.5, llama.cpp defaults 21.5, llama.cpp 32 at a time 21.7, llama.cpp with MTP 18.2, LM Studio 18.2, vLLM 22.0.
GPU memory held by each server with Qwen3.8-27B loaded. MTP added about 2.3 GiB (2,336 MiB) to llama.cpp.

Each server sizes its context differently:

  • Ollama picks the default context from the card's memory: 32k on cards with 24 to 48 GiB, and 4k below 24 GiB.
  • llama.cpp fits the context to the free memory when you don't set it. Here it chose 97,792 tokens and 4 requests at a time.
  • LM Studio loads with an 8,192-token context and 4 requests at a time unless you change them.
  • vLLM reserves a share of the card first and fills what the model doesn't need with cache.

vLLM had the least room, and its room changed between starts. With 18.28 GiB of weights, --max-model-len 32768 and --gpu-memory-utilization 0.93, the most this card allowed with the desktop running, it first had 1.03 GiB of cache and said the longest possible request was 11,760 tokens. The main runs therefore used --max-model-len 8192, which still fits the long prompt. In the later context runs, the same 32,768 setting started in three tries out of three, with 43,690 to 44,373 tokens of cache, and 36,864 did not fit. This test did not find out why the first start had less room.

The number of requests vLLM runs at once has a second limit on this model. Each running request needs one block for the linear-attention layers, and on its first start, with an empty compile cache, vLLM refused --max-num-seqs 32: "max_num_seqs (32) exceeds available Mamba cache blocks (21)". It started with 16. With the compile cache in place, the same command with 32 worked.

Tip: when you set llama.cpp's -c yourself, check that the whole model stayed on the GPU. With -np 32 -c 49152, llama.cpp v1.0 started without an error, but it had moved layers to the CPU, and it wrote 11.5 tokens per second instead of 43.7. Adding -ngl all --fit off keeps every layer on the GPU, and then the server fails to load if the context doesn't fit.

Start-up time: llama.cpp in 4 s, Ollama's first answer after 88 s

ServerTime after docker run
llama.cpp, defaults4.0 s until ready
LM Studio3.9 s until the server answered, then 4.4 s to load the model
vLLM, compile cache on disk46.4 s until ready
vLLM, first start155.1 s until ready
Ollama, defaults5.5 s until the API answered, 88.4 s until the first answer
Ollama, MTP download5.2 s until the API answered, 60.5 s until the first answer
Bar chart of seconds from docker run to a working server: Ollama until the first answer 88.4 s, llama.cpp 4.0 s, LM Studio with the model loaded 8.3 s, vLLM with the compile cache on disk 46.4 s, vLLM on its first start 155.1 s.
Time from docker run until each server could answer. Ollama loads the model on the first request, so its bar runs to the first answer.

Ollama's API answered after 5.5 s, but Ollama loads a model only when the first request arrives. That first answer came 88.4 s after docker run. Ollama counted 32.1 s of it as loading the model, and its log did not break down the rest. Set to 4 at a time, Ollama gave its first answer after 85.7 s, and with the MTP download 60.5 s, of which Ollama counted 31.5 s as loading. Later requests did not wait, because the model stayed loaded. vLLM compiles the model on its first start and keeps the result in a cache folder, so mount that folder to keep later starts near 46 s.

Which one to run on a home server

  • One person chatting: LM Studio, because MTP is on by default and it wrote 65.8 tokens per second without any settings. llama.cpp is as fast or faster with --spec-type draft-mtp, but v1.0 crashed with MTP after a burst of 32 requests in this test.
  • A few clients at once: llama.cpp or LM Studio, which work on 4 requests at a time by default, or Ollama with OLLAMA_NUM_PARALLEL=4. Ollama's default of 1 at a time made the last of four requests wait 19.1 s.
  • Many requests at once: vLLM for the highest total, or llama.cpp with -np 32, which came within 15% and started every request sooner. Give either one a context that fits all the requests you expect.
  • Long prompts from several clients: Ollama or vLLM, which queued requests that didn't fit, or llama.cpp with a -c large enough for all of them.
  • The fewest settings: Ollama. It picked a 32k context for this card and kept the whole model on the GPU, at 43.5 tokens per second. Its qwen3.8:27b-mtp-q4_K_M download turns on MTP and wrote 57.1.

Commands used in this test

All four run in Docker with an NVIDIA GPU once the NVIDIA Container Toolkit is installed. The ports are bound to 127.0.0.1, so other machines on your network can't reach them. The GGUF file is in ~/models.

llama.cpp v1.0 with MTP, using LM Studio's settings:

docker run -d --name llama --gpus all -p 127.0.0.1:8080:8080 \
  -v ~/models:/models:ro ghcr.io/ggml-org/llama.cpp:server-cuda-v1.0.0-b12359-b1b770d \
  -m /models/Qwen3.8-27B-UD-Q4_K_M.gguf --host 0.0.0.0 --port 8080 \
  -c 8192 -np 4 --kv-unified -b 2048 -ub 512 -ngl all --fit off \
  --spec-type draft-mtp --spec-draft-n-max 3

For many requests at once, replace the last two lines with -np 32 -c 32768 --kv-unified -ngl all --fit off, the setting that produced 295.4 tokens per second at 32 requests.

Ollama set to 4 requests at a time, then the model imported from the same GGUF file:

echo "FROM /gguf/Qwen3.8-27B-UD-Q4_K_M.gguf" > Modelfile
docker run -d --name ollama --gpus all -p 127.0.0.1:11434:11434 \
  -e OLLAMA_NUM_PARALLEL=4 -e OLLAMA_CONTEXT_LENGTH=8192 \
  -v ollama:/root/.ollama -v ~/models:/gguf:ro -v $PWD/Modelfile:/Modelfile:ro \
  ollama/ollama:0.40.15
docker exec ollama ollama create qwen38-27b -f /Modelfile

Ollama with MTP, using its own download instead of the imported file:

docker exec ollama ollama pull qwen3.8:27b-mtp-q4_K_M

vLLM with the settings that fit on a 24 GB card. Use --max-num-seqs 16 on the first start, before the compile cache exists:

docker run -d --name vllm --gpus all --ipc=host -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface -v ~/vllm-cache:/root/.cache/vllm \
  vllm/vllm-openai:v0.33.0 --model cyankiwi/Qwen3.8-27B-AWQ-INT4 --language-model-only \
  --max-model-len 8192 --gpu-memory-utilization 0.93 \
  --max-num-seqs 32 --max-cudagraph-capture-size 32

--language-model-only skips the model's image and video part, which this test did not use. LM Studio's headless setup is in the LM Studio vs Ollama comparison; here, lms import linked the same GGUF file and lms load loaded it with its defaults.

Qwen3.8 thinks before it answers unless you turn it off. This test sent "chat_template_kwargs": {"enable_thinking": false} to llama.cpp and vLLM, "think": false to Ollama and "reasoning": "off" to LM Studio.

Frequently asked questions

Which is fastest: Ollama, llama.cpp, LM Studio or vLLM?

For one request with a 70-token prompt, Qwen3.8-27B on an RTX 3090, llama.cpp with MTP turned on wrote 70.9 tokens per second and LM Studio, which turns MTP on by default, 65.8, both with an 8k context. Ollama with its MTP download wrote 57.1 at 32k. Without MTP, Ollama, llama.cpp and vLLM wrote 43.5 to 43.8. With 32 requests at once, vLLM produced the most, 345.8 tokens per second in total, with at most 8k of context per request.

Can Qwen3.8-27B run on an RTX 3090?

Yes. The 4-bit GGUF file is 16.6 GB, and the servers in this test used 16.0 to 22.0 GiB of the card's 24 GB. Without MTP it wrote about 43.5 tokens per second for one request with a 70-token prompt.

How much VRAM does Qwen3.8-27B need?

In 4 bits, the servers used 16.0 GiB with an 8k context and 21.5 GiB with llama.cpp's default 97,792-token context. MTP added about 2.3 GiB. The largest context that fit was 106,496 tokens on llama.cpp, 65,536 with MTP, and 32,768 on vLLM, which loaded 18.28 GiB of weights.

Does LM Studio use speculative decoding by default?

In this test, yes. LM Studio's llama-server ran Qwen3.8-27B with --spec-type draft-mtp --spec-draft-n-max 3 without being asked, and it wrote 65.8 tokens per second at an 8k context, against 43.8 for llama.cpp without MTP at its default 97.8k.

Does vLLM support MTP on a 24 GB card?

vLLM has an MTP setting, but with Qwen3.8-27B in 4 bits on an RTX 3090 it ran out of GPU memory while building the MTP layer. It needed another 2.37 GiB with about 0.8 GiB free.

Do Ollama and LM Studio use llama.cpp?

Yes. Ollama runs llama.cpp's server underneath, and LM Studio's CUDA runtime 2.79.0 is llama.cpp build 12521. vLLM is a separate engine.

Sources

  1. Qwen3.8-27B model card (Qwen, Hugging Face)
  2. Qwen3.8-27B GGUF files (unsloth)
  3. Qwen3.8-27B AWQ INT4 (cyankiwi)
  4. Ollama 0.30.0 release notes
  5. Ollama docs: context length
  6. Ollama library: qwen3.8 tags
  7. Ollama license
  8. llama.cpp: llama-server README at v1.0.0
  9. llama.cpp license
  10. LM Studio docs: speculative decoding
  11. LM Studio app terms
  12. vLLM docs: MTP (multi-token prediction)
  13. vLLM license
  14. NVIDIA Container Toolkit install guide