How to Migrate From Ollama to vLLM

1. Identify the concurrency bottleneck by monitoring request queue times and timeout rates during peak team usage.
2. Audit all current models, quantization levels, and API integration points calling the Ollama endpoint.
3. Verify GPU compatibility (CUDA compute capability ≥ 8.0) and confirm vLLM supports your target models.
4. Deploy vLLM alongside Ollama using Docker Compose with separate ports and shared GPU memory budgets.
5. Migrate API clients incrementally by updating the base URL and model name to the vLLM endpoint.
6. Validate performance with concurrent load tests comparing latency and throughput against your Ollama baseline.
7. Decommission Ollama after all clients are confirmed stable on vLLM and rollback criteria are no longer triggered.

A small team of two or three developers running Ollama for local LLM inference encounters few problems — until the team grows and requests start queuing. This guide provides a concrete decision framework for when migration to vLLM makes sense, followed by a step-by-step process to move with minimal downtime.

Table of Contents

• When Your Team Outgrows Ollama
• Ollama vs vLLM for Production: What's Actually Different
• Signs You've Outgrown Ollama
• Pre-Migration Checklist
• Setting Up vLLM with Docker Compose
• Migrating Your API Integrations
• Validating the Migration
• Post-Migration: Decommissioning Ollama
• When to Stick with Ollama
• Scaling Local LLMs for Your Team

When Your Team Outgrows Ollama

A small team of two or three developers running Ollama for local LLM inference encounters few problems. Requests flow sequentially, responses return fast enough, and the simplicity of ollama pull followed by a quick API call keeps everyone productive. Then the team grows to five, six, eight people. Suddenly, developers are waiting. Responses that took five seconds now take thirty. Someone posts in Slack: "Can you hold off on your query? I'm in the middle of a generation." That is the moment a team has outgrown Ollama.

The root cause is architectural. Ollama was designed primarily for local, single-user development workflows. It serializes inference requests, meaning each prompt waits for the previous one to finish before processing begins. This works fine for individual development but becomes a hard bottleneck when multiple people share the same endpoint. vLLM, by contrast, is a production inference engine built around continuous batching and PagedAttention, specifically designed to handle concurrent requests efficiently.

Ollama was designed primarily for local, single-user development workflows. It serializes inference requests, meaning each prompt waits for the previous one to finish before processing begins.

This article provides a concrete decision framework for when migration makes sense, followed by a step-by-step guide to move from Ollama to vLLM using Docker Compose with minimal downtime, targeting zero downtime on systems with sufficient GPU memory for both services simultaneously. It preserves existing OpenAI-compatible API integrations so the switch requires configuration changes rather than code rewrites.

Ollama vs vLLM for Production: What's Actually Different

Architecture and Design Philosophy

Ollama uses llama.cpp as its primary inference backend, wrapped in a user-friendly package with a built-in model registry, dead-simple CLI commands, and automatic quantization handling. It exposes both a native API (/api/generate, /api/chat) and an OpenAI-compatible endpoint. The server processes one request at a time. When a second request arrives while the first is still generating tokens, it queues. A third request queues behind the second. Under concurrent load, latency grows linearly at best, and at worst triggers timeouts.

vLLM takes a fundamentally different approach. Its core innovation is PagedAttention, a memory management technique that handles the key-value (KV) cache the way an operating system handles virtual memory: through non-contiguous blocks allocated on demand. This eliminates the memory waste that plagues naive KV cache implementations, where pre-allocated contiguous memory for each sequence leads to significant internal fragmentation. On top of PagedAttention, vLLM implements continuous batching, which dynamically adds new requests to an in-flight batch rather than waiting for an entire batch to complete before starting the next one. Under concurrent load, throughput scales sub-linearly rather than collapsing into a linear queue, and GPU utilization stays high even as request counts increase.

Performance Comparison Table

The following table presents qualitative performance characteristics for Ollama and vLLM running Llama 3.1 8B on a single NVIDIA A100 40GB GPU. The figures in this table are illustrative only and are not derived from a single controlled experiment. Do not use them as benchmarks for your hardware. Run the load test script in the Validation section against your own stack to obtain baseline figures. Actual results will vary significantly by hardware, driver version, vLLM version, quantization, prompt length, and generation parameters.

At a single concurrent user, the performance gap is negligible. Ollama often feels snappier at single-user loads due to lower startup overhead and optimized GGUF quantization. The divergence shows up under load.

The Concurrent Users Threshold

The inflection point sits around three simultaneous requests, though this threshold varies with model size, hardware, and generation length. At this level, Ollama's sequential processing causes second and third requests to wait for full completion of prior generations before execution begins. For a typical 512-token generation that takes five seconds at single-user speeds, the third concurrent request waits roughly ten seconds before generation even starts (assuming constant per-request latency, which itself varies with prompt and generation length), resulting in a perceived latency of 15 seconds or more. By five concurrent users, queue depth causes requests to regularly exceed common 30-second timeout defaults.

The decision heuristic is straightforward: if a team regularly has three or more people hitting the LLM endpoint simultaneously, or LLM calls are integrated into CI/CD pipelines or automated workflows where multiple requests fire in parallel, Ollama will be the bottleneck. The serial architecture is not a bug to be patched; it is a design choice aligned with Ollama's intended use case as a local development tool.

Signs You've Outgrown Ollama

Before investing in a migration, confirm the problem is actually concurrency-related rather than a hardware or configuration issue. The following symptoms point specifically to Ollama's architectural ceiling:

• Request timeouts spike during peak team working hours but vanish during off-hours or weekends
• Prompts that return in under five seconds when run in isolation now take 30 seconds or longer under normal team load
• Team members have started informally coordinating LLM usage ("don't run it while I'm using it")
• If someone has proposed running multiple Ollama instances behind a load balancer, that is a workaround that creates model duplication, memory waste, and operational complexity without solving the fundamental batching problem
• Watch for non-developer stakeholders (product managers, designers) requesting model access, which expands the concurrent user base beyond the engineering team
• Applications require streaming responses to multiple clients concurrently, which Ollama cannot support due to its sequential request processing

If three or more of these apply, migration will pay for itself quickly. A rough estimate: if five developers each lose 10 minutes per day to queuing, that is over four hours per week of recovered developer time.

Pre-Migration Checklist

Verify and document everything in this checklist before starting the migration. Teams can use it as a tracking artifact.

Inventory

1. Audit current model usage. List every model currently served through Ollama, including specific quantization levels (Q4_K_M, Q5_K_M, etc.). Note which models are used most frequently and by which workflows.
2. Document all API integration points. Record every service, script, or application that calls the Ollama endpoint. Note whether each uses Ollama's native API (/api/generate, /api/chat) or the OpenAI-compatible endpoint (/v1/chat/completions). Record authentication mechanisms if any proxy is in place.

Compatibility

3. Confirm GPU hardware and driver compatibility. vLLM requires NVIDIA GPUs with CUDA compute capability 8.0 or higher for full production support. Compute capability 7.0 (Volta) may work in limited configurations, but the vLLM team does not officially support Volta for production use. Verify your GPU's compute capability at https://developer.nvidia.com/cuda-gpus or by running nvidia-smi --query-gpu=compute_cap --format=csv,noheader. Check available GPU memory, as vLLM running full-precision or AWQ models typically requires more VRAM than Ollama's GGUF quantizations for the same model. Also verify that nvidia-container-toolkit is installed on the Docker host: docker run --rm --gpus all nvidia/cuda:12.0-base-ubuntu20.04 nvidia-smi should return GPU info. Install guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html.
4. Verify vLLM supports your models. Consult the vLLM supported models list. Most popular architectures (Llama, Mistral, Qwen, Phi) are supported, but not every model variant or quantization format is. vLLM uses HuggingFace safetensors format, not GGUF.

Process

5. Set up a monitoring baseline. Record current average latency, p95 latency, throughput, and error rates under typical team load. This provides the comparison point for validating the migration.
6. Plan a parallel-run window. You will run both Ollama and vLLM simultaneously during migration. Ensure sufficient GPU memory for both (or plan to run them on different GPUs, or stagger their model loading).
7. Identify rollback triggers. Define specific criteria that would cause a rollback: error rate thresholds, latency ceilings, model output quality issues.
8. Notify the team and set a migration window. Communicate the plan, expected timeline, and any brief periods of degraded service.

Setting Up vLLM with Docker Compose

Prerequisites: This guide targets Docker Compose v2 (the docker compose plugin, not legacy docker-compose v1). Docker Engine ≥ 23.0 is recommended. The version field in Compose files is deprecated and omitted from the configurations below.

Base vLLM Configuration

The following Docker Compose file runs vLLM serving Llama 3.1 8B with GPU passthrough and defaults sized for 5-10 concurrent users on a 40 GB GPU.

Before running docker compose up, create a .env file in the same directory as your docker-compose.yml:

# .env file in same directory as docker-compose.yml
HF_TOKEN=hf_your_token_here

Obtain a token at https://huggingface.co/settings/tokens with read scope. You must also accept Meta's Llama 3.1 license agreement on the model's HuggingFace page before the download will succeed.

# Requires Docker Compose v2 (docker compose plugin); version field is deprecated
services:
  vllm:
    image: vllm/vllm-openai:v0.4.3  # Pin to a specific release; check https://github.com/vllm-project/vllm/releases
    container_name: vllm-server
    ports:
      - "8000:8000"
    volumes:
      - ./models:/root/.cache/huggingface
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:?HF_TOKEN must be set in .env}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model meta-llama/Meta-Llama-3.1-8B-Instruct
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --max-num-seqs 16
      --host 0.0.0.0
      --port 8000
    restart: unless-stopped

The volumes mount caches the HuggingFace model download so subsequent container restarts do not re-download multi-gigabyte model weights. Ensure the host path (./models) has at least 20 GB of free space for an 8B parameter model in full precision. The HF_TOKEN environment variable is required for gated models like Llama 3.1, which require accepting Meta's license agreement on HuggingFace before download.

Running Ollama and vLLM Side by Side

The migration strategy runs both services simultaneously, allowing clients to be moved incrementally from Ollama to vLLM. The following configuration places Ollama on port 11434 and vLLM on port 8000.

# Requires Docker Compose v2 (docker compose plugin); version field is deprecated
services:
  ollama:
    image: ollama/ollama:0.3.6  # Pin to a specific release; check https://github.com/ollama/ollama/releases
    container_name: ollama-server
    ports:
      - "11434:11434"
    volumes:
      - ./ollama-models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  vllm:
    image: vllm/vllm-openai:v0.4.3  # Pin to a specific release; check https://github.com/vllm-project/vllm/releases
    container_name: vllm-server
    depends_on:
      ollama:
        condition: service_healthy   # vLLM waits until Ollama is up before claiming VRAM
    ports:
      - "8000:8000"
    volumes:
      - ./hf-models:/root/.cache/huggingface
    environment:
      # Fail fast at compose-up time if token is missing, not at model download time
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:?Set HF_TOKEN in .env}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s
    command: >
      --model meta-llama/Meta-Llama-3.1-8B-Instruct
      --max-model-len 8192
      --gpu-memory-utilization 0.85
      --max-num-seqs 16
      --host 0.0.0.0
      --port 8000
    restart: unless-stopped

A critical point: Ollama uses GGUF model files while vLLM uses HuggingFace safetensors format. These are not interchangeable, so the services use separate model directories. Running both services on a single GPU requires careful memory management. On a single-GPU host, both containers will share the same physical GPU (device 0) — Docker GPU reservations do not enforce memory exclusivity between containers. The depends_on with condition: service_healthy ensures vLLM does not begin its VRAM allocation until Ollama is running and healthy, reducing the risk of simultaneous GPU memory contention during startup. Monitor GPU memory with nvidia-smi after both containers start to confirm combined VRAM usage is within budget before sending any requests. The --gpu-memory-utilization flag for vLLM is set to 0.85 in this parallel configuration to leave room for Ollama (use 0.90 only on a dedicated single-service GPU). If memory is tight, consider loading a smaller quantization in Ollama during the transition or staggering the services across multiple GPUs. Note the longer start_period on the vLLM healthcheck, as initial model loading can take 60 to 120 seconds for an 8B parameter model, depending on storage speed and host memory bandwidth.

Essential vLLM Tuning Parameters

To change vLLM startup flags, update the command: field in your Compose file and recreate the container. Do not use docker exec to reconfigure vLLM startup flags — the container already has a running server process, and launching a second one will fail with a port conflict.

The following command: block demonstrates a production-tuned configuration for a team of five to ten concurrent users on a single GPU:

    command: >
      --model meta-llama/Meta-Llama-3.1-8B-Instruct
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --max-num-seqs 32
      --tensor-parallel-size 1
      --host 0.0.0.0
      --port 8000

To enable --enforce-eager (which disables CUDA graphs, trading throughput for lower memory usage), use a Compose override file rather than a YAML comment inside the command: block. YAML comments inside a command: > block scalar are silently stripped by the YAML parser and never reach the container, so "uncommenting" a #-prefixed flag has no effect.

Create a separate override file:

# docker-compose.eager.yml — apply with:
#   docker compose -f docker-compose.yml -f docker-compose.eager.yml up -d
services:
  vllm:
    command: >
      --model meta-llama/Meta-Llama-3.1-8B-Instruct
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --max-num-seqs 32
      --tensor-parallel-size 1
      --host 0.0.0.0
      --port 8000
      --enforce-eager

After updating the command: field (or applying the override), apply the changes:

docker compose up -d --force-recreate vllm

The key flags and their trade-offs:

• Need to limit memory for longer conversations? --max-model-len sets the maximum context window. Higher values consume more KV cache memory. For most team use cases, 8192 balances capability with memory usage. Reducing this to 4096 frees significant GPU memory for higher concurrency.
• --gpu-memory-utilization controls what fraction of GPU memory vLLM reserves. On a dedicated inference server, 0.90 is appropriate. In a parallel-run configuration sharing a GPU with Ollama, use 0.85 or lower. Values above 0.95 risk out-of-memory errors during peak load.
• The primary concurrency lever is --max-num-seqs, which caps the number of sequences processed simultaneously in a batch. For a team of five to ten, values between 16 and 32 work well. Setting this too high with limited GPU memory causes vLLM to reject requests rather than queue them. Start at 16 and increase incrementally while monitoring GPU memory.
• Running tight on VRAM? --enforce-eager disables CUDA graph optimization. CUDA graphs improve steady-state throughput but consume additional memory during warmup. Enabling eager mode trades some throughput for lower memory usage. Default behavior (CUDA graphs enabled) is preferred when memory allows.
• --tensor-parallel-size enables multi-GPU serving. Set to 1 for single-GPU deployments. For multi-GPU setups, set this to the number of GPUs to shard the model across devices, enabling larger models or higher throughput.

Migrating Your API Integrations

Endpoint Compatibility: What Changes and What Doesn't

Both Ollama and vLLM expose OpenAI-compatible endpoints at /v1/chat/completions. The request body structure is identical. What changes is the base URL and the model name string.

# Ollama request
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "temperature": 0.7,
    "stream": false
  }'

# vLLM request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "temperature": 0.7,
    "stream": false
  }'

The request body is identical aside from the model field. Ollama uses its own naming convention (llama3.1:8b) while vLLM uses the full HuggingFace model identifier (meta-llama/Meta-Llama-3.1-8B-Instruct). Streaming behavior is compatible on both, using server-sent events.

Updating Application Code

For teams already using the OpenAI-compatible endpoint through the openai Python SDK, you migrate by changing two configuration lines:

import os
from openai import OpenAI

# Before: pointing to Ollama
# client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-required")
# model_name = "llama3.1:8b"

# After: pointing to vLLM
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key=os.environ.get("OPENAI_API_KEY", "not-required"),
)
model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"

response = client.chat.completions.create(
    model=model_name,
    messages=[{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    temperature=0.7,
)
print(response.choices[0].message.content)

The api_key parameter is required by the SDK but not validated by either Ollama or vLLM in default configurations. Any non-empty string works for local deployments. Warning: vLLM does not authenticate requests by default. If your server is accessible beyond localhost — including on a VPN, internal network, or cloud instance — place it behind a reverse proxy with authentication (e.g., nginx with HTTP Basic Auth or a bearer token header check) before allowing team access. Teams using Ollama's native /api/generate or /api/chat endpoints will need to refactor those calls to the OpenAI-compatible format, as vLLM does not implement Ollama's native API. This is the one scenario where migration requires more than a configuration change.

Note that while the URL swap is straightforward, default sampling parameters (temperature, top_p, stop tokens) may differ between Ollama and vLLM for the same model. Test your most common prompts against both endpoints and compare outputs before cutting over production traffic.

Teams using Ollama's native /api/generate or /api/chat endpoints will need to refactor those calls to the OpenAI-compatible format, as vLLM does not implement Ollama's native API. This is the one scenario where migration requires more than a configuration change.

Validating the Migration

Smoke Tests

Start with manual validation. Send the same prompt to both the Ollama and vLLM endpoints and compare output quality, format, and completeness. Because the underlying model weights may differ slightly between GGUF quantized (Ollama) and full-precision or differently quantized (vLLM) versions, expect minor output variations between the two. Verify that streaming responses arrive incrementally and that client libraries parse them correctly.

Load Testing Your New Setup

The following Python script fires concurrent requests against the vLLM endpoint and reports latency statistics. Before running, install the required dependency:

pip install aiohttp

import asyncio
import math
import time
import aiohttp

VLLM_URL = "http://localhost:8000/v1/chat/completions"
PAYLOAD = {
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Write a haiku about distributed systems."}],
    "temperature": 0.7,
    "max_tokens": 64,
}
REQUEST_TIMEOUT = aiohttp.ClientTimeout(total=120)


async def send_request(session: aiohttp.ClientSession, request_id: int) -> tuple[float, int]:
    start = time.monotonic()
    try:
        async with session.post(VLLM_URL, json=PAYLOAD) as resp:
            body = await resp.text()
            elapsed = time.monotonic() - start
            if resp.status != 200:
                print(f"[{request_id}] ERROR status={resp.status} body={body[:200]}")
            return elapsed, resp.status
    except asyncio.TimeoutError:
        elapsed = time.monotonic() - start
        print(f"[{request_id}] TIMEOUT after {elapsed:.1f}s")
        return elapsed, 0


def compute_percentile(sorted_data: list[float], pct: float) -> float:
    """
    Nearest-rank percentile.  Requires len(sorted_data) >= 1.
    NOTE: statistically meaningful only for n >= 20.
    """
    if len(sorted_data) == 1:
        return sorted_data[0]
    idx = min(math.ceil(len(sorted_data) * pct / 100) - 1, len(sorted_data) - 1)
    return sorted_data[idx]


async def warm_up(session: aiohttp.ClientSession) -> None:
    """Send one un-timed request to complete CUDA graph warm-up before measurement."""
    try:
        async with session.post(VLLM_URL, json=PAYLOAD) as resp:
            await resp.text()
    except Exception as exc:
        print(f"[warm-up] failed: {exc}")


async def load_test(session: aiohttp.ClientSession, num_concurrent: int) -> None:
    tasks = [send_request(session, i) for i in range(num_concurrent)]
    results = await asyncio.gather(*tasks)
    latencies = sorted(r[0] for r in results)
    errors = sum(1 for r in results if r[1] != 200)
    avg = sum(latencies) / len(latencies)
    p95 = compute_percentile(latencies, 95)
    print(
        f"Concurrent: {num_concurrent:>3} | "
        f"Avg: {avg:.2f}s | "
        f"P95: {p95:.2f}s (n={num_concurrent}; {'valid' if num_concurrent >= 20 else 'low-n, interpret with caution'}) | "
        f"Errors: {errors}"
    )


async def main() -> None:
    async with aiohttp.ClientSession(timeout=REQUEST_TIMEOUT) as session:
        print("Warming up CUDA graphs...")
        await warm_up(session)
        for n in [1, 5, 10]:
            await load_test(session, n)


asyncio.run(main())

Note: the P95 statistic is not meaningful for very small sample sizes. For reliable percentile measurements, increase the concurrency values to 20 or higher.

Run this script against both the Ollama and vLLM endpoints (adjusting the URL and model name accordingly) to produce a direct comparison. The expected result: Ollama's average latency will scale roughly linearly with concurrency (since requests queue), while vLLM's average latency will remain relatively flat up to the configured --max-num-seqs limit.

Confirming Performance Gains

Compare the load test results against the baseline metrics captured in the pre-migration checklist. The primary indicators of a successful migration are near-constant per-request latency as concurrency increases, higher aggregate throughput (total tokens per second across all concurrent users), and zero timeout errors at the team's typical concurrent load.

Post-Migration: Decommissioning Ollama

Once you have migrated all API clients to the vLLM endpoint and load testing confirms acceptable performance, remove the Ollama service from Docker Compose. Rather than deleting the Ollama service definition entirely, comment it out and retain it for one to two weeks as a fast rollback path. Update team documentation, README files, and onboarding guides to reference the new vLLM endpoint and model naming conventions.

When to Stick with Ollama

Ollama remains the better choice in several scenarios. Solo developers or teams of one to two with no concurrent usage get fewer config files, no HuggingFace authentication, and a single ollama pull to serve a model, all with no performance penalty. For rapid model experimentation, Ollama's built-in model library and ollama pull workflow is unmatched for quickly trying different models and quantizations without navigating HuggingFace download flows. CPU-only environments favor Ollama as well, since vLLM is designed for NVIDIA CUDA GPUs and has no production-grade CPU inference path; without a compatible GPU, vLLM is not a viable option. In any environment where setup simplicity and low operational overhead take priority over throughput, Ollama is the pragmatic choice.

Scaling Local LLMs for Your Team

The migration path follows a predictable sequence: identify the concurrency threshold, run Ollama and vLLM in parallel, migrate clients incrementally by changing base URLs and model names, validate with load testing, and decommission Ollama. The key takeaway is not that Ollama is inadequate. It is that Ollama and vLLM serve different operational scales, and the boundary between those scales sits at roughly three concurrent users. As teams grow further toward 20 or more concurrent users or multi-model serving, vLLM's tensor parallelism and, with a reverse proxy such as LiteLLM, its ability to route across multiple model instances will continue to provide headroom that Ollama's sequential architecture does not support.

The key takeaway is not that Ollama is inadequate. It is that Ollama and vLLM serve different operational scales, and the boundary between those scales sits at roughly three concurrent users.