vLLM

Overview

vLLM is a high-performance LLM inference engine developed by UC Berkeley. Its core innovation is PagedAttention, an attention mechanism inspired by operating system virtual memory that significantly reduces KV cache memory waste. vLLM's throughput is 14-24x higher than Hugging Face Transformers and 2-4x higher than Text Generation Inference (TGI). It supports continuous batching, tensor parallelism, streaming output, and more. vLLM is the preferred solution for deploying LLMs in production environments.

Installation

vLLM requires an NVIDIA GPU (CUDA 11.8+) and Python 3.8+. Installation via pip is recommended. Docker deployment is also supported. Dependencies are automatically handled during installation.

# pip install
pip install vllm

# Or install a specific version
pip install vllm==0.6.0

# Docker
docker pull vllm/vllm-openai:latest

# Install from source
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

Quick Start

vLLM provides a simple API to start a service. A single command can start an OpenAI-compatible API server. A Python API for offline inference is also supported.

# Start the API server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000

# Python API offline inference
from vllm import LLM
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
outputs = llm.generate(["Hello, please introduce yourself"])

Serving Models

vLLM provides high-performance model serving. It supports loading Hugging Face models and local models. You can configure the number of GPUs, tensor parallelism, quantization, and more. The service is compatible with the OpenAI API and can seamlessly replace OpenAI.

# Start the service (detailed configuration)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --port 8000

# Test the API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

OpenAI API

vLLM provides an OpenAI-compatible API that can seamlessly replace OpenAI. It supports endpoints such as Chat Completions, Completions, and Embeddings. Existing applications only need to change the API address to use vLLM.

# OpenAI SDK integration
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm"  # vLLM does not require a real key
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="")

Quantization

vLLM supports multiple quantization schemes, reducing model memory usage and improving inference speed. It supports quantization formats such as AWQ, GPTQ, SqueezeLLM, and FP8. Quantization can significantly lower hardware requirements, allowing large models to run on smaller GPUs.

# Use an AWQ quantized model
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-2-13B-AWQ \
  --quantization awq

# Use a GPTQ quantized model
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-2-13B-GPTQ \
  --quantization gptq

# FP8 quantization (Hopper GPU)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --quantization fp8

Distributed Serving

vLLM supports distributed inference, allowing large models to run on multiple GPUs. It supports Tensor Parallelism and Pipeline Parallelism. This enables large models such as 70B and 405B to run on multi-GPU systems.

# Tensor parallelism (single machine, multiple GPUs)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

# Multi-node distributed
# Node 0
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2

# Node 1
VLLM_HOST_IP=192.168.1.2 \
NCCL_SOCKET_IFNAME=eth0 \
python -m vllm.entrypoints.openai.api_server ...

Performance

vLLM's performance advantages come from PagedAttention, continuous batching, and optimized CUDA kernels. Throughput is 14-24x higher than Transformers. It supports high-concurrency requests with low latency. Performance can be further improved through parameter tuning.

# Performance optimization configuration
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --swap-space 16  # GB, CPU swap space

Integration

vLLM can be integrated into various applications. It supports frameworks such as LangChain and LlamaIndex. Through the OpenAI-compatible API, it can replace OpenAI in most applications. It is suitable for building high-performance AI applications.

# LangChain integration
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm",
    model="meta-llama/Llama-3.1-8B-Instruct"
)

# LlamaIndex integration
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(
    model="meta-llama/Llama-3.1-8B-Instruct",
    api_base="http://localhost:8000/v1",
    api_key="vllm"
)

Configuration

vLLM is configured through CLI flags passed to vllm.entrypoints.openai.api_server and environment variables. The most important flags are --model (model id or path), --tensor-parallel-size (GPUs per node), --gpu-memory-utilization (KV cache budget), --max-model-len (context window), and --quantization (awq/gptq/fp8). For persistent settings, write flags into a shell script or systemd unit. The server speaks the OpenAI API, so clients just need base_url and a dummy api_key.

# Core serving flags
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --quantization fp8 \
  --port 8000

# Environment variables
export VLLM_NO_USAGE_STATS=1        # disable usage collection
export HF_TOKEN=hf_xxx              # for gated models
export VLLM_ATTENTION_BACKEND=FLASHINFER  # pick an attention backend

# Run as a background service (systemd unit)
# /etc/systemd/system/vllm.service
# ExecStart=/usr/bin/python -m vllm.entrypoints.openai.api_server \
#   --model meta-llama/Llama-3.1-8B-Instruct

gpu-memory-utilization is the fraction reserved for the KV cache—leave headroom for the model weights, or startup will OOM.

FAQ

Common questions cover GPU/memory requirements, quantization, OpenAI compatibility, distributed serving, and throughput tuning. vLLM needs an NVIDIA GPU with CUDA 11.8+; a 7B model fits on a single 16 GB GPU, while 70B needs 4x80 GB or quantization. The /v1 endpoint is OpenAI-compatible, so most SDKs work after changing base_url. Throughput scales with --max-num-seqs and continuous batching, so raise it for high-concurrency workloads.

Q: What hardware does vLLM need?
A: An NVIDIA GPU with CUDA 11.8+. A 7B/8B model fits on a 16 GB GPU;
   a 70B model needs ~4x80 GB or quantization (AWQ/GPTQ/FP8).

Q: Is the API OpenAI-compatible?
A: Yes—point any OpenAI SDK at http://localhost:8000/v1 with a dummy
   api_key; chat/completions, embeddings, and completions all work.

Q: How do I cut memory usage?
A: Use a quantized model (--quantization awq|gptq|fp8), lower
   --gpu-memory-utilization, shorten --max-model-len, or pick a smaller model.

Q: How do I serve across multiple GPUs?
A: Use --tensor-parallel-size N on one machine, or combine
   --tensor-parallel-size with --pipeline-parallel-size for multi-node.

Q: Why is throughput lower than expected?
A: Raise --max-num-seqs and --max-num-batched-tokens, enable
   --enable-chunked-prefill, and keep --gpu-memory-utilization high.

Monitor nvidia-smi while serving—if GPU memory is not saturated, you can usually raise --max-num-seqs for higher throughput.