VLLM

Running in GPU

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-2-7b-hf
 
 
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-hf",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'
 
 

Enable with VLLM flags:

--log-requests              # Log all request metadata
--log-request-responses     # Also log model responses
--max-log-len 2048         # Characters to log per request

curl http://localhost:8002/metrics | grep cache_hit_rate
 
# Higher = better. Typical:
# < 0.5 = need optimization
# 0.5-0.7 = good
# > 0.7 = excellent (repeated prompts being reused)
 

CPU

docker run -it --rm \
  --privileged=true \
  --shm-size=4g \
  -p 8000:8000 \
  -e VLLM_CPU_KVCACHE_SPACE=8 \
  -e VLLM_CPU_OMP_THREADS_BIND=auto \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.2-1B-Instruct \
  --dtype=bfloat16 \
  --max-model-len 2048
 

Resources

• https://mistral.ai/news/debugging-memory-leak-in-vllm