How to use vLLM to serve LLMs in python

Setup

pip install vllm

from vllm import LLMEngine

Examples

Integration steps

1. Install the vLLM Python package using pip.
2. Import LLMEngine from the vllm library.
3. Initialize LLMEngine with the desired model checkpoint path or Hugging Face model ID.
4. Call the generate method with a list of prompt strings.
5. Process the returned generation results to extract the generated text.
6. Close the engine when done to free resources.

Full code

from vllm import LLMEngine

# Initialize the LLM engine with a model checkpoint or Hugging Face model ID
engine = LLMEngine(model="huggyllama/llama-7b")

# Define prompts to generate completions for
prompts = [
    "Write a short story about a robot learning emotions.",
    "Summarize the benefits of renewable energy."
]

# Generate completions
results = engine.generate(prompts)

# Print generated outputs
for i, result in enumerate(results):
    print(f"Prompt {i+1}: {prompts[i]}")
    print(f"Completion: {result.text}")
    print("---")

# Close the engine to release resources
engine.close()

Prompt 1: Write a short story about a robot learning emotions.
Completion: In a quiet lab, a robot named Aiden discovered feelings for the first time, learning joy and sorrow through human connection.
---
Prompt 2: Summarize the benefits of renewable energy.
Completion: Renewable energy reduces carbon emissions, lowers pollution, and provides sustainable power sources for the future.
---

API trace

{"model": "huggyllama/llama-7b", "prompts": ["Write a short story about a robot learning emotions.", "Summarize the benefits of renewable energy."]}

{"results": [{"text": "In a quiet lab, a robot named Aiden discovered feelings..."}, {"text": "Renewable energy reduces carbon emissions..."}]}

Variants

from vllm import LLMEngine

engine = LLMEngine(model="huggyllama/llama-7b")
prompts = ["Explain quantum computing in simple terms."]

# Stream tokens as they are generated
for token in engine.generate(prompts, stream=True):
    print(token.text, end='', flush=True)

engine.close()

from vllm import LLMEngine

engine = LLMEngine(model="huggyllama/llama-7b")
prompts = ["Describe the water cycle.", "What is the capital of France?"]

results = engine.generate(prompts, max_concurrent_requests=2)
for r in results:
    print(r.text)

engine.close()

Performance

• Use batch generation to amortize overhead across multiple prompts.
• Limit max tokens per generation to reduce latency and memory usage.
• Use smaller models or quantized checkpoints for faster inference.