How to beginner · 3 min read

How to evaluate LLM outputs with LangSmith

Q: How to evaluate LLM outputs with LangSmith

Use the langsmith Python SDK to trace and evaluate LLM outputs by initializing a Client with your API key, then wrapping your LLM calls with the @traceable decorator or manual tracing methods. This enables detailed logging, metrics, and visualization of LLM responses for quality assessment.

Quick answer

Use the langsmith Python SDK to trace and evaluate LLM outputs by initializing a Client with your API key, then wrapping your LLM calls with the @traceable decorator or manual tracing methods. This enables detailed logging, metrics, and visualization of LLM responses for quality assessment.

PREREQUISITES

Python 3.8+
LangSmith API key
pip install langsmith

Setup

Install the langsmith package and set your API key as an environment variable to authenticate your requests.

bash

pip install langsmith

Step by step

This example shows how to initialize the LangSmith client, define a simple LLM call wrapped with the @traceable decorator, and print the output while LangSmith automatically logs the interaction for evaluation.

python

import os
from langsmith import Client, traceable

# Initialize LangSmith client with API key from environment
client = Client(api_key=os.environ["LANGSMITH_API_KEY"])

@traceable
# Define a function that calls the LLM
# For demonstration, we simulate an LLM call
# Replace with your actual LLM call logic

def generate_response(prompt: str) -> str:
    # Simulated LLM output
    return f"Response to: {prompt}"

# Run the function with tracing enabled
response = generate_response("Explain RAG in AI.")
print("LLM output:", response)

output

LLM output: Response to: Explain RAG in AI.

Common variations

You can manually create trace sessions for more control or integrate LangSmith with your existing LLM SDK calls. Async functions and streaming outputs are also supported by wrapping or instrumenting your code with LangSmith's tracing decorators or context managers.

python

import os
import asyncio
from langsmith import Client, traceable

client = Client(api_key=os.environ["LANGSMITH_API_KEY"])

@traceable
async def async_generate(prompt: str) -> str:
    # Simulate async LLM call
    await asyncio.sleep(0.1)
    return f"Async response to: {prompt}"

async def main():
    response = await async_generate("What is LangSmith?")
    print("Async LLM output:", response)

asyncio.run(main())

output

Async LLM output: Async response to: What is LangSmith?

Troubleshooting

If you see authentication errors, verify your LANGSMITH_API_KEY environment variable is set correctly.
If traces do not appear in the LangSmith dashboard, ensure your network allows outbound HTTPS requests to https://cloud.langsmith.com.
For missing or incomplete logs, confirm you are using the @traceable decorator or manual tracing methods properly around your LLM calls.

✅

Key Takeaways

Initialize LangSmith Client with your API key from environment variables.
Use the @traceable decorator to automatically log and evaluate LLM outputs.
LangSmith supports both synchronous and asynchronous LLM calls for flexible integration.
Check environment and network settings if traces do not appear in your LangSmith dashboard.
Manual tracing methods offer fine-grained control over evaluation and logging.

Verified 2026-04

Verify ↗