How to beginner · 3 min read

How to compare experiments in wandb

Q: How to compare experiments in wandb

Use wandb to log your experiments and then compare them via the wandb.Api by fetching runs from a project. You can programmatically retrieve metrics and visualize differences or use the wandb web UI's built-in comparison tools for side-by-side analysis.

Quick answer

Use wandb to log your experiments and then compare them via the wandb.Api by fetching runs from a project. You can programmatically retrieve metrics and visualize differences or use the wandb web UI's built-in comparison tools for side-by-side analysis.

PREREQUISITES

Python 3.8+
wandb account and API key
pip install wandb

Setup

Install the wandb Python package and login to your wandb account to enable experiment tracking and comparison.

bash

pip install wandb
wandb login

Step by step

Log experiments with wandb.init() and then use the wandb.Api to fetch and compare runs programmatically.

python

import os
import wandb

# Initialize wandb API client
api = wandb.Api()

# Define your project and entity
entity = os.environ.get('WANDB_ENTITY', 'your-entity')
project = os.environ.get('WANDB_PROJECT', 'your-project')

# Fetch all runs from the project
runs = api.runs(f"{entity}/{project}")

# Print summary metrics for comparison
for run in runs:
    print(f"Run ID: {run.id}")
    print(f"  Name: {run.name}")
    print(f"  Accuracy: {run.summary.get('accuracy', 'N/A')}")
    print(f"  Loss: {run.summary.get('loss', 'N/A')}")
    print("---")

output

Run ID: 1a2b3c4d
  Name: experiment-1
  Accuracy: 0.92
  Loss: 0.15
---
Run ID: 5e6f7g8h
  Name: experiment-2
  Accuracy: 0.94
  Loss: 0.12
---

Common variations

You can compare experiments using the wandb web UI for visual charts or export run data as CSV for offline analysis. Also, use filters in wandb.Api().runs() to compare specific subsets of experiments.

python

import wandb

api = wandb.Api()
runs = api.runs("your-entity/your-project", filters={"config.learning_rate": {"$gt": 0.001}})

for run in runs:
    print(run.name, run.summary.get('accuracy'))

output

experiment-3 0.95
experiment-5 0.96

Troubleshooting

If you see empty metrics, ensure your runs have completed and metrics are logged with wandb.log(). Also, verify your WANDB_ENTITY and WANDB_PROJECT environment variables are set correctly.

✅

Key Takeaways

Use wandb.Api to programmatically fetch and compare experiment runs.
Leverage the wandb web UI for intuitive visual comparison of metrics and charts.
Filter runs by config or tags to focus comparisons on relevant experiments.

Verified 2026-04

Verify ↗