wandb.errors.CommError
wandb.errors.CommError
Stack trace
wandb.errors.CommError: Failed to sync offline run: network error or API key missing
File "/usr/local/lib/python3.9/site-packages/wandb/sdk/wandb_run.py", line 1234, in sync
raise CommError("Failed to sync offline run")
Why it happens
When wandb is used in offline mode, runs are stored locally and require syncing to the cloud later. This error occurs if the sync process cannot communicate with the wandb server due to missing or invalid API keys, network connectivity issues, or corrupted local run files.
Detection
Monitor wandb sync logs for CommError exceptions and verify that the WANDB_API_KEY environment variable is set and network connectivity to wandb servers is available before syncing.
Causes & fixes
WANDB_API_KEY environment variable is missing or invalid during sync
Set the WANDB_API_KEY environment variable with a valid API key before running wandb sync.
Network connectivity issues prevent communication with wandb servers
Ensure stable internet connection and that firewall or proxy settings allow outbound connections to wandb endpoints.
Local offline run files are corrupted or incomplete
Delete corrupted offline run directories or rerun experiments to regenerate valid offline run data before syncing.
Using an outdated wandb version with known sync bugs
Upgrade wandb to the latest stable version using pip install --upgrade wandb.
Code: broken vs fixed
import wandb
wandb.init(project="my-project", mode="offline")
# ... training code ...
wandb.finish()
# Later sync attempt without API key set
import os
os.environ.pop("WANDB_API_KEY", None) # API key missing
import subprocess
subprocess.run(["wandb", "sync", "./wandb/offline-run-xyz"])
# This triggers wandb.errors.CommErrorimport os
os.environ["WANDB_API_KEY"] = "your_valid_api_key_here" # Set your API key
import wandb
wandb.init(project="my-project", mode="offline")
# ... training code ...
wandb.finish()
import subprocess
subprocess.run(["wandb", "sync", "./wandb/offline-run-xyz"])
print("Sync completed successfully")
# Fixed by setting API key and ensuring network connectivityWorkaround
If immediate sync is failing, manually upload offline run files to wandb cloud via the web interface or retry syncing later after verifying API key and network status.
Prevention
Always set WANDB_API_KEY in your environment and run wandb in online mode when possible; use CI/CD checks to verify network connectivity and API key presence before syncing offline runs.