DatasetNotFoundError
langsmith.evaluation.errors.DatasetNotFoundError
Stack trace
langsmith.evaluation.errors.DatasetNotFoundError: Dataset with ID '1234abcd' not found in LangSmith workspace.
File "app.py", line 45, in run_evaluation
evaluation = client.evaluation.create(dataset_id="1234abcd", model_id="model-xyz")
File "langsmith/client.py", line 102, in create
raise DatasetNotFoundError(f"Dataset with ID '{dataset_id}' not found in LangSmith workspace.") Why it happens
This error occurs when the LangSmith client attempts to evaluate a model using a dataset ID that does not exist or has been deleted from the LangSmith workspace. It can also happen if the dataset ID is mistyped or if the user lacks access permissions to the dataset.
Detection
Check for DatasetNotFoundError exceptions when calling evaluation methods and log the dataset ID being used to verify its existence before proceeding.
Causes & fixes
The dataset ID provided does not exist in the LangSmith workspace.
Verify the dataset ID in the LangSmith dashboard or API and use the correct existing dataset ID in your evaluation call.
The dataset was deleted or moved, making the ID invalid.
Recreate the dataset or update your code to reference a current valid dataset ID.
Insufficient permissions to access the dataset in LangSmith.
Ensure your API key or user account has the necessary permissions to access the dataset.
Typo or formatting error in the dataset ID string.
Double-check the dataset ID string for typos, extra spaces, or incorrect casing.
Code: broken vs fixed
from langsmith import LangSmithClient
client = LangSmithClient(api_key="sk-incorrect")
evaluation = client.evaluation.create(dataset_id="wrong-id", model_id="model-xyz") # Raises DatasetNotFoundErrorimport os
from langsmith import LangSmithClient
client = LangSmithClient(api_key=os.environ["LANGSMITH_API_KEY"])
evaluation = client.evaluation.create(dataset_id="correct-dataset-id", model_id="model-xyz") # Fixed dataset ID
print("Evaluation started successfully")Workaround
Catch DatasetNotFoundError in a try/except block, then list available datasets via the LangSmith API to select a valid dataset ID dynamically.
Prevention
Implement validation logic to confirm dataset existence via LangSmith API before evaluation calls and manage dataset IDs centrally to avoid typos or stale references.