How to Beginner · 3 min read

How to count tokens in Python

Q: How to count tokens in Python

Use the tiktoken library in Python to count tokens for OpenAI models by encoding your text and measuring the token length. Alternatively, use the openai SDK's tokenizer utilities or third-party tokenizers to handle token counting accurately.

Quick answer

Use the tiktoken library in Python to count tokens for OpenAI models by encoding your text and measuring the token length. Alternatively, use the openai SDK's tokenizer utilities or third-party tokenizers to handle token counting accurately.

PREREQUISITES

Python 3.8+
pip install tiktoken
OpenAI API key (optional for SDK usage)

Setup

Install the tiktoken library, which is the official tokenizer used by OpenAI models. This library allows you to encode text and count tokens efficiently.

bash

pip install tiktoken

output

Collecting tiktoken
  Downloading tiktoken-0.4.0-py3-none-any.whl (30 kB)
Installing collected packages: tiktoken
Successfully installed tiktoken-0.4.0

Step by step

Use tiktoken to count tokens for a given text string. Specify the encoding that matches your model (e.g., cl100k_base for GPT-4 and GPT-3.5).

python

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    # Map model to encoding
    if model.startswith("gpt-4o") or model.startswith("gpt-4.1") or model.startswith("gpt-3.5"):
        encoding = tiktoken.get_encoding("cl100k_base")
    else:
        encoding = tiktoken.get_encoding("r50k_base")  # fallback
    tokens = encoding.encode(text)
    return len(tokens)

# Example usage
text = "Hello, how many tokens does this sentence use?"
token_count = count_tokens(text)
print(f"Token count: {token_count}")

output

Token count: 11

Common variations

You can count tokens asynchronously if integrating with an async framework, or use the openai SDK's tokenizer utilities if available. Different models may require different encodings, so always check the model's recommended tokenizer.

python

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Example: Using OpenAI SDK to count tokens (if supported)
# Note: As of 2026-04, direct token counting via SDK may not be exposed,
# so prefer <code>tiktoken</code> for precise counts.

# Async example with tiktoken
import asyncio

async def async_count_tokens(text: str):
    return count_tokens(text)

async def main():
    count = await async_count_tokens("Async token count example.")
    print(f"Async token count: {count}")

asyncio.run(main())

output

Async token count: 5

Troubleshooting

If you get an error like ModuleNotFoundError: No module named 'tiktoken', ensure you installed the package with pip install tiktoken.
If token counts seem off, verify you are using the correct encoding for your model.
For very large texts, consider chunking before counting to avoid memory issues.

✅

Key Takeaways

Use the tiktoken library for accurate token counting in Python.
Match the tokenizer encoding to your specific LLM model for correct counts.
Counting tokens helps manage context window limits and avoid truncation.
Async token counting is possible but typically uses the same synchronous logic.
Always install dependencies and verify your environment to avoid import errors.

Verified 2026-04 · gpt-4o, gpt-4.1, gpt-3.5

Verify ↗