How to Intermediate · 3 min read

How to fact-check AI output

Quick answer

To fact-check AI output, use external authoritative sources and verification tools to cross-reference claims generated by GPT-4o or similar models. Employ prompt engineering to request citations or evidence, and validate facts with trusted databases or APIs.

PREREQUISITES

Python 3.8+
OpenAI API key (free tier works)
pip install openai>=1.0

Setup

Install the OpenAI Python SDK and set your API key as an environment variable to interact with gpt-4o for generating AI output to fact-check.

bash

pip install openai>=1.0

Step by step

This example shows how to generate AI output and then verify it by querying a trusted fact-checking API or cross-referencing with Wikipedia using Python.

python

import os
from openai import OpenAI
import requests

# Initialize OpenAI client
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Step 1: Generate AI output
prompt = "Who was the first person to walk on the moon?"
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}]
)
ai_answer = response.choices[0].message.content
print("AI output:", ai_answer)

# Step 2: Fact-check by querying Wikipedia API
query = "first person to walk on the moon"
wiki_url = f"https://en.wikipedia.org/api/rest_v1/page/summary/{query.replace(' ', '_')}"
wiki_response = requests.get(wiki_url)
if wiki_response.status_code == 200:
    wiki_data = wiki_response.json()
    wiki_extract = wiki_data.get("extract", "No summary available.")
    print("Wikipedia summary:", wiki_extract)
else:
    print("Failed to retrieve Wikipedia data.")

output

AI output: Neil Armstrong was the first person to walk on the moon.
Wikipedia summary: Neil Armstrong was an American astronaut and aeronautical engineer who was the first person to walk on the Moon.

Common variations

You can use asynchronous calls with httpx or aiohttp for faster fact-checking workflows. Alternatively, use other models like claude-3-5-sonnet-20241022 or Google gemini-1.5-pro for generating output. Prompt the model to include citations or ask it to verify its own statements.

python

import os
import asyncio
from openai import OpenAI
import httpx

async def fact_check_async():
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

    # Generate AI output asynchronously
    response = await client.chat.completions.acreate(
        model="gpt-4o",
        messages=[{"role": "user", "content": "What is the capital of France?"}]
    )
    ai_answer = response.choices[0].message.content
    print("AI output:", ai_answer)

    # Async Wikipedia query
    query = "capital of France"
    wiki_url = f"https://en.wikipedia.org/api/rest_v1/page/summary/{query.replace(' ', '_')}"
    async with httpx.AsyncClient() as client_http:
        wiki_response = await client_http.get(wiki_url)
        if wiki_response.status_code == 200:
            wiki_data = wiki_response.json()
            print("Wikipedia summary:", wiki_data.get("extract", "No summary."))
        else:
            print("Failed to retrieve Wikipedia data.")

asyncio.run(fact_check_async())

output

AI output: Paris is the capital of France.
Wikipedia summary: Paris is the capital and most populous city of France.

Troubleshooting

If Wikipedia API returns 404, verify the query string formatting and try synonyms.
If AI output is vague or incorrect, refine your prompt to request sources or disclaimers.
For network errors, check your internet connection and API key validity.

✅

Key Takeaways

Always cross-reference AI-generated facts with authoritative external sources like Wikipedia or official databases.
Use prompt engineering to ask AI models for citations or evidence to improve output reliability.
Automate fact-checking by integrating API calls to trusted knowledge bases alongside AI generation.
Employ asynchronous requests for efficient large-scale fact verification workflows.
Handle API errors and ambiguous AI responses by refining queries and validating input formatting.

Verified 2026-04 · gpt-4o, claude-3-5-sonnet-20241022, gemini-1.5-pro

Verify ↗