Code beginner · 3 min read

How to use cross validation in python

Q: How to use cross validation in python

Use sklearn.model_selection.KFold or StratifiedKFold to split your dataset into training and validation folds, then train and evaluate your PyTorch model on each fold to perform cross validation.

Direct answer

Use sklearn.model_selection.KFold or StratifiedKFold to split your dataset into training and validation folds, then train and evaluate your PyTorch model on each fold to perform cross validation.

Setup

Install

bash

pip install torch scikit-learn numpy

Imports

python

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.model_selection import KFold
import numpy as np

Examples

inDataset with 100 samples, 10-fold cross validation

outFold 1: Validation Accuracy: 0.85 Fold 2: Validation Accuracy: 0.87 ... Average Accuracy: 0.86

inDataset with imbalanced classes, use StratifiedKFold

outFold 1: Validation Accuracy: 0.78 Fold 2: Validation Accuracy: 0.80 ... Average Accuracy: 0.79

inSmall dataset with 5 samples, 2-fold cross validation

outFold 1: Validation Accuracy: 0.60 Fold 2: Validation Accuracy: 0.80 Average Accuracy: 0.70

Integration steps

Prepare your dataset as tensors or numpy arrays.
Initialize a cross validation splitter like KFold from scikit-learn.
For each fold, split the data into training and validation sets.
Create and train your PyTorch model on the training set.
Evaluate the model on the validation set and record metrics.
Aggregate results across folds to estimate model performance.

Full code

python

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.model_selection import KFold
import numpy as np

# Simple dataset: 100 samples, 10 features
X = np.random.rand(100, 10).astype(np.float32)
y = np.random.randint(0, 2, size=(100,)).astype(np.int64)

class SimpleNet(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(SimpleNet, self).__init__()
        self.fc = nn.Linear(input_dim, output_dim)
    def forward(self, x):
        return self.fc(x)

kf = KFold(n_splits=5, shuffle=True, random_state=42)

accuracies = []

for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

    model = SimpleNet(input_dim=10, output_dim=2)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.01)

    # Convert to torch tensors
    X_train_t = torch.from_numpy(X_train)
    y_train_t = torch.from_numpy(y_train)
    X_val_t = torch.from_numpy(X_val)
    y_val_t = torch.from_numpy(y_val)

    # Train for 20 epochs
    model.train()
    for epoch in range(20):
        optimizer.zero_grad()
        outputs = model(X_train_t)
        loss = criterion(outputs, y_train_t)
        loss.backward()
        optimizer.step()

    # Evaluate
    model.eval()
    with torch.no_grad():
        val_outputs = model(X_val_t)
        _, predicted = torch.max(val_outputs, 1)
        accuracy = (predicted == y_val_t).float().mean().item()
        accuracies.append(accuracy)
        print(f"Fold {fold + 1}: Validation Accuracy: {accuracy:.2f}")

print(f"Average Accuracy: {np.mean(accuracies):.2f}")

output

Fold 1: Validation Accuracy: 0.80
Fold 2: Validation Accuracy: 0.85
Fold 3: Validation Accuracy: 0.78
Fold 4: Validation Accuracy: 0.82
Fold 5: Validation Accuracy: 0.79
Average Accuracy: 0.81

API trace

Request

json

{"model": "SimpleNet", "data": {"X_train": "tensor", "y_train": "tensor", "X_val": "tensor", "y_val": "tensor"}, "params": {"epochs": 20, "optimizer": "Adam", "loss": "CrossEntropy"}}

Response

json

{"fold": int, "validation_accuracy": float}

ExtractExtract validation accuracy from each fold's evaluation output and average them.

Variants

StratifiedKFold for imbalanced classification ›

Use when your dataset has imbalanced classes and you want to preserve class distribution in each fold.

python

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.model_selection import StratifiedKFold
import numpy as np

X = np.random.rand(100, 10).astype(np.float32)
y = np.concatenate([np.zeros(80), np.ones(20)]).astype(np.int64)  # Imbalanced

class SimpleNet(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(SimpleNet, self).__init__()
        self.fc = nn.Linear(input_dim, output_dim)
    def forward(self, x):
        return self.fc(x)

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
accuracies = []

for fold, (train_idx, val_idx) in enumerate(skf.split(X, y)):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

    model = SimpleNet(input_dim=10, output_dim=2)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.01)

    X_train_t = torch.from_numpy(X_train)
    y_train_t = torch.from_numpy(y_train)
    X_val_t = torch.from_numpy(X_val)
    y_val_t = torch.from_numpy(y_val)

    model.train()
    for epoch in range(20):
        optimizer.zero_grad()
        outputs = model(X_train_t)
        loss = criterion(outputs, y_train_t)
        loss.backward()
        optimizer.step()

    model.eval()
    with torch.no_grad():
        val_outputs = model(X_val_t)
        _, predicted = torch.max(val_outputs, 1)
        accuracy = (predicted == y_val_t).float().mean().item()
        accuracies.append(accuracy)
        print(f"Fold {fold + 1}: Validation Accuracy: {accuracy:.2f}")

print(f"Average Accuracy: {np.mean(accuracies):.2f}")

Manual train/validation split without sklearn ›

Use when you want a simple train/validation split without cross validation.

python

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

X = np.random.rand(100, 10).astype(np.float32)
y = np.random.randint(0, 2, size=(100,)).astype(np.int64)

split_idx = int(0.8 * len(X))
X_train, X_val = X[:split_idx], X[split_idx:]
y_train, y_val = y[:split_idx], y[split_idx:]

class SimpleNet(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(SimpleNet, self).__init__()
        self.fc = nn.Linear(input_dim, output_dim)
    def forward(self, x):
        return self.fc(x)

model = SimpleNet(input_dim=10, output_dim=2)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

X_train_t = torch.from_numpy(X_train)
y_train_t = torch.from_numpy(y_train)
X_val_t = torch.from_numpy(X_val)
y_val_t = torch.from_numpy(y_val)

model.train()
for epoch in range(20):
    optimizer.zero_grad()
    outputs = model(X_train_t)
    loss = criterion(outputs, y_train_t)
    loss.backward()
    optimizer.step()

model.eval()
with torch.no_grad():
    val_outputs = model(X_val_t)
    _, predicted = torch.max(val_outputs, 1)
    accuracy = (predicted == y_val_t).float().mean().item()
    print(f"Validation Accuracy: {accuracy:.2f}")

Performance

Latency~1-3 seconds per fold for small models on CPU

CostNo monetary cost for local cross validation; compute cost depends on model size and dataset

Rate limitsNot applicable for local PyTorch training

Keep dataset size manageable to reduce training time per fold.
Use fewer epochs per fold during experimentation to speed up cross validation.
Cache dataset preprocessing outside the fold loop.

Approach	Latency	Cost/call	Best for
KFold with PyTorch	~1-3s per fold	Free (local compute)	General cross validation
StratifiedKFold with PyTorch	~1-3s per fold	Free (local compute)	Imbalanced classification
Manual train/val split	<1s	Free (local compute)	Quick validation without folds

✓

Quick tip

Use <code>StratifiedKFold</code> instead of <code>KFold</code> for classification tasks with imbalanced classes to maintain class proportions in each fold.

⚠

Common mistake

Not resetting or reinitializing the PyTorch model weights for each fold, causing data leakage and biased results.

Verified 2026-04 · SimpleNet (custom PyTorch model), KFold, StratifiedKFold

Verify ↗