Benchmark LLMs with lm-evaluation-harness

How to use

Use lm-evaluation-harness when benchmarking models for academic papers, comparing model quality across standard tasks, or tracking training progress.

It also fits when you need reproducible evaluation and reporting of standardized metrics, since everyone uses the same prompts.

For broader evaluation of fairness, efficiency and calibration, consider HELM (Stanford); for instruction-following with LLM judges, AlpacaEval; for conversational multi-turn evaluation, MT-Bench; for domain-specific evaluation, custom scripts.

Hardware: an NVIDIA GPU (CUDA 11.8+) is supported; CPU also works but is very slow.

VRAM needs vary by model: a 7B model needs 16GB in bf16 or 8GB in 8-bit, a 13B model needs 28GB in bf16 or 14GB in 8-bit, and a 70B model requires multi-GPU or quantization.

Approximate evaluation times for a 7B model on a single A100: HellaSwag about 10 minutes, GSM8K about 5 minutes, full MMLU about 2 hours, and HumanEval about 20 minutes.

For frequent evaluation during training, prefer fast benchmarks: HellaSwag takes about 10 minutes on 1 GPU, GSM8K about 5 minutes, and PIQA about 2 minutes.

Avoid MMLU (about 2 hours across 57 subjects) and HumanEval (requires code execution) for frequent evaluation because they are too slow.

You can automate evaluation by integrating it into your training script, or use PyTorch Lightning callbacks instead.

Use lm-evaluation-harness when benchmarking models for academic papers, comparing model quality across standard tasks, or tracking training progress.

It also fits when you need reproducible evaluation and reporting of standardized metrics, since everyone uses the same prompts.

For broader evaluation of fairness, efficiency and calibration, consider HELM (Stanford); for instruction-following with LLM judges, AlpacaEval; for conversational multi-turn evaluation, MT-Bench; for domain-specific evaluation, custom scripts.

Hardware: an NVIDIA GPU (CUDA 11.8+) is supported; CPU also works but is very slow.

VRAM needs vary by model: a 7B model needs 16GB in bf16 or 8GB in 8-bit, a 13B model needs 28GB in bf16 or 14GB in 8-bit, and a 70B model requires multi-GPU or quantization.

Approximate evaluation times for a 7B model on a single A100: HellaSwag about 10 minutes, GSM8K about 5 minutes, full MMLU about 2 hours, and HumanEval about 20 minutes.

For frequent evaluation during training, prefer fast benchmarks: HellaSwag takes about 10 minutes on 1 GPU, GSM8K about 5 minutes, and PIQA about 2 minutes.

Avoid MMLU (about 2 hours across 57 subjects) and HumanEval (requires code execution) for frequent evaluation because they are too slow.

You can automate evaluation by integrating it into your training script, or use PyTorch Lightning callbacks instead.

You can automate evaluation by integrating it into your training script, or use PyTorch Lightning callbacks instead.

Quick start: install the harness.

Install it with pip: pip install lm-eval.

pip install lm-eval

To evaluate any HuggingFace model, run lm_eval with the hf backend, a pretrained model name, tasks such as mmlu,gsm8k,hellaswag, a device, and a batch size.

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag \
  --device cuda:0 \
  --batch_size 8

To view available tasks, run lm-eval ls tasks.

lm-eval ls tasks

Workflow 1: standard benchmark evaluation — evaluate a model on core benchmarks (MMLU, GSM8K, HumanEval).

Copy the checklist: Step 1, choose benchmark suite; Step 2, configure model; Step 3, run evaluation; Step 4, analyze results.

Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model
- [ ] Step 3: Run evaluation
- [ ] Step 4: Analyze results

Step 1: choose a benchmark suite.

For model releases, the recommended standard suite covers mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge.

--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge

Step 2: configure the model.

For a HuggingFace model, run lm_eval with the hf backend, model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16, --tasks mmlu, --device cuda:0 and --batch_size auto to auto-detect the optimal batch size.

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16 \
  --tasks mmlu \
  --device cuda:0 \
  --batch_size auto  # Auto-detect optimal batch size

For a quantized 4-bit/8-bit model, add load_in_4bit=True to the model_args.

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,load_in_4bit=True \
  --tasks mmlu \
  --device cuda:0

For a custom checkpoint, pass pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer in the model_args.

lm_eval --model hf \
  --model_args pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer \
  --tasks mmlu \
  --device cuda:0

Step 3: run the evaluation, for example full MMLU (57 subjects) with --num_fewshot 5 (the standard), --batch_size 8, --output_path results/ and --log_samples to save individual predictions, or run multiple benchmarks at once by listing them after --tasks.

# Full MMLU evaluation (57 subjects)
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu \
  --num_fewshot 5 \  # 5-shot evaluation (standard)
  --batch_size 8 \
  --output_path results/ \
  --log_samples  # Save individual predictions

# Multiple benchmarks at once
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \
  --num_fewshot 5 \
  --batch_size 8 \
  --output_path results/llama2-7b-eval.json

Step 4: analyze the results.

Results are saved to results/llama2-7b-eval.json as JSON with per-task accuracy metrics such as mmlu acc, gsm8k exact_match and hellaswag acc_norm, each with a stderr, plus the run config.

{
  "results": {
    "mmlu": {
      "acc": 0.459,
      "acc_stderr": 0.004
    },
    "gsm8k": {
      "exact_match": 0.142,
      "exact_match_stderr": 0.006
    },
    "hellaswag": {
      "acc_norm": 0.765,
      "acc_norm_stderr": 0.004
    }
  },
  "config": {
    "model": "hf",
    "model_args": "pretrained=meta-llama/Llama-2-7b-hf",
    "num_fewshot": 5
  }
}

Workflow 2: track training progress by evaluating checkpoints during training.

Training Progress Tracking:
- [ ] Step 1: Set up periodic evaluation
- [ ] Step 2: Choose quick benchmarks
- [ ] Step 3: Automate evaluation
- [ ] Step 4: Plot learning curves

Step 1: set up periodic evaluation. In a script like eval_checkpoint.sh, take the checkpoint directory and step as arguments and run lm_eval on checkpoint-$STEP with tasks gsm8k,hellaswag, --num_fewshot 0 (0-shot for speed), --batch_size 16 and --output_path results/step-$STEP.json, evaluating every N training steps.

#!/bin/bash
# eval_checkpoint.sh

CHECKPOINT_DIR=$1
STEP=$2

lm_eval --model hf \
  --model_args pretrained=$CHECKPOINT_DIR/checkpoint-$STEP \
  --tasks gsm8k,hellaswag \
  --num_fewshot 0 \  # 0-shot for speed
  --batch_size 16 \
  --output_path results/step-$STEP.json

Step 2: choose quick benchmarks for frequent evaluation.

Avoid slow benchmarks for frequent eval: MMLU takes about 2 hours (57 subjects) and HumanEval requires code execution.

Fast options for frequent evaluation include HellaSwag (~10 minutes on 1 GPU), GSM8K (~5 minutes) and PIQA (~2 minutes).

Slow options to avoid for frequent eval: MMLU takes about 2 hours (57 subjects) and HumanEval requires code execution.

Step 3: automate evaluation.

Integrate it with your training script by saving a checkpoint every eval_interval steps and running ./eval_checkpoint.sh checkpoints step-{step}.

# In training loop
if step % eval_interval == 0:
    model.save_pretrained(f"checkpoints/step-{step}")

    # Run evaluation
    os.system(f"./eval_checkpoint.sh checkpoints step-{step}")

Alternatively, use PyTorch Lightning callbacks: define an EvalHarnessCallback that on validation epoch end saves the checkpoint and runs lm_eval on it.

from pytorch_lightning import Callback

class EvalHarnessCallback(Callback):
    def on_validation_epoch_end(self, trainer, pl_module):
        step = trainer.global_step
        checkpoint_path = f"checkpoints/step-{step}"

        # Save checkpoint
        trainer.save_checkpoint(checkpoint_path)

        # Run lm-eval
        os.system(f"lm_eval --model hf --model_args pretrained={checkpoint_path} ...")

Step 4: plot learning curves — load all results files, collect the MMLU accuracy per training step, and plot the training-progress curve with matplotlib.

import json
import matplotlib.pyplot as plt

# Load all results
steps = []
mmlu_scores = []

for file in sorted(glob.glob("results/step-*.json")):
    with open(file) as f:
        data = json.load(f)
        step = int(file.split("-")[1].split(".")[0])
        steps.append(step)
        mmlu_scores.append(data["results"]["mmlu"]["acc"])

# Plot
plt.plot(steps, mmlu_scores)
plt.xlabel("Training Step")
plt.ylabel("MMLU Accuracy")
plt.title("Training Progress")
plt.savefig("training_curve.png")

Workflow 3: compare multiple models using a benchmark suite for model comparison.

Model Comparison:
- [ ] Step 1: Define model list
- [ ] Step 2: Run evaluations
- [ ] Step 3: Generate comparison table

Step 1: define the model list, for example a models.txt file with one HuggingFace model name per line.

# models.txt
meta-llama/Llama-2-7b-hf
meta-llama/Llama-2-13b-hf
mistralai/Mistral-7B-v0.1
microsoft/phi-2

Step 2: run the evaluations with a script like eval_all_models.sh that reads models.txt, converts each model name to a file-safe output name, and runs lm_eval with tasks mmlu,gsm8k,hellaswag,truthfulqa, --num_fewshot 5, --batch_size auto and --output_path results/$model_name.json.

#!/bin/bash
# eval_all_models.sh

TASKS="mmlu,gsm8k,hellaswag,truthfulqa"

while read model; do
    echo "Evaluating $model"

    # Extract model name for output file
    model_name=$(echo $model | sed 's/\//-/g')

    lm_eval --model hf \
      --model_args pretrained=$model,dtype=bfloat16 \
      --tasks $TASKS \
      --num_fewshot 5 \
      --batch_size auto \
      --output_path results/$model_name.json

done < models.txt

Step 3: generate a comparison table — load each model's results JSON, take the primary metric per task (acc or exact_match), and build a pandas DataFrame rendered with to_markdown.

import json
import pandas as pd

models = [
    "meta-llama-Llama-2-7b-hf",
    "meta-llama-Llama-2-13b-hf",
    "mistralai-Mistral-7B-v0.1",
    "microsoft-phi-2"
]

tasks = ["mmlu", "gsm8k", "hellaswag", "truthfulqa"]

results = []
for model in models:
    with open(f"results/{model}.json") as f:
        data = json.load(f)
        row = {"Model": model.replace("-", "/")}
        for task in tasks:
            # Get primary metric for each task
            metrics = data["results"][task]
            if "acc" in metrics:
                row[task.upper()] = f"{metrics['acc']:.3f}"
            elif "exact_match" in metrics:
                row[task.upper()] = f"{metrics['exact_match']:.3f}"
        results.append(row)

df = pd.DataFrame(results)
print(df.to_markdown(index=False))

| Model                  | MMLU  | GSM8K | HELLASWAG | TRUTHFULQA |
|------------------------|-------|-------|-----------|------------|
| meta-llama/Llama-2-7b  | 0.459 | 0.142 | 0.765     | 0.391      |
| meta-llama/Llama-2-13b | 0.549 | 0.287 | 0.801     | 0.430      |
| mistralai/Mistral-7B   | 0.626 | 0.395 | 0.812     | 0.428      |
| microsoft/phi-2        | 0.560 | 0.613 | 0.682     | 0.447      |

Workflow 4: evaluate with vLLM for faster inference.

vLLM Evaluation:
- [ ] Step 1: Install vLLM
- [ ] Step 2: Configure vLLM backend
- [ ] Step 3: Run evaluation

Step 1: install vLLM with pip install vllm.

pip install vllm

Step 2: configure the vLLM backend — run lm_eval with --model vllm and model_args like pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8, plus --tasks mmlu and --batch_size auto.

lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \
  --tasks mmlu \
  --batch_size auto

Step 3: run the evaluation. For a 7B model, full MMLU takes about 2 hours with the standard HF backend but about 15–20 minutes with vLLM, for example with tensor_parallel_size=2.

# Standard HF: ~2 hours for MMLU on 7B model
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu \
  --batch_size 8

# vLLM: ~15-20 minutes for MMLU on 7B model
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=2 \
  --tasks mmlu \
  --batch_size auto

For slow evaluations, use the vLLM backend.

Use the vLLM backend: lm_eval --model vllm with model_args such as pretrained=model-name,tensor_parallel_size=2.

lm_eval --model vllm \
  --model_args pretrained=model-name,tensor_parallel_size=2

Or reduce fewshot examples, e.g. --num_fewshot 0 instead of 5.

--num_fewshot 0  # Instead of 5

Or evaluate a subset of MMLU, e.g. --tasks mmlu_stem for only the STEM subjects.

--tasks mmlu_stem  # Only STEM subjects

For out-of-memory issues, reduce the batch size, e.g. --batch_size 1 or --batch_size auto.

--batch_size 1  # Or --batch_size auto

Or use quantization by adding load_in_8bit=True to the model_args.

--model_args pretrained=model-name,load_in_8bit=True

Or enable CPU offloading with model_args like pretrained=model-name,device_map=auto,offload_folder=offload.

--model_args pretrained=model-name,device_map=auto,offload_folder=offload

For inaccurate results, check the fewshot count — most papers use 5-shot (--num_fewshot 5).

--num_fewshot 5  # Most papers use 5-shot

For task-not-found errors, check the exact task name, e.g. --tasks mmlu, not mmlu_direct or mmlu_fewshot.

--tasks mmlu  # Not mmlu_direct or mmlu_fewshot

Verify that the model and tokenizer match, e.g. model_args pretrained=model-name,tokenizer=same-model-name.

--model_args pretrained=model-name,tokenizer=same-model-name

Code-executing tasks (HumanEval, MBPP, etc.) are gated behind an explicit confirmation flag — you must pass --confirm_run_unsafe_code to run them.

lm_eval --model hf \
  --model_args pretrained=model-name \
  --tasks humaneval \
  --confirm_run_unsafe_code  # Required to run tasks that execute generated code

Without this flag, lm-eval refuses to run the task rather than silently skipping code execution.

Links

• references/benchmark-guide.md
• references/custom-tasks.md
• references/api-evaluation.md
• references/distributed-eval.md