Structured LLM Generation with Outlines
Use Outlines when you need guaranteed valid JSON/XML/code from an LLM, type-safe Pydantic outputs, grammar-level token-sampling control, or support for local models via Transformers, llama.cpp, and vLLM.
Requirements
- Requires the outlines package and Pydantic, plus only the backend used by the selected route: Transformers, llama-cpp-python, vLLM, or another supported provider.
How to use
In v1 you pass the desired output type directly as the second argument to the model call.
Outlines supports multiple local and API-based backends through from_* factories.
Outlines has first-class Pydantic support with automatic schema translation: generation returns a JSON string, and you call model_validate_json to get an instance.
This skill targets the Outlines 1.x API, where the pre-1.0 helpers outlines.models.transformers(...) and outlines.generate.json/choice/regex/... have been removed. Create a model with outlines.from_transformers(...) (or from_vllm, from_llamacpp, from_openai) and call the model directly with an output type, as in model(prompt, output_type). JSON/Pydantic outputs are returned as a JSON string — validate them with YourModel.model_validate_json(result).
Install the base outlines package, adding the extra dependency only for the backend you plan to use — transformers for Hugging Face models, llama-cpp-python for llama.cpp, or vllm for high-throughput serving.
# Base installation
pip install outlines
# With specific backends
pip install outlines transformers # Hugging Face models
pip install outlines llama-cpp-python # llama.cpp
pip install outlines vllm # vLLM for high-throughput
Create a basic classification model by wrapping a Hugging Face causal LM and its tokenizer, then call the model directly with a Literal output type to guarantee one of the listed choices.
import outlines
from typing import Literal
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
# v1: wrap a Transformers model + tokenizer
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME),
)
# Call the model directly with an output type
prompt = "Sentiment of 'This product is amazing!': "
sentiment = model(prompt, Literal["positive", "negative", "neutral"])
print(sentiment) # "positive" (guaranteed one of these)
Use a Pydantic model as the output type to extract structured fields; the call returns a JSON string that you parse into the model with model_validate_json.
from pydantic import BaseModel
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
class User(BaseModel):
name: str
age: int
email: str
MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME),
)
# Generate structured output (returns a JSON string)
prompt = "Extract user: John Doe, 30 years old, john@example.com"
result = model(prompt, User, max_new_tokens=200)
user = User.model_validate_json(result) # parse into the Pydantic model
print(user.name) # "John Doe"
print(user.age) # 30
print(user.email) # "john@example.com"
Constrained token sampling makes invalid outputs impossible: pass the output type (such as a Pydantic model) as the second argument and validate the returned JSON string.
import outlines
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer
class Person(BaseModel):
name: str
age: int
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),
AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)
result = model("Generate person: Alice, 25", Person)
person = Person.model_validate_json(result)
For multiple choice, constrain the output to a fixed set of options with Literal, so the result is always one of the listed choices.
from typing import Literal
sentiment = model("Review: This is great!", Literal["positive", "negative", "neutral"])
# Result: one of the three choices
For JSON output, define a Pydantic model with typed fields and parse the returned JSON string into a valid model instance.
from pydantic import BaseModel
class Product(BaseModel):
name: str
price: float
in_stock: bool
result = model("Extract: iPhone 15, $999, available", Product)
product = Product.model_validate_json(result) # valid Product instance
Pass a regex string as the output type to generate text guaranteed to match the pattern, such as a phone number.
# Generate text matching a regex pattern
phone = model("Generate phone number:", r"[0-9]{3}-[0-9]{3}-[0-9]{4}")
# Result: "555-123-4567" (guaranteed to match the pattern)
Pass the Python type directly to guarantee numeric outputs such as an int age or a float price.
# Pass the Python type directly
age = model("Person's age:", int) # guaranteed integer
price = model("Product price:", float) # guaranteed float
For the Transformers (Hugging Face) backend, wrap a pretrained causal LM and tokenizer and call the model with your output type.
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),
AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)
result = model(prompt, YourModel)
For the llama.cpp backend, load a GGUF model and wrap it with from_llamacpp before calling it with your output type.
import outlines
from llama_cpp import Llama
llm = Llama("./models/llama-3.1-8b-instruct.Q4_K_M.gguf", n_gpu_layers=35, n_ctx=4096)
model = outlines.from_llamacpp(llm)
result = model(prompt, YourModel)
For the vLLM backend, suited to high-throughput workloads, wrap an LLM instance with from_vllm and call the model with your output type.
import outlines
from vllm import LLM
llm = LLM("meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=2)
model = outlines.from_vllm(llm)
result = model(prompt, YourModel)
For OpenAI, wrap a client with from_openai; API backends support JSON-schema style server-side constrained output.
import outlines
from openai import OpenAI
client = OpenAI()
model = outlines.from_openai(client, "gpt-4o-mini")
# API backends support JSON-schema style structured output
result = model(prompt, YourModel)
Define basic Pydantic models with typed and described fields, then validate the returned JSON string into an instance.
from pydantic import BaseModel, Field
class Article(BaseModel):
title: str = Field(description="Article title")
author: str = Field(description="Author name")
word_count: int = Field(description="Number of words", gt=0)
tags: list[str] = Field(description="List of tags")
result = model("Generate article about AI", Article, max_new_tokens=300)
article = Article.model_validate_json(result)
print(article.title)
print(article.word_count) # Guaranteed > 0
Compose nested Pydantic models so a generated object includes structured sub-objects, such as an address inside a person.
class Address(BaseModel):
street: str
city: str
country: str
class Person(BaseModel):
name: str
age: int
address: Address # Nested model
result = model("Generate person in New York", Person)
person = Person.model_validate_json(result)
print(person.address.city) # "New York"
Use Enums and Literal types for category fields so values must come from a fixed set.
from enum import Enum
from typing import Literal
class Status(str, Enum):
PENDING = "pending"
APPROVED = "approved"
REJECTED = "rejected"
class Application(BaseModel):
applicant: str
status: Status # Must be one of enum values
priority: Literal["low", "medium", "high"] # Must be one of literals
result = model("Generate application", Application)
app = Application.model_validate_json(result)
print(app.status) # Status.PENDING (or APPROVED/REJECTED)
Apply the data extraction pattern: define a Pydantic model for the fields you need, prompt with the source text, and validate the returned JSON.
from pydantic import BaseModel
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
class CompanyInfo(BaseModel):
name: str
founded_year: int
industry: str
employees: int
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),
AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)
text = """
Apple Inc. was founded in 1976 in the technology industry.
The company employs approximately 164,000 people worldwide.
"""
prompt = f"Extract company information:\n{text}\n\nCompany:"
company = CompanyInfo.model_validate_json(model(prompt, CompanyInfo, max_new_tokens=200))
print(f"Name: {company.name}")
print(f"Founded: {company.founded_year}")
print(f"Industry: {company.industry}")
print(f"Employees: {company.employees}")
Apply the classification pattern using Literal output types for binary or multi-class labels, or a Pydantic model to include a confidence value.
from typing import Literal
from pydantic import BaseModel
# Binary classification
result = model("Email: Buy now! 50% off!", Literal["spam", "not_spam"])
# Multi-class classification
category = model(
"Article: Apple announces new iPhone...",
Literal["technology", "business", "sports", "entertainment"],
)
# With confidence
class Classification(BaseModel):
label: Literal["positive", "negative", "neutral"]
confidence: float
out = model("Review: This product is okay, nothing special", Classification)
result = Classification.model_validate_json(out)
Apply the structured forms pattern to extract a full user profile with typed fields and lists into a validated Pydantic instance.
class UserProfile(BaseModel):
full_name: str
age: int
email: str
phone: str
country: str
interests: list[str]
prompt = """
Extract user profile from:
Name: Alice Johnson
Age: 28
Email: alice@example.com
Phone: 555-0123
Country: USA
Interests: hiking, photography, cooking
"""
profile = UserProfile.model_validate_json(model(prompt, UserProfile, max_new_tokens=250))
print(profile.full_name)
print(profile.interests) # ["hiking", "photography", "cooking"]
Apply the multi-entity extraction pattern with a document-level model holding a list of typed entities.
from typing import Literal
class Entity(BaseModel):
name: str
type: Literal["PERSON", "ORGANIZATION", "LOCATION"]
class DocumentEntities(BaseModel):
entities: list[Entity]
text = "Tim Cook met with Satya Nadella at Microsoft headquarters in Redmond."
prompt = f"Extract entities from: {text}"
result = DocumentEntities.model_validate_json(model(prompt, DocumentEntities, max_new_tokens=300))
for entity in result.entities:
print(f"{entity.name} ({entity.type})")
Apply the code generation pattern by generating a Pydantic-structured function specification and assembling the code from its fields.
class PythonFunction(BaseModel):
function_name: str
parameters: list[str]
docstring: str
body: str
prompt = "Generate a Python function to calculate factorial"
func = PythonFunction.model_validate_json(model(prompt, PythonFunction, max_new_tokens=300))
print(f"def {func.function_name}({', '.join(func.parameters)}):")
print(f' """{func.docstring}"""')
print(f" {func.body}")
For batch processing, v1 accepts a list of prompts in a single call; validate each returned JSON string into a model instance.
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
from pydantic import BaseModel
class Person(BaseModel):
name: str
age: int
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),
AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)
texts = [
"John is 30 years old",
"Alice is 25 years old",
"Bob is 40 years old",
]
# v1 accepts a list of prompts for batched generation
prompts = [f"Extract from: {t}" for t in texts]
outputs = model(prompts, Person, max_new_tokens=100)
people = [Person.model_validate_json(o) for o in outputs]
for person in people:
print(f"{person.name}: {person.age}")
For Transformers, GPU and dtype configuration is set on the Hugging Face model itself when you wrap it with from_transformers.
import outlines
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"
# Basic usage
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME),
)
# GPU + dtype configuration is set on the HF model itself
import torch
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="cuda", torch_dtype=torch.float16),
AutoTokenizer.from_pretrained(MODEL_NAME),
)
# Popular models
for name in [
"meta-llama/Llama-3.1-8B-Instruct",
"mistralai/Mistral-7B-Instruct-v0.3",
"Qwen/Qwen2.5-7B-Instruct",
]:
model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(name, device_map="auto"),
AutoTokenizer.from_pretrained(name),
)
For llama.cpp, configure the context window, GPU layers, and CPU threads on the Llama object; set n_gpu_layers to -1 for full GPU offload.
import outlines
from llama_cpp import Llama
# Load GGUF model
llm = Llama(
"./models/llama-3.1-8b.Q4_K_M.gguf",
n_ctx=4096, # Context window
n_gpu_layers=35, # GPU layers
n_threads=8, # CPU threads
)
model = outlines.from_llamacpp(llm)
# Full GPU offload: set n_gpu_layers=-1 on the Llama object
For vLLM in production, use tensor_parallel_size for multi-GPU setups and quantization options such as AWQ on the LLM instance.
import outlines
from vllm import LLM
# Single GPU
model = outlines.from_vllm(LLM("meta-llama/Llama-3.1-8B-Instruct"))
# Multi-GPU
model = outlines.from_vllm(LLM("meta-llama/Llama-3.1-70B-Instruct", tensor_parallel_size=4))
# With quantization
model = outlines.from_vllm(LLM("meta-llama/Llama-3.1-8B-Instruct", quantization="awq"))
Use specific Pydantic field types such as float, int, and bool instead of strings for everything.
# ✅ Good: Specific types
class Product(BaseModel):
name: str
price: float # Not str
quantity: int # Not str
in_stock: bool # Not str
# ❌ Bad: Everything as string
class Product(BaseModel):
name: str
price: str # Should be float
quantity: str # Should be int
Add Field constraints to your Pydantic models, such as length bounds, numeric ranges, and regex patterns.
from pydantic import Field
# ✅ Good: With constraints
class User(BaseModel):
name: str = Field(min_length=1, max_length=100)
age: int = Field(ge=0, le=120)
email: str = Field(pattern=r"^[\w\.-]+@[\w\.-]+\.\w+$")
# ❌ Bad: No constraints
class User(BaseModel):
name: str
age: int
email: str
Use Enums for fields with a fixed set of categories instead of free-form strings.
# ✅ Good: Enum for fixed set
class Priority(str, Enum):
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
class Task(BaseModel):
title: str
priority: Priority
# ❌ Bad: Free-form string
class Task(BaseModel):
title: str
priority: str # Can be anything
Provide clear context in your prompts rather than sending minimal text alone.
# ✅ Good: Clear context
prompt = """
Extract product information from the following text.
Text: iPhone 15 Pro costs $999 and is currently in stock.
Product:
"""
# ❌ Bad: Minimal context
prompt = "iPhone 15 Pro costs $999 and is currently in stock."
Model fields that may be missing as Optional with defaults so extraction can succeed even when some data is incomplete.
from typing import Optional
# ✅ Good: Optional fields for incomplete data
class Article(BaseModel):
title: str # Required
author: Optional[str] = None # Optional
date: Optional[str] = None # Optional
tags: list[str] = [] # Default empty list
# Can succeed even if author/date missing
Always validate JSON output: v1 returns a JSON string for Pydantic/JSON output types, so parse it with model_validate_json to get an instance.
# v1 returns a JSON string for Pydantic/JSON output types.
result = model(prompt, Article) # str
article = Article.model_validate_json(result) # Article instance
Source: nousresearch/hermes-agent and its contributors. License: MIT.
Copyright (c) 2025 Nous Research
Translation and editorial adaptation by ClawCases.
>Resources
>Related Use Cases
Agent Merge-Conflict Arbiter
Resolve a git merge conflict between two agents' branches as an impartial third party: a neutral arbiter that takes both diffs and their stated intents and produces a merged result without overwriting either side.
Autonomous Project Management with Subagents
Coordinate projects autonomously: a main agent delegates work to PM subagents that track tasks in a shared STATE.yaml, update it as they progress, pick up unblocked work in parallel, and keep all task state in version-controlled files.
Health & Symptom Tracker
This agent logs food intake and symptoms so the user can review possible associations in their own journal. Timestamped entries and reminders support consistent logging without treating correlations as confirmed causes.