Zero-shot image classification and image-text search with CLIP
Use OpenAI's CLIP model to classify images into natural-language categories without any training data.
Requirements
- Install CLIP together with transformers, torch, and pillow; the model runs on Linux, macOS, and Windows and is released under the MIT License.
How to use
CLIP is built for zero-shot classification, image-text matching, semantic image search, content moderation, visual question answering, and cross-modal retrieval between images and text.
For captioning tasks, BLIP-2 is an alternative that performs better.
For vision-language chat, LLaVA is an alternative.
For image segmentation, Segment Anything is an alternative.
Model choice is a trade-off: RN50 (102M parameters) is fast with good quality, ViT-B/32 (151M) is medium-speed with better quality, and ViT-L/14 (428M) is slow but delivers the best quality.
Expect roughly 200 ms per image encoding and 50 ms per text encoding on CPU versus about 20 ms and 5 ms on a V100 GPU, with similarity computation under 1 ms on either.
CLIP is not suited to fine-grained tasks; it works best for broad categories.
It requires descriptive text: vague labels perform poorly.
Because it was trained on web data, results may carry dataset biases.
It cannot produce bounding boxes and works on the whole image only.
Its spatial understanding is limited, so positioning and counting are weak.
The model is distributed under the MIT License.
Install CLIP from its GitHub repository and then install torch, torchvision, ftfy, regex, and tqdm.
pip install git+https://github.com/openai/CLIP.git
pip install torch torchvision ftfy regex tqdm
To classify an image with no training, load a CLIP model on the available device, preprocess your image, tokenize a list of candidate labels, and let the model output a probability for each label.
import torch
import clip
from PIL import Image
# Load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
# Load image
image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)
# Define possible labels
text = clip.tokenize(["a dog", "a cat", "a bird", "a car"]).to(device)
# Compute similarity
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
# Cosine similarity
logits_per_image, logits_per_text = model(image, text)
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
# Print results
labels = ["a dog", "a cat", "a bird", "a car"]
for label, prob in zip(labels, probs[0]):
print(f"{label}: {prob:.2%}")
Pick a model by size and quality: RN50, RN101, ViT-B/32 (recommended), ViT-B/16 (better quality but slower), or ViT-L/14 (best quality but slowest).
# Models (sorted by size)
models = [
"RN50", # ResNet-50
"RN101", # ResNet-101
"ViT-B/32", # Vision Transformer (recommended)
"ViT-B/16", # Better quality, slower
"ViT-L/14", # Best quality, slowest
]
model, preprocess = clip.load("ViT-B/32")
To measure how well an image matches a text description, encode both, normalize the resulting feature vectors, and compute their cosine similarity.
# Compute embeddings
image_features = model.encode_image(image)
text_features = model.encode_text(text)
# Normalize
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
# Cosine similarity
similarity = (image_features @ text_features.T).item()
print(f"Similarity: {similarity:.4f}")
To search a collection of images by meaning, precompute normalized embeddings for every image, encode the text query the same way, and return the top-k images ranked by similarity.
# Index images
image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
image_embeddings = []
for img_path in image_paths:
image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
with torch.no_grad():
embedding = model.encode_image(image)
embedding /= embedding.norm(dim=-1, keepdim=True)
image_embeddings.append(embedding)
image_embeddings = torch.cat(image_embeddings)
# Search with text query
query = "a sunset over the ocean"
text_input = clip.tokenize([query]).to(device)
with torch.no_grad():
text_embedding = model.encode_text(text_input)
text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
# Find most similar images
similarities = (text_embedding @ image_embeddings.T).squeeze(0)
top_k = similarities.topk(3)
for idx, score in zip(top_k.indices, top_k.values):
print(f"{image_paths[idx]}: {score:.3f}")
To screen content, define moderation categories such as safe for work, not safe for work, violent content, and graphic content, then classify the image and report the most likely category with its probability.
# Define categories
categories = [
"safe for work",
"not safe for work",
"violent content",
"graphic content"
]
text = clip.tokenize(categories).to(device)
# Check image
with torch.no_grad():
logits_per_image, _ = model(image, text)
probs = logits_per_image.softmax(dim=-1)
# Get classification
max_idx = probs.argmax().item()
max_prob = probs[0, max_idx].item()
print(f"Category: {categories[max_idx]} ({max_prob:.2%})")
For efficiency, process batches of images and texts at once and obtain a full similarity matrix between them.
# Process multiple images
images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]
images = torch.stack(images).to(device)
with torch.no_grad():
image_features = model.encode_image(images)
image_features /= image_features.norm(dim=-1, keepdim=True)
# Batch text
texts = ["a dog", "a cat", "a bird"]
text_tokens = clip.tokenize(texts).to(device)
with torch.no_grad():
text_features = model.encode_text(text_tokens)
text_features /= text_features.norm(dim=-1, keepdim=True)
# Similarity matrix (10 images × 3 texts)
similarities = image_features @ text_features.T
print(similarities.shape) # (10, 3)
To scale search, store CLIP embeddings in a vector database such as Chroma and query it with a text embedding to retrieve the closest images.
# Store CLIP embeddings in Chroma/FAISS
import chromadb
client = chromadb.Client()
collection = client.create_collection("image_embeddings")
# Add image embeddings
for img_path, embedding in zip(image_paths, image_embeddings):
collection.add(
embeddings=[embedding.cpu().numpy().tolist()],
metadatas=[{"path": img_path}],
ids=[img_path]
)
# Query with text
query = "a sunset"
text_embedding = model.encode_text(clip.tokenize([query]))
results = collection.query(
query_embeddings=[text_embedding.cpu().numpy().tolist()],
n_results=5
)
ViT-B/32 is a good balance for most cases.
Normalize embeddings, which is required for cosine similarity.
Use batch processing for more efficient runs.
Cache embeddings, since recomputing them is expensive.
Use descriptive labels to get better zero-shot performance.
A GPU is recommended and makes processing 10-50× faster.
Preprocess images with the provided preprocess function.
The license is MIT.
Source: nousresearch/hermes-agent and its contributors. License: MIT.
Copyright (c) 2025 Nous Research
Translation and editorial adaptation by ClawCases.
>Resources
>Related Use Cases
Agent Merge-Conflict Arbiter
Resolve a git merge conflict between two agents' branches as an impartial third party: a neutral arbiter that takes both diffs and their stated intents and produces a merged result without overwriting either side.
Autonomous Project Management with Subagents
Coordinate projects autonomously: a main agent delegates work to PM subagents that track tasks in a shared STATE.yaml, update it as they progress, pick up unblocked work in parallel, and keep all task state in version-controlled files.
Controlling Home Assistant conversationally with Hermes
Put a conversational agent in front of Home Assistant so questions like the kitchen temperature or turning off downstairs lights are faster to ask than to find in a dense dashboard.