Benchmark Client

Run this on your machine to benchmark your favorite local LLM. Get concrete numbers for speed, memory, code quality, reasoning, and more. Your results help others find the right LLM for their needs.

Download

Quick Install (Recommended)

Linux/macOS

curl -fsSL https://llm-bench.io/install.sh | bash

Windows

irm https://llm-bench.io/install.ps1 | iex

Security & Code Signing

The script-based install flow above is the recommended install path. Code signing is still in progress.

How it works

You need a local inference server running with at least one model available. The client detects oMLX, Ollama, llama.cpp, LM Studio, Unsloth Studio, vLLM, SGLang and Text Generation Inference automatically. With Unsloth Studio it loads the model you pick at the context length you pick, then puts back whatever was loaded before — so nothing needs preparing there first. Most runs finish in 5–15 minutes; reasoning models left on a high thinking effort take considerably longer (why). Benchmarks test your model against real-world scenarios: code generation, reasoning, roleplay, and more.

1
Run the client
Open a terminal and run llm-benchmark. The installer puts it on your PATH, so it works from anywhere. It starts an interactive prompt.
2
Select your AI tool and model
The client automatically finds any running AI tools on your machine and lists the available models.
3
Wait for the benchmark
It sends a series of test prompts to your model and measures how fast it responds.
4
Results are submitted
Once done, the results are automatically added to the community leaderboard so everyone can compare.
View all benchmark results →

What data is collected

What is sent

  • ✓GPU model and video memory size
  • ✓CPU model and number of cores
  • ✓Total system RAM
  • ✓Operating system name and version
  • ✓AI tool name and version (e.g. Ollama 0.3)
  • ✓Model name and parameter size (e.g. llama3 8B)
  • ✓Benchmark results: generation speed, prompt-processing speed, memory usage, time to first token
  • ✓The benchmark prompt texts and your model's responses to them (for server-side output quality scoring)
  • ✓The inference settings the run actually used: samplers, engine feature flags, context length, and your server's per-model settings, with anything path-shaped removed
  • ✓The reasoning effort the run resolved to, read from the model's chat template

What is never sent

  • ✕Your name, email, or any personal information
  • ✕Your model weights, or the contents of any local file — the client reads your model's chat template on disk to work out its reasoning-effort setting, and uploads only that setting
  • ✕Your own conversations or chat history

What is collected for site operations

  • ◆Your IP address (for rate limiting and server logs, not stored long-term)

Other requests the client makes

  • ◆When your model is named as a Hugging Face repository (for example org/model), the client fetches that repo's chat template from huggingface.co to read its reasoning-effort default. Only the public repository name is sent, and only when the local files do not already answer the question.
The only purpose of data collection is to power the community comparison tables on this site. All submitted results are public and visible to everyone.