When I started testing LLMs on my AMD Radeon RX7900XTX, I realized I needed more than just speed numbers. I needed to know: can this model handle tools? Does it write good code? Can it reason effectively? This is how we benchmark to answer those real questions.
We test models with 4 distinct scenarios, each evaluating different AI capabilities. Every model gets the same four prompts and runs with its own default sampling settings. The benchmark doesn't set a temperature.
Most models ship with recommended sampling settings (temperature, top-p, top-k), and the inference tool applies them unless it's told otherwise. Overriding them would test our choice of settings rather than the model as people actually run it.
Where the tool reports the settings it used, they're recorded with the run and shown on its run page under Inference Configuration, so you can see what a result was produced with.
These metrics measure how fast the model runs and how efficiently it uses hardware. All metrics are collected automatically during benchmark execution.
Unit: tokens per second (tok/s)
The speed at which the model generates output tokens after processing the input. Reasoning tokens count as output: a thinking model's reasoning is generated text like any other.
Higher is better. Typically 10-150 tok/s depending on model size and hardware. Small and mixture-of-experts models, especially with speculative decoding, can go well beyond that.
Below 5 tok/s usually means part of the model is running on the CPU, or the model is too large for the hardware.
generated_tokens ÷ generation_time (seconds). Generation time is the request's total time minus its time to first token, or the server-reported generation duration on Ollama. The run's value is the average over the four scenarios.Unit: tokens per second (tok/s)
The speed at which the model reads the input before it writes anything — the prefill phase. Shown as PP wherever a run is displayed. It is the counterpart to token generation speed: PP is how fast the prompt goes in, tok/s is how fast the answer comes out.
Higher is better, and it is usually several times the generation speed. It matters most for long prompts and agent loops, where each turn re-reads a growing context.
A value close to the generation speed suggests the prompt is being processed token-by-token rather than in a batch, which makes long-context work expensive.
prompt_tokens ÷ prefill_time (seconds). The token count is the real count the server reports, never a word-count estimate. Prefill time is the measured time to first token for streaming backends, or the server-reported prompt evaluation duration on Ollama.Unit: milliseconds (ms)
The latency between sending a request and receiving the first output token. Measures "perceived responsiveness." Interpretation depends on the scenario being benchmarked, as different use cases have different latency expectations.
Lower is better, but thresholds depend on use case: • Code Generation: ≤400ms (IDE/completion integration) • Agent Workflow: ≤500ms (multi-step reasoning) • Role Play & Narrative: ≤300ms (narrative flow) • Research & Analysis: ≤1000ms (analytical work)
Performance threshold depends on use case: • Code Generation: >1200ms (breaks IDE integration experience) • Agent Workflow: >1500ms (compounds across multiple steps) • Role Play & Narrative: >800ms (breaks immersion) • Research & Analysis: >3000ms (frustrating for analytical work)
Directly measured via HTTP streaming: T1 − T0, where T0 = timestamp immediately before sending the request, T1 = timestamp when the first non-empty content chunk arrives. For a thinking model the first token is usually reasoning, so TTFT shows how quickly the model starts, not how long until the visible answer begins. If streaming fails and the client retries without it, a true first-token time can't be measured; the value recorded then is the average time per generated token, which isn't comparable to a streamed TTFT.Unit: gigabytes (GB)
Memory the inference server holds for the model: weights, KV cache and compute buffers, split into GPU memory and system RAM where both are used. Each run records what kind of figure it has: measured, reported by the tool, reserved, or estimated.
Lower is better for given model quality. Allows more models/instances on same hardware.
Usage approaching available VRAM causes slowdown (memory swapping). Usage above available RAM causes out-of-memory errors.
Reported memory (in MB) ÷ 1024 = GBUnit: percentage (%)
What fraction of the model is held in GPU memory rather than system RAM. 100% means the entire model is on the GPU (fastest). Only Ollama reports this; runs from other tools leave it blank.
100% = entire model on GPU = best performance. 50%+ = good performance. Below 50% = significant CPU overhead.
0% = model running entirely on CPU = slowest possible inference.
size_vram ÷ size × 100, as reported by Ollama for the loaded model (bytes, not parameters)Unit: tokens per second per GB (tok/s/GB)
Normalized speed considering memory cost. How many tokens can the model generate per second for each gigabyte of memory used?
Higher is better. Shows quality/performance per unit of hardware cost. 0.5-2.0 is solid range.
Below 0.1 indicates inefficient hardware utilization.
tokenGenerationSpeed ÷ (memoryUsage_in_MB ÷ 1024)Token Estimation: When actual token counts aren't available from the inference server, we estimate tokens as:
estimated_tokens = word_count × 1.3
This 1.3 multiplier is empirically derived as an average ratio of tokens to words in English text.
We use an LLM to evaluate model responses across multiple dimensions. Each scenario has specific evaluation criteria designed to measure what matters most for that task type.
We evaluate whether the agent breaks down complex problems logically, selects appropriate tools for each step, considers error cases, and explains reasoning clearly. The checklist verifies: reasoning coherence (steps connected?), tool appropriateness (correct tools for each step?), decomposition quality (manageable sub-tasks?), and error handling (risks identified and mitigated?).
Character consistency is verified through emotional triggers (does Kestra mention hit authentically?), trust progression (earned gradually, not given freely?), and internal conflict (character torn between mercenary instinct and humanity?). Immersion requires sensory grounding (tavern smells, body language, emotional texture). Dialogue must be voice-authentic and reveal character through language. Narrative arc must show clear progression from resistance through warming to a binary, character-driven choice.
Depth checks: Are all required sections present? Is scaling analyzed per-task (not overall)? Are calculations shown, not claimed? Insight checks: Do comparisons show understanding of task-specific differences? Are patterns explained or just observed? Interpretation checks: Can recommendations be verified from data? Are predictions justified with confidence levels and assumptions? Are limitations quantified (±X percentage points)? Clarity checks: Are recommendations specific and tied to actual scores/tradeoffs?
Correctness requires verifiable code inspection: state management clear (grid, ball, paddle tracked)? Ball physics correct (velocity magnitude, direction initialization)? Collision detection complete (ball-wall, ball-paddle, ball-brick all verified)? Paddle control responsive (keyboard + mobile touch)? Rendering uses Canvas (not DOM)? Game over logic triggers correctly? Frame rate using requestAnimationFrame? We inspect the code structure rather than running it, so bugs are visible: off-by-one collisions, velocity math errors, missing event handlers.
For each dimension, the evaluator model (currently GPT-5.4 mini, at temperature 0.5) assigns a score from 0-100 based on the quality definition for that scenario. Specific criteria and checklists guide the scoring to keep it consistent. If the evaluator's reply is missing the score for a dimension, that dimension counts as 0.
The KPI (Key Performance Indicator) for each scenario is calculated as a weighted average:
KPI = Σ(dimension_score × dimension_weight)
The overall quality score of a run is the average of its four scenario KPIs. A run only gets an overall score once all four scenarios have been scored.
For reasoning models that narrate their chain-of-thought, the evaluator scores only the answer the model actually delivered. Any reasoning wrapped in <think>…</think> (or <thinking>) is stripped before scoring. This means a model is judged on its finished output — a false start or placeholder draft left inside its reasoning can't count against it, and a good final answer isn't diluted by messy scratch work. The same stripping is applied when we render a coding scenario's game in the browser, so the preview matches what was scored. One exception: if removing the reasoning would leave nothing, for example when a run ended mid-thought, the response is scored as it is.
Here's what happens when you run a benchmark:
After one short warm-up request, the client runs the four scenarios one after another. It records the full response, token counts, generation and prompt-processing speed, time to first token, and whatever memory and GPU-offload figures the inference tool provides (see Performance Metrics for which tools provide what). If any scenario fails or runs out of time, the whole run is abandoned and nothing is submitted.
Full benchmark data is sent to /api/benchmarks endpoint. Stored in database. Quality assessment jobs created (status: pending).
A background processor picks up pending assessments every two minutes and sends each scenario's response, with that scenario's assessment prompt, to the evaluator model.
Scores and KPI stored in database with status: completed or failed. Average KPI calculated across all 4 scenarios.
Results appear on the /benchmarks page. Each run also has its own page with its quality scores, the evaluator's summaries and the inference configuration it ran with.
Common questions about our benchmarking process.
Every model gets the exact same four prompts and the same 30-minute budget per scenario, after one warm-up request. Sampling is left at the model's own defaults and no seed is set, so outputs differ from run to run. Each scenario runs once per benchmark. To check a number that looks off, run the benchmark again on the same setup.
No. The evaluator scores only the final answer a model delivers — chain-of-thought wrapped in <think> tags is stripped before scoring (see Quality Assessment above). Reasoning models aren't rewarded or penalized for how much scratch work they show, only for what they produce.
Each scenario has a 30-minute budget so a single slow, thinking-heavy model/hardware combination can't stall a whole run indefinitely. It's generous enough that legitimate long generations finish. If a scenario doesn't finish in time, the whole run is abandoned and nothing is submitted. There is no partial or did-not-finish result: a model and hardware combination that can't finish a scenario simply doesn't appear.
Partly. Every submission comes from a registered device, and the benchmark client signs it with that device's key. The server rejects submissions with missing fields or non-numeric metrics and limits how often one address can submit. There is no automatic plausibility check on the numbers yet. Runs that are clearly wrong are flagged by hand and left out of model pages, hardware pages, comparisons and site statistics.