Benchmark result

Qwen3.8-27B-oQ4-mtp on Apple M5 Max — 39.7 tok/s

Measured with oMLX 0.6.0.dev1 on August 17, 2026.

ShareRedditX

LLM / Model

Model Size
27B
Architecture
qwen
Memory Kind
model
Tool
oMLX v0.6.0.dev1
Scenarios

LLM Quality Assessment

Agent WorkflowopenclawCOMPLETED
Overall Quality Score
89.60
coherence
92
decomposition
90
error_handling
86
tool_selection
88

The plan is logically ordered, meets the exact 4-step constraint, and clearly separates retrieval, fallback, and synthesis. Tool choices are mostly appropriate, though read_file is somewhat speculative because the cache file is not guaranteed. Error handling is solid with a backup web search and fallback metrics strategy, but the plan could better specify how to ensure truly comparable speed and accuracy figures across model categories.

Code Generationcoding_agentCOMPLETED▶ Play Game
Overall Quality Score
76.20
correctness
78
performance
72
code_quality
74
completeness
84

A strong, mostly playable Breakout implementation that covers most core and mobile requirements, but it has some spec compliance concerns, relies on an external font, and its collision logic is not fully robust for all edge cases.

Role Play & NarrativeroleplayCOMPLETED
Overall Quality Score
81.35
dialogue
84
immersion
88
consistency
82
narrative_arc
61

Atmospheric and character-faithful in the opening, with strong tension and believable suspicion, but it remains only the first beat of the required interaction rather than a complete arc.

Research & AnalysisresearchCOMPLETED
Overall Quality Score
88.70
depth
92
clarity
93
insight
88
interpretation
84

Strong, well-structured analysis with accurate calculations, clear task-by-task comparisons, and actionable recommendations. The main weakness is methodological: it treats heterogeneous model families as if parameter count were the only driver, so the scaling conclusions are suggestive rather than causal.

GPU Acceleration

GPU Model
Apple M5 Max
VRAM
64 GB

Performance Metrics

Generation Speed
39.7
tokens/sec
TTFT
1334
milliseconds
Memory Usage
17.83
GB
Memory (Raw)
18260
MB

Context Information

Max context used
18886
Context Window
65,536

Throughput

Total Runtime
15m 54s

CPU & Memory

CPU Model
Apple M5 Max
CPU Threads
18
Physical Cores
18
CPU Frequency
N/A
Total RAM
64 GB

Operating System

OS Name
macos
OS Version
26.5.2
Kernel Version
25.5.0

Submission Details

Submitted:
8/17/2026, 7:34:34 PM
Client:
v0.4.36+97
Benchmark Result ID:
cmsxmuj3h009h01p415bwb7to