Benchmark result

Qwen3.8-27B-oQ8e-mtp on Apple M5 Max — 35.4 tok/s

Measured with oMLX 0.31.3 on August 30, 2026.

ShareRedditX

LLM / Model

Model Size
27B
Architecture
qwen
Memory Kind
model
Tool
oMLX v0.31.3
Scenarios

LLM Quality Assessment

Agent WorkflowopenclawCOMPLETED
1,793 output tokens · 48.6s
Overall Quality Score
58.30
coherence
58
decomposition
71
error_handling
49
tool_selection
54

The plan is structurally complete and follows a logical discovery-to-delivery flow, but it contains major issues: Step 2 relies on an invalid read_file target (a URL treated like a local file), the backup strategy is vague and partly conflicts with tool constraints, and the identification of the unnecessary tool is inconsistent. Task decomposition is decent, but tool choice and error handling are only moderately sound.

Code Generationcoding_agentCOMPLETED▶ Play Game
8,840 output tokens · 208.7s
Overall Quality Score
61.00
correctness
58
performance
55
code_quality
72
completeness
52

Polished and visually rich implementation, but it misses several core spec requirements: correct start state, exact collision behavior, win-state handling, and some required initialization details. Playability is decent, but the solution is not fully compliant with the prompt.

Role Play & NarrativeroleplayCOMPLETED
610 output tokens · 23.4s
Overall Quality Score
84.10
dialogue
82
immersion
91
consistency
88
narrative_arc
63

An evocative and character-faithful opening that nails the tell and atmosphere, but it stops before the required arc develops, leaving the trust-building and decision points unresolved.

Research & AnalysisresearchCOMPLETED
1,738 output tokens · 53.7s
Overall Quality Score
73.25
depth
78
clarity
82
insight
74
interpretation
63

Well-structured and reasonably thorough, with clear recommendations and useful task comparisons. However, the scaling characterization overstates certainty, the extrapolation is weakly supported, and some numerical/causal inferences exceed what the sparse data can justify.

GPU Acceleration

GPU Model
Apple M5 Max
VRAM
64 GB

Performance Metrics

Generation Speed
35.4
tokens/sec
TTFT
1452
milliseconds
Memory Usage
35.15
GB
Memory (Raw)
35992
MB

Context Information

Max context used
9679
Context Window
262,144

Throughput

Total Output Tokens
12,981
Total Runtime
5m 40s

CPU & Memory

CPU Model
Apple M5 Max
CPU Threads
18
Physical Cores
18
CPU Frequency
N/A
Total RAM
64 GB

Operating System

OS Name
macos
OS Version
26.6.2
Kernel Version
25.6.0

Submission Details

Submitted:
8/30/2026, 11:11:17 AM
Client:
v0.4.46+97
Benchmark Result ID:
cmtfpldae002c01nx943ugvcz