[v0.16.2] Qwen3 30B A3B Q40 on 2 x OrangePi Plus 16GB #266

oeschlberger · 2025-10-15T07:44:39Z

oeschlberger
Oct 15, 2025

[v0.16.2] Qwen3 30B A3B Q40 on 2 x OrangePi Plus 16GB

Setup

The OrangePis are connected directly via 2.5Gb LAN Ports.
OrangePi1 ═ 2.5Gb ═ OrangePi2

Device: 2 x OrangePi Plus 16GB (connected directly via 2.5Gb LAN)
Distributed Llama version: 0.16.2
Model: qwen3_30b_a3b_q40

Run Commands

Worker node (OrangePi)

./dllama worker --model dllama_model_qwen3_30b_a3b_q40.m --tokenizer dllama_tokenizer_qwen3_30b_a3b_q40.t --port 9999 --nthreads 4

Inference (Root node)

 ./dllama inference --prompt "<|im_start|>user
Please explain me where is Poland as I have 1 year<|im_end|>
<|im_start|>assistant
" --steps 128 --model models/qwen3_30b_a3b_q40/dllama_model_qwen3_30b_a3b_q40.m --tokenizer models/qwen3_30b_a3b_q40/dllama_tokenizer_qwen3_30b_a3b_q40.t --buffer-float-type q80 --nthreads 4 --max-seq-len 4096 --workers 192.168.2.10:9999

Benchmark Evaluation

Device	Prediction Speed (tok/s)	Benchmark Speed (tok/s)
2 x OrangePi Plus 16GB	7.29	11.55

Output

./dllama inference --prompt "<|im_start|>user
Please explain me where is Poland as I have 1 year<|im_end|>
<|im_start|>assistant
" --steps 128 --model models/qwen3_30b_a3b_q40/dllama_model_qwen3_30b_a3b_q40.m --tokenizer models/qwen3_30b_a3b_q40/dllama_tokenizer_qwen3_30b_a3b_q40.t --buffer-float-type q80 --nthreads 4 --max-seq-len 4096 --workers 192.168.2.10:9999
📄 AddBos: 0
📄 BosId: 151643 (<|endoftext|>)
📄 EosId: 151645 (<|im_end|>)
📄 RegularVocabSize: 151643
📄 SpecialVocabSize: 26
Tokenizer vocab size (151669) does not match the model vocab size (151936)
💡 Arch: Qwen3 MoE
💡 HiddenAct: Silu
💡 Dim: 2048
💡 HeadDim: 128
💡 QDim: 4096
💡 KvDim: 512
💡 HiddenDim: 6144
💡 VocabSize: 151936
💡 nLayers: 48
💡 nHeads: 32
💡 nKvHeads: 4
💡 OrigSeqLen: 262144
💡 nExperts: 128
💡 nActiveExperts: 8
💡 MoeHiddenDim: 768
💡 SeqLen: 4096
💡 NormEpsilon: 0.000001
💡 RopeType: Falcon
💡 RopeTheta: 10000000
📀 RequiredMemory: 9765 MB
⭕ Socket[0]: connecting to 192.168.2.10:9999 worker
⭕ Socket[0]: connected
⭕ Network is initialized
🧠 CPU: neon dotprod fp16
💿 Loading weights...
💿 Loaded 11/48
💿 Loaded 12/48
💿 Loaded 13/48
💿 Loaded 14/48
💿 Loaded 15/48
💿 Loaded 16/48
💿 Loaded 17/48
💿 Loaded 18/48
💿 Loaded 19/48
💿 Loaded 20/48
💿 Loaded 21/48
💿 Loaded 22/48
💿 Loaded 23/48
💿 Loaded 24/48
💿 Loaded 25/48
💿 Loaded 26/48
💿 Loaded 27/48
💿 Loaded 28/48
💿 Loaded 29/48
💿 Loaded 30/48
💿 Loaded 31/48
💿 Loaded 32/48
💿 Loaded 33/48
💿 Loaded 34/48
💿 Loaded 35/48
💿 Loaded 36/48
💿 Loaded 37/48
💿 Loaded 38/48
💿 Loaded 39/48
💿 Loaded 40/48
💿 Loaded 41/48
💿 Loaded 42/48
💿 Loaded 43/48
💿 Loaded 44/48
💿 Loaded 45/48
💿 Loaded 46/48
💿 Loaded 47/48
💿 Loaded 48/48
💿 Weights loaded
🚁 Network is in non-blocking mode
<|im_start|>user
Please explain me where is Poland as I have 1 year<|im_end|>
<|im_start|>assistant

Evaluation
nBatches: 32
nTokens: 19
tokens/s: 11.55 (86.62 ms/tok)
Prediction
nTokens: 109
tokens/s: 7.29 (137.27 ms/tok)
⭕ Network is closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[v0.16.2] Qwen3 30B A3B Q40 on 2 x OrangePi Plus 16GB #266

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

[v0.16.2] Qwen3 30B A3B Q40 on 2 x OrangePi Plus 16GB #266

Uh oh!

oeschlberger Oct 15, 2025

[v0.16.2] Qwen3 30B A3B Q40 on 2 x OrangePi Plus 16GB

Setup

Run Commands

Worker node (OrangePi)

Inference (Root node)

Benchmark Evaluation

Output

Replies: 0 comments

oeschlberger
Oct 15, 2025