[Speed] Qwen3-1.7B prefill speed is very slow on Android

[Msyu1020]

🏎️ Speed Report

• The model code: Qwen3-1.7B-q4f16_1

• The model configuration (e.g. quantization mode, running data type, etc.): q4f16_1

• Device (e.g. MacBook Pro M2, PC+RTX 3080): Xiaomi 14 Pro(Snapdragon 8 gen3)

• OS (if applicable): Android

• Encode speed (Token/s): 1 tokens/s (110 tokens/s for Llama-3.2-3B-Instruct-q4f16_0)

• Decode speed (Token/s): 12 tokens/s (12 tokens/s for Llama-3.2-3B-Instruct-q4f16_0)

• Memory usage (if applicable):

//
//

Llama-3.2-3B-Instruct-q4f16_0

I'm unsure if there is an issue during the compilation of the Qwen3-1.7B-q4f16_1 model. The Could anyone help me identify if something is wrong?

compile_models_log.txt

[MasterJH5574]

Hi @Msyu1020, could you please also try Qwen3-1.7B-q4f16_0? It's observed to have better performance than q4f16_1.

[Msyu1020]

Hi @Msyu1020, could you please also try Qwen3-1.7B-q4f16_0? It's observed to have better performance than q4f16_1.

Thanks so much, the prefill speed achieves over 100 tokens/s when I change the model to Qwen3-1.7B-q4f16_0. Could you please explain why the two quantisation methods could make such a difference?

[MasterJH5574]

@Msyu1020 The main difference between q4f16_0 and q4f16_1 is the weight layout. One is transposed and the other one is not

mlc-llm/python/mlc_llm/quantization/quantization.py

Lines 69 to 90 in 4084e7f

	"q4f16_0": GroupQuantize(
	name="q4f16_0",
	kind="group-quant",
	group_size=32,
	quantize_dtype="int4",
	storage_dtype="uint32",
	model_dtype="float16",
	linear_weight_layout="KN",
	quantize_embedding=True,
	quantize_final_fc=True,
	),
	"q4f16_1": GroupQuantize(
	name="q4f16_1",
	kind="group-quant",
	group_size=32,
	quantize_dtype="int4",
	storage_dtype="uint32",
	model_dtype="float16",
	linear_weight_layout="NK",
	quantize_embedding=True,
	quantize_final_fc=True,
	),

[Msyu1020]

@MasterJH5574 Oh, thanks! I'm shocked that simply transposing a matrix can cause such a significant drop in speed.

[MasterJH5574]

@Msyu1020 It is also related to the memory layout of the adreno chips on android phones, which prefer the q4f16_0 layout we use.