Chat template not applied for vllm models

[Tim-Siu]

It seems that if we run run eval_kk.py with argument use_vllm, the script will fedd raw prompt to vllm, without applying chat template like <|im_start|>...

mem-kk-logic/utils.py

Lines 25 to 48 in 1022068

	def batch_decode_vllm(llm, prompts, batch_size=32):
	"""
	Perform batch decoding using vLLM.
	Args:
	- llm: The vLLM model instance
	- prompts: List of prompts to process
	- batch_size: Number of prompts to process in each batch
	Returns:
	- List of generated responses
	"""
	from vllm import SamplingParams # type: ignore
	all_responses = []
	for i in range(0, len(prompts), batch_size):
	batch_prompts = prompts[i : i + batch_size]
	sampling_params = SamplingParams(max_tokens=llm.max_tokens, temperature=0)
	outputs = llm.model.generate(
	batch_prompts, sampling_params
	)
	responses = [output.outputs[0].text for output in outputs]
	all_responses.extend(responses)
	return all_responses