for now we haven't tried streaming mode. In the buffered mode, the response from LLM has to be fully received before the response is received. If you can add an issue to track TTFT in streaming mode, that'll be great.
Originally posted by @rootfs in #126 (comment)