api issue

[geraldstanje1]

hi,

i see the format is different as expected "text_input", [1, 1],: https://github.com/npuichigo/openai_trtllm/blob/main/src/routes/completions.rs#L201

but inflight batcher expects:

name: "text_input"
data_type: TYPE_STRING
dims: [ 1 ]

https://github.com/NVIDIA/TensorRT-LLM/blob/main/triton_backend/all_models/inflight_batcher_llm/ensemble/config.pbtxt#L34

is that a bug or tensorrtllm api changed?

[npuichigo]

Here includes the batch size I think.

[geraldstanje1]

it looks like batch_size = 0 will not work and i need batch_size = 1:

name: "gemma_chat"
backend: "python"
max_batch_size: 1

input [
  {
    name: "text_input"
    data_type: TYPE_STRING
    dims: [1]
  },
  {
    name: "max_tokens"
    data_type: TYPE_INT32
    dims: [1]
  },
 {
   name: "bad_words"
   data_type: TYPE_STRING
   dims: [-1]
   optional: true
  },
  {
   name: "stop_words"
   data_type: TYPE_STRING
   dims: [-1]
   optional: true
  },
  {
    name: "end_id"
    data_type: TYPE_INT32
    dims: [1]
    optional: true
  },
  {
    name: "pad_id"
    data_type: TYPE_INT32
    dims: [1]
    optional: true
  },
  {
    name: "top_k"
    data_type: TYPE_INT32
    dims: [1]
    optional: true
  },
  {
    name: "top_p"
    data_type: TYPE_FP32
    dims: [1]
    optional: true
  },
  {
    name: "temperature"
    data_type: TYPE_FP32
    dims: [1]
    optional: true
  },
  {
    name: "length_penalty"
    data_type: TYPE_FP32
    dims: [1]
    optional: true
  },
  {
    name: "repetition_penalty"
    data_type: TYPE_FP32
    dims: [1]
    optional: true
  },
  {
    name: "min_length"
    data_type: TYPE_INT32
    dims: [1]
    optional: true
  },
  {
    name: "presence_penalty"
    data_type: TYPE_FP32
    dims: [1]
    optional: true
  },
  {
    name: "frequency_penalty"
    data_type: TYPE_FP32
    dims: [1]
    optional: true
  },
  {
    name: "random_seed"
    data_type: TYPE_UINT64
    dims: [1]
    optional: true
  },
  {
    name: "return_log_probs"
    data_type: TYPE_BOOL
    dims: [1]
    optional: true
  },
  {
    name: "return_context_logits"
    data_type: TYPE_BOOL
    dims: [1]
    optional: true
  },
  {
    name: "return_generation_logits"
    data_type: TYPE_BOOL
    dims: [1]
    optional: true
  },
  {
    name: "beam_width"
    data_type: TYPE_INT32
    dims: [1]
    optional: true
  },
  {
    name: "stream"
    data_type: TYPE_BOOL
    dims: [1]
    optional: true
  },
  {
    name: "prompt_embedding_table"
    data_type: TYPE_FP16
    dims: [-1]
    optional: true
  },
  {
    name: "prompt_vocab_size"
    data_type: TYPE_INT32
    dims: [1]
    optional: true
  },
  {
      name: "embedding_bias_words"
      data_type: TYPE_STRING
      dims: [-1]
      optional: true
  },
  {
      name: "embedding_bias_weights"
      data_type: TYPE_FP32
      dims: [-1]
      optional: true
  }
]

output [
  {
    name: "text_output"
    data_type: TYPE_STRING
    dims: [1]
  }
]

instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
]

@npuichigo it works now - who sets the prompt_tokens, completion_tokens, total_tokens? - they are all 0 - see response
request - just saw its not supported yet: https://github.com/npuichigo/openai_trtllm/blob/main/src/routes/chat.rs#L205

curl -X POST "http://localhost:3000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer test" \
  -d '{
    "model": "gemma_chat",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Write me a poem about Machine Learning."
      }
    ],
    "max_tokens": 50
  }'

response:

{
   "id":"cmpl-3da285ac-fe5b-49a2-9d6c-ebeef579deac",
   "object":"text_completion",
   "created":1756753981,
   "model":"gemma_chat",
   "system_fingerprint":null,
   "choices":[
      {
         "index":0,
         "message":{
            "role":"assistant",
            "content":"System: You are a helpful assistant.\nUser: Write me a poem about Machine Learning.\nASSISTANT: Okay, here's a poem about Machine Learning:\n\n**The Algorithmic Bloom**\n\nIn realms of data, vast and deep,\nA silent learning, secrets keep.\nMachine Learning, a curious art,\nTo find the patterns"
         },
         "finish_reason":"stop"
      }
   ],
   "usage":{
      "prompt_tokens":0,
      "completion_tokens":0,
      "total_tokens":0
   }
}

[npuichigo]

Yeah. Triton doesn't expose a way to get these statistics that time so is not supported. Not sure if we can get that now.