How many can optillm handle? #263
Replies: 2 comments 5 replies
-
|
THe number of requests shouldn't be an issue, I have used it for synthetic data generation with several parallel requests and servers as in - https://x.com/asankhaya/status/1972233930117144943 Are you using all optimizations with the same model and which model is it? |
Beta Was this translation helpful? Give feedback.
-
|
some errors are like: while most is like the one i pasted above |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I am using 4 optimizations (moa, leap, bon, self_concistency) using |.
I tried using 5 requests sometimes i got 500:
optillm-openrouter | 2025-10-27 09:25:50,172 - INFO - Generated 1 initial completions using n parameter. Tokens used: 502
optillm-openrouter | 2025-10-27 09:26:04,917 - INFO - 127.0.0.1 - - [27/Oct/2025 09:26:04] "GET /health HTTP/1.1" 200 -
optillm-openrouter | 2025-10-27 09:26:34,985 - INFO - 127.0.0.1 - - [27/Oct/2025 09:26:34] "GET /health HTTP/1.1" 200 -
optillm-openrouter | 2025-10-27 09:26:39,811 - ERROR - Error processing request: Expecting value: line 327 column 1 (char 1793)
optillm-openrouter | 2025-10-27 09:26:39,811 - INFO - 172.1.7.5 - - [27/Oct/2025 09:26:39] "POST /v1/chat/completions HTTP/1.1" 500 -
optillm-openrouter | 2025-10-27 09:26:40,832 - INFO - Received request to /v1/chat/completions
was there any limit on how many requests the optillm can handle?
Can it run 5-10 requests simultaneously using 3-5 optimizations on each? If so, where do you think the 500 is coming from?
Beta Was this translation helpful? Give feedback.
All reactions