Skip to content

Batcher simplifications #394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

declark1
Copy link
Collaborator

@declark1 declark1 commented May 9, 2025

While working on the chat completion detection streaming implementation, I realized that we can simplify some things related to the batcher. As the ChatCompletionBatcher implementation ended up being simpler than expected, we do not actually need generic batches. It's essentially the same as the MaxProcessedIndexBatcher (a single batcher could actually be used for both), but we can keep separate implementations in case chat completion batching requirements evolve.

Changes related to this:

  • Drop associated type Batch from DetectionBatcher and Batch generics throughout
  • Add Batch = (u32, Chunk, Detections) type alias; we can use this for all batchers
  • Integrate single detection stream optimization into DetectionBatchStream and drop process_detection_stream() functions
    • We only need process_batch_detection_stream() now rather than separate functions
    • We couldn't integrate this previously because DetectionBatchStream returned a generic type, while DetectionStream returned a static type (even if they are the same)

Other changes:

  • Drop detector_id from DetectionStream and DetectionBatcher::push(); it's redundant as detector_id is set in Detection
  • Drop NoopBatcher (not needed for testing)
  • Drop InputId type alias and just use u32

…etector_id from DetectionStream, integrate single detection stream optimization

Signed-off-by: declark1 <44146800+declark1@users.noreply.github.com>
@declark1 declark1 marked this pull request as ready for review May 9, 2025 18:22
Signed-off-by: declark1 <44146800+declark1@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant