[MODULE] - Tiktoken Length Classifier

[jhoetter]

Please describe the module you would like to add to bricks
See here: https://platform.openai.com/tokenizer

Do you already have an implementation?
Tiktoken has a basic implementation in their docs, e.g. for gpt-3.5.-turbo. We just need to calculate the number of tokens of a text input.

Additional context
Also use this to calculate the complexity of a paragraph for RAG; for instance, > 1000 tokens means a high complexity.

[jhoetter]

Those are two bricks, if I think about it. one to just calculate the number of tokens (i.e. returning an integer), and one brick that categorizes the text into discrete classes like "short", "medium", "long" etc.

[LeonardPuettmannKern]

@jhoetter I will rename this issue to "Tiktoken Length Classifier" and open a separate issue for a "Tiktoken Token Counter" generator brick!

[LeonardPuettmannKern]

See here: #359

[LeonardPuettmannKern]

Implemented with #360