Module tiktoken_rs::tokenizer

Enums

• Tokenizer
  Enum representing the available tokenizers for different OpenAI models.

Functions

• get_tokenizer
  Returns the tokenizer type used by a model.