Function tiktoken_rs::cl100k_base
source · pub fn cl100k_base() -> Result<CoreBPE>Expand description
Use for ChatGPT models, text-embedding-ada-002
Initializes and returns a new instance of the cl100k_base tokenizer.
pub fn cl100k_base() -> Result<CoreBPE>Use for ChatGPT models, text-embedding-ada-002
Initializes and returns a new instance of the cl100k_base tokenizer.