Function tiktoken_rs::r50k_base
source · pub fn r50k_base() -> Result<CoreBPE>Expand description
Use for GPT-3 models like davinci
Initializes and returns a new instance of the r50k_base tokenizer (also known as gpt2)
pub fn r50k_base() -> Result<CoreBPE>Use for GPT-3 models like davinci
Initializes and returns a new instance of the r50k_base tokenizer (also known as gpt2)