[go: up one dir, main page]

Function tiktoken_rs::p50k_base

source · 
pub fn p50k_base() -> Result<CoreBPE>
Expand description

Use for Code models, text-davinci-002, text-davinci-003 Initializes and returns a new instance of the p50k_base tokenizer.

Examples found in repository?
examples/count_tokens.rs (line 6)
5
6
7
8
9
10
11

fn main() -> Result<(), Box<dyn Error>> {
    let bpe = p50k_base().unwrap();
    let tokens = bpe.encode_with_special_tokens("This is a test         with a lot of spaces");
    println!("Token count: {}", tokens.len());

    Ok(())
}