pub struct Lexer<'a> { /* private fields */ }Expand description
Implementations§
source§impl<'a> Lexer<'a>
impl<'a> Lexer<'a>
sourcepub fn new(input: &str) -> Lexer<'_>
pub fn new(input: &str) -> Lexer<'_>
Creates a new lexer which will lex the input source string.
sourcepub fn allow_confusing_unicode(&mut self, allow: bool) -> &mut Self
pub fn allow_confusing_unicode(&mut self, allow: bool) -> &mut Self
Configures whether “confusing” unicode characters are allowed while lexing.
If allowed then no error will happen if these characters are found, but otherwise if disallowed a lex error will be produced when these characters are found. Confusing characters are denied by default.
For now “confusing characters” are primarily related to the “trojan source” problem where it refers to characters which cause humans to read text differently than this lexer, such as characters that alter the left-to-right display of the source code.
sourcepub fn parse(&self, pos: &mut usize) -> Result<Option<Token>, Error>
pub fn parse(&self, pos: &mut usize) -> Result<Option<Token>, Error>
Lexes the next at the byte position pos in the input.
Returns Some if a token is found or None if we’re at EOF.
The pos argument will be updated to point to the next token on a
successful parse.
Errors
Returns an error if the input is malformed.
sourcepub fn iter(
&self,
pos: usize
) -> impl Iterator<Item = Result<Token, Error>> + '_
pub fn iter( &self, pos: usize ) -> impl Iterator<Item = Result<Token, Error>> + '_
Returns an iterator over all tokens in the original source string
starting at the pos specified.
sourcepub fn annotation(&self, pos: usize) -> Option<&'a str>
pub fn annotation(&self, pos: usize) -> Option<&'a str>
Returns whether an annotation is present at pos and the name of the
annotation.