pub struct Token {
pub kind: TokenKind,
pub offset: usize,
pub len: u32,
}Expand description
A single token parsed from a Lexer.
Fields§
§kind: TokenKindThe kind of token this represents, such as whether it’s whitespace, a keyword, etc.
offset: usizeThe byte offset within the original source for where this token came from.
len: u32The byte length of this token as it resides in the original source.
Implementations§
Source§impl Token
impl Token
Sourcepub fn id<'a>(&self, s: &'a str) -> Result<Cow<'a, str>, Error>
pub fn id<'a>(&self, s: &'a str) -> Result<Cow<'a, str>, Error>
Returns the identifier, without the leading $ symbol, that this token
represents.
Note that this method returns the contents of the identifier. With a string-based identifier this means that escapes have been resolved to their string-based equivalent.
Should only be used with TokenKind::Id.
§Errors
Returns an error if this is a string-based identifier (e.g. $"...")
which is invalid utf-8.
Sourcepub fn annotation<'a>(&self, s: &'a str) -> Result<Cow<'a, str>, Error>
pub fn annotation<'a>(&self, s: &'a str) -> Result<Cow<'a, str>, Error>
Returns the annotation, without the leading @ symbol, that this token
represents.
Note that this method returns the contents of the identifier. With a string-based identifier this means that escapes have been resolved to their string-based equivalent.
Should only be used with TokenKind::Annotation.
§Errors
Returns an error if this is a string-based identifier (e.g. $"...")
which is invalid utf-8.
Sourcepub fn keyword<'a>(&self, s: &'a str) -> &'a str
pub fn keyword<'a>(&self, s: &'a str) -> &'a str
Returns the keyword this token represents.
Should only be used with TokenKind::Keyword.
Sourcepub fn reserved<'a>(&self, s: &'a str) -> &'a str
pub fn reserved<'a>(&self, s: &'a str) -> &'a str
Returns the reserved string this token represents.
Should only be used with TokenKind::Reserved.
Sourcepub fn string<'a>(&self, s: &'a str) -> Cow<'a, [u8]>
pub fn string<'a>(&self, s: &'a str) -> Cow<'a, [u8]>
Returns the parsed string that this token represents.
This returns either a raw byte slice into the source if that’s possible or an owned representation to handle escaped characters and such.
Should only be used with TokenKind::String.
Sourcepub fn float<'a>(&self, s: &'a str, kind: FloatKind) -> Float<'a>
pub fn float<'a>(&self, s: &'a str, kind: FloatKind) -> Float<'a>
Returns the decomposed float token that this represents.
This will slice up the float token into its component parts and return a description of the float token in the source.
Should only be used with TokenKind::Float.
Sourcepub fn integer<'a>(&self, s: &'a str, kind: IntegerKind) -> Integer<'a>
pub fn integer<'a>(&self, s: &'a str, kind: IntegerKind) -> Integer<'a>
Returns the decomposed integer token that this represents.
This will slice up the integer token into its component parts and return a description of the integer token in the source.
Should only be used with TokenKind::Integer.