pub struct Token {
pub kind: TokenKind,
pub offset: usize,
pub len: u32,
}Expand description
A single token parsed from a Lexer.
Fields§
§kind: TokenKindThe kind of token this represents, such as whether it’s whitespace, a keyword, etc.
offset: usizeThe byte offset within the original source for where this token came from.
len: u32The byte length of this token as it resides in the original source.
Implementations§
source§impl Token
impl Token
sourcepub fn id<'a>(&self, s: &'a str) -> &'a str
pub fn id<'a>(&self, s: &'a str) -> &'a str
Returns the identifier, without the leading $ symbol, that this token
represents.
Should only be used with TokenKind::Id.
sourcepub fn keyword<'a>(&self, s: &'a str) -> &'a str
pub fn keyword<'a>(&self, s: &'a str) -> &'a str
Returns the keyword this token represents.
Should only be used with TokenKind::Keyword.
sourcepub fn reserved<'a>(&self, s: &'a str) -> &'a str
pub fn reserved<'a>(&self, s: &'a str) -> &'a str
Returns the reserved string this token represents.
Should only be used with TokenKind::Reserved.
sourcepub fn string<'a>(&self, s: &'a str) -> Cow<'a, [u8]>
pub fn string<'a>(&self, s: &'a str) -> Cow<'a, [u8]>
Returns the parsed string that this token represents.
This returns either a raw byte slice into the source if that’s possible or an owned representation to handle escaped characters and such.
Should only be used with TokenKind::String.
sourcepub fn float<'a>(&self, s: &'a str, kind: FloatKind) -> Float<'a>
pub fn float<'a>(&self, s: &'a str, kind: FloatKind) -> Float<'a>
Returns the decomposed float token that this represents.
This will slice up the float token into its component parts and return a description of the float token in the source.
Should only be used with TokenKind::Float.
sourcepub fn integer<'a>(&self, s: &'a str, kind: IntegerKind) -> Integer<'a>
pub fn integer<'a>(&self, s: &'a str, kind: IntegerKind) -> Integer<'a>
Returns the decomposed integer token that this represents.
This will slice up the integer token into its component parts and return a description of the integer token in the source.
Should only be used with TokenKind::Integer.