lang_sparql::lang::tokenizer

Function tokenize

Source 
pub fn tokenize(st: &str) -> (Vec<Spanned<Token>>, Vec<Simple<char>>)