atproto_mlf/lexer

Grapheme-by-grapheme scanner turning mlf source into a token stream with 1-indexed line/col spans. Triple-slash /// doc comments become significant DocComment tokens; plain // and # comments (and the #! shebang) are discarded. String and integer/float literals, identifiers, backtick-escaped raw identifiers, keywords, and single punctuation each map to one token. The stream always ends with Eof.

pub fn tokenize(
  source: String,
) -> Result(List(#(token.Token, token.Span)), error.ParseError)