atproto_mlf/token

The token vocabulary produced by the lexer, plus a 1-indexed source Span. mlf keywords, literals, punctuation, and the significant triple-slash doc comment each get their own variant. Plain // and # comments are discarded by the lexer and never appear here.

Types

Span

A source position: 1-indexed line and column, pointing at the first grapheme of the token it is attached to.

pub type Span {
  Span(line: Int, col: Int)
}

Constructors

```
Span(line: Int, col: Int)
```

Token

pub type Token {
  KwAs
  KwBlob
  KwBoolean
  KwBytes
  KwConstrained
  KwDef
  KwError
  KwInline
  KwInteger
  KwNull
  KwProcedure
  KwQuery
  KwRecord
  KwSelf
  KwString
  KwSubscription
  KwToken
  KwType
  KwUnknown
  KwUse
  KwTrue
  KwFalse
  Ident(name: String)
  StringLit(value: String)
  IntLit(value: Int)
  FloatLit(value: Float)
  DocComment(text: String)
  Colon
  Comma
  Dot
  Pipe
  Bang
  Star
  At
  Equals
  Semicolon
  LBrace
  RBrace
  LBracket
  RBracket
  LParen
  RParen
  Eof
}

Constructors

```
KwAs
```
```
KwBlob
```
```
KwBoolean
```
```
KwBytes
```
```
KwConstrained
```
```
KwDef
```
```
KwError
```
```
KwInline
```
```
KwInteger
```
```
KwNull
```
```
KwProcedure
```
```
KwQuery
```
```
KwRecord
```
```
KwSelf
```
```
KwString
```
```
KwSubscription
```
```
KwToken
```
```
KwType
```
```
KwUnknown
```
```
KwUse
```
```
KwTrue
```
```
KwFalse
```
```
Ident(name: String)
```
```
StringLit(value: String)
```
```
IntLit(value: Int)
```
```
FloatLit(value: Float)
```
```
DocComment(text: String)
```
```
Colon
```
```
Comma
```
```
Dot
```
```
Pipe
```
```
Bang
```
```
Star
```
```
At
```
```
Equals
```
```
Semicolon
```
```
LBrace
```
```
RBrace
```
```
LBracket
```
```
RBracket
```
```
LParen
```
```
RParen
```
```
Eof
```

Values

code_point

pub fn code_point(g: String) -> Int

Grapheme classifiers for the mlf identifier grammar, shared by the lexer (scanning source) and the printer (deciding whether a name needs backtick-quoting), so the two stay in sync by construction.

is_digit

pub fn is_digit(g: String) -> Bool

is_ident_continue

pub fn is_ident_continue(g: String) -> Bool

is_ident_start

pub fn is_ident_start(g: String) -> Bool

to_string

pub fn to_string(token: Token) -> String

A human-readable rendering of a token, used in parse-error messages (the “found X” half). Mirrors the surface syntax the token came from.