HTML Tokenizer
HTML Tokenizer
The tokenizer is the first stage of HTML parsing: it converts markup text into a stream of standards-defined tokens. It is public API, so tools that care about tokens rather than trees — linters, sanitizer pre-passes, syntax highlighters — can consume the stream directly.
Tokenizing Markup
Tokenizer.tokenize() processes complete input; Tokenizer.tokenize_fragment() applies fragment tokenization rules. Both emit the standard token vocabulary.
Token Classes
Each token is a typed object: StartTagToken and EndTagToken for tags (with attributes), CharacterToken for text, CommentToken, DoctypeToken, and EofToken marking end of input.
Tokenizer States
TokenizerState models the standards state machine — the same states the specification names — and Tokenizer.set_state() lets fragment and context-sensitive callers start in the right state.
Character References
Named and numeric character references resolve through resolve_named_char_ref(), resolve_numeric_char_ref(), and find_named_char_ref_match() — the full named-reference table, not a shortcut subset.
Tips and Best Practices
- Use
tokenize_fragment()with the correct initial state when tokenizing content that would appear inside a specific element - Treat
EofTokenas the definitive end signal rather than exhausting the iterator by exception - For sanitizers, inspect
StartTagTokenattributes at the token level before any tree is built
Common Issues
| Issue | Cause | Fix |
|---|---|---|
| Script/style content tokenizes as markup | Wrong initial state for context | Set the appropriate TokenizerState via set_state() |
| Entities appear unresolved | Consuming raw text without reference resolution | Use the char-ref resolver functions |
| Fragment output differs from full parse | Fragment rules intentionally differ | Use tokenize() for full documents |
FAQ
Can I use the tokenizer without building a tree?
Yes — the token stream is a public, standalone output.
Are tokenizer states the standard ones?
TokenizerState mirrors the specification’s state machine and is settable via set_state().
How are named character references handled?
Through resolve_named_char_ref() and find_named_char_ref_match() against the full named-reference table.
API Reference Summary
| Class/Method | Description |
|---|---|
Tokenizer.tokenize | Tokenize complete input |
Tokenizer.tokenize_fragment | Tokenize with fragment rules |
Tokenizer.set_state | Set the initial tokenizer state |
StartTagToken | Start-tag token with attributes |
TokenizerState | Standards state machine states |
resolve_named_char_ref | Resolve a named character reference |