BPE Tokenizer

Task

Implement bpe_tokenize(text: str, merges: list[tuple[str, str]], vocabulary: dict[str, int], unknown_token_id: int) -> tuple[list[str], list[int]]. Build an end-to-end character-level BPE tokenizer: split text into words, apply deterministic ranked merges within each word, then encode the final symbols with a vocabulary.

Requirements

• Split text with text.split(), so leading, trailing, and repeated Unicode whitespace does not produce tokens.
• Initialize each non-empty word as its Unicode code points followed by the reserved </w> boundary symbol.
• Treat each merge rule's list index as its rank; lower ranks have higher priority.
• When a pair appears more than once in merges, its first occurrence defines the rank and later duplicates are ignored.
• On each round, find the currently adjacent mergeable pair with the lowest rank.
• Merge every non-overlapping occurrence of that selected pair from left to right in the same round.
• Do not reconsider newly merged symbols until the next round, then search all adjacent pairs again.
• Stop a word when none of its adjacent pairs has a merge rank, and never merge across word boundaries.
• Flatten final symbols in word order and map each one with vocabulary.get(token, unknown_token_id).
• Return the token strings and token IDs as a pair of lists without mutating merges or vocabulary.
• Return two empty lists for empty or whitespace-only text.

Example

merges = [
    ("l", "o"),
    ("lo", "w"),
    ("low", "</w>"),
]
vocabulary = {"low</w>": 42}

bpe_tokenize("low", merges, vocabulary, unknown_token_id=0)
# (["low</w>"], [42])