Tokenizer Benchmark

Prompt

Implement a complete, correct byte-level BPE tokenizer in Python 3 using only the standard library. Requirements: - Initialize the vocabulary with all 256 possible byte values. - Train BPE merges from a corpus, with configurable vocabulary size and minimum pair frequency. - Use deterministic tie-breaking and never merge across separate samples. - Implement `train(corpus, vocab_size, min_frequency=2)`, `encode(text)`, `decode(ids)`, `save(path)`, `load(path)`, and a `vocab_size` property. - Preserve exact round-trip reconstruction for arbitrary valid Unicode text, including whitespace, newlines, emoji, and unseen text. - Serialize all vocabulary and merge data to valid JSON; loading must reproduce identical token IDs. - Handle empty corpora, empty strings, invalid IDs, malformed files, and vocabulary sizes below 256 safely. - Include a runnable `unittest` test suite, a usage example, and an honest complexity analysis. - Avoid unnecessary inefficiency, but prioritize correctness. Return the complete implementation in one Python code block, followed by the tests and complexity analysis. Do not claim code was executed or tests passed unless they actually were. Evaluation priorities: BPE correctness, Unicode and whitespace preservation, deterministic training, persistence, edge-case handling, test quality, and instruction completeness.