v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
- Ollama: 58 events in the last 90 days
- Ollama: 55th Release in the last 90 days
- Previous: 2 days earlier · v0.35.1-rc2
What happened
mlx: match publisher tokenizer semantics Honor pretokenizer stage order, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges. Handle empty added tokens and empty Metaspace input consistently. Add shared Go/Python reference cases using published tokenizers, pulling missing models directly and failing on errors, plus focused regressions for configuration precedence, byte fallback, and…
Summary assembled by rule from the sources below