- Speech tokens are not all trying to preserve the same thing.
- Semantic tokens keep words and meaning (useful for ASR). Acoustic tokens keep timbre, prosody and fine sound detail (useful for generation).
- Speech-to-speech systems usually need a mixed representation. If the tokenizer drops speaker or timing information, the language model cannot recover it later.
Connections
- part_of Speech Language Model