- Does not need to rebuild the language model. A speech encoder turns audio into embeddings, an adapter projects them to the LLM’s embedding size, and those vectors replace an audio placeholder in the prompt.
- Useful mental model: reshape speech until the LLM can attend to it like text. The encoder handles perception; the LLM handles the task.
- Temporal subsampling matters because raw audio produces far too many frames for an LLM context window.
Connections
- related_to Large Language Model
- uses Modality Adapter
- uses Speech Encoder