Speech & Voice AI
│
├── Audio Signal Fundamentals
│ ├── Waveform — Raw time-domain representation of sound as amplitude over time
│ ├── Sampling Rate — Number of samples per second (e.g., 16kHz, 44.1kHz)
│ ├── Spectrogram — 2D visualization of frequency content over time via STFT
│ ├── Mel Spectrogram — Spectrogram mapped to Mel scale matching human hearing perception
│ ├── MFCC (Mel-Frequency Cepstral Coefficients) — Compact features capturing vocal tract shape
│ ├── Filter Banks — Triangular filters on Mel scale for frequency decomposition
│ ├── Pitch / F0 — Fundamental frequency of vocal cord vibration
│ ├── Formants — Resonant frequencies that distinguish vowels and consonants
│ └── Prosody — Rhythm, stress, and intonation patterns in speech
│
├── Automatic Speech Recognition (ASR)
│ │ — Converts spoken audio into written text
│ │
│ ├── Traditional / Statistical ASR
│ │ ├── HMM (Hidden Markov Models) — Statistical model for sequential speech states
│ │ ├── GMM-HMM — Gaussian Mixture Models for acoustic modeling within HMM framework
│ │ ├── Acoustic Model — Maps audio features to phonemes or sub-word units
│ │ ├── Pronunciation Model (Lexicon) — Maps phoneme sequences to dictionary words
│ │ ├── Language Model (N-gram) — Predicts word sequences using statistical co-occurrence
│ │ └── WFST Decoding — Weighted Finite State Transducers for efficient search over ASR graph
│ │
│ ├── Hybrid Neural ASR
│ │ ├── DNN-HMM — Deep neural nets replace GMMs but keep HMM alignment framework
│ │ ├── CNN-HMM — Convolutional layers for local pattern extraction in spectrograms
│ │ └── LSTM-HMM — Recurrent layers capture temporal context within HMM pipeline
│ │
│ ├── End-to-End ASR
│ │ ├── CTC (Connectionist Temporal Classification) — Loss function that handles
│ │ │ alignment-free sequence prediction from audio to text
│ │ ├── Attention-based Encoder-Decoder — Seq2seq model (e.g., Listen Attend Spell)
│ │ │ that uses attention to align audio frames with output characters
│ │ ├── RNN-Transducer (RNN-T) — Combines encoder, prediction, and joint networks
│ │ │ for streaming ASR with monotonic alignment
│ │ └── Conformer — Hybrid CNN+Transformer capturing both local and global audio patterns
│ │
│ ├── Self-Supervised ASR Foundation Models
│ │ ├── CPC (Contrastive Predictive Coding) — Predicts future audio frames in latent
│ │ │ space using contrastive loss
│ │ ├── wav2vec 2.0 — Self-supervised Transformer encoder with masked prediction
│ │ │ and quantized targets on raw audio
│ │ ├── HuBERT — Hidden-Unit BERT; uses offline k-means clustering to create
│ │ │ pseudo-labels for masked prediction training
│ │ ├── WavLM — Extends HuBERT with denoising and speaker-overlapped speech
│ │ │ for full-stack speech tasks
│ │ ├── data2vec — Unified self-distillation framework that works across speech,
│ │ │ vision, and text modalities
│ │ └── UniSpeech-SAT — Speaker-aware pre-training extension of HuBERT
│ │
│ └── Notable ASR Models
│ ├── DeepSpeech (Baidu) — Early end-to-end CTC-based model using RNNs
│ ├── Whisper (OpenAI) — Encoder-decoder Transformer trained on 680K hours,
│ │ 99+ languages with zero-shot multilingual capability
│ ├── Deepgram Nova — Commercial ASR with 30% lower WER than predecessors
│ ├── gpt-4o-transcribe — OpenAI LLM-integrated transcription with
│ │ superior accuracy in noisy conditions
│ └── Voxtral (Mistral) — Audio-native LLM outperforming Whisper by up to 50%
│ in multilingual transcription
│
├── Text-to-Speech (TTS)
│ │ — Converts written text into natural-sounding audio
│ │
│ ├── Classical TTS
│ │ ├── Formant Synthesis — Rule-based generation using resonant frequency models
│ │ ├── Concatenative Synthesis — Joins pre-recorded speech segments (diphones/triphones)
│ │ └── Statistical Parametric (HMM-TTS) — HMMs predict pitch, duration, and spectral features
│ │
│ ├── Neural TTS (Spectrogram Generation)
│ │ ├── Tacotron — Seq2seq model mapping characters directly to spectrograms
│ │ ├── Tacotron 2 — RNN + attention generating Mel spectrograms fed to a vocoder
│ │ ├── FastSpeech / FastSpeech 2 — Non-autoregressive; predicts duration explicitly
│ │ │ for parallel, fast synthesis
│ │ ├── 🆕 GlowTTS — Flow-based TTS with monotonic alignment for parallel generation
│ │ └── 🆕 Grad-TTS — Score-based diffusion model for spectrogram generation
│ │
│ ├── Neural Vocoders (Spectrogram → Waveform)
│ │ ├── WaveNet (DeepMind, 2016) — Autoregressive CNN generating one audio sample at a time
│ │ │ with unprecedented realism
│ │ ├── WaveRNN — Lightweight single-layer RNN vocoder for real-time synthesis
│ │ ├── WaveGlow — Flow-based model for parallel, non-autoregressive waveform generation
│ │ ├── HiFi-GAN — GAN-based vocoder achieving high-fidelity audio with fast inference
│ │ └── 🆕 BigVGAN — Large-scale universal vocoder with anti-aliased activations
│ │
│ ├── End-to-End TTS
│ │ ├── VITS — Fully end-to-end: VAE + normalizing flows + adversarial training;
│ │ │ text directly to waveform with stochastic duration
│ │ └── 🆕 NaturalSpeech Series — Microsofts progressively improved end-to-end
│ │ TTS achieving human-level quality
│ │
│ ├── 🆕 Neural Codec TTS (Zero-Shot Voice Cloning)
│ │ ├── VALL-E (Microsoft) — Neural codec language model; clones voice from 3-second
│ │ │ clip using discrete audio tokens
│ │ ├── VALL-E X — Cross-lingual extension for multi-language zero-shot TTS
│ │ ├── VALL-E 2 — Achieves human parity on LibriSpeech with repetition-aware sampling
│ │ ├── Voicebox (Meta) — Non-autoregressive flow-matching model for speech generation,
│ │ │ editing, and noise removal
│ │ ├── XTTS-v2 (Coqui) — Voice cloning with just 6 seconds of audio across 17 languages
│ │ ├── Bark — Transformer-based; can generate speech, music, and sound effects
│ │ ├── Tortoise TTS — Multi-step autoregressive + diffusion for high-quality cloning
│ │ └── 🆕 gpt-4o-mini-tts (OpenAI) — Steerable TTS where developers control
│ │ speaking style (tone, emotion, pace) via text instructions
│ │
│ └── 🆕 Conversational Speech Models
│ ├── Sesame CSM — Multimodal text+speech model using RVQ tokens with
│ │ conversational context for human-like pauses and intonation
│ └── 🆕 VibeVoice (Microsoft) — Open-source audio stack handling TTS, ASR,
│ and voice cloning in one unified model
│
├── 🆕 Speech Tokenizers & Neural Audio Codecs
│ │ — Convert continuous audio into discrete tokens for language model processing
│ │
│ ├── EnCodec (Meta) — RVQ-based neural audio codec compressing speech into
│ │ discrete token sequences at various bitrates
│ ├── SpeechTokenizer — Unified tokenizer disentangling semantic (content) and
│ │ acoustic (timbre) information across RVQ layers
│ ├── Mimi (Kyutai) — Streaming neural codec combining semantic and acoustic
│ │ tokens via knowledge distillation from WavLM
│ ├── 🆕 UniCodec — Ultra-low bitrate codec with global, local semantic,
│ │ and residual token decomposition
│ └── 🆕 DAC (Descript Audio Codec) — High-fidelity general audio codec
│ supporting 44.1kHz audio
│
├── 🆕 Speech Language Models (Speech LLMs)
│ │ — LLMs that directly process and/or generate speech tokens
│ │
│ ├── Architecture Paradigms
│ │ ├── Cascaded (ASR + LLM + TTS) — Traditional pipeline; transcribe → reason → synthesize
│ │ ├── S2T (Speech-to-Text LLM) — Audio encoder feeds into LLM for text output
│ │ ├── ST2T (Speech+Text-to-Text) — Both audio and text prompts → text output
│ │ │ (most common for speech understanding)
│ │ └── ST2ST (Speech+Text-to-Speech+Text) — Full duplex: speech in → speech out
│ │ with text as intermediate reasoning
│ │
│ ├── Key Components
│ │ ├── Speech Encoder — Extracts representations from audio (e.g., Whisper encoder, wav2vec)
│ │ ├── Modality Adapter — Projects speech embeddings into LLM token space
│ │ ├── Cross-Modal Alignment — Training to map audio representations to text space
│ │ └── Audio Decoder / Vocoder — Converts LLM output tokens back to speech waveform
│ │
│ ├── Notable Models
│ │ ├── SpeechGPT — Early model connecting speech tokens to GPT-style LLM
│ │ ├── Spirit-LM (Meta) — Interleaves speech and text tokens during generation
│ │ ├── Moshi (Kyutai) — First real-time full-duplex speech LLM; two audio
│ │ │ streams (user + model) with "Inner Monologue" text prediction;
│ │ │ 7B Temporal Transformer + Depth Transformer; ~160ms latency
│ │ ├── GPT-4o (OpenAI) — End-to-end omni model processing text, audio, image,
│ │ │ and video with ~320ms voice response latency
│ │ ├── Gemini Live (Google) — Real-time multimodal conversation including voice
│ │ └── Voxtral (Mistral) — Audio-native LLM family (3B and 24B) for
│ │ transcription, summarization, and conversational understanding
│ │
│ └── 🆕 Training Techniques for Speech LLMs
│ ├── Cross-Modal Instruction Fine-Tuning — Append instructions to paired ASR data
│ ├── Chain-of-Modality Fine-Tuning — Synthesize speech-in/speech-out datasets via TTS
│ └── Inner Monologue — Predict text tokens before audio tokens to improve
│ linguistic quality (pioneered in Moshi)
│
├── 🆕 Downstream Speech Tasks
│ ├── Speaker Diarization — Identifying "who spoke when" in multi-speaker audio;
│ │ uses embeddings + clustering (e.g., PyAnnote, AssemblyAI)
│ ├── Speaker Verification — Confirming if audio matches a known speakers voiceprint
│ ├── Speaker Identification — Determining which known speaker is talking
│ ├── Speech Emotion Recognition (SER) — Classifying emotions (happy, sad, angry)
│ │ from speech using MFCC features + neural classifiers
│ ├── Voice Activity Detection (VAD) — Detecting speech vs. silence/noise segments;
│ │ uses energy-based, model-based, or hybrid methods
│ ├── Voice Conversion — Transforming one speakers voice to sound like another
│ │ while preserving content (e.g., RVC)
│ ├── Keyword Spotting — Detecting specific wake words or commands in audio stream
│ ├── Speech Enhancement — Removing noise/reverb to improve audio clarity
│ ├── Speech Separation — Isolating individual speakers from mixed audio (cocktail party problem)
│ └── Speech Translation — Translating spoken language directly to another languages text or speech
│
└── 🆕 Evaluation Metrics
├── WER (Word Error Rate) — Standard ASR accuracy metric
├── CER (Character Error Rate) — Character-level ASR accuracy
├── MOS (Mean Opinion Score) — Human subjective rating of speech quality (1-5)
├── PESQ — Perceptual Evaluation of Speech Quality (automated MOS approximation)
├── Speaker Similarity Score — Cosine similarity between speaker embeddings
└── DER (Diarization Error Rate) — Measures speaker diarization accuracy
