Speech ML · a companion note

From GMMs to Whisper: working with voice in Python

The same Gaussian Mixture Model from the leak-detection note was, for two decades, the engine of speech recognition itself. This traces the arc from that GMM to modern ASR — turning voice into words, into meaning, and flagging voice that doesn't belong — with the citation embedded in each code block.

The through-line. Every model here reads the same front-end: MFCCs — a handful of numbers per short frame of audio. A GMM, a Whisper transformer, and an Isolation Forest are just different back-ends bolted onto that shared representation. Learn the front-end once and the rest is swappable.

raw audio (.wav)
     │
     ├─ MFCC features ──┬─▶ GMM / Isolation Forest      # anomaly & speaker ID
     │   (librosa)      │
     │                  └─▶ Whisper ─▶ "the cat sat"     # words
     │                                      │
     └──────────────────────────────────────▶ NLP       # meaning: intent, sentiment, entities

Method 00 · the origin

Before neural nets, a speaker was a GMM

The direct ancestor of the leak-detection GMM. Classic speech systems fit one Gaussian Mixture per speaker (or, via GMM-HMM, per phoneme) over MFCC frames, then identified a voice by whichever model gave the highest likelihood. Same sklearn.mixture.GaussianMixture, pointed at audio.

[1] Reynolds & Rose (1995), IEEE Trans. Speech & Audio Processing 3(1):72–83 · doi:10.1109/89.365379

"""
The original speech model: a Gaussian Mixture over MFCC features.
Ref: Reynolds, D. A. & Rose, R. C. (1995). "Robust text-independent
speaker identification using Gaussian mixture speaker models." IEEE
Trans. Speech & Audio Processing, 3(1), 72-83. doi:10.1109/89.365379
"""
import librosa, numpy as np
from sklearn.mixture import GaussianMixture

def mfcc_frames(path, n_mfcc=20):
    y, sr = librosa.load(path, sr=16000)               # load + resample to 16 kHz
    m = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=n_mfcc) # shape (n_mfcc, T)
    return m.T                                          # one row per audio frame

# fit one GMM per enrolled speaker, on that speaker's frames only
enrolled = {name: GaussianMixture(n_components=8, covariance_type="diag",
                                  reg_covar=1e-6, random_state=0).fit(mfcc_frames(f))
            for name, f in {"alice": "alice.wav", "bob": "bob.wav"}.items()}

# identify an unknown clip: highest mean log-likelihood wins
frames = mfcc_frames("unknown.wav")
who = max(enrolled, key=lambda name: enrolled[name].score(frames))
print("most likely speaker:", who)

Method 01 · voice → words

Whisper, the model that replaced the GMM-HMM

Modern ASR collapses the whole GMM-HMM-plus-language-model stack into one network. In practice this is a one-liner — pip install openai-whisper — and it returns text with per-segment timestamps.

[2] Radford et al. (2023), arXiv:2212.04356 · OpenAI Whisper

"""
Voice -> words. One model replaces the entire GMM-HMM pipeline.
Ref: Radford, A. et al. (2023). "Robust Speech Recognition via
Large-Scale Weak Supervision." arXiv:2212.04356 (OpenAI Whisper).
"""
# pip install openai-whisper
import whisper

model  = whisper.load_model("base")     # tiny | base | small | medium | large
result = model.transcribe("voice_memo.wav")

print(result["text"])                    # the full transcript
for seg in result["segments"]:           # ...with timestamps
    print(f'[{seg["start"]:6.1f}s] {seg["text"].strip()}')

# --- same thing inside the Hugging Face ecosystem ---
# from transformers import pipeline
# asr = pipeline("automatic-speech-recognition", model="openai/whisper-base")
# print(asr("voice_memo.wav")["text"])

Method 02 · voice → meaning

Chain the transcript into ordinary NLP

There's no such thing as "NLP on raw audio" — you transcribe first, then run text models. The moment you have a string, sentiment, entities, intent, and summarization are all off-the-shelf.

[3] Baevski et al. (2020), wav2vec 2.0, arXiv:2006.11477 · NeurIPS 2020

"""
Voice -> meaning. Transcribe, then treat it as text.
Chain: Whisper (audio->text) -> sentiment + named entities.
"""
from transformers import pipeline

asr       = pipeline("automatic-speech-recognition", model="openai/whisper-base")
sentiment = pipeline("sentiment-analysis")
entities  = pipeline("ner", grouped_entities=True)

text = asr("voice_memo.wav")["text"]
print("words   :", text)
print("feeling :", sentiment(text)[0])   # {'label': 'POSITIVE', 'score': 0.99}
print("entities:", entities(text))       # people / places / orgs

# Want meaning WITHOUT transcribing? wav2vec 2.0 gives embeddings
# straight from audio -- Baevski et al. (2020), arXiv:2006.11477.

Method 03 · the synthesis

Isolation Forest for voice anomaly detection

This closes the loop: reuse the exact MFCC front-end from Method 00, summarize each clip into one fixed-length vector, and let an Isolation Forest flag clips that don't resemble the rest — an unfamiliar speaker, a spoof, a corrupted file.

[4] Liu, Ting & Zhou (2008), Isolation Forest, IEEE ICDM · doi:10.1109/ICDM.2008.17

"""
Voice anomaly detection. SAME MFCC front-end as the GMM above --
only the back-end model changes.
Ref: Liu, F. T., Ting, K. M. & Zhou, Z.-H. (2008). "Isolation Forest."
IEEE Int. Conf. on Data Mining (ICDM). doi:10.1109/ICDM.2008.17
"""
import librosa, numpy as np
from sklearn.ensemble import IsolationForest

def clip_vector(path, n_mfcc=20):
    y, sr = librosa.load(path, sr=16000)
    m = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=n_mfcc)
    # collapse the whole clip to one vector: per-coeff mean + spread
    return np.concatenate([m.mean(axis=1), m.std(axis=1)])   # -> 2*n_mfcc dims

clips = ["a.wav", "b.wav", "c.wav", "suspect.wav"]
X = np.vstack([clip_vector(f) for f in clips])

det   = IsolationForest(contamination=0.1, random_state=0).fit(X)
flag  = det.predict(X)                # -1 = anomaly, 1 = normal
score = det.decision_function(X)      # higher = more normal
for f, fl, s in zip(clips, flag, score):
    print(f"{f:12s} {s:+.3f}  " + ("ANOMALY" if fl == -1 else "ok"))

# A GMM is a drop-in back-end on the SAME X -- the leak-detection code:
#   from sklearn.mixture import GaussianMixture
#   g  = GaussianMixture(4).fit(X)
#   ll = g.score_samples(X)          # low log-likelihood = anomaly

Reference

The five libraries this actually needs

Job	Library	Why it, not something else
Audio features (MFCC, spectrogram)	librosa	The standard audio front-end; feeds every model here.
Classical models — GMM, Isolation Forest	scikit-learn	Both live here; interchangeable on the same feature matrix.
Anomaly-detector toolbox	pyod	Optional. ~40 detectors, one API — only if comparing.
Speech → text (ASR)	openai-whisper / transformers	Pretrained; no training needed to transcribe.
Training your own nets	torch / tensorflow / jax	General DL engines — not vision-specific. Only when you train.

Which library, concretely. Use sklearn.ensemble.IsolationForest — it's the standard, and it's the same one everything else wraps. Reach for PyOD only if you want to compare many detectors behind one API. You touch PyTorch/TensorFlow/JAX only if you're training your own neural model — never just to run one.

References

Reynolds, D. A., & Rose, R. C. (1995). Robust text-independent speaker identification using Gaussian mixture speaker models. IEEE Trans. Speech and Audio Processing, 3(1), 72–83. doi:10.1109/89.365379
Radford, A., et al. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356 (ICML 2023).
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. NeurIPS 2020; arXiv:2006.11477.
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008). Isolation Forest. IEEE ICDM, 413–422. doi:10.1109/ICDM.2008.17
McFee, B., et al. (2015). librosa: Audio and Music Signal Analysis in Python. Proc. 14th Python in Science Conf. (SciPy). doi:10.25080/Majora-7b98e3ed-003