Article

Meeting transcription without the cloud: Whisper and speaker separation

Avenex editorial team · 2026-10-13

Meeting recordings often cannot be sent to an external service: they contain agreements, figures and names. Below is how we built a transcription setup that runs entirely on a work computer, and what we get out of it. This describes our own working tool, not a packaged product.

What the pipeline consists of

First, ffmpeg converts the recording to 16 kHz WAV, the format speech recognition models expect. Whisper large-v3 then recognises the text. On Macs with Apple Silicon we use the mlx-whisper build, which runs on the GPU cores through Metal. The model detects the language itself, but you can also set it explicitly.

Speaker separation (diarization) is a separate step, handled by pyannote speaker-diarization-3.1. Whisper answers “what was said”, pyannote answers “who said it”. The results of both steps are merged into one text with speaker labels and timecodes.

Working without internet access

The pyannote models are hosted on Hugging Face; the first download requires accepting their terms of use and providing a token. After that, the models are stored locally. Our tool has an offline mode: if even one model file is missing from local storage, transcription stops with an error instead of quietly trying to download it.

For a laptop without a dedicated GPU there is another option. In our dictation assistant, recognition runs either through Whisper with GPU acceleration or through the Parakeet model, optimised for an ordinary CPU. Before recognition, a voice activity detector (VAD) discards silence: less wasted work and fewer “phantom” phrases in empty stretches.

Recognising familiar voices

On its own, pyannote does not know names: it outputs labels such as SPEAKER_00, SPEAKER_01 and so on. Labelling speakers by hand every time is tedious, so we keep a gallery of voiceprints. For each speaker, pyannote returns an averaged voice vector, which is compared with the gallery using cosine similarity.

A name is assigned only if the best candidate scores at least 0.5 and leads the runner-up by a clear margin of at least 0.1. Otherwise the label stays SPEAKER_XX: an honest label is better than confidently naming the wrong person. A person’s voice is added to the gallery once, from a recording that has already been labelled. If the number of participants is known, it can be passed to the diarization as a hint.

From transcript to decisions and tasks

A raw transcript is not yet a set of minutes. Before drawing conclusions, we mark who joined and left, and separate reliable speech from silence, overlapping voices and recognition repeats. Every conclusion keeps its timecode so it can be checked against the recording. A fragment recognised with low confidence can become a follow-up question, but not the basis for a task assigned to a specific person.

The most common mistake is to record any future-tense phrase as a decision. We distinguish several levels: an idea was mentioned, a proposal without agreement, an accepted next step with an owner and a deadline, a conditional action, and a final decision. For example, “if the budget is approved, we register” is a conditional target, not a decision.

Where to keep the results

Transcripts and voiceprints are sensitive data. In our tool, each project’s data, credentials and outputs are kept separately, and processing runs only with an explicitly specified project profile. The internal portal is built from reviewed records — decisions, tasks, meetings — not from raw transcripts.

If you set up something similar, decide in advance where recordings are stored and for how long, who sees the full transcript and who sees only the resulting tasks, and who checks the minutes before they are sent out.