@remotion/whisper-webgpuv4.0.518

Transcribe audio locally in the browser using timestamped Whisper models and WebGPU through Transformers.js, and convert the result to @remotion/captions.

Installation

Remotion CLI

npx remotion add @remotion/whisper-webgpu

npm

npm i --save-exact @remotion/[email protected]

This assumes you are currently using v4.0.518 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

bun

pnpm i @remotion/[email protected]

This assumes you are currently using v4.0.518 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

pnpm

bun i @remotion/[email protected]

This assumes you are currently using v4.0.518 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

yarn

yarn --exact add @remotion/[email protected]

This assumes you are currently using v4.0.518 of Remotion.
Also update remotion and all `@remotion/*` packages to the same version.
Remove all ^ character in front of the version numbers of it as it can lead to a version conflict.

Example

transcribe.ts
import {clearStaleModels, resampleTo16Khz, toCaptions, transcribe} from '@remotion/whisper-webgpu';

await clearStaleModels();

const file = new File([], 'audio.wav');
const channelWaveform = await resampleTo16Khz({file});

const transcription = await transcribe({
  channelWaveform,
  model: 'small.en',
  language: 'en',
});

const {captions} = toCaptions({whisperWebGpuOutput: transcription});
console.log(captions);

The model is downloaded on the first call and cached by the browser. Call loadWhisperModel() first to display model download progress.

APIs

canUseWhisperWebGpu()

Check whether WebGPU is available

getAvailableModels()

List timestamped models and their download sizes

clearStaleModels()

Remove models discontinued by newer versions

isWhisperModelCached()

Check whether a model is downloaded

loadWhisperModel()

Download and initialize a model

transcribe()

Transcribe a waveform with word-level timestamps

toCaptions()

Convert a transcription to Remotion captions

resampleTo16Khz()

Decode and resample browser audio

disposeWhisperModel()

Release model memory

Requirements

WebGPU requires HTTPS in production or localhost during development. Use canUseWhisperWebGpu() to check for a usable adapter.

License

MIT