Voice in the browser

Introduction

You will add read aloud and dictation to a web chat. The speech models run in the browser, download only when the user turns a feature on, and none of the voice code loads for a user who never does. Most of this recipe is Vite configuration, because each setting fixes a failure that gives no hint of its cause.

What you need

• A chat from Add chat to a React app.
• Optional peers installed: @ricky0123/vad-web, onnxruntime-web and @huggingface/transformers.

Steps

1. Build a lazy backend

lazyVoiceBackend runs your factory the first time voice is needed. assets are the URLs where your app serves the worklet, the model and the WebAssembly files (see step 3).

src/voice/voice-kit.ts

import { lazyVoiceBackend, voicePreferencesStore, type KeyValueStorage } from '@pilotoo/voice';
import type { WebVoiceAssets } from '@pilotoo/voice/web';

export function createVoiceKit(storage: KeyValueStorage, assets: WebVoiceAssets) {
  return {
    backend: lazyVoiceBackend(async () => {
      const web = await import('@pilotoo/voice/web');
      const backend = web.createWebVoiceBackend({ assets, device: await web.detectInferenceDevice() });
      // Read the model cache first, so settings never show a downloaded model as missing.
      await backend.models.loaded;
      return backend;
    }),
    preferences: voicePreferencesStore(storage),
  };
}

2. Wrap the app in VoiceProvider

Both switches (readAloud, dictation) are off until the user turns them on in settings. onOpenSettings is where the buttons lead while a feature is off or missing its model.

src/main.tsx

import { VoiceProvider } from '@pilotoo/ui/voice';

const voice = createVoiceKit(storage, voiceAssetUrls(new URL(import.meta.env.BASE_URL, document.baseURI).href));

<VoiceProvider
  backend={voice.backend}
  preferences={voice.preferences}
  onOpenSettings={() => navigate('/settings')}
>
  <App />
</VoiceProvider>;

Add VoiceSettingsSection and AudioSettingsSection from @pilotoo/ui/settings to your settings page so users can switch features on and download models.

3. Serve the assets

The web backend loads its files from your own origin. Give it one folder per group.

src/voice/voice-assets.ts

import type { WebVoiceAssets } from '@pilotoo/voice/web';

export function voiceAssetUrls(base: string): WebVoiceAssets {
  const folder = (name: keyof WebVoiceAssets) => new URL(`voice/${name}/`, base).href;
  return { vad: folder('vad'), onnxWasm: folder('onnxWasm'), transformersWasm: folder('transformersWasm') };
}

The files to copy into those folders, from node_modules:

Folder	Files	Comes from
voice/vad/	vad.worklet.bundle.min.js, silero_vad_v6.onnx	@ricky0123/vad-web/dist
voice/onnxWasm/	ort-wasm-simd-threaded.mjs, .wasm, .asyncify.mjs, .asyncify.wasm	onnxruntime-web/dist
voice/transformersWasm/	ort-wasm-simd-threaded.asyncify.mjs, .asyncify.wasm	the onnxruntime-web that @huggingface/transformers installs

The demo web app does this with a small Vite plugin (apps/web/build/voice-assets.ts) that serves the files in dev and copies them in a build.

4. Configure Vite

Each line names the symptom it fixes.

vite.config.ts

import { defineConfig, defaultClientConditions } from 'vite';

export default defineConfig({
  // Symptom: "The voice worker failed to start".
  // @pilotoo/voice starts its workers with new Worker(new URL(..., import.meta.url)).
  // A pre-bundled copy moves the file and the URL no longer points at the worker.
  optimizeDeps: {
    exclude: ['@pilotoo/voice'],
    // Symptom: "does not provide an export named" for vad-web in dev. It is CommonJS.
    include: ['@ricky0123/vad-web'],
  },

  // The workers are ES modules (type: 'module'); build them as ES modules to match.
  worker: { format: 'es' },

  resolve: {
    // Symptom: 14-27 MB of onnxruntime wasm copied into the build twice, one set never loaded.
    // The default entries embed new URL('ort-*.wasm', import.meta.url); this condition leaves
    // the files to the folders from step 3.
    conditions: ['onnxruntime-web-use-extern-wasm', ...defaultClientConditions],
    // One copy of React for the kit and your app.
    dedupe: ['react', 'react-dom'],
  },
});

Tip

Some hosts cap the size of each file they serve. Because the wasm is served from separate folders, you can put those folders on a CDN and point assets at it.

How it works

• Read aloud puts a button under each assistant message. Dictation puts a button in the composer; it writes into the input and never sends.
• Speech-to-text is Whisper base in a worker, text-to-speech is Supertonic 3 in a worker, and voice activity is Silero in the page. Models download only when the user asks.
• Because the factory is lazy and every voice module is imported dynamically, the bundle a user without voice downloads contains none of it.

Next steps

• Voice: backends, models, licences and the Tauri native backend.
• Packages: voice: the web and native entry points.
• Desktop app with a local backend: voice in a desktop shell.