Voice

Introduction

Voice is optional and runs on the user’s device. Read aloud speaks an assistant message, and dictate types what the user says into the composer. Nothing renders, loads or downloads until you wrap your app in VoiceProvider and the user turns a feature on in settings.

The demo web app wires voice for the browser and for the desktop shell. This page follows it.

Wiring the provider

VoiceProvider (@pilotoo/ui/voice, also re-exported from @pilotoo/ui/chat) takes a backend and the user’s preferences.

import { VoiceProvider } from '@pilotoo/ui/voice';
import { voicePreferencesStore } from '@pilotoo/voice';

<VoiceProvider
  backend={backend}                                  // a VoiceBackend, see below
  preferences={voicePreferencesStore(storage)}       // your KeyValueStorage
  onOpenSettings={() => navigate('/settings')}       // where read aloud and dictation lead while off or missing their model
  playCue={(sound, kind) => previewCue(sound, kind)} // optional: start/stop sounds and their preview
  microphones={microphones}                          // optional: { list(), subscribe(), requestAccess() }
>
  <App />
</VoiceProvider>

The demo web app bundles those inputs in a small helper, createVoiceKit (apps/web/src/features/voice/voice-kit.ts, app code rather than a package export). It builds a lazy backend, the preferences store and the device helpers once per window.

apps/web/src/main.tsx

voice: createVoiceKit({
  source: globalThis,
  storage,
  assets: voiceAssetUrls(new URL(import.meta.env.BASE_URL, document.baseURI).href),
  engines: browserVoiceEngines,
  devices: browserVoiceDevices(),
}),

Choosing a backend

The backend decides where the engines run.

Backend	Voice activity, speaker	Speech-to-text	Text-to-speech
createWebVoiceBackend (@pilotoo/voice/web)	Silero VAD, Web Audio, in the page	Whisper base, in a worker	Supertonic 3, in a worker (int8 on WebAssembly, fp32 on WebGPU)
createNativeVoiceBackend (@pilotoo/voice-native)	The same page pieces, passed as page	Parakeet TDT 0.6B v3, native	Supertonic 3 int8, native

Build the backend lazily so none of it loads until voice is used. Both backends offer openSpeaker (text-to-speech without the microphone) and openDictation (microphone and speech-to-text without the speaker). Models download only when the user asks, from settings.

apps/web/src/features/voice/browser-voice.ts

import { lazyVoiceBackend } from '@pilotoo/voice';

const backend = lazyVoiceBackend(async () => {
  const web = await import('@pilotoo/voice/web');
  const backend = web.createWebVoiceBackend({ assets, device: await web.detectInferenceDevice() });
  await backend.models.loaded; // read the model cache first, so settings never show a downloaded model as missing
  return backend;
});

assets is { vad, onnxWasm, transformersWasm }: three URLs of folders your own app serves (WebVoiceAssets). The demo app builds them with voiceAssetUrls(base) and serves them under <base>voice/<folder>/.

Serving the voice assets and configuring Vite

The web backend needs the vad-web worklet and Silero model, and onnxruntime-web’s WebAssembly files, served from your origin. These Vite settings are what make that work. Each one fails in a way that is hard to trace, so copy them whole.

apps/web/vite.config.ts

import { defaultClientConditions, defineConfig } from 'vite';
import { voiceAssets } from './build/voice-assets.ts';

export default defineConfig({
  plugins: [voiceAssets(fileURLToPath(new URL('./package.json', import.meta.url)))],
  worker: { format: 'es' },
  resolve: {
    conditions: ['onnxruntime-web-use-extern-wasm', ...defaultClientConditions],
  },
});

worker.format must be es

@pilotoo/voice/web starts its model workers as module workers (type: 'module'). Build them as ES modules to match, or the worker fails to start.

onnxruntime-web-use-extern-wasm

onnxruntime-web’s default entries embed new URL('ort-*.wasm', import.meta.url). Without this resolve condition the build copies a second, never-loaded set of its 14 to 27 MB runtimes. With it, voice loads them from /voice/.

The voiceAssets plugin

voiceAssets() (apps/web/build/voice-assets.ts) serves the asset files in development and copies them next to the bundle in a build. Copy it with apps/web/src/features/voice/voice-assets.ts, which lists the files per folder. It is skipped in library mode, because the embed ships without voice.

optimizeDeps.exclude

If your app consumes @pilotoo/voice as an installed package, add it to optimizeDeps.exclude. The workers locate their files with new URL(..., import.meta.url), and a pre-bundled copy breaks that with “The voice worker failed to start”. The CommonJS @ricky0123/vad-web goes in optimizeDeps.include, and resolve.dedupe: ['react', 'react-dom'] keeps a single React.

The optional peers are @ricky0123/vad-web, onnxruntime-web and @huggingface/transformers.

Using voice in a Tauri app

@pilotoo/voice-native is a Tauri 2 plugin (tauri-plugin-pilotoo-voice) plus a TypeScript backend. It sends speech-to-text and text-to-speech to the plugin and keeps voice activity and the speaker in the page.

The desktop shell tells the page to use it by setting window.__PILOTOO_VOICE__ = { engine: 'native' } before any page script runs. The demo app reads it in apps/web/src/features/voice/voice-handoff.ts. Without it the page uses the browser engines, and a malformed hand-off is an error, never a silent fallback.

Model id	Role	Download	Licence
parakeet-tdt-0.6b-v3-int8	speech-to-text	487 MB	CC-BY-4.0 (attribute NVIDIA)
supertonic-3-int8	text-to-speech	129 MB	OpenRAIL-M

Models live in one shared folder, <data dir>/pilotoo/voice-models/<id>/, so a model one Pilotoo app downloaded is ready in the others. Downloads are checked against size and SHA-256 and renamed into place, so a model is either whole or missing. Engines load on first use and unload after five idle minutes. To keep an app’s models apart, register the plugin with init_with_models_dir(dir) instead of init().

1. Add the Cargo dependency in src-tauri/Cargo.toml:

   [dependencies]
   tauri-plugin-pilotoo-voice = { path = "../../../packages/voice-native" }

2. Register the plugin:

   tauri::Builder::default()
       .plugin(tauri_plugin_pilotoo_voice::init())

3. Add "pilotoo-voice:default" to the window capability’s permissions.
4. On macOS, set NSMicrophoneUsageDescription in Info.plist and the entitlement com.apple.security.device.audio-input to true. Without them a hardened-runtime build is denied the microphone.
5. In the CSP, connect-src must allow ipc: and http://ipc.localhost. The page half typically needs 'wasm-unsafe-eval' in script-src and worker-src 'self'.
6. Build the page side:

   const backend = lazyVoiceBackend(async () => {
     const [{ createNativeVoiceBackend }, web] = await Promise.all([import('@pilotoo/voice-native'), import('@pilotoo/voice/web')]);
     const page = web.createWebPageVoice({ assets });
     await page.models.loaded;
     return createNativeVoiceBackend({ page });
   });

The factory is async because it reads the plugin’s model list first. sherpa-onnx links statically and adds about 19 MB to a stripped macOS arm64 release binary. Offline builds set SHERPA_ONNX_ARCHIVE_DIR or SHERPA_ONNX_LIB_DIR (see packages/voice-native/README.md).

Licences

Supertonic 3 is OpenRAIL-M. Commercial use is allowed, and its use restrictions must be carried into your app’s terms.

What the user gets

Feature	Behaviour
Read aloud	A button under each assistant message reads its text (not code or tool calls), one message at a time. While off or missing its model it leads to settings. Turning it off stops a reading.
Dictate	A button in the composer writes what the user says into the input and never sends. Each voice segment is transcribed on the device as it ends. Stopping finishes the segment being transcribed before the microphone is released.

The preferences hold two independent switches, readAloud and dictation, both off until the user turns them on. Each needs only its own model: read aloud the text-to-speech model, dictation the speech-to-text model (VOICE_FEATURE_ROLES and missingModels(backend, language, features) in @pilotoo/voice).

Dictated text goes through the composer’s own draft by appendDictation(draft, text): the trimmed text alone into an empty draft, otherwise after one space (none when the draft already ends in whitespace). An empty transcription changes nothing.

Settings screens

From @pilotoo/ui/settings: VoiceSettingsSection (the two switches, language, voice, speed and preview, and the models with size, licence and download progress) and AudioSettingsSection (microphone priority, cues, echo cancellation, noise suppression, volume). A saved microphone is matched by device id, else by name, since WebKit can change device ids between sessions.

Using the pieces directly

Need	Use
Read any text	SpeechReader from @pilotoo/voice, or ReadAloudButton / useReadAloud inside a VoiceProvider
Dictate elsewhere	Dictation from @pilotoo/voice, or ComposerDictation / useDictation inside a VoiceProvider

Next steps

• Presentation reads slides aloud when voice is ready.
• Theming for the settings screens’ styles.
• Embedding the web chat: the embed ships without voice.