D-Robotics Official Website
1.0.0
D-Robotics Official Website
1.0.0
English

Whisper On-Device Execution

This section uses the whisper-medium model as an example to introduce the on-device execution flow. This process applies to scenarios where a quantized model is deployed and executed directly on the device. Please first read the On-device Preparation section to complete the necessary setup and understand the correspondence between models and sample files.

Usage Limitations

whisper-mediumonly supports 16 kHz, single-channel audio. The maximum audio duration allowed for a single inference is 30 seconds.

Sample Files Description

Within theoellm_runtime directory, the files and folders used by whisper-mediumare as follows:

. ├── configs │ └── Whisper_Medium_config # Tokenizer files ├── examples │ └── whisper_demo │ ├── essentia # Essentia library header files │ ├── build_whisper.sh # Cross-compilation script │ ├── CMakeLists.txt │ ├── whisper_demo.cc # Source code of the executable │ ├── whisper # Executable file │ ├── 0.wav # Audio test sample │ ├── run_whisper.sh # On-device execution script │ └── whisper_config.json # On-device execution config file ├── include ├── lib └── model ├── whisper_medium │ ├── whisper-medium_audio_encode_duration_30s_sr_16k_w8_nash-p_corenum_4.hbm │ └── whisper-medium_audio_decode_w8_nash-p_corenum_1_1.hbm └── resolve_model_nash-p.md ## Model download instructions for on-device use

Configuration File Description

The on-device configuration file whisper_config.json includes the following parameters:

Parameter NameDescriptionRequired / Optional
model_dirDescription: Directory path where the model files are stored on the device
Type: string		Required
encode_model_fileDescription: Filename of the encoder model
Type: string		Required
decode_model_fileDescription: Filename of the decoder model
Type: string		Required
encode_bpu_coreDescription: BPU cores used by the encoder; specify cores in the format[0,1,2,3]when using multiple
Type: [int]		Required
decode_bpu_coreDescription: BPU cores used by the decoder; specify cores in the format[0,1,2,3]when using multiple
Type: [int]		Required
vocabulary_pathDescription: Path to the tokenizer configuration
Type: string		Required
languageDescription: Target language for transcription
Type: string
Options: zh: Chineseen: English
Default: zh		Optional

Example configuration:

{ "model_dir": "../../model/whisper_medium/", "encode_model_file": "whisper-medium_audio_encode_duration_30s_sr_16k_w8_nash-p_corenum_4.hbm", "decode_model_file": "whisper-medium_audio_decode_w8_nash-p_corenum_1_1.hbm", "encode_bpu_core": [ 0,1,2,3 ], "decode_bpu_core": [ 0 ], "vocabulary_path": "../../configs/Whisper_Medium_config/", "language": "en" }

On-Device Execution Guide

A one-click execution script run_whisper.sh is provided.

cd /userdata/oellm_runtime/examples/whisper_demo/ bash run_whisper.sh

Contents of the script:

export LD_LIBRARY_PATH=../../lib:$LD_LIBRARY_PATH # Specify dynamic libraries export HB_DNN_USER_DEFINED_L2M_SIZES=6:6:6:6 # Allocate L2M size for BPU ./whisper --config_path ./whisper_config.json --audio_path ./0.wav

Executable parameter info:

Usage: ./whisper --config_path <config_path> --audio_path <audio_path> [options] Options: -c, --config_path <config_path> Path to the whisper config file (required) -a, --audio_path <audio_path> Path to the local audio file (required) -h, --help Show this help message Examples: ./whisper --config_path ./whisper_config.json --audio_path ./0.wav

Example Output

Execution result for the test sample0.wav

[Transcription] Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel. [Performance] TTFT: 110.557000 ms TPS: 61.050993 tokens/s decode_tokens: 21

Additional Notes

  1. whisper_demo can directly read local audio files and supports common formats such as mp3,wav and flac. The program uses Essentia's MonoLoader to resample audio to 16 kHz, convert it into a float32 (-1.0 ~ 1.0) 1-D vector, and pass it to the ASR model through the xlm_feed_audio_online interface.
  2. xlm_feed_audio_online supports a maximum of 30 seconds of audio input. Any excess duration is discarded before inference, along with a warning.
  3. If your audio data is in int16_t format, convert it to float32 first—typically by dividing by 32768 and casting the data type.