D-Robotics Official Website
1.0.0
D-Robotics Official Website
1.0.0
English

VLM On-Device Execution

This section uses the Qwen3-VL-4B-Instruct model as an example to illustrate the on-device execution workflow. This workflow applies to quantized models running directly on the device. Please first refer to the On-device Preparation section to complete the environment setup and understand the correspondence between the model and example.

Usage Limitations

The current vlm_demoonly supports single-image input. Multi-image input will be supported in future releases.

Sample Files Description

In the oellm_runtime directory, the VLM example uses the following files and folders:

. ├── configs # Tokenizer files │ ├── InternVL2_2B_config │ ├── Qwen2.5_VL_config │ └── Qwen3_VL_config ├── examples │ └── vlm_demo │ ├── build_vlm.sh # Cross-compilation script │ ├── vlm_demo.cc # Source code of executable │ ├── CMakeLists.txt │ ├── image0.jpg # Test images │ ├── image1.jpg │ ├── run_vlm.sh # On-device execution script │ ├── vlm # Executable │ ├── internvl2_2b_config.json # On-device config files │ ├── qwen2.5vl_3b_config.json │ ├── qwen2.5vl_7b_config.json │ ├── qwen3vl_2b_config.json │ ├── qwen3vl_4b_config.json │ └── qwen3vl_8b_config.json ├── include ├── lib └── model ├── Qwen3-VL-4B-Instruct │ ├── Qwen3-VL-4B-Instruct_vision_448x448_w8-4_nash-p_corenum_4.hbm │ ├── Qwen3-VL-4B-Instruct_language_chunk_512_cache_1024_w4_nash-p_corenum_4_4.hbm │ └── Qwen3-VL-4B-Instruct_embed_tokens_w4_fp16.bin └── resolve_model_nash-p.md # On-device model download instructions

Configuration File Description

The on-device configuration file {model}_config.json contains user-configurable parameters as follows:

ParameterDescriptionRequired / Optional
model_typeDescription: Model type
Type: string
Values: Qwen2.5-VL, Qwen3-VL, InternVL		Required
model_dirDescription: Folder path storing on-device model files
Type: string		Required
vit_model_fileDescription: Vision model filename
Type: string		Required
llm_model_fileDescription: Language model filename
Type: string		Required
embed_weight_file_pathDescription: Embedding weight filename
Type: string		Required
vit_bpu_coreDescription: BPU cores used for vision model. Multiple cores specified as [0,1,2,3]
Type: [int]		Required
prefill_bpu_coreDescription: BPU cores used by the language model during the prefill stage. Specify multiple cores as [0,1,2,3]
Type: [int]		Required
decode_bpu_coreDescription: BPU cores used by the language model during the decode stage. Specify multiple cores as [0,1,2,3]
Type: [int]		Required
vocabulary_pathDescription: Tokenizer configuration directory path
Type: string		Required
warning

Do not modify other configuration parameters, as it may cause inference errors.

Example qwen3vl_4b_config.json:

{ "model_type": "Qwen3-VL", "model_dir": "../../model/Qwen3-VL-4B-Instruct/", "vit_model_file": "Qwen3-VL-4B-Instruct_vision_448x448_w8-4_nash-p_corenum_4.hbm", "llm_model_file": "Qwen3-VL-4B-Instruct_language_chunk_512_cache_1024_w4_nash-p_corenum_4_4.hbm", "embed_weight_file_path": "Qwen3-VL-4B-Instruct_embed_tokens_w4_fp16.bin", "vit_bpu_core": [ 0,1,2,3 ], "prefill_bpu_core": [ 0,1,2,3 ], "decode_bpu_core": [ 0,1,2,3 ], "vocabulary_path" : "../../configs/Qwen3_VL_4B_config", "text_end_token" : "<|endoftext|>", "img_start_token": "<|vision_start|>", "img_end_token": "<|vision_end|>", "img_context_token": "<|image_pad|>", "mask_pad_value": -32768, "temporal_patch_size": 2, "patch_size": 16, "vocab_size": 151936, "embed_dim": 2560, "image_height": 448, "image_width": 448, "image_net_mean": [ 0.5, 0.5, 0.5 ], "image_net_std": [ 0.5, 0.5, 0.5 ] }

On-Device Execution Guide

A one-click script run_vlm.shis provided. Example commands:

cd /userdata/oellm_runtime/examples/vlm_demo/ # Initialize model only bash run_vlm.sh qwen3vl_4b_config.json # Initialize model and load a single image bash run_vlm.sh qwen3vl_4b_config.json image0.jpg

Script contents:

#!/bin/sh export LD_LIBRARY_PATH=../../lib:$LD_LIBRARY_PATH # Specify dynamic library export HB_DNN_USER_DEFINED_L2M_SIZES=6:6:6:6 # Allocate L2M size for BPU config_file=$1 image_file=$2 if [ "$#" -ge 2 ]; then image_file=$2 ./vlm -c $config_file -i $image_file else ./vlm -c $config_file fi

Executable parameters:

Usage: ./vlm --config_path <config_path> [options] Options: -c, --config_path <config_path> Path to the vlm config file (required) -i, --image_path <image_path> Path to the local image file -h, --help Show this help message Examples: ./vlm --config_path ./qwen3vl_4b_config.json

Conversation Control Commands

CommandDescription
<prompt>Input text prompt
/image <image_path>Load local image, replacing previously loaded image
regenRegenerate the current response
resetClear memory and previously loaded image information
exitExit the program

Multi-Modal Input

Image Input

Method 1. Load image during model initialization:

bash run_vlm.sh qwen3vl_4b_config.json image0.jpg

Method 2. Load image during interactive session:

[User] <<< /image image1.jpg

Text Input

Enter text directly after [User] <<< prompt, for example:

[User] <<< Briefly describe this image

Example Output

xlm init success Hello, this is the on-device multi-modal LLM demo - Input text: <prompt> - Load image: /image <image_path> - Regenerate: regen - Clear memory: reset - Exit: exit [User] <<< /image image1.jpg [Assistant] >>> <<< ===== performance ===== ===== vit cost: 34.988000 ms, vit infer cost: 34.988000 ms ===== [User] <<< Briefly describe this image [Assistant] >>> This image shows a Red Panda climbing or resting on a wooden structure. It has characteristic reddish-brown fur, white facial markings, and black eyes and nose. The Red Panda is looking up at the camera with an alert or curious expression. The background consists of natural trees and branches, suggesting it may be in its natural habitat or an outdoor zoo environment. Its posture and expression indicate observation and adaptation to its surroundings. <<< ===== prefill token num: 512 prefill cost: 153.039000 ms, prefill speed: 3345.552441 tokens/s ===== ===== decode token num: 94 cost per token: 19.266138 ms, decode speed: 51.904538 tokens/s ===== ===== end_to_end cost: 1965.197000 ms =====