English

LLM On-Device Execution

This section uses the Qwen3-1.7B model as an example to illustrate the on-device execution workflow. This workflow applies to quantized models running directly on the device. Please first refer to the On-device Preparation section to complete the environment setup and understand the correspondence between the model and example.

Sample Files Description

In the oellm_runtime directory, the LLM example uses the following files and folders:

.
├── configs                            # Tokenizer files
│   ├── DeepSeek_R1_Distill_Qwen_1.5B_config
│   └── Qwen3_config
├── examples
│   └── llm_demo
│       ├── build_llm.sh               # Cross-compilation script
│       ├── llm_demo.cc                # Source code of executable
│       ├── CMakeLists.txt
│       ├── llm                        # Executable
│       ├── run_llm.sh                 # On-device execution script
│       ├── qwen3_0.6b_config.json     # On-device config files
│       ├── qwen3_1.7b_config.json
│       ├── qwen3_4b_config.json
│       ├── qwen3_8b_config.json
│       └── deepseek_1.5b_config.json
├── include
├── lib
└── model
    ├── Qwen3_1.7B
    │   └── Qwen3-1.7B_language_chunk_512_cache_4096_w4_nash-p_corenum_4_4.hbm
    └── resolve_model_nash-p.md        # On-device model download instructions

Configuration File Description

The on-device configuration file {model}_config.json contains the following parameters:

Parameter	Description	Required / Optional
`hbm_path`	Description: Path to the model on-device Type: `string`	Required
`bpu_core`	Description: BPU cores used for inference. For multiple cores, specify as `[0,1,2,3]` Type: `[int]`	Required
`tokenizer_dir`	Description: Path to the tokenizer configuration directory Type: `string`	Required
`model_type`	Description: Model type Type: `int` Values: `1`: deepseek, `9`: qwen3	Required
`enable_multi_turn`	Description: Enable multi-turn conversation Type: `bool` Values: `true`, `false` Default: `false`	Optional
`enable_thinking`	参数Description描述: Enable think mode Type: `bool` Values: `true`, `false` Default: `true`	Optional

Example qwen3_1.7b_config.json:

{
  "hbm_path": "../../model/Qwen3_1.7B/Qwen3-1.7B_language_chunk_512_cache_4096_w4_nash-p_corenum_4_4.hbm",
  "bpu_core": [
    0,1,2,3
  ],
  "tokenizer_dir": "../../configs/Qwen3_config/",
  "model_type": 9,
  "enable_multi_turn": false,
  "enable_thinking": true
}

On-Device Execution Guide

A one-click execution script run_llm.shis provided. Example command:

cd /userdata/oellm_runtime/examples/llm_demo/
bash run_llm.sh

Script contents:

export LD_LIBRARY_PATH=../../lib:$LD_LIBRARY_PATH   # Specify dynamic library path
export HB_DNN_USER_DEFINED_L2M_SIZES=6:6:6:6        # Allocate L2M size for BPU

./llm -c qwen3_1.7b_config.json

Executable parameters:

Usage:
  ./llm --config_path <config_path> [options]

Options:
  -c, --config_path <config_path>   Path to the llm config file (required)
  -h, --help                        Show this help message

Examples:
  ./llm --config_path ./qwen3_1.7b_config.json

Conversation Control Commands

Command	Description
`<prompt>`	Input text prompt
`regen`	Regenerate the current response
`reset`	Clear memory, resetting multi-turn history
`exit`	Exit the program

Example Output

With think mode enabled

Hello, this is the on-device LLM demo
- Input text: <prompt>
- Regenerate: regen
- Clear memory: reset
- Exit: exit
[User] <<< Briefly introduce the future development of AI
[Assistant] >>> <think>
Hmm, the user asked for a brief introduction to AI’s future development. First, I need to clarify the user’s needs. They may be interested in recent AI trends or the outlook from the tech industry.

Next, I should consider the user’s background. They could be students, researchers, or general enthusiasts. The explanation should be easy to understand, avoiding overly technical terms.

Key points for future development include deep learning, NLP, robotics, ethics, and quantum computing. Also consider applications in healthcare, transportation, education, and potential challenges like data privacy and algorithmic bias.

Finally, ensure the response is structured clearly, concise, and complete.
</think>

The future of artificial intelligence (AI) will revolve around technological breakthroughs, application expansion, ethics, and global collaboration, mainly in the following aspects:

1. **Technological breakthroughs**
   - **Deep Learning**: Neural networks will become more efficient, combined with quantum computing and increased compute, enhancing image recognition and NLP capabilities.
   - **Multimodal Fusion**: AI will process text, images, and audio simultaneously for more precise cross-domain applications.

2. **Application Expansion**
   - **Healthcare**: AI-assisted diagnosis, drug discovery, and personalized treatment.
   - **Transportation**: Autonomous driving, intelligent traffic management, improving urban traffic flow and safety.
   - **Education**: Personalized learning recommendations, virtual teachers, AI-assisted assessment.

3. **Ethics and Regulation**
   - **Ethical Frameworks**: AI decisions must be fair, transparent, and explainable, avoiding bias and privacy breaches.
   - **Legislation**: Countries will establish AI ethics and legal frameworks, clarifying responsibility and usage standards.

4. **Global Collaboration**
   - **Cross-border Cooperation**: Sharing AI technology and standards to address global challenges.
   - **Technology Accessibility**: Open-source and policy support to make AI benefits widely available.

5. **Challenges and Opportunities**
   - **Technical Challenges**: Data security, model explainability, AI-human ethical boundaries.
   - **Opportunities**: AI will reshape employment, promote sustainable development, and solve global issues.

**Summary**: The future of AI will be smarter, more efficient, and safer, requiring continuous evolution across technology, ethics, and legislation, achieving harmonious human-AI interaction.

Performance prefill: 4266.67 tokens/s    decode: 74.59 tokens/s

Without think mode

Hello, this is the on-device LLM demo
[User] <<< Briefly introduce the future development of AI
[Assistant] >>>

The future development of AI will show the following key trends:

1. **Enhanced Computing Power**: With quantum computing and advanced chips, AI can handle more complex tasks with higher efficiency and accuracy.

2. **Multimodal Learning**: AI can process text, images, and audio simultaneously, achieving comprehensive perception and understanding.

3. **Ethics and Safety**: AI applications will comply with ethical and safety standards, preventing misuse.

4. **Ubiquity and Deep Applications**: AI will be further applied in healthcare, education, transportation, improving social efficiency.

5. **Autonomous Decision-making**: AI will have stronger self-learning and decision-making capabilities for complex scenarios.

6. **Cross-domain Integration**: AI will integrate with other technologies (IoT, blockchain) to form a cohesive ecosystem.

These trends will enable AI to play a greater role across industries while highlighting societal and ethical considerations.

Performance prefill: 4376.07 tokens/s    decode: 73.51 tokens/s

Multi-Turn Conversation

Hello, this is the on-device LLM demo
[User] <<< Briefly introduce AI technology
[Assistant] >>>

AI technology refers to the use of Artificial Intelligence to achieve intelligent processing and decision-making. Core technologies include machine learning, deep learning, NLP, and computer vision. These allow computers to analyze and process data automatically.

Applications include speech recognition, image recognition, autonomous driving, recommendation systems, and intelligent customer service.

Performance prefill: 4376.07 tokens/s    decode: 73.28 tokens/s

[User] <<< How does this technology integrate with robots
[Assistant] >>>

AI integrates with robotics by providing perception, decision-making, and execution capabilities, enabling robots to perform tasks autonomously. Machine learning allows robots to understand instructions and execute actions, improving flexibility and adaptability.

Performance prefill: 5626.37 tokens/s    decode: 73.19 tokens/s

Exiting Multi-Turn

Hello, this is the on-device LLM demo
[User] <<< Briefly introduce basketball
[Assistant] >>>

Basketball is a team sport played on a court, where two teams compete to score points by putting the ball in the opponent's hoop. Rules include shooting, passing, and defense. Players need physical fitness, teamwork, and tactical awareness.

Performance prefill: 4376.07 tokens/s    decode: 72.92 tokens/s

[User] <<< How many players are required for an official match
[Assistant] >>>

An official match typically requires **2 or more players**, depending on the sport’s rules. For example:

- **Table Tennis**: Usually 2 players; some matches may have 4.
- **Tennis**: Usually 2 players; some doubles matches have 4.
- **Soccer**: Usually 11 players; some variations allow 10.

The exact number depends on the sport and competition format. Specific sports can be queried for detailed requirements.

Performance prefill: 5688.89 tokens/s    decode: 73.50 tokens/s

On This Page

LLM Development Guide

VLM Development Guide

VLA Development Guide

ASR Development Guide

Appendix

API Details Introduction

LLM On-Device Execution

Sample Files Description

Configuration File Description

On-Device Execution Guide

Example Output

With think mode enabled

Without think mode

Multi-Turn Conversation

Exiting Multi-Turn

#LLM On-Device Execution

#Sample Files Description

#Configuration File Description

#On-Device Execution Guide

#Example Output

#With think mode enabled

#Without think mode

#Multi-Turn Conversation

#Exiting Multi-Turn

LLM On-Device Execution

Sample Files Description

Configuration File Description

On-Device Execution Guide

Example Output

With think mode enabled

Without think mode

Multi-Turn Conversation

Exiting Multi-Turn