Pi0 Simulation and On-Device Execution
This section uses the Pi0 model as an example to introduce the workflow for simulation and on-device execution.
Overview
VLA models are tightly coupled with the robot hardware.
The model takes the robot’s current state—visual input and joint sensor data—as input, and its output directly updates the robot’s state. This forms a continuously interactive, iterative process.
To enable convenient evaluation of Pi0 on the S600 platform, the demo adopts a hybrid workflow that combines on-device inference on S600 with simulation on a PC. The workflow consists of two major components:
Deploying the RoboTwin simulation environment on the PC, which is responsible for generating model inputs, executing the model outputs, and visualizing the Pi0 inference results.
Deploying the Pi0 model on the S600 device, which performs on-device inference and reports performance metrics.
Communication between the two sides is achieved through Ethernet, and the S600 must be able to connect to the PC via the network.
The following diagram illustrates the interaction workflow between the S600 and the PC simulation:
【 S600 】 【 PC 】
+-------------------+ +-------------------+
| OELLM Deploymen | | x86 Simulation |
+---------+---------+ +---------+---------+
| |
v v
+---------+---------+ +---------+---------+
| Run Model | | Run Simulation |
+---------+---------+ +---------+---------+
| |
v v
+---------+---------+ +---------+---------+
| Wait for Stat |<-------------------+ Send State |
+---------+---------+ +---------+---------+
| ^
v |
+---------+---------+ +---------+---------+
| Model Inference | | Simulation Step |
+---------+---------+ +-----------+-------+
| ^
v |
+---------+---------+ +---------+---------+
| Send Action +------------------->| Receive Action |
+-------------------+ +-------------------+
PC Simulation Deployment
Environment Setup
- Clone the repository:
git clone https://github.com/D-Robotics/RoboTwin.git
cd RoboTwin/
git checkout pi0pub
- Set up the virtual environment:
# Under the RoboTwin directory
conda create -n RoboTwin python=3.10 -y
conda activate RoboTwin
pip install uv
# Under RoboTwin/policy/pi0
cd policy/pi0
GIT_LFS_SKIP_SMUDGE=1 uv sync
cp -r ./src/openpi/models_pytorch/transformers_replace/* .venv/lib/python3.11/site-packages/transformers/
- Install curobo:
# Under RoboTwin/policy/pi0
conda deactivate
source .venv/bin/activate
# At this point, you should be in the (openpi) environment
cd ../../envs
git clone https://github.com/NVlabs/curobo.git
cd curobo
export CUROBO_DISABLE_CUDA_EXT=0
pip install -e . --no-build-isolation # This step takes a long time
- Download assets:
# Go to the Robotwin/assets directory
# With a download speed of 2 MB/s, this step takes more than 2 hours
bash download.sh
Common Issues
- Curobo installation failure
Try:
export CUROBO_DISABLE_CUDA_EXT=1
pip install -e . --no-build-isolation
- CUDA version incompatibility
If you are using CUDA 13.0 or above, you need to update the PyTorch version after running uv sync.
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu130
Modify Configuration
Update the RoboTwin/task_config/config.yaml file as follows:
# Testing
sample: 0 # Sample index
rnd: false # Whether to enable random sampling; false means using the sample specified by 'sample'
restore: false # Whether to enable resume-from-checkpoint
# Deployment
stage: 3 # Evaluation stage: OBS=0, SKIP, ACTION, FULL
use_cpp: true # Use on-device inference only; simulation does not run inference (optional when stage is FULL)
torch_model: "policy/pi0/torch_model/pi0_aloha_pytorch_hammer_vispruner_1126"
# Path to the PyTorch model
port: 8888 # Port to listen for socket communication
# Filtering
filter: 0 # Filter type: 0=disabled; 1=ButterWorth; 2=FIR; 3=ZeroPhase
cutoff: 3 # Cutoff frequency
fs: 50 # Sampling frequency
channels: 14 # Number of filter channels
# Inference
chunk: 50 # Chunk size per iteration (max=50)
visp: true # Enable vispruner vocabulary compression
do_preproc: false # Send preprocessed OBS
do_postproc: false # Perform post-processing on received actions
debug: false
# Rendering / Display
use_video: true # Whether to save simulation video
spcam: false # Whether to use a custom camera (resolution is set in _camera_config.yaml; angle in eval_policy.py)
raw_tri: false
cat_dim: 1 # Image concatenation dimension: 0 vertical, 1 horizontal
fresh: 0 # Image refresh interval; 0 disables on-screen display
hd: 2 # Image display scaling factor
Key Parameters to Pay Attention To:
- port: Communication port used to connect with the on-device system; must match the on-device configuration.
- use_video: Whether to save simulation video. Enabled by default. Disabling it may slightly improve performance.
- fresh: Whether to display the simulation scene in real time. Enabled by default. Disabling it may slightly improve performance.
- sample and rnd: Control how samples are selected for evaluation.
sample=0, rnd=false: Use the existing dataset sample 0. sample=0, rnd=true: Randomly resample dataset 0 (overwrite if it already exists).
- restore: Whether to resume from the previous interrupted evaluation.
true: Load the last saved evaluation progress and configuration for this task. false: Start a new evaluation cycle.
Simulation Execution
Enter the RoboTwin/policy/pi0 directory and run the following command:
bash eval.sh beat_block_hammer demo_clean pi0_base_aloha_robotwin_full demo_clean 0 0
S600 On-Device Deployment
Environment Setup
Please refer to the On-device Preparation section to complete the on-device environment configuration.
Example Files
In the oellm_runtimedirectory, the Pi0 example uses the following files and folders:
.
├── configs # Tokenizer files
│ └── Pi0_config
├── examples
│ └── pi0_demo
│ ├── proto # Proto message definitions
│ ├── msg.proto
│ ├── build_pi0.sh # Cross-compilation script
│ ├── CMakeLists.txt
│ ├── pi0_demo.cc # Source code of the executable
│ ├── pi0 # Executable file
│ ├── run_pi0.sh # On-device execution script
│ └── pi0_config.json # On-device configuration file
├── include
├── lib
└── model
├── Pi0
│ ├── Pi0_hammer-beat-block_vision_224x224_w8_nash-p_corenum_1.hbm
│ ├── Pi0_hammer-beat-block_language_w8_nash-p_corenum_4.hbm
│ └── Pi0_hammer-beat-block_action_w8_nash-p_corenum_4.hbm
└── resolve_model_nash-p.md # On-device model download instructions
Configuration File Description
The on-device configuration file {model}_config.json contains the following parameters:
| Parameter | Description | Required / Optional |
|---|
siglip_hbm_path | Description: Path to the vision model
Type: string | Required |
paligemma_hbm_path | Description: Path to the language model
Type: string | Required |
action_hbm_path | Description: Path to the action model
Type: string | Required |
siglip_bpu_core | Description: BPU cores used for the vision model. For multiple cores, specify as [0,1,2,3]
Type: [int] | Required |
paligemma_bpu_core | Description: BPU cores used for the language model. For multiple cores, specify as [0,1,2,3]
Type: [int] | Required |
action_bpu_core | Description: BPU cores used for the action model. For multiple cores, specify as [0,1,2,3]
Type: [int] | Required |
tokenizer_dir | Description: Path to the tokenizer configuration directory
Type: string | Required |
norm_stats_path | Description: Path to the norm stats file
Type: string | Required |
server_ip | Description: Server port for communication; must match the config.yaml port on the simulation side
Type: int
Default: 8888 | Optional |
stage | Description: Execution mode. Usually no change is needed; recommended to keep as 6
Type: int | Required |
preproc | Description: Whether to perform preprocessing on-device; recommended true
Type: bool | Required |
postproc | Description: Whether to perform post-processing on-device; recommended true
Type: bool | Required |
fs | Description: Sampling frequency for action filtering. Adjust based on action performance; 0 disables filtering
Type: double | Required |
Example pi0_config.json Configuration:
{
"siglip_hbm_path": "../../model/Pi0/Pi0_hammer-beat-block_vision_224x224_w8_nash-p_corenum_1.hbm",
"paligemma_hbm_path": "../../model/Pi0/Pi0_hammer-beat-block_language_w8_nash-p_corenum_4.hbm",
"action_hbm_path": "../../model/Pi0/Pi0_hammer-beat-block_action_w8_nash-p_corenum_4.hbm",
"siglip_bpu_core": [
0,1,2
],
"paligemma_bpu_core": [
0,1,2,3
],
"action_bpu_core": [
0,1,2,3
],
"tokenizer_dir": "../../configs/Pi0_config/",
"norm_stats_path": "../../configs/Pi0_config/norm_stats.json",
"server_ip": "120.48.157.2",
"server_port": 30001,
"stage": 6,
"preproc": true,
"postproc": true,
"fs": 15
}
On-Device Execution Guide
A sample execution scriptrun_pi0.sh is provided.
Example command to run:
cd /userdata/oellm_runtime/examples/pi0_demo/
bash run_pi0.sh
Contents of the script:
export LD_LIBRARY_PATH=../../lib:$LD_LIBRARY_PATH # Specify dynamic library path
export HB_DNN_USER_DEFINED_L2M_SIZES=6:6:6:6 # Allocate L2M size for the BPU
./pi0 --config pi0_config.json
Executable parameters:
Usage:
./pi0 --config_path <config_path> [options]
Options:
-c, --config_path <config_path> Path to the pi0 config file (required)
-h, --help Show this help message
Examples:
./pi0 --config_path ./pi0_config.json
Execution Results
S600
...
Filter fs: 15
xlm init success
Starting connection: 120.48.157.2:30001
Connection established: 120.48.157.2:30001 ...
Received successfully, length: 691,484 bytes
===== Parsing Header Information =====
Sequence Number: 0
Timestamp : 1770106723.66053628
Reset Simulation: 1
===== Parsing Body Information =====
Received 3 image tensors:
Type=1, Shape=3×240×320
Type=1, Shape=3×240×320
Type=1, Shape=3×240×320
Received 1 language tensor:
Type=2, Shape=-
Received 1 state tensor:
Type=0, Shape=14
===== Starting Pi0 Inference =====
Preprocess time : 1.573 ms
Siglip infer time: 18.468 ms
Paligemma infer time: 38.965 ms
Action infer time : 38.578 ms
Postprocess time : 0.059 ms
Pi0 Total time : 97.888 ms
===========================
Sent successfully, length: 5,651 bytes
...
PC
...
Render Well
============= Config =============
Messy Table: False
Random Background: False
Random Light: False
Random Table Height: 0
Random Head Camera Distance: 0
Head Camera Config: D435, True
Wrist Camera Config: D435, True
Embodiment Config: aloha-agilex
==================================
missing pytorch3d
No model loaded!
Load tokenizer success!
NoFilter
NoFilter
Load model success!
Task Name: beat_block_hammer
Policy Name: pi0
Start new eval!
...
Server started successfully, waiting for client connection... (Port: 30001)
Client connected: IP=221.226.80.67, Port=58799
Sent successfully, length: 691,484 bytes
Received successfully, length: 5,651 bytes
===== Parsing Header Information =====
Sequence Number : 1
Timestamp : 1770106723.508825449
====== Parsing Body Information ======
Received 0 image tensors
Received 1 embedding tensor:
Language 0: Type=0, Shape=1×50×14
Received 0 state tensors
=============================
Step: 1 / 400
...
A full video of the robotic arm grasping process will also be generated in the pi0/RoboTwin/eval_resultdirectory.
Below are some screenshots from the video:
