D-Robotics Official Website
1.0.0
D-Robotics Official Website
1.0.0
English

Sampling Configuration File Instruction

When running the model on the board, a sampling configuration file is required to sample the logits output by the LLM model.

This configuration file is located in the deployment package oellm_runtime/configs/{model}_config directory, and is named generation_config.json. It contains the sampling strategy used.

Sampling Configuration File Format

The following parameters can be set in generation_config.json:

generation_config.json
{ "do_sample": true, "temperature": 1.0, "top_k": 50, "top_p": 0.9, "typ_p": 0.9, "min_p": 0.1, "penalty_last_n": 32, "frequency_penalty": 0.3, "presence_penalty": 0.4, "repetition_penalty": 1.05, "min_keep": 1 }

The following is a detailed explanation of each parameter in the sampling configuration file:

Parameter NameParameter TypeParameter Description
do_sampleboolWhether to enable random sampling. If not enabled, argmax will be used. Value range: true, false.
temperaturefloatTemperature coefficient. Value range: [0.0, 2.0], recommended setting range: [0.6, 1.0].
top_kintSelect the k tokens with the highest probability for sampling. Setting it to 1 is equivalent to greedy sampling. Setting range: [1, +∞), recommended setting range: [20, 80].
top_pfloatCumulative probability sampling threshold. Value range: [0.0, 1.0], recommended setting range: [0.85, 0.95].
typ_pfloatInformation theory-based typicality threshold. Value range: [0.0, 1.0], recommended setting range: [0.90, 0.98].
min_pfloatMinimum probability threshold, filters low probability tokens. Value range: [0.0, 1.0], recommended setting range: [0.05, 0.15].
penalty_last_nintPenalty window. Value range: [0, 2048], recommended setting value: 32, 64, 128.
frequency_penaltyfloatFrequency penalty. Value range: [0.0, 2.0], Recommended setting range: [0.1, 0.5].
presence_penaltyfloatPresence penalty. Value range: [0.0, 2.0], Recommended setting range: [0.2, 0.6].
repetition_penaltyfloatRepetition penalty. Value range: [0.0, 2.0], Recommended setting range: [1.05, 1.2].
min_keepintMinimum number of candidate words to retain. Value range: [1, +∞).
Note

  • The execution order of the sampling algorithm is the same as the setting order of the sampling parameters, different orders will affect the sampling results.

  • The parameters in the sampling configuration file can be adjusted according to actual needs to obtain the best generation effect.

Different models may have varying sensitivities to sampling parameters; thorough testing is recommended when adjusting parameters.