7.5.7.5
Configuring TIS and MIS in TRL
Learn the TRL configuration options for enabling vLLM importance-sampling correction and selecting TIS or MIS.
What is it?
TRL provides configuration options for enabling vLLM importance-sampling correction and selecting TIS or MIS. The source demonstrates sequence-level truncation for TIS and sequence-level masking for MIS.
Key points
Enable use_vllm.|Enable vllm_importance_sampling_correction.|Use sequence_truncate for TIS.|Use sequence_mask for MIS.|The source examples use caps of 5.0 for TIS and 3.0 for MIS.
Configuring TIS and MIS in TRL
from trl import GRPOConfig
config_tis = GRPOConfig(
use_vllm=True,
vllm_importance_sampling_correction=True,
vllm_importance_sampling_mode="sequence_truncate",
vllm_importance_sampling_cap=5.0,
)
config_mis = GRPOConfig(
use_vllm=True,
vllm_importance_sampling_correction=True,
vllm_importance_sampling_mode="sequence_mask",
vllm_importance_sampling_cap=3.0,
)
Real-world application
Apply the relevant configuration when using vLLM with a GRPO training setup in a compatible TRL version.
Key takeaway
TRL exposes separate modes for truncated and masked importance-sampling correction.