Optimize multimodal deployments with vLLM

When deploying multimodal models (e.g., models that handle images, video, or audio inputs) with vLLM, there are several parameters that allow you to control how media inputs are processed. These parameters help you optimize memory usage, limit resource consumption, and configure media preprocessing to match your specific use case.

In Dell Enterprise Hub, you will find these parameters set in the deployment snippet for multimodal models. However, you can adjust them to better fit your requirements.

Multimodal Parameters

Video Pruning Rate

Controls token pruning for video inputs via Efficient Video Sampling. This parameter determines approximately what fraction of media tokens from each video should be removed, reducing memory usage and improving throughput when processing video inputs.

• Environment Variable: VIDEO_PRUNING_RATE
• Type: Float
• Range: [0, 1)
• Default: 0 (no pruning)

Example

A value of 0.5 means approximately 50% of video tokens will be pruned. Higher values reduce memory consumption but may affect output quality.

docker run \
    ...    
    -e VIDEO_PRUNING_RATE=0.5
    ...

Limit Media Items Per Prompt

Sets the maximum number of media items allowed per prompt for each modality. This is useful for controlling resource consumption and ensuring predictable memory usage across requests.

• Environment Variable: LIMIT_MM_PER_PROMPT
• Type: JSON dictionary as string where keys are modality names (image, video, audio) and values are the maximum count allowed per prompt.
• Default: 999 for each modality when unspecified

Example

This limits each prompt to at most 1 video, 1 image, and 1 audio input.

docker run \
    ...    
    -e LIMIT_MM_PER_PROMPT='{"video": 1, "image": 1, "audio": 1}'
    ...

Extra Media IO Kwargs

Passes additional arguments for media processing, organized by modality type. This enables fine-grained control on how media inputs are preprocessed before being fed to the model.

• Environment Variable: MEDIA_IO_KWARGS
• Type: JSON dictionary as string where keys are modality names (image, video, audio) and values are dictionaries of modality-specific preprocessing arguments.
• Default: {} (no additional arguments)

Example

This configures the video processor to sample at 2 frames per second, with a maximum of 256 frames.

docker run \
    ... 
-e MEDIA_IO_KWARGS='{"video": {"fps": 2, "num_frames": 256}}'
    ...

Access Local Media

Specifies a directory on the server filesystem from which the API is allowed to read local media files (images, videos, audio). When set, API requests can reference local file paths instead of URLs or base64-encoded data.

• Environment Variable: ALLOWED_LOCAL_MEDIA_PATH
• Type: String (directory path)
• Default: "" (disabled)

This is a security-sensitive parameter. Enabling it allows the API to access files on the server filesystem. Only enable it in trusted environments and restrict the path to the minimum necessary directory.

Example

To use local media files, you need to:

1. Mount a host directory into the container using Docker's -v flag.
2. Set --allowed-local-media-path to the container-side mount path so vLLM is allowed to read from it.

For example, if your media files are stored on the host at /data/media, you can mount them into the container at /media and point vLLM to that path:

docker run \
    -v /data/media:/media \
    -e ALLOWED_LOCAL_MEDIA_PATH='/media' \
    ...

Then, in your API requests, you can reference files using their container path (e.g., /media/video.mp4).

Make sure the mounted directory contains all the media files your prompts will reference. The container can only access files within the mounted path — it cannot reach other directories on the host filesystem.

Example: Deploying nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8

Below is an example of how these parameters come together in a deployment snippet for serving nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8, a multimodal model:

docker run \
    -it \
    --runtime=nvidia \
    --gpus all \
    -p 8000:8000 \
    -v /data/media:/media \
    -e VIDEO_PRUNING_RATE=0.7 \
    -e LIMIT_MM_PER_PROMPT='{"video": 1, "image": 3, "audio": 3}' \
    -e ALLOWED_LOCAL_MEDIA_PATH='/media' \
    -e MEDIA_IO_KWARGS='{"video": {"fps": 4, "num_frames": 256}}' \
    registry.dell.huggingface.co/enterprise-dell-inference-nvidia-nemotron-3-nano-omni-30b-a3b-reasoning-fp8:latest

Tuning Recommendations

• Start with the defaults provided in the deployment snippets for your model and platform. These have been tested and optimized for common use cases.
• Adjust --video-pruning-rate if you experience memory pressure with video inputs. Increasing the pruning rate reduces memory usage at the cost of some quality.
• Use --limit-mm-per-prompt to enforce strict limits on media inputs if your deployment requirements demand it.
• Configure --media-io-kwargs to control each modality in a more fine-grained way. For example for video processing, lowering num_frames or fps reduces compute and memory requirements, while increasing them improves the model's ability to understand temporal details in video.

For the full list of vLLM serve parameters, refer to the official vLLM documentation.