Goodput Scenarios

When working with Large Language Models applications, the most commonly used metric to evaluate the performance of a system and minimize costs is usually throughput (measured as requests/tokens served per second). However, real downstream applications may come with more complex requirements, and depending on the use case we may want to optimize for a variety of service level objectives (SLO) such as context or concurrency requirements.

Based on this, we define Goodput as the number of served requests/s (throughput) that a system can reach while meeting the required SLOs for a specific use-case or scenario.

Scenarios definition

To give users a starting point for their deployments, Dell Enterprise Hub offers optimized configurations for three different Goodput scenarios:

• Balanced: This scenario is ideal for applications that require a balance between context length and concurrency, while keeping compute resources at an intermediate level. It is suitable for a wide range of applications and offers a good starting point for further optimization.
• High concurrency: This scenario is designed for applications with a high number of concurrent requests, but with a lower context length requirement. It optimizes for served requests per second in exchange for a high resources utilization.
• Long context: This scenario is tailored for applications that require a long context length, but with a lower concurrency requirement. It optimizes for context length in exchange for a lower throughput and a higher resource utilization.
• Performance: This scenario is a single optimized configuration tailored to Dell AI PC platforms. It represents common use cases for these platforms, supporting a long context window for a small number of concurrent users.

Platform-specific SLOs definition

The SLOs for each scenario are defined based on the computing capabilities of the underlying hardware platform. For example, a r760xa-nvidia-l40s may have different SLOs for the same scenario compared to a xe8640-nvidia-h100.

Selecting the most appropriate scenario for your use case may be difficult, so in this guide you will find the SLOs associated with each scenario for the different platforms available in Dell Enterprise Hub, aggregated by GPU type. This way, you can easily compare the requirements and select the scenario that best fits the needs and constraints of your application.

To give you a better understanding of what represents each SLO, here you can find a brief description:

• Max model context: This SLO defines the maximum context length that the model can handle while meeting the requirements of the scenario. It is measured in number of tokens.
• Virtual users: This SLO defines the number of concurrent users sending requests that the deployment should handle.
• Input tokens: The range of input tokens that are used to simulate requests during the benchmarking runs. For each request, we sample a random number of input tokens within this range.
• Output tokens: The range of output tokens that are used during the benchmarking runs. For each request, we sample a random number of output tokens within this range.

SLOs for NVIDIA L40s

SLOs for NVIDIA H100

SLOs for NVIDIA H200

SLOs for NVIDIA B300

SLOs for NVIDIA GB10

Note that Goodput Scenarios may not be available for some platforms yet. We are continuously working on expanding our optimized deployment configurations to cover more platforms and models. In the meantime, if you have any specific request that is not covered yet, please reach out to us and we will try to work on it.

Optimized deployment configurations

As mentioned, Dell Enterprise Hub offers optimized deployment configurations tailored to Dell platforms and pre-defined goodput scenarios. These configurations are designed to help you achieve the best possible goodput for your use case.

Once the SLOs are defined for a specific scenario and platform combination, we run a set of benchmarks where we test our inference containers with different deployment configurations. Based on these results, we offer the configuration that maximizes throughput while being complient with the defined requirements.

These optimized deployment configurations are available in a specific section of each model view in Dell Enterprise Hub. There you will find the recommended parameters values for each scenario. You can set those parameters in your deployment snippet as environment variables.

For example, if you want to deploy google/gemma-3-1b-it on r760xa-nvidia-l40s platform, you can find the optimized deployment configuration for the Balanced scenario in the model view, and set the parameters in your deployment snippet as follows:

docker run \
    -it \
    -p 80:80 \
    --shm-size 1g \
    --gpus 1 \
    -e NUM_GPUS=1 \
    -e ENABLE_CHUNKED_PREFILL=true \
    -e ENABLE_PREFIX_CACHING=true \
    -e MAX_MODEL_LEN=8192 \
    -e MAX_NUM_SEQS=256 \
    -e MAX_NUM_BATCHED_TOKENS=2048 \
    -e GPU_MEMORY_UTILIZATION=0.97 \
    registry.dell.huggingface.co/enterprise-dell-inference-google-gemma-3-1b-it:vllm-v0.11.2

Benchmarking tools

The scenarios and configurations we provide are not exhaustive, but rather a starting point to guide you in selecting the most appropriate deployment configuration for your use case. We recommend you to customize these configurations based on your unique requirements and constraints.

If you still want to optimize further and run your own benchmarks, you can check the following benchmarking tools to help you assess the performance of your deployment for your specific SLOs:

• inference-benchmarker
• vLLM benchmarks
• NVIDIA AIperf