Using pre-downloaded model weights

By default, when you run a Dell Enterprise Hub container with a MODEL_ID environment variable set to a Hugging Face model identifier (e.g., meta-llama/Llama-3.1-8B-Instruct), the container will automatically download the model weights from the Hugging Face Hub during startup. While this approach works well for getting started quickly, it has a significant drawback: the model weights are downloaded every time a new container instance is created. For large language models that can be tens or hundreds of gigabytes in size, this results in substantial network bandwidth usage, longer startup times, and potential rate limiting issues when deploying multiple replicas or frequently restarting containers.

Downloading model weights in a local folder

To prevent from pulling the model weights from the Hugging Face Hub on container startup, we strongly recommend pulling those locally via the huggingface_hub Python CLI into a local directory with the following command:

hf download <MODEL_ID> --local-dir <DOWNLOAD_DIR>

Docker

Once the model weights are downloaded locally, you can mount those in the container and set MODEL_ID environment variable pointing to that path, so that those are not pulled on every container startup.

docker run --gpus all --shm-size 1g -p <ENGINE_DEFAULT_PORT>:80 \
  -v <DOWNLOAD_DIR>:/data \
  -e MODEL_ID=/data \
  registry.dell.huggingface.co/enterprise-dell-inference-<model>:<tag>

The docker run command above is just an example, what you should actually do is to copy the docker run command on the Dell Enterprise Hub (DEH) for a given model and then add the -v <DOWNLOAD_DIR>:/data and update the default value for the MODEL_ID environment variable to point to /data (or whatever path you chose within the container). Note that you should still respect and keep the rest of the configuration for the given model, meaning all the environment variables or container arguments that are defined in the original command.

Kubernetes

When working with Kubernetes, if you already downloaded and stored the model weights on a local path or in a shared network filesystem such as NFS, you can mount it into your Kubernetes pods, ensuring all nodes can access the directory (your storage must support at least ReadOnlyMany (ROX) access). To do so, you can use PersistentVolumes in your Kubernetes manifest.

This is an example of a Kubernetes manifest to deploy a model with pre-downloaded weights stored on an NFS server:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: model-weights-nfs-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: <NFS_SERVER_IP_OR_HOST>
    path: <DOWNLOAD_DIR> # path where model weights have been downloaded
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights-pvc
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 100Gi
  volumeName: model-weights-nfs-pv # binds to the PV above
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: server
  template:
    metadata:
      labels:
        app: server
        hf.co/model: <MODEL_ID>
        hf.co/task: text-generation
    spec:
      containers:
        - name: container
          image: registry.dell.huggingface.co/enterprise-dell-inference-<model>:<tag>
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: MODEL_ID
              value: "/mnt/data"
            - name: NUM_SHARD
              value: "1"
          volumeMounts:
            - name: model-weights
              mountPath: /mnt/data
              readOnly: true
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: model-weights
          persistentVolumeClaim:
            claimName: model-weights-pvc
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 1Gi
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3

Instead of using NFS, you can also use hostPath to mount a local path. In this case, make sure the model weights are available at the specified path on all nodes of your cluster. Then, just replace nfs section in the PersistentVolume definition with the following:

hostPath:
    path: "<PATH>"
    type: Directory

Using model weights from Hugging Face cache

An alternative for downloading the model weights before running the container consists on re-using the default Hugging Face cache and mount it, so that the container points to the cache mounted path via the environment variable HF_HUB_CACHE. This way you can benefit from using your existing local Hugging Face cache for mounting those, whilst still keeping the MODEL_ID default value, given that Hugging Face will resolve the MODEL_ID to whatever is on the cache.

To download the model weights in the cache, you can either just run hf download <MODEL_ID> and those will automatically be downloaded in the cache (i.e. ~/.cache/huggingface/hub), or rather specigy the --cache-dir argument pointing to another directory to act as the Hugging Face cache. Note that the file structure and hierarchy is different when using the cache versus when just pulling locally via --local-dir as previously mentioned.

hf download <MODEL_ID> --cache-dir <DOWNLOAD_DIR>

This command will download the model weights into <DOWNLOAD_DIR>/models--<MODEL_ID>/snapshots/.... This is needed if we want the container to check if the specified MODEL_ID is already cached.

Kubernetes

You can update the previous Kubernetes manifest to use the Hugging Face cache as follows:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: model-weights-nfs-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadOnlyMany
  nfs:
    server: <NFS_SERVER_IP_OR_HOST>
    path: <DOWNLOAD_DIR> # or ~/.cache/huggingface/hub if downloaded to de default cache using `hf download <MODEL_ID>`.
---
# ...
          env:
            - name: MODEL_ID
              value: "<MODEL_ID>"
            - name: HF_HUB_CACHE # no need to set it if mounting to the default cache path as shown below.
              value: "/mnt/data"
          volumeMounts:
            - name: model-weights
              mountPath: /mnt/data # or /root/.cache/huggingface/hub to mount to the default cache path.
              readOnly: true
# ...

Docker

You can update and run your Docker container as follows:

docker run --gpus all --shm-size 1g -p <ENGINE_DEFAULT_PORT>:80 \
  -v <DOWNLOAD_DIR>:/data/ \
  -e HF_HUB_CACHE=/data/ \
  -e MODEL_ID=<MODEL_ID> \
  registry.dell.huggingface.co/enterprise-dell-inference-<model>:<tag>