How to run GPU workloads in Docker containers on a GPU Server

Published on: November 6, 2026

Containers let you package GPU software together with the CUDA libraries it was built with, so applications that need different CUDA versions can run side by side on the same GPU Server. The server itself needs only the NVIDIA driver and the NVIDIA Container Toolkit, which gives containers access to the GPU.

This guide covers Docker Engine on Ubuntu 24.04, which the operating system with the NVIDIA driver preinstalled is built on. It walks you through setting up Docker and the NVIDIA Container Toolkit, choosing an image version that suits your driver, and running a PyTorch workload with docker run and Docker Compose.

Before you start

You need:

A GPU Server with the NVIDIA driver installed that you can connect to over SSH. To deploy one, see How to get started with GPU Servers. On Ubuntu Server 24.04 LTS, install the driver first, as described in How to install and update NVIDIA drivers on a GPU Server.
For the PyTorch example, about 9 GiB of free disk space on the server.

The docker commands in this guide run as root, the user you connect as in the getting started guide. If you connect as another user, add sudo in front of them.

Set up Docker for the GPU

The operating system with the NVIDIA driver preinstalled comes with Docker Engine and the NVIDIA Container Toolkit, and Docker is already set up to use the GPU. If your server runs it, go straight to Run a GPU container. On Ubuntu Server 24.04 LTS, you install and set up both yourself.

Install Docker Engine

Docker Engine comes from Docker's own apt repository. First, add the repository's key:

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Then add the repository and update the package lists:

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt update

Install Docker Engine and the Docker Compose plugin:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Docker starts as soon as it's installed. For other ways to install it, see Docker's Install Docker Engine on Ubuntu.

Install the NVIDIA Container Toolkit

The NVIDIA Container Toolkit comes from NVIDIA's CUDA repository, which you added when you installed the driver:

sudo apt install nvidia-container-toolkit

Configure Docker to use the GPU

Add the toolkit's NVIDIA runtime to Docker's configuration, then restart Docker to load it:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Output
INFO[0000] Config file does not exist; using empty config
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.

Run a GPU container

A container gets access to the GPU when you start it with --gpus all. To see the GPU from a container, run nvidia-smi in NVIDIA's CUDA base image:

docker run --rm --gpus all nvidia/cuda:13.2.1-base-ubuntu24.04 nvidia-smi

Docker downloads the image the first time, and nvidia-smi shows the same GPU and driver as on the server itself:

Output
Unable to find image 'nvidia/cuda:13.2.1-base-ubuntu24.04' locally
13.2.1-base-ubuntu24.04: Pulling from nvidia/cuda
...
Status: Downloaded newer image for nvidia/cuda:13.2.1-base-ubuntu24.04
Sat Oct 31 10:17:21 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.58.03              Driver Version: 595.58.03      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:00:06.0 Off |                    0 |
| N/A   31C    P8             25W /  350W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
...

Choose an image version

The CUDA version an image is built for needs to suit the driver on the server. The version is part of the image's tag, such as 13.2.1 in NVIDIA's CUDA images or cuda13.2 in PyTorch's. The newest version the driver supports is in the header of nvidia-smi, labelled CUDA Version, or CUDA UMD Version on the 615 driver branch. The driver runs images for that version or an older one.

Images for a newer version with the same major version usually run too, because of CUDA's minor version compatibility. A driver that shows CUDA 13.2, for example, also runs CUDA 13.4 images. Software that relies on features of the newer version can still need a newer driver, and NVIDIA's CUDA Compatibility documentation describes the limits. To update the driver to a branch that supports a newer CUDA version, see How to install and update NVIDIA drivers on a GPU Server.

Run a PyTorch workload

PyTorch's images come with PyTorch and the CUDA libraries it needs. Create a folder for this example's files and go to it:

mkdir gpu-check
cd gpu-check

Create a file named gpu_check.py, which multiplies two large matrices on the GPU:

import torch

print("PyTorch", torch.__version__, "with CUDA", torch.version.cuda)
print("GPU available:", torch.cuda.is_available(), torch.cuda.get_device_name(0))

a = torch.rand(8192, 8192, device="cuda")
b = torch.rand(8192, 8192, device="cuda")
c = a @ b
torch.cuda.synchronize()
print("Result:", tuple(c.shape), c.dtype)

Run the script in PyTorch's image for CUDA 13.2:

docker run --rm --gpus all -v "$PWD":/work -w /work pytorch/pytorch:2.15.0-cuda13.2-cudnn9-runtime python gpu_check.py

-v mounts the current folder in the container as /work, and -w makes it the container's working folder. Once Docker has downloaded the image, the script runs on the GPU:

Output
Unable to find image 'pytorch/pytorch:2.15.0-cuda13.2-cudnn9-runtime' locally
2.15.0-cuda13.2-cudnn9-runtime: Pulling from pytorch/pytorch
...
Status: Downloaded newer image for pytorch/pytorch:2.15.0-cuda13.2-cudnn9-runtime
PyTorch 2.15.0+cu132 with CUDA 13.2
GPU available: True NVIDIA L40S
Result: (8192, 8192) torch.float32

Use the GPU with Docker Compose

In Docker Compose, a service gets access to the GPU through a device reservation. To run the same PyTorch script as a Compose service, create a file named compose.yaml in the gpu-check folder:

services:
  pytorch:
    image: pytorch/pytorch:2.15.0-cuda13.2-cudnn9-runtime
    command: python gpu_check.py
    working_dir: /work
    volumes:
      - .:/work
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

The devices entry asks for NVIDIA GPUs, and count: all gives the service every GPU of the server, the same as --gpus all. Start the service:

docker compose up

Each line of the script's output starts with pytorch-1, the name of the service's container:

Output
[+] up 2/2
 ✔ Network gpu-check_default     Created
 ✔ Container gpu-check-pytorch-1 Created
Attaching to pytorch-1
pytorch-1  | PyTorch 2.15.0+cu132 with CUDA 13.2
pytorch-1  | GPU available: True NVIDIA L40S
pytorch-1  | Result: (8192, 8192) torch.float32
pytorch-1 exited with code 0

To remove the container once the script has ended, run:

docker compose down

Choose which GPUs a container uses

On a plan with more than one GPU, you can run several workloads side by side by giving each container only the GPUs it needs. nvidia-smi -L on the server lists the GPUs with their indexes:

nvidia-smi -L

Output
GPU 0: NVIDIA L40S (UUID: GPU-ca907630-ad71-6b08-e003-29eafa84d166)
GPU 1: NVIDIA L40S (UUID: GPU-e3b21e05-6508-a287-5306-d809451262f1)

--gpus also takes a number of GPUs, or device= followed by the index of a GPU. The first of these containers gets one GPU, and the second gets the GPU with index 1:

docker run --rm --gpus 1 nvidia/cuda:13.2.1-base-ubuntu24.04 nvidia-smi -L
docker run --rm --gpus device=1 nvidia/cuda:13.2.1-base-ubuntu24.04 nvidia-smi -L

Output
GPU 0: NVIDIA L40S (UUID: GPU-ca907630-ad71-6b08-e003-29eafa84d166)
GPU 0: NVIDIA L40S (UUID: GPU-e3b21e05-6508-a287-5306-d809451262f1)

Inside a container, the GPUs are numbered from 0, so the second container lists the server's GPU 1 as its GPU 0. Select GPUs by their index rather than their UUID, because a GPU Server can get different GPUs when you stop and start it. To give a container several GPUs, put the list in quotes, for example --gpus '"device=0,1"'.

In Docker Compose, replace count: all with the number of GPUs, or with device_ids and the indexes of the GPUs. For example, this gives the service the GPU with index 1:

          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

Troubleshooting

docker: Error response from daemon: failed to discover GPU vendor from CDI: no known GPU vendor found
The NVIDIA Container Toolkit isn't installed. Install it and configure Docker to use it, as described in Install the NVIDIA Container Toolkit.
docker: Error response from daemon: AMD CDI spec not found
Docker isn't configured to use the NVIDIA Container Toolkit. Configure it and restart Docker as described in Configure Docker to use the GPU.
docker: Error response from daemon: failed to create task for container: ... exec: "nvidia-smi": executable file not found in $PATH
The container was started without --gpus all, so it has no access to the GPU, and the CUDA image doesn't include nvidia-smi. Add --gpus all to the docker run command.
docker: Error response from daemon: cannot set both Count and DeviceIDs on device request
The list of GPU indexes after --gpus device= isn't in quotes. Put it in quotes, for example --gpus '"device=0,1"'.

Next steps

How to install and update NVIDIA drivers on a GPU Server explains how to keep the driver up to date and move to a branch that supports a newer CUDA version.
Getting Started with GPU Workloads in Managed Kubernetes explains how to run GPU containers in an UpCloud Managed Kubernetes cluster.
Recommended software for GPU Servers lists other software for GPU work.
Docker's Run Docker Compose services with GPU access describes the other device reservation options.

NVIDIA is a registered trademark of NVIDIA Corporation.

Contributed by: Samir Haliru