GPU Support With Docker Compose

Fri, 01 Jan 2021 03:52:55 +0000

I use Docker quite a lot as it allows me to install, test, remove libraries or components without any risk to break my main system. VM is another option but even if a linux install is now just a few clicks in virt-manager GUI, it takes time and I don’t need the full isolation provided for this use-case. One thing that is quite difficult to setup in docker is the host GPU sharing. Same issue with the VM, unless having 2 cards installed in the host, giving access to the guest system either have an heavy performance impact, either require unlinking the card from the host to bind it to the guest system (and the host is now headless in the meantime, so hope the switch works without fail or you’ll be blind-fixing the issue in front of a dark screen).

It is usually not a problem as most of the containers are running services, and my emacs can easily be displayed on the host Xserver even when running inside a container. A Jupyter notebook with Tensorflow and GPU support or an instance of Leela to review a Go game is another story…

nvidia-docker, nvidia-docker2 or nvidia-container provided somewhat working solutions but replacing the default docker runtime broke some of my other containers, some were ok with nvidia-docker while others only with nvidia-container, and docker-compose was not usable with them at all. Of course, switching between the different approach required to uninstall/install the correct set of packages with matching versions. I had to rely to wrapper scripts to run the correct docker commands, which is basically the job of docker-compose… Somewhat working, but quite a messy solution.

GPU support was discussed for a long time and it seems we’re near fixing all of this docker-compose 1.28.0-rc1 is announcing “Support for Nvidia GPUs via device requests”.

I’ll update this with a more detailled setup when 1.28 is released, as things may change. The first part is the setup described here, moved to a container. The rest is a few lines to add to the compose yaml

My docker daemon config only move docker files out of /var and define a local registry:

> cat /etc/docker/daemon.json
{
  "data-root": "/home/docker",
  "registry-mirrors": [ "https://192.168.1.178:5000" ]
}

On the host the buster-backports nvidia-driver and nvidia-container-runtime are installed. In the Dockerfile, install nvidia-cuda-dev from buster-backports (thus we match the host driver version).

Tensorflow require cuDNN, so install the matching libcudnn8 too. I have a copy of this in a local debian repository: adding a source.list file to the image then apt-get install is faster than pushing it from the docker context (these libraries are huge).

In the docker-compose yaml, add:

jupyter:
  ...
  deploy:
    resources:
      reservations:
        devices:
        - 'driver': 'nvidia'
          'count': 1
          'capabilities': ['gpu', 'utility']
  devices:
    - /dev/nvidia-uvm:/dev/nvidia-uvm
    - /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
  ...

The deploy parameters will act as the –gpus option. The nvidia-uvm module is not always loaded, running modprobe beforehand (or force loading the module at boottime) may be required. Cuda will fail to initialize if these devices are not present. Once the devices are available on the host, nvidia-container doesn’t map them into the container, so we manually add them in the compose yaml.

Tensorflow with GPU acceleration on Debian Buster

Wed, 18 Dec 2019 04:23:17 +0000

A bit early for christmas presents, but I got myself a new machine (AMD Ryzen 3900X / 32G / RTX2060 Super, by the way Ryzen CPUs are real monsters). Now, let’s try to make use of this.

Nvidia drivers and tools

Assuming the Debian backports are configured, we can choose to use a more up to date driver version.

> cat /etc/apt/sources.list.d/backports.list
deb http://deb.debian.org/debian buster-backports main contrib non-free

Install the nvidia-driver (currently 450.80) and the nvidia-cuda-toolkit (currently 11.1)

cuDNN packages

Tensorflow require the cuDNN components but they are not available in the usual repository so we have to download the packages from Nvidia. cuDNN is not available for the Debian distribution but Ubuntu ones are compatible enough. The packages we need are :

libcudnn8_8.0.5.39-1+cuda11.1_amd64.deb
libcudnn8-dev_8.0.5.39-1+cuda11.1_amd64.deb

We’ll install them manually with a dpkg -i as configuring the Nvidia repository will probably fetch conflicting packages too.

Jupyter notebook

Most of the documentation on the internet related to machine learning is using the python API,

> python3 -mvenv jupyter-env
> cd jupyter-env
> . ./bin/activate
> pip install -U pip setuptools wheel
> pip install jupyterlab tensorflow-gpu

Then check if the GPU is available to Tensorflow :

> python3
>>> import tensorflow as tf
2020-12-26 11:19:12.763644: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
>>> tf.config.list_physical_devices()
2020-12-26 11:19:50.377149: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-12-26 11:19:50.382895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2020-12-26 11:19:50.471910: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-26 11:19:50.472295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: GeForce RTX 2060 SUPER computeCapability: 7.5
coreClock: 1.695GHz coreCount: 34 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-12-26 11:19:50.472310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-12-26 11:19:50.485168: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-12-26 11:19:50.485199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-12-26 11:19:50.491058: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-12-26 11:19:50.494631: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-12-26 11:19:50.494706: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-12-26 11:19:50.498531: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-12-26 11:19:50.498616: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-12-26 11:19:50.498625: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
>>>

And, we failed ! cuDNN was loaded, GPU is detected but Tensorflow wants an older version of libcusolver: nvidia-cuda-toolkit installed the version 11 but we need the version 10 (no idea why as other libraries use cuda11, which is the correct version for tensorflow 2.4).

~~Not a big issue as a libcusolver10 package is available in the backports. Let’s install that then give it another try:~~

Update: Now, libcusolver11 is no longer available in the buster-backports and tensorflow-gpu keep requesting the v10, it will probably be fixed in next tensorflow or nvidia drivers release but in the meantime, it seems the workaround is to use the cuda11 library through a symlink:

cd /usr/lib/x86_64-linux-gnu
ln -sf libcusolver.so.11 libcusolver.so.10
ldconfig


...
2020-12-26 11:21:28.182053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>>>

Now launch a notebook with jupyter lab, and run some tutorial code

...
2020-12-26 11:27:51.811618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6736 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060 SUPER, pci bus id: 0000:0a:00.0, compute capability: 7.5)
...

Yep, seems good now.

Nvidia | Cyrille Froehlich

GPU Support With Docker Compose

Tensorflow with GPU acceleration on Debian Buster

Nvidia drivers and tools

cuDNN packages

Jupyter notebook