I am able to run a tensorflow container w/ access to the GPU from the command line w/ the following command
$ sudo docker run --runtime=nvidia --rm gcr.io/tensorflow/tensorflow:latest-gpu
I would like to be able to run this container from docker-compose. Is it possible to specify the --runtime flag from docker-compose.yml?
Currently (Aug 2018), NVIDIA container runtime for Docker (nvidia-docker2) supports Docker Compose.
Yes, use Compose format 2.3 and add runtime: nvidia to your GPU service. Docker Compose must be version 1.19.0 or higher.
Example docker-compose.yml:
version: '2.3'
services:
nvsmi:
image: ubuntu:16.04
runtime: nvidia
environment:
- NVIDIA_VISIBLE_DEVICES=all
command: nvidia-smi
More example from NVIDIA blog uses Docker Compose to show how to launch multiple GPU containers with the NVIDIA Container Runtime.
You should edit /etc/docker/daemon.json, adding the first level key "default-runtime": "nvidia", restart docker daemon (ex. "sudo service docker restart") and then all containers on that host will run with nvidia runtime.
More info on daemon.json here
Or better: using systemd and assuming the path is /usr/libexec/oci/hooks.d/nvidia
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/nvidia-containers.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -D --add-runtime nvidia=/usr/libexec/oci/hooks.d/nvidia --default-runtime=nvidia
EOF
systemctl daemon-reload
systemctl restart docker
Don't need to specify --runtime=nvidia since we set default-runtime=nvidia in the configuration step.
docker run --rm gcr.io/tensorflow/tensorflow:latest-gpu
Solution Inspired from my tutorial about KATA runtime.
Setting the default runtime seemed the most reasonable to me. However Gabriel's answer did not work for me.
Instead I found that the nvidia container toolkit can automatically configure the daemon.json [1].
sudo nvidia-ctk runtime configure --runtime=docker --set-as-default
sudo service docker restart
This should be more future proof, since it now adds a different section to the daemon.json.
[1] https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.12.1/user-guide.html
