Note that there are timeouts in place - this is a demo pod which runs only for 24 hours and an interactive session also has a time limit, so it is better to build a custom run script which is executed when the container in the pod starts. See the Kubernetes documentation on pods and jobs for more details. TODO: link to respective doc.
'''Important known issue:Currently, on at least one of the compute nodes (zariel), the nVidia GPU device is not always created as "/dev/nvidia0", but sometimes as "/dev/nvidiaX", where X is another number (likely the device ID on the host). This makes Tensorflow fail to detect the GPU. While I am searching for a an actual solution, a workaround is'''
<syntaxhighlight>
(shell inside the container)