CCU:GPU Cluster Quick Start

Overview

The GPU cluster runs on Kubernetes, which is a container orchestrator. That means that users can run docker containers, which are essentially light-weight virtual machines without the overhead of an operating system, i.e. they mostly make use of the OS of the host machine. The additional layer in between allows the container to bring their own libraries with them, and shields the host OS from interference from the container. The containers are assigned to the host machines automatically, but the user has some options to specify which machine or which kind of machine they want to end up on. There is a global file system which is running on a Ceph cluster, which is mounted on every host. The details are not important for you, but it means that there is plenty of fast NVMe storage available which you can use for your code and datasets. You have to mount the directories which you want to use inside the container.

The typical workflow if you want to run your own applications is as follows:

1. Log in to the cluster and configure kubectl, the command line tool to talk to Kubernetes, to use your login credentials and namespace.
2. Create a persistent container to access the file systems, and mount the Ceph volumes inside it. Use this container to transfer code and data to the cluster and back.
3. (optional): Create your own custom container image with special libraries etc. which you need to run your code.
4. Create a GPU-enabled container based on your own image or one of the ready-made images with Deep Learning toolkits or whatever workload you want to run.
5. Start your workloads by logging into the container and running your code manually (only good for debugging), or by defining a job script which automatically runs a specified command inside the container until successful completion (recommended).

We will cover these points in more detail below.

Log in to the cluster and configure kubectl

You first need a working version of kubectl on your system. The cluster runs Kubernetes 1.20.1, the version of kubectl should match this. Check out installation instructions in the official Kubernetes documentation.

The login page to the cluster is here. Enter your credentials, you will get back an authorization token. Click on "full kubeconfig" on the left, and copy the content of this to a new file named ".kube/config" in your home directory. Note that the default namespace still has the template name "user-<firstname>-<lastname>". Replace this text with your username, so that your kubeconfig looks like this:

apiVersion: v1
kind: Config
preferences: {}
current-context: ccu-k8s
contexts:
- name: ccu-k8s
  context:
    user: your.name
    cluster: ccu-k8s.inf.uni-konstanz.de
    namespace: user-your-name
clusters:
- name: ccu-k8s.inf.uni-konstanz.de
  cluster:
    server: https://ccu-k8s.inf.uni-konstanz.de:7443
    certificate-authority-data: LS0tLS1C ... <many more characters>
    insecure-skip-tls-verify: false
users:
- name: your.name
  user:
    auth-provider:
      config:
        idp-issuer-url: https://ccu-k8s.inf.uni-konstanz.de:31000/dex
        client-id: loginapp
        id-token: eyJhbGciOiJSU ... <many more characters>
        client-secret: 4TORGiNV9M54BTk1v7dNuFSaI6hUjfjq <many more characters>
        refresh-token: ChlveGR ...
      name: oidc

The namespace "user-your-name" is your personal namespace within the Kubernetes cluster, and (so far) the only one you have access to. Conversely, no one else can access resources within your namespace. The kubeconfig above will make sure that all kubectl commands use your private namespace by default. Test your connection to the cluster now by running

> kubectl config use-context ccu-k8s
> kubectl get pods
No resources found in namespace user-your-name.

It is now time to create your first pod, which is a wrapper for one or more containers to run on the cluster. You might also want to become more familiar with the "kubectl" command at some point, check out the kubectl cheat sheet. A very good idea is to install bash autocompletion for kubectl, which is the very first tip on that page.

Note 1: There will very likely be occasions where your login credentials become invalid - they might time out, services might have been updated, certificates have been renewed, etc. In this case, please login again and update your kubeconfig with the new credentials. You then only need to update the block with your user data. If this still does not work, please report immediately, as there might be a problem with the login services.

Note 2: It is not supported to store separate credentials on two different computers. What will happen in this case is that one of them will consume the refresh token, which will then become invalid on the other one. If you need to access the cluster from a second computer, it is advised to use a ssh connection to your primary one where you store the credentials.

Create a pod to access the file systems

After login and adjusting the kubeconfig to the new cluster and user namespace, you should be able to start your first pod. Create a work directory on your machine, and a file "access-pod.yaml" with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-test-pod
spec:
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ["sleep", "1d"]
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 1
        memory: 1Gi
    volumeMounts:
      - mountPath: /abyss/home
        name: cephfs-home
        readOnly: false
      - mountPath: /abyss/shared
        name: cephfs-shared
        readOnly: false
      - mountPath: /abyss/datasets
        name: cephfs-datasets
        readOnly: true
  volumes:
    - name: cephfs-home
      hostPath:
        path: "/cephfs/abyss/home/<your-username>"
        type: Directory
    - name: cephfs-shared
      hostPath:
        path: "/cephfs/abyss/shared"
        type: Directory
    - name: cephfs-datasets
      hostPath:
        path: "/cephfs/abyss/datasets"
        type: Directory

When you run this on the cluster, it will create a pod for you which runs a container using the latest Ubuntu container image, and the ceph filesystems mounted into it. Use the following commands to create the pod and check out its status:

> kubectl apply -f access-pod.yaml
> kubectl get pods
> kubectl describe pod access-pod

Pay close attention to the event messages given at the end of the "describe pod" command, they give hints what might be wrong if the pod does not start up.

When the pod finally gets the status "running", you can log into the container just as in a remote server to obtain a shell prompt. Do this and verify that the filesystems have been mounted successfully:

> kubectl exec -it access-pod -- /bin/bash
# cd /abyss/home/
# ls
<might already contain stuff which was automatically copied from volumes on the old cluster.
#

From within the container, you have access to the internet, can install packages which are still missing, and also copy over your code and data via rsync or pulling it with e.g. git or svn. You can also push stuff into the container from your local machine using kubectl.

> kubectl cp <my-files> access-pod:/abyss/home/

This works also in the other direction to get stuff out of the pod. For more ideas for what you can do with kubectl, which is a powerful and complex tool, please refer to the basic kubectl cheat sheet or a more advanced version here.

The file systems you are mounting into the pod are available on every node in the cluster. The following directories can be used by anyone:

• /cephfs/abyss/home/<your-username>: this is your personal home directory which you can use any way you like.
• /cephfs/abyss/shared: a shared directory where every user has read/write access. It's a standard unix filesystem and everyone has an individual user id but is (for now) in the same user group. You can set the permission for files and directories you create accordingly to restrict or allow access. To not have total anarchy in this filesystem, please give sensible names and organize in subdirectories. For example, put personal files which you want to make accessible to everyone in "/abyss/shared/users/<username>". I will monitor how it works out and whether we need more rules here.
• /cephfs/abyss/datasets: directory for static datasets, mounted read-only. These are large general-interest datasets for which we only want to store one copy on the filesystem (no separate imagenets for everyone, please). So whenever you have a well-known public dataset in your shared directory which you think is useful to have in the static tree, please contact me and I move it to the read-only region.

Running actual workloads on the cluster

You can now verify that you can start a GPU-enabled pod. Try to create a pod with the following specs to allocate 1 GPU for you somewhere on the cluster. The image we use is provided by nVidia and has Tensorflow/Keras pre-installed. There are many other useful base images around which you can use instead.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: gpu-container
    image: nvcr.io/nvidia/tensorflow:20.09-tf2-py3
    command: ["sleep", "1d"]
    resources:
      requests:
        cpu: 1
        nvidia.com/gpu: 1
        memory: 10Gi
      limits:
        cpu: 1
        nvidia.com/gpu: 1
        memory: 10Gi
    volumeMounts:
      - mountPath: /abyss/home
        name: cephfs-home
        readOnly: false
      - mountPath: /abyss/shared
        name: cephfs-shared
        readOnly: false
      - mountPath: /abyss/datasets
        name: cephfs-datasets
        readOnly: true
  volumes:
    - name: cephfs-home
      hostPath:
        path: "/cephfs/abyss/home/<username>"
        type: Directory
    - name: cephfs-shared
      hostPath:
        path: "/cephfs/abyss/shared"
        type: Directory
    - name: cephfs-datasets
      hostPath:
        path: "/cephfs/abyss/datasets"
        type: Directory

See the catalog of containers by nVidia for more options for base images (e.g. PyTorch), or Google around for containers of your favourite application. Make sure you only run containers from trusted sources!

Please note (very important): The versions 20.09 of the deep learning frameworks on nvcr.io work on all hosts in the cluster. While there are newer images available, they require drivers >= 455, which are not available for all machines yet. So please stick to 20.09 unless you target a very specific host. I will soon provide a table with driver versions for all hosts once they are upgraded and moved to the new cluster. As a general rule, everything which is made for Cuda 11.0 and driver version >= 450 should work fine on the Cluster.

You can again switch to a shell in the container and verify GPU capabilities:

> kubectl exec -it gpu-pod /bin/bash
# nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0    51W / 400W |      4MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

Combine with the volume mounts above, and you already have a working environment. For example, you could transfer some code and data of yours to your home directory, and run it in interactive mode in the container as a quick test. Remember to adjust paths to data sets or to mount the directories in the locations expected by your code.

> kubectl exec -it gpu-pod -- /bin/bash
# cd /abyss/home/<your-code-repo>
# python ./main.py

Note that there are timeouts in place - this is a demo pod which runs only for 24 hours and an interactive session also has a time limit, so it is better to build a custom run script which is executed when the container in the pod starts. A job is a wrapper for a pod spec, which can for example make sure that the pod is restarted until it has at least one successful completion. This is useful for long deep learning work loads, where a pod failure might happen in between (for example due to a node reboot). See Kubernetes docs for pods or jobs for more details.

If you do not have your code ready, you can do a quick test if GPU execution works by running demo code from this tutorial as follows:

> kubectl exec -it gpu-pod -- /bin/bash
# cd /abyss/home
# git clone https://github.com/dragen1860/TensorFlow-2.x-Tutorials.git
# cd TensorFlow-2.x-Tutorials/12_VAE
# ls
README.md  images  main.py  variational_autoencoder.png
# pip3 install pillow matplotlib
# python ./main.py

Accessing ports on the pod from your own system

Some monitoring tools for deep learning use ports on the pod to convey information via a browser interface, an example being Tensorboard. You can forward these ports to your own local host using kubectl as a proxy. Follow the tutorial here to learn how it works. Syntax for port-forwarding:

> kubectl port-forward <pod-name> <dest-port>:<source-port>

kubectl will now continue running as a proxy. While it is running, you can access the pod service on "localhost:<dest-port>" in the browser on your own machine. You could even create containers which provide interactive environments via a web interface, e.g. a Jupyter notebook server.