CCU:GPU Cluster Quick Start

Overview

The GPU cluster runs on Kubernetes, which is a container orchestrator. That means that users can run docker containers, which are essentially light-weight virtual machines without the overhead of an operating system, i.e. they mostly make use of the OS of the host machine. The additional layer in between allows the container to bring their own libraries with them, and shields the host OS from interference from the container. The containers are assigned to the host machines automatically, but the user has some options to specify which machine or which kind of machine they want to end up on. There is a global file system which is running on a Ceph cluster, which is mounted on every host. The details are not important for you, but it means that there is plenty of fast NVMe storage available which you can use for your code and datasets. You have to mount the directories which you want to use inside the container.

The typical workflow if you want to run your own applications is as follows:

1. Log in to the cluster and configure kubectl, the command line tool to talk to Kubernetes, to use your login credentials and namespace.
2. Create a persistent container to access the file systems, and mount the Ceph volumes inside it. Use this container to transfer code and data to the cluster and back.
3. (optional): Create your own custom container image with special libraries etc. which you need to run your code.
4. Create a GPU-enabled container based on your own image or one of the ready-made images with Deep Learning toolkits or whatever workload you want to run.
5. Start your workloads by logging into the container and running your code manually (only good for debugging), or by defining a job script which automatically runs a specified command inside the container until successful completion (recommended).

We will cover these points in more detail below.

Log in to the cluster and configure kubectl

You first need a working version of kubectl on your system. The cluster runs Kubernetes 1.28.2, the version of kubectl should match this. Check out installation instructions in the official Kubernetes documentation.

The login page to the cluster is here. Enter your credentials, you will get back an authorization token. Click on "full kubeconfig" on the left, and copy the content of this to a new file named ".kube/config" in your home directory. Note that the default namespace still has the template name "user-<firstname>-<lastname>". Replace this text with your username, so that your kubeconfig looks like this:

apiVersion: v1
kind: Config
preferences: {}
current-context: ccu-k8s
contexts:
- name: ccu-k8s
  context:
    user: your.name
    cluster: ccu-k8s.inf.uni-konstanz.de
    namespace: user-your-name
clusters:
- name: ccu-k8s.inf.uni-konstanz.de
  cluster:
    server: https://ccu-k8s.inf.uni-konstanz.de:7443
    certificate-authority-data: LS0tLS1C ... <many more characters>
    insecure-skip-tls-verify: false
users:
- name: your.name
  user:
    auth-provider:
      config:
        idp-issuer-url: https://ccu-k8s.inf.uni-konstanz.de:31000/dex
        client-id: loginapp
        id-token: eyJhbGciOiJSU ... <many more characters>
        client-secret: 4TORGiNV9M54BTk1v7dNuFSaI6hUjfjq <many more characters>
        refresh-token: ChlveGR ...
      name: oidc

The namespace "user-your-name" is your personal namespace within the Kubernetes cluster, and (so far) the only one you have access to. Conversely, no one else can access resources within your namespace. The kubeconfig above will make sure that all kubectl commands use your private namespace by default. Test your connection to the cluster now by running

> kubectl config use-context ccu-k8s
> kubectl get pods
No resources found in namespace user-your-name.

It is now time to create your first pod, which is a wrapper for one or more containers to run on the cluster. You might also want to become more familiar with the "kubectl" command at some point, check out the kubectl cheat sheet. A very good idea is to install bash autocompletion for kubectl, which is the very first tip on that page.

Note 1: There will very likely be occasions where your login credentials become invalid - they might time out, services might have been updated, certificates have been renewed, etc. In this case, please login again and update your kubeconfig with the new credentials. You then only need to update the block with your user data. If this still does not work, please report immediately, as there might be a problem with the login services.

Note 2: It is not supported to store separate credentials on two different computers. What will happen in this case is that one of them will consume the refresh token, which will then become invalid on the other one. If you need to access the cluster from a second computer, it is advised to use a ssh connection to your primary one where you store the credentials.

Create a pod to access the file systems

After login and adjusting the kubeconfig to the new cluster and user namespace, you should be able to start your first pod. Create a work directory on your machine, and a file "ubuntu-test-pod.yaml" with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-test-pod
spec:
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ["sleep", "1d"]
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 1
        memory: 1Gi
    volumeMounts:
      - mountPath: /abyss/home
        name: cephfs-home
        readOnly: false
      - mountPath: /abyss/shared
        name: cephfs-shared
        readOnly: false
      - mountPath: /abyss/datasets
        name: cephfs-datasets
        readOnly: true
  volumes:
    - name: cephfs-home
      hostPath:
        path: "/cephfs/abyss/home/<your-username>"
        type: Directory
    - name: cephfs-shared
      hostPath:
        path: "/cephfs/abyss/shared"
        type: Directory
    - name: cephfs-datasets
      hostPath:
        path: "/cephfs/abyss/datasets"
        type: Directory

When you run this on the cluster, it will create a pod for you which runs a container using the latest Ubuntu container image, and the ceph filesystems mounted into it. Use the following commands to create the pod and check out its status:

> kubectl apply -f ubuntu-test-pod.yaml
> kubectl get pods
> kubectl describe pod ubuntu-test-pod

Pay close attention to the event messages given at the end of the "describe pod" command, they give hints what might be wrong if the pod does not start up.

When the pod finally gets the status "running", you can log into the container just as in a remote server to obtain a shell prompt. Do this and verify that the filesystems have been mounted successfully:

> kubectl exec -it ubuntu-test-pod -- /bin/bash
# cd /abyss/home/
# ls
<might already contain stuff which was automatically copied from volumes on the old cluster.
#

From within the container, you have access to the internet, can install packages which are still missing, and also copy over your code and data via rsync or pulling it with e.g. git or svn. You can also push stuff into the container from your local machine using kubectl.

> kubectl cp <my-files> ubuntu-test-pod:/abyss/home/

This works also in the other direction to get stuff out of the pod. For more ideas for what you can do with kubectl, which is a powerful and complex tool, please refer to the basic kubectl cheat sheet or a more advanced version here.

The file systems you are mounting into the pod are available on every node in the cluster. The following directories can be used by anyone:

• /cephfs/abyss/home/<your-username>: this is your personal home directory which you can use any way you like.
• /cephfs/abyss/shared: a shared directory where every user has read/write access, so your data is not secure here from manipulation or deletion. To not have total anarchy in this filesystem, please give sensible names and organize in subdirectories. For example, put personal files which you want to make accessible to everyone in "/abyss/shared/users/<username>". Be considerate towards other users. I will monitor how it works out and whether we need more rules here. If you need more private storage shared only between all members of a trusted work group, please contact me.
• /cephfs/abyss/datasets: directory for static datasets, mounted read-only. These are large general-interest datasets for which we only want to store one copy on the filesystem (no separate imagenets for everyone, please). So whenever you have a well-known public dataset in your shared directory which you think is useful to have in the static tree, please contact me and I move it to the read-only region.

In addition, you can use a directory local to each host, which depending on your workload might be much faster than cephfs, but also ties you to a specific machine:

• /raid/local-data/<your-username>: your personal directory on the local SSD raid of the machine. Make sure to set "type: DirectoryOrCreate", at it is not guaranteed to exist yet.

Please refer to the persistent storage documentation for more details.

Running actual workloads on the cluster

You can now verify that you can start a GPU-enabled pod. Try to create a pod with the following specs to allocate 1 GPU for you somewhere on the cluster. The image we use is provided by nVidia and has Tensorflow/Keras pre-installed. There are many other useful base images around which you can use instead.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: gpu-container
    image: nvcr.io/nvidia/tensorflow:20.09-tf2-py3
    command: ["sleep", "1d"]
    resources:
      requests:
        cpu: 1
        nvidia.com/gpu: 1
        memory: 10Gi
      limits:
        cpu: 1
        nvidia.com/gpu: 1
        memory: 10Gi
    volumeMounts:
      - mountPath: /abyss/home
        name: cephfs-home
        readOnly: false
      - mountPath: /abyss/shared
        name: cephfs-shared
        readOnly: false
      - mountPath: /abyss/datasets
        name: cephfs-datasets
        readOnly: true
  volumes:
    - name: cephfs-home
      hostPath:
        path: "/cephfs/abyss/home/<username>"
        type: Directory
    - name: cephfs-shared
      hostPath:
        path: "/cephfs/abyss/shared"
        type: Directory
    - name: cephfs-datasets
      hostPath:
        path: "/cephfs/abyss/datasets"
        type: Directory

See the catalog of containers by nVidia for more options for base images (e.g. PyTorch), or Google around for containers of your favourite application. Make sure you only run containers from trusted sources!

Please note (very important): The versions 20.09 of the deep learning frameworks on nvcr.io work on all hosts in the cluster. While there are newer images available, they require drivers >= 455, which are not available for all machines yet. For guaranteed compability, you must stick to 20.09, but you can target a specific host with newer drivers.

At the bottom of the GPU cluster status page, there is the nvidia-smi output for each node, where you can check individual driver and CUDA version. You can also switch to a shell in the container and verify GPU capabilities:

> kubectl apply -f gpu-pod.yaml
... wait until pod is created, check with "kubectl describe pod gpu-pod" or "kubectl get pods"
> kubectl exec -it gpu-pod -- /bin/bash
# nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0    51W / 400W |      4MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

To check compabitility with specific nVidia containers, please refer to the official compatibility matrix. Note that all nodes have datacenter drivers installed, which should give a large amount of compability. If in doubt, just try it out.

Combine with the volume mounts above, and you already have a working environment. For example, you could transfer some code and data of yours to your home directory, and run it in interactive mode in the container as a quick test. Remember to adjust paths to data sets or to mount the directories in the locations expected by your code.

> kubectl exec -it gpu-pod -- /bin/bash
# cd /abyss/home/<your-code-repo>
# python ./main.py

Note that there are timeouts in place - this is a demo pod which runs only for 24 hours and an interactive session also has a time limit, so it is better to build a custom run script which is executed when the container in the pod starts. A job is a wrapper for a pod spec, which can for example make sure that the pod is restarted until it has at least one successful completion. This is useful for long deep learning work loads, where a pod failure might happen in between (for example due to a node reboot). See Kubernetes docs for pods or jobs for more details.

If you do not have your code ready, you can do a quick test if GPU execution works by running demo code from this tutorial as follows:

> kubectl exec -it gpu-pod -- /bin/bash
# cd /abyss/home
# git clone https://github.com/dragen1860/TensorFlow-2.x-Tutorials.git
# cd TensorFlow-2.x-Tutorials/12_VAE
# ls
README.md  images  main.py  variational_autoencoder.png
# pip3 install pillow matplotlib
# python ./main.py

Remember to clean up resources which you are not using anymore, this includes pods and jobs. For example, when your pod has finished what ever it is supposed to be doing, run

> kubectl delete -f gpu-pod.yaml

using the same manifest file you used to create the resource with kubectl apply.

Targeting specific nodes and GPU capabilities

By default, your pods will be scheduled on the lowest class of GPUs (in terms of memory available, they are mostly still quite decent). Please refer to the documentation on compute nodes for information how to target different nodes with higher capability.

Accessing ports on the pod from your own system

Some monitoring tools for deep learning use ports on the pod to convey information via a browser interface, an example being Tensorboard. You can forward these ports to your own local host using kubectl as a proxy. Follow the tutorial here to learn how it works. Syntax for port-forwarding:

> kubectl port-forward <pod-name> <dest-port>:<source-port>

kubectl will now continue running as a proxy. While it is running, you can access the pod service on "localhost:<dest-port>" in the browser on your own machine. You could even create containers which provide interactive environments via a web interface, e.g. a Jupyter notebook server.

Create, push and pull docker images to and from the CCU repository

Please follow our tutorial on how to create, push and pull docker images to and from our CCU repository:

• How to use the CCU image repository

Mount your custom, or Data Management Plan (DMP) provided, cifs storage

• How to mount cifs storage