CCU:GPU Cluster Quick Start

Overview

The GPU cluster runs on Kubernetes, which is a container orchestrator. That means that users can run docker containers, which are essentially light-weight virtual machines without the overhead of an operating system, i.e. they mostly make use of the OS of the host machine. The additional layer in between allows the container to bring their own libraries with them, and shields the host OS from interference from the container. The containers are assigned to the host machines automatically, but the user has some options to specify which machine or which kind of machine they want to end up on. There is a global file system which is running on a Ceph cluster, which is mounted on every host. The details are not important for you, but it means that there is plenty of fast NVMe storage available which you can use for your code and datasets. You have to mount the directories which you want to use inside the container.

The typical workflow if you want to run your own applications is as follows:

1. Log in to the cluster and configure kubectl, the command line tool to talk to Kubernetes, to use your login credentials.
2. Create a persistent container to access the file systems, and mount the Ceph volumes inside it. Use this container to transfer code and data to the cluster and back.
3. (optional): Create your own custom container image with special libraries etc. which you need to run your code.
4. Create a GPU-enabled container based on your own image or one of the ready-made images with Deep Learning toolkits or whatever workload you want to run.
5. Start your workloads by logging into the container and running your code manually (only good for debugging), or by defining a job script which automatically runs a specified command inside the container until successful completion (recommended).

We will cover these points in more detail below.

Log in to the cluster and configure kubectl

You first need a working version of kubectl on your system. The cluster runs Kubernetes 1.20.1, the version of kubectl should match this. Check out installation instructions in the (https://kubernetes.io/docs/tasks/tools/install-kubectl/ official Kubernetes documentation).

The login page to the cluster is here. Enter your credentials, you will get back an authorization token. Click on "full kubeconfig" on the left, and copy the content of this to a new file named ".kube/config" in your home directory. Note that the default namespace has the template name "user-<firstname>-<lastname>". Replace this text with your username, so that your kubeconfig looks like this:

apiVersion: v1
kind: Config
preferences: {}
current-context: ccu-k8s
contexts:
- name: ccu-k8s
  context:
    user: your.name
    cluster: ccu-k8s.inf.uni-konstanz.de
    namespace: user-your-name
clusters:
- name: ccu-k8s.inf.uni-konstanz.de
  cluster:
    server: https://ccu-k8s.inf.uni-konstanz.de:7443
    certificate-authority-data: LS0tLS1C ... <many more characters>
    insecure-skip-tls-verify: false
users:
- name: your.name
  user:
    auth-provider:
      config:
        idp-issuer-url: https://ccu-k8s.inf.uni-konstanz.de:31000/dex
        client-id: loginapp
        id-token: eyJhbGciOiJSU ... <many more characters>
        client-secret: 4TORGiNV9M54BTk1v7dNuFSaI6hUjfjq <many more characters>
        refresh-token: ChlveGR ...
      name: oidc

The namespace "user-your-name" is your personal namespace within the Kubernetes cluster, and (so far) the only one you have access to. Conversely, no one else can access resources within your namespace. The kubeconfig above will make sure that all kubectl commands use your private namespace by default. Test your connection to the cluster now by running

> kubectl get pods
No resources found in namespace .

It is now time to create your first pod, which is a wrapper for one or more containers to run on the cluster. You might also want to become more familiar with the "kubectl" command at some point, check out the kubectl cheat sheet. A very good idea is to install bash autocompletion for kubectl, which is the very first tip on that page.

Note 1: There will very likely be occasions where your login credentials become invalid - they might time out, services might have been updated, certificates have been renewed, etc. In this case, please login again and update your kubeconfig with the new credentials. You then only need to update the block with your user data. If this still does not work, please report immediately, as there might be a problem with the login services.

Note 2: It is not supported to store separate credentials on two different computers. What will happen in this case is that one of them will consume the refresh token, which will then become invalid on the other one. If you need to access the cluster from a second computer, it is advised to use a ssh connection to your primary one where you store the credentials.

Pod configuration on the new cluster

User namespace, pod security and quotas

Each user works in their own namespace now, which is auto-generated when your login is created. The naming convention is as follows:

• Login ID : firstname.lastname
• Username : firstname-lastname
• Namespace: user-firstname-lastname

That means you replace all '.'s in your login ID with a '-' to obtain the username, and prepend "user-" to obtain the namespace.

Thus, you should set your default namespace in the kubeconfig accordingly, and perhaps have to update pod configurations. For security reasons, containers are forced to run with your own user id and a group id of "10000". These will also be the ids used to create files and directories, and decide the permissions you have on the file system. The pod security policy which is active for your namespace will automatically fill in this data. Note that the security policy for pods is very restrictive for now to detect all problematic cases. In particular, you can not switch to root inside containers anymore. Please inform me if security policies disrupt your usual workflow so that we can work something out.

Finally, there is now a mechanism in place to set resource quotas for individual users. The preset is quite generous at the moment since we have plenty of resources, but if you believe your account is too limited, please contact me.

Persistent volume management (or lack thereof)

The ceph storage cluster provides a file system which is mounted on every node in the cluster. Pods are allowed to mount a subset of the filesystem as a host path, see the example pod below. The following directories can be mounted:

• /cephfs/abyss/home/<username>: this is your personal home directory which you can use any way you like.
• /cephfs/abyss/shared: a shared directory where every user has read/write access. It's a standard unix filesystem and everyone has an individual user id but is (for now) in the same user group. You can set the permission for files and directories you create accordingly to restrict or allow access. To not have total anarchy in this filesystem, please give sensible names and organize in subdirectories. For example, put personal files which you want to make accessible to everyone in "/abyss/shared/users/<username>". I will monitor how it works out and whether we need more rules here.
• /cephfs/abyss/datasets: directory for static datasets, mounted read-only. These are large general-interest datasets for which we only want to store one copy on the filesystem (no separate imagenets for everyone, please). So whenever you have a well-known public dataset in your shared directory which you think is useful to have in the static tree, please contact me and I move it to the read-only region.

Copy data from the old cluster into the new filesystem

The shared file system can be mounted as an nfs volume on the node "Vecna" on the old cluster, so you can create a pod on Vecna which mounts both the new filesystem as well as your PVs from the old cluster. Please use the following pod configuration as a template and add additional mounts for the PVs you want to copy over:

apiVersion: v1
kind: Pod
metadata:
  name: <your-username>-transfer-pod
  namespace: exc-cb
spec:
  nodeSelector:
    kubernetes.io/hostname: vecna
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ["sleep", "1d"]
    volumeMounts:
      - mountPath: /abyss/shared
        name: cephfs-shared
        readOnly: false
  volumes:
    - name: cephfs-shared
      nfs:
        path: /cephfs/abyss/shared
        server: ccu-node1

Afterwards, run a shell in the container and copy your stuff over to /abyss/shared/users/<your-username>. Make sure to set a group ownership id of 10000 with rw permissions for the group (rwx for directories) so you have read/write access on the new cluster. The following should do the trick:

> kubectl exec -it <your-username>-transfer-pod /bin/bash
# cd /cephfs/abyss/shared/users/<your-username>
# cp -r <all-my-stuff> ./
# chgrp -R 10000 *
# chown -R 10000 *    (replace with your real user ID if you already know it from logging into the new cluster, see below)
# chmod -R g+w *

Getting started on the new cluster

Login to the new cluster and update your kubeconfig

The frontend for the cluster and login services is located here:

https://ccu-k8s.inf.uni-konstanz.de/

Please choose "login to the cluster" and enter your credentials to obtain the kubeconfig data. Choose "full kubeconfig" on the left for all the details you need. Either backup your old kubeconfig and use this as a new one, or merge them both into a new kubeconfig which allows you to easily switch context between both clusters. In the beginning, this might be useful as you maybe have forgotten some data, and also still need to clean up once everything works.

A kubeconfig for both clusters has the following structure (note this needs to be saved in "~/.kube/config"):

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRV ... <many more characters>
    server: https://134.34.224.84:6443
  name: ccu-old
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRV ... <many more characters>
    server: https://ccu-k8s.inf.uni-konstanz.de:7443
  name: ccu-new
contexts:
- context:
    cluster: ccu-old
    namespace: exc-cb
    user: credentials-old
  name: ccu-old
- context:
    cluster: ccu-new
    namespace: <your-namespace>
    user: credentials-new
  name: ccu-new
current-context: ccu-new
kind: Config
preferences: {}
users:
- name: credentials-old
  <all the data below your username returned from the old loginapp goes here>
- name: credentials-new
  <all the data below your username returned from the new loginapp goes here>

Both the long CA data string and user credentials are returned from the respective loginapps of the clusters. Note: the CA data is different for both clusters, although the first couple of characters are the same. If you have such a kubeconfig for multiple contexts, you can easily switch between the clusters:

> kubectl config use-context ccu-old
> <... work with old cluster>
> kubectl config use-context ccu-new
> <... work with new cluster>

Defining different contexts is also a good way to switch between namespaces or users (which should not be necessary for the average user).

Running the first test container on the new cluster

After login and adjusting the kubeconfig to the new cluster and user namespace, you should be able to start your first pod. The following example pod mounts the ceph filesystems into an Ubuntu container image.

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-test-pod
spec:
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ["sleep", "1d"]
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 1
        memory: 1Gi
    volumeMounts:
      - mountPath: /abyss/home
        name: cephfs-home
        readOnly: false
      - mountPath: /abyss/shared
        name: cephfs-shared
        readOnly: false
      - mountPath: /abyss/datasets
        name: cephfs-datasets
        readOnly: true
  volumes:
    - name: cephfs-home
      hostPath:
        path: "/cephfs/abyss/home/<username>"
        type: Directory
    - name: cephfs-shared
      hostPath:
        path: "/cephfs/abyss/shared"
        type: Directory
    - name: cephfs-datasets
      hostPath:
        path: "/cephfs/abyss/datasets"
        type: Directory

Save this into a "test-pod.yaml", start the pod and verify that it has been created correcly and the filesystems have been mounted successfully, for example with the below commands. You can also check whether you can access the data you have copied over and obtain the numeric user- and group-id for filesystem permissions.

> kubectl apply -f test-pod.yaml
> kubectl get pods
> kubectl describe pod ubuntu-test-pod
> kubectl exec -it ubuntu-test-pod /bin/bash
$ ls /abyss/shared/<the directory you created for your data>
$ id
uid=10000 gid=10000 groups=10000

Moving your workloads to the new cluster

You can now verify that you can start a GPU-enabled pod. Try to create a pod with the following specs to allocate 1 GPU for you somewhere on the cluster.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
  - name: gpu-container
    image: docker.io/nvidia/cuda:11.0-base
    command: ["sleep", "1d"]
    resources:
      requests:
        cpu: 1
        nvidia.com/gpu: 1
        memory: 100Mi
      limits:
        cpu: 1
        nvidia.com/gpu: 1
        memory: 1Gi

You can again switch to a shell in the container and verify GPU capabilities:

> kubectl exec -it gpu-pod /bin/bash
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0    51W / 400W |      4MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

Combine with the volume mounts above, and you already have a working environment. For example, you could transfer some code and data of yours to your home directory, and run it in interactive mode in the container as a quick test. Note that there are timeouts in place and an interactive session does not last forever, so it is better to build a custom run script which is executed when the container in the pod starts. See the documentation for more details. TODO: link to respective doc.

Cleaning up

Once everything works for you on the new cluster, please clean up your presence on the old one.

In particular:

• Delete all running pods
• Delete all persistent volume claims. This is the most important step, as it shows me which of the local filesystems of the nodes are not in use anymore, so I can transfer the node over to the new cluster.