CCU:New GPU Cluster

Overview

In January, the old GPU cluster will gradually be dismantled and integrated into a new Kubernetes cluster. The reason is a massive hardware upgrades of the backbone infrastructure:

• New Ceph-based storage cluster with currenly 210TB of NVMe storage to supply all compute nodes with data.
• New network backbone: HDR infiniband (200 GB/s).
• Triple-redundant servers to supply basic services and serve API requests, so that downtime should be minimized.
• As a cherry on top, another GPU server with 4x A100.

Since we reinstall everything from scratch, the usage of the Cluster will also change slightly, both for easier access to storage (getting rid of the somewhat cumbersome need to allocate persistent volumes) and improved security (separate user namespaces).

We first provide a comprehensive list of changes in how to use the cluster, then give a detailed manual for how to move over your data and pods.

Pod configuration on the new cluster

User namespace, pod security and quotas

Each user works in their own namespace now, which is auto-generated when your login is created. The naming convention is "firstname-lastname", i.e. you replace all '.'s in your cluster username with '-'. Thus, you need to set your default namespace in the kubeconfig accordingly, and perhaps update pod configurations. For security reasons, containers are forced to run with your own user id and your user group. To make configuration easy, a pod preset which sets all required options (in addition to mounting basic filesystems) is available in your namespace, see examples below for details.

Otherwise, the security policy for pods is now pretty restrictive, in particular, you can not switch to root inside containers anymore. If this presents problems, please contact me so we can work out a solution.

Finally, there is now a mechanism in place to set resource quotas for individual users. The preset is quite generous at the moment since we have plenty of resources, but if you believe your account is too limited, please contact me.

Persistent volume management (or lack thereof)

The ceph storage cluster provides a file system which is mounted on every node in the cluster. Pods are allowed to mount a subset of the filesystem as a host path, which is done automatically if you use the preconfigured pod preset in your namespace, see below. The following directories will be mounted in each of your containers which use the preset:

• /abyss/home: this is your personal home directory which you can use any way you like.
• /abyss/shared: a shared directory where every user has read/write access. It's a standard unix filesystem and everyone has an individual user id but is (for now) in the same user group. You can set the permission for files and directories you create accordingly to restrict or allow access. To not have total anarchy in this filesystem, please give sensible names and organize in subdirectories. For example, put personal files which you want to make accessible to everyone in "/abyss/shared/users/<your-namespace>". I will monitor how it works out and whether we need more rules here.
• /abyss/datasets: directory for static datasets, mounted read-only. These are large general-interest datasets for which we only want to store one copy on the filesystem (no separate imagenets for everyone, please). So whenever you have a well-known public dataset in your shared directory which you think is useful to have in the static tree, please contact me and I move it to the read-only region.

Copy data from the old cluster into the new filesystem

The shared file system can be mounted as a host path on the node "Vecna" on the old cluster, so you can create a pod on Vecna which mounts both the new filesystem as well as your PVs from the old cluster. Please use the following pod configuration as a template and add additional mounts for the PVs your want to copy over:

Afterwards, run a shell in the container and copy your stuff over to /abyss/shared/users/<your-namespace>. Make sure to set a group ownership id of 10000 with rw permissions for the group (rwx for directories) so you have read/write access on the new cluster. See commands below for an example how to do it.

Getting started on the new cluster

Login to the new cluster and update your kubeconfig

The frontend for the cluster and login services is located here:

https://ccu-k8s.inf.uni-konstanz.de/

Please follow instructions there to obtain credentials and cluster data for your kubeconfig.