Changes

# Log in to the cluster and configure kubectl, the command line tool to talk to Kubernetes, to use your login credentialsand namespace.# Create a persistent container to access the global file systemsystems, and mount the Ceph volumes inside it. Use this container to transfer code and data to the cluster and back.

== Pod configuration on Log in to the new cluster and configure kubectl ==

=== User namespaceYou first need a working version of kubectl on your system. The cluster runs Kubernetes 1.28.2, pod security and quotas ===the version of kubectl should match this. Check out installation instructions in the [https://kubernetes.io/docs/tasks/tools/install-kubectl/ official Kubernetes documentation].

Each user works in their own namespace now, which is auto-generated when your login is created. The naming convention is as follows: * Login ID : firstname.lastname* Username : firstname-lastname* Namespace: user-firstname-lastname That means you replace all '.'s in your login ID with a '-' to obtain the username, and prepend "user-" page to obtain the namespace.  Thus, you should set your default namespace in the kubeconfig accordingly, and perhaps have to update pod configurations. For security reasons, containers are forced to run with your own user id and a group id of "10000". These will also be the ids used to create files and directories, and decide the permissions you have on the file system. The pod security policy which is active for your namespace will automatically fill in this data. Note that the security policy for pods is very restrictive for now to detect all problematic cases. In particular, you can not switch to root inside containers anymore. Please inform me if security policies disrupt your usual workflow so that we can work something out. Finally, there is now a mechanism in place to set resource quotas for individual users. The preset is quite generous at the moment since we have plenty of resources, but if you believe your account is too limited, please contact me. === Persistent volume management (or lack thereof) === The ceph storage cluster provides a file system which is mounted on every node in the cluster. Pods are allowed to mount a subset of the filesystem as a host path, see the example pod below. The following directories can be mounted[https: * '''/cephfs/abyss/home/<username>''': this is your personal home directory which you can use any way you likeccu-k8s.* '''/cephfs/abyss/shared''': a shared directory where every user has read/write accessinf. It's a standard unix filesystem and everyone has an individual user id but is (for now) in the same user groupuni-konstanz. You can set the permission for files and directories you create accordingly to restrict or allow accessde here]. To not have total anarchy in this filesystem, please give sensible names and organize in subdirectories. For exampleEnter your credentials, put personal files which you want to make accessible to everyone in "/abyss/shared/users/<username>". I will monitor how it works out and whether we need more rules here.* '''/cephfs/abyss/datasets''': directory for static datasets, mounted read-onlyget back an authorization token. These are large general-interest datasets for which we only want to store one copy Click on the filesystem (no separate imagenets for everyone, please). So whenever you have a well-known public dataset in your shared directory which you think is useful to have in the static tree, please contact me and I move it to the read-only region. == Copy data from the old cluster into the new filesystem == The shared file system can be mounted as an nfs volume on the node "Vecnafull kubeconfig" on the old clusterleft, so you can create a pod on Vecna which mounts both the new filesystem as well as your PVs from the old cluster. Please use the following pod configuration as a template and add additional mounts for copy the PVs you want content of this to copy over: <syntaxhighlight>apiVersion: v1kind: Podmetadata: name: <your-username>-transfer-pod namespace: exc-cbspec: nodeSelector: kubernetesa new file named ".iokube/hostname: vecna containers: - name: ubuntu image: ubuntu:20.04 command: ["sleep", "1dconfig"] volumeMounts: - mountPath: /abyss/shared name: cephfs-shared readOnly: false volumes: - name: cephfs-shared nfs: path: /cephfs/abyss/shared server: ccu-node1</syntaxhighlight> Afterwards, run a shell in the container and copy your stuff over to /abyss/shared/users/<your-username>home directory. Make sure to set a group ownership id of 10000 with rw permissions for Note that the group (rwx for directories) so you have read/write access on default namespace still has the new cluster. The following should do the trick: <syntaxhighlight>> kubectl exec template name "user-it <your-usernamefirstname>-transfer-pod /bin/bash# cd /cephfs/abyss/shared/users/<your-username># cp -r <all-my-stufflastname> "./# chgrp -R 10000 *# chown -R 10000 * (replace Replace this text with your real user ID if you already know it from logging into the new clusterusername, see below)# chmod -R g+w *</syntaxhighlight> == Getting started on the new cluster == === Login to the new cluster and update so that your kubeconfig === The frontend for the cluster and login services is located here: https://ccu-k8s.inf.uni-konstanz.de/ Please choose "login to the cluster" and enter your credentials to obtain the kubeconfig data. Choose "full kubeconfig" on the left for all the details you need. Either backup your old kubeconfig and use looks like this as a new one, or merge them both into a new kubeconfig which allows you to easily switch context between both clusters. In the beginning, this might be useful as you maybe have forgotten some data, and also still need to clean up once everything works. A kubeconfig for both clusters has the following structure (note this needs to be saved in "~/.kube/config"):

- clustername: certificateccu-authority-data: LS0tLS1CRUdJTiBDRV k8s.inf.uni-konstanz. <many more characters> server: https://134.34.224.84:6443de name: ccu-old- cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRV ... <many more characters>

name: ccu-newcontexts:- context: cluster: ccu-old namespace: exc-cb user: credentials-old name: ccucertificate-old- context: cluster: ccuauthority-new namespacedata: LS0tLS1C ... <your-namespacemany more characters> user: credentialsinsecure-new name: ccuskip-newcurrenttls-context: ccu-newkind: Configpreferencesverify: {}false

- name: credentialsyour.name user: auth-provider: config: idp-issuer-url: https://ccu-k8s.inf.uni-konstanz.de:31000/dex client-oldid: loginapp id-token: eyJhbGciOiJSU ... <all the data below your username returned from the old loginapp goes heremany more characters> client- namesecret: credentials4TORGiNV9M54BTk1v7dNuFSaI6hUjfjq <many more characters> refresh-newtoken: ChlveGR ... <all the data below your username returned from the new loginapp goes here> name: oidc

 Both the long CA data string and The namespace "user credentials are returned from the respective loginapps of the clusters. Note: -your-name" is your personal namespace within the CA data is different for both clustersKubernetes cluster, although the first couple of characters are and (so far) the same. If only one you have such a kubeconfig for multiple contextsaccess to. Conversely, you no one else can easily switch between access resources within your namespace. The kubeconfig above will make sure that all kubectl commands use your private namespace by default. Test your connection to the clusters:cluster now by running

> kubectl config use-context ccu-oldk8s> <... work with old cluster>kubectl get pods> kubectl config useNo resources found in namespace user-context ccuyour-new> <.name.. work with new cluster>

Defining different contexts It is now time to create your first pod, which is a wrapper for one or more containers to run on the cluster. You might also a want to become more familiar with the "kubectl" command at some point, check out the [https://kubernetes.io/docs/reference/kubectl/cheatsheet/ kubectl cheat sheet]. A very good way idea is to switch between namespaces or users (install bash autocompletion for kubectl, which should is the very first tip on that page. '''Note 1:''' There will very likely be occasions where your login credentials become invalid - they might time out, services might have been updated, certificates have been renewed, etc. In this case, please login again and update your kubeconfig with the new credentials. You then only need to update the block with your user data. If this still does not work, please report immediately, as there might be necessary for a problem with the average user)login services.

=== Running '''Note 2:''' It is not supported to store separate credentials on two different computers. What will happen in this case is that one of them will consume the first test container refresh token, which will then become invalid on the new other one. If you need to access the cluster ===from a second computer, it is advised to use a ssh connection to your primary one where you store the credentials.

== Create a pod to access the file systems == After login and adjusting the kubeconfig to the new cluster and user namespace, you should be able to start your first pod. The following example Create a work directory on your machine, and a file "ubuntu-test-pod mounts .yaml" with the ceph filesystems into an Ubuntu container image.following content:

path: "/cephfs/abyss/home/<your-username>"

Save this into a "test-pod.yaml"From within the container, start you have access to the pod internet, can install packages which are still missing, and also copy over your code and verify that data via rsync or pulling it has been created correcly and the filesystems have been mounted successfully, for example with the below commandse.g. git or svn. You can also check whether you can access push stuff into the data you have copied over and obtain the numeric user- and group-id for filesystem permissionscontainer from your local machine using kubectl.

> kubectl apply cp <my-f test-pod.yamlfiles> kubectl get pods> kubectl describe pod ubuntu-test-pod> kubectl exec -it ubuntu-test-pod /bin/bash$ ls :/abyss/sharedhome/<the directory you created for your data>$ iduid=10000 gid=10000 groups=10000

=== Moving your workloads This works also in the other direction to get stuff out of the pod. For more ideas for what you can do with kubectl, which is a powerful and complex tool, please refer to the new cluster ===basic [https://kubernetes.io/docs/reference/kubectl/cheatsheet/ kubectl cheat sheet] or a more [https://github.com/dennyzhang/cheatsheet-kubernetes-A4 advanced version here].

The file systems you are mounting into the pod are available on every node in the cluster. The following directories can be used by anyone: * '''/cephfs/abyss/home/<your-username>''': this is your personal home directory which you can use any way you like.* '''/cephfs/abyss/shared''': a shared directory where every user has read/write access, so your data is not secure here from manipulation or deletion. To not have total anarchy in this filesystem, please give sensible names and organize in subdirectories. For example, put personal files which you want to make accessible to everyone in "/abyss/shared/users/<username>". Be considerate towards other users. I will monitor how it works out and whether we need more rules here. If you need more private storage shared only between all members of a trusted work group, please contact me.* '''/cephfs/abyss/datasets''': directory for static datasets, mounted read-only. These are large general-interest datasets for which we only want to store one copy on the filesystem (no separate imagenets for everyone, please). So whenever you have a well-known public dataset in your shared directory which you think is useful to have in the static tree, please contact me and I move it to the read-only region. In addition, you can use a directory local to each host, which depending on your workload might be much faster than cephfs, but also ties you to a specific machine: * '''/raid/local-data/<your-username>''': your personal directory on the local SSD raid of the machine. Make sure to set "type: DirectoryOrCreate", at it is not guaranteed to exist yet. Please refer to [[CCU:Perstistent storage on the Kubernetes cluster|the persistent storage documentation]] for more details. == Running actual workloads on the cluster == You can now verify that you can start a GPU-enabled pod. Try to create a pod with the following specs to allocate 1 GPU for you somewhere on the cluster. The image we use is provided by nVidia and has Tensorflow/Keras pre-installed. There are many other useful base images around which you can use instead.

image: dockernvcr.io/nvidia/cudatensorflow:1120.009-basetf2-py3

memory: 100Mi10Gi

memory: 1Gi10Gi volumeMounts: - mountPath: /abyss/home name: cephfs-home readOnly: false - mountPath: /abyss/shared name: cephfs-shared readOnly: false - mountPath: /abyss/datasets name: cephfs-datasets readOnly: true volumes: - name: cephfs-home hostPath: path: "/cephfs/abyss/home/<username>" type: Directory - name: cephfs-shared hostPath: path: "/cephfs/abyss/shared" type: Directory - name: cephfs-datasets hostPath: path: "/cephfs/abyss/datasets" type: Directory

See [https://www.nvidia.com/en-us/gpu-cloud/containers/ the catalog of containers by nVidia] for more options for base images (e.g. [https://ngc.nvidia.com/catalog/containers/nvidia:pytorch PyTorch]), or Google around for containers of your favourite application. '''Make sure you only run containers from trusted sources!''' '''Please note (very important): The versions 20.09 of the deep learning frameworks on nvcr.io work on all hosts in the cluster. While there are newer images available, they require drivers >= 455, which are not available for all machines yet. For guaranteed compability, you must stick to 20.09, but you can target a specific host with newer drivers.'''  At the bottom of the GPU cluster status page, there is the nvidia-smi output for each node, where you can check individual driver and CUDA version. You can again also switch to a shell in the container and verify GPU capabilities:

> kubectl apply -f gpu-pod.yaml... wait until pod is created, check with "kubectl describe pod gpu-pod" or "kubectl get pods"> kubectl exec -it gpu-pod -- /bin/bash$ # nvidia-smi

Combine with the volume mounts above, and you already have a working environment. For example== Create, you could transfer some code push and data of yours pull docker images to your home directory, and run it in interactive mode in from the container as a quick test. Note that there are timeouts in place and an interactive session does not last forever, so it is better to build a custom run script which is executed when the container in the pod starts. See the documentation for more details. TODO: link to respective doc.CCU repository ==

Once everything works for you on * [[Tutorials:Link_to_container_registry_on_our_server | How to use the new cluster, please clean up your presence on the old one.CCU image repository]]

* Delete all running pods * Delete all persistent volume claims. This is the most important step, as it shows me which of the local filesystems of the nodes are not in use anymore, so I can transfer the node over [[Tutorials:Mount_cifs_storage_in_a_pod | How to the new cluster.mount cifs storage]]