Tutorials:Persistent volumes on the Kubernetes cluster

• • • • THIS IS OUTDATED INFORMATION, PLEASE REFER TO CCU:Perstistent storage on the Kubernetes cluster instead.

Prerequisites

• Pre-requisited from previous tutorial.
• Sample code from previous tutorial.

Global dataset storage for large, static datasets

The first cluster node exports an NFS filesystem on a large NVMe-Raid, which is reasonably fast and can be used as a global dataset storage. It can be mounted into a pod as follows:

apiVersion: v1
kind: Pod
metadata:
  name: your-username-test-global-storage
spec:
  containers:
  - name: your-username-test-global-storage

    # we use a small ubuntu base to access the PVC
    image: ubuntu:18.04
    # make sure that we have some time until the container quits by itself
    command: ['sleep', '6h']

    volumeMounts:
    # Path to mount the NFS volume to
    - mountPath: "/mnt/datasets"
      name: datasets-nfs
      # NFS is exported read-only
      readOnly: true

  volumes:
    # Volume which mounts the NFS server exported to the cluster by ccu-node1
    - name: datasets-nfs
      nfs:
        server: ccu-node1
        path: /raid/datasets

Please see the page on global storage for a list of available datasets and the method to upload your own.

Persistent volumes

A persistent volume in Kubernetes is a cluster resource which can be requested by a container. For this, you have to claim a persistent volume (PV) using a persistent volume claim (PVC), which you apply in your namespace. The persistent volume claim can then be mounted to directories within a container. The important point is that the PVC survives the end of the container, i.e. the data in the PV will be permanent until the PVC is released. If the PVC is mounted again to a new container, the data will still be present. A persistent volume which is bound to a claim can not be assigned to any other claim. If the PVC is released, the PV is also released and immediately and automatically wiped clean of all data. If you want to keep your data, copy it to some other permanent storage first.

On the cluster, there are two types of persistent volumes currently configured:

• Local persistent volumes
• Global persistent volumes

Note: the cluster will soon get large, fast global storage, at this point local persistent volumes will be phased out and probably not available anymore. Tensorboard monitoring should be done using service exports, as explained below, and not make use of local PVs.

Local persistent volumes

These are persistent volumes which are mapped to special folders of the host filesystem of the node. Each node exposes several persistent volumes which can be claimed. The user can not control exactly which volume is bound to a claim, but can request a minimum size. A persistent volume claim for a local PV is configured like this. Code examples can be found in the subdirectory "kubernetes/example_2" of the tutorial sample code, File:Kubernetes samples.zip.

WARNING: Once a local persistent volume has been bound to a specific node, all pods which make use this volume are forced to also run on this node. This means you have to rely on resources (e.g. GPUs) being available on exactly that particular node.

NOTE: The storage class "local-ssd" which was previously used for local persistent volumes is now obsolete, since a better driver with automatic provisioning has been installed. From now on, please use "local-path" instead, which will give you a PV on the fastest local device (usually SSD/NVMe RAID). No new volumes of class "local-ssd" can be claimed. Please copy over your data from old PVCs if you have the opportunity, or delete old PVCs not in use anymore. As soon as there are no more PVCs of the old class in use, it will be deleted from the cluster. Also, check out "global-datasets" below, which gives you a new opportunity to store large, static datasets on a very fast device.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # the name of the PVC, we refer to this in the container configuration
  name: tf-mnist-pvc

spec:
  resources:
    requests:
      # storage resource request. This PVC can only be bound to volumes which
      # have at least 8 GiB of storage available.
      storage: 8Gi

  # the requested storage class, see tutorial.
  storageClassName: local-path

  # leave these unchanged, they must match the PV type, otherwise binding will fail
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem

The following storage classes are configured in the cluster:

When the claim is defined to your satisfaction, apply it like this:

> kubectl apply -f pvc.yaml

You can check on the status of this (and every other) claim:

> kubectl get pvc
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
tf-mnist-pvc   Pending                                      local-path     11s

Since the claim has not been used by a container yet, it is not yet bound to a persitent volume (PV).

Global persistent volumes

In contrast, global persistent volumes are provided cluster-wide and are accessible from any node (managed internally with rook-ceph). They reside on SSDs and thus should be reasonably fast, however, depending on where the volume ends up, data will probably be transferred across the network to/from the node. Thus, they are slower than local-ssd, but leave you considerably more flexible, as they do not require pods to run on specific nodes. Also, there is no constraint on maximum size except for physical limitations. Currently, there is a total of 20 TB of cluster-wide SSD storage, which we plan to increase considerably in the near future.

Compared to creating local persistent volumes, the only thing which needs to be changed is the storage class to "ceph-ssd".

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # the name of the PVC, we refer to this in the container configuration
  name: tf-mnist-global-pvc

spec:
  resources:
    requests:
      # storage resource request. This PVC can only be bound to volumes which
      # have at least 8 GiB of storage available.
      storage: 8Gi

  # the requested storage class, see tutorial.
  storageClassName: ceph-ssd

  # access mode is mandatory
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  # For me (Felix) it worked only with the additional following line:
  volumeMode: Filesystem

Since anyone can mount global persistent volumes in the same namespace, they can and should be used to share datasets. The name of a PVC which contains a useful dataset should start with "dataset-" and be descriptive, so that it can easily be found by other users. Also, the root of the PVC should contain a README with informations about the dataset (at least the source and what exactly it is).

A note on mounting. Currently (will change in the near future), ceph volumes can be either mounted ReadWrite by a single pod only, or ReadOnly by multiple pods. Thus, the workflow for a static dataset is to create the PVC, then create a pod to write all the data to it, then delete this pod and mount it read only from now on so it can be used in multiple pods.

Reading/writing the contents of a persistent volume

You can access a PV which is bound to a PVC by mounting it into a container. For a demonstration, we use the simple container image "ubuntu:18.04", which runs a minimalistic Ubuntu, and keep it in a very long wait after container startup.

# Test pod to mount a PV bound to a PVC into a container
# Before starting this pod, apply the PVC with kubectl apply -f pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: your-username-pvc-access-pod
spec:
  containers:
    - name: pvc-access-container

      # we use a small ubuntu base to access the PVC
      image: ubuntu:18.04
      # make sure that we have some time until the container quits by itself
      command: ['sleep', '6h']

      # list of mount paths within the container which will be
      # bound to persistent volumes.
      volumeMounts:
      - mountPath: "/mnt/pvc-mnist"
        # name of the volume for this path (from the below list)
        name: pvc-mnist

  volumes:
    # User-defined name of the persistent volume within this configuration.
    # This can be different from the name of the PVC.
    - name: pvc-mnist
      persistentVolumeClaim:
        # name of the PVC this volume binds to
        claimName: your-username-tf-mnist-pvc

After the PVC is applied, spin up the test pod with

> kubectl apply -f pvc-access-pod.yaml

You now have several options to get data to and from the container.

1. Copying data from within the container

You can get a root shell inside the container as usual (insert the correct pod name you used below):

> kubectl exec -it pvc-access-pod /bin/bash

Your pod has internet access. Thus, an option to get data to/from the pod, in particular into the persistent volume, is to use scp, which first needs to be installed inside the pod:

# apt-get update && apt install openssh-client rsync
# cd /my-pvc-mount-path
# scp your.username@external-server:/path/to/data/. ./

An even better variant would be "rsync -av" instead of scp, as this only copies files which are different or do not exist in the destination. By reversing source and destination, you can also copy data out of the container this way.

2. Copying data from the outside

From the outside world, you can directly copy data to and from the container using kubectl cp, which has a very similar syntax as scp:

# to get data into the container, substitute name with correct id obtained from kubectl get pods
> kubectl cp /path/to/data/. pvc-access-pod:/my-pvc-mount/path/data
# to get data from the container
> kubectl cp pvc-access-pod:/my-pvc-mount/path/. /path/to/output/

Read up on Kubernetes "kubectl cp" documentation to check how it handles directories, it's a bit unusual and slightly different from scp.

Note: kubectl cp internally uses tar and some compression to speed up network transfer. However, this means that your access pod needs a certain amount of memory, in particular when transferring large files. If you run into "error 137" (out of memory), increase memory limits of the access pod or use scp from within the pod.