ccu, Administrators
684
edits
Changes
Jump to navigation
Jump to search
Large training data sets Compared to creating local persistent volumes, the only thing which are required by many different users are stored permanently in needs to be changed is the filesystem of several nodesstorage class to "ceph-ssd". They can be claimed with a PVC as follows:
TODO[[Category: Will finish this part soon, for now, read up on Kubernetes "kubectl cp" documentation to copy stuff to/from a PV.Tutorials]]
no edit summary
**** THIS IS OUTDATED INFORMATION, PLEASE REFER TO [[CCU:Perstistent storage on the Kubernetes cluster]] instead.
== Prerequisites ==
* Pre-requisited from [[Tutorials:Run_the_example_container_on_the_cluster|previous tutorial]].
* Sample code from [[Tutorials:Run_the_example_container_on_the_cluster|previous tutorial]].
== Global dataset storage for large, static datasets ==
The first cluster node exports an NFS filesystem on a large NVMe-Raid, which is reasonably fast and can be used as a global dataset storage. It can be mounted into a pod as follows:
<syntaxhighlight lang="yaml">
apiVersion: v1
kind: Pod
metadata:
name: your-username-test-global-storage
spec:
containers:
- name: your-username-test-global-storage
# we use a small ubuntu base to access the PVC
image: ubuntu:18.04
# make sure that we have some time until the container quits by itself
command: ['sleep', '6h']
volumeMounts:
# Path to mount the NFS volume to
- mountPath: "/mnt/datasets"
name: datasets-nfs
# NFS is exported read-only
readOnly: true
volumes:
# Volume which mounts the NFS server exported to the cluster by ccu-node1
- name: datasets-nfs
nfs:
server: ccu-node1
path: /raid/datasets
</syntaxhighlight>
Please see the page [[CCU:Global dataset storage|on global storage]] for a list of available datasets and the method to upload your own.
== Persistent volumes ==
On the cluster, there are two types of persistent volumes currently configured:
* Local persistent volumes
* Host directoriesGlobal persistent volumesLocal Note: the cluster will soon get large, fast global storage, at this point local persistent volumes will be phased out and probably not available anymore. Tensorboard monitoring should be used to import training data done using service exports, as explained below, and store results and log files not make use of your training. There are special local PVs for monitoring the training using Tensorboard. Host directories are meant for common training data sets stored permanently on the host. They are always read only.
=== Local persistent volumes ===
These are persistent volumes which are mapped to special folders of the host filesystem of the node. Each node exposes several persistent volumes which can be claimed. The user can not control exactly which volume is bound to a claim, but can request a minimum size. A persistent volume claim for a local PV is configured like this. Code examples can be found in the subdirectory "kubernetes/example_2" of the tutorial sample code, [[File:Kubernetes_samples.zip|Kubernetes samples]]. '''WARNING: Once a local persistent volume has been bound to a specific node, all pods which make use this volume are forced to also run on this node. This means you have to rely on resources (e.g. GPUs) being available on exactly that particular node.''' '''NOTE: The storage class "local-ssd" which was previously used for local persistent volumes is now obsolete, since a better driver with automatic provisioning has been installed. From now on, please use "local-path" instead, which will give you a PV on the fastest local device (usually SSD/NVMe RAID). No new volumes of class "local-ssd" can be claimed.''' Please copy over your data from old PVCs if you have the opportunity, or delete old PVCs not in use anymore. As soon as there are no more PVCs of the old class in use, it will be deleted from the cluster. Also, check out "global-datasets" below, which gives you a new opportunity to store large, static datasets on a very fast device.
<syntaxhighlight lang="yaml">
# the requested storage class, see tutorial.
storageClassName: local-ssdpath
# leave these unchanged, they must match the PV type, otherwise binding will fail
<syntaxhighlight lang="yaml">
> kubectl apply -f pvc.ymlyaml
</syntaxhighlight>
> kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
tf-mnist-pvc Pending local-ssd path 11s
</syntaxhighlight>
Since the claim has not been used by a container yet, it is not yet bound to a persitent volume (PV).
=== Host directories Global persistent volumes === In contrast, global persistent volumes are provided cluster-wide and are accessible from any node (managed internally with rook-ceph). They reside on SSDs and thus should be reasonably fast, however, depending on where the volume ends up, data will probably be transferred across the network to/from the node. Thus, they are slower than local-ssd, but leave you considerably more flexible, as they do not require pods to run on specific nodes. Also, there is no constraint on maximum size except for physical limitations. Currently, there is a total of 20 TB of cluster-wide SSD storage, which we plan to increase considerably in the near future.
<syntaxhighlight lang="yaml">
apiVersion: v1kind: PersistentVolumeClaimmetadata: # the name of the PVC, we refer to this in the container configuration name: tf-mnist-global-pvc spec: resources: requests: # storage resource request.This PVC can only be bound to volumes which # have at least 8 GiB of storage available. storage: 8Gi # the requested storage class, see tutorial. storageClassName: ceph-ssd # access mode is mandatory accessModes: - ReadWriteOnce - ReadOnlyMany # For me (Felix) it worked only with the additional following line: volumeMode: Filesystem
</syntaxhighlight>
Since anyone can mount global persistent volumes in the same namespace, they can and should be used to share datasets. The name of a PVC which contains a useful dataset should start with "dataset-" and be descriptive, so that it can easily be found by other users. Also, the root of the PVC should contain a README with informations about the dataset (at least the source and what exactly it is).
A note on mounting. Currently (will change in the near future), ceph volumes can be either mounted ReadWrite by a single pod only, or ReadOnly by multiple pods. Thus, the workflow for a static dataset is to create the PVC, then create a pod to write all the data to it, then delete this pod and mount it read only from now on so it can be used in multiple pods.
== Reading/writing the contents of a persistent volume ==
<syntaxhighlight lang="yaml">
# sudo apt-get update && sudo apt install openssh-clientrsync
# cd /my-pvc-mount-path
# scp your.username@external-server:/path/to/data/. ./
<syntaxhighlight lang="yaml">
# to get data into the container, substitute name with correct id obtained from kubectl get pods
> kubectl cp /path/to/data/. pvc-access-pod:/my-pvc-mount/path/data
# to get data from the container
> kubectl cp pvc-access-pod:/my-pvc-mount/path/. /path/to/output/
</syntaxhighlight>
Read up on Kubernetes "kubectl cp" documentation to check how it handles directories, it's a bit unusual and slightly different from scp.
Note: kubectl cp internally uses tar and some compression to speed up network transfer. However, this means that your access pod needs a certain amount of memory,
in particular when transferring large files. If you run into "error 137" (out of memory), increase memory limits of the access pod or use scp from within the pod.
