ccu, Administrators
684
edits
Changes
Jump to navigation
Jump to search
m
This file system is usually very fast, but only if it is used for workloads it is designed for. Remember that it is a distributed storage, this means that metadata access is over a database and can be a bottleneck. In effect, performance breaks down dramatically if writing or accessing many small files, or having many small files in a single directory (which forces metadata to be stored on a single server). '''TL;DR, and this is very important: when using CephFS, make sure to organize your dataset in few large files (e.g. HDF5), and not many small ones !''' If this is not possible for you, then you need to resort to persistent volumes residing on local storage on a single node, which for small files is orders of magnitude faster, but you are bound to a particular node (or have to duplicate the data in different local filesystems). A tutorial follows. <syntaxhighlight lang== Persistent volumes == A persistent volume in Kubernetes is a cluster resource which can be requested by a container. For this, you have to claim a persistent volume (PV) using a persistent volume claim (PVC), which you apply in your namespace. The persistent volume claim can then be mounted to directories within a container. The important point is that the PVC survives the end of the container, i.e. the data in the PV will be permanent until the PVC is released. If the PVC is mounted again to a new container, the data will still be present. A persistent volume which is bound to a claim can not be assigned to any other claim. '''If the PVC is released, the PV is also released and immediately and automatically wiped clean of all data'''. If you want to keep your data, copy it to some other permanent storage first. On the cluster, there are two types of persistent volumes currently configured:* Local persistent volumes* Global persistent volumes Note: the cluster will soon get large, fast global storage, at this point local persistent volumes will be phased out and probably not available anymore. Tensorboard monitoring should be done using service exports, as explained below, and not make use of local PVs. === Local persistent volumes === These are persistent volumes which are mapped to special folders of the host filesystem of the node. Each node exposes several persistent volumes which can be claimed. The user can not control exactly which volume is bound to a claim, but can request a minimum size. A persistent volume claim for a local PV is configured like this. Code examples can be found in the subdirectory "kubernetes/example_2bash" of the tutorial sample code, [[File:Kubernetes_samples.zip|Kubernetes samples]].> '''WARNING: Once a local persistent volume has been bound to a specific node, all pods which make use this volume are forced to also run on this node. This means you have to rely on resources (e.g. GPUs) being available on exactly that particular node.''' '''NOTE: The storage class "local-ssd" which was previously used for local persistent volumes is now obsolete, since a better driver with automatic provisioning has been installed. From now on, please use "local-path" instead, which will give you a PV on the fastest local device (usually SSD/NVMe RAID). No new volumes of class "local-ssd" can be claimed.''' Please copy over cephfs/abyss/home/<your data from old PVCs if you have the opportunity, or delete old PVCs not in use anymore. As soon as there are no more PVCs of the old class in use, it will be deleted from the cluster. Also, check out "global-datasets" below, which gives you a new opportunity to store large, static datasets on a very fast device. username></syntaxhighlight lang="yaml">apiVersion: v1kind: PersistentVolumeClaimmetadata: # the name of the PVC, we refer to this in the container configuration name: tf-mnist-pvc spec: resources: requests: # storage resource request. This PVC can only be bound to volumes which # have at least 8 GiB of storage available. storage: 8Gi # the requested storage class, see tutorial. storageClassName: local-path
# leave these unchangedThis file system is usually quite fast, they must match but only if it is used for workloads it is designed for. It is a distributed storage, where the filesystem metadata is stored in databases on different servers, and the actual content of the PV typefiles on other ones. This means that metadata access (such as reading file attributes, otherwise binding will fail accessModes: - ReadWriteOnce volumeMode: Filesystem</syntaxhighlightor on which server to look for a specific file) can be a bottleneck. In effect, the task of reading the metadata for a small file is orders of magnitude more expensive than reading the actual contents of the file itself. This means that performance breaks down dramatically if writing or accessing many small files. In particular, having many small files in a single directory (say >10k) makes any simple filesystem operations such as directory listings take ages, and in particular automated backup jobs might run into problems.
The following storage classes '''TL;DR, and this is very important: when using CephFS, make sure to organize your dataset in few large files (e.g. HDF5), and not many small ones ! If you really have to have individual files, then make sure they are configured stored in the cluster:subdirectories which do not become too large. '''
When the claim is defined to your satisfaction, apply it like this:== CephFS capacity and backup strategy ==
<syntaxhighlight lang="yaml">> kubectl apply The storage on the Ceph filesystem is quite expensive due to redundancy built in (if any server reboots or is otherwise unavailable, the others can still serve all of the data). The contents of the home directories are also backed up daily onto a backup server with a file history -f pvcif you ever accidentally overwrite or otherwise lose an extremely important file, you can contact me and check if I have an old copy in a backup.yaml</syntaxhighlight>
You can check Currently, there is sufficient space left, however, I kindly ask you to not keep data you do not use anymore on the status Ceph filesystem for too long. In particular, please delete old checkpoints of training runs you will never need again - I have seeen people use several Terabytes for their training histories. If you still need these, please move them onto your own computers. If you really want to keep old stuff lying around on the cluster filesystem, maybe because you are not sure whether you will need it again later on, then please put it into a folder which is not backed up. For this (and , every other) claim:user can mount the Ceph directory
Since which can be used as an archive. Make sure that the claim has directory is created if it does not been used exist, by a container yet, it is not yet bound to a persitent volume (PV)specifying "type: DirectoryOrCreate".
In contrast, global persistent volumes are provided cluster-wide and are accessible from any node (managed internally with rook-ceph). They reside on SSDs and thus should be reasonably fast, however, depending on where the volume ends up, data will probably be transferred across the network to/from the node. Thus, they are slower than The path for local-ssd, but leave you considerably more flexible, as they do not require pods to run on specific nodes. Also, there is no constraint on maximum size except storage for physical limitations. Currently, there each user is a total of 20 TB of cluster-wide SSD storage, which we plan to increase considerably in the near future.
Compared to creating local persistent volumes, the only thing which needs to be changed is the storage class to "ceph-ssd". <syntaxhighlight lang="yamlbash">apiVersion: v1kind: PersistentVolumeClaimmetadata: # the name of the PVC, we refer to this in the container configuration name: tf/raid/local-mnistdata/<your-global-pvc spec: resources: requests: # storage resource request. This PVC can only be bound to volumes which # have at least 8 GiB of storage available. storage: 8Gi # the requested storage class, see tutorial. storageClassName: ceph-ssd # access mode is mandatory accessModes: - ReadWriteOnce - ReadOnlyMany # For me (Felix) it worked only with the additional following line: volumeMode: Filesystemusername>
Since anyone can mount global The data will remain persistent volumes in on the same namespacehost, they can and should be used to share datasets. The name of a PVC which contains a useful dataset should start with "dataset-" and be descriptive, so but note that it can easily be found by other usersalso only exists on this particular host. AlsoIf you need to access it again, you need to make sure the root of pod always ends up on the PVC should contain same specific node. See example below. Otherwise, write your scripts in such a README with informations about way that they check for existence of the dataset (at least the source local data, and what exactly if it is)not there yet, copy it over from somewhere on the internet.
A note '''In contrast to Ceph storage, local paths on mounting. Currently (will change in the near future), ceph volumes can be either mounted ReadWrite by a single pod only, or ReadOnly by multiple podshosts are not backed up. Thus, the workflow for a static dataset is to create the PVC, then create a pod to write all the data to it, then delete this pod and mount it read only from now on so it can be used in multiple podsYou have been warned.'''
You can The following example creates an access a PV which is bound to a PVC by mounting it into a container. For a demonstration, we use pod on the simple container image compute node "ubuntu:18.04tiamat", which runs a minimalistic Ubuntu, and keep it mounts the local storage as well as all your personal directories in a very long wait after container startup.the ceph file system:
- name: pvc-access-container # we use a small ubuntu base to access the PVC image: ubuntu:1820.04 # make sure that we have some time until the container quits by itself command: ['"sleep'", '6h'"1d"] resources: # list of mount paths within the container which will berequests: cpu: 100m memory: 100Mi # bound to persistent volumes.limits: cpu: 1 memory: 1Gi volumeMounts: volumeMounts- mountPath: /abyss/home name: cephfs-home readOnly:false - mountPath: "/mntabyss/pvcshared name: cephfs-mnist"shared readOnly: false - mountPath: /abyss/datasets # name of the volume for this path (from the below list): cephfs-datasets readOnly: true - mountPath: /local name: pvclocal-mniststorage readOnly: false
# User-defined name of the persistent volume within this configuration.: cephfs-home hostPath: path: "/cephfs/abyss/home/<your-username>" type: Directory # This can be different from the - name of the PVC.: cephfs-shared hostPath: path: "/cephfs/abyss/shared" type: Directory - name: pvccephfs-mnistdatasets persistentVolumeClaimhostPath: # path: "/cephfs/abyss/datasets" type: Directory - name of the PVC this volume binds to: local-storage hostPath: claimNamepath: "/raid/local-data/<your-username-tf-mnist-pvc>" type: DirectoryOrCreate
After == Reading/writing to the PVC is applied, spin up directories in the test pod with <syntaxhighlight lang="yaml">> kubectl apply -f pvc-access-pod.yaml</syntaxhighlight>=
You now After you have created the access pod with "kubectl apply -f <filename>.yaml", you have several options to get data to and from the container.
→Local storage on the node
== The CephFS file system ==
As explained in the [[CCU:GPU Cluster Quick Start|quick start tutorial]], every user can mount certain local host paths inside their pods, which refer to a global distributed Ceph file system.Reminder, the primary home directory is
For example, if you have a million images of the form abcdef.jpg in a single directory, you better distribute them over a directory tree a/b/c/def.jpg, so that it is only 1000 files per directory.
An interesting option if you have a dataset consisting of many small files might be to keep it in a tar archive and mount that archive using [https://github.com/mxmlnkn/ratarmount ratarmount].
If this is not possible for you, then you need to use the local SSD storage on a single node, which for small files is orders of magnitude faster, but you are bound to a particular node (or have to duplicate the data in different local filesystems). See below for details on local filesystems.
<syntaxhighlight lang="yamlbash">/cephfs/abyss/archive/nobackup/<your-username> kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEtf-mnist-pvc Pending local-path 11s
</syntaxhighlight>
=== Global persistent volumes =Local storage on the node ==
</syntaxhighlight>
You can mount it as a hostPath, but have to make sure that the directory is created if it does not exist, by specifying "type: DirectoryOrCreate".
== Reading/writing the contents of a persistent volume Example ==
<syntaxhighlight lang="yamlbash"># Test pod to mount a PV bound to a PVC into a container# Before starting this pod, apply the PVC with kubectl apply -f pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: your-username-pvcstorage-access-pod-tiamat
spec:
nodeSelector:
kubernetes.io/hostname: tiamat
containers:
volumes:
</syntaxhighlight>
=== 1. Copying data from within the container ===
You can get a root shell inside the container as usual (insert the correct pod name you used below):
<syntaxhighlight lang="yaml">
> kubectl exec -it pvc-access-pod -- /bin/bash
</syntaxhighlight>
Your pod has internet access. Thus, an option to get data to/from the pod, in particular into the persistent volume, is to use scp, which first needs might need to be installed inside the pod:
<syntaxhighlight lang="yaml">
An even better variant would be "rsync -av" instead of scp, as this only copies files which are different or do not exist in the destination. By reversing source and destination, you can also copy data out of the container this way.
=== 2. Copying data from the outside your local machine ===
From the outside worldlocal machine which has kubectl access to the cluster, you can directly copy data to and from the container using kubectl cp, which has a very similar syntax as scp:
<syntaxhighlight lang="yaml">
