ccu, Administrators
684
edits
Changes
Jump to navigation
Jump to search
m
To find the dataset, just tip
<syntaxhighlight lang="bash">
> kubectl get pvc
</syntaxhighlight>
and you will get the information like following:
<syntaxhighlight lang="bash">pvc name: dataset-kitti, capacity: 100Gi, access modes: RWO, storageclass:cephTBC-ssd </syntaxhighlight> To access to the dataset, you can get a root shell inside the container as usual (insert the correct pod name you used below): <syntaxhighlight lang="bash">> kubectl exec -it user-name-pvc-access-pod /bin/bash</syntaxhighlight> Then access into the dataset like following: <syntaxhighlight lang="bash">root@user-name-pvc-access-pod:/# cd mnt/'''nuScenes dataset-kitti/root@user-name-pvc-access-pod:/mnt/dataset-kitti# </syntaxhighlight>'''
If you want to bind your project to this global dataset, please change your '''job-script-pvc.yaml''' like following:
# list of mount paths within the container which will benuScenes Dataset # bound to persistent volumes. volumeMounts: # └── /tmpraid/data is where mnist.py will download the training data, unless it is # already there. This is the path where we mount the persistent volume. # Thus, for the second run of this container, the data will # not be downloaded again. - mountPath: "datasets/mntgeneral/dataset_kitti" # # name of the volume for this path (from the below list) name: dataset-kitti-user # login credentials to the docker registry. # for convenience, a readonly credential is provided as a secret in each namespace. imagePullSecrets: - name: registry-ro-login # containers will never restart restartPolicy: Never volumes: # User-defined name of the persistent volume within this configuration. # This can be different from the name of the PVC. - name: dataset-kitti-user persistentVolumeClaim: # name of the PVC this volume binds to claimName: dataset-kittinuscenes
TBC->'''nuScenes dataset'''
→List of datasets in global storage
== Overview ==
The global dataset storage is intended for large, static datasets, in particular those which benefit multiple users (but feel free to also use it for your own data which only you need). Write access is very slow since it is tunneled over a slow filesystem for security and backup reasons (see below for technical details), so it will take a while until your datasets actually show up on the cluster. Read access, however, should be very fast (the NVMe RAID where it resides has 1.9 GB/s read speed, it is accessed over a 10 GBit/s Network from nodes other than the DGX-2), and might in some cases even surpass local storage.
The global storage can be easily mounted in any container on any node as a read-only volume, while you have to write to it using certain rsync commands on the master node. See below for detailed instructions. Every user has their own subdirectory within the global storage (readable by everyone, writeable only by that user). In addition, there is a user-independent directory subtree with common machine learning datasets. If you believe you have a dataset in your own subdirectory which is static and beneficial for many users, please contact me to move it to the common tree.
== Writing your data to the global storage ==
The global storage is populated from the subdirectory "datasets/cluster" in your home directory on the CCU master node Lolth (ccu-master.inf.uni-konstanz.de, IP 134.34.224.84). If you try to login there with ssh, the shell is extremely limited for security reasons. However, you can run an rsync command to the server as follows. Assume the current directory contains a subdirectory "my_dataset" which you want to copy to the global storage on the cluster, then run
<syntaxhighlight lang="bash">
> rsync -avz --info=progress2 ./my_dataset your.username@ccu-master.inf.uni-konstanz.de:datasets/cluster/
</syntaxhighlight>
You now have a copy of your dataset on Lolth. What happens now is that roughly every hour, the datasets on Lolth are synced to the directory "/raid/datasets/your.username" on an NFS server.
This directory is exported and you can mount it into any container running on the cluster. Note that every user has read access to the whole directory tree, so you can use this method to share data between users as well.
As a side effect, you now also have two backups of your data on two different machines (however in the same rack, so not really fire-proof).
You can also delete data from Lolth by ssh'ing into the machine and using rm to delete stuff in the "datasets/cluster" subdirectory. During the hourly sync, data not present here will also be deleted from the global cluster storage.
== Accessing the global storage from within a container ==
* Benchmark: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d
* Source code on TensorFlow: http://tfds.object_detection.Kitti
* Source code on PyTorch: https://github.com/poodarchu/Det3D
* Dataset size: 85 GiB
* The dataset is totally downloaded and organized as follows
<syntaxhighlight lang="bash">
# KITTI Dataset
└── /mntraid/dataset_kittidatasets/general/kitti
├── training <-- 7481 train data
| ├── image_2 <-- for visualization
<syntaxhighlight lang="bash">
</syntaxhighlight>
