ccu, Administrators
684
edits
Changes
Jump to navigation
Jump to search
Created page with "== Tensorboard support on the GPU cluster == Tensorboard is a monitoring tool for machine learning training, which provides a web browser interface on a port of the server (6..."
== Tensorboard support on the GPU cluster ==
Tensorboard is a monitoring tool for machine learning training, which provides a web browser interface on a port of the server (6116 in our cluster). Each compute node has its own instance of Tensorboard running, which is thus exported on node-address:6116. Tensorboard parses the content of a particular directory of the node, which can be mounted as a certain persistent volume storage class.
TODO: following was copy/pasted for reference, finish tutorial page.
A persistent volume in Kubernetes is a cluster resource which can be requested by a container. For this, you have to claim a persistent volume (PV) using a persistent volume claim (PVC), which you apply in your namespace. The persistent volume claim can then be mounted to directories within a container. The important point is that the PVC survives the end of the container, i.e. the data in the PV will be permanent until the PVC is released. If the PVC is mounted again to a new container, the data will still be present. A persistent volume which is bound to a claim can not be assigned to any other claim. '''If the PVC is released, the PV is also released and immediately and automatically wiped clean of all data'''. If you want to keep your data, copy it to some other permanent storage first.
On the cluster, there are two types of persistent volumes currently configured:
* Local persistent volumes
* Host directories
Local persistent volumes should be used to import training data and store results and log files of your training. There are special PVs for monitoring the training using Tensorboard. Host directories are meant for common training data sets stored permanently on the host. They are always read only.
=== Local persistent volumes ===
These are persistent volumes which are mapped to special folders of the host filesystem of the node. Each node exposes several persistent volumes which can be claimed. The user can not control exactly which volume is bound to a claim, but can request a minimum size. A persistent volume claim for a local PV is configured like this. Code examples can be found in the subdirectory "kubernetes/example_2" of the tutorial sample code, [[File:Kubernetes_samples.zip|Kubernetes samples]].
<syntaxhighlight lang="yaml">
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
# the name of the PVC, we refer to this in the container configuration
name: tf-mnist-pvc
spec:
resources:
requests:
# storage resource request. This PVC can only be bound to volumes which
# have at least 8 GiB of storage available.
storage: 8Gi
# the requested storage class, see tutorial.
storageClassName: local-ssd
# leave these unchanged, they must match the PV type, otherwise binding will fail
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
</syntaxhighlight>
The following storage classes are configured in the cluster:
When the claim is defined to your satisfaction, apply it like this:
<syntaxhighlight lang="yaml">
> kubectl apply -f pvc.yml
</syntaxhighlight>
You can check on the status of this (and every other) claim:
<syntaxhighlight lang="yaml">
> kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
tf-mnist-pvc Pending local-ssd 11s
</syntaxhighlight>
Since the claim has not been used by a container yet, it is not yet bound to a persitent volume (PV).
=== Host directories ===
Large training data sets which are required by many different users are stored permanently in the filesystem of several nodes. They can be claimed with a PVC as follows:
<syntaxhighlight lang="yaml">
...
</syntaxhighlight>
== Reading/writing the contents of a persistent volume ==
You can access a PV which is bound to a PVC by mounting it into a container. For a demonstration, we use the simple container image "ubuntu:18.04", which runs a minimalistic Ubuntu, and keep it in a very long wait after container startup.
<syntaxhighlight lang="yaml">
# Test pod to mount a PV bound to a PVC into a container
# Before starting this pod, apply the PVC with kubectl apply -f pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: your-username-pvc-access-pod
spec:
containers:
- name: pvc-access-container
# we use a small ubuntu base to access the PVC
image: ubuntu:18.04
# make sure that we have some time until the container quits by itself
command: ['sleep', '6h']
# list of mount paths within the container which will be
# bound to persistent volumes.
volumeMounts:
- mountPath: "/mnt/pvc-mnist"
# name of the volume for this path (from the below list)
name: pvc-mnist
volumes:
# User-defined name of the persistent volume within this configuration.
# This can be different from the name of the PVC.
- name: pvc-mnist
persistentVolumeClaim:
# name of the PVC this volume binds to
claimName: your-username-tf-mnist-pvc
</syntaxhighlight>
After the PVC is applied, spin up the test pod with
<syntaxhighlight lang="yaml">
> kubectl apply -f pvc-access-pod.yaml
</syntaxhighlight>
You now have several options to get data to and from the container.
=== 1. Copying data from within the container ===
You can get a root shell inside the container as usual (insert the correct pod name you used below):
<syntaxhighlight lang="yaml">
> kubectl exec -it pvc-access-pod /bin/bash
</syntaxhighlight>
Your pod has internet access. Thus, an option to get data to/from the pod, in particular into the persistent volume, is to use scp, which first needs to be installed inside the pod:
<syntaxhighlight lang="yaml">
# apt-get update && apt install openssh-client rsync
# cd /my-pvc-mount-path
# scp your.username@external-server:/path/to/data/. ./
</syntaxhighlight>
An even better variant would be "rsync -av" instead of scp, as this only copies files which are different or do not exist in the destination. By reversing source and destination, you can also copy data out of the container this way.
=== 2. Copying data from the outside ===
From the outside world, you can directly copy data to and from the container using kubectl cp, which has a very similar syntax as scp:
<syntaxhighlight lang="yaml">
# to get data into the container, substitute name with correct id obtained from kubectl get pods
> kubectl cp /path/to/data/. pvc-access-pod:/my-pvc-mount/path/data
# to get data from the container
> kubectl cp pvc-access-pod:/my-pvc-mount/path/. /path/to/output/
</syntaxhighlight>
TODO: Will finish this part soon, for now, read up on Kubernetes "kubectl cp" documentation to copy stuff to/from a PV.
[[Category:Tutorials]]
Tensorboard is a monitoring tool for machine learning training, which provides a web browser interface on a port of the server (6116 in our cluster). Each compute node has its own instance of Tensorboard running, which is thus exported on node-address:6116. Tensorboard parses the content of a particular directory of the node, which can be mounted as a certain persistent volume storage class.
TODO: following was copy/pasted for reference, finish tutorial page.
A persistent volume in Kubernetes is a cluster resource which can be requested by a container. For this, you have to claim a persistent volume (PV) using a persistent volume claim (PVC), which you apply in your namespace. The persistent volume claim can then be mounted to directories within a container. The important point is that the PVC survives the end of the container, i.e. the data in the PV will be permanent until the PVC is released. If the PVC is mounted again to a new container, the data will still be present. A persistent volume which is bound to a claim can not be assigned to any other claim. '''If the PVC is released, the PV is also released and immediately and automatically wiped clean of all data'''. If you want to keep your data, copy it to some other permanent storage first.
On the cluster, there are two types of persistent volumes currently configured:
* Local persistent volumes
* Host directories
Local persistent volumes should be used to import training data and store results and log files of your training. There are special PVs for monitoring the training using Tensorboard. Host directories are meant for common training data sets stored permanently on the host. They are always read only.
=== Local persistent volumes ===
These are persistent volumes which are mapped to special folders of the host filesystem of the node. Each node exposes several persistent volumes which can be claimed. The user can not control exactly which volume is bound to a claim, but can request a minimum size. A persistent volume claim for a local PV is configured like this. Code examples can be found in the subdirectory "kubernetes/example_2" of the tutorial sample code, [[File:Kubernetes_samples.zip|Kubernetes samples]].
<syntaxhighlight lang="yaml">
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
# the name of the PVC, we refer to this in the container configuration
name: tf-mnist-pvc
spec:
resources:
requests:
# storage resource request. This PVC can only be bound to volumes which
# have at least 8 GiB of storage available.
storage: 8Gi
# the requested storage class, see tutorial.
storageClassName: local-ssd
# leave these unchanged, they must match the PV type, otherwise binding will fail
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
</syntaxhighlight>
The following storage classes are configured in the cluster:
When the claim is defined to your satisfaction, apply it like this:
<syntaxhighlight lang="yaml">
> kubectl apply -f pvc.yml
</syntaxhighlight>
You can check on the status of this (and every other) claim:
<syntaxhighlight lang="yaml">
> kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
tf-mnist-pvc Pending local-ssd 11s
</syntaxhighlight>
Since the claim has not been used by a container yet, it is not yet bound to a persitent volume (PV).
=== Host directories ===
Large training data sets which are required by many different users are stored permanently in the filesystem of several nodes. They can be claimed with a PVC as follows:
<syntaxhighlight lang="yaml">
...
</syntaxhighlight>
== Reading/writing the contents of a persistent volume ==
You can access a PV which is bound to a PVC by mounting it into a container. For a demonstration, we use the simple container image "ubuntu:18.04", which runs a minimalistic Ubuntu, and keep it in a very long wait after container startup.
<syntaxhighlight lang="yaml">
# Test pod to mount a PV bound to a PVC into a container
# Before starting this pod, apply the PVC with kubectl apply -f pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: your-username-pvc-access-pod
spec:
containers:
- name: pvc-access-container
# we use a small ubuntu base to access the PVC
image: ubuntu:18.04
# make sure that we have some time until the container quits by itself
command: ['sleep', '6h']
# list of mount paths within the container which will be
# bound to persistent volumes.
volumeMounts:
- mountPath: "/mnt/pvc-mnist"
# name of the volume for this path (from the below list)
name: pvc-mnist
volumes:
# User-defined name of the persistent volume within this configuration.
# This can be different from the name of the PVC.
- name: pvc-mnist
persistentVolumeClaim:
# name of the PVC this volume binds to
claimName: your-username-tf-mnist-pvc
</syntaxhighlight>
After the PVC is applied, spin up the test pod with
<syntaxhighlight lang="yaml">
> kubectl apply -f pvc-access-pod.yaml
</syntaxhighlight>
You now have several options to get data to and from the container.
=== 1. Copying data from within the container ===
You can get a root shell inside the container as usual (insert the correct pod name you used below):
<syntaxhighlight lang="yaml">
> kubectl exec -it pvc-access-pod /bin/bash
</syntaxhighlight>
Your pod has internet access. Thus, an option to get data to/from the pod, in particular into the persistent volume, is to use scp, which first needs to be installed inside the pod:
<syntaxhighlight lang="yaml">
# apt-get update && apt install openssh-client rsync
# cd /my-pvc-mount-path
# scp your.username@external-server:/path/to/data/. ./
</syntaxhighlight>
An even better variant would be "rsync -av" instead of scp, as this only copies files which are different or do not exist in the destination. By reversing source and destination, you can also copy data out of the container this way.
=== 2. Copying data from the outside ===
From the outside world, you can directly copy data to and from the container using kubectl cp, which has a very similar syntax as scp:
<syntaxhighlight lang="yaml">
# to get data into the container, substitute name with correct id obtained from kubectl get pods
> kubectl cp /path/to/data/. pvc-access-pod:/my-pvc-mount/path/data
# to get data from the container
> kubectl cp pvc-access-pod:/my-pvc-mount/path/. /path/to/output/
</syntaxhighlight>
TODO: Will finish this part soon, for now, read up on Kubernetes "kubectl cp" documentation to copy stuff to/from a PV.
[[Category:Tutorials]]
