Changes

Jump to navigation Jump to search
Tensorboard support on the GPU cluster
== Tensorboard support on the GPU cluster ==
Tensorboard is a monitoring tool for machine learning training, which provides a web browser interface on a port of the server (6116 in our cluster). Each compute node has its own instance of Tensorboard running, which is thus exported exposed on node-addressdomain:6116. Tensorboard parses the content of a particular directory of the node, which . Subdirectories can be mounted as a certain the persistent volume storage class"local-tensorboard" and used to write logs.
TODO: following was copy/pasted === Local persistent volumes for reference, finish tutorial page.Tensorboard logging ===
A The following obtains a persistent volume in Kubernetes is a cluster resource which can be requested by a container. For this, you have to claim for a persistent volume (local PV) using a persistent volume claim (PVC), which you apply in your namespace. The persistent volume claim can then be mounted to directories within a container. The important point is that the PVC survives the end of the container, i.e. the for data in the PV will be permanent until the PVC is released. If the PVC is mounted again to a new containerstorage, the data will still be present. A persistent volume which is bound to as well as a claim can not be assigned to any other claim. '''If the PVC is released, the PV is also released and immediately and automatically wiped clean of all data'''. If you want to keep your data, copy it to some other permanent storage first. On the cluster, there are two types of persistent volumes currently configured:* Local persistent volumes* Host directoriesLocal persistent volumes should be used to import training data and store results and log files of your training. There are special PVs for monitoring the training using Tensorboardlogging. Host directories are meant for common training data sets stored permanently on the host. They are always read only.  === Local persistent volumes === These are persistent volumes which are mapped to special folders of the host filesystem of the node. Each node exposes several persistent volumes which Note that both can be claimed. The user can not control exactly which volume is bound to done with a claim, but can request a minimum size. A persistent volume claim for a local PV is configured like thissingle config file. Code examples can be found in the subdirectory "kubernetes/example_2example_3" of the tutorial sample code, [[File:Kubernetes_samples.zip|Kubernetes samples]].
<syntaxhighlight lang="yaml">
metadata:
# the name of the PVC, we refer to this in the container configuration
name: your-username-tf-mnist-pvc
spec:
storage: 8Gi
# the requested storage class, see tutorialhere is fast data storage.
storageClassName: local-ssd
- ReadWriteOnce
volumeMode: Filesystem
</syntaxhighlight>---apiVersion: v1kind: PersistentVolumeClaimmetadata: # the second claim is for tensorboard logging, it needs its own ID. name: your-username-tf-mnist-tb-pvc
The following spec: resources: requests: # Tensorboard logging typically requires not that much storage. storage classes are configured in the cluster:2Gi
# this storage class is parsed by the local Tensorboard instance
# exposed to the network at port 6116.
storageClassName: local-tensorboard
# leave these unchanged, they must match the PV type, otherwise binding will fail
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
</syntaxhighlight>
 Remember to prepend names with your username to make them unique. When the claim is defined to your satisfaction, apply it like this:
<syntaxhighlight lang="yaml">
</syntaxhighlight>
You can again check on the status of this (and every other) claim:
<syntaxhighlight lang="yaml">
> kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEtf-mnist-pvc Pending local-ssd 11stf-mnist-tb-pvc Pending local-ssd tensorboard 11s
</syntaxhighlight>
Since the claim has not been used by a container yet, it is not yet bound to a persitent volume (PV).
 
=== Host directories ===
 
Large training data sets which are required by many different users are stored permanently in the filesystem of several nodes. They can be claimed with a PVC as follows:
 
<syntaxhighlight lang="yaml">
...
</syntaxhighlight>
 
 
== Reading/writing the contents of a persistent volume ==

Navigation menu