Tutorials:Monitoring with Tensorboard on the GPU cluster

Tensorboard support on the GPU cluster

Tensorboard is a monitoring tool for machine learning training, which provides a web browser interface on a port of the server (6116 in our cluster). Each compute node has its own instance of Tensorboard running, which is exposed on node-domain:6116. Tensorboard parses the content of a particular directory of the node. Subdirectories can be mounted as the persistent volume storage class "local-tensorboard" and used to write logs.

Local persistent volumes for Tensorboard logging

The following obtains a persistent volume claim for a local PV for data storage, as well as a PV for Tensorboard logging. Note that both can be done with a single config file. Code examples can be found in the subdirectory "kubernetes/example_3" of the tutorial sample code, File:Kubernetes samples.zip.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # the name of the PVC, we refer to this in the container configuration
  name: your-username-tf-mnist-pvc

spec:
  resources:
    requests:
      # storage resource request. This PVC can only be bound to volumes which
      # have at least 8 GiB of storage available.
      storage: 8Gi

  # the requested storage class here is fast data storage.
  storageClassName: local-ssd

  # leave these unchanged, they must match the PV type, otherwise binding will fail
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # the second claim is for tensorboard logging, it needs its own ID.
  name: your-username-tf-mnist-tb-pvc

spec:
  resources:
    requests:
      # Tensorboard logging typically requires not that much storage.
      storage: 2Gi

  # this storage class is parsed by the local Tensorboard instance
  # exposed to the network at port 6116.
  storageClassName: local-tensorboard

  # leave these unchanged, they must match the PV type, otherwise binding will fail
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem

Remember to prepend names with your username to make them unique. When the claim is defined to your satisfaction, apply it like this:

> kubectl apply -f pvc.yml

You can again check on the status of this (and every other) claim:

> kubectl get pvc
NAME              STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS        AGE
tf-mnist-pvc      Pending                                      local-ssd           11s
tf-mnist-tb-pvc   Pending                                      local-tensorboard   11s

Since the claim has not been used by a container yet, it is not yet bound to a persitent volume (PV).

Reading/writing the contents of a persistent volume

You can access a PV which is bound to a PVC by mounting it into a container. For a demonstration, we use the simple container image "ubuntu:18.04", which runs a minimalistic Ubuntu, and keep it in a very long wait after container startup.

# Test pod to mount a PV bound to a PVC into a container
# Before starting this pod, apply the PVC with kubectl apply -f pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: your-username-pvc-access-pod
spec:
  containers:
    - name: pvc-access-container

      # we use a small ubuntu base to access the PVC
      image: ubuntu:18.04
      # make sure that we have some time until the container quits by itself
      command: ['sleep', '6h']

      # list of mount paths within the container which will be
      # bound to persistent volumes.
      volumeMounts:
      - mountPath: "/mnt/pvc-mnist"
        # name of the volume for this path (from the below list)
        name: pvc-mnist

  volumes:
    # User-defined name of the persistent volume within this configuration.
    # This can be different from the name of the PVC.
    - name: pvc-mnist
      persistentVolumeClaim:
        # name of the PVC this volume binds to
        claimName: your-username-tf-mnist-pvc

After the PVC is applied, spin up the test pod with

> kubectl apply -f pvc-access-pod.yaml

You now have several options to get data to and from the container.

1. Copying data from within the container

You can get a root shell inside the container as usual (insert the correct pod name you used below):

> kubectl exec -it pvc-access-pod /bin/bash

Your pod has internet access. Thus, an option to get data to/from the pod, in particular into the persistent volume, is to use scp, which first needs to be installed inside the pod:

# apt-get update && apt install openssh-client rsync
# cd /my-pvc-mount-path
# scp your.username@external-server:/path/to/data/. ./

An even better variant would be "rsync -av" instead of scp, as this only copies files which are different or do not exist in the destination. By reversing source and destination, you can also copy data out of the container this way.

2. Copying data from the outside

From the outside world, you can directly copy data to and from the container using kubectl cp, which has a very similar syntax as scp:

# to get data into the container, substitute name with correct id obtained from kubectl get pods
> kubectl cp /path/to/data/. pvc-access-pod:/my-pvc-mount/path/data
# to get data from the container
> kubectl cp pvc-access-pod:/my-pvc-mount/path/. /path/to/output/

TODO: Will finish this part soon, for now, read up on Kubernetes "kubectl cp" documentation to copy stuff to/from a PV.