ccu
2
edits
Changes
Jump to navigation
Jump to search
no edit summary
== Viewing the Tensorboard of the job ==
=== Option 1: Using compute node global Tensorboard instance ===
As stated earlier, each compute node has its own instance of Tensorboard running. This instance will automatically display all Tensorboard summary files contained in persistent volumes with <code>storageClassName: local-tensorboard</code>.
First, find out the compute node your pod was allocated to.
This means that your logs will be the ones prefixed by "glasya-pv-tb-25gb-2" in the Tensorboard instance.
=== Option 2: Run your own Tensorboard instance ===
Another option is to create a pod running your own Tensorboard instance which is exposed via a Kubernetes service.
First create a pod running Tensorboard which is listening to your summary directory. In order to do so, we can simply use the [https://hub.docker.com/r/tensorflow/tensorflow/ latest Tensorflow container] from Docker Hub:<code>tensorflow/tensorflow:latest-py3</code>. The corresponding pod should look like this:
<syntaxhighlight lang="yaml">
apiVersion: v1
kind: Pod
metadata:
name: your-username-tb-pod
labels:
run: your-username-tb-0
spec:
containers:
- name: your-username-tb-container
image: tensorflow/tensorflow:latest-py3
# Execute Tensorboard in your mounted summaries folder. This will make the pod run indefinitely if no errors occur. Make sure to delete the pod if you do not use it anymore.
command: ["/bin/bash", "-C"]
args: ["cd /mnt/tensorboard/; tensorboard --logdir ."]
# Mount the persistent volume where you log Tensorboard summaries to
volumeMounts:
- mountPath: "/mnt/tensorboard"
name: your-username-tb
# Expose Tensorboard port, which is 6006 by default.
ports:
- containerPort: 6006
protocol: TCP
restartPolicy: Never
volumes:
- name: your-username-tb
persistentVolumeClaim:
claimName: your-username-tb-pvc
</syntaxhighlight>
Run the pod as usual. Next, we need to create a Kubernetes service mapping the Tensorboard pod IP and port to some fixed service IP and expose it publicly. This can be done using the <code>kubectl expose</code> command:
<syntaxhighlight lang="python">
kubectl expose pod *pod-name* --type=NodePort --name=*your-username-service-name*
</syntaxhighlight>
Replace *pod-name* with the name of the Tensorboard pod you just started and give the Service some name. You can check all running services with <code>kubectl get svc</code>. Your service should be in this list.
Kubernetes will automatically choose a port (NodePort) to expose, which we need to access Tensorboard. Get the NodePort with:
<syntaxhighlight lang="python">
kubectl describe svc *your-username-service-name*| grep NodePort
Type: NodePort:
NodePort: <unset> *NodePort*/TCP
</syntaxhighlight>
At last, find out the IP of the cluster node the Tensorboard pod is running on like described in Option 1. Then, your Tensorboard instance can be accessed via <code>*cluster-node-ip*:*service-node-port*</code>. For more general information on how to expose an Application running in a Kubernetes pod see [https://kubernetes.io/docs/tasks/access-application-cluster/service-access-application-cluster/ this].
[[Category:Tutorials]]
