ccu, Administrators
684
edits
Changes
Jump to navigation
Jump to search
m
Set up according to [https://github.com/krishnapmv/k8s-ldap this tutorial]<syntaxhighlight lang="bash">> kubectl get pvwith customized install scripts in kubernetes/init/dex</syntaxhighlight>
→Kubernetes and pre-requisites (every node)
== Kubernetes and pre-requisites (every node) ==
Install Kubernetes on Ubuntu 18.04. Assuming version 1.14.3 is pulled, check how to fix version. On new systems, copy over the install script from the master node.
<syntaxhighlight lang="bash">
> cd init
> ./install_kubernetes.sh
</syntaxhighlight>
Reconfigure docker runtime. Edit /etc/docker/daemon.json as follows:
<syntaxhighlight lang="bash">
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
</syntaxhighlight>
On nodes with an nVidia GPU, add the following:
<syntaxhighlight lang="bash">
"default-runtime": "nvidia",
"default-shm-size": "1g",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
</syntaxhighlight>
Restart docker daemon:
<syntaxhighlight lang="bash">
> mkdir -p /etc/systemd/system/docker.service.d
> systemctl daemon-reload
> systemctl restart docker
</syntaxhighlight>
Make sure swap is off
<syntaxhighlight lang="bash">
> sudo swapoff -a
</syntaxhighlight>
Check /etc/fstab if swap is still configured there, delete if this is the case.
== Spin up the master node ==
Use kubeadm with vanilla defaults to initialize the control plane.
<syntaxhighlight lang="bash">
> sudo systemctl enable docker.service
> sudo kubeadm init
</syntaxhighlight>
If this fails at any point, use kubeadm reset after problems have been fixed before trying to re-initialize.
* Post-init steps to setup admin user on this account
<syntaxhighlight lang="bash">
> cd init
> ./finalize_master.sh
</syntaxhighlight>
== Update kubelet configuration for master node ==
Edit /etc/kubernetes/manifests/kube-controller-manager.yaml:
<syntaxhighlight lang="bash">
spec:
containers:
- command:
# add these two
- --allocate-node-cidrs=true
- --cluster-cidr=10.244.0.0/16
</syntaxhighlight>
Copy certs/ca.crt (certificate for ccu.uni-konstanz.de) to /usr/share/ca-certificates/ca-dex.pem.
Edit /etc/kubernetes/manifests/kube-apiserver.yaml:
<syntaxhighlight lang="bash">
spec:
containers:
- command:
# add these five
- --oidc-issuer-url=https://ccu.uni-konstanz.de:32000/dex
- --oidc-client-id=loginapp
- --oidc-ca-file=/usr/share/ca-certificates/ca-dex.pem
- --oidc-username-claim=name
- --oidc-groups-claim=groups
</syntaxhighlight>
== Daemonsets on Master node ==
=== Flannel daemonset (pod network for communication) ===
<syntaxhighlight lang="bash">
> cd init
> ./start_pod_network.sh
</syntaxhighlight>
=== nVidia daemonset ===
<syntaxhighlight lang="bash">
> cd init
> ./deploy_nvidia_device_plugin.sh
</syntaxhighlight>
The daemonset should be active on any node with an nVidia GPU.
== Authentication systems ==
The master node should now login to the docker registry of the cluster. <syntaxhighlight lang="bash">> docker login https://ccu.uni-konstanz.de:5000Username: bastian.goldlueckePassword:</syntaxhighlight> Also, we need to provide the read-only secret for the docker registry in every namespace. TODO: howto. Finally, we need to set up all the rules for rbac. <syntaxhighlight lang="bash">> cd rbac# generate namespaces for user groups> ./generate_namespaces.sh# label all compute nodes for which namespace they serve# (after they are up, needs to be redone when new nodes are added)> ./label_nodes.sh# set up access rights for namespaces> kubectl apply -f rbac.yaml# set up rights for which namespaces can access which compute node> kubectl apply -f node_to_groups.yaml</syntaxhighlight> = DEX with LDAP =Persistent volumes == === Local persistent volumes === Check directory local_storage:* clone the git repository for the provisioner using clone_provisioner.sh (delete first if already here).* install helm: install_helm.sh, get_helm.sh. Do NOT run helm init (unsafe and soon obsolete).* set up and run provisioner: <syntaxhighlight lang="bash">> cd install> generate_config.sh> kubectl apply -f install_storageclass.yaml> kubectl apply -f install_service.yaml> kubectl apply -f provisioner_generated.yaml</syntaxhighlight> After local persistent volumes on the nodes have been generated in /mnt/kubernetes, they should show up under
