
I recently acquired an Nvidia Jetson Nano 2GB Developer Kit and had some fun with it! In this series I present the overview of the articles I created:

* [Setting up an Nvidia Jetson Nano Node](/post/infrastructure/nvidia-jetson-setup)
* [Creating your own Nvidia Jetson Cluster](/post/infrastructure/nvidia-jetson-cluster)
* [Deploying Kubernetes on your Nvidia Jetson Nano Cluster](/post/infrastructure/nvidia-jetson-k3s)
* [Adding Dapr and Grafana to the Nvidia Jetson Nano Kubernetes Cluster](/post/infrastructure/nvidia-jetson-k3s-dapr-and-grafana)


Learn how to configure the Jetson Nano, Deploy a Kubernetes server and finally monitor it with Grafana

An Nvidia Jetson Nano Series


Since Kubernetes has now been installed, we can continue to run a demonstration! For this demonstration we are going to deploy the following:

1. [Dapr](https://dapr.io) as our Distributed Application Runtime
2. Deploy an Hello World Example
3. [Grafana](https://grafana.com)

So let's get started!

## Installing Dapr

To install Dapr we can utilize its `install.sh` script which we execute:

```bash
wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
```

Whereafter we can run `dapr --version` to validate that it is installed:

```bash
# Output of dapr --version
CLI version: 1.2.0
Runtime version: n/a
```

Finally, we configure Dapr on our Kubernetes cluster by initializing it

```bash
dapr init -k
```

Which we again verify by running `dapr status -k` 

```bash
# Output of dapr status -k
NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE  CREATED
dapr-sentry            dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-operator          dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-dashboard         dapr-system  True     Running  1         0.6.0    33s  2021-05-30 11:27.57
dapr-sidecar-injector  dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-placement-server  dapr-system  True     Running  1         1.2.0    32s  2021-05-30 11:27.58
```

## Installing a Demo Application

kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

## Grafana Dashboard

Installing Grafana is a bit more complicated as it requires a store to gather our telemetry. For this we will be utilizing Prometheus.

### Install Prometheus

For configuring Prometheus, we utilize the commands below that will set it up with a persistent volume.

> 💡 To disable persistent volumes, append `--set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false` to the end of the `helm install` command.

```bash
# Create a dedicated namespace
kubectl create namespace dapr-monitoring

# Install Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring

# Verify that it is running
kubectl get pods -n dapr-monitoring -w
```

### Install Grafana

Since we have our store set-up, we can now install our dashboard

> 💡 We can again disable persistent volumes by appending `--set persistence.enabled=false` to the `helm install` command

```bash
# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n dapr-monitoring

# Verify that it is running
kubectl get pods -n dapr-monitoring -w
```

### Grafana Admin Password

Once Grafana is up and running, gather its admin password by running the command below

```bash
kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

### Access Grafana

Now Grafana is up and running, we can access it by port forwarding its port ot our [local system](/nvidia-jetson-k3s#Installing-and-Configuring-Kubectl).

```bash
# Forward Grafana from our cluster to localhost
kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring

# Open browser at:
# http://localhost:8080/
```

We can now login with:

* **Username:** `admin`
* **Password:** See Grafana Admin Password (`kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo`)

### Configure Grafana with Prometheus

Finally we need to set-up our Grafana Dashboard. Luckily for us, Dapr provides some [out-of-the-box dashboards](https://github.com/dapr/dapr/tree/master/grafana)

To start, we load the dashboard and add our Prometheus Server

![](./grafana.png)

![](./grafana-prometheus.png)

As a URL we fetch the Dapr Prometheus Server url by viewing the services and finding the name of the deployment. Which will become `<DEPLOYMENT_NAME>.<NAMESPACE>`

```bash
kubectl get svc -n dapr-monitoring
```

![](./grafana-prometheus-server.png)

In our case this becomes `http://dapr-prom-prometheus-server.dapr-monitoring`

![](./grafana-configure.png)

Now we can `Save & Test` and go towads the import screen, where we import the dashboard that we copied from ]Dapr its GitHub page.](https://raw.githubusercontent.com/dapr/dapr/master/grafana/grafana-system-services-dashboard.json)

![](./grafana-import.png)
![](./grafana-import-2.png)

If everything is done correctly, we can now see our monitoring

![](./grafana-final.png)

Learn how you can install Dapr and Grafana on your own Kubernetes Cluster powered by Nvidia Jetson Nanos

Installing Grafana on your Kubernetes Cluster


We now have [a beautiful cluster assembled and running](/post/infrastructure/nvidia-jetson-cluster), but no software running on it yet. So let's get started and provision Kubernetes on top of it so we can manage our nodes easier!

## Introducing K3s

For our Kubernetes setup we want a lightweight Kubernetes installation seeing the required Edge computation. For this, we can utilize [K3S](https://k3s.io/) which is a distribution built specifically to run on IoT & Edge devices.

> Personally, the installation feels similar to how we installed and configured Docker Swarm before. Install it everywhere and join nodes to the master - quick and easy! 

## Installing K3s

Installing K3S is super trivial! There is however an easier way if you want to automate all of it, by utilizing the CLI and adding just 1 parameter.

### Master Node Setup

To configure our Master Node, we execute the following command that will pass the secret we define:

```bash
curl -sfL https://get.k3s.io | sh -s server --cluster-init --token "XAVIER_HIS_SECRET_TOKEN" --write-kubeconfig-mode 644
```

> 💡 As you may have noticed, we are utilizing `--write-kubeconfig-mode 644` to allow our cluster to be imported in rancher later.

### Worker Node Setup

Finally, we can add our Worker (or Agent) nodes by running the following line on each of the nodes:

```bash
curl -sfL https://get.k3s.io | sh -s agent --server https://MASTER_NODE_IP_ADDRESS:6443 --token "XAVIER_HIS_SECRET_TOKEN"
```

### Verification

When we now execute `sudo k3s kubectl get nodes -w` on our master node, we will see our nodes appearing.

![](./k3s-node-setup.png)

## Installing and Configuring Kubectl

> ⚠ Execute this on your local machine! (WSL in this case)

### Installing Kubectl

Since we now have a Kubernetes cluster running, it is time to configure it to run on our local system. For that I am utilizing the `WSL` or `Windows Subsystem for Linux` system as explained in a [previous article](https://xaviergeerinck.com/post/coding/python/windows-development-environment-setup).

To install WSL we execute the following:

```bash
# Download Kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Install it
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Verify
kubectl version --client
```

### Configuring Kubectl

Now this is installed, we can configure it towards our remote cluster:

```bash
# Copy over the Kubeconfig file to our local machine
scp xavier@xavier-nvidia-node-1.local:/etc/rancher/k3s/k3s.yaml nvidia-node-1-config.yaml

# Replace localhost with the location of the master node
sed -ie s/127.0.0.1/xavier-nvidia-node-1.local/g nvidia-node-1-config.yaml

# Verify that it works
kubectl --kubeconfig nvidia-node-1-config.yaml get pods -A

# Then copy it over to kubectl directory and activate it
# we can now run kubectl get pods -A
cp nvidia-node-1-config.yaml ~/.kube/config
```

## Troubleshooting

### Agent does not want to join

Sometimes an agent does not want to join, in that case try utilizing the IP address directly.

### Removing K3S

⚠ In case something happened, you can uninstall K3S with the following commands:

```bash
# Uninstall Server Node
/usr/local/bin/k3s-uninstall.sh

# Uninstall Agent Node
/usr/local/bin/k3s-agent-uninstall.sh
```

## Conclusion

We have now set-up our Nvidia Jetson nodes and are running a Kubernetes cluster on it! To keep this article light, I have decided to move the demonstration to a new article where you will learn to [run Dapr as your distributed application runtime as well as Grafana to monitor your nodes](/post/infrastructure/nvidia-jetson-k3s-dapr-and-grafana)

Learn how you can install Kubernetes on a small GPU Cluster for edge computation purposes

Installing Kubernetes on an Nvidia Jetson Cluster


[In my previous article](/post/infrastructure/nvidia-jetson-setup) I explained how we can get one single Nvidia Jetson Nano 2GB Node up and running. In this article I would love to go in more detail on how we can create a cluster.

## Requirements

To create my cluster, I utilized the materials below:

> **Note:** These requirements were created for 3 Nodes and receive internet through a Mobile Pocket router that can be put in Client mode to connect LAN over WiFi (very useful for portability).

* 3x [Jetson Nano](https://developer.nvidia.com/embedded/jetson-nano-2gb-developer-kit) ~195 Eur Total
* 1x [Switch](https://eu.dlink.com/be/nl/products/dgs-105-5-port-gigabit-ethernet-switch) ~19.95 Eur
* 1x [Standoffs](https://www.amazon.nl/gp/product/B0778JRXPS) ~8.99 Eur
* 1x [Mobile Router](https://www.tp-link.com/nl/home-networking/wifi-router/tl-wr902ac/) (it is end of life, but super cheap and still working!) ~34.15 Eur
* 4x 0.5m UTP Cables (I had those, but typically ~ 4eur for 4)
* 4x [0.3m USB C to USB A cables](https://www.blitzwolf.com/BlitzWolf-BW-TC13-PVC-3A-USB-Type-C-Charging-Data-Cable-1ft0.3m-22-AWG-Power-+-30-AWG-Data-Wire-Cores-p-276.html) ~19.68 Eur
* 1x [6 port USB Hub](https://m.blitzwolf.com/60W-QC3.0-6-Ports-Charger-Adapter-p-450.html) ~18.87 Eur
* 4x [128GB SD Card](https://www.samsung.com/be/memory-storage/memory-card/evo-plus-microsd-card-with-sd-adapter-128gb-mb-mc128ga-eu/) ~58Eur Total (minimum should be 32GB)

**Total Bill:** 358.64 Eur (or 163.64 Eur without the Nvidia Jetsons)

## While Assembling

To make efficient use of our time, we can flash the SD Cards with the Operating System as explained [before](/post/infrastructure/nvidia-jetson-setup). They take around 20 minutes each.

## Assembling

While the SD Cards are flashing, we can get started with assembly! This is quite straightforward, so mount the nodes on top of the switch and connect them through the UTP cables. Then take the Mobile Pocket Router and connect that to the first port of the switch.

![](./node-standoffs.jpg)

![](./nodes-standoffs-stacked.jpg)

![](./switch.jpg)

![](./nodes-ethernet.jpg)

Once this is done, we can provide power by connecting the nodes and the mobile Router to our USB Hub. Finally, power on the switch and the USB Hub to get started!

![](./cluster-assembled.jpg)

## Connecting to your Nvidia Jetson Nanos

Once the Nvidia Jetson Nanos have been set-up, you can quite easily detect them by installing [Angry IP Scanner](https://angryip.org/download/#windows) and scanning your home network. This will show the nodes and their respective IP Addresses.

![](./ip-scan.png)

## Configuring Windows Terminal

As mentioned earlier, you can easily login on all the nodes through a shortcut and entering the command below:

```bash
CMD /C wt.exe wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-1.local"; split-pane -H wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-2.local"; split-pane -V wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-3.local"
```

## Conclusion

With a small bill we have built our own GPU Cluster that has a combined 384 Cores and 1.416 TFLOPS! (which is similar to 1 Jetson TX2)

![](./cluster-finalized.jpg)

Learn how you can create your own Nvidia Jetson Nano GPU Cluster