
Recently I've been working a lot with different AI Models for a variety of use cases, ranging from simple object classification to massive crowd counting models. While working with these models, I saw that it is always difficult to implement a model in custom code since the Architecture is never included in the weights themselves, thus each model works differently.

That's when I came across a format named [ONNX](https://onnx.ai/) which instantly peaked my interest with promising points such as:
* Improved Inference Performance 
* Interoperability

## Prerequisites

To get started with ONNX, we have two tasks that should be done

1. Configure the [ONNX Execution Provider](https://onnxruntime.ai/docs/execution-providers/)
2. Get an ONNX Model to infer

### Configuring the ONNX Execution Provider for CUDA 

To be able to run on our GPU I chose to utilize the [CUDA Execution Provider](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html). Which has 2 major components that should be installed:

* CUDA
* cuDNN

#### Installing CUDA

CUDA is the easiest of the two to install since it is publically available without a login wall. Before we start, check if CUDA is already installed by running the below, which should finish correctly:

```bash
# Check if CUDA is available
nvidia-smi
nvcc --version
```

If the above is not available, we can install it (**on WSL 2**) by running the below:

> If you want to install it on another target, feel free to check the original [link](https://developer.nvidia.com/cuda-toolkit-archive)

```bash
# View all: https://developer.nvidia.com/cuda-toolkit-archive
# View ONNX Supported: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0

# Install CUDA
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo apt-key add /var/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

# Add CUDA to Path
# https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
# Note: add those to ~/.bashrc at the bottom
export PATH=/usr/local/cuda-11.5/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

> Note: Do not forget to add the export links to your ~/.bashrc file

#### Installing cuDNN

Now up to the other requirement, to install `cuDNN`.

```bash
OS="wsl-ubuntu"
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0
CUDNN_VERSION_MAJOR=8
CUDNN_VERSION_MINOR=2
CUDNN_VERSION_PATCH=4

wget https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/cuda-${OS}.pin 

sudo mv cuda-${OS}.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/ /"
sudo apt-get update
sudo apt-get install libcudnn8=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
sudo apt-get install libcudnn8-dev=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
```

## Running the YOLOX Model

Now we have all the required software installed for the CUDA Execution Provider, we can start running a model and infer from it. Luckily for us, the [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) team provides a [ready-to-use ONNX model](https://github.com/Megvii-BaseDetection/YOLOX/tree/main/demo/ONNXRuntime) that we can use. So let's download it and use it!

To keep the code part short, you can find the entire project on [my public repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/AI/ONNX/yolox) the main part to know is how we can infer an ONNX Model.

Infering an ONNX Model is as easy as creating a session that defines the model path as well as the provider to use

```python
# Create an ONNX Inference Session
session = onnxruntime.InferenceSession(
    path_or_bytes=args.model,
    providers=["CUDAExecutionProvider"]
)
```

After that is done, we just need to provide it an input:

```python
ort_inputs = {session.get_inputs()[0].name: img[None, :, :, :]}
output = session.run(None, ort_inputs)
```

Whereafter we can run all of this and receive our result!

```bash
# Create directories
mkdir model; mkdir images; mkdir output

# Download Model
wget -O ./model/yolox_l.onnx https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_l.onnx 

# Download Example
wget -O ./images/example.jpg https://www.ikea.com/ext/ingkadam/m/7f8f282fb240f466/original/PH179208-crop001.jpg?f=m

# Run
python src/main.py -m ./model/yolox_l.onnx -i ./images/example.jpg -o ./output -s 0.3 --input_shape 640,640
```

## Conclusion

As you can see from the above, it is quite easy to infer an ONNX Model and get results from it! I personally think I will utilize ONNX more in the future, reducing the complexity of my AI pipeline codes even further.

Learn how to run AI inferencing using ONNX on your GPU

AI Model Inferencing with ONNX on GPU (CUDA)


For a use case I wanted to utilize the Nvidia Jetson for edge inference. One of the bottlenecks here was that my software required a Python version that is greater than 3.6. When looking at the [Nvidia Jetson packages for PyTorch](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch) it was seen that this was only created for version 3.6.

Searching around, I could find quite a lot of people struggling with this (@TODO: PROOF) which made me to look into **a solution** that would help me automate the building of ARM Wheel of **PyTorch that can run on Nvidia Devices** (e.g. Nvidia Jetson Nano) and thus **support CUDA**.

The entire process above took me around ~11 full days, starting of with figuring out how to build the Dockerfile and finally automating the CI process.

> Note: I also decided to utilize this article as an entry for a running Hackathon by [Dev.to for GitHub Actions](https://dev.to/devteam/join-us-for-the-2021-github-actions-hackathon-on-dev-4hn4)

You would wonder why we want to automate this? Well when compiling this initially on my Nvidia Jetson Nano, I couldn't get it to compile past ~80% when using 16Gb of Swap Space (the device only has 2 Gb). After building it on my personal PC, this took ~6h to get it to compile completely. Making it large and long enough to automate it.

> The final source code can be found on [GitHub](https://github.com/XavierGeerinck/Jetson-Linux-PyTorch)

## Contributions

In any kind of project of this size, there are specific contributions that were made. In my project I believe to have made the following:

* Install CUDA on non-GPU devices
* Compile PyTorch with CUDA enabled on non-GPU devices
* Compile PyTorch for Python > 3.6
* Build for ARM with CI through Docker Buildx

## Project Outline

As a best practice, I always love to include the project outline of how I tackled the issue above (to share my thought process):

1. Create a Dockerfile that builds the Wheel
    * How do I build for CUDA? (Hardest part)
    * How do I cross build?
2. Create GitHub Action
    * How do I cross build? Can I run for ARM specifically?

## Dockerfile Creation - Building PyTorch for ARM and Python > 3.6

The hardest part BY FAR is how to compile PyTorch for ARM and Python > 3.6 with CUDA enabled. Because we are going to run on a non-GPU device, thus CUDA is not available on there. I splitted up the Dockerfile into 3 specific sections that can be ran in parallel:

1. Set up CUDA
2. Set up PyTorch (the cloning takes a while)
3. Set up Python 3.9
4. Compile PyTorch
    * I have Nvidia Jetson optimisations included here, thanks [QEngineering](https://qengineering.eu/install-pytorch-on-jetson-nano.html)!
4. Dockerfile Result

Below you can find the explanation of the respective steps. For the final `Dockerfile` optimizations where made to decrease the docker layer sizes (by grouping RUN in 1 command)

### Setting up CUDA

For CUDA, we do not have the CUDA libraries, nor do we have access to them! There is however a trick that allows us to get CUDA loaded in. We copy over the public key of NVIDIA Jetson and we authorize ourselves to the repository. Then we use our package manager to install them:

```bash
V_CUDA_DASH=10-2

# Add the public key
echo "[Builder] Adding the Jetson Public Key"
curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc
echo "deb https://repo.download.nvidia.com/jetson/common ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
echo "deb https://repo.download.nvidia.com/jetson/t186 ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

# Install the CUDA Libraries
echo "[Builder] Installing CUDA System"
apt-get update
apt-get install -y --no-install-recommends \
    cuda-libraries-$V_CUDA_DASH \
    cuda-libraries-dev-$V_CUDA_DASH \
    cuda-nvtx-$V_CUDA_DASH \
    cuda-minimal-build-$V_CUDA_DASH \
    cuda-license-$V_CUDA_DASH \
    cuda-command-line-tools-$V_CUDA_DASH \
    libnvvpi1 vpi1-dev

# Link CUDA to /usr/local/cuda
ln -s /usr/local/cuda-$CUDA /usr/local/cuda
```

When we eventually start compiling, we will see CUDA enabled in our CMAKE output 🥳

```bash
#   USE_CUDA              : ON
#     Split CUDA          : OFF
#     CUDA static link    : OFF
#     USE_CUDNN           : OFF
#     USE_EXPERIMENTAL_CUDNN_V8_API: OFF
#     CUDA version        : 10.2
#     CUDA root directory : /usr/local/cuda
#     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
#     cudart library      : /usr/local/cuda/lib64/libcudart.so
#     cublas library      : /usr/local/cuda/lib64/libcublas.so
#     cufft library       : /usr/local/cuda/lib64/libcufft.so
#     curand library      : /usr/local/cuda/lib64/libcurand.so
#     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
#     CUDA include path   : /usr/local/cuda/include
#     NVCC executable     : /usr/local/cuda/bin/nvcc
#     NVCC flags          : <CUT>
#     CUDA host compiler  : /usr/bin/clang
#     NVCC --device-c     : OFF
#     USE_TENSORRT        : OFF
```

### Setting up PyTorch

In a separate docker step, we set up PyTorch and clone it to the working repository (in our case `/build/pytorch`)

```bash
V_PYTORCH=v1.10.0

# Downloads PyTorch to /build/pytorch
git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch /build/pytorch
```

### Setting up Python 3.9

We configure our Python version through the `deadsnakes` ppa and link it as the default one.

> Best practice we should have a `venv` but since I am running it in a Docker container this should suffice.

```bash
# Setting up Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python3 \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python
```

### Compiling PyTorch

The last step in the Dockerfile is to compile PyTorch. For this we set the correct environment variabels to enable CUDA and to optimise the building process by disabling some other parts (e.g. MKLDNN, NNPACK, XNNPACK, ... to be turned off).

We also configure it to use `clang` as the Nvidia Jetson has [NEON registers](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_462) and clang supports those (GCC doesn't).

For our source, we utilize the other layer we created and just copy it from there.

```dockerfile
COPY --from=downloader-pytorch /build/pytorch /build/pytorch
WORKDIR /build/pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Build
rm build/CMakeCache.txt || : \
sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
pip install -r requirements.txt
python setup.py bdist_wheel
```

### Copying the result as an Artifact

Docker Buildx is amazing in the sense that we can utilize the `--output type-local,dest=.` command to output files to our local filesystem, making it such that docker builds and we can export the build result as an artifact.

To achieve this, we pull from the `scratch` image and copy over our result to it from the other docker layer. Our `/` path will then contain all the build wheels of PyTorch (e.g. `torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl`)

```dockerfile
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

### Dockerfile Result

Finally, the full Dockerfile will look like this:

```dockerfile
# ##################################################################################
# Setup Nvidia CUDA for Jetson
# ##################################################################################
FROM ubuntu:18.04 as cuda-devel

# Configuration Arguments
ARG V_CUDA_MAJOR=10
ARG V_CUDA_MINOR=2
ARG V_L4T_MAJOR=32
ARG V_L4T_MINOR=6

ENV V_CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}
ENV V_CUDA_DASH=${V_CUDA_MAJOR}-${V_CUDA_MINOR}
ENV V_L4T=r${V_L4T_MAJOR}.${V_L4T_MINOR}

# Expose environment variables everywhere
ENV CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Fix CUDA info
ARG DPKG_STATUS

# Add NVIDIA repo/public key and install VPI libraries
RUN echo "$DPKG_STATUS" >> /var/lib/dpkg/status \
    && echo "[Builder] Installing Prerequisites" \
    && apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates software-properties-common curl gnupg2 apt-utils \
    && echo "[Builder] Installing CUDA Repository" \
    && curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc \
    && echo "deb https://repo.download.nvidia.com/jetson/common ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "deb https://repo.download.nvidia.com/jetson/t186 ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "[Builder] Installing CUDA System" \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
    cuda-libraries-${V_CUDA_DASH} \
    cuda-libraries-dev-${V_CUDA_DASH} \
    cuda-nvtx-${V_CUDA_DASH} \
    cuda-minimal-build-${V_CUDA_DASH} \
    cuda-license-${V_CUDA_DASH} \
    cuda-command-line-tools-${V_CUDA_DASH} \
    libnvvpi1 vpi1-dev \
    && ln -s /usr/local/cuda-${V_CUDA} /usr/local/cuda \
    && rm -rf /var/lib/apt/lists/*

# Update environment
ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs

RUN ln -fs /usr/share/zoneinfo/Europe/Brussels /etc/localtime

# ##################################################################################
# Create PyTorch Docker Layer
# We do this seperately since else we need to keep rebuilding
# ##################################################################################
FROM --platform=$BUILDPLATFORM ubuntu:18.04 as downloader-pytorch

# Configuration Arguments
# https://github.com/pytorch/pytorch
ARG V_PYTORCH=v1.10.0
# https://github.com/pytorch/vision
ARG V_PYTORCHVISION=v0.11.1
# https://github.com/pytorch/audio
ARG V_PYTORCHAUDIO=v0.10.0

# Install Git Tools
RUN apt-get update \
    && apt-get install -y --no-install-recommends software-properties-common apt-utils git \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Clone Source
RUN git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch

# ##################################################################################
# Build PyTorch for Jetson (with CUDA)
# ##################################################################################
FROM cuda-devel as builder

# Configuration Arguments
ARG V_PYTHON_MAJOR=3
ARG V_PYTHON_MINOR=9

ENV V_PYTHON=${V_PYTHON_MAJOR}.${V_PYTHON_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Download Common Software
RUN apt-get update \
    && apt-get install -y clang build-essential bash ca-certificates git wget cmake curl software-properties-common ffmpeg libsm6 libxext6 libffi-dev libssl-dev xz-utils zlib1g-dev liblzma-dev

# Setting up Python 3.9
WORKDIR /install

RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python${V_PYTHON_MAJOR} \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python

# PyTorch - Build - Source Code Setup 
# copy everything from the downloader-pytorch layer /torch to /torch on this one
COPY --from=downloader-pytorch /pytorch /pytorch
WORKDIR /pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Set path to ccache
ARG PATH=/usr/lib/ccache:$PATH

# Other arguments
ARG USE_CUDA=ON
ARG USE_CUDNN=ON
ARG BUILD_CAFFE2_OPS=0
ARG USE_FBGEMM=0
ARG USE_FAKELOWP=0
ARG BUILD_TEST=0
ARG USE_MKLDNN=0
ARG USE_NNPACK=0
ARG USE_XNNPACK=0
ARG USE_QNNPACK=0
ARG USE_PYTORCH_QNNPACK=0
ARG TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
ARG USE_NCCL=0
ARG USE_SYSTEM_NCCL=0
ARG USE_OPENCV=0
ARG USE_DISTRIBUTED=0

# Build
RUN cd /pytorch \
    && rm build/CMakeCache.txt || : \
    && sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
    && pip install -r requirements.txt \
    && python setup.py bdist_wheel \
    && cd ..

# ##################################################################################
# Prepare Artifact
# ##################################################################################
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

## GitHub Action Creation

Since PyTorch is now finally compiling! It is time to start automating this and publishing them to an Artifact in GitHub (this way we can always trigger it ourselves and kick of the building process). I want to start building automatically, as soon as a Release is published! So for our action, we have the following outline:

### Action Outline

1. When a release is created trigger the action
2. Clone the repository
3. Setup Docker with Buildx
4. Run our container
5. Copy over the Built Wheel to an artifact on GitHub

### Used Actions

As for actions, the following actions could be reused:

1. [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action)
    * I cross-compile for ARM on AMD64 machines in the pipeline
2. [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action)
    * Configure QEMU to be able to compile for ARM and install the QEMU static binaries
3. [actions/checkout](https://github.com/actions/checkout)
    * Check out a repo
4. [actions/cache](https://github.com/actions/cache)
    * Allow us to cache the docker layers
5. [actions/upload-artifact](https://github.com/actions/upload-artifact)
    * Upload the output of a directory to GitHub artifacts

### Result

Finally resulting in the following GitHub action:

```yaml
name: ci

# https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows
on:
  push:
    branches: [ main ]
  release:
    types: [ created ]

jobs:
  build_wheels:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v2
        
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v1
        
    - name: Set up Docker Buildx
      id: buildx
      uses: docker/setup-buildx-action@v1
    
    - name: Cache Docker layers
      uses: actions/cache@v2
      with:
      path: /tmp/.buildx-cache
      key: ${{ runner.os }}-buildx-${{ github.sha }}
      restore-keys: |
        ${{ runner.os }}-buildx-
    
    - name: Build Docker Image
      run: |
      docker buildx build \
          --platform=linux/arm64 \
          --progress=plain \
          --output type=local,dest=./wheels \
          --file Dockerfile.jetson .

    - name: Upload Artifacts
      uses: actions/upload-artifact@v2
      with:
      name: wheels
      path: |
        wheels/*.whl

    # This ugly bit is necessary if you don't want your cache to grow forever
    # till it hits GitHub's limit of 5GB.
    # Temp fix
    # https://github.com/docker/build-push-action/issues/252
    # https://github.com/moby/buildkit/issues/1896
    - name: Move cache
      run: |
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache
```

## Future Work

### Extra Optimizations to be made

Some support could still be added for the Nvidia Jetson Nano by adapting the source code, but this is currently out of scope of this project. These optimisations can be found in [QEngineering their post](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_425)

### GitHub Actions Improvements

Currently, Build Arguments are included but not yet used. In theory, the following can be added to the `docker buildx` command to build for other PyThon versions:

**Python 3.8**

```bash
docker buildx build \
    --platform=linux/arm64 \
    --progress=plain \
    --build-arg PYTHON_MAJOR=3 \
    --build-arg PYTHON_MINOR=8 \
    --output type=local,dest=./wheels \
    --file Dockerfile.jetson .
```

## Conclusion

This project was definitely not easy, taking a long time between builds, figuring out where to build, automating it, ... by sharing this I hope to help the community utilize GPUs more easily with the latest versions.

In a next article, I hope to show you how to run an actual AI model with CUDA enabled on the Nvidia Jetson nano and Python > 3.6 😉 

## References

All of the above was possible by some contributions of others:

* [https://github.com/soerensen3/buildx-pytorch-jetson](https://github.com/soerensen3/buildx-pytorch-jetson) helped me with some of the Dockerfile code (but didn't had CUDA support)






Learn how you can compile PyTorch to run on the Nvidia Jetson with a Python version > 3.6

Enabling CUDA with PyTorch on Nvidia Jetson and Python 3.9


I recently acquired an Nvidia Jetson Nano 2GB Developer Kit and had some fun with it! In this series I present the overview of the articles I created:

* [Setting up an Nvidia Jetson Nano Node](/post/2021/05/30/nvidia-jetson-setup)
* [Creating your own Nvidia Jetson Cluster](/post/2021/05/30/nvidia-jetson-cluster)
* [Deploying Kubernetes on your Nvidia Jetson Nano Cluster](/post/2021/05/30/nvidia-jetson-k3s)
* [Adding Dapr and Grafana to the Nvidia Jetson Nano Kubernetes Cluster](/post/2021/05/30/nvidia-jetson-k3s-dapr-and-grafana)


Learn how to configure the Jetson Nano, Deploy a Kubernetes server and finally monitor it with Grafana

An Nvidia Jetson Nano Series


Since Kubernetes has now been installed, we can continue to run a demonstration! For this demonstration we are going to deploy the following:

1. [Dapr](https://dapr.io) as our Distributed Application Runtime
2. Deploy an Hello World Example
3. [Grafana](https://grafana.com)

So let's get started!

## Installing Dapr

To install Dapr we can utilize its `install.sh` script which we execute:

```bash
wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
```

Whereafter we can run `dapr --version` to validate that it is installed:

```bash
# Output of dapr --version
CLI version: 1.2.0
Runtime version: n/a
```

Finally, we configure Dapr on our Kubernetes cluster by initializing it

```bash
dapr init -k
```

Which we again verify by running `dapr status -k` 

```bash
# Output of dapr status -k
NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE  CREATED
dapr-sentry            dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-operator          dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-dashboard         dapr-system  True     Running  1         0.6.0    33s  2021-05-30 11:27.57
dapr-sidecar-injector  dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-placement-server  dapr-system  True     Running  1         1.2.0    32s  2021-05-30 11:27.58
```

## Installing a Demo Application

kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

## Grafana Dashboard

Installing Grafana is a bit more complicated as it requires a store to gather our telemetry. For this we will be utilizing Prometheus.

### Install Prometheus

For configuring Prometheus, we utilize the commands below that will set it up with a persistent volume.

> 💡 To disable persistent volumes, append `--set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false` to the end of the `helm install` command.

```bash
# Create a dedicated namespace
kubectl create namespace dapr-monitoring

# Install Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring

# Verify that it is running
kubectl get pods -n dapr-monitoring -w
```

### Install Grafana

Since we have our store set-up, we can now install our dashboard

> 💡 We can again disable persistent volumes by appending `--set persistence.enabled=false` to the `helm install` command

```bash
# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n dapr-monitoring

# Verify that it is running
kubectl get pods -n dapr-monitoring -w
```

### Grafana Admin Password

Once Grafana is up and running, gather its admin password by running the command below

```bash
kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

### Access Grafana

Now Grafana is up and running, we can access it by port forwarding its port ot our [local system](/post/2021/5/30/infrastructure-nvidia-nvidia-jetson-k3s#Installing-and-Configuring-Kubectl).

```bash
# Forward Grafana from our cluster to localhost
kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring

# Open browser at:
# http://localhost:8080/
```

We can now login with:

* **Username:** `admin`
* **Password:** See Grafana Admin Password (`kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo`)

### Configure Grafana with Prometheus

Finally we need to set-up our Grafana Dashboard. Luckily for us, Dapr provides some [out-of-the-box dashboards](https://github.com/dapr/dapr/tree/master/grafana)

To start, we load the dashboard and add our Prometheus Server

![](./grafana.png)

![](./grafana-prometheus.png)

As a URL we fetch the Dapr Prometheus Server url by viewing the services and finding the name of the deployment. Which will become `<DEPLOYMENT_NAME>.<NAMESPACE>`

```bash
kubectl get svc -n dapr-monitoring
```

![](./grafana-prometheus-server.png)

In our case this becomes `http://dapr-prom-prometheus-server.dapr-monitoring`

![](./grafana-configure.png)

Now we can `Save & Test` and go towads the import screen, where we import the dashboard that we copied from ]Dapr its GitHub page.](https://raw.githubusercontent.com/dapr/dapr/master/grafana/grafana-system-services-dashboard.json)

![](./grafana-import.png)
![](./grafana-import-2.png)

If everything is done correctly, we can now see our monitoring

![](./grafana-final.png)

Learn how you can install Dapr and Grafana on your own Kubernetes Cluster powered by Nvidia Jetson Nanos

Installing Grafana on your Kubernetes Cluster


We now have [a beautiful cluster assembled and running](/post/2021/5/30/infrastructure-nvidia-nvidia-jetson-cluster), but no software running on it yet. So let's get started and provision Kubernetes on top of it so we can manage our nodes easier!

## Introducing K3s

For our Kubernetes setup we want a lightweight Kubernetes installation seeing the required Edge computation. For this, we can utilize [K3S](https://k3s.io/) which is a distribution built specifically to run on IoT & Edge devices.

> Personally, the installation feels similar to how we installed and configured Docker Swarm before. Install it everywhere and join nodes to the master - quick and easy! 

## Installing K3s

Installing K3S is super trivial! There is however an easier way if you want to automate all of it, by utilizing the CLI and adding just 1 parameter.

### Master Node Setup

To configure our Master Node, we execute the following command that will pass the secret we define:

```bash
curl -sfL https://get.k3s.io | sh -s server --cluster-init --token "XAVIER_HIS_SECRET_TOKEN" --write-kubeconfig-mode 644 --docker
```

> 💡 As you may have noticed, we are utilizing `--write-kubeconfig-mode 644` to allow our cluster to be imported in rancher later.

> 💡 `--docker` will enable us to use our host's Docker registry and use local images

### Worker Node Setup

Finally, we can add our Worker (or Agent) nodes by running the following line on each of the nodes:

```bash
curl -sfL https://get.k3s.io | sh -s agent --server https://MASTER_NODE_IP_ADDRESS:6443 --token "XAVIER_HIS_SECRET_TOKEN"
```

### Verification

When we now execute `sudo k3s kubectl get nodes -w` on our master node, we will see our nodes appearing.

![](./k3s-node-setup.png)

## Installing and Configuring Kubectl

> ⚠ Execute this on your local machine! (WSL in this case)

### Installing Kubectl

Since we now have a Kubernetes cluster running, it is time to configure it to run on our local system. For that I am utilizing the `WSL` or `Windows Subsystem for Linux` system as explained in a [previous article](/post/2021/1/4/coding-python-windows-development-environment-setup).

To install WSL we execute the following:

```bash
# Download Kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Install it
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Verify
kubectl version --client
```

### Configuring Kubectl

Now this is installed, we can configure it towards our remote cluster:

```bash
# Copy over the Kubeconfig file to our local machine
scp xavier@xavier-nvidia-node-1.local:/etc/rancher/k3s/k3s.yaml nvidia-node-1-config.yaml

# Replace localhost with the location of the master node
sed -ie s/127.0.0.1/xavier-nvidia-node-1.local/g nvidia-node-1-config.yaml

# Verify that it works
kubectl --kubeconfig nvidia-node-1-config.yaml get pods -A

# Then copy it over to kubectl directory and activate it
# we can now run kubectl get pods -A
cp nvidia-node-1-config.yaml ~/.kube/config
```

## Troubleshooting

### Agent does not want to join

Sometimes an agent does not want to join, in that case try utilizing the IP address directly.

### Removing K3S

⚠ In case something happened, you can uninstall K3S with the following commands:

```bash
# Uninstall Server Node
/usr/local/bin/k3s-uninstall.sh

# Uninstall Agent Node
/usr/local/bin/k3s-agent-uninstall.sh
```

## Conclusion

We have now set-up our Nvidia Jetson nodes and are running a Kubernetes cluster on it! To keep this article light, I have decided to move the demonstration to a new article where you will learn to [run Dapr as your distributed application runtime as well as Grafana to monitor your nodes](/post/2021/5/30/infrastructure-nvidia-nvidia-jetson-k3s-dapr-and-grafana)

Learn how you can install Kubernetes on a small GPU Cluster for edge computation purposes

Installing Kubernetes on an Nvidia Jetson Cluster


[In my previous article](/post/2021/5/30/infrastructure-nvidia-nvidia-jetson-setup) I explained how we can get one single Nvidia Jetson Nano 2GB Node up and running. In this article I would love to go in more detail on how we can create a cluster.

## Requirements

To create my cluster, I utilized the materials below:

> **Note:** These requirements were created for 3 Nodes and receive internet through a Mobile Pocket router that can be put in Client mode to connect LAN over WiFi (very useful for portability).

* 3x [Jetson Nano](https://developer.nvidia.com/embedded/jetson-nano-2gb-developer-kit) ~195 Eur Total
* 1x [Switch](https://eu.dlink.com/be/nl/products/dgs-105-5-port-gigabit-ethernet-switch) ~19.95 Eur
* 1x [Standoffs](https://www.amazon.nl/gp/product/B0778JRXPS) ~8.99 Eur
* 1x [Mobile Router](https://www.tp-link.com/nl/home-networking/wifi-router/tl-wr902ac/) (it is end of life, but super cheap and still working!) ~34.15 Eur
* 4x 0.5m UTP Cables (I had those, but typically ~ 4eur for 4)
* 4x [0.3m USB C to USB A cables](https://www.blitzwolf.com/BlitzWolf-BW-TC13-PVC-3A-USB-Type-C-Charging-Data-Cable-1ft0.3m-22-AWG-Power-+-30-AWG-Data-Wire-Cores-p-276.html) ~19.68 Eur
* 1x [6 port USB Hub](https://m.blitzwolf.com/60W-QC3.0-6-Ports-Charger-Adapter-p-450.html) ~18.87 Eur
* 4x [128GB SD Card](https://www.samsung.com/be/memory-storage/memory-card/evo-plus-microsd-card-with-sd-adapter-128gb-mb-mc128ga-eu/) ~58Eur Total (minimum should be 32GB)

**Total Bill:** 358.64 Eur (or 163.64 Eur without the Nvidia Jetsons)

## While Assembling

To make efficient use of our time, we can flash the SD Cards with the Operating System as explained [before](/post/2021/5/30/infrastructure-nvidia-nvidia-jetson-setup). They take around 20 minutes each.

## Assembling

While the SD Cards are flashing, we can get started with assembly! This is quite straightforward, so mount the nodes on top of the switch and connect them through the UTP cables. Then take the Mobile Pocket Router and connect that to the first port of the switch.

![](./node-standoffs.jpg)

![](./nodes-standoffs-stacked.jpg)

![](./switch.jpg)

![](./nodes-ethernet.jpg)

Once this is done, we can provide power by connecting the nodes and the mobile Router to our USB Hub. Finally, power on the switch and the USB Hub to get started!

![](./cluster-assembled.jpg)

## Connecting to your Nvidia Jetson Nanos

Once the Nvidia Jetson Nanos have been set-up, you can quite easily detect them by installing [Angry IP Scanner](https://angryip.org/download/#windows) and scanning your home network. This will show the nodes and their respective IP Addresses.

![](./ip-scan.png)

## Configuring Windows Terminal

As mentioned earlier, you can easily login on all the nodes through a shortcut and entering the command below:

```bash
CMD /C wt.exe wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-1.local"; split-pane -H wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-2.local"; split-pane -V wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-3.local"
```

## Conclusion

With a small bill we have built our own GPU Cluster that has a combined 384 Cores and 1.416 TFLOPS! (which is similar to 1 Jetson TX2)

![](./cluster-finalized.jpg)

Learn how you can create your own Nvidia Jetson Nano GPU Cluster

Creating an Nvidia Jetson Nano Portable Cluster


I recently acquired an Nvidia Jetson Nano 2GB Developer Kit, so I'd love to walk you through my process of getting it up and running!

## Installing the Jetpack SDK SD Card Image

To help us, Nvidia created an SD Card Image with the [Jetpack SDK](https://developer.nvidia.com/embedded/jetpack) built in. We can use this Image to create an SD Card that can be used to boot up the Nvidia Jetson Developer Kit! So let's go ahead and [download the image](https://developer.nvidia.com/jetson-nano-2gb-sd-card-image).

> Note: I have the 2GB Developer Kit version

Once the image is downloaded, we will have to flash it to the SD Card. For this we can utilize [Balena Etcher](https://www.balena.io/etcher/). 

So boot up Balena and select your target device and OS that you just downloaded.

![](./balena-etcher-nvidia-jetson.png)

![](./balena-etcher-nvidia-jetson-2.png)

![](./balena-etcher-nvidia-jetson-3.png)
s
> Note: this can take quite a while (~30min in my case) depending on the SD Card speed. I had some old ones lying around and they are definitely not fast

## Configuring your Nvidia Jetson Nano (headless)

Once the SD Card has been inserted and the device has been powered on with the USB-C cable, we need to connect to it. For this NVIDIA gave us [2 possible ways](https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-2gb-devkit#setup):
* Screen Mode (with Display)
* Headless Mode (without Display)

I chose for the headless mode, since it's quite annoying to setup a device this way. So let's follow these steps:
1. Plug-in your Nvidia Jetson through Micro-USB on your PC
2. Situate the COM port it is connected to (see Device Manager - for me COM4).
3. Open Properties and go to details, then select Hardwareids and ensure the `PID (Product ID) = 7020` and the `VID (Vendor ID) is 0955`
4. Open "PuTTY" (`winget install PuTTY.PuTTY` if you don't have it)
5. Select `Baud Rate = 115200`; `Serial Line = COM4`; `Connection Type = Serial` and click "Open"

> Note: I provided the hostname: `xavier-nvidia-node-1` since I will create a cluster later.

> ⚠ When I tried to configure the system, I received an error: `System configuration setup wasn't completed. Please reboot device and try again.`

![](./device-manager-com.png)

![](./device-manager-com-hardwareids.png)

![](./putty-config.png)

![](./system-config.png)

## Configuring SSH

Our node has now been configured and we are able to connect to it. Get the IP address you configured earlier and utilize `ssh <USER>@<YOUR_IP>` to login on the node. 

To make our lives easier, we can apply the following command with your `id_rsa.pub` key to skip having to enter a password next time.

```bash
mkdir -p ~/.ssh/; touch ~/.ssh/authorized_keys; echo "YOUR ID_RSA.PUB KEY" >> ~/.ssh/authorized_keys
```

## Configuring Windows Terminal

As a final configuration, I am utilizing `Windows Terminal` and an Elegato StreamDeck for easy hotkey access. Since I do not always want to enter my 3 nodes and reconfigure my terminal, I utilize the following command to automatically login on all the nodes:

```bash
CMD /C wt.exe wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-1.local"; split-pane -H wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-2.local"; split-pane -V wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-3.local"
```

## Finalization

Congratulations! You should now be able to see your NVIDIA Jetson powered up an a prompt to start working on it!

Learn how to get started with the Nvidia Jetson Nano Developer Kit