
For a use case I wanted to utilize the Nvidia Jetson for edge inference. One of the bottlenecks here was that my software required a Python version that is greater than 3.6. When looking at the [Nvidia Jetson packages for PyTorch](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch) it was seen that this was only created for version 3.6.

Searching around, I could find quite a lot of people struggling with this (@TODO: PROOF) which made me to look into **a solution** that would help me automate the building of ARM Wheel of **PyTorch that can run on Nvidia Devices** (e.g. Nvidia Jetson Nano) and thus **support CUDA**.

The entire process above took me around ~11 full days, starting of with figuring out how to build the Dockerfile and finally automating the CI process.

> Note: I also decided to utilize this article as an entry for a running Hackathon by [Dev.to for GitHub Actions](https://dev.to/devteam/join-us-for-the-2021-github-actions-hackathon-on-dev-4hn4)

You would wonder why we want to automate this? Well when compiling this initially on my Nvidia Jetson Nano, I couldn't get it to compile past ~80% when using 16Gb of Swap Space (the device only has 2 Gb). After building it on my personal PC, this took ~6h to get it to compile completely. Making it large and long enough to automate it.

> The final source code can be found on [GitHub](https://github.com/XavierGeerinck/Jetson-Linux-PyTorch)

## Contributions

In any kind of project of this size, there are specific contributions that were made. In my project I believe to have made the following:

* Install CUDA on non-GPU devices
* Compile PyTorch with CUDA enabled on non-GPU devices
* Compile PyTorch for Python > 3.6
* Build for ARM with CI through Docker Buildx

## Project Outline

As a best practice, I always love to include the project outline of how I tackled the issue above (to share my thought process):

1. Create a Dockerfile that builds the Wheel
    * How do I build for CUDA? (Hardest part)
    * How do I cross build?
2. Create GitHub Action
    * How do I cross build? Can I run for ARM specifically?

## Dockerfile Creation - Building PyTorch for ARM and Python > 3.6

The hardest part BY FAR is how to compile PyTorch for ARM and Python > 3.6 with CUDA enabled. Because we are going to run on a non-GPU device, thus CUDA is not available on there. I splitted up the Dockerfile into 3 specific sections that can be ran in parallel:

1. Set up CUDA
2. Set up PyTorch (the cloning takes a while)
3. Set up Python 3.9
4. Compile PyTorch
    * I have Nvidia Jetson optimisations included here, thanks [QEngineering](https://qengineering.eu/install-pytorch-on-jetson-nano.html)!
4. Dockerfile Result

Below you can find the explanation of the respective steps. For the final `Dockerfile` optimizations where made to decrease the docker layer sizes (by grouping RUN in 1 command)

### Setting up CUDA

For CUDA, we do not have the CUDA libraries, nor do we have access to them! There is however a trick that allows us to get CUDA loaded in. We copy over the public key of NVIDIA Jetson and we authorize ourselves to the repository. Then we use our package manager to install them:

```bash
V_CUDA_DASH=10-2

# Add the public key
echo "[Builder] Adding the Jetson Public Key"
curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc
echo "deb https://repo.download.nvidia.com/jetson/common ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
echo "deb https://repo.download.nvidia.com/jetson/t186 ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

# Install the CUDA Libraries
echo "[Builder] Installing CUDA System"
apt-get update
apt-get install -y --no-install-recommends \
    cuda-libraries-$V_CUDA_DASH \
    cuda-libraries-dev-$V_CUDA_DASH \
    cuda-nvtx-$V_CUDA_DASH \
    cuda-minimal-build-$V_CUDA_DASH \
    cuda-license-$V_CUDA_DASH \
    cuda-command-line-tools-$V_CUDA_DASH \
    libnvvpi1 vpi1-dev

# Link CUDA to /usr/local/cuda
ln -s /usr/local/cuda-$CUDA /usr/local/cuda
```

When we eventually start compiling, we will see CUDA enabled in our CMAKE output 🥳

```bash
#   USE_CUDA              : ON
#     Split CUDA          : OFF
#     CUDA static link    : OFF
#     USE_CUDNN           : OFF
#     USE_EXPERIMENTAL_CUDNN_V8_API: OFF
#     CUDA version        : 10.2
#     CUDA root directory : /usr/local/cuda
#     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
#     cudart library      : /usr/local/cuda/lib64/libcudart.so
#     cublas library      : /usr/local/cuda/lib64/libcublas.so
#     cufft library       : /usr/local/cuda/lib64/libcufft.so
#     curand library      : /usr/local/cuda/lib64/libcurand.so
#     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
#     CUDA include path   : /usr/local/cuda/include
#     NVCC executable     : /usr/local/cuda/bin/nvcc
#     NVCC flags          : <CUT>
#     CUDA host compiler  : /usr/bin/clang
#     NVCC --device-c     : OFF
#     USE_TENSORRT        : OFF
```

### Setting up PyTorch

In a separate docker step, we set up PyTorch and clone it to the working repository (in our case `/build/pytorch`)

```bash
V_PYTORCH=v1.10.0

# Downloads PyTorch to /build/pytorch
git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch /build/pytorch
```

### Setting up Python 3.9

We configure our Python version through the `deadsnakes` ppa and link it as the default one.

> Best practice we should have a `venv` but since I am running it in a Docker container this should suffice.

```bash
# Setting up Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python3 \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python
```

### Compiling PyTorch

The last step in the Dockerfile is to compile PyTorch. For this we set the correct environment variabels to enable CUDA and to optimise the building process by disabling some other parts (e.g. MKLDNN, NNPACK, XNNPACK, ... to be turned off).

We also configure it to use `clang` as the Nvidia Jetson has [NEON registers](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_462) and clang supports those (GCC doesn't).

For our source, we utilize the other layer we created and just copy it from there.

```dockerfile
COPY --from=downloader-pytorch /build/pytorch /build/pytorch
WORKDIR /build/pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Build
rm build/CMakeCache.txt || : \
sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
pip install -r requirements.txt
python setup.py bdist_wheel
```

### Copying the result as an Artifact

Docker Buildx is amazing in the sense that we can utilize the `--output type-local,dest=.` command to output files to our local filesystem, making it such that docker builds and we can export the build result as an artifact.

To achieve this, we pull from the `scratch` image and copy over our result to it from the other docker layer. Our `/` path will then contain all the build wheels of PyTorch (e.g. `torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl`)

```dockerfile
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

### Dockerfile Result

Finally, the full Dockerfile will look like this:

```dockerfile
# ##################################################################################
# Setup Nvidia CUDA for Jetson
# ##################################################################################
FROM ubuntu:18.04 as cuda-devel

# Configuration Arguments
ARG V_CUDA_MAJOR=10
ARG V_CUDA_MINOR=2
ARG V_L4T_MAJOR=32
ARG V_L4T_MINOR=6

ENV V_CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}
ENV V_CUDA_DASH=${V_CUDA_MAJOR}-${V_CUDA_MINOR}
ENV V_L4T=r${V_L4T_MAJOR}.${V_L4T_MINOR}

# Expose environment variables everywhere
ENV CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Fix CUDA info
ARG DPKG_STATUS

# Add NVIDIA repo/public key and install VPI libraries
RUN echo "$DPKG_STATUS" >> /var/lib/dpkg/status \
    && echo "[Builder] Installing Prerequisites" \
    && apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates software-properties-common curl gnupg2 apt-utils \
    && echo "[Builder] Installing CUDA Repository" \
    && curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc \
    && echo "deb https://repo.download.nvidia.com/jetson/common ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "deb https://repo.download.nvidia.com/jetson/t186 ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "[Builder] Installing CUDA System" \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
    cuda-libraries-${V_CUDA_DASH} \
    cuda-libraries-dev-${V_CUDA_DASH} \
    cuda-nvtx-${V_CUDA_DASH} \
    cuda-minimal-build-${V_CUDA_DASH} \
    cuda-license-${V_CUDA_DASH} \
    cuda-command-line-tools-${V_CUDA_DASH} \
    libnvvpi1 vpi1-dev \
    && ln -s /usr/local/cuda-${V_CUDA} /usr/local/cuda \
    && rm -rf /var/lib/apt/lists/*

# Update environment
ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs

RUN ln -fs /usr/share/zoneinfo/Europe/Brussels /etc/localtime

# ##################################################################################
# Create PyTorch Docker Layer
# We do this seperately since else we need to keep rebuilding
# ##################################################################################
FROM --platform=$BUILDPLATFORM ubuntu:18.04 as downloader-pytorch

# Configuration Arguments
# https://github.com/pytorch/pytorch
ARG V_PYTORCH=v1.10.0
# https://github.com/pytorch/vision
ARG V_PYTORCHVISION=v0.11.1
# https://github.com/pytorch/audio
ARG V_PYTORCHAUDIO=v0.10.0

# Install Git Tools
RUN apt-get update \
    && apt-get install -y --no-install-recommends software-properties-common apt-utils git \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Clone Source
RUN git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch

# ##################################################################################
# Build PyTorch for Jetson (with CUDA)
# ##################################################################################
FROM cuda-devel as builder

# Configuration Arguments
ARG V_PYTHON_MAJOR=3
ARG V_PYTHON_MINOR=9

ENV V_PYTHON=${V_PYTHON_MAJOR}.${V_PYTHON_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Download Common Software
RUN apt-get update \
    && apt-get install -y clang build-essential bash ca-certificates git wget cmake curl software-properties-common ffmpeg libsm6 libxext6 libffi-dev libssl-dev xz-utils zlib1g-dev liblzma-dev

# Setting up Python 3.9
WORKDIR /install

RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python${V_PYTHON_MAJOR} \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python

# PyTorch - Build - Source Code Setup 
# copy everything from the downloader-pytorch layer /torch to /torch on this one
COPY --from=downloader-pytorch /pytorch /pytorch
WORKDIR /pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Set path to ccache
ARG PATH=/usr/lib/ccache:$PATH

# Other arguments
ARG USE_CUDA=ON
ARG USE_CUDNN=ON
ARG BUILD_CAFFE2_OPS=0
ARG USE_FBGEMM=0
ARG USE_FAKELOWP=0
ARG BUILD_TEST=0
ARG USE_MKLDNN=0
ARG USE_NNPACK=0
ARG USE_XNNPACK=0
ARG USE_QNNPACK=0
ARG USE_PYTORCH_QNNPACK=0
ARG TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
ARG USE_NCCL=0
ARG USE_SYSTEM_NCCL=0
ARG USE_OPENCV=0
ARG USE_DISTRIBUTED=0

# Build
RUN cd /pytorch \
    && rm build/CMakeCache.txt || : \
    && sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
    && pip install -r requirements.txt \
    && python setup.py bdist_wheel \
    && cd ..

# ##################################################################################
# Prepare Artifact
# ##################################################################################
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

## GitHub Action Creation

Since PyTorch is now finally compiling! It is time to start automating this and publishing them to an Artifact in GitHub (this way we can always trigger it ourselves and kick of the building process). I want to start building automatically, as soon as a Release is published! So for our action, we have the following outline:

### Action Outline

1. When a release is created trigger the action
2. Clone the repository
3. Setup Docker with Buildx
4. Run our container
5. Copy over the Built Wheel to an artifact on GitHub

### Used Actions

As for actions, the following actions could be reused:

1. [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action)
    * I cross-compile for ARM on AMD64 machines in the pipeline
2. [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action)
    * Configure QEMU to be able to compile for ARM and install the QEMU static binaries
3. [actions/checkout](https://github.com/actions/checkout)
    * Check out a repo
4. [actions/cache](https://github.com/actions/cache)
    * Allow us to cache the docker layers
5. [actions/upload-artifact](https://github.com/actions/upload-artifact)
    * Upload the output of a directory to GitHub artifacts

### Result

Finally resulting in the following GitHub action:

```yaml
name: ci

# https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows
on:
  push:
    branches: [ main ]
  release:
    types: [ created ]

jobs:
  build_wheels:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v2
        
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v1
        
    - name: Set up Docker Buildx
      id: buildx
      uses: docker/setup-buildx-action@v1
    
    - name: Cache Docker layers
      uses: actions/cache@v2
      with:
      path: /tmp/.buildx-cache
      key: ${{ runner.os }}-buildx-${{ github.sha }}
      restore-keys: |
        ${{ runner.os }}-buildx-
    
    - name: Build Docker Image
      run: |
      docker buildx build \
          --platform=linux/arm64 \
          --progress=plain \
          --output type=local,dest=./wheels \
          --file Dockerfile.jetson .

    - name: Upload Artifacts
      uses: actions/upload-artifact@v2
      with:
      name: wheels
      path: |
        wheels/*.whl

    # This ugly bit is necessary if you don't want your cache to grow forever
    # till it hits GitHub's limit of 5GB.
    # Temp fix
    # https://github.com/docker/build-push-action/issues/252
    # https://github.com/moby/buildkit/issues/1896
    - name: Move cache
      run: |
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache
```

## Future Work

### Extra Optimizations to be made

Some support could still be added for the Nvidia Jetson Nano by adapting the source code, but this is currently out of scope of this project. These optimisations can be found in [QEngineering their post](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_425)

### GitHub Actions Improvements

Currently, Build Arguments are included but not yet used. In theory, the following can be added to the `docker buildx` command to build for other PyThon versions:

**Python 3.8**

```bash
docker buildx build \
    --platform=linux/arm64 \
    --progress=plain \
    --build-arg PYTHON_MAJOR=3 \
    --build-arg PYTHON_MINOR=8 \
    --output type=local,dest=./wheels \
    --file Dockerfile.jetson .
```

## Conclusion

This project was definitely not easy, taking a long time between builds, figuring out where to build, automating it, ... by sharing this I hope to help the community utilize GPUs more easily with the latest versions.

In a next article, I hope to show you how to run an actual AI model with CUDA enabled on the Nvidia Jetson nano and Python > 3.6 😉 

## References

All of the above was possible by some contributions of others:

* [https://github.com/soerensen3/buildx-pytorch-jetson](https://github.com/soerensen3/buildx-pytorch-jetson) helped me with some of the Dockerfile code (but didn't had CUDA support)






Learn how you can compile PyTorch to run on the Nvidia Jetson with a Python version > 3.6

Enabling CUDA with PyTorch on Nvidia Jetson and Python 3.9


The reworked Dapr Javascript SDK has been released a couple of weeks ago! Together with a brand new and easier to use SDK, we have been thinking in the community of reworking the documentation to simplify the examples, more specifically the Actors examples.

After some brainstorming together, we decided to follow a scenario that was explained by [Mark Russinovich](https://www.youtube.com/watch?v=eJCu6a-x9uo&t=3785) detailing **how you can create your own Parking Garage Spot Monitoring System!**

## Scenario

Microsoft is growing to 1.000.000 employees. To accomodate for this, they have parking garages to hold the cars of these employees. For simplicity reasons of the demonstration, we take into account 2 parking garages that can hold each 500.000 employee cars and each have their own sensor monitoring the spot. If a car drives into a spot, an event is sent that updates the value of the sensor itself. Each of the sensors is using Azure Sphere.

## Actors

Now we could implement this with a dedicated application for each of the sensors, bake in the application and let it have its own state management system. An easier system however is to have the sensor send out a packet that we can ingest and process. Since each sensor is "standalone" this should have its own isolated set of actions and state! Which fits the description of an "Actor".

## Approach

To implement this parking garage example, we need to do the following:

* Create an actor that represents the latest state of a sensor at any given time
* Create a database made for holding historical data (`influxdb`)
* Create a dashboard that shows our points with color codes on a map (`grafana`)

## Implementation

Let's start by implementing this! First we should define the actions our actor can take, in the case of a Parking Garage with sensors, we will have 2 methods: `carEnter` and `carLeave` to define the state of the sensor. So let's draw up the interface for this:

### Defining the Actor Methods

```javascript
export default interface ParkingSensorInterface {
  carEnter(): Promise<void>;
  carLeave(): Promise<void>;
}
```

### Defining the Actor Implementation

Once the interface has been created, we need to implement the actual logic of our actor. For this actor, we thus implement the interface described above and we add some extra helper methods.

**Outline:**

1. We create an `onActivate` method which acts as the constructor when the actor is created. This will initialize the mock location of our sensor in a parking garage (latitude and longitude) through the `generateRandomPoint` method.
2. We implement the actor interface methods `carEnter` and `carLeave`
    * `carEnter`: Will set our state to `true` defining that a car is parked
    * `carLeave`: Will set our state to `false` defining that a spot is free
3. We add binding calls to `influxdb` that when a state changes also sends the state to `influxdb` for visualization purposes.

> **Note:** the power of Dapr is exactly that of the above, we implement actors and need to sink a point to a database, rather than having to implement and configure yet another API we can simply call the client with the binding and that will take care of it for us!

**Code:**

```javascript
import { AbstractActor } from "dapr-client";
import ParkingSensorInterface from "./ParkingSensorInterface";

const STATE_NAME_PARKING_SENSOR_VALUE = "parking-sensor-value";
const STATE_NAME_PARKING_SENSOR_LOCATION_LAT = "parking-sensor-location-lat";
const STATE_NAME_PARKING_SENSOR_LOCATION_LNG = "parking-sensor-location-lng";

// Coordinates pool to pick from as center of the radius circle
// we take multiple so we can distribute points more
// this is the Microsoft Garage Building 99
// 47.64174525422637, -122.14032265575015
const COORDINATES = [
  {
    lat: 47.64167040541433, lng: -122.1398701709982
  },
  {
    lat: 47.64177786440955, lng: -122.14077604924732
  }
]

export default class ParkingSensorImpl extends AbstractActor implements ParkingSensorInterface {
  async carEnter(): Promise<void> {
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_VALUE, true);

    const sensorLocationLat = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT);
    const sensorLocationLng = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG);

    await this.getDaprClient().binding.send("binding-influxdb", "create", {
      measurement: "sensor-states",
      tags: `sensor=${this.getId().getId()}`,
      values: `lat=${sensorLocationLat},lng=${sensorLocationLng},isParked=1`
    });
  }

  async carLeave(): Promise<void> {
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_VALUE, true);

    const sensorLocationLat = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT);
    const sensorLocationLng = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG);

    await this.getDaprClient().binding.send("binding-influxdb", "create", {
      measurement: "sensor-states",
      tags: `sensor=${this.getId().getId()}`,
      values: `lat=${sensorLocationLat},lng=${sensorLocationLng},isParked=0`
    });
  }

  async getInfo(): Promise<object> {
    const sensorValue = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_VALUE);
    const sensorLocationLat = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT);
    const sensorLocationLng = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG);

    return {
      sensorValue,
      sensorLocationLat,
      sensorLocationLng
    }
  }

  async onActivate(): Promise<void> {
    const coordIdx = Math.floor(Math.random() * COORDINATES.length);
    const lat = COORDINATES[coordIdx]["lat"];
    const lng = COORDINATES[coordIdx]["lng"];
    const spotLocation = this.generateRandomPoint({ lat, lng }, 20);

    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_VALUE, false);
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT, spotLocation.lat);
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG, spotLocation.lng);
  }

  generateRandomPoint(center: { lat: number, lng: number }, radius: number) {
    var x0 = center.lng;
    var y0 = center.lat;

    // Convert Radius from meters to degrees.
    var rd = radius / 111300;

    var u = Math.random();
    var v = Math.random();

    var w = rd * Math.sqrt(u);
    var t = 2 * Math.PI * v;
    var x = w * Math.cos(t);
    var y = w * Math.sin(t);

    var xp = x / Math.cos(y0);

    // Resulting point.
    return { 'lat': y + y0, 'lng': xp + x0 };
  }
}
```






## Creating our Application

We now have the actor implemented and created but we still need to call it to represent our sensors! So let's create a simple implementation that:

1. Starts a server with our Actor registered
2. Create X amount of actors representing Parking Spot Sensors
3. Set the Parking Spot Sensor randomly to active or inactive every `5s`

> **Note:** In a real-world scenario, we would call this actor based on a pub/sub that receives an event from the sensors. Thus completely separating logic from hardware.

```javascript
import { DaprServer, DaprClient } from "dapr-client";
import ParkingSensorImpl from "./ParkingSensorImpl";

const daprHost = "127.0.0.1";
const daprPort = "50000"; // Dapr Sidecar Port of this Example Server
const serverHost = "127.0.0.1"; // App Host of this Example Server
const serverPort = "50001"; // App Port of this Example Server

async function sleep(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function start() {
  const server = new DaprServer(serverHost, serverPort, daprHost, daprPort);
  const client = new DaprClient(daprHost, daprPort);

  console.log("===============================================================");
  console.log("SERVER INITIALIZING")
  console.log("===============================================================");
  await server.actor.init(); // Let the server know we need actors
  server.actor.registerActor(ParkingSensorImpl); // Register the actor
  await server.startServer(); // Start the server

  console.log("===============================================================");
  console.log("CLIENT EXECUTION");
  console.log("===============================================================");
  const amount = 100;
  console.log(`Creating ${amount} Actors`);
  for (let i = 0; i < amount; i++) {
    await client.actor.invoke("GET", ParkingSensorImpl.name, `parking-sensor-${i}`, "getInfo");
  }

  console.log("===============================================================");
  console.log("CHANGING STATE OF SENSORS");
  console.log("===============================================================");
  console.log("Waiting 10 seconds before starting");
  await sleep(10000);

  while (true) {
    console.log("Simulating cars entering and leaving for 5% of the population");

    const populationSize = Math.floor(amount * 0.05);

    // Randomly put 10 on parked
    for (let i = 0; i < populationSize; i++) {
      const id = Math.floor(Math.random() * amount);
      await client.actor.invoke("PUT", ParkingSensorImpl.name, `parking-sensor-${id}`, "carEnter");
    }

    // Randomly put 10 on empty
    for (let i = 0; i < populationSize; i++) {
      const id = Math.floor(Math.random() * amount);
      await client.actor.invoke("PUT", ParkingSensorImpl.name, `parking-sensor-${id}`, "carLeave");
    }

    await sleep(5000);
  }
}

start().catch((e) => {
  console.error(e);
  process.exit(1);
});
```


## Visualization

### Creating the Containers 

For our visualization we will create an `influxdb` and a `grafana` instance! To quickly boot up these containers in a *non-persistent* way, we can utilize the commands below:

```bash
# Run InfluxDB
# Note: it auto removes after shutdown
# Note: non-persistent volume, add "-v influxdb2:/var/lib/influxdb2" to make it persistent
docker run --rm -it -d \
    -e DOCKER_INFLUXDB_INIT_MODE=setup \
    -e DOCKER_INFLUXDB_INIT_USERNAME=admin \
    -e DOCKER_INFLUXDB_INIT_PASSWORD=MyAdmin@123! \
    -e DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-token \
    -e DOCKER_INFLUXDB_INIT_ORG=my-parking-garage \
    -e DOCKER_INFLUXDB_INIT_BUCKET=my-sensors \
    --net=host \
    --name influxdb \
    influxdb:2.0

# Run Telegraf
docker run --rm -it -d \
    --net=host \
    --name=telegraf \
    telegraf

# Run Grafana
# Note: non-persistent volume, add "-v influxdb2:/var/lib/influxdb2" to make it persistent
docker run --rm -it -d \
    --name=grafana \
    --net=host \
    grafana/grafana
```

> ⚠️ the above is not-persistent, add volume binds and remove `--rm` to make them persistent.

### Configuring the Data Source

Now open up `http://localhost:3000` in your browser to open up Grafana. Here we need to configure a datasource:

![](./grafana-configure-datasource.png)

![](./grafana-configure-datasource-2.png)

![](./grafana-configure-datasource-3.png)

![](./grafana-configure-datasource-4.png)

![](./grafana-configure-datasource-5.png)

### Creating the Dashboard

Above, we configured the data source, so let's now create our visualization for it! To keep this easy I have preconfigured the dashboard before that we can import into Grafana.

```json
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 1,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 25,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "options": {
        "basemap": {
          "config": {},
          "type": "osm-standard"
        },
        "controls": {
          "mouseWheelZoom": true,
          "showAttribution": true,
          "showDebug": false,
          "showScale": false,
          "showZoom": true
        },
        "layers": [
          {
            "config": {
              "color": {
                "field": "isParked",
                "fixed": "dark-green"
              },
              "fillOpacity": 0.4,
              "shape": "circle",
              "showLegend": true,
              "size": {
                "fixed": 3,
                "max": 15,
                "min": 2
              }
            },
            "location": {
              "mode": "auto"
            },
            "type": "markers"
          }
        ],
        "view": {
          "id": "coords",
          "lat": 47.641728,
          "lon": -122.140278,
          "zoom": 19.24
        }
      },
      "targets": [
        {
          "query": "from(bucket: \"my-sensors\")\r\n  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\r\n  |> filter(fn: (r) => r[\"_measurement\"] == \"sensor-states\")\r\n  |> pivot(columnKey: [\"_field\"], rowKey: [\"_time\"], valueColumn: \"_value\")\r\n  |> group()\r\n  |> yield(name: \"last\")",
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "type": "geomap"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 31,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Parking Garage",
  "uid": "Px36rBv7z",
  "version": 5
}
```

For illustration purposes, this is the query that is used to create the visualization:

```
from(bucket: "my-sensors")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "sensor-states")
  |> pivot(columnKey: ["_field"], rowKey: ["_time"], valueColumn: "_value")
  |> group()
  |> yield(name: "last")
```

## Running

Everything has been implemented! So let's start up the demo! For that, simply follow these steps:

1. Run `npm run start:dapr` on your application to start the application
2.open the folder to the example and run: `npm run start:dapr` which will boot up the demo. Finally, open up the dashboard and see the demo running.

![](./result.png)


## Conclusion

By using Dapr we can easily create an implementation of a Parking Garage using its Actor Framework. Without having to worry about certain pesky configurations or coding that we otherwise would have to do.

![](./demo-small-parking-garage.gif)

Learn how you can get started to create your own Parking Garage spot monitoring system using Dapr, Actors and the Javascript SDK!

Parking Garage Sensor implementation using Dapr Actors


I have been a user of Vercel for a long time, but recently I wanted more control over the deployment process of my blog! Which is why I decided to move everything to Azure for this, seeing that they have a [new Static Site hosting option](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website?WT.mc_id=DOP-MVP-5004263) which is simply AMAZING (and free 😉).

## Typical Blog Setup

Typically when setting up a blog, we need to take care of a couple of things:

* Storage Account (expose this to allow public access)
* CDN (for fast worldwide delivery)
* CI/CD setup (e.g. GitHub Actions)

## Azure Static Site with Bicep

Now recently a new offering was made available that bundles all of the above and automatically bundles your application and publishes your website. The only thing we have to do is provide the repository URL and how to build our website! 🤯 Now let's create our Azure setup with an Infrastructure-as-a-Code tool named [Azure Bicep](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview?WT.mc_id=DOP-MVP-5004263)

### Creating Bicep Files

The first file we have to create is our main file that contains the modules we want to deploy and some general overview. In here we use `subscription scoping` to define what has to happen at a *subscription level*

> We split up subscription level resources and resource group level resources in different files. Resource groups are created at a subscription level, while resources are created at a resource group level.

Our `main.bicep` file will thus define some parameters and their defaults and create a resource group. After creating this resource group we will tell it to initiate our `static-website.bicep` module that takes care of creating the static website setup itself.

**main.bicep**

```bicep
param environment string = 'dev'
param location string = 'westeurope'
param resourcePrefix string = 'yourresourceprefix'

// Parameters for static website
param repositoryUrl string
param repositoryToken string // the repository token to setup actions workflow and api secrets (https://github.com/settings/tokens)
param repositoryBranch string = 'master'

// This file operates at a subscription level
targetScope = 'subscription'

resource MainRG 'Microsoft.Resources/resourceGroups@2021-04-01' = {
  name: 'rg-${resourcePrefix}-${environment}'
  location: location
}

module ModuleStaticWebsite 'static-website.bicep' = {
  // Change deployment context to RG
  name: 'static-website'
  scope: resourceGroup(MainRG.name)
  params: {
    environment: environment
    resourcePrefix: resourcePrefix
    repositoryUrl: repositoryUrl
    repositoryBranch: repositoryBranch
    repositoryToken: repositoryToken
  }
}
```

Now our resource group is created, we define our `static-website.bicep` module. This module is quite straightforward as we just have to define the SKU, the Tier and our repository settings as well as how to build our application!

> Note: I am using a Next.js deployment, so I define that my site output will be situated in the `out` folder starting from the `/` after building through `npm run build`

**static-website.bicep**

```bicep
param environment string
param resourcePrefix string
param location string = resourceGroup().location

param repositoryBranch string
param repositoryUrl string

// the repository token to setup actions workflow and api secrets (https://github.com/settings/tokens)
// needs to have repo and workflow access
param repositoryToken string
param skuName string = 'Free'
param skuTier string = 'Free'

// https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep
resource staticWebApp 'Microsoft.Web/staticSites@2021-01-15' = {
  name: 'ss-${resourcePrefix}-${environment}'
  location: location
  sku: {
    name: skuName
    tier: skuTier
  }
  properties: {
    provider: 'GitHub' // required to ensure success in follow up deployments
    repositoryUrl: repositoryUrl
    repositoryToken: repositoryToken
    branch: repositoryBranch
    buildProperties: {
      outputLocation: 'out' // next.js will export to the out directory
      appLocation: '/'
      apiLocation: ''
    }
  }
}

// Outputs
output deployment_token string = listSecrets(staticWebApp.id, staticWebApp.apiVersion).properties.apiKey
```

And that's it! 😱 in just these few lines I have automated the entire infrastructure definition and can run this.

> What is even more amazing is that the Static Website implementation on Azure will add a `.github` workflow that automatically deploys our website to it!

### Deploying Bicep Template

As a last step we have to deploy all of the code we just wrote. So execute the code below to deploy this to your Azure account:

```bash
# Login to Azure
az login

# Set Subscription
az account set --subscription <your_sub_id>

# Deploy Bicep
az deployment sub create --location westeurope --template-file ./main.bicep
```

## Conclusion

In almost under 10 minutes you should have been able to:

1. Deploy new Cloud Infrastructure
2. Take code from your (private) repository
3. Build this code
4. Deploy the result on a static website
5. Automate all of it through a CI/CD pipeline

Which is now powering my personal blog! 😍

Learn how you can automated your Blog Deployment with a CI/CD pipeline with Azure Bicep in just 10 minutes!

Use Bicep and Azure Static Site for your Blog Deployment


Did you find yourself ever having to set-up your development environment all the time? Well I did as well! What if I could tell you, you could save many minutes each day by automating this? Enter the world of Windows Terminal and WSL! 😎

## Requirements

### WSL Environment

For this post we will need an initialized WSL environment. To get started feel free to follow my [previous blog post](https://xaviergeerinck.com/post/coding/python/windows-development-environment-setup) which gives you full control of where you want WSL to be installed! 

### Windows Terminal

If you don't have it yet (you should) go download [Windows Terminal](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701) or install it with `winget install Microsoft.WindowsTerminal`

## Tips and Tricks

### Automating Development Environment Startup

Windows Terminal has a lot of [CLI tricks up its sleeves](https://docs.microsoft.com/en-us/windows/terminal/command-line-arguments?tabs=windows&WT.mc_id=DT-MVP-5004263), one of these is that you can use 1 single command to open up your entire development environment. Typically, I make a shortcut with the following command in it that opens up my environment, starts processes and lets me know how I can start my other programs:

```powershell
CMD /C wt.exe --title "Services" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && sudo service postgresql start && sudo service docker start && exec bash"; new-tab --title "Site" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/frontend && sleep 3 && npm run dev"; split-pane -V wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && sleep 3 && npm run dev"; new-tab --title "Flow Engine" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/backend/dotnet/API && sleep 1 && ./start.sh"; focus-tab -t 0
```

In the following sections I will dissect some of the tricks used above:

### Starting a Service

When WSL boots, not all your services are started automatically. You want to do this but not always adapt the `~/.bashrc` file, so add yourself to sudoers first so you don't have to enter a password all the time and then execute the line below on Windows.

```powershell
# on WSL
echo "$USER ALL=(ALL:ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$USER

# on Windows
CMD /C wt.exe --title "Services" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && sudo service postgresql start && sudo service docker start && exec bash";
```

### Starting a Program

Besides starting a service you also might want to start your program automatically after navigating to a specific directory. The trick here is to **make bash interactive** by appending `-i`

```powershell
CMD /C wt.exe --title "Services" wsl -d "Ubuntu-20.04" -- bash -i -c "cd ~/ && sudo service postgresql start && sudo service docker start && exec bash";
```

### Printing to Console

You can print to console by just using echo, just don't forget to open an interactive shell afterwards so it doesn't closes.

```powershell
CMD /C wt.exe --title "Demo" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && echo 'hello world' && exec bash"
```

### Open interactive Bash

Sometimes you want to have an interactive bash screen and not just your program running. For this just append `exec bash` behind it:

```powershell
CMD /C wt.exe --title "Demo" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && exec bash";
```

Elevate your developer experience with Windows Terminal and WSL

Windows Terminal and WSL Tips and Tricks


So you are working on a production scale and suddenly notice that you don't have to provision just 1, not 10 but 1000s of devices. What do you do?

This is where you start automating stuff, rather than doing everything manually, you ideally:

1. Create an image
2. Flash an SD Card
3. Install the SD Card
4. Boot-up the device

Applying this to Azure IoT Edge, I will walk you through how you can create your own Azure IoT Edge image for your devices. This way we can follow a previous blog post to [automatically register our devices on boot with IoT Hub !](/posts/iot/iot-dps)

## Packer Installation

To do so, we will be utilizing a tool named [Packer](https://www.packer.io/). Packer is a tool created by Hashicorp to create "Golden Images" for a target platform through **Builders**. These *builders* are responsible for generating an image for that target platform. (e.g. we create an Image and save it to Azure, we create an Image to run on Hyper-V, ...)

* [VirtualBox](https://www.packer.io/docs/builders/virtualbox)
* [Hyper-V](https://www.packer.io/docs/builders/hyperv)
* [Azure](https://www.packer.io/docs/builders/azure)
* [AWS](https://www.packer.io/docs/builders/amazon)
* [GCP](https://www.packer.io/docs/builders/googlecompute)
* [ARM](https://github.com/mkaczanowski/packer-builder-arm)
* ...

## Installing Packer with ARM support

In our case, we want to utilize Packer to create an image that can run on our target devices which are ARM based. Packer didn't have this option natively, but provided a [community supported module that can do just this](https://github.com/mkaczanowski/packer-builder-arm).

### Packer

Let's get started on creating our own image by installing Packer itself.

```bash
# Install Go (required for Packer)
# See: https://github.com/canha/golang-tools-install-script
wget -q -O - https://git.io/vQhTU | bash

# Install dependencies
sudo apt install -y git unzip qemu qemu-user-static e2fsprogs dosfstools

# Install Packer
# see version at: https://www.packer.io/downloads
PACKER_VERSION=1.7.4
wget "https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip"
unzip packer_${PACKER_VERSION}_linux_amd64.zip 
sudo mv packer /usr/local/bin/
```

### ARM for Packer

Since ARM is not natively installed, we will have to install a plugin for this. 

```bash
# Download the Packer ARM Builder
git clone https://github.com/mkaczanowski/packer-builder-arm

# Build it
cd packer-builder-arm
go mod download
go build

# Install to /usr/local/bin (packer will discover plugins with prefix packer-*)
sudo mv packer-builder-arm /usr/local/bin
```

## Configuring

Once packer is installed and the ARM plugin was added we can get started on creating our own custom image. Let's recap on how our image should look like:

1. **Use a Jetson Nano with Ubuntu:** We want to utilize a base Ubuntu image as our source compiled for a Jetson nano.
2. **Add a custom script for IoT Edge with DPS configuration:** We utilize DPS to set-up our device, this script will take in the ScopeID and Primary Key of a Group Enrollment and automatically provision our Device in IoT Hub.
3. **Basic System Configuration:** We want to configure time settings, ssh access, admin users, cleanup, ... so we add that as well.

Create a folder structure that looks like the below:

```bash                  
.
├── jetson-azure-iot-edge.json    # Packer Configuration Details
├── scripts
│   ├── 00-clean-nvidia-jetson.sh # Cleanup Nvidia Jetson base image (remove GUI)
│   ├── 01-configure-system.sh    # Configure the system (hostname, DNS, upgrading)
│   ├── 02-configure-ssh.sh       # Allow SSH Access
│   ├── 03-configure-chrony.sh    # Time Synchronisation
│   ├── 04-create-admin-user.sh   # Admin Login
│   └── 05-install-iot-edge.sh    # IoT Edge Configuration on Boot
```

The main `jetson-azure-iot-edge.json` file will contain the configuration for how Packer should build our image. Here we will detail that:

* Variables for our scripts
* It should use the Nvidia Jetson Nano SD Card Image
* Copy our scripts to `/tmp/packer/scripts`
* Execute the different scripts with the variables passed as environment variables at build.

> For the full source code, feel free to check my [Public Repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/Azure-IoT-Edge/PackerImage/NvidiaJetsonNano)

Our Packer JSON Image (`jetson-azure-iot-edge.json`) will then look like:

**jetson-azure-iot-edge.json:**

```javascript
{
    "variables": {
        "dps_group_enrollment_key": "NOT_SET_PASS_BY_CLI",
        "dps_scope_id": "NOT_SET_PASS_BY_CLI",
        "admin_create": "true",
        "admin_user": "admin",
        "admin_group": "admin",
        "admin_passwd": "admin",
        "admin_uid": "1000",
        "admin_gid": "1000",
        "admin_groups": "cdrom,floppy,sudo,audio,dip,video,plugdev,netdev,systemd-journal",
        "admin_shell": "/bin/bash",
        "admin_gecos": "Administrator",
        "admin_ssh_authorised_key": ""
    },
    "builders": [
        {
            "type": "arm", // The builder type
            "file_urls": [ // Our base image URL
                "https://developer.nvidia.com/jetson-nano-sd-card-image-r3221"
            ],
            "file_checksum": "7b87e26d59c560ca18692a1ba282d842", // Our base image checksum
            "file_checksum_type": "md5", // Our base image checksum type
            "file_target_extension": "zip", // The extension/compression of the image
            "image_build_method": "reuse", // Reuse disk image or create from scratch?
            "image_path": "jetson-nano.img", // Where to unpack to
            "image_size": "12G", // The size of the image
            "image_type": "dos", // Partition Table Scheme
            "image_partitions": [ // Specifications of the partitions of the image
                {
                    "name": "root",
                    "type": "83",
                    "start_sector": "24576",
                    "filesystem": "ext4",
                    "size": "12G",
                    "mountpoint": "/"
                }
            ],
            "image_chroot_env": [ // The shell environment that is passed to chroot
                "PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"
            ],
            "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static", // Where is qemu situated?
            "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static" // Where will copy it to?
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "mkdir -p /tmp/packer/scripts"
            ]
        },
        {
            "type": "file",
            "source": "./scripts",
            "destination": "/tmp/packer/"
        },
        {
            "type": "shell",
            "remote_folder": "/tmp/packer",
            "skip_clean": true,
            "environment_vars": [
                "ADMIN_CREATE={{user `admin_create`}}",
                "ADMIN_GECOS={{user `admin_gecos`}}",
                "ADMIN_GID={{user `admin_gid`}}",
                "ADMIN_GROUP={{user `admin_group`}}",
                "ADMIN_GROUPS={{user `admin_groups`}}",
                "ADMIN_PASSWD={{user `admin_passwd`}}",
                "ADMIN_SHELL={{user `admin_shell`}}",
                "ADMIN_SSH_AUTHORISED_KEY={{user `admin_ssh_authorised_key`}}",
                "ADMIN_UID={{user `admin_uid`}}",
                "ADMIN_USER={{user `admin_user`}}",
                "DPS_GROUP_ENROLLMENT_KEY={{user `dps_group_enrollment_key`}}",
                "DPS_SCOPE_ID={{user `dps_scope_id`}}"
            ],
            "scripts": [
                "./scripts/00-clean-nvidia-jetson.sh",
                "./scripts/01-configure-system.sh",
                "./scripts/02-configure-ssh.sh",
                "./scripts/03-configure-chrony.sh",
                "./scripts/04-create-admin-user.sh",
                "./scripts/05-install-iot-edge.sh"
            ]
        }
    ],
    "post-processors": []
}
```

## Building

Finally, to build our image it is as simple as running the command below. Which sets the variables as defined in the variables section, more specifically variables `dps_scope_id` and `dps_group_enrollment_key`.

We can adapt the other variables as well the same way by passing them in the CLI. Otherwise, Packer will use their default values.

```bash
sudo packer build -var 'dps_scope_id=<YOUR_DPS_SCOPE_ID>' -var 'dps_group_enrollment_key=<YOUR_DPS_GROUP_KEY>' jetson-nano.json 
```

This should take around 12 minutes (mainly due to removal of packages and upgrading the system), but once that is finishes you will have a `jetson-nano.img` image created that can be flashed to an SD Card!

## Conclusion

After following this article, you should be left with an Image that can be installed on an SD Card. Which, when booted completely configures a brand new device and registers it with the IoT Device Provisioning Service (DPS). All without us having to log in on the device, configure network, install DPS, configure DPS, configure hostname, ... 

If we would have to do the above on 1000s of devices, this would save us ~10min per device (the build time) thus saving ~10.000 minutes!

## Troubleshooting

When working with complex software as the above (more specifically `qemu` virtualization), errors are bound to happen. In my case I receive the following errors sometimes:

* **==> arm: Failed to find binfmt_misc for /usr/bin/qemu-aarch64-static under /proc/sys/fs/binfmt_misc**
  * To resolve this I had to reinstall the `qemu-user-static` environment with `sudo apt install --reinstall qemu-user-static` more information can be found in the [WSL Interop Issue](https://github.com/microsoft/WSL/issues/2103)

## References

I always love to include some of the references I checked that were precious in me writing this. Below you can find the ones I used for this blog post.

* [https://linuxhit.com/build-a-raspberry-pi-image-packer-packer-builder-arm/](https://linuxhit.com/build-a-raspberry-pi-image-packer-packer-builder-arm/)
* [https://www.thegeekdiary.com/how-to-create-virtual-block-device-loop-device-filesystem-in-linux/](https://www.thegeekdiary.com/how-to-create-virtual-block-device-loop-device-filesystem-in-linux/)

Learn how you can create your own OS Golden Image to setup your IoT Edge Devices and get them provisioned automatically!

Creating an Azure IoT Edge Golden Image


Once you get further in the world of the Internet of Things (IoT) you will notice that you have to setup devices in bulk rather than device-per-device. This is where Azure helps you through a service named **Device Provisioning Service**.

## Requirements

* Azure IoT Hub (Standard Tier or higher)
* A device with Ubuntu installed

## The Device Provisioning Process

Before we get started with creating and configuring DPS, let's go through the basics of how Device Provisioning works.

> Device Provisioning is the process of getting a device ready for hand-off to an end-user (i.e. the device is ready for bootup and usage)

In the case of provisioning a device with DPS we have 2 main steps:

1. The creation of the device with a bare OS
2. The configuration of the OS to connect to cloud 

While step 1 is trivial, step 2 is considerably more complex as a lot of security aspects have to be taken into account. The diagram below explains [how Step 2 is handled through DPS](https://docs.microsoft.com/en-us/azure/iot-dps/about-iot-dps?WT.mc_id=DT-MVP-5004263):

![](https://docs.microsoft.com/en-us/azure/iot-dps/media/about-iot-dps/dps-provisioning-flow.png)

1. Device manufacturer adds the device registration information to the enrollment list in the Azure portal.
2. Device contacts the DPS endpoint set at the factory. The device passes the identifying information to DPS to prove its identity.
3. DPS validates the identity of the device by validating the registration ID and key against the enrollment list entry using either a nonce challenge (Trusted Platform Module) or standard X.509 verification (X.509).
4. DPS registers the device with an IoT hub and populates the device's desired twin state.
5. The IoT hub returns device ID information to DPS.
6. DPS returns the IoT hub connection information to the device. The device can now start sending data directly to the IoT hub.
7. The device connects to IoT hub.
8. The device gets the desired state from its device twin in IoT hub.

## Creating and Configuring DPS

### Creating DPS and Linking IoT Hub

Now we know how device provisioning works, let's create and configure our DPS.

To get started we need to create and configure the Device Provisioning Service (DPS). To do this, open up the Azure Portal, search for "Device Provisioning Service" and create one.

![](./dps-create-1.png)

Once the Device Provisioning Service has been created, we need to **link our IoT Hub** to it. So open it up and navigate to "Linked IoT Hubs" and add the created IoT Hub from the requirements.

![](./dps-create-2.png)

### Creating a Device Enrollment

We now have an IoT Hub linked to the DPS and are able to get started on **creating an *Enrollment* for our device.** Open up the Azure Portal under DPS and navigate towards "Manage Enrollments"

![](./dps-configure-1.png)

Here we can pick Individual Enrollment or Group Enrollment:
* **Individual Enrollment:** This is for a single device
* **Group Enrollment:** This is for a group of devices sharing a common attestation (i.e. how will the devices confirm their identity)

Personally I find it more interesting to cover **Group Enrollments** as this is what we would typically see in production environments. So let's continue with this one.

Head over to **Add Enrollment Group** and select the configuration you want to use. Here I used the configuration to pick an *Edge Device*, *Symmetric Keys* and give it a specific *Name*.

![](./dps-configure-2.png)

⚠️ Finally, don't click save yet, but **configure the *Initial Device Twin State*** to contain a tag. In our case I picked: `"deviceType": "nvidia-cluster"`

> We will utilize this tag later to let IoT Hub know which modules should be provisioned to which set of devices. DPS will help us connect the device to the correct IoT Hub with the correct initial state, but IoT Hub will add our modules.

![](./dps-configure-3.png)

## Configuring our IoT Edge Modules to be Deployed

Ok, so let's recap. We now have:

1. Created a DPS 
2. Configured it to link to our IoT Hub
3. Created an enrollment for a group of devices based on an attestation
4. Configured it so the devices provisioned get a specific Twin State with them

If we would connect a device now, we will see that it will get provisioned with the tag we provided earlier (`"deviceType": "nvidia-cluster"`). Now we have to use this tag to let IoT Hub know which modules should get deployed on this set of devices.

Open up your IoT Hub and go towards **IoT Edge** and click **Add Deployment** in the IoT Edge Deployments tab.

![](./iothub-deployment-1.png)

For the initial name, feel free to pick something unique for yourself.

![](./iothub-deployment-create-1.png)

The next step is the first of the 2 most important steps. Here we will define which modules should get deploy. So open up the "Add" pane and add an "IoT Edge Module"

> Note: here you can also configure Azure Container Registry credentials if you need those.

![](./iothub-deployment-create-2.png)

In our case we want a simple simulation, so add the public Simulation device from the Docker Hub

* **Image URI:** `mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0`

![](./iothub-deployment-create-3.png)

Afterwards, we specify the routes (send everything upstream)

![](./iothub-deployment-create-4.png)

Now the second most important part comes: **specifying the target devices**. Here we specify which devices should get these modules. In our case we will thus configure `tags.deviceType = 'nvidia-cluster'` as we defined before (remember where we put the initial twin state to have `"tags": { "deviceType": "nvidia-cluster" }`)

![](./iothub-deployment-create-5.png)

Finally, validate the deployment and you will get redirect back to your IoT Hub and see your deployment standing (but with 0 devices for now!)

![](./iothub-deployment-create-6.png)
![](./iothub-deployment-create-7.png)

## Configuring our Devices

As a last step, we need to configure our devices to know where they should connect to and fetch their information. We could do this manually for each device, but for production workloads this is a long cumbersome task. Therefore, we will utilize the [iot-edge-config](https://github.com/Azure/iot-edge-config?WT.mc_id=DT-MVP-5004263) tool that will make our lives so much easier!

> Note: since we are using a group enrollment key, we have to [derive a new key from it](https://docs.microsoft.com/en-us/azure/iot-edge/how-to-auto-provision-symmetric-keys?view=iotedge-2020-11&tabs=linux&WT.mc_id=DT-MVP-5004263#derive-a-key-on-linux)

Open up your device and run the following commands:

```bash
# DPS Enrollment Group Key
DPS_GROUP_ENROLLMENT_KEY=<YOUR_GROUP_ENROLLMENT_KEY>
DPS_SCOPE_ID=<YOUR_DPS_SCOPE_ID>

# Get the MAC Address
MAC_ADDR=$(ifconfig eth0 | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}' | sed -En "s/:/-/gp")

# Specify the Registration ID
# Note: in our case we utilize the Mac Address
REG_ID=$MAC_ADDR

# Compute a Derived key
keybytes=$(echo $DPS_GROUP_ENROLLMENT_KEY | base64 --decode | xxd -p -u -c 1000)
DERIVED_KEY=$(echo -n $REG_ID | openssl sha256 -mac HMAC -macopt hexkey:$keybytes -binary | base64)

# Configure
wget https://github.com/Azure/iot-edge-config/releases/latest/download/azure-iot-edge-installer.sh -O azure-iot-edge-installer.sh 
chmod +x azure-iot-edge-installer.sh 
sudo -H ./azure-iot-edge-installer.sh -s $DPS_SCOPE_ID -r $MAC_ADDR -k $DERIVED_KEY
rm -rf azure-iot-edge-installer.sh
```

![](./device-install-2.png)

We can now check the status by running `sudo iotedge system logs` which should not show any errors. If we then open our DPS and go to Enrollment Group Details -> Registration Records we can see our device being registered.

![](./device-install-3.png)

Finally, open up IoT Hub and go to IoT Edge devices. There you will now see your device automatically being provisioned! After a couple of minutes, the modules should as well be available on it.

![](./device-install-4.png)

Seeing that we configured a simulated sensor to be deployed, we can open up our VSCode, go to the IoT Hub Extension and listen for incoming messages to view our messages.

> Note: The simulated sensor module sends 1 message every 5 seconds and automatically stops after 500 to preserve your quota. It could be that you have to restart the container if it stops.

![](./iothub-view-messages-1.png)

![](./iothub-view-messages-2.png)

![](./iothub-view-messages-3.png)


Learn how you can automatically provision your IoT Device using Azure Device Provisioning Service (DPS)

Automatically Provision an IoT Device with DPS

Troubleshoot your Azure IoT Edge solution with the useful commands and locations!

Azure IoT Edge Troubleshooting


Autonomous Platforms are being deployed all around us, optimizing our daily processes and improving their efficiency and stability without the need of human intelligence. They learn by themself, iteratively improving themselves and staying ahead of the competition.

One of the biggest hurdles for Autonomous Platforms to be deployed is the access to a simulation environment that can be configured to the real-world use case and scaled to millions of instances to train on. Achieving 100s of iterations per second in training.

This is where the [Azure Project Bonsai platform](https://www.microsoft.com/en-us/ai/autonomous-systems-project-bonsai) comes into the picture, helping customers to teach their machines to achieve super-human optimization.

Through the following articles, I introduce you to how you can get started yourself with this amazing topic:

* [1. Platform Introduction & Overview](/post/ai/rl/bonsai/introduction) (+ [update!](/post/ai/rl/bonsai/introduction-revisiting))
* [2. Create and integrate your first "Brain" - CartPole Problem](/post/ai/rl/bonsai/brain-integration)
* 3 Create your own Simulation Environment from scratch - Self-Balancing Cube Problem [Coming Soon]
* 4 3D Model and Teach your own Robot to Walk - Custom Robotics Project [Coming Soon]

Learn how you can create an autonomous system through Machine Teaching with the Azure Project Bonsai platform.

Machine Teaching through Azure Project Bonsai - Series Overview

In a [previous article](/posts/infrastructure/azure-arc) I went in detail on how you can get started running your own Azure Arc setup and connect your cross-cloud and on-premises systems and Kubernetes clusters. In this article I would like to go in more detail how you are now able to utilize the previously created infrastructure and run Azure App Service on it! Bringing the ecosystem of Azure towards your multi-cloud environment.

## Overview

I'd love to explain the above through a graphic as illustrated below. Here you have several components:
* Multi-Cloud / Hybrid Cloud environment (illustrated as "Hosting" below)
* Azure Application Services with Azure Arc (the top block)

Making it clear that Arc is there to enable us to run Azure Application Services across our cloud environments, no matter the infrastructure layer beneath it. 

![](./arc-overview.png)

## Hands-On

After the above, let's actually do this ourselves! To deploy an Azure Service we need to have an Azure Arc enabled cluster or server available and execte the following steps:

1. Enable an Extension (e.g. App Service)
2. Create a Custom Location (our own "cloud" location!)
3. Create an App Service

### Enable an Extension (App Service)

First we start by enabling an extension on the target cluster. For this we open up the cluster in Azure Arc, navigate to "Extensions" and find the extension we want to install. In our case we will install the App Service extension.

![](./azure-arc-cluster-extensions.png)
![](./azure-arc-cluster-extensions-add-1.png)
![](./azure-arc-cluster-extensions-add-2.png)

Now we get to the details pane, where a public IP is required. This is the static IP created when you create the Kubernetes Cluster, to find this in GCP we can simply navigate to our GKE instance and locate the public IP endpoint.

![](./azure-arc-cluster-extensions-add-5-enter-details.png)
![](./azure-arc-cluster-extensions-add-3-gcp-public-ip-finding.png)
![](./azure-arc-cluster-extensions-add-4-gcp-public-ip-finding.png)

All details are now entered and a script will be generated for us. Execute this by copying this to a PowerShell instance of your choice that has access to the cluster and AZ CLI.

![](./azure-arc-cluster-extensions-add-6-run-script.png)
![](./azure-arc-cluster-extensions-add-7-copy-script-to-az-shell.png)

### Create a Custom Location

Since the Extension is now enabled, we are able to create a custom location. Navigate to Azure Arc "Custom Locations" and click the "Add" button.

![](./azure-arc-custom-location-overview.png)

This will open up the interface to add a custom location. Just navigate through this, selecting the settings you require.

![](./azure-arc-custom-location-create.png)
![](./azure-arc-custom-location-create-2.png)
![](./azure-arc-custom-location-create-3-created.png)

Once this has been created, we can verify that it has been created by navigating to a `kubectl` shell and checking the deployed pods. We will see a new namespace being created containing the resources.

![](./azure-arc-custom-location-create-4-created-gcp-overview.png)

### App Service Creation

We are almost there! So far we created a App Service enabled Kubernetes cluster on GCP and added the GCP Kubernetes cluster as a target location for Azure App Service instances. Finally we need to deploy App Service on it! 

Open Azure Arc and navigate towards "App Service" on the left, creating it in the previously created custom location with the default image.

![](./azure-arc-appsvc-1.png)
![](./azure-arc-appsvc-2.png)
![](./azure-arc-appsvc-3.png)
![](./azure-arc-appsvc-4.png)
![](./azure-arc-appsvc-5-deployed.png)

**Congratulations**, App Service is now up and running on your own instance! To verify it, open up the GKE console again and verify that an extra pod is running.

![](./azure-arc-appsvc-6-gcp-confirm.png)



## Conclusion

As shown in this article, it is trivial to deploy your own Azure Services across your existing hardware! This opens up a lot of scenarios where customers can connect their existing infrastructure running across multiple clouds and easily managing it from an easy to understand portal! Personally, I am a big fan of this solution and cannot wait for it to expand and improve more. Some of my personal learnings:

* Some errors exist and might be difficult to work with (e.g. I had a stamp issue as well as a Storage Class not existing, requiring some manual intervention - which can be automated). Tickets are being raised and the service is maturing more and more (it is still in preview!)
* I would love to see more automation happening. Currently a lot of the commands have to be ran manually, but it is prone to making mistakes.

In any case, if you have existing infrastructure and want a good control plane, Azure Arc is definitely a promising service to check out!

Learn how you can deploy Azure App Service across your Multi-Cloud environment on GCP through the Azure Arc control plane

Xavier Geerinck

Did you enjoy reading? Or do you want to stay up-to-date of new Articles?