
In my [previous post](/post/2021/12/20/coding-ai-dapr-container-apps) I explained how you can get started with Container Apps. The one missing piece of that puzzle is how you can utilize a queue behind to dynamically scale your workload based on items in a queue! This has been one my most difficult blog items so far, taking me through a range of different issues with Container Apps and Dapr, but which definitely aided those services and myself in becoming more mature. 

The following items have been created because of this:

* [Dapr: Python SDK - Input Bindings for HTTP](https://github.com/dapr/python-sdk/issues/342)
* [Log Analytics: Difference between TimeGenerated and _timestamp_d #79](https://github.com/microsoft/azure-container-apps/issues/79)
* [Support for setting Dapr App Protocol to gRPC](https://github.com/microsoft/azure-container-apps/issues/70)
* [Container Apps seems to be missing appProtocol](https://github.com/Azure/azure-rest-api-specs/issues/17144)
* [Azure Container Apps not Updating Docker Container](https://github.com/pulumi/pulumi-azure/issues/1007)
* [Metadata (and queryString?) should use an Array Type instead of Tuple Type](https://github.com/dapr/python-sdk/issues/328)
* [Azure Container App Demonstration](https://github.com/dapr/python-sdk/issues/316)

This entire process took me around 2 months to complete, but once figured out it's super simple!

## Source Code

The entire source code for this project can be found on my [Public GitHub repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/Azure-Container-Apps/AI-Model-YoloV5-Dapr-Queue)

## Using a Queue - Made easy through Dapr

We will be using a queue in our source code. Luckily for us, this can be done easily by utilizing [Dapr](https://dapr.io) which takes away a lot of the implementation for the different queueing systems and their retry logic. For this project, I decided to utilize the [Service Bus Queue](https://docs.dapr.io/reference/components-reference/supported-bindings/servicebusqueues/).

To set this up, we create a component spec such as the below:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: my-queue
  namespace: default
spec:
  type: bindings.azure.servicebusqueues
  version: v1
  metadata:
  - name: connectionString # Required
    value: "YOUR_CONNECTION_STRING"
  - name: queueName
    value: worker-items
  - name: ttlInSeconds
    value: 60
```

> **Note:** The component spec for Container Apps is defined differently, but more about that soon.

Once this is created, we can hook up our application to the queue by creating an [Input Binding](https://docs.dapr.io/developing-applications/building-blocks/bindings/howto-triggers/) that will monitor the Service Bus queue and call the defined POST endpoint. When looking to the component spec above, we defined the name `my-queue` which will result in a POST call to `/my-queue`. 

The code below shows us how we can process items from this binding.

```python
from pydantic import BaseModel
from fastapi import FastAPI
from dapr.ext.fastapi import DaprApp

# Define Request DTO
class WorkerItemRequestDto(BaseModel):
    id: str
    image_url: str

# Add our Routes
@app.options("/my-queue")
async def processor_worker_item_options():
    return

@app.post("/my-queue")
async def processor_worker_item(item: WorkerItemRequestDto):
    return
```

## Implementing YoloX

Now we know how to get items in, we just have to initialize our model and run the item through it. As we did before, we initialize the model, download our image and print the result:

```python
# First load the model as we will always need this
print("Loading Model", flush=True)
model = torch.hub.load("ultralytics/yolov5", "yolov5s", pretrained=True)
model.eval()

# ... Other code

@app.post("/my-queue")
async def processor_worker_item(item: WorkerItemRequestDto):
    return handler_image_url(item.image_url)

# Handling the Image URL
def handler_image_url(image_url):
    print(f'Received: image_url="{image_url}"', flush=True)

    if not image_url:
        return

    # Download the image
    response = requests.get(image_url)
    img = Image.open(io.BytesIO(response.content))

    # Infer result
    results = model(img, size=640)
    bboxes = results.pandas().xyxy[0].to_json(orient="records")
    bboxes = json.loads(bboxes)

    # Print the Result
    items = {}
    for bbox in bboxes:
        items[bbox["name"]] = 1 if bbox["name"] not in items else items[bbox["name"]] + 1
    
    print(f'Results:', flush=True)
    for key, value in items.items():
        print(f'- {key}: {value}', flush=True)
        
    return
```

## Running Locally

The above we can run locally by utilizing the `dapr run` command:

```bash
# Send 5 Events to Service Bus Queue
dapr run --app-id=send-events --app-protocol http --components-path=./example/components python example/send.py 5

# Start Server to Process the Events
cd app/code/src
dapr run --app-id=yolox --app-protocol http --components-path=../../../example/components -- uvicorn main:app --host 0.0.0.0 --port 5000
```

Resulting in our application receiving events and processing them! 🤩

![](./demo-local.gif)

## Running in Cloud with Container Apps

Now the power of all of this is that we can enable Serverless AI! To achieve this, we will utilize [Azure Container Apps](https://azure.microsoft.com/en-us/services/container-apps/) that run our container whenever there is an item in our queue. When there are no items in the queue, we use a [KEDA scaler](https://keda.sh) to scale down to 0 instances!

The below illustrates how we utilize secrets that configure the Dapr Component as well as the KEDA scaler:

```javascript
const containerApp = new web.ContainerApp("app", {
  resourceGroupName: resourceGroup.name,
  location: glbLocation,
  kubeEnvironmentId: kubeEnv.id,
  configuration: {
    // ... see source code
    activeRevisionsMode: ActiveRevisionsMode.Single,
    secrets: [
      {
        name: "acr-pwd",
        value: outAcrAdminPassword,
      },
      {
        name: "sb-connection-string",
        value: serviceBusNamespaceKeys.primaryConnectionString,
      }
    ],
  },
  template: {
    revisionSuffix: `${(new Date()).getTime()}`,
    containers: [{
      name: `container-${GLB_APP_ID}`,
      image: image.name,
      env: [
        // ... see source code
      ]
    }],
    // https://keda.sh/docs/2.5/scalers/azure-service-bus/
    scale: {
      minReplicas: 0,
      maxReplicas: 3,
      rules: [{
        name: "rule-servicebus-scaler",
        custom: {
          type: "azure-servicebus",
          metadata: {
            queueName: "worker-items",
            messageCount: "2",
          },
          auth: [{
            secretRef: "sb-connection-string",
            triggerParameter: "connection"
          }]
        }
      }]
    },
    dapr: {
      enabled: true,
      appId: GLB_APP_ID,
      appPort: GLB_APP_PORT,
      // appProtocol: 'http', // Disabled for now since unsupported by ARM CLI
      components: [
        {
          name: 'my-queue',
          type: 'bindings.azure.servicebusqueues',
          version: 'v1',
          metadata: [
            {
              name: 'connectionString',
              secretRef: "sb-connection-string"
            },
            {
              name: 'queueName',
              value: 'worker-items'
            },
            {
              name: 'ttlInSeconds',
              value: '60'
            }
          ]
        }
      ]
    }
  }
});
```

To set up this infrastructure, we can simply run the following command:

```bash
pulumi up \                                                                                                
    --stack demo-cae-yolox \
    -c glb-location=northeurope \
    -c glb-project-name=demo-cae-yolox \
    -c glb-project-env=prd \
    -c sb-topics="worker-items"
```

> For setting up the entire infrastructure, I refer to the Pulumi file created in the [source code](https://github.com/XavierGeerinck/PublicProjects/blob/master/Azure-Container-Apps/AI-Model-YoloV5-Dapr-Queue/index.ts).

We can now view the logs of our application in Log Analytics with the query:

```bash
ContainerAppConsoleLogs_CL 
    | project TimeCreated = unixtime_milliseconds_todatetime(_timestamp_d * 1000), ContainerAppName_s, Log_s
    | order by TimeCreated desc 
    | take 150
```

![](./demo-azure.png)

## Conclusion

This article might not sound as something worth spending 2 months on! However, when looking at the benefits Container Apps brings in the field of Serverless AI, it's well worth spending this time! We can now finally run models, inference a result and shut it down again without needing to have a compute instance running 24/7 which can end up being expensive when using GPUs.

I hope you enjoyed this article, feel free to put a comment below or contact me!


Serverless AI is a game changer in reducing operational costs! But how can we achieve this? Learn more about it here

Enabling Serverless AI with Azure Container Apps and Dapr


Recently I've been working a lot with different AI Models for a variety of use cases, ranging from simple object classification to massive crowd counting models. While working with these models, I saw that it is always difficult to implement a model in custom code since the Architecture is never included in the weights themselves, thus each model works differently.

That's when I came across a format named [ONNX](https://onnx.ai/) which instantly peaked my interest with promising points such as:
* Improved Inference Performance 
* Interoperability

## Prerequisites

To get started with ONNX, we have two tasks that should be done

1. Configure the [ONNX Execution Provider](https://onnxruntime.ai/docs/execution-providers/)
2. Get an ONNX Model to infer

### Configuring the ONNX Execution Provider for CUDA 

To be able to run on our GPU I chose to utilize the [CUDA Execution Provider](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html). Which has 2 major components that should be installed:

* CUDA
* cuDNN

#### Installing CUDA

CUDA is the easiest of the two to install since it is publically available without a login wall. Before we start, check if CUDA is already installed by running the below, which should finish correctly:

```bash
# Check if CUDA is available
nvidia-smi
nvcc --version
```

If the above is not available, we can install it (**on WSL 2**) by running the below:

> If you want to install it on another target, feel free to check the original [link](https://developer.nvidia.com/cuda-toolkit-archive)

```bash
# View all: https://developer.nvidia.com/cuda-toolkit-archive
# View ONNX Supported: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0

# Install CUDA
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo apt-key add /var/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

# Add CUDA to Path
# https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
# Note: add those to ~/.bashrc at the bottom
export PATH=/usr/local/cuda-11.5/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

> Note: Do not forget to add the export links to your ~/.bashrc file

#### Installing cuDNN

Now up to the other requirement, to install `cuDNN`.

```bash
OS="wsl-ubuntu"
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0
CUDNN_VERSION_MAJOR=8
CUDNN_VERSION_MINOR=2
CUDNN_VERSION_PATCH=4

wget https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/cuda-${OS}.pin 

sudo mv cuda-${OS}.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/ /"
sudo apt-get update
sudo apt-get install libcudnn8=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
sudo apt-get install libcudnn8-dev=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
```

## Running the YOLOX Model

Now we have all the required software installed for the CUDA Execution Provider, we can start running a model and infer from it. Luckily for us, the [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) team provides a [ready-to-use ONNX model](https://github.com/Megvii-BaseDetection/YOLOX/tree/main/demo/ONNXRuntime) that we can use. So let's download it and use it!

To keep the code part short, you can find the entire project on [my public repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/AI/ONNX/yolox) the main part to know is how we can infer an ONNX Model.

Infering an ONNX Model is as easy as creating a session that defines the model path as well as the provider to use

```python
# Create an ONNX Inference Session
session = onnxruntime.InferenceSession(
    path_or_bytes=args.model,
    providers=["CUDAExecutionProvider"]
)
```

After that is done, we just need to provide it an input:

```python
ort_inputs = {session.get_inputs()[0].name: img[None, :, :, :]}
output = session.run(None, ort_inputs)
```

Whereafter we can run all of this and receive our result!

```bash
# Create directories
mkdir model; mkdir images; mkdir output

# Download Model
wget -O ./model/yolox_l.onnx https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_l.onnx 

# Download Example
wget -O ./images/example.jpg https://www.ikea.com/ext/ingkadam/m/7f8f282fb240f466/original/PH179208-crop001.jpg?f=m

# Run
python src/main.py -m ./model/yolox_l.onnx -i ./images/example.jpg -o ./output -s 0.3 --input_shape 640,640
```

## Conclusion

As you can see from the above, it is quite easy to infer an ONNX Model and get results from it! I personally think I will utilize ONNX more in the future, reducing the complexity of my AI pipeline codes even further.

Learn how to run AI inferencing using ONNX on your GPU

AI Model Inferencing with ONNX on GPU (CUDA)


For a new project I am working on, I am utilizing a Queuing architecture where jobs come in and a cluster of containers is processing work on this queue. Each job takes an item from a queue, runs the item through its AI pipeline and spits out a JSON response.

Setting up an infrastructure to accomodate the above is quite "interesting" as it requires:

* Creating a Kubernetes cluster
* Creating and hosting a queueing solution
* Autoscaling your cluster on a hardware level depending on the items in your queue
* Allowing Microservice cross invocation

Typically we would solve the above by utilizing a Queue (e.g. RabbitMQ), a Kubernetes cluster that autoscales (e.g. with with KEDA) and an Image that pulls from this queue in an abstracted way (e.g. with Dapr) with additionally cross Microservice invocation with Dapr.

In the above, the main annoying part stays: "Scaling your hardware to a 0-usage based on a queue". KEDA makes this easy, but it's still annoying and requires some work.

Luckily this is where ["Container Apps" comes in!](https://docs.microsoft.com/en-gb/azure/container-apps/get-started)

Let's see how we can utilize Container Apps to expose the YoloV5 model through an API.

## Building a YoloV5 Container with API

### Code

Create a `main.py` file with the following code, that will pull the model from Torch Hub and run the model on port 5000 with endpoint `/v1/object-detection/yolov5s`

```python
import argparse
import io
from PIL import Image

import torch
from flask import Flask, request

app = Flask(__name__)

@app.route("/", methods=["GET"])
def hello():
    return {
        "hello": "world"
    }

@app.route("/v1/object-detection/yolov5s", methods=["POST"])
def predict():
    if not request.method == "POST":
        return

    if request.files.get("image"):
        image_file = request.files["image"]
        image_bytes = image_file.read()

        img = Image.open(io.BytesIO(image_bytes))

        results = model(img, size=640)
        data = results.pandas().xyxy[0].to_json(orient="records")
        return data


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Flask api exposing yolov5 model")
    parser.add_argument("--port", default=5000, type=int, help="port number")
    args = parser.parse_args()

    model = torch.hub.load("ultralytics/yolov5", "yolov5s", pretrained=True, force_reload=True).autoshape()
    model.eval()
    app.run(host="0.0.0.0", port=args.port) 
```

As a last piece of code, we add our `requirements.txt`:

```
flask
requests
black

matplotlib>=3.2.2
numpy>=1.18.5
opencv-python>=4.1.2
Pillow
PyYAML>=5.3.1
scipy>=1.4.1
torch>=1.7.0
torchvision>=0.8.1
tqdm>=4.41.0

tensorboard>=2.4.1

seaborn>=0.11.0
pandas

thop  # FLOPs computation
```

### Packaging it

Finally, we can package the above as a container by creating a `Dockerfile` with our content:

```bash
FROM python:3.8-slim-buster

RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6  -y

WORKDIR /app
ADD requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt

WORKDIR /app
ADD . /app

EXPOSE 5000

CMD ["python", "main.py", "--port=5000"]
```

This can build the container by running:

```bash
docker build -t api-yolov5 .
```

To make things easier, I have already [pushed this container to the docker hub.](https://hub.docker.com/repository/docker/thebillkidy/api-yolov5)

## Creating our Azure Container Apps

Now our container has been created, we can get started hosting it! To do this, let's first configure our Azure Container App access in the CLI and then publish our container above.

### Prerequisites

First configure our AZ CLI to include the extension and be logged in:

```bash
# Login
az login
az account set --subscription <id>

# Add CLI Extensions
az extension add \
  --source https://workerappscliextension.blob.core.windows.net/azure-cli-extension/containerapp-0.2.0-py2.py3-none-any.whl 

# Register the Web Provider
az provider register --namespace Microsoft.Web
```

### Create

```bash
RESOURCE_GROUP="demo"
LOCATION="northeurope"
LOG_ANALYTICS_WORKSPACE="my-container-apps-logs"
CONTAINERAPPS_ENVIRONMENT="my-environment"

az group create --name $RESOURCE_GROUP --location "$LOCATION"

az monitor log-analytics workspace create \
  --resource-group $RESOURCE_GROUP \
  --workspace-name $LOG_ANALYTICS_WORKSPACE

# Get Log Analytics Client ID and Client Secret
LOG_ANALYTICS_WORKSPACE_CLIENT_ID=`az monitor log-analytics workspace show --query customerId -g $RESOURCE_GROUP -n $LOG_ANALYTICS_WORKSPACE --out tsv`
LOG_ANALYTICS_WORKSPACE_CLIENT_SECRET=`az monitor log-analytics workspace get-shared-keys --query primarySharedKey -g $RESOURCE_GROUP -n $LOG_ANALYTICS_WORKSPACE --out tsv`

# Create Container App Environment
# this is where all the container apps are deployed
az containerapp env create \
  --name $CONTAINERAPPS_ENVIRONMENT \
  --resource-group $RESOURCE_GROUP \
  --logs-workspace-id $LOG_ANALYTICS_WORKSPACE_CLIENT_ID \
  --logs-workspace-key $LOG_ANALYTICS_WORKSPACE_CLIENT_SECRET \
  --location "$LOCATION"

# Create our Container App
az containerapp create \
  --name my-container-app \
  --resource-group $RESOURCE_GROUP \
  --environment $CONTAINERAPPS_ENVIRONMENT \
  --image thebillkidy/api-yolov5:latest \
  --target-port 5000 \
  --ingress 'external' \
  --query configuration.ingress.fqdn
```

## Testing it

Once we ran the last command (`az containerapp create`) we will get a URL back that looks like this: `my-container-app.unique_id.region.azurecontainerapps.io`. When we call it in the browser, we will get:

```json
{
    "hello": "world"
}
```

Which means it is up and running! Executing the inference endpoint now, we can run: 

```bash
curl -X POST -F image=@demo.jpg 'https://my-container-app.unique_id.region.azurecontainerapps.io/v1/object-detection/yolov5s'
```

The above will send the local `demo.jpg` picture to our endpoint and return the infered result, which in this case shows that we have 4 chairs, 4 potted plants, 2 persons and 1 couch on this image!

```bash
[
    {"xmin":103.2421798706,"ymin":292.3749694824,"xmax":437.8124694824,"ymax":615.8124389648,"confidence":0.8725585938,"class":56,"name":"chair"},
    {"xmin":774.8436889648,"ymin":246.4374847412,"xmax":1126.8748779297,"ymax":567.5311889648,"confidence":0.841796875,"class":56,"name":"chair"},
    {"xmin":0.3515624702,"ymin":244.3281097412,"xmax":150.3515472412,"ymax":483.6249389648,"confidence":0.8344726562,"class":56,"name":"chair"},
    {"xmin":752.3436889648,"ymin":97.0234298706,"xmax":821.7186889648,"ymax":220.5390472412,"confidence":0.8227539062,"class":0,"name":"person"},
    {"xmin":536.2499389648,"ymin":107.1015548706,"xmax":659.0624389648,"ymax":210.4609222412,"confidence":0.7124023438,"class":0,"name":"person"},
    {"xmin":356.0155944824,"ymin":218.4296722412,"xmax":984.3749389648,"ymax":491.1249389648,"confidence":0.6181640625,"class":57,"name":"couch"},
    {"xmin":85.8984298706,"ymin":154.0937347412,"xmax":185.5077972412,"ymax":284.8749694824,"confidence":0.5854492188,"class":58,"name":"potted plant"},
    {"xmin":674.5311889648,"ymin":121.9843673706,"xmax":729.8436889648,"ymax":205.1874847412,"confidence":0.384765625,"class":58,"name":"potted plant"},
    {"xmin":233.5546722412,"ymin":220.0702972412,"xmax":367.0312194824,"ymax":321.9062194824,"confidence":0.2978515625,"class":56,"name":"chair"},
    {"xmin":440.3905944824,"ymin":0.0,"xmax":626.7186889648,"ymax":173.1952972412,"confidence":0.2956542969,"class":58,"name":"potted plant"},
    {"xmin":0.9082030654,"ymin":146.59375,"xmax":44.5898399353,"ymax":170.0312347412,"confidence":0.2568359375,"class":58,"name":"potted plant"}
]
```

## Conclusion

It has never been easier to create a workload that dynamically scales based on the incoming requests! In a next post, I will explain how we can combine this with [Dapr](https://docs.microsoft.com/en-us/azure/container-apps/microservices-dapr?tabs=bash) to get items from a queue instead of a HTTP endpoint.

Learn how you run your AI workload with Azure Container Apps, enabling autoscaling

Running your AI Workload through Azure Container Apps


For a use case I wanted to utilize the Nvidia Jetson for edge inference. One of the bottlenecks here was that my software required a Python version that is greater than 3.6. When looking at the [Nvidia Jetson packages for PyTorch](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch) it was seen that this was only created for version 3.6.

Searching around, I could find quite a lot of people struggling with this (@TODO: PROOF) which made me to look into **a solution** that would help me automate the building of ARM Wheel of **PyTorch that can run on Nvidia Devices** (e.g. Nvidia Jetson Nano) and thus **support CUDA**.

The entire process above took me around ~11 full days, starting of with figuring out how to build the Dockerfile and finally automating the CI process.

> Note: I also decided to utilize this article as an entry for a running Hackathon by [Dev.to for GitHub Actions](https://dev.to/devteam/join-us-for-the-2021-github-actions-hackathon-on-dev-4hn4)

You would wonder why we want to automate this? Well when compiling this initially on my Nvidia Jetson Nano, I couldn't get it to compile past ~80% when using 16Gb of Swap Space (the device only has 2 Gb). After building it on my personal PC, this took ~6h to get it to compile completely. Making it large and long enough to automate it.

> The final source code can be found on [GitHub](https://github.com/XavierGeerinck/Jetson-Linux-PyTorch)

## Contributions

In any kind of project of this size, there are specific contributions that were made. In my project I believe to have made the following:

* Install CUDA on non-GPU devices
* Compile PyTorch with CUDA enabled on non-GPU devices
* Compile PyTorch for Python > 3.6
* Build for ARM with CI through Docker Buildx

## Project Outline

As a best practice, I always love to include the project outline of how I tackled the issue above (to share my thought process):

1. Create a Dockerfile that builds the Wheel
    * How do I build for CUDA? (Hardest part)
    * How do I cross build?
2. Create GitHub Action
    * How do I cross build? Can I run for ARM specifically?

## Dockerfile Creation - Building PyTorch for ARM and Python > 3.6

The hardest part BY FAR is how to compile PyTorch for ARM and Python > 3.6 with CUDA enabled. Because we are going to run on a non-GPU device, thus CUDA is not available on there. I splitted up the Dockerfile into 3 specific sections that can be ran in parallel:

1. Set up CUDA
2. Set up PyTorch (the cloning takes a while)
3. Set up Python 3.9
4. Compile PyTorch
    * I have Nvidia Jetson optimisations included here, thanks [QEngineering](https://qengineering.eu/install-pytorch-on-jetson-nano.html)!
4. Dockerfile Result

Below you can find the explanation of the respective steps. For the final `Dockerfile` optimizations where made to decrease the docker layer sizes (by grouping RUN in 1 command)

### Setting up CUDA

For CUDA, we do not have the CUDA libraries, nor do we have access to them! There is however a trick that allows us to get CUDA loaded in. We copy over the public key of NVIDIA Jetson and we authorize ourselves to the repository. Then we use our package manager to install them:

```bash
V_CUDA_DASH=10-2

# Add the public key
echo "[Builder] Adding the Jetson Public Key"
curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc
echo "deb https://repo.download.nvidia.com/jetson/common ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
echo "deb https://repo.download.nvidia.com/jetson/t186 ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

# Install the CUDA Libraries
echo "[Builder] Installing CUDA System"
apt-get update
apt-get install -y --no-install-recommends \
    cuda-libraries-$V_CUDA_DASH \
    cuda-libraries-dev-$V_CUDA_DASH \
    cuda-nvtx-$V_CUDA_DASH \
    cuda-minimal-build-$V_CUDA_DASH \
    cuda-license-$V_CUDA_DASH \
    cuda-command-line-tools-$V_CUDA_DASH \
    libnvvpi1 vpi1-dev

# Link CUDA to /usr/local/cuda
ln -s /usr/local/cuda-$CUDA /usr/local/cuda
```

When we eventually start compiling, we will see CUDA enabled in our CMAKE output 🥳

```bash
#   USE_CUDA              : ON
#     Split CUDA          : OFF
#     CUDA static link    : OFF
#     USE_CUDNN           : OFF
#     USE_EXPERIMENTAL_CUDNN_V8_API: OFF
#     CUDA version        : 10.2
#     CUDA root directory : /usr/local/cuda
#     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
#     cudart library      : /usr/local/cuda/lib64/libcudart.so
#     cublas library      : /usr/local/cuda/lib64/libcublas.so
#     cufft library       : /usr/local/cuda/lib64/libcufft.so
#     curand library      : /usr/local/cuda/lib64/libcurand.so
#     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
#     CUDA include path   : /usr/local/cuda/include
#     NVCC executable     : /usr/local/cuda/bin/nvcc
#     NVCC flags          : <CUT>
#     CUDA host compiler  : /usr/bin/clang
#     NVCC --device-c     : OFF
#     USE_TENSORRT        : OFF
```

### Setting up PyTorch

In a separate docker step, we set up PyTorch and clone it to the working repository (in our case `/build/pytorch`)

```bash
V_PYTORCH=v1.10.0

# Downloads PyTorch to /build/pytorch
git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch /build/pytorch
```

### Setting up Python 3.9

We configure our Python version through the `deadsnakes` ppa and link it as the default one.

> Best practice we should have a `venv` but since I am running it in a Docker container this should suffice.

```bash
# Setting up Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python3 \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python
```

### Compiling PyTorch

The last step in the Dockerfile is to compile PyTorch. For this we set the correct environment variabels to enable CUDA and to optimise the building process by disabling some other parts (e.g. MKLDNN, NNPACK, XNNPACK, ... to be turned off).

We also configure it to use `clang` as the Nvidia Jetson has [NEON registers](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_462) and clang supports those (GCC doesn't).

For our source, we utilize the other layer we created and just copy it from there.

```dockerfile
COPY --from=downloader-pytorch /build/pytorch /build/pytorch
WORKDIR /build/pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Build
rm build/CMakeCache.txt || : \
sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
pip install -r requirements.txt
python setup.py bdist_wheel
```

### Copying the result as an Artifact

Docker Buildx is amazing in the sense that we can utilize the `--output type-local,dest=.` command to output files to our local filesystem, making it such that docker builds and we can export the build result as an artifact.

To achieve this, we pull from the `scratch` image and copy over our result to it from the other docker layer. Our `/` path will then contain all the build wheels of PyTorch (e.g. `torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl`)

```dockerfile
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

### Dockerfile Result

Finally, the full Dockerfile will look like this:

```dockerfile
# ##################################################################################
# Setup Nvidia CUDA for Jetson
# ##################################################################################
FROM ubuntu:18.04 as cuda-devel

# Configuration Arguments
ARG V_CUDA_MAJOR=10
ARG V_CUDA_MINOR=2
ARG V_L4T_MAJOR=32
ARG V_L4T_MINOR=6

ENV V_CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}
ENV V_CUDA_DASH=${V_CUDA_MAJOR}-${V_CUDA_MINOR}
ENV V_L4T=r${V_L4T_MAJOR}.${V_L4T_MINOR}

# Expose environment variables everywhere
ENV CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Fix CUDA info
ARG DPKG_STATUS

# Add NVIDIA repo/public key and install VPI libraries
RUN echo "$DPKG_STATUS" >> /var/lib/dpkg/status \
    && echo "[Builder] Installing Prerequisites" \
    && apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates software-properties-common curl gnupg2 apt-utils \
    && echo "[Builder] Installing CUDA Repository" \
    && curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc \
    && echo "deb https://repo.download.nvidia.com/jetson/common ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "deb https://repo.download.nvidia.com/jetson/t186 ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "[Builder] Installing CUDA System" \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
    cuda-libraries-${V_CUDA_DASH} \
    cuda-libraries-dev-${V_CUDA_DASH} \
    cuda-nvtx-${V_CUDA_DASH} \
    cuda-minimal-build-${V_CUDA_DASH} \
    cuda-license-${V_CUDA_DASH} \
    cuda-command-line-tools-${V_CUDA_DASH} \
    libnvvpi1 vpi1-dev \
    && ln -s /usr/local/cuda-${V_CUDA} /usr/local/cuda \
    && rm -rf /var/lib/apt/lists/*

# Update environment
ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs

RUN ln -fs /usr/share/zoneinfo/Europe/Brussels /etc/localtime

# ##################################################################################
# Create PyTorch Docker Layer
# We do this seperately since else we need to keep rebuilding
# ##################################################################################
FROM --platform=$BUILDPLATFORM ubuntu:18.04 as downloader-pytorch

# Configuration Arguments
# https://github.com/pytorch/pytorch
ARG V_PYTORCH=v1.10.0
# https://github.com/pytorch/vision
ARG V_PYTORCHVISION=v0.11.1
# https://github.com/pytorch/audio
ARG V_PYTORCHAUDIO=v0.10.0

# Install Git Tools
RUN apt-get update \
    && apt-get install -y --no-install-recommends software-properties-common apt-utils git \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Clone Source
RUN git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch

# ##################################################################################
# Build PyTorch for Jetson (with CUDA)
# ##################################################################################
FROM cuda-devel as builder

# Configuration Arguments
ARG V_PYTHON_MAJOR=3
ARG V_PYTHON_MINOR=9

ENV V_PYTHON=${V_PYTHON_MAJOR}.${V_PYTHON_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Download Common Software
RUN apt-get update \
    && apt-get install -y clang build-essential bash ca-certificates git wget cmake curl software-properties-common ffmpeg libsm6 libxext6 libffi-dev libssl-dev xz-utils zlib1g-dev liblzma-dev

# Setting up Python 3.9
WORKDIR /install

RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python${V_PYTHON_MAJOR} \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python

# PyTorch - Build - Source Code Setup 
# copy everything from the downloader-pytorch layer /torch to /torch on this one
COPY --from=downloader-pytorch /pytorch /pytorch
WORKDIR /pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Set path to ccache
ARG PATH=/usr/lib/ccache:$PATH

# Other arguments
ARG USE_CUDA=ON
ARG USE_CUDNN=ON
ARG BUILD_CAFFE2_OPS=0
ARG USE_FBGEMM=0
ARG USE_FAKELOWP=0
ARG BUILD_TEST=0
ARG USE_MKLDNN=0
ARG USE_NNPACK=0
ARG USE_XNNPACK=0
ARG USE_QNNPACK=0
ARG USE_PYTORCH_QNNPACK=0
ARG TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
ARG USE_NCCL=0
ARG USE_SYSTEM_NCCL=0
ARG USE_OPENCV=0
ARG USE_DISTRIBUTED=0

# Build
RUN cd /pytorch \
    && rm build/CMakeCache.txt || : \
    && sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
    && pip install -r requirements.txt \
    && python setup.py bdist_wheel \
    && cd ..

# ##################################################################################
# Prepare Artifact
# ##################################################################################
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

## GitHub Action Creation

Since PyTorch is now finally compiling! It is time to start automating this and publishing them to an Artifact in GitHub (this way we can always trigger it ourselves and kick of the building process). I want to start building automatically, as soon as a Release is published! So for our action, we have the following outline:

### Action Outline

1. When a release is created trigger the action
2. Clone the repository
3. Setup Docker with Buildx
4. Run our container
5. Copy over the Built Wheel to an artifact on GitHub

### Used Actions

As for actions, the following actions could be reused:

1. [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action)
    * I cross-compile for ARM on AMD64 machines in the pipeline
2. [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action)
    * Configure QEMU to be able to compile for ARM and install the QEMU static binaries
3. [actions/checkout](https://github.com/actions/checkout)
    * Check out a repo
4. [actions/cache](https://github.com/actions/cache)
    * Allow us to cache the docker layers
5. [actions/upload-artifact](https://github.com/actions/upload-artifact)
    * Upload the output of a directory to GitHub artifacts

### Result

Finally resulting in the following GitHub action:

```yaml
name: ci

# https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows
on:
  push:
    branches: [ main ]
  release:
    types: [ created ]

jobs:
  build_wheels:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v2
        
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v1
        
    - name: Set up Docker Buildx
      id: buildx
      uses: docker/setup-buildx-action@v1
    
    - name: Cache Docker layers
      uses: actions/cache@v2
      with:
      path: /tmp/.buildx-cache
      key: ${{ runner.os }}-buildx-${{ github.sha }}
      restore-keys: |
        ${{ runner.os }}-buildx-
    
    - name: Build Docker Image
      run: |
      docker buildx build \
          --platform=linux/arm64 \
          --progress=plain \
          --output type=local,dest=./wheels \
          --file Dockerfile.jetson .

    - name: Upload Artifacts
      uses: actions/upload-artifact@v2
      with:
      name: wheels
      path: |
        wheels/*.whl

    # This ugly bit is necessary if you don't want your cache to grow forever
    # till it hits GitHub's limit of 5GB.
    # Temp fix
    # https://github.com/docker/build-push-action/issues/252
    # https://github.com/moby/buildkit/issues/1896
    - name: Move cache
      run: |
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache
```

## Future Work

### Extra Optimizations to be made

Some support could still be added for the Nvidia Jetson Nano by adapting the source code, but this is currently out of scope of this project. These optimisations can be found in [QEngineering their post](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_425)

### GitHub Actions Improvements

Currently, Build Arguments are included but not yet used. In theory, the following can be added to the `docker buildx` command to build for other PyThon versions:

**Python 3.8**

```bash
docker buildx build \
    --platform=linux/arm64 \
    --progress=plain \
    --build-arg PYTHON_MAJOR=3 \
    --build-arg PYTHON_MINOR=8 \
    --output type=local,dest=./wheels \
    --file Dockerfile.jetson .
```

## Conclusion

This project was definitely not easy, taking a long time between builds, figuring out where to build, automating it, ... by sharing this I hope to help the community utilize GPUs more easily with the latest versions.

In a next article, I hope to show you how to run an actual AI model with CUDA enabled on the Nvidia Jetson nano and Python > 3.6 😉 

## References

All of the above was possible by some contributions of others:

* [https://github.com/soerensen3/buildx-pytorch-jetson](https://github.com/soerensen3/buildx-pytorch-jetson) helped me with some of the Dockerfile code (but didn't had CUDA support)






Learn how you can compile PyTorch to run on the Nvidia Jetson with a Python version > 3.6

Enabling CUDA with PyTorch on Nvidia Jetson and Python 3.9


The reworked Dapr Javascript SDK has been released a couple of weeks ago! Together with a brand new and easier to use SDK, we have been thinking in the community of reworking the documentation to simplify the examples, more specifically the Actors examples.

After some brainstorming together, we decided to follow a scenario that was explained by [Mark Russinovich](https://www.youtube.com/watch?v=eJCu6a-x9uo&t=3785) detailing **how you can create your own Parking Garage Spot Monitoring System!**

## Scenario

Microsoft is growing to 1.000.000 employees. To accomodate for this, they have parking garages to hold the cars of these employees. For simplicity reasons of the demonstration, we take into account 2 parking garages that can hold each 500.000 employee cars and each have their own sensor monitoring the spot. If a car drives into a spot, an event is sent that updates the value of the sensor itself. Each of the sensors is using Azure Sphere.

## Actors

Now we could implement this with a dedicated application for each of the sensors, bake in the application and let it have its own state management system. An easier system however is to have the sensor send out a packet that we can ingest and process. Since each sensor is "standalone" this should have its own isolated set of actions and state! Which fits the description of an "Actor".

## Approach

To implement this parking garage example, we need to do the following:

* Create an actor that represents the latest state of a sensor at any given time
* Create a database made for holding historical data (`influxdb`)
* Create a dashboard that shows our points with color codes on a map (`grafana`)

## Implementation

Let's start by implementing this! First we should define the actions our actor can take, in the case of a Parking Garage with sensors, we will have 2 methods: `carEnter` and `carLeave` to define the state of the sensor. So let's draw up the interface for this:

### Defining the Actor Methods

```javascript
export default interface ParkingSensorInterface {
  carEnter(): Promise<void>;
  carLeave(): Promise<void>;
}
```

### Defining the Actor Implementation

Once the interface has been created, we need to implement the actual logic of our actor. For this actor, we thus implement the interface described above and we add some extra helper methods.

**Outline:**

1. We create an `onActivate` method which acts as the constructor when the actor is created. This will initialize the mock location of our sensor in a parking garage (latitude and longitude) through the `generateRandomPoint` method.
2. We implement the actor interface methods `carEnter` and `carLeave`
    * `carEnter`: Will set our state to `true` defining that a car is parked
    * `carLeave`: Will set our state to `false` defining that a spot is free
3. We add binding calls to `influxdb` that when a state changes also sends the state to `influxdb` for visualization purposes.

> **Note:** the power of Dapr is exactly that of the above, we implement actors and need to sink a point to a database, rather than having to implement and configure yet another API we can simply call the client with the binding and that will take care of it for us!

**Code:**

```javascript
import { AbstractActor } from "dapr-client";
import ParkingSensorInterface from "./ParkingSensorInterface";

const STATE_NAME_PARKING_SENSOR_VALUE = "parking-sensor-value";
const STATE_NAME_PARKING_SENSOR_LOCATION_LAT = "parking-sensor-location-lat";
const STATE_NAME_PARKING_SENSOR_LOCATION_LNG = "parking-sensor-location-lng";

// Coordinates pool to pick from as center of the radius circle
// we take multiple so we can distribute points more
// this is the Microsoft Garage Building 99
// 47.64174525422637, -122.14032265575015
const COORDINATES = [
  {
    lat: 47.64167040541433, lng: -122.1398701709982
  },
  {
    lat: 47.64177786440955, lng: -122.14077604924732
  }
]

export default class ParkingSensorImpl extends AbstractActor implements ParkingSensorInterface {
  async carEnter(): Promise<void> {
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_VALUE, true);

    const sensorLocationLat = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT);
    const sensorLocationLng = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG);

    await this.getDaprClient().binding.send("binding-influxdb", "create", {
      measurement: "sensor-states",
      tags: `sensor=${this.getId().getId()}`,
      values: `lat=${sensorLocationLat},lng=${sensorLocationLng},isParked=1`
    });
  }

  async carLeave(): Promise<void> {
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_VALUE, true);

    const sensorLocationLat = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT);
    const sensorLocationLng = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG);

    await this.getDaprClient().binding.send("binding-influxdb", "create", {
      measurement: "sensor-states",
      tags: `sensor=${this.getId().getId()}`,
      values: `lat=${sensorLocationLat},lng=${sensorLocationLng},isParked=0`
    });
  }

  async getInfo(): Promise<object> {
    const sensorValue = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_VALUE);
    const sensorLocationLat = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT);
    const sensorLocationLng = await this.getStateManager().getState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG);

    return {
      sensorValue,
      sensorLocationLat,
      sensorLocationLng
    }
  }

  async onActivate(): Promise<void> {
    const coordIdx = Math.floor(Math.random() * COORDINATES.length);
    const lat = COORDINATES[coordIdx]["lat"];
    const lng = COORDINATES[coordIdx]["lng"];
    const spotLocation = this.generateRandomPoint({ lat, lng }, 20);

    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_VALUE, false);
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_LOCATION_LAT, spotLocation.lat);
    await this.getStateManager().setState(STATE_NAME_PARKING_SENSOR_LOCATION_LNG, spotLocation.lng);
  }

  generateRandomPoint(center: { lat: number, lng: number }, radius: number) {
    var x0 = center.lng;
    var y0 = center.lat;

    // Convert Radius from meters to degrees.
    var rd = radius / 111300;

    var u = Math.random();
    var v = Math.random();

    var w = rd * Math.sqrt(u);
    var t = 2 * Math.PI * v;
    var x = w * Math.cos(t);
    var y = w * Math.sin(t);

    var xp = x / Math.cos(y0);

    // Resulting point.
    return { 'lat': y + y0, 'lng': xp + x0 };
  }
}
```






## Creating our Application

We now have the actor implemented and created but we still need to call it to represent our sensors! So let's create a simple implementation that:

1. Starts a server with our Actor registered
2. Create X amount of actors representing Parking Spot Sensors
3. Set the Parking Spot Sensor randomly to active or inactive every `5s`

> **Note:** In a real-world scenario, we would call this actor based on a pub/sub that receives an event from the sensors. Thus completely separating logic from hardware.

```javascript
import { DaprServer, DaprClient } from "dapr-client";
import ParkingSensorImpl from "./ParkingSensorImpl";

const daprHost = "127.0.0.1";
const daprPort = "50000"; // Dapr Sidecar Port of this Example Server
const serverHost = "127.0.0.1"; // App Host of this Example Server
const serverPort = "50001"; // App Port of this Example Server

async function sleep(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function start() {
  const server = new DaprServer(serverHost, serverPort, daprHost, daprPort);
  const client = new DaprClient(daprHost, daprPort);

  console.log("===============================================================");
  console.log("SERVER INITIALIZING")
  console.log("===============================================================");
  await server.actor.init(); // Let the server know we need actors
  server.actor.registerActor(ParkingSensorImpl); // Register the actor
  await server.startServer(); // Start the server

  console.log("===============================================================");
  console.log("CLIENT EXECUTION");
  console.log("===============================================================");
  const amount = 100;
  console.log(`Creating ${amount} Actors`);
  for (let i = 0; i < amount; i++) {
    await client.actor.invoke("GET", ParkingSensorImpl.name, `parking-sensor-${i}`, "getInfo");
  }

  console.log("===============================================================");
  console.log("CHANGING STATE OF SENSORS");
  console.log("===============================================================");
  console.log("Waiting 10 seconds before starting");
  await sleep(10000);

  while (true) {
    console.log("Simulating cars entering and leaving for 5% of the population");

    const populationSize = Math.floor(amount * 0.05);

    // Randomly put 10 on parked
    for (let i = 0; i < populationSize; i++) {
      const id = Math.floor(Math.random() * amount);
      await client.actor.invoke("PUT", ParkingSensorImpl.name, `parking-sensor-${id}`, "carEnter");
    }

    // Randomly put 10 on empty
    for (let i = 0; i < populationSize; i++) {
      const id = Math.floor(Math.random() * amount);
      await client.actor.invoke("PUT", ParkingSensorImpl.name, `parking-sensor-${id}`, "carLeave");
    }

    await sleep(5000);
  }
}

start().catch((e) => {
  console.error(e);
  process.exit(1);
});
```


## Visualization

### Creating the Containers 

For our visualization we will create an `influxdb` and a `grafana` instance! To quickly boot up these containers in a *non-persistent* way, we can utilize the commands below:

```bash
# Run InfluxDB
# Note: it auto removes after shutdown
# Note: non-persistent volume, add "-v influxdb2:/var/lib/influxdb2" to make it persistent
docker run --rm -it -d \
    -e DOCKER_INFLUXDB_INIT_MODE=setup \
    -e DOCKER_INFLUXDB_INIT_USERNAME=admin \
    -e DOCKER_INFLUXDB_INIT_PASSWORD=MyAdmin@123! \
    -e DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=my-token \
    -e DOCKER_INFLUXDB_INIT_ORG=my-parking-garage \
    -e DOCKER_INFLUXDB_INIT_BUCKET=my-sensors \
    --net=host \
    --name influxdb \
    influxdb:2.0

# Run Telegraf
docker run --rm -it -d \
    --net=host \
    --name=telegraf \
    telegraf

# Run Grafana
# Note: non-persistent volume, add "-v influxdb2:/var/lib/influxdb2" to make it persistent
docker run --rm -it -d \
    --name=grafana \
    --net=host \
    grafana/grafana
```

> ⚠️ the above is not-persistent, add volume binds and remove `--rm` to make them persistent.

### Configuring the Data Source

Now open up `http://localhost:3000` in your browser to open up Grafana. Here we need to configure a datasource:

![](./grafana-configure-datasource.png)

![](./grafana-configure-datasource-2.png)

![](./grafana-configure-datasource-3.png)

![](./grafana-configure-datasource-4.png)

![](./grafana-configure-datasource-5.png)

### Creating the Dashboard

Above, we configured the data source, so let's now create our visualization for it! To keep this easy I have preconfigured the dashboard before that we can import into Grafana.

```json
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 1,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": null,
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 25,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "options": {
        "basemap": {
          "config": {},
          "type": "osm-standard"
        },
        "controls": {
          "mouseWheelZoom": true,
          "showAttribution": true,
          "showDebug": false,
          "showScale": false,
          "showZoom": true
        },
        "layers": [
          {
            "config": {
              "color": {
                "field": "isParked",
                "fixed": "dark-green"
              },
              "fillOpacity": 0.4,
              "shape": "circle",
              "showLegend": true,
              "size": {
                "fixed": 3,
                "max": 15,
                "min": 2
              }
            },
            "location": {
              "mode": "auto"
            },
            "type": "markers"
          }
        ],
        "view": {
          "id": "coords",
          "lat": 47.641728,
          "lon": -122.140278,
          "zoom": 19.24
        }
      },
      "targets": [
        {
          "query": "from(bucket: \"my-sensors\")\r\n  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\r\n  |> filter(fn: (r) => r[\"_measurement\"] == \"sensor-states\")\r\n  |> pivot(columnKey: [\"_field\"], rowKey: [\"_time\"], valueColumn: \"_value\")\r\n  |> group()\r\n  |> yield(name: \"last\")",
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "type": "geomap"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 31,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Parking Garage",
  "uid": "Px36rBv7z",
  "version": 5
}
```

For illustration purposes, this is the query that is used to create the visualization:

```
from(bucket: "my-sensors")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "sensor-states")
  |> pivot(columnKey: ["_field"], rowKey: ["_time"], valueColumn: "_value")
  |> group()
  |> yield(name: "last")
```

## Running

Everything has been implemented! So let's start up the demo! For that, simply follow these steps:

1. Run `npm run start:dapr` on your application to start the application
2.open the folder to the example and run: `npm run start:dapr` which will boot up the demo. Finally, open up the dashboard and see the demo running.

![](./result.png)


## Conclusion

By using Dapr we can easily create an implementation of a Parking Garage using its Actor Framework. Without having to worry about certain pesky configurations or coding that we otherwise would have to do.

![](./demo-small-parking-garage.gif)

Learn how you can get started to create your own Parking Garage spot monitoring system using Dapr, Actors and the Javascript SDK!

Parking Garage Sensor implementation using Dapr Actors


I have been a user of Vercel for a long time, but recently I wanted more control over the deployment process of my blog! Which is why I decided to move everything to Azure for this, seeing that they have a [new Static Site hosting option](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website?WT.mc_id=DOP-MVP-5004263) which is simply AMAZING (and free 😉).

## Typical Blog Setup

Typically when setting up a blog, we need to take care of a couple of things:

* Storage Account (expose this to allow public access)
* CDN (for fast worldwide delivery)
* CI/CD setup (e.g. GitHub Actions)

## Azure Static Site with Bicep

Now recently a new offering was made available that bundles all of the above and automatically bundles your application and publishes your website. The only thing we have to do is provide the repository URL and how to build our website! 🤯 Now let's create our Azure setup with an Infrastructure-as-a-Code tool named [Azure Bicep](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview?WT.mc_id=DOP-MVP-5004263)

### Creating Bicep Files

The first file we have to create is our main file that contains the modules we want to deploy and some general overview. In here we use `subscription scoping` to define what has to happen at a *subscription level*

> We split up subscription level resources and resource group level resources in different files. Resource groups are created at a subscription level, while resources are created at a resource group level.

Our `main.bicep` file will thus define some parameters and their defaults and create a resource group. After creating this resource group we will tell it to initiate our `static-website.bicep` module that takes care of creating the static website setup itself.

**main.bicep**

```bicep
param environment string = 'dev'
param location string = 'westeurope'
param resourcePrefix string = 'yourresourceprefix'

// Parameters for static website
param repositoryUrl string
param repositoryToken string // the repository token to setup actions workflow and api secrets (https://github.com/settings/tokens)
param repositoryBranch string = 'master'

// This file operates at a subscription level
targetScope = 'subscription'

resource MainRG 'Microsoft.Resources/resourceGroups@2021-04-01' = {
  name: 'rg-${resourcePrefix}-${environment}'
  location: location
}

module ModuleStaticWebsite 'static-website.bicep' = {
  // Change deployment context to RG
  name: 'static-website'
  scope: resourceGroup(MainRG.name)
  params: {
    environment: environment
    resourcePrefix: resourcePrefix
    repositoryUrl: repositoryUrl
    repositoryBranch: repositoryBranch
    repositoryToken: repositoryToken
  }
}
```

Now our resource group is created, we define our `static-website.bicep` module. This module is quite straightforward as we just have to define the SKU, the Tier and our repository settings as well as how to build our application!

> Note: I am using a Next.js deployment, so I define that my site output will be situated in the `out` folder starting from the `/` after building through `npm run build`

**static-website.bicep**

```bicep
param environment string
param resourcePrefix string
param location string = resourceGroup().location

param repositoryBranch string
param repositoryUrl string

// the repository token to setup actions workflow and api secrets (https://github.com/settings/tokens)
// needs to have repo and workflow access
param repositoryToken string
param skuName string = 'Free'
param skuTier string = 'Free'

// https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep
resource staticWebApp 'Microsoft.Web/staticSites@2021-01-15' = {
  name: 'ss-${resourcePrefix}-${environment}'
  location: location
  sku: {
    name: skuName
    tier: skuTier
  }
  properties: {
    provider: 'GitHub' // required to ensure success in follow up deployments
    repositoryUrl: repositoryUrl
    repositoryToken: repositoryToken
    branch: repositoryBranch
    buildProperties: {
      outputLocation: 'out' // next.js will export to the out directory
      appLocation: '/'
      apiLocation: ''
    }
  }
}

// Outputs
output deployment_token string = listSecrets(staticWebApp.id, staticWebApp.apiVersion).properties.apiKey
```

And that's it! 😱 in just these few lines I have automated the entire infrastructure definition and can run this.

> What is even more amazing is that the Static Website implementation on Azure will add a `.github` workflow that automatically deploys our website to it!

### Deploying Bicep Template

As a last step we have to deploy all of the code we just wrote. So execute the code below to deploy this to your Azure account:

```bash
# Login to Azure
az login

# Set Subscription
az account set --subscription <your_sub_id>

# Deploy Bicep
az deployment sub create --location westeurope --template-file ./main.bicep
```

## Conclusion

In almost under 10 minutes you should have been able to:

1. Deploy new Cloud Infrastructure
2. Take code from your (private) repository
3. Build this code
4. Deploy the result on a static website
5. Automate all of it through a CI/CD pipeline

Which is now powering my personal blog! 😍

Learn how you can automated your Blog Deployment with a CI/CD pipeline with Azure Bicep in just 10 minutes!

Use Bicep and Azure Static Site for your Blog Deployment


Did you find yourself ever having to set-up your development environment all the time? Well I did as well! What if I could tell you, you could save many minutes each day by automating this? Enter the world of Windows Terminal and WSL! 😎

## Requirements

### WSL Environment

For this post we will need an initialized WSL environment. To get started feel free to follow my [previous blog post](https://xaviergeerinck.com/post/coding/python/windows-development-environment-setup) which gives you full control of where you want WSL to be installed! 

### Windows Terminal

If you don't have it yet (you should) go download [Windows Terminal](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701) or install it with `winget install Microsoft.WindowsTerminal`

## Tips and Tricks

### Automating Development Environment Startup

Windows Terminal has a lot of [CLI tricks up its sleeves](https://docs.microsoft.com/en-us/windows/terminal/command-line-arguments?tabs=windows&WT.mc_id=DT-MVP-5004263), one of these is that you can use 1 single command to open up your entire development environment. Typically, I make a shortcut with the following command in it that opens up my environment, starts processes and lets me know how I can start my other programs:

```powershell
CMD /C wt.exe --title "Services" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && sudo service postgresql start && sudo service docker start && exec bash"; new-tab --title "Site" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/frontend && sleep 3 && npm run dev"; split-pane -V wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && sleep 3 && npm run dev"; new-tab --title "Flow Engine" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/backend/dotnet/API && sleep 1 && ./start.sh"; focus-tab -t 0
```

In the following sections I will dissect some of the tricks used above:

### Starting a Service

When WSL boots, not all your services are started automatically. You want to do this but not always adapt the `~/.bashrc` file, so add yourself to sudoers first so you don't have to enter a password all the time and then execute the line below on Windows.

```powershell
# on WSL
echo "$USER ALL=(ALL:ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$USER

# on Windows
CMD /C wt.exe --title "Services" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && sudo service postgresql start && sudo service docker start && exec bash";
```

### Starting a Program

Besides starting a service you also might want to start your program automatically after navigating to a specific directory. The trick here is to **make bash interactive** by appending `-i`

```powershell
CMD /C wt.exe --title "Services" wsl -d "Ubuntu-20.04" -- bash -i -c "cd ~/ && sudo service postgresql start && sudo service docker start && exec bash";
```

### Printing to Console

You can print to console by just using echo, just don't forget to open an interactive shell afterwards so it doesn't closes.

```powershell
CMD /C wt.exe --title "Demo" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && echo 'hello world' && exec bash"
```

### Open interactive Bash

Sometimes you want to have an interactive bash screen and not just your program running. For this just append `exec bash` behind it:

```powershell
CMD /C wt.exe --title "Demo" wsl -d "Ubuntu-20.04" -- bash -c "cd ~/ && exec bash";
```

Elevate your developer experience with Windows Terminal and WSL

Windows Terminal and WSL Tips and Tricks


So you are working on a production scale and suddenly notice that you don't have to provision just 1, not 10 but 1000s of devices. What do you do?

This is where you start automating stuff, rather than doing everything manually, you ideally:

1. Create an image
2. Flash an SD Card
3. Install the SD Card
4. Boot-up the device

Applying this to Azure IoT Edge, I will walk you through how you can create your own Azure IoT Edge image for your devices. This way we can follow a previous blog post to [automatically register our devices on boot with IoT Hub !](/post/2021/8/12/azure-iot-iot-dps)

## Packer Installation

To do so, we will be utilizing a tool named [Packer](https://www.packer.io/). Packer is a tool created by Hashicorp to create "Golden Images" for a target platform through **Builders**. These *builders* are responsible for generating an image for that target platform. (e.g. we create an Image and save it to Azure, we create an Image to run on Hyper-V, ...)

* [VirtualBox](https://www.packer.io/docs/builders/virtualbox)
* [Hyper-V](https://www.packer.io/docs/builders/hyperv)
* [Azure](https://www.packer.io/docs/builders/azure)
* [AWS](https://www.packer.io/docs/builders/amazon)
* [GCP](https://www.packer.io/docs/builders/googlecompute)
* [ARM](https://github.com/mkaczanowski/packer-builder-arm)
* ...

## Installing Packer with ARM support

In our case, we want to utilize Packer to create an image that can run on our target devices which are ARM based. Packer didn't have this option natively, but provided a [community supported module that can do just this](https://github.com/mkaczanowski/packer-builder-arm).

### Packer

Let's get started on creating our own image by installing Packer itself.

```bash
# Install Go (required for Packer)
# See: https://github.com/canha/golang-tools-install-script
wget -q -O - https://git.io/vQhTU | bash

# Install dependencies
sudo apt install -y git unzip qemu qemu-user-static e2fsprogs dosfstools

# Install Packer
# see version at: https://www.packer.io/downloads
PACKER_VERSION=1.7.4
wget "https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip"
unzip packer_${PACKER_VERSION}_linux_amd64.zip 
sudo mv packer /usr/local/bin/
```

### ARM for Packer

Since ARM is not natively installed, we will have to install a plugin for this. 

```bash
# Download the Packer ARM Builder
git clone https://github.com/mkaczanowski/packer-builder-arm

# Build it
cd packer-builder-arm
go mod download
go build

# Install to /usr/local/bin (packer will discover plugins with prefix packer-*)
sudo mv packer-builder-arm /usr/local/bin
```

## Configuring

Once packer is installed and the ARM plugin was added we can get started on creating our own custom image. Let's recap on how our image should look like:

1. **Use a Jetson Nano with Ubuntu:** We want to utilize a base Ubuntu image as our source compiled for a Jetson nano.
2. **Add a custom script for IoT Edge with DPS configuration:** We utilize DPS to set-up our device, this script will take in the ScopeID and Primary Key of a Group Enrollment and automatically provision our Device in IoT Hub.
3. **Basic System Configuration:** We want to configure time settings, ssh access, admin users, cleanup, ... so we add that as well.

Create a folder structure that looks like the below:

```bash                  
.
├── jetson-azure-iot-edge.json    # Packer Configuration Details
├── scripts
│   ├── 00-clean-nvidia-jetson.sh # Cleanup Nvidia Jetson base image (remove GUI)
│   ├── 01-configure-system.sh    # Configure the system (hostname, DNS, upgrading)
│   ├── 02-configure-ssh.sh       # Allow SSH Access
│   ├── 03-configure-chrony.sh    # Time Synchronisation
│   ├── 04-create-admin-user.sh   # Admin Login
│   └── 05-install-iot-edge.sh    # IoT Edge Configuration on Boot
```

The main `jetson-azure-iot-edge.json` file will contain the configuration for how Packer should build our image. Here we will detail that:

* Variables for our scripts
* It should use the Nvidia Jetson Nano SD Card Image
* Copy our scripts to `/tmp/packer/scripts`
* Execute the different scripts with the variables passed as environment variables at build.

> For the full source code, feel free to check my [Public Repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/Azure-IoT-Edge/PackerImage/NvidiaJetsonNano)

Our Packer JSON Image (`jetson-azure-iot-edge.json`) will then look like:

**jetson-azure-iot-edge.json:**

```javascript
{
    "variables": {
        "dps_group_enrollment_key": "NOT_SET_PASS_BY_CLI",
        "dps_scope_id": "NOT_SET_PASS_BY_CLI",
        "admin_create": "true",
        "admin_user": "admin",
        "admin_group": "admin",
        "admin_passwd": "admin",
        "admin_uid": "1000",
        "admin_gid": "1000",
        "admin_groups": "cdrom,floppy,sudo,audio,dip,video,plugdev,netdev,systemd-journal",
        "admin_shell": "/bin/bash",
        "admin_gecos": "Administrator",
        "admin_ssh_authorised_key": ""
    },
    "builders": [
        {
            "type": "arm", // The builder type
            "file_urls": [ // Our base image URL
                "https://developer.nvidia.com/jetson-nano-sd-card-image-r3221"
            ],
            "file_checksum": "7b87e26d59c560ca18692a1ba282d842", // Our base image checksum
            "file_checksum_type": "md5", // Our base image checksum type
            "file_target_extension": "zip", // The extension/compression of the image
            "image_build_method": "reuse", // Reuse disk image or create from scratch?
            "image_path": "jetson-nano.img", // Where to unpack to
            "image_size": "12G", // The size of the image
            "image_type": "dos", // Partition Table Scheme
            "image_partitions": [ // Specifications of the partitions of the image
                {
                    "name": "root",
                    "type": "83",
                    "start_sector": "24576",
                    "filesystem": "ext4",
                    "size": "12G",
                    "mountpoint": "/"
                }
            ],
            "image_chroot_env": [ // The shell environment that is passed to chroot
                "PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin"
            ],
            "qemu_binary_source_path": "/usr/bin/qemu-aarch64-static", // Where is qemu situated?
            "qemu_binary_destination_path": "/usr/bin/qemu-aarch64-static" // Where will copy it to?
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "mkdir -p /tmp/packer/scripts"
            ]
        },
        {
            "type": "file",
            "source": "./scripts",
            "destination": "/tmp/packer/"
        },
        {
            "type": "shell",
            "remote_folder": "/tmp/packer",
            "skip_clean": true,
            "environment_vars": [
                "ADMIN_CREATE={{user `admin_create`}}",
                "ADMIN_GECOS={{user `admin_gecos`}}",
                "ADMIN_GID={{user `admin_gid`}}",
                "ADMIN_GROUP={{user `admin_group`}}",
                "ADMIN_GROUPS={{user `admin_groups`}}",
                "ADMIN_PASSWD={{user `admin_passwd`}}",
                "ADMIN_SHELL={{user `admin_shell`}}",
                "ADMIN_SSH_AUTHORISED_KEY={{user `admin_ssh_authorised_key`}}",
                "ADMIN_UID={{user `admin_uid`}}",
                "ADMIN_USER={{user `admin_user`}}",
                "DPS_GROUP_ENROLLMENT_KEY={{user `dps_group_enrollment_key`}}",
                "DPS_SCOPE_ID={{user `dps_scope_id`}}"
            ],
            "scripts": [
                "./scripts/00-clean-nvidia-jetson.sh",
                "./scripts/01-configure-system.sh",
                "./scripts/02-configure-ssh.sh",
                "./scripts/03-configure-chrony.sh",
                "./scripts/04-create-admin-user.sh",
                "./scripts/05-install-iot-edge.sh"
            ]
        }
    ],
    "post-processors": []
}
```

## Building

Finally, to build our image it is as simple as running the command below. Which sets the variables as defined in the variables section, more specifically variables `dps_scope_id` and `dps_group_enrollment_key`.

We can adapt the other variables as well the same way by passing them in the CLI. Otherwise, Packer will use their default values.

```bash
sudo packer build -var 'dps_scope_id=<YOUR_DPS_SCOPE_ID>' -var 'dps_group_enrollment_key=<YOUR_DPS_GROUP_KEY>' jetson-nano.json 
```

This should take around 12 minutes (mainly due to removal of packages and upgrading the system), but once that is finishes you will have a `jetson-nano.img` image created that can be flashed to an SD Card!

## Conclusion

After following this article, you should be left with an Image that can be installed on an SD Card. Which, when booted completely configures a brand new device and registers it with the IoT Device Provisioning Service (DPS). All without us having to log in on the device, configure network, install DPS, configure DPS, configure hostname, ... 

If we would have to do the above on 1000s of devices, this would save us ~10min per device (the build time) thus saving ~10.000 minutes!

## Troubleshooting

When working with complex software as the above (more specifically `qemu` virtualization), errors are bound to happen. In my case I receive the following errors sometimes:

* **==> arm: Failed to find binfmt_misc for /usr/bin/qemu-aarch64-static under /proc/sys/fs/binfmt_misc**
  * To resolve this I had to reinstall the `qemu-user-static` environment with `sudo apt install --reinstall qemu-user-static` more information can be found in the [WSL Interop Issue](https://github.com/microsoft/WSL/issues/2103)

## References

I always love to include some of the references I checked that were precious in me writing this. Below you can find the ones I used for this blog post.

* [https://linuxhit.com/build-a-raspberry-pi-image-packer-packer-builder-arm/](https://linuxhit.com/build-a-raspberry-pi-image-packer-packer-builder-arm/)
* [https://www.thegeekdiary.com/how-to-create-virtual-block-device-loop-device-filesystem-in-linux/](https://www.thegeekdiary.com/how-to-create-virtual-block-device-loop-device-filesystem-in-linux/)

Learn how you can create your own OS Golden Image to setup your IoT Edge Devices and get them provisioned automatically!

Creating an Azure IoT Edge Golden Image


Once you get further in the world of the Internet of Things (IoT) you will notice that you have to setup devices in bulk rather than device-per-device. This is where Azure helps you through a service named **Device Provisioning Service**.

## Requirements

* Azure IoT Hub (Standard Tier or higher)
* A device with Ubuntu installed

## The Device Provisioning Process

Before we get started with creating and configuring DPS, let's go through the basics of how Device Provisioning works.

> Device Provisioning is the process of getting a device ready for hand-off to an end-user (i.e. the device is ready for bootup and usage)

In the case of provisioning a device with DPS we have 2 main steps:

1. The creation of the device with a bare OS
2. The configuration of the OS to connect to cloud 

While step 1 is trivial, step 2 is considerably more complex as a lot of security aspects have to be taken into account. The diagram below explains [how Step 2 is handled through DPS](https://docs.microsoft.com/en-us/azure/iot-dps/about-iot-dps?WT.mc_id=DT-MVP-5004263):

![](https://docs.microsoft.com/en-us/azure/iot-dps/media/about-iot-dps/dps-provisioning-flow.png)

1. Device manufacturer adds the device registration information to the enrollment list in the Azure portal.
2. Device contacts the DPS endpoint set at the factory. The device passes the identifying information to DPS to prove its identity.
3. DPS validates the identity of the device by validating the registration ID and key against the enrollment list entry using either a nonce challenge (Trusted Platform Module) or standard X.509 verification (X.509).
4. DPS registers the device with an IoT hub and populates the device's desired twin state.
5. The IoT hub returns device ID information to DPS.
6. DPS returns the IoT hub connection information to the device. The device can now start sending data directly to the IoT hub.
7. The device connects to IoT hub.
8. The device gets the desired state from its device twin in IoT hub.

## Creating and Configuring DPS

### Creating DPS and Linking IoT Hub

Now we know how device provisioning works, let's create and configure our DPS.

To get started we need to create and configure the Device Provisioning Service (DPS). To do this, open up the Azure Portal, search for "Device Provisioning Service" and create one.

![](./dps-create-1.png)

Once the Device Provisioning Service has been created, we need to **link our IoT Hub** to it. So open it up and navigate to "Linked IoT Hubs" and add the created IoT Hub from the requirements.

![](./dps-create-2.png)

### Creating a Device Enrollment

We now have an IoT Hub linked to the DPS and are able to get started on **creating an *Enrollment* for our device.** Open up the Azure Portal under DPS and navigate towards "Manage Enrollments"

![](./dps-configure-1.png)

Here we can pick Individual Enrollment or Group Enrollment:
* **Individual Enrollment:** This is for a single device
* **Group Enrollment:** This is for a group of devices sharing a common attestation (i.e. how will the devices confirm their identity)

Personally I find it more interesting to cover **Group Enrollments** as this is what we would typically see in production environments. So let's continue with this one.

Head over to **Add Enrollment Group** and select the configuration you want to use. Here I used the configuration to pick an *Edge Device*, *Symmetric Keys* and give it a specific *Name*.

![](./dps-configure-2.png)

⚠️ Finally, don't click save yet, but **configure the *Initial Device Twin State*** to contain a tag. In our case I picked: `"deviceType": "nvidia-cluster"`

> We will utilize this tag later to let IoT Hub know which modules should be provisioned to which set of devices. DPS will help us connect the device to the correct IoT Hub with the correct initial state, but IoT Hub will add our modules.

![](./dps-configure-3.png)

## Configuring our IoT Edge Modules to be Deployed

Ok, so let's recap. We now have:

1. Created a DPS 
2. Configured it to link to our IoT Hub
3. Created an enrollment for a group of devices based on an attestation
4. Configured it so the devices provisioned get a specific Twin State with them

If we would connect a device now, we will see that it will get provisioned with the tag we provided earlier (`"deviceType": "nvidia-cluster"`). Now we have to use this tag to let IoT Hub know which modules should get deployed on this set of devices.

Open up your IoT Hub and go towards **IoT Edge** and click **Add Deployment** in the IoT Edge Deployments tab.

![](./iothub-deployment-1.png)

For the initial name, feel free to pick something unique for yourself.

![](./iothub-deployment-create-1.png)

The next step is the first of the 2 most important steps. Here we will define which modules should get deploy. So open up the "Add" pane and add an "IoT Edge Module"

> Note: here you can also configure Azure Container Registry credentials if you need those.

![](./iothub-deployment-create-2.png)

In our case we want a simple simulation, so add the public Simulation device from the Docker Hub

* **Image URI:** `mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0`

![](./iothub-deployment-create-3.png)

Afterwards, we specify the routes (send everything upstream)

![](./iothub-deployment-create-4.png)

Now the second most important part comes: **specifying the target devices**. Here we specify which devices should get these modules. In our case we will thus configure `tags.deviceType = 'nvidia-cluster'` as we defined before (remember where we put the initial twin state to have `"tags": { "deviceType": "nvidia-cluster" }`)

![](./iothub-deployment-create-5.png)

Finally, validate the deployment and you will get redirect back to your IoT Hub and see your deployment standing (but with 0 devices for now!)

![](./iothub-deployment-create-6.png)
![](./iothub-deployment-create-7.png)

## Configuring our Devices

As a last step, we need to configure our devices to know where they should connect to and fetch their information. We could do this manually for each device, but for production workloads this is a long cumbersome task. Therefore, we will utilize the [iot-edge-config](https://github.com/Azure/iot-edge-config?WT.mc_id=DT-MVP-5004263) tool that will make our lives so much easier!

> Note: since we are using a group enrollment key, we have to [derive a new key from it](https://docs.microsoft.com/en-us/azure/iot-edge/how-to-auto-provision-symmetric-keys?view=iotedge-2020-11&tabs=linux&WT.mc_id=DT-MVP-5004263#derive-a-key-on-linux)

Open up your device and run the following commands:

```bash
# DPS Enrollment Group Key
DPS_GROUP_ENROLLMENT_KEY=<YOUR_GROUP_ENROLLMENT_KEY>
DPS_SCOPE_ID=<YOUR_DPS_SCOPE_ID>

# Get the MAC Address
MAC_ADDR=$(ifconfig eth0 | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}' | sed -En "s/:/-/gp")

# Specify the Registration ID
# Note: in our case we utilize the Mac Address
REG_ID=$MAC_ADDR

# Compute a Derived key
keybytes=$(echo $DPS_GROUP_ENROLLMENT_KEY | base64 --decode | xxd -p -u -c 1000)
DERIVED_KEY=$(echo -n $REG_ID | openssl sha256 -mac HMAC -macopt hexkey:$keybytes -binary | base64)

# Configure
wget https://github.com/Azure/iot-edge-config/releases/latest/download/azure-iot-edge-installer.sh -O azure-iot-edge-installer.sh 
chmod +x azure-iot-edge-installer.sh 
sudo -H ./azure-iot-edge-installer.sh -s $DPS_SCOPE_ID -r $MAC_ADDR -k $DERIVED_KEY
rm -rf azure-iot-edge-installer.sh
```

![](./device-install-2.png)

We can now check the status by running `sudo iotedge system logs` which should not show any errors. If we then open our DPS and go to Enrollment Group Details -> Registration Records we can see our device being registered.

![](./device-install-3.png)

Finally, open up IoT Hub and go to IoT Edge devices. There you will now see your device automatically being provisioned! After a couple of minutes, the modules should as well be available on it.

![](./device-install-4.png)

Seeing that we configured a simulated sensor to be deployed, we can open up our VSCode, go to the IoT Hub Extension and listen for incoming messages to view our messages.

> Note: The simulated sensor module sends 1 message every 5 seconds and automatically stops after 500 to preserve your quota. It could be that you have to restart the container if it stops.

![](./iothub-view-messages-1.png)

![](./iothub-view-messages-2.png)

![](./iothub-view-messages-3.png)


Learn how you can automatically provision your IoT Device using Azure Device Provisioning Service (DPS)

Xavier Geerinck

Did you enjoy reading? Or do you want to stay up-to-date of new Articles?