
At my current customer [Proximus](https://proximus.com) I am working on putting Edge AI in use for the vision of "TODO-INSERT-VISION". To achieve this, we had to enable CUDA on the Nvidia Jetson devices that will analyze and infer our sensors.

## Prerequisites

Let's first go through the prerequisites. These are important as else we won't be able to correctly run our container.

### Requirements

1. Have [JetPack](https://developer.nvidia.com/embedded/jetpack) installed on your Nvidia Jetson Nano
2. Have [PyTorch compiled for Python 3.9](/post/2021/11/25/infrastructure-nvidia-ai-nvidia-building-pytorch)

### The Nvidia on Docker packages

To have GPU support in your containers, it is important to understand the different packages out there and how they work together in achieving this.

All packages below are summarized under the term `nvidia-docker` which refers to a collection of components:

* `libnvidia-container`: Main required package, it is `container-runtime` agnostic and provides a wrapper CLI that different runtimes can invoke to inject NVIDIA GPU support into their containers.
* `nvidia-container-toolkit`: Includes a sccript that implements the interface required by `runC` `prestart` hook.
* `nvidia-container-runtime`: Wrapper around `runC` and injects the `nvidia-container-toolkit` script in it.
* `nvidia-docker2`: Takes script associated with `nvidia-container-runtime` and installs it into docker's `/etc/docker/daemon.json` so we can run `docker run --runtime=nvidia ...`

It is important to note that all are needed even though some documentation links state only to install `nvidia-docker2`. This is because Kubernetes with Docker 19.03 needs nvidia-docker2 to be able to pass GPU information since it doesn't support the `--gpus` flag yet.

> 💡 More details can be found in [this post](https://github.com/NVIDIA/nvidia-docker/issues/1268#issuecomment-632692949) by Kevin Klues. 

Ensure the packages described above are available on the system

```bash
dpkg -l | grep nvidia
```

### Verification

To verify the above, we can utilize the following blocks that help us check CUDA.

#### Check Docker Runtime

Run the command below

```bash
sudo docker info | grep Runtime
```

That should show the below, with **Default Runtime: nvidia**

```
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia runc
 Default Runtime: nvidia
WARNING: No blkio weight support
WARNING: No blkio weight_device support
```

If this is not the case, make sure `nvidia-docker` is installed correctly and that the default runtime was configured in `/etc/docker/daemon.json` 

```json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
```

#### Nvidia Jetson Board Details

We can get detailed information of the board we are running through `jetsonhacks` its utilities. Run the following for this:

```bash
git clone https://github.com/jetsonhacks/jetsonUtilities
cd jetsonUtilities
python jetsonInfo.py
```

Which will give something such as:

```
NVIDIA Jetson Nano (Developer Kit Version)
 L4T 32.6.1 [ JetPack 4.6 ]
   Ubuntu 18.04.5 LTS
   Kernel Version: 4.9.253-tegra
 CUDA 10.2.300
   CUDA Architecture: 5.3
 OpenCV version: 4.1.1
   OpenCV Cuda: NO
 CUDNN: 8.2.1.32
 TensorRT: 8.0.1.6
 Vision Works: 1.6.0.501
 VPI: ii libnvvpi1 1.1.11 arm64 NVIDIA Vision Programming Interface library
 Vulcan: 1.2.70
```

#### Verify Nvidia Runtime for Containers

If you have followed the prerequisites, you should be able to run the below.

```bash
# Introduced with JetPack 4.2
# Allow Docker containers access to underlying GPU
sudo nvidia-docker version
```

## Running the Device Query example on our GPU

Let's get started with the first example now running in a container. For this we will utilize the commonly used [`deviceQuery` example](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/deviceQuery) that will query our device and show specifications about the found devices. 

Create a `Dockerfile` with the content below:

```dockerfile
FROM nvcr.io/nvidia/l4t-base:r32.6.1

ARG V_CUDA=10.2
ARG ARCH=aarch64

RUN apt-get update \
        && apt-get install -y --no-install-recommends make g++ git ca-certificates

RUN git clone --branch v${V_CUDA} https://github.com/NVIDIA/cuda-samples /tmp/cuda-samples

WORKDIR /tmp/cuda-samples/Samples/deviceQuery

RUN make clean && make TARGET_ARG=${ARCH}

CMD [ "./deviceQuery" ]
```

Finally, we can build this container now and run it:

```bash
# Build the docker image
sudo docker build -t m18x/nvidia-jetson-device-query .

# Run the docker image
sudo docker run --rm -it m18x/nvidia-jetson-device-query
```

## Running PyTorch with Python 3.9 on our GPU

> Note: for this example, you need to build [PyTorch for Python 3.9](/post/2021/11/25/infrastructure-nvidia-ai-nvidia-building-pytorch)

Going to a more complex example, let's build a container that invokes PyTorch and prints out the version.

Create a **main.py** file with the following content:

```python
import torch

print(f"Torch Version: {torch.__version__}")
print(f"Torch CUDA Is Available: {torch.cuda.is_available()}")
print(f"Torch CUDA Version: {torch.version.cuda}")
```

Package it all up with a `Dockerfile` that will install Python 3.9, add our Python specific PyTorch wheel with CUDA enabled and run our application.

```dockerfile
FROM nvcr.io/nvidia/l4t-base:r32.6.1

ARG V_CUDA=10.2
ARG ARCH=aarch64

ARG V_PYTHON_MAJOR=3
ARG V_PYTHON_MINOR=9
ENV V_PYTHON=${V_PYTHON_MAJOR}.${V_PYTHON_MINOR}

ENV DEBIAN_FRONTEND=noninteractive

# Install Dependencies
# Required for PyTorch: libomp5 libopenblas-dev
RUN apt-get update \
        && apt-get install -y --no-install-recommends software-properties-common apt-utils curl libomp5 libopenblas-dev

# Configure Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python3 \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python

# Install PyTorch
COPY torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl /tmp/torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl
RUN pip install /tmp/torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl

# Install Dependencies
RUN pip install numpy

# Application
WORKDIR /code
COPY main.py /code/main.py

CMD [ "python", "main.py" ]
```

Finally, we can again build this container and run it with:

```bash
# Build the docker image
sudo docker build -t m18x/nvidia-jetson-pytorch-test .

# Run the docker image
sudo docker run --rm -it m18x/nvidia-jetson-pytorch-test
```

k3s agent -s https://192.168.1.52:6443 -t ${NODE_TOKEN}

## Running the PyTorch example on Kubernetes

### Kubernetes Setup for GPU

As a final step, we need to be able to run PyTorch examples on a Kubernetes enabled cluster. This might sound trivial, but as seen above, Kubernetes does not natively support the `--gpus` flag yet. But by following the commands below we can quite easily get it working on Kubernetes.

First start by creating your Kubernetes cluster. Since we are running on ARM, I typically use [`K3S`](https://k3s.io) which is more lightweight. We can set this up through the following command:

```bash
# Server:
# --kubelet-arg=--feature-gates=DevicePlugins=true: Enables us to use the nvidia-device-plugin to get nvidia.com/gpu available
curl -sfL https://get.k3s.io | K3S_TOKEN="XAVIER_HIS_SECRET_TOKEN" INSTALL_K3S_EXEC="--cluster-init --docker --write-kubeconfig-mode 644 -write-kubeconfig $HOME/.kube/config --kubelet-arg=feature-gates=DevicePlugins=true" sh -s -

# Agent:
curl -sfL https://get.k3s.io | sh -s agent --server https://192.168.1.52:6443 --token "XAVIER_HIS_SECRET_TOKEN" --docker
curl -sfL https://get.k3s.io | sh -s agent --server https://your-hostname:6443 --token "XAVIER_HIS_SECRET_TOKEN" --docker

# On the master, label the worker nodes
kubectl label nodes node-1 node-2 tinx.ai/type=worker
```

Once the command above finishes, we should now be able to execute a NVIDIA container on Kubernetes:

```bash
sudo kubectl run -i -t nvidia --rm --image=m18x/nvidia-jetson-device-query --restart=Never
```

Which if successfully completes should output our Device Information:

```bash
admin@node-masked:~$ sudo kubectl run -i -t nvidia --rm --image=m18x/nvidia-jetson-device-query --restart=Never
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Tegra X1"
  CUDA Driver Version / Runtime Version          10.2 / 10.2
  CUDA Capability Major/Minor version number:    5.3
  Total amount of global memory:                 1980 MBytes (2076119040 bytes)
  ( 1) Multiprocessors, (128) CUDA Cores/MP:     128 CUDA Cores
  GPU Max Clock rate:                            922 MHz (0.92 GHz)
  Memory Clock rate:                             13 Mhz
  Memory Bus Width:                              64-bit
  L2 Cache Size:                                 262144 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 32768
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.2, NumDevs = 1
Result = PASS
pod "nvidia" deleted
```

Or if we run the other container we build:

```bash
admin@node-masked:~$ sudo kubectl run -i -t nvidia --rm --image=m18x/nvidia-jetson-pytorch-test --restart=Never
If you don't see a command prompt, try pressing enter.
Torch Version: 1.10.0a0+git36449ea
Torch CUDA Is Available: True
Torch CUDA Version: 10.2
pod "nvidia" deleted
```

### Enabling Nvidia Device Plugin for POD YAML Deployments with GPU support

> ⚠️  The below will not work and finish with an error: `Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "--fail-on-init-error=false": executable file not found in $PATH: unknown`. I am however, including this for future reference and illustration of the state of support for the Jetson. For now, you should be able to utilize the GPU by tagging your nodes for GPU use and having the default runtime set.

The above will start up a container but it won't be registered under pods automatically. For that we need to create a manifest, but how can we request GPUs in there? Normally we simply would use the [Nvidia Device Plugin](https://github.com/NVIDIA/k8s-device-plugin) for this that will expose the resource limit tag `nvidia.com/gpu` for our YAML. However, this DaemonSet depends on `NVML` which in its turn depends on `nvidia-smi` which is [not available for Tegra devices](https://gitlab.com/nvidia/kubernetes/device-plugin/-/merge_requests/61)

Luckily the people and [WindRiver](https://blogs.windriver.com/wind_river_blog/2020/06/nvidia-k8s-device-plugin-for-wind-river-linux/) figured out how to get this plugin available for us.


```bash
# Config your user first
git config --global user.email "xavier.geerinck@gmail.com"

# Clone the Device Plugin repository and apply the patches
git clone -b 1.0.0-beta6 https://github.com/nvidia/k8s-device-plugin.git
cd k8s-device-plugin
wget https://labs.windriver.com/downloads/0001-arm64-add-support-for-arm64-architectures.patch
wget https://labs.windriver.com/downloads/0002-nvidia-Add-support-for-tegra-boards.patch
wget https://labs.windriver.com/downloads/0003-main-Add-support-for-tegra-boards.patch
git am 000*.patch

# Build the device plugin container
sudo docker build -t nvidia/k8s-device-plugin:1.0.0-beta6 -f docker/arm64/Dockerfile.ubuntu16.04 .

# Apply it on the cluster
kubectl apply -f nvidia-device-plugin.yml

```

> There is a prebuilt container available at `m18x/k8s-device-plugin:1.0.0-beta6`



Learn how you can compile PyTorch to run on the Nvidia Jetson with the latest Python version

Enabling CUDA with PyTorch on Nvidia Jetson and Python 3.9


# Video Analytics - Edge

For a recent use case I want to run an AI Edge Model that infers a real-time video stream as quickly as possible. Normally this is quite difficult as the inference might lack behind on the realtime stream and we have to stamp the stream to solve a lag behind inferencing model. Luckily for us, Azure has the ["Azure Video Analyzer" service](https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/overview) that can help us with this. But how do we develop a module with this and deploy it on IoT Edge?

## Prerequisites

Before we get started, let's do some prerequisites:

### Create Video Analyzer account in azure

Open up the Azure Portal and create the Video Analyzer Service with an Edge Module in it. In there we need to copy the provisioning token:

![](./video-analyzer-create-1.png)

![](./video-analyzer-create-2.png)

### Create IoT Edge Device

Now we have that, we need to create an IoT Device that we want to deploy the module on. Not only will we use this for a real deployment, but also for simulating to ease development (I will be running under WSL initially).

1. `VS Code` -> `CTRL + P` -> Create Azure IoT Edge Device
2. Save the `Connection String` somewhere (e.g. the info below) more speci

> If you can't find the connection string anymore, you can right click the device if you have the Azure IoT Hub extension installed in VS Code

### Configure IoT Edge Simulator Locally

As a last prerequisite, we have to configure the [IoT Edge Simulator](https://github.com/Azure/iotedgehubdev) so we can develop locally. So run the commands below for this.

> AWAITING https://github.com/Azure/iotedgehubdev/issues/352

```bash
# Install Python
pyenv install 3.8.10
pyenv global 3.8.10

# Install iotedgehubdev (latest)
# pip install iotedgehubdev --upgrade --force-reinstall
pip install git+https://github.com/Azure/iotedgehubdev@master

# Install Docker Compose (Linux Only)
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Enable PyEnv binaries under sudo
# required for sudo iotedgehubdev
sed -r -i -e "s#Defaults\s+secure_path=\"([^\"]+)\"#Defaults secure_path=\"\1:$HOME/.pyenv/shims\"#" /etc/sudoers

# Configure the Device
sudo iotedgehubdev setup -c "<DEVICE_CONN_STRING>"
```

Now the simulator is setup and we can get started!

> In case of issues, you can remove it completely through this link: https://github.com/Azure/iotedgehubdev/issues/221#issuecomment-536382620

> If you broke WSL its /etc/sudoers as I did, you can fix it by running `wsl -u root visudo` in a Windows Terminal

## 1. IoT Edge - Video Stream Simulator

The first module we will be creating is a Video Stream simulator that allows us to transform a video file into an RTSP stream. A normal webcam or camera will output this, but by having a simple simulator we don't have to connect a webcam each time during development.

### Configuring the device

To configure this device, we can run the script below that will create a local user account and initialize the folders containing our video files:

> Note: this was taken from https://aka.ms/ava-edge/prep_device which we could run through `bash -c "$(curl -sL https://aka.ms/ava-edge/prep_device)"`

```bash
#!/usr/bin/env bash

####################################################################################################

# This script is designed to run an edge device.
# 
# It does the following:
# - for the rtspsim module, it creates a `samples` folders and downloads sample media into it
# - it creates folders on the host that the edge module will mount
# - it create a local user and group with restricted access for the edge module to use

####################################################################################################

# create the local group and user for the edge module
# these are mapped from host to container in the deployment manifest in the desire properties for the module
sudo groupadd -g 1010 localedgegroup
sudo useradd --home-dir /home/localedgeuser --uid 1010 --gid 1010 localedgeuser
sudo mkdir -p /home/localedgeuser

# create folders to be used by the rtspsim module
sudo mkdir -p /home/localedgeuser/samples
sudo mkdir -p /home/localedgeuser/samples/input

sudo curl https://lvamedia.blob.core.windows.net/public/camera-300s.mkv --output /home/localedgeuser/samples/input/camera-300s.mkv
sudo curl https://lvamedia.blob.core.windows.net/public/lots_284.mkv --output /home/localedgeuser/samples/input/lots_284.mkv
sudo curl https://lvamedia.blob.core.windows.net/public/lots_015.mkv --output /home/localedgeuser/samples/input/lots_015.mkv
sudo curl https://lvamedia.blob.core.windows.net/public/t2.mkv --output /home/localedgeuser/samples/input/t2.mkv
sudo curl https://lvamedia.blob.core.windows.net/public/retailshop-15fps.mkv --output /home/localedgeuser/samples/input/retailshop-15fps.mkv

# give the local user access
sudo chown -R localedgeuser:localedgegroup /home/localedgeuser/

# set up folders for use by the Video Analyzer module
# these are mounted in the deployment manifest

# !NOTE! these folder locations are must match the folders used in `deploy-modules.sh` and ultimately the IoT edge deployment manifest

# general app data for the module
sudo mkdir -p /var/lib/videoanalyzer 
sudo chown -R localedgeuser:localedgegroup /var/lib/videoanalyzer/
sudo mkdir -p /var/lib/videoanalyzer/tmp/ 
sudo chown -R localedgeuser:localedgegroup /var/lib/videoanalyzer/tmp/
sudo mkdir -p /var/lib/videoanalyzer/logs
sudo chown -R localedgeuser:localedgegroup /var/lib/videoanalyzer/logs

# output folder for file sink
sudo mkdir -p /var/media
sudo chown -R localedgeuser:localedgegroup /var/media/
```

## 2. IoT Solution - Deployment Manifest

We can now deploy our entire IoT Solution on our device (or on the simulator). So create a file named `deployment.template.json` in the root with the content from the [Azure Video Analyzer repository](https://github.com/Azure/video-analyzer/blob/main/setup/general-sample-setup.modules.json)

> In my case I adapted the Hub version to `1.2.0`

Since we copy over the file as the boilerplate, we can see that [some variables have been inserted that still require replacement](https://github.com/Azure/video-analyzer/blob/main/setup/deploy-modules.sh). 

So let's just use the value replacement:

```bash
PROVISIONING_TOKEN="<YOUR_AVA_PROVISIONING_TOKEN>" # the token used for provisioing the edge module
VIDEO_OUTPUT_FOLDER_ON_DEVICE="/home/xanrin/Projects/iot-prototyping-team/customers/thinkthings/video-analytics-edge/VideoAnalyticsEdge/demo" # the folder where the file sink will store clips
VIDEO_INPUT_FOLDER_ON_DEVICE="/home/xanrin/Projects/iot-prototyping-team/customers/thinkthings/video-analytics-edge/VideoAnalyticsEdge/demo" # the folder where where rtspsim will look for sample clips
APPDATA_FOLDER_ON_DEVICE="/home/xanrin/Projects/iot-prototyping-team/customers/thinkthings/video-analytics-edge/VideoAnalyticsEdge/demo" # the folder where Video Analyzer module will store state

# Make sure the directory exists
mdkir -p /home/xanrin/Projects/iot-prototyping-team/customers/thinkthings/video-analytics-edge/VideoAnalyticsEdge/demo

sed -i "s@\$AVA_PROVISIONING_TOKEN@${PROVISIONING_TOKEN}@g" deployment.template.json
sed -i "s@\$VIDEO_OUTPUT_FOLDER_ON_DEVICE@${VIDEO_OUTPUT_FOLDER_ON_DEVICE}@g" deployment.template.json
sed -i "s@\$VIDEO_INPUT_FOLDER_ON_DEVICE@${VIDEO_INPUT_FOLDER_ON_DEVICE}@g" deployment.template.json
sed -i "s@\$APPDATA_FOLDER_ON_DEVICE@${APPDATA_FOLDER_ON_DEVICE}@g" deployment.template.json
```


## 3. Run in Simulator

We can now run the IoT Solution in our simulator by running `VS Code` -> `CTRL + P` -> `Build and Run IoT Edge Solution in Simulator` -> Select the `deployment.template.json` file.

This will boot up everything and run it in our simulator.

## Reference

https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/get-started-detect-motion-emit-events-portal
https://github.com/Azure/video-analyzer/
[https://github.com/Azure/video-analyzer/tree/main/setup](https://github.com/Azure/video-analyzer/tree/main/setup) has scripts for setting up demo resources
[https://github.com/Azure/video-analyzer/blob/main/setup/general-sample-setup.modules.json](https://github.com/Azure/video-analyzer/blob/main/setup/general-sample-setup.modules.json) deployment manifest for IoT Hub

## Old

### Local running container:

```json
"avaedge": {
    "version": "1.0",
    "type": "docker",
    "status": "running",
    "restartPolicy": "always",
    "settings": {
        "image": "${MODULES.avaedge}",
        "createOptions": {}
    }
},
```


## TO STRUCTURE

1. We create Sim
    * Should have brebaked video files
    * Should output RTSP stream

-> Copied from avaedge and made some adaptations to remove the bind and make user experience more smooth

2. We create AVA Edge and configure it

* I had to edit the Twin Config?! Since properties.desired was not taken over into the module twin...
    * Right click the moodule in the Azure IoT Hub device explorer
    * Edit Module Twin Config
    * Add the lines under avaedge -> properties.desired
    * In prioperties.desired add the lines
    * Click "Update Module Twin" on the left top in VS Code
    * Restart the entire sim

3. We create custom AVA Edge AI Model

## TO STRUCTURE - DEBUGGING

-> container is mcr.microsoft.com/media/video-analyzer
* Versions: https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/release-notes


-> we have 2 binds in /var/lib/videoanalyzer and /var/media
* /var/lib/videoanalyzer
    * /var/lib/videoanalyzer/debuglogs: seems to hold the logs?
* /var/media: seems to be used for recording video to a folder
    * can be controlled through the environment variable: VIDEO_OUTPUT_FOLDER_ON_DEVICE 

-> the rtsp sim module:
* github: https://github.com/Azure/video-analyzer/tree/main/edge-modules/sources/rtspsim-live555

more: 

they have a usb to rtsp as well: https://github.com/Azure/video-analyzer/tree/main/edge-modules/sources/USB-to-RTSP
how to use custom ai module: https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/analyze-live-video-custom-vision?pivots=programming-language-csharp
line crossing explained: https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/use-line-crossing

https://github.com/Azure-Samples/video-analyzer-iot-edge-csharp/tree/main/src/edge
https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/production-readiness
https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-docs/http-extension-protocol
https://thewindowsupdate.com/2021/08/24/design-an-ai-enabled-nvr-with-ava-edge-and-intel-openvino/

Streams


Learn how you can get started to create your own Parking Garage spot monitoring system using Dapr, Actors and the Javascript SDK!

Parking Garage Sensor implementation using Dapr Actors

For a customer, I have been researching a way to reduce the time it takes us to integrate different sensors on-edge, while still being flexible and configure everything remotely. Next to that, we do not want to spend much time on integrating custom connectors.

> Use Case: **"Reduce the time it takes us to integrate different sensors on-edge"**

## Requirements

* Support on-edge flows with a focus on
  * Protocols (MQTT, AMQP, Websockets, Webhooks are commonly used by sensor vendors)
  * On-Edge computation (AI Inferencing)
* Offline support (cloud connection should be optional)
* Possible connectors for multi-cloud support (some of our customers require this)

## Meet Logic Apps Standard

One of the possible solutions that caught my eye is **Logic Apps**. Now you might wonder: "Doesn't Logic App require an Azure Connection?" Well this was true until BUILD 2021, where [**Logic Apps Standard**](https://techcommunity.microsoft.com/t5/azure-developer-community-blog/azure-logic-apps-announcement-ga-of-single-tenant-standard-sku/ba-p/2382460?WT.mc_id=DT-MVP-5004263) was introduced, which is a SKU designed to run as a containerized workload that can **run anywhere**. Causing a lot of excitement in the community! (myself included)

https://twitter.com/brendandburns/status/1397224740486623245

https://twitter.com/logicappsio/status/1397217019473416195



Looking at this announcement, the biggest requirement of all would now be possible through Logic Apps: "Offline Support"! But what is required for this and how far is Microsoft to make this reality?

## Making Logic Apps Standard Available Offline

Well, if you have worked with Logic Apps before, you will know that Logic Apps is quite dependent on Azure Storage for its `AzureWebJobsStorage` setting. This configuration uses the provided connection string for *normal operation*, connecting to an Azure Storage Account. We could potentially utilize the Azure Storage Emulator, but this is not a production recommended configuration and thus not a good solution to work offline.

Luckily for us, Microsoft also announced that they are working on **SQL Support to enable workloads to run locally with no Azure dependency on storage**❗ Which is available as a [Private Preview](http://aka.ms/logicappssql), so sign-up to get a sneak peek at it! 😉

> For my use case I requested access to it and wanted to show you how this works!

So let's dive into how we are able to create a Logic Apps workflow that can run entirely offline.

## Creating a Logic Apps Workflow

### Requirements

Before we can get started creating our Logic Apps workflow, we need to install some dependencies:

> ⚠ I ran this on WSL but I included the Windows installation commands

1. Install .NET Core SDK (3.1.X)
  * **Windows:** `winget install Microsoft.dotnet --version 3.1.411` find the latest version in the [winget-pkgs repository](https://github.com/microsoft/winget-pkgs/tree/master/manifests/m/Microsoft/dotnet)
  * **Linux:** `wget -q https://dot.net/v1/dotnet-install.sh -O - | /bin/bash -s -- -c 3.1` (use -c Current for latest)
2. Install Node.js ([the current supported one is 14](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-node?tabs=v2&WT.mc_id=DT-MVP-5004263#node-version))
  * **Windows:** `nvm install v14.17.3`
  * **Linux:** `nvm install v14.17.3`
3. Install Azure Functions Core Tools 
  * **Windows:** `winget install Microsoft.AzureFunctionsCoreTools`
  * **Linux:** [Add the repository](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?WT.mc_id=DT-MVP-5004263&tabs=linux%2Ccsharp%2Cbash#v2) and run `sudo apt-get install azure-functions-core-tools-3`
4. Install [VSCode Extension - Azure Logic Apps (Standard)](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurelogicapps)
5. Install [VSCode Extension - Azure Functions](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions)
6. Create `global.json` file in the root folder to configure the dotnet version it should use with the content below

**global.json**

```json
{
    "sdk": {
       "version": "3.1.411",
       "rollForward": "disable"
    }
}
```

### Creating a Logic Apps project

We are finally ready to get started on our Logic Apps project! 🤩 To create a Logic Apps workflow, execute the following in VSCode:

1. Create and open a new Project Folder
2. Press `CTRL + P` -> `Azure Logic Apps: Create New Project` -> Current Folder -> Stateless Workflow -> `DemoSQLStorageProvider`
3. Right Click `workflow.json` -> Open in Designer -> Skip for now (for connectors)

The Logic Apps workflow design time api will now start up and run `func host start --port 8000` in the background.

![](./vscode-logic-apps-designer.png)

### Configure SQL Storage Provider for Logic Apps Offline Support

Now we have all our dependencies installed, we can continue setting up the Storage Provider we will use. In this case that will be the [SQL Storage Provider (Private Preview)](https://github.com/Azure/logic-apps-sql-provider/blob/main/GETTING_STARTED.md).

#### Starting SQL Server

Of course we need a SQL Server instance running for our SQL Storage Provider. For development purposes, we can simply run this with:

```bash
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=******' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-CU11-ubuntu-20.04
```

Which will allow us to connect through tools such as [Azure Data Studio](https://docs.microsoft.com/en-us/sql/azure-data-studio/download-azure-data-studio?WT.mc_id=DT-MVP-5004263).

![](./data-studio-mssql.png)

Create a database here and shape it into a connection string 

```bash
# Create Database Query:
CREATE DATABASE LogicAppDemo

# Example Connection String:
# Server=localhost;Database=LogicAppDemo;User Id=sa;Password=*****
```

#### Configuring Logic Apps to utilize SQL Server

Now SQL Server is running, we can go back to our Logic Apps project and adapt the `local.settings.json` file to include the following:

```js
{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "", // Not required since we are using SQL Server as Storage Provider
    "AzureWebJobsSecretStorageType": "Files", // Store secrets in local file system (https://docs.microsoft.com/en-gb/azure/azure-functions/functions-app-settings#azurewebjobssecretstoragetype)
    "Workflows.Sql.ConnectionString": "Server=localhost;Database=LogicAppDemo;User Id=sa;Password=*****", // SQL Server connection information we use User ID and Password instead lf Integrated Security=True
    "FUNCTIONS_WORKER_RUNTIME": "node", // The language worker runtime to load (https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings#functions_worker_runtime)
    "FUNCTIONS_V2_COMPATIBILITY_MODE": "true", // No Change
    "FUNCTIONS_EXTENSIONBUNDLE_SOURCE_URI": "https://workflowscdn.azureedge.net/2021-05-unix-preview" // SQL Storage Provider extension bundle
  }
}
```

Logic Apps is now configured to utilize the SQL Storage Provider.

#### Testing SQL Server Storage Provider

Before we continue, let's create a simple application that takes in a HTTP Request and then sends a HTTP Response back with "Hello World".

![](./vscode-logic-apps-designer-hello-world.png)

> ⚠ Do not forget to save!

Then start the Logic Apps runtime to ensure that we are able to run our application. Press `CTRL + F5` (or Run -> Continue without debugging) to start up the Logic App workflow or utilize `func start --verbose` in a terminal window.

> ⚠ When it asks which runtime to use, select `Node.js`

You should now see your function app running

![](./vscodelogic-apps-start-1.png)

To invoke our endpoint, Right click on the `workflow.json` file and select "overview"

![](./vscodelogic-apps-start-2.png)

Now select the URL and load it in your browser to receive the expected result!

![](./vscodelogic-apps-start-4.png)

#### Troubleshooting

When you have an issue (e.g. `Unable to load shared library 'sni.dll' or one of its dependencies`) try to check the following

* `dotnet --version`: is this showing `3.x.x`?
* Remove the core tools: `rm -rf ~/.azure-functions-core-tools` and try again

Learn how you can run Logic Apps locally without requiring an active internet connection or Azure Storage Account with the SQL Storage Provider

Run Logic Apps locally with the SQL Storage Provider



When looking at a series of posts for Project Bonsai in more detail
So far the main part lacking was how you can actually **train your own custom environment** utilizing the Azure Project Bonsai environment! So let's do just that! 

## Project

For this project I will be working on training the cube below. The purpose of this cube **"To balance on one of its corners*"**

![](./cube.jpg)

Learn how you can teach a cube machine how to self-balance itself by training in a simulation environment and transfer it into the a real working example.

Teaching a cube to self-balance with Azure Project Bonsai

TODO TODO TODO

For a customer, I have been researching a way to reduce the time it takes us to integrate different sensors on-edge, while still being flexible and configure everything remotely. Next to that, we do not want to spend much time on integrating custom connectors.

> Use Case: **"Reduce the time it takes us to integrate different sensors on-edge"**

## Requirements

* Support on-edge flows with a focus on
  * Protocols (MQTT, AMQP, Websockets, Webhooks are commonly used by sensor vendors)
  * On-Edge computation (AI Inferencing)
* Offline support (cloud connection should be optional)
* Possible connectors for multi-cloud support (some of our customers require this)

## Meet Logic Apps Standard

One of the possible solutions that caught my eye is **Logic Apps**. Now you might wonder: "Doesn't Logic App require an Azure Connection?" Well this was true until BUILD 2021 that introduced [**Logic Apps Standard**](https://techcommunity.microsoft.com/t5/azure-developer-community-blog/azure-logic-apps-announcement-ga-of-single-tenant-standard-sku/ba-p/2382460?WT.mc_id=DT-MVP-5004263) which is a SKU designed to run as a containerized workload that can **run anywhere**.

https://twitter.com/brendandburns/status/1397224740486623245

https://twitter.com/logicappsio/status/1397217019473416195

![](./logic-apps-example.png)

## Making Logic Apps Standard Available Offline

If you have worked with Logic Apps before, you will know that Logic Apps is quite dependent on Azure Storage for its `AzureWebJobsStorage` setting, which uses this connection string for *normal operation*. This is an Azure connection string or an Azure Storage Emulator connection string. It's thus not a good solution to work offline.

Luckily for us, Microsoft also announced that they are working on **SQL Support to enable workloads to run locally with no Azure dependency on storage**❗ Which is available as a [Private Preview](http://aka.ms/logicappssql), so sign-up to get a sneak peek at it! 😉

> For my use case I requested access to it and wanted to show you how this works!

## Creating a Logic Apps Workflow

### Requirements

Before we can get started creating our Logic Apps workflow, we need to install some dependencies:

> ⚠ I ran this on Windows where I had access to Winget for easy installation

1. Install .NET Core SDK (`winget install Microsoft.dotnet --version 3.1.409`)
  * Windows:
  * Linux: 
  * find the latest version in the [winget-pkgs repository](https://github.com/microsoft/winget-pkgs/tree/master/manifests/m/Microsoft/dotnet)
2. Install Azure Functions Core Tools (`winget install Microsoft.AzureFunctionsCoreTools`)
3. Install [VSCode Extension - Azure Logic Apps (Standard)](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurelogicapps)
4. Install [VSCode Extension - Azure Functions](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions)
5. Create `global.json` file in the root folder to configure the dotnet version it should use with the content below

**global.json**

```json
{
    "sdk": {
       "version": "3.1.409",
       "rollForward": "disable"
    }
}
```

### Configure SQL Storage Provider for Logic Apps

Now we have all our dependencies installed, we can continue setting up the Storage Provider we will use. In this case that will be the [SQL Storage Provider (Private Preview)](https://github.com/Azure/logic-apps-sql-provider/blob/main/GETTING_STARTED.md).

Of course we need a SQL Server instance running for our SQL Storage Provider. For development purposes, we can simply run this with:

```bash
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=AStrongPassword@123' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-CU11-ubuntu-20.04
```

Which will allow us to connect through tools such as [Azure Data Studio](https://docs.microsoft.com/en-us/sql/azure-data-studio/download-azure-data-studio?WT.mc_id=DT-MVP-5004263).

![](./data-studio-mssql.png)

Create a database here and shape it into a connection string 

```bash
# Create Database Query:
CREATE DATABASE LogicAppDemo

# Example Connection String:
Server=localhost\\SQLEXPRESS;Database=LogicAppDemo;Integrated Security=True;
```

Create and Integrate your Internet of Things (IoT) use case on-edge with Logic Apps and Dapr

Internet of Things (IoT) on-edge with Logic Apps and Dapr

Learn how you can create your own Nvidia Jetson Nano GPU Cluster

Creating an Nvidia Jetson Nano Portable Cluster


As any Machine Learning algorithm a compute pool is required to train your model. For Reinforcement Learning, this is exactly the same. Through cloud, we now have access to on-demand compute, such that we only have to pay for what we use. But how do we actually reap the benefits of this? In this post I will explain you how you can setup your own Reinforcement Learning cluster utilizing Azure Kubernetes Service and Spot-Instances!

## Why Spot Instances

Before we start I would like to introduce [Azure Spot Virtual Machines](https://docs.microsoft.com/en-us/azure/virtual-machines/spot-vms?WT.mc_id=DT-MVP-5004263)! You might wonder why we are setting up an Azure Kubernetes Cluster with spot instances? Well the reason for this is that Spot Instances allow us to utilize "unused" capacity, while getting a significant cost saving in return! 

Now, for Machine Learning use cases where we have a lot of workers typically the costs can raise quite high, but our jobs don't mind that a worker sometimes drops out. When utilizing spot instances, we will then create Virtual Machines for our workers, but they can then be taken away when someone else requires this instance. However in our use cases, they will then be replaced by another instance. This typically costs us a bit of time, but the cost savings generated by it are well worth this!

## Prerequisites

Enough about the Spot Instances! How can we actually get started on creating this ourselves? For this, I created the following upfront:

* A `service principal` having access rights on our subscription
* Python version that is the same to the head node (in our example this is `3.7.7`) which we can do by running `pyenv install 3.7.7; pyenv local 3.7.7; pip install --upgrade pip` [see my previous article to setup pyenv](/post/2021/1/4/coding-python-windows-development-environment-setup)

## Creating our Cluster

For the automation part, I utilized a pre-generated script that we can find on [GitHub](https://github.com/terraform-providers/terraform-provider-azurerm/tree/master/examples/kubernetes/spot-node-pool) which spins up a Kubernetes cluster with a `Standard_DS2_V2` node pool (our main Ray node) and a node pool containing spot instances.

In the default case, it will utilize `Standard_DS2_V2` instances, but feel free to tune this to utilize heavier instances!

A simple `terraform init` and `terraform apply` will allow us to now create the cluster and return the context!

Once this is done, I configured the `kubectl` context through editing the `~/.kube/config` file and adding a context in there

## Install Ray RLLib on Kubernetes

We can now start training after creating the cluster! For this install some dependencies first:

```bash
pip install kubernetes
pip install ray[rllib]
```

Once these are installed, we can utilize a [demo configuration from the Ray project](https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/azure/example-full.yaml) that will setup a small cluster containing one head node pod and 2 autoscaling worker node pods, only requiring 1 CPU and 0.5GiB RAM each.

So save the file above under `ray-example.yaml` and open it!

> ⚠ Make sure to adapt the file since else pods won't get scheduled due to the spot instances!

When opened, add the following under the `spec:` definition for the `worker` pods:

```yaml
tolerations:
- key: "kubernetes.azure.com/scalesetpriority"
  operator: "Equal"
  value: "spot"
  effect: "NoSchedule"
```

When done, execute `ray up ray-example.yaml` which will spin-up our cluster!

![](./ray-1.png)

![](./ray-2.png)

Finally, after this finishes, we are able to run `ray monitor ray-example.yaml` to get the statistics of our cluster running! Or when we prefer a GUI we can run `ray dashboard ray-example.yaml` to forward the cluster dashboard port locally.

![](./ray-3.png)

## Running an example on our cluster

Ok, so our cluster is set-up, let's now run a simple experiment on it!

To do this we can perform several actions:
1. We package a container and run our code through this one on the Kubernetes cluster
2. We forward the Ray head service port and configure `ray.init` as `ray.utils.connect("127.0.0.1:10001")` and forward with `kubectl -n ray port-forward svc/example-cluster-ray-head 10001:10001`

```python

```

Discover how you can spin up an AKS cluster with on-demand resources for your Reinforcement Learning use case utilizing Terraform

Reinforcement Learning with Azure Kubernetes Service (AKS) with Ray RLLib


Next to spinning up a Ray cluster on a Kubernetes Cluster, it's also possible to deploy it on Azure! (actually it's even easier). 

## Prerequisites

Make sure you have ray installed and are connected to Azure.

```bash
# Install ray + azure
pip install ray azure-cli azure-core

# Authenticate on Azure + set subscription id
az login
az account set -s <SUBSCRIPTION_ID>
```

## Installing Ray on Azure

To start on this, we can utilize the [Ray AutoScaler from the repository](https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/azure/example-full.yaml). Download this file and edit it to your wishes, afterwards execute the command below.

> ⚠️ You can edit the `example-full.yaml` file to set another subscription than the one logged in with (default).

In our example, I added the following EXTRA commands:

```yaml
head_setup_commands:
    - pip install azure-cli-core==2.20.0 azure-mgmt-compute==19.0.0 azure-mgmt-msi==1.0.0 azure-mgmt-network==18.0.0
    - sudo apt-get install libglib2.0-0
worker_setup_commands: []
setup_commands: []
```

Finally, start the cluster with:

```
ray up example-full.yaml
```

> ⚠️ At the time of writing this article, there was a small [issue (PR #14750)](https://github.com/ray-project/ray/pull/14750#issuecomment-802696221) in the upstream version. To correctly deploy this, make sure to include `worker_setup_commands: []` and `setup_comands: []` in your deploy file, else they will be overwritten .

Once this setup finishes, we will see something like this as output:

```bash
Acquiring an up-to-date head node
  Launched a new head node
  Fetching the new head node
  
<1/1> Setting up head node
  Prepared bootstrap config
  New status: waiting-for-ssh
  [1/7] Waiting for SSH to become available
    Running `uptime` as a test.
    Fetched IP: <MASKED_IP>
ssh: connect to host <MASKED_IP> port 22: Connection refused
    SSH still not available (SSH command failed.), retrying in 5 seconds.
# -- snipped
Warning: Permanently added '<MASKED_IP>' (ECDSA) to the list of known hosts.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

 09:45:57 up 1 min,  1 user,  load average: 2.38, 0.77, 0.28
Shared connection to <MASKED_IP> closed.
    Success.
  Updating cluster configuration. [hash=60c1cc4dff2c06f8a558dd628bc149cd3fad461d]
  New status: syncing-files
  [2/7] Processing file mounts
Shared connection to <MASKED_IP> closed.
    /home/ubuntu/.ssh/id_rsa.pub from /home/xavier/.ssh/id_rsa.pub
Shared connection to <MASKED_IP> closed.
Shared connection to <MASKED_IP> closed.
  [3/7] No worker file mounts to sync
  New status: setting-up
  [4/7] Running initialization commands
Warning: Permanently added '<MASKED_IP>' (ECDSA) to the list of known hosts.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Connection to <MASKED_IP> closed.
Warning: Permanently added '<MASKED_IP>' (ECDSA) to the list of known hosts.
Connection to <MASKED_IP> closed.
  [5/7] Initalizing command runner
Warning: Permanently added '<MASKED_IP>' (ECDSA) to the list of known hosts.
Shared connection to <MASKED_IP> closed.
nightly-py37: Pulling from rayproject/ray
5d3b2c2d21bb: Pull complete 
3fc2062ea667: Pull complete 
75adf526d75b: Pull complete 
cb9cc0ffd7d7: Pull complete 
20e6bba2821c: Pull complete 
5f94c257d7a8: Pull complete 
8d2d31defa88: Pull complete 
0dc6a7b56a50: Pull complete 
96fa1d3e5cdd: Pull complete 
Digest: sha256:f3f7961c9b2fba6f870027b279fada0f5f53bdd02b23c95310c57bf6ab4c154c
Status: Downloaded newer image for rayproject/ray:nightly-py37
docker.io/rayproject/ray:nightly-py37
Shared connection to <MASKED_IP> closed.
NVIDIA-SMI has failed because it could not communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Shared connection to <MASKED_IP> closed.
2021-03-19 10:47:53,745 WARNING command_runner.py:901 -- Nvidia Container Runtime is present, but no GPUs found.
Shared connection to <MASKED_IP> closed.
bc7d51d7d412b7286c45dc1c1ac5ba256f31b322619d7102c55d6dd6815c0d32
Shared connection to <MASKED_IP> closed.
# -- snipped python packages installation
Successfully installed PyJWT-1.7.1 azure-cli-core-2.20.0 azure-mgmt-compute-19.0.0 azure-mgmt-core-1.2.2 azure-mgmt-network-18.0.0 cryptography-3.3.2 knack-0.8.0rc2 msal-1.10.0
Shared connection to <MASKED_IP> closed.
  [7/7] Starting the Ray runtime
Did not find any active Ray processes.
Shared connection to <MASKED_IP> closed.
Local node IP: <LOCAL_NODE_IP>
2021-03-19 02:48:49,516 INFO services.py:1256 -- View the Ray dashboard at http://127.0.0.1:8265

--------------------
Ray runtime started.
--------------------

Next steps
  To connect to this Ray runtime from another node, run
    ray start --address='<LOCAL_NODE_IP>:6379' --redis-password='<REDIS_PW>'
  
  Alternatively, use the following Python code:
    import ray
    ray.init(address='auto', _redis_password='<REDIS_PW>')
  
  If connection fails, check your firewall settings and network configuration.
  
  To terminate the Ray runtime, run
    ray stop
Shared connection to <MASKED_IP> closed.
  New status: up-to-date

Useful commands
  Monitor autoscaling with
    ray exec /home/xavier/Projects/azure-rllib/rw-train/azure/deploy.yaml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*'
  Connect to a terminal on the cluster head:
    ray attach /home/xavier/Projects/azure-rllib/rw-train/azure/deploy.yaml
  Get a remote shell to the cluster manually:
    ssh -tt -o IdentitiesOnly=yes -i ~/.ssh/id_rsa ubuntu@<MASKED_IP> docker exec -it ray_container /bin/bash
```

## Running a Test

Since our cluster is now installed, it's useful to check out what it can do! So let's start by running a test.

Create a python file named `cartpole.py` with the following content:

```python
```

Once that is created, we can submit it to the cluster with:

```
ray submit deploy.yaml cartpoly.py --start --tmux
ray attach deploy.yaml --tmux
```

This command will take care of starting the cluster when needed and execute our command resulting in the below as output:

## Conclusion

The Ray library is simply amazing in what it does and how it does it. Running distributed compute clusters in cloud has been made super easy. Together with Spot instances, it's a clear choice to utilize Ray whenever we are working with for example Reinforcement Learning!


In my last article about ML-Agents I went through an introduction of what it was and how we can use it. Now I wanted to get some more hands-on experience and found this awesome course on [Unity Learn](https://learn.unity.com/tutorial/traditional-game-ai-vs-ml-agents-hummingbird-course-preview) that went in detail on how to create a Hummingbird. After following this course, I wanted to do my own and work on a more industrial focused example.



Xavier Geerinck

Latest Posts

Video Analytics - Edge