
> 💡[Check Out all Dapr Posts](/tag/dapr)

Creating a stream processor is always an interesting task, you want to be able to take incoming message and process them one by one, executing some actions along the way. Typically, we would use Apache Spark or Apache Flink to process these incoming streams, but these tools can be quite complex. What if you just have to be able to take incoming messages, read them and write them to a sink while doing some data manipulation along the way?

## Use Case

For this use case, we will be streaming events to [Azure IoT Hub](https://docs.microsoft.com/en-us/azure/iot-hub/) and pick up those events and sink them towards a [CosmosDB](https://docs.microsoft.com/en-us/azure/cosmos-db/).

## Difficulties & Solution

When reading the above, developers instantly think: Damn, we have to integrate IoT Hub (which is annoying in Javascript due to the SDK ecosystem) and sink it towards CosmosDB (yet another API to integrate).

Now, there is a solution for this! By utilizing [Dapr](https://dapr.io) and the [JS SDK](https://github.com/dapr/js-sdk), we can simply configure our 2 connections and utilize the Dapr SDK to connect to Azure IoT Hub and write those events to CosmosDB! If we want to utilize another PubSub or Sink, we can simply redefine our components and it will magically work!

## Creating our Solution


### Directory Layout

In this case start of by [creating a Typescript project](/post/2021/03/24/typescript-getting-started) and adding a `components/` folder, so that your layout will look like this:

```
components/
src/
package.json
README.md
tsconfig.json
```

### Dapr Components

In the layout above, we configure 2 Dapr Components

#### PubSub - IoT Hub

**components/stream.yaml**

```yaml
# https://docs.dapr.io/reference/components-reference/supported-bindings/eventhubs/
# https://docs.dapr.io/reference/components-reference/supported-pubsub/setup-azure-eventhubs/#subscribing-to-azure-iot-hub-events
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: stream
  namespace: default
spec:
  type: pubsub.azure.eventhubs
  version: v1
  metadata:
  - name: connectionString      # Azure EventHubs connection string
    value: "IOTHUB_EH_CONNECTION_STRING"
  - name: consumerID         # EventHubs consumer group
    value: "$Default"
  - name: storageAccountName    # Azure Storage Account Name
    value: "CHECKPOINT_ACCOUNT_NAME"
  - name: storageAccountKey     # Azure Storage Account Key
    value: "CHECKPOINT_ACCOUNT_KEY"
  - name: storageContainerName  # Azure Storage Container Name
    value: "CHECKPOINT_CONTAINER_NAME"
```

#### Binding - CosmosDB

**components/sink.yaml**

```yaml
# https://docs.dapr.io/reference/components-reference/supported-bindings/cosmosdb/
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: sink
  namespace: default
spec:
  type: bindings.azure.cosmosdb
  version: v1
  metadata:
  - name: url
    value: COSMOSDB_URL
  - name: masterKey
    value: COSMOSDB_KEY
  - name: database
    value: COSMOSDB_DATABASE
  - name: collection
    value: COSMOSDB_COLLECTION
  - name: partitionKey
    value: COSMOSDB_PARTITIONKEY
```

### Coding 

After creating the components are configured we can create our source code that will connect to the IoT Hub stream through the JS SDK `await server.pubsub.receive()` and sink it to CosmosDB with `await client.binding.send()`

```javascript
import { DaprClient, DaprServer } from "@dapr/dapr";
import Config from "./config";

const client = new DaprClient(Config.dapr.sidecar.host, Config.dapr.sidecar.portHttp);

async function main() {
  console.log(`Initializing Server (App Port: ${Config.dapr.app.port}, Sidecar Port: ${Config.dapr.sidecar.portHttp}, Sidecar Protocol: HTTP)`);
  const server = new DaprServer(Config.dapr.app.host, Config.dapr.app.port, Config.dapr.sidecar.host, Config.dapr.sidecar.portHttp);

  // Initialize the server to subscribe (listen)
  await server.pubsub.subscribe(Config?.dapr?.components?.pubsub?.eventhub?.name, Config?.dapr?.components?.pubsub?.eventhub?.topic, async (data: any) => {
    await writeToSink(data);
  });

  await server.start();
}

async function writeToSink(message: any) {
  // Add PartitionID
  const partitionId = message.body.my_partition_key; // @todo: set your partition key
  message.body.partitionID = partitionId;

  // Sink the body to CosmosDB
  await client.binding.send(Config?.dapr?.components?.bindings?.cosmos?.name, "create", message.body);
}


main().catch(e => console.error(e));
```

### Testing

Finally, we can test the code above by adding some scripts to the `package.json`:

```
"start": "npm run build && node dist/index.js",
"start:dapr": "dapr run --app-id stream-processor --app-port 10000 --dapr-http-port 10001 --dapr-grpc-port 10002 --app-protocol http --components-path ./components/ npm run start",
```

This will start up dapr, and you will be able to see incoming messages!

Learn how you can create an Azure IoT Hub Stream Processor that sinks data towards CosmosDB

Create an Azure IoT Hub Stream Processor with Dapr


I am working on a Mobile Application (spoiler) that I am developing through [React-Native](https://reactnative.dev/) and [Expo](https://expo.dev/). 

Now to develop this application, I want the mobile app to connect to my backend and frontend seeing that I need to implement authentication (that already gets handled by [Tinx.ai](https://tinx.ai) in the backend)

Up until now I was utilizing [loca.lt](https://theboroer.github.io/localtunnel-www/) instead of the popular [ngrok](https://ngrok.com/) as it allowed me to simply run `lt --port 8000 --subdomain my-domain` and provide me an URL to work with.

This worked perfectly, but had 2 optimizations that I wanted to do:

1. Google was not happy as the redirect URL was not owned by me (it is hosted under loca.lt)
2. The custom subdomains had a small timeout after disconnect, which made it harder to restart them.

Which is why I wanted to look at alternatives and came across [Cloudflare Tunnels](https://www.cloudflare.com/en-gb/products/tunnel/)


## Installing Cloudflared

First get started by installing the Cloudflare CLI tool by [downloading it](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation/)

As I installed this on the WSL environment I could run:

```bash
# Download the latest version
curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb

# Install the package
sudo dpkg -i cloudflared.deb 
```

## Cloudflared Login

Next up, login to your account by running

```bash
cloudflared tunnel login
```

This will let you open an URL and authorize cloudflared for a specific zone.

## Configuring your Tunnel

Now the most important step, let's create a tunnel and route the traffic to it.

Start of by creating a tunnel:

```bash
cloudflared tunnel create my-tunnel
```

The tunnel is now created and will be visible on the Cloudflare [Zero Trust dashboard](https://dash.teams.cloudflare.com/):

![](./dashboard.png)

Once the tunnel is created, route traffic to it:

```bash
# my-tunnel: the tunnel name we created
# my.domain.com: the domain we want to connect to
cloudflared tunnel route dns my-tunnel my1.domain.com
cloudflared tunnel route dns my-tunnel my2.domain.com
```

Before we can start it we need to configure `cloudflared` its configuration file situated at `~/.cloudflared/config.yml`:

```yaml
# Get the tunnel id by running `cloudflared tunnel list`
tunnel: YOUR_TUNNEL_ID
credentials-file: /home/YOUR_USER/.cloudflared/YOUR_TUNNEL_ID.json
warp-routing:
  enabled: true
ingress:
 - hostname: my1.domain.com
   service: http://localhost:3000
 - hostname: my2.domain.com
   service: http://localhost:8000
 - service: http_status:404
```

## Starting your Tunnel

First run the below to configure a [larger UDP buffer size](https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size)

```bash
sudo sysctl -w net.core.rmem_max=2500000
```

Finally, you can start your tunnel by running:

```bash
# my-tunnel: the tunnel name we created
cloudflared tunnel run my-tunnel
```


Learn how you can set up and replace ngrok or loca.lt by Cloudflare Tunnels

Setting up Cloudflare Tunnels


For a recent preparation, I wanted to revisit the creation of a stack in Javascript and how we can do this, taking into consideration changing design aspects as well as efficiency.

## Definition

A stack is a data structure in computer science that works according to the Last-in, First-out (LIFO) principle. The example of a waiter is often given, who is taking plates and stacking them on top of each other. There is only one possibility to take them of, which is to start at the top again, the first plate is thus the last one to go.

## Use Cases

* You want items to go out in the reverse order that you put them in
* "undo" functionality

## Methods

We require a few basic methods to use this data structure:

* `push`: Adding Items to the top
* `pop`: Removing items from the top
* `isEmpty`: See if there are items on the stack
* `isFull`: see if the stack is full
* `peek`: Check the item on the top without removing it

## Attention Points

Before even starting to code, we always have to remember some attention points. Some things I always try to think about:

* **Check input parameters:** Ensure you check to see if the input parameters are correct.
    * I remember a quote here from Facebook stating that if your code has a chance of failing 0.001% of the time, it will fail at scale!
* **Check Performance for your Use Case:** Performance is crucial, but sometimes not needed. When building something, do you need performance (typically takes a bit longer due to thinking process and sometimes implementation)? Or do you need speed of delivery? Don't waste spending hourse on over-optimizing, but design for your use case
* **Think before you do:** before even starting with writing a letter of code, think what you want to do, achieve and what possible ways are. If you can't come up with enough ideas, brainstorm it around.
* **Tests:** Write tests for everything you do, covering the following cases:
    * **Main Case:** The case your test is designed for
    * **Edge Case:** Edge boundaries (e.g. if you expect a number between 0 an 100, check 0 and 100 specifically)
    * **Stretch Case:** What do you think might get hit but is very unlikely?

## Implementation

```javascript
class Stack {
    constructor(size) {
        this.size = size;
        this.items = [];
    }

    push(item) {
        if (!item) {
            throw new Error("INVALID_ITEM");
        }

        if (this.isFull()) {
            throw new Error("STACK_IS_FULL");
        }

        this.items.push(item);
    }

    pop() {
        if (isEmpty()) {
            return null;
        }

        return this.items.pop();
    }

    peek() {
        if (isEmpty()) {
            return null;
        }

        return this.items[this.items.length - 1];
    }

    isEmpty() {
        return this.items.length > 0;
    }

    isFull() {
        return this.items.length >= this.size;
    }
}
```

## Evolution #1: Remove from bottom

We now are adding the evolution of our class and want to be able to remove items from the bottom, how would we do this? How can we do this performantly?

Implementing this can be done quite easily in Javascript. We can utilize either the `shift()` or the `splice()` method for our convenience. Using this `shift()` method, our method named `popFromBottom()` could look like this:

```javascript
shift() {
    if (this.isEmpty()) {
        return null;
    }

    this.items.shift();
}
```

Assessing the performance is tricky now as it includes the `shift()` method which is created by Javascript for our convenience. We thus would need to know the performance of the underlying implementation for this. In any case, we know we have 1 check and 1 method call.

If we would use the `splice()` method in Javascript, we would cut up our array in 2 pieces and "glue" them together afterwards, forming a new array.

## Evolution #2: Remove from the bottom without using `shift()` or `splice()`

Another evolution we can do on this is to forbid certain methods. Languages such as Javascript add a convenience method to underlying data structures which help us. So we should as well know how we can do this ourselves.

E.g. when we forbid the usage of `shift()` and `splice()`, thus we need to think how we can implement the method in an efficient way without any of these methods.

Since we can neither use the `splice` or `shift` methods, we could think of the following naively:
1. Create a new empty array (the size of the existing one - 1 element)
2. Loop over the existing array and add each item, skipping the one at the given index (in our case 0 as it's the beginning)

Can we do better at this point in time with what we know? Can we think of other data structure and algorithms that come to mind?
* Data Structures: Queue, Deque (Double Ended Queue), Linked List, ...
* Algorithms:
    * If size 1, we can push at the end and swap the items in memory (O(1))
    * If size n, we can push at the end and swap towards the front (O(n))
    * If size n, we can copy to a new array and append the items there


Which would result in code looking like this:

```javascript
let newArr = [];
let idxToRemove = 0; // Remove the first item

for (let i = 0; i < this.items.length; i++) {
    if (i == idxToRemove) {
        continue;
    }

    newArr.push(this.items[i]);
}

this.items = newArr;
```

## O Table for Common Data Structures - Average Cases

As a reminder, I also include the typical data structures used to remind myself.

|Data Structure|Access|Search|Inseration|Deletion|
|-|:-:|:-:|:-:|:-:|
|Array|O(1)|O(n)|O(n)|O(n)|
|Stack|O(n)|O(n)|O(1)|O(1)|
|Queue|O(n)|O(n)|O(1)|O(1)|
|Sigly-Linked List|O(n)|O(n)|O(1)|O(1)|
|Doubly-Linked List|O(n)|O(n)|O(1)|O(1)|
|Hash Table|N/A|O(1)|O(1)|O(1)|
|Binary Search Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|
|B-Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|
|Red-Black Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|
|Splay Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|

My reasoning behind implementing a simple stack explained through Javascript to prepare for a coding interview

Coding Interview Preparation - Stack Implementation


I always wanted to get more into .NET programming as together with Typescript it is one of my favourite languages as it is flexible and more importantly allows for cross-platform development. Finally, it is adopting a more simplified syntax as well which is coming in .NET 7.

For cross platform mobile app development, .NET had Xamarin and Xamarin.Forms which allowed us to create mobile apps easily. However, some things were lacking (such as Desktop support) and it was more of a standalone product that was not integrated in the .NET BCL (Base Class Library). Which is why Microsoft decided to rewrite the entire approach and name it **MAUI** ([more info can be found here](https://ballardchalmers.com/2021/10/04/what-is-net-maui-and-how-does-it-differ-from-xamarin/) - Thanks [Clifford](https://twitter.com/CliffordAgius) for the amazing write up!)

![](./architecture.png)

You can view a [detailed roadmap of MAUI here](https://github.com/dotnet/maui/wiki/roadmap)

So let's get started and create a mobile application!

## Getting Started

### Prerequisites

- XCode (Make sure to start it once to accept the EULA)
- XCode Terminal Tool
  - If dotnet doesn't build, check if it is set by opening Xcode -> Preferences -> Locations -> Command Line Tools and ensure it is set.
- Android Studio
- Visual Studio 2022 Preview (7+) for Mac
  - Mac sure to install iOS, Android and macOS tools

### Installing MAUI

1. Go to the dotnet website and [download and install the latest dotnet LTS version](https://dotnet.microsoft.com/en-us/download/dotnet)
   - Make sure `dotnet --list-sdks` shows it is installed under `/usr/local/share/dotnet/sdk`
2. Once installed, install maui by running `dotnet workload install maui`

> Warning: Do not install dotnet through brew as this will render you [unable to install maui correctly on mac](https://github.com/dotnet/maui/issues/5939)

## Creating and Running your App

### Create your App

1. Create a directory with `mkdir mobile-app`
2. Create a maui app with `cd mobile-app; dotnet new maui`

### Get Device Identifier

If we would try to run our application, we would get the error as shown below:

![](./vs-error.png)

To resolve this, we should do another thing, which is to get our Device Identifier that we will use to target our device. So follow these steps:

1. Open up Xcode
2. Go to Window
3. Devices and Simulators
4. Select your Device
5. Right Click
6. Copy Identifier

Now run the command below with your identifier:

```bash
dotnet build -t:Run -f net6.0-ios -p:\_DeviceName=:v2:udid=4C4906B6-B3DB-490A-A322-8153E2DBCA85
```

If it opens up the simulator, you can close it again.

### Run your App

Now we can open our app with Visual Studio.

If you were able to run all the steps above, you should now be able to see an interface that looks like the below:

![](./vs-overview.png)

When we now click the big Play button on the top, the simulator will start and we will be able to start working on our application! Including Hot-Reload of the XAML interface, so each change we make will be reflected in the simulator

![](./vs-working.png)

## Conclusion

That's it! See how simple it is to start creating your cross platform mobile application using .NET.


Learn how you can create a cross platform working mobile application utilizing MAU and the .NET ecosystem

Creating a Cross Platform Mobile App with Maui .NET on Mac M1


As the official maintainer of the [Dapr JS SDK](https://github.com/dapr/js-sdk), I am always looking on how we can improve our documentation, tooling and API to integrate Dapr through Javascript. 

Recently, I started working on [Tinx.ai](https://tinx.ai) where I am using Dapr to host the Backend and send events to my event queue. So in this blog article, I wanted to dive a bit deeper on how you can integrate Dapr utilizing [Nestjs](https://nestjs.com/)

## File Structure

The first thing we should do is to create our file structure containing our NestJS project:

> 💡 To learn more on how you can bootstrap a NestJS project, check out their [documentation](https://docs.nestjs.com/first-steps)

```bash
src/
    main.ts             # NestJS Bootstrap (see https://docs.nestjs.com/first-steps)
    app.module.ts       # App Module
    config/config.ts    # Config 
    dapr/
        dapr.module.ts  # Dapr module main entry point
        dapr.service.ts # Dapr Service containing our functions
```

## Create a Nest Dapr Module

Once our file structure is created we can configure our module and make it available to NestJS

### src/dapr/dapr.module.ts

```javascript
import { Module } from "@nestjs/common";
import { ConfigModule } from "@nestjs/config";
import { DaprService } from "./dapr.service";

@Module({
    // List of imported modules
    imports: [ ConfigModule ],

    // List of controllers
    controllers: [ ],

    // Providers that will be instantiated by the Nest injector and that may be shared at least across this module
    providers: [ DaprService ],
    
    // which providers can be used in other modules?
    exports: [ DaprService ]
})
export class DaprModule {}
```

The code above will utilize the Config module (which we use to inject configuration into our service later on) as well as the Dapr Service that we will create that holds the Dapr JS SDK methods.

Finally, register this module in the `app.module.ts` file:

```javascript
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import configuration from '../config/config';
import { DaprModule } from './dapr/dapr.module';

@Module({
  imports: [
    ConfigModule.forRoot({
      load: [configuration],
    }), // https://docs.nestjs.com/techniques/configuration
    DaprModule
  ],
  controllers: [],
  providers: [],
})
export class AppModule;
```

### src/dapr/dapr.service.ts

Now we have our module registered, let's create the service class that helps us access the Dapr JS SDK:

```javascript
import { Injectable, Logger } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { DaprClient } from 'dapr-client';

@Injectable()
export class DaprService {
  daprClient: DaprClient;
  private readonly logger = new Logger(DaprService.name);

  constructor(
    private readonly configService: ConfigService
  ) {
    const daprHost = this.configService.get<string>('third_party.dapr.host');
    const daprPort = this.configService.get<string>('third_party.dapr.port');

    this.logger.log(`Initializing DaprClient("${daprHost}", ${daprPort})`);
    this.daprClient = new DaprClient(daprHost, daprPort);
  }
}
```

As you can see, we are accessing `third_party.dapr.host` and `third_party.dapr.port` here that pull information from the `config/config.ts` file. So go ahead and utilize the following config:

```javascript
export default () => ({
  third_party: {
    dapr: {
      host: process.env.DAPR_SIDECAR_HOST || '127.0.0.1',
      port: process.env.DAPR_SIDECAR_PORT || '3500',
    }
  },
});
```

## Utilize the Nest Module

Now we created our module, we can import it to any of our Nest modules (add it under `imports: [ DaprModule ]`, [more info here](https://docs.nestjs.com/modules)) and start utilizing it.

```javascript
import { Controller, Get, HttpCode, Req, Logger } from '@nestjs/common';
import { ApiTags } from '@nestjs/swagger';
import { DaprService } from '../dapr/dapr.service';

@Controller('demo')
@ApiTags('demo') // Attaches controller to a specific tag - https://docs.nestjs.com/websites/swagger#tags
export class DemoController {
  private readonly logger = new Logger(DemoController.name);

  constructor(
    private readonly daprService: DaprService,
  ) { }

  @Get('/')
  @HttpCode(200)
  async demo(@Req() req): Promise<void> {
    await this.daprService.daprClient.binding.send(`my-component`, "create", { hello: "world" });
  }
```


## Start Nest with Dapr

To start all of this, we can now utilize the `dapr run` command, which will create our process with Dapr in it.

```
dapr run --app-id my-application --app-protocol http --app-port 50001 --dapr-http-port 3500 --components-path ./components npm run start
```

Learn how you can utilize Dapr with Nest.js and the Dapr JS SDK

Integrate Dapr with Nest.js and the Dapr JS SDK


When wanting to work with AI models on my latest Macbook Pro 2021 with M1 Architecture, I received the following error while running `pip install onnxruntime`:

```
ERROR: Could not find a version that satisfies the requirement onnxruntime>=1.10.0 (from tinx) (from versions: none)
```

After doing some research, this error was not unknown. But how can we now get it resolved on this Macbook? When we look at the GitHub repository of [onnxruntime](https://github.com/microsoft/onnxruntime/releases/tag/v1.10.0) we can see that an OSX version is actually being released. 

Luckily, PyPi has a "Test" repository as well which we should try when the normal doesn't work, as it often hosts packages that are not found on the public one yet. So we can simply run the below which will install `onnxruntime`. Note, that we install `flatbuffers` as well, since we will else encounter an error stating that it is depending on flatbuffers.

```bash
# ONNX Depends on this, so we should first install it
pip install flatbuffers

# Then we can install onnxruntime
pip install -i https://test.pypi.org/simple/ onnxruntime
```

If we now run the below, we will see the `onnxruntime` printed and we will be able to infer AI models on our Macbook.

```python
import onnxruntime as rt
print(rt.__version__)
```

Getting ONNX to work on your Macbook is tricky as it throws back quite some errors. Learn here how you can get it working

Running ONNX with onnruntime on your M1 Mac


In my [previous post](/post/2021/12/20/coding-ai-dapr-container-apps) I explained how you can get started with Container Apps. The one missing piece of that puzzle is how you can utilize a queue behind to dynamically scale your workload based on items in a queue! This has been one my most difficult blog items so far, taking me through a range of different issues with Container Apps and Dapr, but which definitely aided those services and myself in becoming more mature.

The following items have been created because of this:

- [Dapr: Python SDK - Input Bindings for HTTP](https://github.com/dapr/python-sdk/issues/342)
- [Log Analytics: Difference between TimeGenerated and \_timestamp_d #79](https://github.com/microsoft/azure-container-apps/issues/79)
- [Support for setting Dapr App Protocol to gRPC](https://github.com/microsoft/azure-container-apps/issues/70)
- [Container Apps seems to be missing appProtocol](https://github.com/Azure/azure-rest-api-specs/issues/17144)
- [Azure Container Apps not Updating Docker Container](https://github.com/pulumi/pulumi-azure/issues/1007)
- [Metadata (and queryString?) should use an Array Type instead of Tuple Type](https://github.com/dapr/python-sdk/issues/328)
- [Azure Container App Demonstration](https://github.com/dapr/python-sdk/issues/316)

This entire process took me around 2 months to complete, but once figured out it's super simple!

## Source Code

The entire source code for this project can be found on my [Public GitHub repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/Azure-Container-Apps/AI-Model-YoloV5-Dapr-Queue)

## Using a Queue - Made easy through Dapr

We will be using a queue in our source code. Luckily for us, this can be done easily by utilizing [Dapr](https://dapr.io) which takes away a lot of the implementation for the different queueing systems and their retry logic. For this project, I decided to utilize the [Service Bus Queue](https://docs.dapr.io/reference/components-reference/supported-bindings/servicebusqueues/).

To set this up, we create a component spec such as the below:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: my-queue
  namespace: default
spec:
  type: bindings.azure.servicebusqueues
  version: v1
  metadata:
    - name: connectionString # Required
      value: "YOUR_CONNECTION_STRING"
    - name: queueName
      value: worker-items
    - name: ttlInSeconds
      value: 60
```

> **Note:** The component spec for Container Apps is defined differently, but more about that soon.

Once this is created, we can hook up our application to the queue by creating an [Input Binding](https://docs.dapr.io/developing-applications/building-blocks/bindings/howto-triggers/) that will monitor the Service Bus queue and call the defined POST endpoint. When looking to the component spec above, we defined the name `my-queue` which will result in a POST call to `/my-queue`.

The code below shows us how we can process items from this binding.

```python
from pydantic import BaseModel
from fastapi import FastAPI
from dapr.ext.fastapi import DaprApp

# Define Request DTO
class WorkerItemRequestDto(BaseModel):
    id: str
    image_url: str

# Add our Routes
@app.options("/my-queue")
async def processor_worker_item_options():
    return

@app.post("/my-queue")
async def processor_worker_item(item: WorkerItemRequestDto):
    return
```

## Implementing YoloX

Now we know how to get items in, we just have to initialize our model and run the item through it. As we did before, we initialize the model, download our image and print the result:

```python
# First load the model as we will always need this
print("Loading Model", flush=True)
model = torch.hub.load("ultralytics/yolov5", "yolov5s", pretrained=True)
model.eval()

# ... Other code

@app.post("/my-queue")
async def processor_worker_item(item: WorkerItemRequestDto):
    return handler_image_url(item.image_url)

# Handling the Image URL
def handler_image_url(image_url):
    print(f'Received: image_url="{image_url}"', flush=True)

    if not image_url:
        return

    # Download the image
    response = requests.get(image_url)
    img = Image.open(io.BytesIO(response.content))

    # Infer result
    results = model(img, size=640)
    bboxes = results.pandas().xyxy[0].to_json(orient="records")
    bboxes = json.loads(bboxes)

    # Print the Result
    items = {}
    for bbox in bboxes:
        items[bbox["name"]] = 1 if bbox["name"] not in items else items[bbox["name"]] + 1

    print(f'Results:', flush=True)
    for key, value in items.items():
        print(f'- {key}: {value}', flush=True)

    return
```

## Running Locally

The above we can run locally by utilizing the `dapr run` command:

```bash
# Send 5 Events to Service Bus Queue
dapr run --app-id=send-events --app-protocol http --components-path=./example/components python example/send.py 5

# Start Server to Process the Events
cd app/code/src
dapr run --app-id=yolox --app-protocol http --components-path=../../../example/components -- uvicorn main:app --host 0.0.0.0 --port 5000
```

Resulting in our application receiving events and processing them! 🤩

![](./demo-local.gif)

## Running in Cloud with Container Apps

Now the power of all of this is that we can enable Serverless AI! To achieve this, we will utilize [Azure Container Apps](https://azure.microsoft.com/en-us/services/container-apps/) that run our container whenever there is an item in our queue. When there are no items in the queue, we use a [KEDA scaler](https://keda.sh) to scale down to 0 instances!

The below illustrates how we utilize secrets that configure the Dapr Component as well as the KEDA scaler:

```javascript
const containerApp = new web.ContainerApp("app", {
  resourceGroupName: resourceGroup.name,
  location: glbLocation,
  kubeEnvironmentId: kubeEnv.id,
  configuration: {
    // ... see source code
    activeRevisionsMode: ActiveRevisionsMode.Single,
    secrets: [
      {
        name: "acr-pwd",
        value: outAcrAdminPassword,
      },
      {
        name: "sb-connection-string",
        value: serviceBusNamespaceKeys.primaryConnectionString,
      },
    ],
  },
  template: {
    revisionSuffix: `${new Date().getTime()}`,
    containers: [
      {
        name: `container-${GLB_APP_ID}`,
        image: image.name,
        env: [
          // ... see source code
        ],
      },
    ],
    // https://keda.sh/docs/2.5/scalers/azure-service-bus/
    scale: {
      minReplicas: 0,
      maxReplicas: 3,
      rules: [
        {
          name: "rule-servicebus-scaler",
          custom: {
            type: "azure-servicebus",
            metadata: {
              queueName: "worker-items",
              messageCount: "2",
            },
            auth: [
              {
                secretRef: "sb-connection-string",
                triggerParameter: "connection",
              },
            ],
          },
        },
      ],
    },
    dapr: {
      enabled: true,
      appId: GLB_APP_ID,
      appPort: GLB_APP_PORT,
      // appProtocol: 'http', // Disabled for now since unsupported by ARM CLI
      components: [
        {
          name: "my-queue",
          type: "bindings.azure.servicebusqueues",
          version: "v1",
          metadata: [
            {
              name: "connectionString",
              secretRef: "sb-connection-string",
            },
            {
              name: "queueName",
              value: "worker-items",
            },
            {
              name: "ttlInSeconds",
              value: "60",
            },
          ],
        },
      ],
    },
  },
});
```

To set up this infrastructure, we can simply run the following command:

```bash
pulumi up \
    --stack demo-cae-yolox \
    -c glb-location=northeurope \
    -c glb-project-name=demo-cae-yolox \
    -c glb-project-env=prd \
    -c sb-topics="worker-items"
```

> For setting up the entire infrastructure, I refer to the Pulumi file created in the [source code](https://github.com/XavierGeerinck/PublicProjects/blob/master/Azure-Container-Apps/AI-Model-YoloV5-Dapr-Queue/index.ts).

We can now view the logs of our application in Log Analytics with the query:

```bash
ContainerAppConsoleLogs_CL
    | project TimeCreated = unixtime_milliseconds_todatetime(_timestamp_d * 1000), ContainerAppName_s, Log_s
    | order by TimeCreated desc
    | take 150
```

![](./demo-azure.png)

## Conclusion

This article might not sound as something worth spending 2 months on! However, when looking at the benefits Container Apps brings in the field of Serverless AI, it's well worth spending this time! We can now finally run models, inference a result and shut it down again without needing to have a compute instance running 24/7 which can end up being expensive when using GPUs.

I hope you enjoyed this article, feel free to put a comment below or contact me!


Serverless AI is a game changer in reducing operational costs! But how can we achieve this? Learn more about it here

Enabling Serverless AI with Azure Container Apps and Dapr


Recently I've been working a lot with different AI Models for a variety of use cases, ranging from simple object classification to massive crowd counting models. While working with these models, I saw that it is always difficult to implement a model in custom code since the Architecture is never included in the weights themselves, thus each model works differently.

That's when I came across a format named [ONNX](https://onnx.ai/) which instantly peaked my interest with promising points such as:
* Improved Inference Performance 
* Interoperability

## Prerequisites

To get started with ONNX, we have two tasks that should be done

1. Configure the [ONNX Execution Provider](https://onnxruntime.ai/docs/execution-providers/)
2. Get an ONNX Model to infer

### Configuring the ONNX Execution Provider for CUDA 

To be able to run on our GPU I chose to utilize the [CUDA Execution Provider](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html). Which has 2 major components that should be installed:

* CUDA
* cuDNN

#### Installing CUDA

CUDA is the easiest of the two to install since it is publically available without a login wall. Before we start, check if CUDA is already installed by running the below, which should finish correctly:

```bash
# Check if CUDA is available
nvidia-smi
nvcc --version
```

If the above is not available, we can install it (**on WSL 2**) by running the below:

> If you want to install it on another target, feel free to check the original [link](https://developer.nvidia.com/cuda-toolkit-archive)

```bash
# View all: https://developer.nvidia.com/cuda-toolkit-archive
# View ONNX Supported: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0

# Install CUDA
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo apt-key add /var/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

# Add CUDA to Path
# https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
# Note: add those to ~/.bashrc at the bottom
export PATH=/usr/local/cuda-11.5/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

> Note: Do not forget to add the export links to your ~/.bashrc file

#### Installing cuDNN

Now up to the other requirement, to install `cuDNN`.

```bash
OS="wsl-ubuntu"
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0
CUDNN_VERSION_MAJOR=8
CUDNN_VERSION_MINOR=2
CUDNN_VERSION_PATCH=4

wget https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/cuda-${OS}.pin 

sudo mv cuda-${OS}.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/ /"
sudo apt-get update
sudo apt-get install libcudnn8=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
sudo apt-get install libcudnn8-dev=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
```

## Running the YOLOX Model

Now we have all the required software installed for the CUDA Execution Provider, we can start running a model and infer from it. Luckily for us, the [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) team provides a [ready-to-use ONNX model](https://github.com/Megvii-BaseDetection/YOLOX/tree/main/demo/ONNXRuntime) that we can use. So let's download it and use it!

To keep the code part short, you can find the entire project on [my public repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/AI/ONNX/yolox) the main part to know is how we can infer an ONNX Model.

Infering an ONNX Model is as easy as creating a session that defines the model path as well as the provider to use

```python
# Create an ONNX Inference Session
session = onnxruntime.InferenceSession(
    path_or_bytes=args.model,
    providers=["CUDAExecutionProvider"]
)
```

After that is done, we just need to provide it an input:

```python
ort_inputs = {session.get_inputs()[0].name: img[None, :, :, :]}
output = session.run(None, ort_inputs)
```

Whereafter we can run all of this and receive our result!

```bash
# Create directories
mkdir model; mkdir images; mkdir output

# Download Model
wget -O ./model/yolox_l.onnx https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_l.onnx 

# Download Example
wget -O ./images/example.jpg https://www.ikea.com/ext/ingkadam/m/7f8f282fb240f466/original/PH179208-crop001.jpg?f=m

# Run
python src/main.py -m ./model/yolox_l.onnx -i ./images/example.jpg -o ./output -s 0.3 --input_shape 640,640
```

## Conclusion

As you can see from the above, it is quite easy to infer an ONNX Model and get results from it! I personally think I will utilize ONNX more in the future, reducing the complexity of my AI pipeline codes even further.

Learn how to run AI inferencing using ONNX on your GPU

AI Model Inferencing with ONNX on GPU (CUDA)


For a new project I am working on, I am utilizing a Queuing architecture where jobs come in and a cluster of containers is processing work on this queue. Each job takes an item from a queue, runs the item through its AI pipeline and spits out a JSON response.

Setting up an infrastructure to accomodate the above is quite "interesting" as it requires:

* Creating a Kubernetes cluster
* Creating and hosting a queueing solution
* Autoscaling your cluster on a hardware level depending on the items in your queue
* Allowing Microservice cross invocation

Typically we would solve the above by utilizing a Queue (e.g. RabbitMQ), a Kubernetes cluster that autoscales (e.g. with with KEDA) and an Image that pulls from this queue in an abstracted way (e.g. with Dapr) with additionally cross Microservice invocation with Dapr.

In the above, the main annoying part stays: "Scaling your hardware to a 0-usage based on a queue". KEDA makes this easy, but it's still annoying and requires some work.

Luckily this is where ["Container Apps" comes in!](https://docs.microsoft.com/en-gb/azure/container-apps/get-started)

Let's see how we can utilize Container Apps to expose the YoloV5 model through an API.

## Building a YoloV5 Container with API

### Code

Create a `main.py` file with the following code, that will pull the model from Torch Hub and run the model on port 5000 with endpoint `/v1/object-detection/yolov5s`

```python
import argparse
import io
from PIL import Image

import torch
from flask import Flask, request

app = Flask(__name__)

@app.route("/", methods=["GET"])
def hello():
    return {
        "hello": "world"
    }

@app.route("/v1/object-detection/yolov5s", methods=["POST"])
def predict():
    if not request.method == "POST":
        return

    if request.files.get("image"):
        image_file = request.files["image"]
        image_bytes = image_file.read()

        img = Image.open(io.BytesIO(image_bytes))

        results = model(img, size=640)
        data = results.pandas().xyxy[0].to_json(orient="records")
        return data


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Flask api exposing yolov5 model")
    parser.add_argument("--port", default=5000, type=int, help="port number")
    args = parser.parse_args()

    model = torch.hub.load("ultralytics/yolov5", "yolov5s", pretrained=True, force_reload=True).autoshape()
    model.eval()
    app.run(host="0.0.0.0", port=args.port) 
```

As a last piece of code, we add our `requirements.txt`:

```
flask
requests
black

matplotlib>=3.2.2
numpy>=1.18.5
opencv-python>=4.1.2
Pillow
PyYAML>=5.3.1
scipy>=1.4.1
torch>=1.7.0
torchvision>=0.8.1
tqdm>=4.41.0

tensorboard>=2.4.1

seaborn>=0.11.0
pandas

thop  # FLOPs computation
```

### Packaging it

Finally, we can package the above as a container by creating a `Dockerfile` with our content:

```bash
FROM python:3.8-slim-buster

RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6  -y

WORKDIR /app
ADD requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt

WORKDIR /app
ADD . /app

EXPOSE 5000

CMD ["python", "main.py", "--port=5000"]
```

This can build the container by running:

```bash
docker build -t api-yolov5 .
```

To make things easier, I have already [pushed this container to the docker hub.](https://hub.docker.com/repository/docker/thebillkidy/api-yolov5)

## Creating our Azure Container Apps

Now our container has been created, we can get started hosting it! To do this, let's first configure our Azure Container App access in the CLI and then publish our container above.

### Prerequisites

First configure our AZ CLI to include the extension and be logged in:

```bash
# Login
az login
az account set --subscription <id>

# Add CLI Extensions
az extension add \
  --source https://workerappscliextension.blob.core.windows.net/azure-cli-extension/containerapp-0.2.0-py2.py3-none-any.whl 

# Register the Web Provider
az provider register --namespace Microsoft.Web
```

### Create

```bash
RESOURCE_GROUP="demo"
LOCATION="northeurope"
LOG_ANALYTICS_WORKSPACE="my-container-apps-logs"
CONTAINERAPPS_ENVIRONMENT="my-environment"

az group create --name $RESOURCE_GROUP --location "$LOCATION"

az monitor log-analytics workspace create \
  --resource-group $RESOURCE_GROUP \
  --workspace-name $LOG_ANALYTICS_WORKSPACE

# Get Log Analytics Client ID and Client Secret
LOG_ANALYTICS_WORKSPACE_CLIENT_ID=`az monitor log-analytics workspace show --query customerId -g $RESOURCE_GROUP -n $LOG_ANALYTICS_WORKSPACE --out tsv`
LOG_ANALYTICS_WORKSPACE_CLIENT_SECRET=`az monitor log-analytics workspace get-shared-keys --query primarySharedKey -g $RESOURCE_GROUP -n $LOG_ANALYTICS_WORKSPACE --out tsv`

# Create Container App Environment
# this is where all the container apps are deployed
az containerapp env create \
  --name $CONTAINERAPPS_ENVIRONMENT \
  --resource-group $RESOURCE_GROUP \
  --logs-workspace-id $LOG_ANALYTICS_WORKSPACE_CLIENT_ID \
  --logs-workspace-key $LOG_ANALYTICS_WORKSPACE_CLIENT_SECRET \
  --location "$LOCATION"

# Create our Container App
az containerapp create \
  --name my-container-app \
  --resource-group $RESOURCE_GROUP \
  --environment $CONTAINERAPPS_ENVIRONMENT \
  --image thebillkidy/api-yolov5:latest \
  --target-port 5000 \
  --ingress 'external' \
  --query configuration.ingress.fqdn
```

## Testing it

Once we ran the last command (`az containerapp create`) we will get a URL back that looks like this: `my-container-app.unique_id.region.azurecontainerapps.io`. When we call it in the browser, we will get:

```json
{
    "hello": "world"
}
```

Which means it is up and running! Executing the inference endpoint now, we can run: 

```bash
curl -X POST -F image=@demo.jpg 'https://my-container-app.unique_id.region.azurecontainerapps.io/v1/object-detection/yolov5s'
```

The above will send the local `demo.jpg` picture to our endpoint and return the infered result, which in this case shows that we have 4 chairs, 4 potted plants, 2 persons and 1 couch on this image!

```bash
[
    {"xmin":103.2421798706,"ymin":292.3749694824,"xmax":437.8124694824,"ymax":615.8124389648,"confidence":0.8725585938,"class":56,"name":"chair"},
    {"xmin":774.8436889648,"ymin":246.4374847412,"xmax":1126.8748779297,"ymax":567.5311889648,"confidence":0.841796875,"class":56,"name":"chair"},
    {"xmin":0.3515624702,"ymin":244.3281097412,"xmax":150.3515472412,"ymax":483.6249389648,"confidence":0.8344726562,"class":56,"name":"chair"},
    {"xmin":752.3436889648,"ymin":97.0234298706,"xmax":821.7186889648,"ymax":220.5390472412,"confidence":0.8227539062,"class":0,"name":"person"},
    {"xmin":536.2499389648,"ymin":107.1015548706,"xmax":659.0624389648,"ymax":210.4609222412,"confidence":0.7124023438,"class":0,"name":"person"},
    {"xmin":356.0155944824,"ymin":218.4296722412,"xmax":984.3749389648,"ymax":491.1249389648,"confidence":0.6181640625,"class":57,"name":"couch"},
    {"xmin":85.8984298706,"ymin":154.0937347412,"xmax":185.5077972412,"ymax":284.8749694824,"confidence":0.5854492188,"class":58,"name":"potted plant"},
    {"xmin":674.5311889648,"ymin":121.9843673706,"xmax":729.8436889648,"ymax":205.1874847412,"confidence":0.384765625,"class":58,"name":"potted plant"},
    {"xmin":233.5546722412,"ymin":220.0702972412,"xmax":367.0312194824,"ymax":321.9062194824,"confidence":0.2978515625,"class":56,"name":"chair"},
    {"xmin":440.3905944824,"ymin":0.0,"xmax":626.7186889648,"ymax":173.1952972412,"confidence":0.2956542969,"class":58,"name":"potted plant"},
    {"xmin":0.9082030654,"ymin":146.59375,"xmax":44.5898399353,"ymax":170.0312347412,"confidence":0.2568359375,"class":58,"name":"potted plant"}
]
```

## Conclusion

It has never been easier to create a workload that dynamically scales based on the incoming requests! In a next post, I will explain how we can combine this with [Dapr](https://docs.microsoft.com/en-us/azure/container-apps/microservices-dapr?tabs=bash) to get items from a queue instead of a HTTP endpoint.

Learn how you run your AI workload with Azure Container Apps, enabling autoscaling

Running your AI Workload through Azure Container Apps


For a use case I wanted to utilize the Nvidia Jetson for edge inference. One of the bottlenecks here was that my software required a Python version that is greater than 3.6. When looking at the [Nvidia Jetson packages for PyTorch](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-pytorch) it was seen that this was only created for version 3.6.

Searching around, I could find quite a lot of people struggling with this (@TODO: PROOF) which made me to look into **a solution** that would help me automate the building of ARM Wheel of **PyTorch that can run on Nvidia Devices** (e.g. Nvidia Jetson Nano) and thus **support CUDA**.

The entire process above took me around ~11 full days, starting of with figuring out how to build the Dockerfile and finally automating the CI process.

> Note: I also decided to utilize this article as an entry for a running Hackathon by [Dev.to for GitHub Actions](https://dev.to/devteam/join-us-for-the-2021-github-actions-hackathon-on-dev-4hn4)

You would wonder why we want to automate this? Well when compiling this initially on my Nvidia Jetson Nano, I couldn't get it to compile past ~80% when using 16Gb of Swap Space (the device only has 2 Gb). After building it on my personal PC, this took ~6h to get it to compile completely. Making it large and long enough to automate it.

> The final source code can be found on [GitHub](https://github.com/XavierGeerinck/Jetson-Linux-PyTorch)

## Contributions

In any kind of project of this size, there are specific contributions that were made. In my project I believe to have made the following:

* Install CUDA on non-GPU devices
* Compile PyTorch with CUDA enabled on non-GPU devices
* Compile PyTorch for Python > 3.6
* Build for ARM with CI through Docker Buildx

## Project Outline

As a best practice, I always love to include the project outline of how I tackled the issue above (to share my thought process):

1. Create a Dockerfile that builds the Wheel
    * How do I build for CUDA? (Hardest part)
    * How do I cross build?
2. Create GitHub Action
    * How do I cross build? Can I run for ARM specifically?

## Dockerfile Creation - Building PyTorch for ARM and Python > 3.6

The hardest part BY FAR is how to compile PyTorch for ARM and Python > 3.6 with CUDA enabled. Because we are going to run on a non-GPU device, thus CUDA is not available on there. I splitted up the Dockerfile into 3 specific sections that can be ran in parallel:

1. Set up CUDA
2. Set up PyTorch (the cloning takes a while)
3. Set up Python 3.9
4. Compile PyTorch
    * I have Nvidia Jetson optimisations included here, thanks [QEngineering](https://qengineering.eu/install-pytorch-on-jetson-nano.html)!
4. Dockerfile Result

Below you can find the explanation of the respective steps. For the final `Dockerfile` optimizations where made to decrease the docker layer sizes (by grouping RUN in 1 command)

### Setting up CUDA

For CUDA, we do not have the CUDA libraries, nor do we have access to them! There is however a trick that allows us to get CUDA loaded in. We copy over the public key of NVIDIA Jetson and we authorize ourselves to the repository. Then we use our package manager to install them:

```bash
V_CUDA_DASH=10-2

# Add the public key
echo "[Builder] Adding the Jetson Public Key"
curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc
echo "deb https://repo.download.nvidia.com/jetson/common ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
echo "deb https://repo.download.nvidia.com/jetson/t186 ${L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

# Install the CUDA Libraries
echo "[Builder] Installing CUDA System"
apt-get update
apt-get install -y --no-install-recommends \
    cuda-libraries-$V_CUDA_DASH \
    cuda-libraries-dev-$V_CUDA_DASH \
    cuda-nvtx-$V_CUDA_DASH \
    cuda-minimal-build-$V_CUDA_DASH \
    cuda-license-$V_CUDA_DASH \
    cuda-command-line-tools-$V_CUDA_DASH \
    libnvvpi1 vpi1-dev

# Link CUDA to /usr/local/cuda
ln -s /usr/local/cuda-$CUDA /usr/local/cuda
```

When we eventually start compiling, we will see CUDA enabled in our CMAKE output 🥳

```bash
#   USE_CUDA              : ON
#     Split CUDA          : OFF
#     CUDA static link    : OFF
#     USE_CUDNN           : OFF
#     USE_EXPERIMENTAL_CUDNN_V8_API: OFF
#     CUDA version        : 10.2
#     CUDA root directory : /usr/local/cuda
#     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
#     cudart library      : /usr/local/cuda/lib64/libcudart.so
#     cublas library      : /usr/local/cuda/lib64/libcublas.so
#     cufft library       : /usr/local/cuda/lib64/libcufft.so
#     curand library      : /usr/local/cuda/lib64/libcurand.so
#     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
#     CUDA include path   : /usr/local/cuda/include
#     NVCC executable     : /usr/local/cuda/bin/nvcc
#     NVCC flags          : <CUT>
#     CUDA host compiler  : /usr/bin/clang
#     NVCC --device-c     : OFF
#     USE_TENSORRT        : OFF
```

### Setting up PyTorch

In a separate docker step, we set up PyTorch and clone it to the working repository (in our case `/build/pytorch`)

```bash
V_PYTORCH=v1.10.0

# Downloads PyTorch to /build/pytorch
git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch /build/pytorch
```

### Setting up Python 3.9

We configure our Python version through the `deadsnakes` ppa and link it as the default one.

> Best practice we should have a `venv` but since I am running it in a Docker container this should suffice.

```bash
# Setting up Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python3 \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python
```

### Compiling PyTorch

The last step in the Dockerfile is to compile PyTorch. For this we set the correct environment variabels to enable CUDA and to optimise the building process by disabling some other parts (e.g. MKLDNN, NNPACK, XNNPACK, ... to be turned off).

We also configure it to use `clang` as the Nvidia Jetson has [NEON registers](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_462) and clang supports those (GCC doesn't).

For our source, we utilize the other layer we created and just copy it from there.

```dockerfile
COPY --from=downloader-pytorch /build/pytorch /build/pytorch
WORKDIR /build/pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Build
rm build/CMakeCache.txt || : \
sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
pip install -r requirements.txt
python setup.py bdist_wheel
```

### Copying the result as an Artifact

Docker Buildx is amazing in the sense that we can utilize the `--output type-local,dest=.` command to output files to our local filesystem, making it such that docker builds and we can export the build result as an artifact.

To achieve this, we pull from the `scratch` image and copy over our result to it from the other docker layer. Our `/` path will then contain all the build wheels of PyTorch (e.g. `torch-1.10.0a0+git36449ea-cp39-cp39-linux_aarch64.whl`)

```dockerfile
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

### Dockerfile Result

Finally, the full Dockerfile will look like this:

```dockerfile
# ##################################################################################
# Setup Nvidia CUDA for Jetson
# ##################################################################################
FROM ubuntu:18.04 as cuda-devel

# Configuration Arguments
ARG V_CUDA_MAJOR=10
ARG V_CUDA_MINOR=2
ARG V_L4T_MAJOR=32
ARG V_L4T_MINOR=6

ENV V_CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}
ENV V_CUDA_DASH=${V_CUDA_MAJOR}-${V_CUDA_MINOR}
ENV V_L4T=r${V_L4T_MAJOR}.${V_L4T_MINOR}

# Expose environment variables everywhere
ENV CUDA=${V_CUDA_MAJOR}.${V_CUDA_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Fix CUDA info
ARG DPKG_STATUS

# Add NVIDIA repo/public key and install VPI libraries
RUN echo "$DPKG_STATUS" >> /var/lib/dpkg/status \
    && echo "[Builder] Installing Prerequisites" \
    && apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates software-properties-common curl gnupg2 apt-utils \
    && echo "[Builder] Installing CUDA Repository" \
    && curl https://repo.download.nvidia.com/jetson/jetson-ota-public.asc > /etc/apt/trusted.gpg.d/jetson-ota-public.asc \
    && echo "deb https://repo.download.nvidia.com/jetson/common ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "deb https://repo.download.nvidia.com/jetson/t186 ${V_L4T} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
    && echo "[Builder] Installing CUDA System" \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
    cuda-libraries-${V_CUDA_DASH} \
    cuda-libraries-dev-${V_CUDA_DASH} \
    cuda-nvtx-${V_CUDA_DASH} \
    cuda-minimal-build-${V_CUDA_DASH} \
    cuda-license-${V_CUDA_DASH} \
    cuda-command-line-tools-${V_CUDA_DASH} \
    libnvvpi1 vpi1-dev \
    && ln -s /usr/local/cuda-${V_CUDA} /usr/local/cuda \
    && rm -rf /var/lib/apt/lists/*

# Update environment
ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs

RUN ln -fs /usr/share/zoneinfo/Europe/Brussels /etc/localtime

# ##################################################################################
# Create PyTorch Docker Layer
# We do this seperately since else we need to keep rebuilding
# ##################################################################################
FROM --platform=$BUILDPLATFORM ubuntu:18.04 as downloader-pytorch

# Configuration Arguments
# https://github.com/pytorch/pytorch
ARG V_PYTORCH=v1.10.0
# https://github.com/pytorch/vision
ARG V_PYTORCHVISION=v0.11.1
# https://github.com/pytorch/audio
ARG V_PYTORCHAUDIO=v0.10.0

# Install Git Tools
RUN apt-get update \
    && apt-get install -y --no-install-recommends software-properties-common apt-utils git \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Clone Source
RUN git clone --recursive --branch ${V_PYTORCH} http://github.com/pytorch/pytorch

# ##################################################################################
# Build PyTorch for Jetson (with CUDA)
# ##################################################################################
FROM cuda-devel as builder

# Configuration Arguments
ARG V_PYTHON_MAJOR=3
ARG V_PYTHON_MINOR=9

ENV V_PYTHON=${V_PYTHON_MAJOR}.${V_PYTHON_MINOR}

# Accept default answers for everything
ENV DEBIAN_FRONTEND=noninteractive

# Download Common Software
RUN apt-get update \
    && apt-get install -y clang build-essential bash ca-certificates git wget cmake curl software-properties-common ffmpeg libsm6 libxext6 libffi-dev libssl-dev xz-utils zlib1g-dev liblzma-dev

# Setting up Python 3.9
WORKDIR /install

RUN add-apt-repository ppa:deadsnakes/ppa \
    && apt-get update \
    && apt-get install -y python${V_PYTHON} python${V_PYTHON}-dev python${V_PYTHON}-venv python${V_PYTHON_MAJOR}-tk \
    && rm /usr/bin/python \
    && rm /usr/bin/python${V_PYTHON_MAJOR} \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python \
    && ln -s $(which python${V_PYTHON}) /usr/bin/python${V_PYTHON_MAJOR} \
    && curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | python

# PyTorch - Build - Source Code Setup 
# copy everything from the downloader-pytorch layer /torch to /torch on this one
COPY --from=downloader-pytorch /pytorch /pytorch
WORKDIR /pytorch

# PyTorch - Build - Prerequisites
# Set clang as compiler
# clang supports the ARM NEON registers
# GNU GCC will give "no expression error"
ARG CC=clang
ARG CXX=clang++

# Set path to ccache
ARG PATH=/usr/lib/ccache:$PATH

# Other arguments
ARG USE_CUDA=ON
ARG USE_CUDNN=ON
ARG BUILD_CAFFE2_OPS=0
ARG USE_FBGEMM=0
ARG USE_FAKELOWP=0
ARG BUILD_TEST=0
ARG USE_MKLDNN=0
ARG USE_NNPACK=0
ARG USE_XNNPACK=0
ARG USE_QNNPACK=0
ARG USE_PYTORCH_QNNPACK=0
ARG TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
ARG USE_NCCL=0
ARG USE_SYSTEM_NCCL=0
ARG USE_OPENCV=0
ARG USE_DISTRIBUTED=0

# Build
RUN cd /pytorch \
    && rm build/CMakeCache.txt || : \
    && sed -i -e "/^if(DEFINED GLIBCXX_USE_CXX11_ABI)/i set(GLIBCXX_USE_CXX11_ABI 1)" CMakeLists.txt \
    && pip install -r requirements.txt \
    && python setup.py bdist_wheel \
    && cd ..

# ##################################################################################
# Prepare Artifact
# ##################################################################################
FROM scratch as artifact
COPY --from=builder /pytorch/dist/* /
```

## GitHub Action Creation

Since PyTorch is now finally compiling! It is time to start automating this and publishing them to an Artifact in GitHub (this way we can always trigger it ourselves and kick of the building process). I want to start building automatically, as soon as a Release is published! So for our action, we have the following outline:

### Action Outline

1. When a release is created trigger the action
2. Clone the repository
3. Setup Docker with Buildx
4. Run our container
5. Copy over the Built Wheel to an artifact on GitHub

### Used Actions

As for actions, the following actions could be reused:

1. [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action)
    * I cross-compile for ARM on AMD64 machines in the pipeline
2. [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action)
    * Configure QEMU to be able to compile for ARM and install the QEMU static binaries
3. [actions/checkout](https://github.com/actions/checkout)
    * Check out a repo
4. [actions/cache](https://github.com/actions/cache)
    * Allow us to cache the docker layers
5. [actions/upload-artifact](https://github.com/actions/upload-artifact)
    * Upload the output of a directory to GitHub artifacts

### Result

Finally resulting in the following GitHub action:

```yaml
name: ci

# https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows
on:
  push:
    branches: [ main ]
  release:
    types: [ created ]

jobs:
  build_wheels:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v2
        
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v1
        
    - name: Set up Docker Buildx
      id: buildx
      uses: docker/setup-buildx-action@v1
    
    - name: Cache Docker layers
      uses: actions/cache@v2
      with:
      path: /tmp/.buildx-cache
      key: ${{ runner.os }}-buildx-${{ github.sha }}
      restore-keys: |
        ${{ runner.os }}-buildx-
    
    - name: Build Docker Image
      run: |
      docker buildx build \
          --platform=linux/arm64 \
          --progress=plain \
          --output type=local,dest=./wheels \
          --file Dockerfile.jetson .

    - name: Upload Artifacts
      uses: actions/upload-artifact@v2
      with:
      name: wheels
      path: |
        wheels/*.whl

    # This ugly bit is necessary if you don't want your cache to grow forever
    # till it hits GitHub's limit of 5GB.
    # Temp fix
    # https://github.com/docker/build-push-action/issues/252
    # https://github.com/moby/buildkit/issues/1896
    - name: Move cache
      run: |
      rm -rf /tmp/.buildx-cache
      mv /tmp/.buildx-cache-new /tmp/.buildx-cache
```

## Future Work

### Extra Optimizations to be made

Some support could still be added for the Nvidia Jetson Nano by adapting the source code, but this is currently out of scope of this project. These optimisations can be found in [QEngineering their post](https://qengineering.eu/install-pytorch-on-jetson-nano.html#imTextObject_80_425)

### GitHub Actions Improvements

Currently, Build Arguments are included but not yet used. In theory, the following can be added to the `docker buildx` command to build for other PyThon versions:

**Python 3.8**

```bash
docker buildx build \
    --platform=linux/arm64 \
    --progress=plain \
    --build-arg PYTHON_MAJOR=3 \
    --build-arg PYTHON_MINOR=8 \
    --output type=local,dest=./wheels \
    --file Dockerfile.jetson .
```

## Conclusion

This project was definitely not easy, taking a long time between builds, figuring out where to build, automating it, ... by sharing this I hope to help the community utilize GPUs more easily with the latest versions.

In a next article, I hope to show you how to run an actual AI model with CUDA enabled on the Nvidia Jetson nano and Python > 3.6 😉 

## References

All of the above was possible by some contributions of others:

* [https://github.com/soerensen3/buildx-pytorch-jetson](https://github.com/soerensen3/buildx-pytorch-jetson) helped me with some of the Dockerfile code (but didn't had CUDA support)






Learn how you can compile PyTorch to run on the Nvidia Jetson with a Python version > 3.6

Xavier Geerinck

Are you struggling with testing and integrating AI into your product?

Featured article

Create an Azure IoT Hub Stream Processor with Dapr