
I am a firm believer of making the entry to barrier as low as possible for starting development environments. Even so much, that in the past I created something named [Brewr](/project/brewr) which had the sole purpose of allowing development environments to be pulled and started from an organisation repository. (Now I think of it, it might be something I will pick up again in the future!). 

## Current Problem

Anyway, coming back to lowering this barrier to entry. Currently, I am working on a project that requires the following scripts:
- Backend
- Frontend
- AI Model
- AI Model Processor
- Tunnel
- Mobile App

As you can imagine, any developer that has to start up these scripts is hesitant to get started. So if we can automate this with one button, they would be less reluctant to do so. For this we have some options, we could use `docker-compose` and wrap them all in a container, we could use a script with `Windows Terminal` but that is Windows Only. So while I am working on my Mac, I was thinking on how we could use `VS Code Terminal` and automate it in there.

## Solution

While researching this, I came across the [`Task`](https://code.visualstudio.com/docs/editor/tasks) option in VS Code that allows [Automating Launching Terminals](https://code.visualstudio.com/docs/editor/integrated-terminal#_automating-launching-of-terminals).

To make this file as simply as possible, just create a `scripts/` directory where you add the files that start-up the respective program. Then combine them all together by creating a task in the `./.vscode/tasks.json` file. For example, we can start our frontend like so:

```json
{
  "label": "Frontend",
  "type": "shell",
  "command": "./scripts/mac/start-frontend.sh",
},
```

Next, we often want to "combine programs" into a logical group, that make it easier to show what belongs together (e.g. a frontend and backend).

```json
{
  "label": "Frontend",
  "type": "shell",
  "command": "./scripts/mac/start-frontend.sh",
  "presentation": {
    "group": "Site"
  }
},
```

Combining all of this together, I created the following `./.vscode/tasks.json` that starts up the programs described earlier:

**tasks.json**

```json
{
  "version": "2.0.0",
  "presentation": {
    "echo": false,
    "reveal": "always",
    "focus": false,
    "panel": "dedicated",
    "showReuseMessage": true
  },
  "tasks": [
    {
      "label": "Start Dev Environment",
      "dependsOn": [
        "Frontend",
        "Backend",
        "Model",
        "Result Processor",
        "Tunnels",
        "Mobile App"
      ],
      "runOptions": {
        "runOn": "default"
      }
    },
    {
      "label": "Frontend",
      "type": "shell",
      "command": "./scripts/mac/start-frontend.sh",
      "presentation": {
        "group": "Site"
      }
    },
    {
      "label": "Backend",
      "type": "shell",
      "command": "./scripts/mac/start-backend.sh",
      "presentation": {
        "group": "Site"
      }
    },
    {
      "label": "Model",
      "type": "shell",
      "command": "./models/models/img-matting/deploy/development/run.sh",
      "presentation": {
        "group": "AI"
      }
    },
    {
      "label": "Result Processor",
      "type": "shell",
      "command": "./scripts/mac/start-model-processor.sh",
      "presentation": {
        "group": "AI"
      }
    },
    {
      "label": "Tunnels",
      "type": "shell",
      "command": "./scripts/mac/start-tunnels.sh",
      "presentation": {
        "group": "Tunnels"
      }
    },
    {
      "label": "Mobile App",
      "type": "shell",
      "command": "./scripts/mac/start-mobile.sh",
      "presentation": {
        "group": "Mobile App"
      }
    }
  ]
}

```

Once added, we can simply run `CTRL + SHIFT + P` and select `Run Task: Start Dev Environment` which will launch the task that we defined.

Automate the starting up of your VS Code Terminal with the respective terminal windows to automatically start your development environment

Automate your Development Environment in VS Code Terminal


> 💡[Check Out all Dapr Posts](/tag/dapr)

Creating a stream processor is always an interesting task, you want to be able to take incoming message and process them one by one, executing some actions along the way. Typically, we would use Apache Spark or Apache Flink to process these incoming streams, but these tools can be quite complex. What if you just have to be able to take incoming messages, read them and write them to a sink while doing some data manipulation along the way?

## Use Case

For this use case, we will be streaming events to [Azure IoT Hub](https://docs.microsoft.com/en-us/azure/iot-hub/) and pick up those events and sink them towards a [CosmosDB](https://docs.microsoft.com/en-us/azure/cosmos-db/).

## Difficulties & Solution

When reading the above, developers instantly think: Damn, we have to integrate IoT Hub (which is annoying in Javascript due to the SDK ecosystem) and sink it towards CosmosDB (yet another API to integrate).

Now, there is a solution for this! By utilizing [Dapr](https://dapr.io) and the [JS SDK](https://github.com/dapr/js-sdk), we can simply configure our 2 connections and utilize the Dapr SDK to connect to Azure IoT Hub and write those events to CosmosDB! If we want to utilize another PubSub or Sink, we can simply redefine our components and it will magically work!

## Creating our Solution


### Directory Layout

In this case start of by [creating a Typescript project](/post/2021/03/24/typescript-getting-started) and adding a `components/` folder, so that your layout will look like this:

```
components/
src/
package.json
README.md
tsconfig.json
```

### Dapr Components

In the layout above, we configure 2 Dapr Components

#### PubSub - IoT Hub

**components/stream.yaml**

```yaml
# https://docs.dapr.io/reference/components-reference/supported-bindings/eventhubs/
# https://docs.dapr.io/reference/components-reference/supported-pubsub/setup-azure-eventhubs/#subscribing-to-azure-iot-hub-events
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: stream
  namespace: default
spec:
  type: pubsub.azure.eventhubs
  version: v1
  metadata:
  - name: connectionString      # Azure EventHubs connection string
    value: "IOTHUB_EH_CONNECTION_STRING"
  - name: consumerID         # EventHubs consumer group
    value: "$Default"
  - name: storageAccountName    # Azure Storage Account Name
    value: "CHECKPOINT_ACCOUNT_NAME"
  - name: storageAccountKey     # Azure Storage Account Key
    value: "CHECKPOINT_ACCOUNT_KEY"
  - name: storageContainerName  # Azure Storage Container Name
    value: "CHECKPOINT_CONTAINER_NAME"
```

#### Binding - CosmosDB

**components/sink.yaml**

```yaml
# https://docs.dapr.io/reference/components-reference/supported-bindings/cosmosdb/
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: sink
  namespace: default
spec:
  type: bindings.azure.cosmosdb
  version: v1
  metadata:
  - name: url
    value: COSMOSDB_URL
  - name: masterKey
    value: COSMOSDB_KEY
  - name: database
    value: COSMOSDB_DATABASE
  - name: collection
    value: COSMOSDB_COLLECTION
  - name: partitionKey
    value: COSMOSDB_PARTITIONKEY
```

### Coding 

After creating the components are configured we can create our source code that will connect to the IoT Hub stream through the JS SDK `await server.pubsub.receive()` and sink it to CosmosDB with `await client.binding.send()`

```javascript
import { DaprClient, DaprServer } from "@dapr/dapr";
import Config from "./config";

const client = new DaprClient(Config.dapr.sidecar.host, Config.dapr.sidecar.portHttp);

async function main() {
  console.log(`Initializing Server (App Port: ${Config.dapr.app.port}, Sidecar Port: ${Config.dapr.sidecar.portHttp}, Sidecar Protocol: HTTP)`);
  const server = new DaprServer(Config.dapr.app.host, Config.dapr.app.port, Config.dapr.sidecar.host, Config.dapr.sidecar.portHttp);

  // Initialize the server to subscribe (listen)
  await server.pubsub.subscribe(Config?.dapr?.components?.pubsub?.eventhub?.name, Config?.dapr?.components?.pubsub?.eventhub?.topic, async (data: any) => {
    await writeToSink(data);
  });

  await server.start();
}

async function writeToSink(message: any) {
  // Add PartitionID
  const partitionId = message.body.my_partition_key; // @todo: set your partition key
  message.body.partitionID = partitionId;

  // Sink the body to CosmosDB
  await client.binding.send(Config?.dapr?.components?.bindings?.cosmos?.name, "create", message.body);
}


main().catch(e => console.error(e));
```

### Testing

Finally, we can test the code above by adding some scripts to the `package.json`:

```
"start": "npm run build && node dist/index.js",
"start:dapr": "dapr run --app-id stream-processor --app-port 10000 --dapr-http-port 10001 --dapr-grpc-port 10002 --app-protocol http --components-path ./components/ npm run start",
```

This will start up dapr, and you will be able to see incoming messages!

Learn how you can create an Azure IoT Hub Stream Processor that sinks data towards CosmosDB

Create an Azure IoT Hub Stream Processor with Dapr


I am working on a Mobile Application (spoiler) that I am developing through [React-Native](https://reactnative.dev/) and [Expo](https://expo.dev/). 

Now to develop this application, I want the mobile app to connect to my backend and frontend seeing that I need to implement authentication (that already gets handled by [Tinx.ai](https://tinx.ai) in the backend)

Up until now I was utilizing [loca.lt](https://theboroer.github.io/localtunnel-www/) instead of the popular [ngrok](https://ngrok.com/) as it allowed me to simply run `lt --port 8000 --subdomain my-domain` and provide me an URL to work with.

This worked perfectly, but had 2 optimizations that I wanted to do:

1. Google was not happy as the redirect URL was not owned by me (it is hosted under loca.lt)
2. The custom subdomains had a small timeout after disconnect, which made it harder to restart them.

Which is why I wanted to look at alternatives and came across [Cloudflare Tunnels](https://www.cloudflare.com/en-gb/products/tunnel/)


## Installing Cloudflared

First get started by installing the Cloudflare CLI tool by [downloading it](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation/)

As I installed this on the WSL environment I could run:

```bash
# Download the latest version
curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb

# Install the package
sudo dpkg -i cloudflared.deb 
```

## Cloudflared Login

Next up, login to your account by running

```bash
cloudflared tunnel login
```

This will let you open an URL and authorize cloudflared for a specific zone.

## Configuring your Tunnel

Now the most important step, let's create a tunnel and route the traffic to it.

Start of by creating a tunnel:

```bash
cloudflared tunnel create my-tunnel
```

The tunnel is now created and will be visible on the Cloudflare [Zero Trust dashboard](https://dash.teams.cloudflare.com/):

![](./dashboard.png)

Once the tunnel is created, route traffic to it:

```bash
# my-tunnel: the tunnel name we created
# my.domain.com: the domain we want to connect to
cloudflared tunnel route dns my-tunnel my1.domain.com
cloudflared tunnel route dns my-tunnel my2.domain.com
```

Before we can start it we need to configure `cloudflared` its configuration file situated at `~/.cloudflared/config.yml`:

```yaml
# Get the tunnel id by running `cloudflared tunnel list`
tunnel: YOUR_TUNNEL_ID
credentials-file: /home/YOUR_USER/.cloudflared/YOUR_TUNNEL_ID.json
warp-routing:
  enabled: true
ingress:
 - hostname: my1.domain.com
   service: http://localhost:3000
 - hostname: my2.domain.com
   service: http://localhost:8000
 - service: http_status:404
```

## Starting your Tunnel

First run the below to configure a [larger UDP buffer size](https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size)

```bash
sudo sysctl -w net.core.rmem_max=2500000
```

Finally, you can start your tunnel by running:

```bash
# my-tunnel: the tunnel name we created
cloudflared tunnel run my-tunnel
```


Learn how you can set up and replace ngrok or loca.lt by Cloudflare Tunnels

Setting up Cloudflare Tunnels


For a recent preparation, I wanted to revisit the creation of a stack in Javascript and how we can do this, taking into consideration changing design aspects as well as efficiency.

## Definition

A stack is a data structure in computer science that works according to the Last-in, First-out (LIFO) principle. The example of a waiter is often given, who is taking plates and stacking them on top of each other. There is only one possibility to take them of, which is to start at the top again, the first plate is thus the last one to go.

## Use Cases

* You want items to go out in the reverse order that you put them in
* "undo" functionality

## Methods

We require a few basic methods to use this data structure:

* `push`: Adding Items to the top
* `pop`: Removing items from the top
* `isEmpty`: See if there are items on the stack
* `isFull`: see if the stack is full
* `peek`: Check the item on the top without removing it

## Attention Points

Before even starting to code, we always have to remember some attention points. Some things I always try to think about:

* **Check input parameters:** Ensure you check to see if the input parameters are correct.
    * I remember a quote here from Facebook stating that if your code has a chance of failing 0.001% of the time, it will fail at scale!
* **Check Performance for your Use Case:** Performance is crucial, but sometimes not needed. When building something, do you need performance (typically takes a bit longer due to thinking process and sometimes implementation)? Or do you need speed of delivery? Don't waste spending hourse on over-optimizing, but design for your use case
* **Think before you do:** before even starting with writing a letter of code, think what you want to do, achieve and what possible ways are. If you can't come up with enough ideas, brainstorm it around.
* **Tests:** Write tests for everything you do, covering the following cases:
    * **Main Case:** The case your test is designed for
    * **Edge Case:** Edge boundaries (e.g. if you expect a number between 0 an 100, check 0 and 100 specifically)
    * **Stretch Case:** What do you think might get hit but is very unlikely?

## Implementation

```javascript
class Stack {
    constructor(size) {
        this.size = size;
        this.items = [];
    }

    push(item) {
        if (!item) {
            throw new Error("INVALID_ITEM");
        }

        if (this.isFull()) {
            throw new Error("STACK_IS_FULL");
        }

        this.items.push(item);
    }

    pop() {
        if (isEmpty()) {
            return null;
        }

        return this.items.pop();
    }

    peek() {
        if (isEmpty()) {
            return null;
        }

        return this.items[this.items.length - 1];
    }

    isEmpty() {
        return this.items.length > 0;
    }

    isFull() {
        return this.items.length >= this.size;
    }
}
```

## Evolution #1: Remove from bottom

We now are adding the evolution of our class and want to be able to remove items from the bottom, how would we do this? How can we do this performantly?

Implementing this can be done quite easily in Javascript. We can utilize either the `shift()` or the `splice()` method for our convenience. Using this `shift()` method, our method named `popFromBottom()` could look like this:

```javascript
shift() {
    if (this.isEmpty()) {
        return null;
    }

    this.items.shift();
}
```

Assessing the performance is tricky now as it includes the `shift()` method which is created by Javascript for our convenience. We thus would need to know the performance of the underlying implementation for this. In any case, we know we have 1 check and 1 method call.

If we would use the `splice()` method in Javascript, we would cut up our array in 2 pieces and "glue" them together afterwards, forming a new array.

## Evolution #2: Remove from the bottom without using `shift()` or `splice()`

Another evolution we can do on this is to forbid certain methods. Languages such as Javascript add a convenience method to underlying data structures which help us. So we should as well know how we can do this ourselves.

E.g. when we forbid the usage of `shift()` and `splice()`, thus we need to think how we can implement the method in an efficient way without any of these methods.

Since we can neither use the `splice` or `shift` methods, we could think of the following naively:
1. Create a new empty array (the size of the existing one - 1 element)
2. Loop over the existing array and add each item, skipping the one at the given index (in our case 0 as it's the beginning)

Can we do better at this point in time with what we know? Can we think of other data structure and algorithms that come to mind?
* Data Structures: Queue, Deque (Double Ended Queue), Linked List, ...
* Algorithms:
    * If size 1, we can push at the end and swap the items in memory (O(1))
    * If size n, we can push at the end and swap towards the front (O(n))
    * If size n, we can copy to a new array and append the items there


Which would result in code looking like this:

```javascript
let newArr = [];
let idxToRemove = 0; // Remove the first item

for (let i = 0; i < this.items.length; i++) {
    if (i == idxToRemove) {
        continue;
    }

    newArr.push(this.items[i]);
}

this.items = newArr;
```

## O Table for Common Data Structures - Average Cases

As a reminder, I also include the typical data structures used to remind myself.

|Data Structure|Access|Search|Inseration|Deletion|
|-|:-:|:-:|:-:|:-:|
|Array|O(1)|O(n)|O(n)|O(n)|
|Stack|O(n)|O(n)|O(1)|O(1)|
|Queue|O(n)|O(n)|O(1)|O(1)|
|Sigly-Linked List|O(n)|O(n)|O(1)|O(1)|
|Doubly-Linked List|O(n)|O(n)|O(1)|O(1)|
|Hash Table|N/A|O(1)|O(1)|O(1)|
|Binary Search Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|
|B-Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|
|Red-Black Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|
|Splay Tree|O(log(n))|O(log(n))|O(log(n))|O(log(n))|

My reasoning behind implementing a simple stack explained through Javascript to prepare for a coding interview

Coding Interview Preparation - Stack Implementation


I always wanted to get more into .NET programming as together with Typescript it is one of my favourite languages as it is flexible and more importantly allows for cross-platform development. Finally, it is adopting a more simplified syntax as well which is coming in .NET 7.

For cross platform mobile app development, .NET had Xamarin and Xamarin.Forms which allowed us to create mobile apps easily. However, some things were lacking (such as Desktop support) and it was more of a standalone product that was not integrated in the .NET BCL (Base Class Library). Which is why Microsoft decided to rewrite the entire approach and name it **MAUI** ([more info can be found here](https://ballardchalmers.com/2021/10/04/what-is-net-maui-and-how-does-it-differ-from-xamarin/) - Thanks [Clifford](https://twitter.com/CliffordAgius) for the amazing write up!)

![](./architecture.png)

You can view a [detailed roadmap of MAUI here](https://github.com/dotnet/maui/wiki/roadmap)

So let's get started and create a mobile application!

## Getting Started

### Prerequisites

- XCode (Make sure to start it once to accept the EULA)
- XCode Terminal Tool
  - If dotnet doesn't build, check if it is set by opening Xcode -> Preferences -> Locations -> Command Line Tools and ensure it is set.
- Android Studio
- Visual Studio 2022 Preview (7+) for Mac
  - Mac sure to install iOS, Android and macOS tools

### Installing MAUI

1. Go to the dotnet website and [download and install the latest dotnet LTS version](https://dotnet.microsoft.com/en-us/download/dotnet)
   - Make sure `dotnet --list-sdks` shows it is installed under `/usr/local/share/dotnet/sdk`
2. Once installed, install maui by running `dotnet workload install maui`

> Warning: Do not install dotnet through brew as this will render you [unable to install maui correctly on mac](https://github.com/dotnet/maui/issues/5939)

## Creating and Running your App

### Create your App

1. Create a directory with `mkdir mobile-app`
2. Create a maui app with `cd mobile-app; dotnet new maui`

### Get Device Identifier

If we would try to run our application, we would get the error as shown below:

![](./vs-error.png)

To resolve this, we should do another thing, which is to get our Device Identifier that we will use to target our device. So follow these steps:

1. Open up Xcode
2. Go to Window
3. Devices and Simulators
4. Select your Device
5. Right Click
6. Copy Identifier

Now run the command below with your identifier:

```bash
dotnet build -t:Run -f net6.0-ios -p:\_DeviceName=:v2:udid=4C4906B6-B3DB-490A-A322-8153E2DBCA85
```

If it opens up the simulator, you can close it again.

### Run your App

Now we can open our app with Visual Studio.

If you were able to run all the steps above, you should now be able to see an interface that looks like the below:

![](./vs-overview.png)

When we now click the big Play button on the top, the simulator will start and we will be able to start working on our application! Including Hot-Reload of the XAML interface, so each change we make will be reflected in the simulator

![](./vs-working.png)

## Conclusion

That's it! See how simple it is to start creating your cross platform mobile application using .NET.


Learn how you can create a cross platform working mobile application utilizing MAU and the .NET ecosystem

Creating a Cross Platform Mobile App with Maui .NET on Mac M1


As the official maintainer of the [Dapr JS SDK](https://github.com/dapr/js-sdk), I am always looking on how we can improve our documentation, tooling and API to integrate Dapr through Javascript. 

Recently, I started working on [Tinx.ai](https://tinx.ai) where I am using Dapr to host the Backend and send events to my event queue. So in this blog article, I wanted to dive a bit deeper on how you can integrate Dapr utilizing [Nestjs](https://nestjs.com/)

## File Structure

The first thing we should do is to create our file structure containing our NestJS project:

> 💡 To learn more on how you can bootstrap a NestJS project, check out their [documentation](https://docs.nestjs.com/first-steps)

```bash
src/
    main.ts             # NestJS Bootstrap (see https://docs.nestjs.com/first-steps)
    app.module.ts       # App Module
    config/config.ts    # Config 
    dapr/
        dapr.module.ts  # Dapr module main entry point
        dapr.service.ts # Dapr Service containing our functions
```

## Create a Nest Dapr Module

Once our file structure is created we can configure our module and make it available to NestJS

### src/dapr/dapr.module.ts

```javascript
import { Module } from "@nestjs/common";
import { ConfigModule } from "@nestjs/config";
import { DaprService } from "./dapr.service";

@Module({
    // List of imported modules
    imports: [ ConfigModule ],

    // List of controllers
    controllers: [ ],

    // Providers that will be instantiated by the Nest injector and that may be shared at least across this module
    providers: [ DaprService ],
    
    // which providers can be used in other modules?
    exports: [ DaprService ]
})
export class DaprModule {}
```

The code above will utilize the Config module (which we use to inject configuration into our service later on) as well as the Dapr Service that we will create that holds the Dapr JS SDK methods.

Finally, register this module in the `app.module.ts` file:

```javascript
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import configuration from '../config/config';
import { DaprModule } from './dapr/dapr.module';

@Module({
  imports: [
    ConfigModule.forRoot({
      load: [configuration],
    }), // https://docs.nestjs.com/techniques/configuration
    DaprModule
  ],
  controllers: [],
  providers: [],
})
export class AppModule;
```

### src/dapr/dapr.service.ts

Now we have our module registered, let's create the service class that helps us access the Dapr JS SDK:

```javascript
import { Injectable, Logger } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { DaprClient } from 'dapr-client';

@Injectable()
export class DaprService {
  daprClient: DaprClient;
  private readonly logger = new Logger(DaprService.name);

  constructor(
    private readonly configService: ConfigService
  ) {
    const daprHost = this.configService.get<string>('third_party.dapr.host');
    const daprPort = this.configService.get<string>('third_party.dapr.port');

    this.logger.log(`Initializing DaprClient("${daprHost}", ${daprPort})`);
    this.daprClient = new DaprClient(daprHost, daprPort);
  }
}
```

As you can see, we are accessing `third_party.dapr.host` and `third_party.dapr.port` here that pull information from the `config/config.ts` file. So go ahead and utilize the following config:

```javascript
export default () => ({
  third_party: {
    dapr: {
      host: process.env.DAPR_SIDECAR_HOST || '127.0.0.1',
      port: process.env.DAPR_SIDECAR_PORT || '3500',
    }
  },
});
```

## Utilize the Nest Module

Now we created our module, we can import it to any of our Nest modules (add it under `imports: [ DaprModule ]`, [more info here](https://docs.nestjs.com/modules)) and start utilizing it.

```javascript
import { Controller, Get, HttpCode, Req, Logger } from '@nestjs/common';
import { ApiTags } from '@nestjs/swagger';
import { DaprService } from '../dapr/dapr.service';

@Controller('demo')
@ApiTags('demo') // Attaches controller to a specific tag - https://docs.nestjs.com/websites/swagger#tags
export class DemoController {
  private readonly logger = new Logger(DemoController.name);

  constructor(
    private readonly daprService: DaprService,
  ) { }

  @Get('/')
  @HttpCode(200)
  async demo(@Req() req): Promise<void> {
    await this.daprService.daprClient.binding.send(`my-component`, "create", { hello: "world" });
  }
```


## Start Nest with Dapr

To start all of this, we can now utilize the `dapr run` command, which will create our process with Dapr in it.

```
dapr run --app-id my-application --app-protocol http --app-port 50001 --dapr-http-port 3500 --components-path ./components npm run start
```

Learn how you can utilize Dapr with Nest.js and the Dapr JS SDK

Integrate Dapr with Nest.js and the Dapr JS SDK


When wanting to work with AI models on my latest Macbook Pro 2021 with M1 Architecture, I received the following error while running `pip install onnxruntime`:

```
ERROR: Could not find a version that satisfies the requirement onnxruntime>=1.10.0 (from tinx) (from versions: none)
```

After doing some research, this error was not unknown. But how can we now get it resolved on this Macbook? When we look at the GitHub repository of [onnxruntime](https://github.com/microsoft/onnxruntime/releases/tag/v1.10.0) we can see that an OSX version is actually being released. 

Luckily, PyPi has a "Test" repository as well which we should try when the normal doesn't work, as it often hosts packages that are not found on the public one yet. So we can simply run the below which will install `onnxruntime`. Note, that we install `flatbuffers` as well, since we will else encounter an error stating that it is depending on flatbuffers.

```bash
# ONNX Depends on this, so we should first install it
pip install flatbuffers

# Then we can install onnxruntime
pip install -i https://test.pypi.org/simple/ onnxruntime
```

If we now run the below, we will see the `onnxruntime` printed and we will be able to infer AI models on our Macbook.

```python
import onnxruntime as rt
print(rt.__version__)
```

Getting ONNX to work on your Macbook is tricky as it throws back quite some errors. Learn here how you can get it working

Running ONNX with onnruntime on your M1 Mac


In my [previous post](/post/2021/12/20/coding-ai-dapr-container-apps) I explained how you can get started with Container Apps. The one missing piece of that puzzle is how you can utilize a queue behind to dynamically scale your workload based on items in a queue! This has been one my most difficult blog items so far, taking me through a range of different issues with Container Apps and Dapr, but which definitely aided those services and myself in becoming more mature.

The following items have been created because of this:

- [Dapr: Python SDK - Input Bindings for HTTP](https://github.com/dapr/python-sdk/issues/342)
- [Log Analytics: Difference between TimeGenerated and \_timestamp_d #79](https://github.com/microsoft/azure-container-apps/issues/79)
- [Support for setting Dapr App Protocol to gRPC](https://github.com/microsoft/azure-container-apps/issues/70)
- [Container Apps seems to be missing appProtocol](https://github.com/Azure/azure-rest-api-specs/issues/17144)
- [Azure Container Apps not Updating Docker Container](https://github.com/pulumi/pulumi-azure/issues/1007)
- [Metadata (and queryString?) should use an Array Type instead of Tuple Type](https://github.com/dapr/python-sdk/issues/328)
- [Azure Container App Demonstration](https://github.com/dapr/python-sdk/issues/316)

This entire process took me around 2 months to complete, but once figured out it's super simple!

## Source Code

The entire source code for this project can be found on my [Public GitHub repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/Azure-Container-Apps/AI-Model-YoloV5-Dapr-Queue)

## Using a Queue - Made easy through Dapr

We will be using a queue in our source code. Luckily for us, this can be done easily by utilizing [Dapr](https://dapr.io) which takes away a lot of the implementation for the different queueing systems and their retry logic. For this project, I decided to utilize the [Service Bus Queue](https://docs.dapr.io/reference/components-reference/supported-bindings/servicebusqueues/).

To set this up, we create a component spec such as the below:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: my-queue
  namespace: default
spec:
  type: bindings.azure.servicebusqueues
  version: v1
  metadata:
    - name: connectionString # Required
      value: "YOUR_CONNECTION_STRING"
    - name: queueName
      value: worker-items
    - name: ttlInSeconds
      value: 60
```

> **Note:** The component spec for Container Apps is defined differently, but more about that soon.

Once this is created, we can hook up our application to the queue by creating an [Input Binding](https://docs.dapr.io/developing-applications/building-blocks/bindings/howto-triggers/) that will monitor the Service Bus queue and call the defined POST endpoint. When looking to the component spec above, we defined the name `my-queue` which will result in a POST call to `/my-queue`.

The code below shows us how we can process items from this binding.

```python
from pydantic import BaseModel
from fastapi import FastAPI
from dapr.ext.fastapi import DaprApp

# Define Request DTO
class WorkerItemRequestDto(BaseModel):
    id: str
    image_url: str

# Add our Routes
@app.options("/my-queue")
async def processor_worker_item_options():
    return

@app.post("/my-queue")
async def processor_worker_item(item: WorkerItemRequestDto):
    return
```

## Implementing YoloX

Now we know how to get items in, we just have to initialize our model and run the item through it. As we did before, we initialize the model, download our image and print the result:

```python
# First load the model as we will always need this
print("Loading Model", flush=True)
model = torch.hub.load("ultralytics/yolov5", "yolov5s", pretrained=True)
model.eval()

# ... Other code

@app.post("/my-queue")
async def processor_worker_item(item: WorkerItemRequestDto):
    return handler_image_url(item.image_url)

# Handling the Image URL
def handler_image_url(image_url):
    print(f'Received: image_url="{image_url}"', flush=True)

    if not image_url:
        return

    # Download the image
    response = requests.get(image_url)
    img = Image.open(io.BytesIO(response.content))

    # Infer result
    results = model(img, size=640)
    bboxes = results.pandas().xyxy[0].to_json(orient="records")
    bboxes = json.loads(bboxes)

    # Print the Result
    items = {}
    for bbox in bboxes:
        items[bbox["name"]] = 1 if bbox["name"] not in items else items[bbox["name"]] + 1

    print(f'Results:', flush=True)
    for key, value in items.items():
        print(f'- {key}: {value}', flush=True)

    return
```

## Running Locally

The above we can run locally by utilizing the `dapr run` command:

```bash
# Send 5 Events to Service Bus Queue
dapr run --app-id=send-events --app-protocol http --components-path=./example/components python example/send.py 5

# Start Server to Process the Events
cd app/code/src
dapr run --app-id=yolox --app-protocol http --components-path=../../../example/components -- uvicorn main:app --host 0.0.0.0 --port 5000
```

Resulting in our application receiving events and processing them! 🤩

![](./demo-local.gif)

## Running in Cloud with Container Apps

Now the power of all of this is that we can enable Serverless AI! To achieve this, we will utilize [Azure Container Apps](https://azure.microsoft.com/en-us/services/container-apps/) that run our container whenever there is an item in our queue. When there are no items in the queue, we use a [KEDA scaler](https://keda.sh) to scale down to 0 instances!

The below illustrates how we utilize secrets that configure the Dapr Component as well as the KEDA scaler:

```javascript
const containerApp = new web.ContainerApp("app", {
  resourceGroupName: resourceGroup.name,
  location: glbLocation,
  kubeEnvironmentId: kubeEnv.id,
  configuration: {
    // ... see source code
    activeRevisionsMode: ActiveRevisionsMode.Single,
    secrets: [
      {
        name: "acr-pwd",
        value: outAcrAdminPassword,
      },
      {
        name: "sb-connection-string",
        value: serviceBusNamespaceKeys.primaryConnectionString,
      },
    ],
  },
  template: {
    revisionSuffix: `${new Date().getTime()}`,
    containers: [
      {
        name: `container-${GLB_APP_ID}`,
        image: image.name,
        env: [
          // ... see source code
        ],
      },
    ],
    // https://keda.sh/docs/2.5/scalers/azure-service-bus/
    scale: {
      minReplicas: 0,
      maxReplicas: 3,
      rules: [
        {
          name: "rule-servicebus-scaler",
          custom: {
            type: "azure-servicebus",
            metadata: {
              queueName: "worker-items",
              messageCount: "2",
            },
            auth: [
              {
                secretRef: "sb-connection-string",
                triggerParameter: "connection",
              },
            ],
          },
        },
      ],
    },
    dapr: {
      enabled: true,
      appId: GLB_APP_ID,
      appPort: GLB_APP_PORT,
      // appProtocol: 'http', // Disabled for now since unsupported by ARM CLI
      components: [
        {
          name: "my-queue",
          type: "bindings.azure.servicebusqueues",
          version: "v1",
          metadata: [
            {
              name: "connectionString",
              secretRef: "sb-connection-string",
            },
            {
              name: "queueName",
              value: "worker-items",
            },
            {
              name: "ttlInSeconds",
              value: "60",
            },
          ],
        },
      ],
    },
  },
});
```

To set up this infrastructure, we can simply run the following command:

```bash
pulumi up \
    --stack demo-cae-yolox \
    -c glb-location=northeurope \
    -c glb-project-name=demo-cae-yolox \
    -c glb-project-env=prd \
    -c sb-topics="worker-items"
```

> For setting up the entire infrastructure, I refer to the Pulumi file created in the [source code](https://github.com/XavierGeerinck/PublicProjects/blob/master/Azure-Container-Apps/AI-Model-YoloV5-Dapr-Queue/index.ts).

We can now view the logs of our application in Log Analytics with the query:

```bash
ContainerAppConsoleLogs_CL
    | project TimeCreated = unixtime_milliseconds_todatetime(_timestamp_d * 1000), ContainerAppName_s, Log_s
    | order by TimeCreated desc
    | take 150
```

![](./demo-azure.png)

## Conclusion

This article might not sound as something worth spending 2 months on! However, when looking at the benefits Container Apps brings in the field of Serverless AI, it's well worth spending this time! We can now finally run models, inference a result and shut it down again without needing to have a compute instance running 24/7 which can end up being expensive when using GPUs.

I hope you enjoyed this article, feel free to put a comment below or contact me!


Serverless AI is a game changer in reducing operational costs! But how can we achieve this? Learn more about it here

Enabling Serverless AI with Azure Container Apps and Dapr


Recently I've been working a lot with different AI Models for a variety of use cases, ranging from simple object classification to massive crowd counting models. While working with these models, I saw that it is always difficult to implement a model in custom code since the Architecture is never included in the weights themselves, thus each model works differently.

That's when I came across a format named [ONNX](https://onnx.ai/) which instantly peaked my interest with promising points such as:
* Improved Inference Performance 
* Interoperability

## Prerequisites

To get started with ONNX, we have two tasks that should be done

1. Configure the [ONNX Execution Provider](https://onnxruntime.ai/docs/execution-providers/)
2. Get an ONNX Model to infer

### Configuring the ONNX Execution Provider for CUDA 

To be able to run on our GPU I chose to utilize the [CUDA Execution Provider](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html). Which has 2 major components that should be installed:

* CUDA
* cuDNN

#### Installing CUDA

CUDA is the easiest of the two to install since it is publically available without a login wall. Before we start, check if CUDA is already installed by running the below, which should finish correctly:

```bash
# Check if CUDA is available
nvidia-smi
nvcc --version
```

If the above is not available, we can install it (**on WSL 2**) by running the below:

> If you want to install it on another target, feel free to check the original [link](https://developer.nvidia.com/cuda-toolkit-archive)

```bash
# View all: https://developer.nvidia.com/cuda-toolkit-archive
# View ONNX Supported: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0

# Install CUDA
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local_${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}.${CUDA_VERSION_PATCH}-1_amd64.deb
sudo apt-key add /var/cuda-repo-wsl-ubuntu-${CUDA_VERSION_MAJOR}-${CUDA_VERSION_MINOR}-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

# Add CUDA to Path
# https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions
# Note: add those to ~/.bashrc at the bottom
export PATH=/usr/local/cuda-11.5/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

> Note: Do not forget to add the export links to your ~/.bashrc file

#### Installing cuDNN

Now up to the other requirement, to install `cuDNN`.

```bash
OS="wsl-ubuntu"
CUDA_VERSION_MAJOR=11
CUDA_VERSION_MINOR=4
CUDA_VERSION_PATCH=0
CUDNN_VERSION_MAJOR=8
CUDNN_VERSION_MINOR=2
CUDNN_VERSION_PATCH=4

wget https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/cuda-${OS}.pin 

sudo mv cuda-${OS}.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/7fa2af80.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/${OS}/x86_64/ /"
sudo apt-get update
sudo apt-get install libcudnn8=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
sudo apt-get install libcudnn8-dev=${CUDNN_VERSION_MAJOR}.${CUDNN_VERSION_MINOR}.${CUDNN_VERSION_PATCH}-1+cuda${CUDA_VERSION_MAJOR}.${CUDA_VERSION_MINOR}
```

## Running the YOLOX Model

Now we have all the required software installed for the CUDA Execution Provider, we can start running a model and infer from it. Luckily for us, the [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) team provides a [ready-to-use ONNX model](https://github.com/Megvii-BaseDetection/YOLOX/tree/main/demo/ONNXRuntime) that we can use. So let's download it and use it!

To keep the code part short, you can find the entire project on [my public repository](https://github.com/XavierGeerinck/PublicProjects/tree/master/AI/ONNX/yolox) the main part to know is how we can infer an ONNX Model.

Infering an ONNX Model is as easy as creating a session that defines the model path as well as the provider to use

```python
# Create an ONNX Inference Session
session = onnxruntime.InferenceSession(
    path_or_bytes=args.model,
    providers=["CUDAExecutionProvider"]
)
```

After that is done, we just need to provide it an input:

```python
ort_inputs = {session.get_inputs()[0].name: img[None, :, :, :]}
output = session.run(None, ort_inputs)
```

Whereafter we can run all of this and receive our result!

```bash
# Create directories
mkdir model; mkdir images; mkdir output

# Download Model
wget -O ./model/yolox_l.onnx https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_l.onnx 

# Download Example
wget -O ./images/example.jpg https://www.ikea.com/ext/ingkadam/m/7f8f282fb240f466/original/PH179208-crop001.jpg?f=m

# Run
python src/main.py -m ./model/yolox_l.onnx -i ./images/example.jpg -o ./output -s 0.3 --input_shape 640,640
```

## Conclusion

As you can see from the above, it is quite easy to infer an ONNX Model and get results from it! I personally think I will utilize ONNX more in the future, reducing the complexity of my AI pipeline codes even further.

Learn how to run AI inferencing using ONNX on your GPU

Xavier Geerinck

Latest Posts