
In my previous post I revisited the Bonsai platform. Now it's time to actually start preparing to put it into production! (this will become more clearer over time 😉) To start off with a spoiler: Project Bonsai - It's impressive! 😎

Everything in this article is the result of around 100 hours of work! Starting entirely from scratch with the following tasks that were executed:
* Acquiring the hardware
* Creating own Firmware for the hardware
  * Custom OLED screen software
  * Motor driver
  * Motor Encoder handler
  * Angular Displacement Encoder handler
  * Serial communication
  * ...
* Training a Bonsai Brain
* Deploying a Bonsai Brain
* Creating the Serial Interface to Bonsai Brain
  * Note: The Bonsai Docker Container is running on WSL 2, while Serial is not.

But all of this made it worth it, producing the following:

<iframe width="560" height="315" src="https://www.youtube.com/embed/Eq0X_50fEx8" title="YouTube video player" frameborder="0" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

So let's kick-off where we ended last time, after [training a brain to keep a CartPole balanced.](/post/ai/rl/bonsai-introduction-revisiting)

## Brain (Model) Export

### Creating a Docker Container

We now have a model trained to our specifications and as we can see, it performs quite well in the simulator! 

> 💡 It's important to note that a simulator != the real world! This means that certain aspects such as motor stutter, correct duty cycle measurement, ... are not going to act as we expect them to in the simulator!

Since we are satisfied with the results, we can now **export our brain**. By exporting our brain, we are basically creating a Docker container (automatically through the Bonsai Platform) and upload it on an [Azure Container Registry](https://azure.microsoft.com/en-us/services/container-registry/).

![](./bonsai-export.png)

Once this upload is finished, we will be given the ability to pull it locally through a few simple commands.

![](./bonsai-export-3.png)

### Diving deeper in the Docker Container

Of course we do not want to run this blindly and go a bit more in-depth on what this container actually contains! So let's boot it up and run a quick `exec` to get into it:

```bash
docker exec -it your_container /bin/bash 
```

![](./bonsai-docker-1.png)

When comparing the root folder structure with `ls -la` to a normal Linux image, we can instantly see some extra folders and files that are not there by default!

![](./bonsai-docker-2.png)

Dissecting these files:

* **info.json**: Contains some information about Project Bonsai their inference API
* **brain.pb**: Not verified, but after a dissassembly, it seems that this contains the Protobuf definitions to interact with the created model in an easy way. This would make sense seeing that we programmed everything in Inkling and want to interact with the API now in an easy way.
* **saved_model.tflite**: Our created model saved as a [Tensorflow Lite](https://www.tensorflow.org/lite) format making it so it will run fast on smaller devices!
* **sample/** Contains some demo code, but seems to be a leftover, so we can discard this.

All of the above is interesting, but what is our container actually executing? When we look at the process, we can see a process running named `/usr/local/bin/predict`.

> 💡 I'd love to go more in-depth on this one, but it seems to be C code, so not going to bother decompiling and analyzing it. 🙂

![](./bonsai-docker-4.png)

### Calling our Brain (Model) API

Now we know what is running, we can see that a HTTP server is being started on port `5000` which loads in our brain file. Documentation for this is being exposed on the url: `http://localhost:5000/v1/doc/index.html` in a Swagger interface.

There we can see that we can call our API by executing a `POST` or `GET` call to the endpoint `http://localhost:5000/v1/prediction` with our JSON body. In more detail:

```bash
# Full Example
POST|GET http://localhost:5000/v1/prediction

{
  "cart_position": 0,
  "cart_velocity": 0,
  "pole_angle": 0,
  "pole_angular_velocity": 0
}

# Curl Example
curl -X POST "http://localhost:5000/v1/prediction" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"cart_position\":0,\"cart_velocity\":0,\"pole_angle\":0,\"pole_angular_velocity\":0}"
```

Which will return the command (result) to be executed by our Agent:

```json
{
    "command": -0.015466135926544667
}
```

When running this through Postman we get:

![](./bonsai-api-1.png)

## Running a Real CartPole Example

So everything is set for us to run this on a physical CartPole example! For that I have build my own Firmware on a [premade CartPole that I bought on Aliexpress](https://nl.aliexpress.com/item/1000006732135.html). This Firmware sends observations and takes the commands as actions to drive the motor!

> ⚠️ I have built my own Hardware Firmware to interact with the CartPole over Serial, but will not be covering that in this post! 

To communicate over the Serial Interface, I wrote a quick Serial interaction program in Javascript that sends the received Serial Observations towards the Bonsai Container and writes back the received command over Serial.

```javascript
// Bonsai
const req = await fetch('http://<YOUR_IP>:5001/v1/prediction', {
  method: 'POST',
  body: JSON.stringify({
    "cart_position": x,
    "cart_velocity": v,
    "pole_angle": theta,
    "pole_angular_velocity": omega
  })
})

const res = await req.json();
const cmd = res.command;

// We utilize write since it converts floats so we can send it over UART
await arduino.writeFloat(cmd);
```

All of this gets printed to console in lines such as the ones below. Showing an impressive `35ms` total roundtrip! Of these 35ms, **25ms are a delay in the main loop (for motor stability)**, giving an average delay of `10ms` for Bonsai inference roundtrip over the HTTP endpoint!

```bash
# n (idx), x (cart position), v (cart velocity), θ (pole angle), ω (pole velocity), a (brain action), time previous, time current, difference 
1945;0.018535396084189415;0.3903769850730896;0.006132812704890966;-0.7007526755332947;0.19268926978111267;1618763600647;1618763600682;35;
```

If I now run this, I get a CartPole that is being steered by the Bonsai Brain! Resulting in the following video:

<iframe width="560" height="315" src="https://www.youtube.com/embed/qP6fRIIKy6o" title="YouTube video player" frameborder="0" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>


## Lessons and Conclusion

All of the above did not happen overnight as mentioned earlier and took around 100 hours to complete. 

### Lessons / Takeaways

Since I am not a hardware programmer by profession, it was quite interesting to see how I could interact with all of this. Some of my main lessons are:

* Do not reinvent the wheel, utilize good frameworks such as Arduino to help you
* Next time when acquiring hardware, check the flash memory size... 32Kb is definitely not enough anymore for larger frameworks such as RTOS
* Motors are interesting to steer, PWM steering is not that easy initially and you should start with understanding how clocks work
  * Here I utilized an STM32F103C8 module with the internal clock TIM3 to steer the motor!
* Start out with smaller projects dedicated for sub-parts (e.g. motor driver, oled screen steering, ...)
* Think about the numbers! If your pole goes left and negative, make sure that when the cart goes left it's also along the negative axis... (I swapped them which lost me 15h 😅)

### Conclusion

Finally, I would like to say that I am impressed by what the Bonsai Platform has to offer! Integrating this was a breeze through the delivered HTTP interface, and I was not limited at all by the performance! From what I could tell (I should measure this) inference took only 1ms and the rest was overhead / delays I introduced to make all of this work. I am very happy they made the choice to go the Protobuf route for this! (interesting fact, I actually did this myself through [Roadwork-RL](https://github.com/roadwork/roadwork-rl) with Protobuf, and it's a pain to do so!)

Once our simulation training has been finished and our Reinforcement Learning model was created, it's now time to integrate it with the physical world!

Azure Project Bonsai - Physical CartPole Project


It has been almost an entire year ago since I have visited the Bonsai platform that I covered in my [previous post](/post/ai/rl/bonsai-introduction). Since then a lot changed on the platform, so the time has come to revisit it and see in more detail what exactly changed (on the surface of course)!

> Next to that, I have been working on a new project - more details soon 😉 - which made me revisit the introduction I made last time of the Bonsai platform.

From personal experience, I can definitely say that user experience is everything in platforms such as this one! It is not easy to combine the personas of a Data Scientist, Application Developer and Subject Matter Expert (SME) into a platform that allows the latter to define their own experiments.

## Azure Deployment

Deploying is as easy as going to the Azure portal, searching for Bonsai and filling in the parameters!

![](./bonsai-create-1.png)

![](./bonsai-create-2.png)

## Bonsai - First Launch

Upon launching the Bonsai platform for the first time, we are greated by an amazing getting started page! So let's click on the Cartpole example again and create our cartpole example.

![](./bonsai-getting-started-1.png)

![](./bonsai-getting-started-2.png)

## Bonsai - Interface

Now our example has been deployed, let's see what we notice on first sight when comparing them to the old version!

### Initial Findings

The first thing I noticed is that the code got defined a bit simpler, removing the boundaries that were put on there (which we would mathematically represent as `[ -30, 30 ]` for example).

![](./bonsai-diff-1.png)

But the biggest difference is the concept graph! 😍 This one became super sleek! Clearly defining the inputs, what it will stop upon and the outputs.

> Note: My OCD did catch a out-of-center point definition 😉

![](./bonsai-diff-2.png)

### Concept Graph - A more in-depth view

Going a bit deeper on the Concept Graph, we can clearly see:

* The amount of inputs
* The Goals on which is should learn on -> what do we want to avoid? 
  * Pole should not fall over
  * Cart should stay within the track length

![](./bonsai-deep-dive.png)

More amazingly is that when we add inputs and outputs, the interface changes accordingly!

![](./bonsai-deep-dive-2.png)

## Training

The biggest change is the friendliness in training. When we are training, we can now clearly see our reward and the blue line more expressed showing what we as a user should care about the most (average performance) and our goal score.

![](./bonsai-training-diff-1.png)

Next to that, the concept graph was implemented here as well for progress tracking. What is unclear for me however is what "assessments" mean.

![](./bonsai-training-diff-2.png)

In more detail we can see training below

![](./bonsai-training-1.png)

![](./bonsai-training-4.png)

Which lets us check our experiment running and the parameters it is seeing in more detail! It's simply amazing to be able to see why certain actions are being taken based on the observations received.

![](./bonsai-training-2.png)

![](./bonsai-training-3.png)

## Remarks

* Personally, I think that **Inkling is necessary for the ecosystem, but might be scaring people off**. It's an easy language to pick up, if you are a programmer. But when you are new to it and never coded before, it might be too much. I would personally love to see this evolve to a no-code solution.
* When first starting the interface, the concept of "Brain" is used. Knowing this platform from before, I am comfortable with it, but it might be confusing for newer people.
* The live visualization is very cool but its working could be clearer, which instance is it showing? Why is it only changing once every 10 seconds? ...
* **Spinning up a certain amount of simulators is slow** with no progress showing what is happening (it just states that it is looking for simulators)
* **Performance in training increased a lot!** I only had to spend 7 minutes to get to a decent level
* **Export brain / integration is not as straightforward** with just a simple small button on the right top that looks hidden. **I would love to see a 3th tab that says "integrate"**

## Summary

The Bonsai platform definitely evolved for the better and became more user friendly! Before the concept graph was not that clear, not showing the amount of inputs or outputs, which has been tackled now. 

In this revisitation of the platform I just wanted to see the instant changes when opening the platform. Of course this is not the only thing, so stay tuned for more in-depth coverage as I continue using it 😉





Let’s take a look at the Azure Project Bonsai platform to see what changed the past year.

The Azure Project Bonsai Platform Revisited


For a project I needed a way to automatically schedule pods to be deleted every 3 hours. This was due to the nature of the project where we wanted a cleanup in memory due to zombie processes appearing. It's not a clean fix, but it was necessary for production to run stable while we were patching things.

Setting this up is quite easy actually! Just create the script below and save it to a file of your preferences (e.g. `deploy-cronjob.yaml`), finally, applying it through `kubectl apply -f deploy-cronjob.yaml`.

What this YAML does:
1. It creates the necessary Service Accounts and Roles and Role Bindings to have the permissions to let an automated job run `kubectl delete` 
2. It spins up a `bitnami/kubectl` container
3. It runs the `/bin/sh -c <your_command>` that will get executed

> ⚠️ We utilize `/bin/sh -c` since we utilize command substitution which is only available in the Shell.

**deploy-cronjob.yaml**

```yaml
# Apply With:
# kubectl apply -f deploy-cronjob.yaml

# Validate with:
# kubectl get cronjob

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: deleting-pods
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deleting-pods
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "patch", "list", "watch", "delete"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deleting-pods
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: deleting-pods
subjects:
  - kind: ServiceAccount
    name: deleting-pods

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: deleting-pod-myname
  namespace: default
spec:
  concurrencyPolicy: Forbid
  schedule: "0 */3 * * *" # At minute 0 past every 6 hours
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 600
      template:
        spec:
          serviceAccountName: deleting-pods
          restartPolicy: Never
          containers:
            - name: kubectl
              image: bitnami/kubectl
              command: [ "/bin/sh", "-c" ]
              args: 
                - 'kubectl delete pod $(kubectl get pod -l app=<your_label> -o jsonpath="{.items[0].metadata.name}")'
```

Learn how to schedule a cronjob in Kubernetes that will delete pods automatically and restart it through the deployment

Schedule Kubernetes CronJob to restart pods automatically


Once in a while a new project comes out that I am totally hyped about! [Dapr](https://dapr.io) is such a Project! In this article I would like to go in-depth on how we can quickly (< 10min!!!) create a Microservice that will send an email through a Microservice invocation mechanism!
## Dapr Introduction

In my own words, [Dapr](https://dapr.io) takes away the burden of integrating all these pesky APIs and allows you to focus on what counts: your business logic. Next to that, [Dapr](https://dapr.io) also implements building blocks for Service-to-Service Invocation, State Management, Publish and Subscribe, Actors, Observability, Secret Management all while being extensible!

## What will we create?

In this article I want to create an Email Microservice which should be able to do the following:

* We should be able to call a `send` endpoint from *another* microservice
* We should be able to provide `subject` and `emailTo` email 
* I want to utilize [Twilio Sendgrid](https://www.twilio.com/sendgrid/email-api)

> 💡 I am utilizing Sendgrid, but Dapr allows us to *instantly* swap it out for any other supported binding as described in [the documentation.])(https://docs.dapr.io/operations/components/setup-bindings/supported-bindings/).

## Bootstrapping our project

For this project I will be utilizing `Typescript`, since this is not the scope of the article to set this up, I recommend reading [my previous article - bootstrapping a Typescript Project](/post/coding/javascript/typescript-getting-started) on how you can set this up yourself in a few commands.

## Creating an Email Microservice

Once our initial Typescript project is bootstrapped, add a `components` folder as well for our Dapr components to be stored in. The final directory structure will then look like this:

```bash
.
├── README.md
├── components  # Holds all our bindings for Dapr
├── dist        # Holds the compiled code
├── node_modules
├── package-lock.json
├── package.json
├── src         # Holds the source code
└── tsconfig.json
```

### Initializing our Dapr Component

In the `components` folder we just created we keep all the `.yaml` descriptions for Dapr to know how to configure its components. For sendgrid we create the following file:

**components/binding-sendgrid.yaml**

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: binding-sendgrid # the name of the component
  namespace: default
spec:
  type: bindings.twilio.sendgrid
  version: v1
  metadata:
  - name: emailFrom
    value: "info@youremail.com" # optional 
  - name: apiKey
    value: "<YOUR_SENDGRID_KEY>" # required, this is your SendGrid key
```

### Creating the Email Service

Once that is done, we need to create the email service itself. The way this works is:

1. We install the Javascript SDK (`npm i -S @roadwork/dapr-js-sdk`)
1. We initialize the Javascript SDK (`let client = new Dapr(...)`)
2. We listen on incoming requests from the Dapr Invoker (`client.invoker.listen(...)`)

Therefore, we put the following in our `index.ts` file:

**src/index.ts**

```javascript
import Dapr, { InvokerListenOptionsMethod, Req, Res } from "@roadwork/dapr-js-sdk";

// Dapr is running completely internal, so we connect to 127.0.0.1:3500
const client = new Dapr("127.0.0.1", 3500);

// Implements the onSend method for when the Dapr invocation happened
async function onSend(req: Req, res: Res) {
  const subject = req.body?.subject;
  const emailTo = req.body?.emailTo;

  if (!subject || !emailTo) {
    throw new Error(JSON.stringify({
      error: "Missing subject or emailTo fields"
    }));
  }

  await client.binding.send("binding-sendgrid", "<html><body>Hello World</body></html>", {
    emailTo,
    subject
  });

  console.log("Done Sending Email")

  return res.json({
    isDone: true
  })
}

// Initialize the SDK to listen on the POST /send endpoint
// Note: in the SDK this spins up a server that listens on a port. This port is defined through the DAPR_APP_PORT environment variable, so we need to open it when starting Dapr!
async function main() {
  client.invoker.listen("send", onSend, { method: InvokerListenOptionsMethod.POST })
}

main()
  .catch((e) => {
    console.error(e);
    process.exit();
  })
```

Now this is done, we can start our application utilizing the command below. This command might seem complex, but it's actually quite straightforward:

* We start our application `npm run start:dev`
* We ensure that Dapr waits on our application listening on its internal port, which is set in the SDK by the environment variable `DAPR_APP_PORT` so we configure this and set the Dapr `--app-port 3001` as well
* We want Dapr running on port 3500 for debugging purposes (we can send local requests to http://127.0.0.1:3500 as the dapr endpoint)
* We define our components path as `./components/` where Dapr will look for the bindings (or other components such as PubSub)

```bash
DAPR_APP_PORT=3001 dapr run --app-id msvc-email --app-port 3001 --dapr-http-port 3500 --components-path ./components/ npm run start:dev
```

## Testing the Email Microservice

Everything is in place now to run the code, but we should still create some piece of code to quickly test this. Well, wrong! In normal SDKs we would have to do this, but Dapr actually implemented that with the CLI!!

We thus install the Dapr CLI ([see documentation](https://github.com/dapr/cli)) which we initialize with `dapr init` 

> 💡 This initializes Dapr self-hosted on our container environment

Once that is done we can utilize the internal `invoke` command to trigger our application (a dream come through for testing 😉) so run the command below to trigger this:

```bash
dapr invoke --app-id msvc-email --method send --data '{"subject": "my-subject", "emailTo": "info@your-email.com"}'
```

Which will kick of the Sendgrid binding and send an email towards your inbox!

## Conclusion

So what seemed complex actually wasn't! In just 40 lines of code (which we can simplify to ~3 if we are not counting validation, ...) we created a Microservice that can be called and implements an email sending service. Next to that we have the added benefits that scale-out is trivial, monitoring is out of the box, SDKs can be utilized for the invocation and most importantly service discovery is done for us! Thus, not requiring a central repository.

Create your own Email sending Microservice with Typescript / Javascript utilizing Dapr!

Tutorial - Creating an Email Microservice with Typescript and Dapr


Since I write a lot of Typescript projects, it's quite handy to have an explanation on how to set-up a typescript project quickly.

For this I start by **creating a working directory** for my project. E.g. **my-project/** and navigate to it.

## 1. Installing NPM + Typescript

To set-up a Typescript project, execute the following and then following the specific steps for your Shell of choice:

```bash
# Init NPM
npm init -y

# Install Dependencies
npm install typescript ts-node nodemon rimraf --save-dev
npm install @types/node --save
./node_modules/.bin/tsc --init

# Create Base File
mkdir -p src
touch src/index.ts

# Send content to the file
cat <<EOF >> ./src/index.ts
async function main() {
  console.log("Hello World")
}

main()
.catch((e) => {
  console.error(e);
  process.exit();
})
EOF
```

## 2. Configuring Package.json 

After the project is set-up, we have to adapt our `package.json` so that we are able to execute `npm run start:dev` to start developing. Therefore, add the following lines to the `scripts` section:

```json
"build": "rimraf ./dist && tsc",
"start": "npm run build && node dist/index.js",
"start:dev": "npm run build && nodemon --ext \".ts,.js\" --watch \"./src\" --exec \"ts-node ./src/index.ts\""
```

## 3. Configuring tsconfig.json

Next to the `package.json` we will also see a `tsconfig.json` being created. In here, change the following:

```json
"outDir": "./dist"
"rootDir": "./src"
```

Wich will compile our typescript project into the `./dist` folder and read the source code from the `./src` folder.

## Validation

After executing the steps above, you should now be able to create code, save and have auto-reload working. 

![./typescript.png](./typescript.png)

Get started with creating your first Typescript project

Tutorial - Getting started with Typescript


Reinforcement Learning is something I have been covering a while now! Lately, I have seen this space growing tremendously but keep on seeing the same issue arising each time: ***"How can I as a company with an existing infrastructure / applications (= brownfield) adopt Reinfocement Learning?"***

To answer this question, I would love to share a learning I made a while ago after reading Facebook's amazing [ReAgent](https://reagent.ai/) platform. Where they explain an amazing philosophy on how you can easily go from an existing hardcoded rule engine towards full Reinforcement Learning models as illustrated below.

![](./rl-adoption-lifecycle.png)

## Identifying Use Cases

Adopting Reinforcement Learning should be seen as a phased adoption. This way you maintain full control, while at the same time improving your application. Often however, many people already get stuck on discovering the correct candidates for Reinforcement Learning applications.

The rule of thumb is often easily stated as: "When you can model it as a Markov Decision Process, it's suitable for Reinforcement Learning", meaning that when we are in a certain state, we transition to the next state s' by taking action a with a specific probability. Making it such that the next state depends on our current state. The current state and action however, are coditionally independent of all the previous states and actions (= it adheres to the Markov Property).

> ***An application is a good fit for a Reinforcement Learning model when it can be modelled as a Markov Decision Process***

Now, when you are just starting out with Reinforcement Learning, it's not as straightforward to remember this. So lets explain this with an example: 

When living the life, you are either hungy or full. Depending on the state (hungry or full) you thus take an action (eat or don't eat) which results in a transition. We can go from hungry to full by eating, but we can also stay hungry by not eating. No matter where we start, we always know where we will be able to go to.

As a more practical example, let's take the process of balancing a balloon on the tip of our hand. Depending how the balloon is being balanced, we move our hand in any of the X-Y-Z coordinates. Which is a continuous problem that has to be solved.

## Adopting Reinforcement Learning

Since we identified our use cases, it's now time to actually adopt Reinforcement Learning. Let's go over the different phases as illustrated above. 

### Phase 1: Hardcoded Rules

The first thing we do is that we try to hardcode our rules. Remember our Balloon? We now hardcode it stating that `IF` happens, `THEN` we perform.

Observations we should make here:

* We are not training based on data, we coded all our personal knowledge in it
* We are not utilizing any training engine, it was all pre-determined by us
* If we did not cover all the rules, something will happen that we cannot catch.

### Phase 2: Parameter Store

We then go more advanced, actually storing the parameters, such that we can incorporate feedback when it arises. Again back to the balloon: when we take a certain action, how will it have impacted the balloon from being balanced or not? 

E.g. we have a parametered model that based on a state takes a certain action, when we then receive a reward, we adapt this parameter.

Our observations now changed from before:

* We are actually training (we change our parameters) based on feedback that we received
* We are utilizing a training engine that looks at the state
* What we however did not do yet, is make sense of this state! I.e. is it windy or not for our balloon?

### Phase 3: Full RL-Model

For the last observation, we want to solve the solution of "context". This is where a full-blown Reinforcement Learning model comes into play. We now add context to our model, such that we can start training on specific parts of the environment and the nuances that we encountered.

E.g. when a state states that it is windy and the balloon is in a specific position, we take a different action than when it is not windy but in the same state.

Finally our observations look as follows:

* We are training (we change our parameters) based on the feedback 
* We are utilizing a more complex training engine that does not look only at the current state but also at the context it is in

## Conclusion

Looking at the above, we can say that an application can gradually be adopted to incorporate a Reinforcement Learning model. By going through the different phases, we can adopt it in such a way that we understand what is happening, as well as being able to control when something goes wrong.

Learn how to gradually adopt Autonomous Systems and Reinforcement Learning in Production Environments

Gradually adopting Autonomous Systems in Production


I have been following Azure Synapse Analytics since it was still in development and the mock-ups were not finished yet. The vision that Azure Synapse Analytics brings is simply amazing, bringing an End-to-End analytics platform that seamlessly mixes SQL Dedicated, SQL On-Demand, and Spark to prepare, analyze and publish your data needs.

## Introduction

For a new project I am working on, I wanted to analyze the performance of my ML Models, but had some trouble finding the right documentation to read analyze my Azure Data Lake storage accounts. Therefore, I wanted to share this through a new blog article.

> Please note that this article goes on how we can utilize Azure Synapse to analyze this data and does not goes into how we can create the RL trained models 😊

## Prerequisites

### Generated Data from Random Agent and Trained Agent

To get started, I generated [trajectory sequences](https://docs.ray.io/en/master/rllib-offline.html) for my Reinforcement Learning use case. These sequences are JSON files containing lines with `(s, a, r, s', ...)` tuples, represented as [SampleBatch](https://github.com/ray-project/ray/blob/master/rllib/policy/sample_batch.py) objects in [RLLib](https://docs.ray.io/en/master/rllib.html)

> ⚠ I dumped this data into an Azure Data Lake Store! You will need the storage account key, storage account container name, and storage account URL later

I created those for 2 agents:

* Random Agent: Agent that takes random actions
* Trained Agent: Agent trained with PPO

As an example, the data would be represented as follows in several chunked JSON files:

```json
{"type": "SampleBatch", "t": [0], "eps_id": [0], "agent_index": [0], "obs": "BCJNGGhAkAAAAAAAAAB5iwAAAFKABZWFAAEA8hmMEm51bXB5LmNvcmUubnVtZXJpY5SMC19mcm9tYnVmZmVylJOUKJYQLgDxBQAGw0I97xhFvG7DerymTD07lIwFQADxFpSMBWR0eXBllJOUjAJmNJSJiIeUUpQoSwOMATyUTk5OSv////8FAPAFSwB0lGJLAUsEhpSMAUOUdJRSlC4AAAAA", "actions": [1], "action_prob": [1.0], "action_logp": [0.0], "rewards": [1.0], "prev_actions": [0], "prev_rewards": [0], "dones": [false], "infos": [{}], "new_obs": "BCJNGGhAkAAAAAAAAAB5iwAAAFKABZWFAAEA8hmMEm51bXB5LmNvcmUubnVtZXJpY5SMC19mcm9tYnVmZmVylJOUKJYQLgDxBQC9xkE9JbU7PiHReby005a+lIwFQADxFpSMBWR0eXBllJOUjAJmNJSJiIeUUpQoSwOMATyUTk5OSv////8FAPAFSwB0lGJLAUsEhpSMAUOUdJRSlC4AAAAA", "unroll_id": [0]}
{"type": "SampleBatch", "t": [1], "eps_id": [0], "agent_index": [0], "obs": "BCJNGGhAkAAAAAAAAAB5iwAAAFKABZWFAAEA8hmMEm51bXB5LmNvcmUubnVtZXJpY5SMC19mcm9tYnVmZmVylJOUKJYQLgDxBQC9xkE9JbU7PiHReby005a+lIwFQADxFpSMBWR0eXBllJOUjAJmNJSJiIeUUpQoSwOMATyUTk5OSv////8FAPAFSwB0lGJLAUsEhpSMAUOUdJRSlC4AAAAA", "actions": [1], "action_prob": [1.0], "action_logp": [0.0], "rewards": [1.0], "prev_actions": [1], "prev_rewards": [1.0], "dones": [false], "infos": [{}], "new_obs": "BCJNGGhAkAAAAAAAAAB5iwAAAFKABZWFAAEA8hmMEm51bXB5LmNvcmUubnVtZXJpY5SMC19mcm9tYnVmZmVylJOUKJYQLgDxBQD+ylA9pt3BPk8srbyzjxe/lIwFQADxFpSMBWR0eXBllJOUjAJmNJSJiIeUUpQoSwOMATyUTk5OSv////8FAPAFSwB0lGJLAUsEhpSMAUOUdJRSlC4AAAAA", "unroll_id": [1]}
...
```

### Azure Synapse Analytics Workspace

Create an Azure Synapse Analytics workspace as explained on the [Documentation website](https://docs.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace)

### Spark Pool

Once an Azure Synapse Analytics workspace is created, add a Spark Pool to it (see screenshots below).

![](./spark-pool-create-1.png)
![](./spark-pool-create-2.png)
![](./spark-pool-create-3.png)
![](./spark-pool-create-4.png)

## Gathering our Insights

### Creating a notebook

Now we are ready to do the heavy lifting! So open your Synapse Studio and create a new Notebook.

![](./synapse-1.png)
![](./synapse-2.png)

### Executing our code

In this notebook, we will create some cells that will connect to the storage account. To connect to a storage account we can utilize the following snippet:

```python
spark.conf.set("fs.azure.account.key.rwdemoadls.blob.core.windows.net", "<YOUR_STORAGE_ACCOUNT_KEY>")
```

After which we load our datasets into their respective data frames:

```python
df_prod = spark.read.format("json").load("wasbs://telemetry@rwdemoadls.blob.core.windows.net/production/f748fb93-7cc3-4dc9-814d-d1e733c216c8-non-trained/2021/03/05/12/30/*.json")
df_test = spark.read.format("json").load("wasbs://telemetry@rwdemoadls.blob.core.windows.net/production/f748fb93-7cc3-4dc9-814d-d1e733c216c8-non-trained/2021/03/05/12/30/*.json")
```

To make it easier to consume this data, I register it as a table so I can query it:

```python
df_prod.createOrReplaceTempView("rw_prod")
df_test.createOrReplaceTempView("rw_test")
```

This way, I can run the following query to create new dataframes containing my new data

```python
df_prod_eps_reward = spark.sql("SELECT eps_id, SUM(rewards[0]) as reward  FROM rw_prod GROUP BY eps_id ORDER BY eps_id")
df_test_eps_reward = spark.sql("SELECT eps_id, SUM(rewards[0]) as reward  FROM rw_test GROUP BY eps_id ORDER BY eps_id")
```

That I can visualize through matplotlib

```python
import matplotlib.pyplot as plt

plt.style.use('ggplot')

fig, ax = plt.subplots()

df_prod_eps_reward.toPandas().plot(x="eps_id", y="reward", ax=ax, label="Random Agent")
df_test_eps_reward.toPandas().plot(x="eps_id", y="reward", ax=ax, label="Trained Agent")

ax.set_title("Optimization Score")
ax.set_ylabel("Reward")

plt.show()
```

Finally showing

![](./result.png)

> For the full screenshots, see below

### Executing our code - Screenshots

![](./result_screens-1.png)
![](./result_screens-2.png)

## Conclusion & Feedback on Azure Synapse

In this article, I went through how you can easily utilize Azure Synapse Spark Pools to analyze your data. 

Personally, I am very impressed with the value Azure Synapse can bring for analyzing your data. Having an on-demand cluster that runs while you are utilizing it and provides an end-to-end integrated experience is exactly what I needed for most of my data use cases.

Next to that, my general feedback for the Azure Synapse team would be:
* Notebook UI is amazing, crystal clear, and amazing views for task outputs
* Notebook UI could use some tweaking to not always show "Clear Cell" or "Convert to Markdown", it sits in the way sometimes
* Notebook UI could benefit from "snippets" to connect to storage accounts (I did remember seeing this in mockups, so I am not sure if I missed it though)
* The initial getting started with it is quite difficult, coming from Databricks, I had to figure out which spark code to utilize and it was quite hard to find documentation for my use case (though I might have used wrong keywords)
* I had to use Spark Pools since JSON was not supported in SQL On-Demand which I found sad

Finally, I recommend anyone that is working with data to give Azure Synapse a try! You might get surprised by how powerful it is!

Discover how you can utilize Apache Spark Pools in Azure Synapse to analyze ML Model performance

Analyzing ML/RL Model Performance with Azure Synapse Analytics Spark Pools


MLOps is the continuation of DevOps, extended for Machine Learning. It allows data scientists, data engineers, application developers, and the operations team to collaborate, reducing the time from model creation towards first production deployment.

![](./mlops-lifecycle.png)

> ***"Putting Machine Learning models in production is easy when you have just a few, however when you work at scale and have to manage thousands of them, such technologies do not exist yet."*** - Xavier Geerinck

It was seen by Gartner in the phase of "Peak of Inflated Expectations" in its 2020 Hype Cycle for Data Science and Machine Learning report, with the plateau being reached in 2 to 5 years. Illustrating the expectations that are being set. MLOps is just as DevOps, not a technology, but rather a methodology that should be followed to increase efficiency and stability.

![](./gartner-hype-cycle-2020.png)

Now you may wonder: "Why would I adopt MLOps in my organization?". To answer that question, we should start by looking at the typical lifecycle in more detail.

## MLOps Lifecycle

![](./mlops-lifecycle-full.png)

In the illustration above, we can see that a typical lifecycle starts with a data scientist experimenting and eventually creating a machine learning model. This however is not the bulk of the work, typically taking 1 week to create such a model.

> 1 week is an underestimation, not including the time for data gathering, extensive data processing, ...

Once a model is created, collaboration is needed to promote it to production. This is what leads to the biggest chunk of the work, as different parties have to be involved to do so. 

It starts by working together with an application developer, whose responsibility is typically to define parameters, define startup and finally package the model ready for deployment. A model is typically integrated through a REST API interface, such that anyone requesting the model can easily interface with it, send the request and receive a response (often a prediction or classification) that can be consumed by the demanding application.

Afterward, the model is provided to the operations team. They will take the packaged model and put it in production. Creating a production deployment is then similar to a typical deployment. Taking into account aspects such as scaling, tuning, caching, versioning, ...

While the last block is similar to MLOps, where we create a pipeline that allows us to deploy models to production and manage them. The biggest nuance is in the type of system we are deploying here. In Machine Learning, we often work with "closed-loop" systems (ref. Closed-Loop Control Systems). Where feedback has to be gathered and taken into account for a future iteration.

## MLOps Value

But where does the value of MLOps lie? Just as in DevOps, MLOps focuses on streamlining the production process, allowing for a continuous cycle. When this cycle is implemented well, MLOps will allow your organization to gain several benefits (including but not limited to):

1. Reduction in model publishing cost
2. Improved monitoring
3. Correct feedback loop implementation
4. Cross-team collaboration

## MLOps at Scale

MLOps is still in its infancy stage, with Microsoft stating: "Putting Machine Learning models in production is easy when you have just a few, however when you have thousands of them, it becomes a hurdle"

> ***"Putting Machine Learning models in production is easy when you have just a few, however when you work at scale and have to manage thousands of them, such technologies do not exist yet."*** - Xavier Geerinck

Because of this, companies such as [Microsoft](https://azure.microsoft.com/en-us/services/machine-learning/#features), [Uber](https://eng.uber.com/michelangelo-machine-learning-platform/), [Airbnb](https://databricks.com/session/zipline-airbnbs-machine-learning-data-management-platform), and others build these platforms themselves. Often releasing this knowledge to the public through their cloud platforms, technology sessions, or open-source code available on GitHub.

## Conclusion

If your company is utilizing Machine Learning today and you have multiple models in production but are not yet utilizing the MLOps methodology, it might be worth look into this! The benefits MLOps brings in the long-term will outweigh the disadvantages.

Machine Learning Operations - MLOps explained


A couple of years ago, I decided to buy a Raspberry PI cluster with [Clusterhat](https://clusterhat.com/). In all that time however, I never too the time to configure it correctly to make it able to run Kubernetes. So now I decided to invest some time and get some learnings back from it!

> Note: Right now the ecosystem is utilizing ARMv7 mainly, while as the Pi Zeros are still running ARMv6. It will thus be an interesting task to get Kubernetes running with the Pi Zeros as nodes. Do note that supporting legacy hardware is an important skill that a lot of companies (especially Manufacturing ones) value a lot! Seeing that the devices they have running are often even older! 

So let's get started on how you can set this up! 

// @todo: insert picture clusterhat and pi zero
![]()

## Flashing the Images

We download the Images from https://clusterctrl.com/setup-software utilizing the CNAT version.

> The CNAT version has NAT configured (Network Address Translation) which will automatically share the Ethernet or Wifi on the controller with the Pi Zeros.

Download Links (for backup purposes):

* CNAT Lite Controller: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-CNAT.zip
* P1: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p1.zip
* P2: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p2.zip
* P3: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p3.zip
* P4: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p4.zip

Once the images are downloaded, you can utilize a flashing tool to flash them to the SD Cards. For this I utilized the excellent tool called [Etcher](https://www.balena.io/etcher/) and flashed the different images for the controller and PI Zeros. 

![](./etcher.png)

After each image was flashed, I added a `SSH` file (without any content) to the SD Cards root.

> By adding the `SSH` file, we are able to SSH to the different Nodes.

![](./create-ssh-file.png)

Now the hardest part is done, and we can connect the Ethernet cable and boot up the Controller!

> Note: Make sure to utilize a Power Adapter instead of a USB Cable connected to the PC.

> Note 2: I connected the Zeros from Power Adapter back labeled 1, 2, 3 and 4

## Connecting to SSH

After waiting a couple of minutes, we should be able to access the Raspberry PI Controller. To make things easier and remove the need to connect a screen + keyboard, I used the tool [Advanced IP Scanner](https://www.advanced-ip-scanner.com/) find out the IP Address of the controller by quickly scanning the network.

![](./ip-scanner.png)

Through this way, I discovered that mine was running under `192.168.1.56`. So go ahead and log in on it then through `ssh pi@<your_ip>` with username `pi` and password `clusterctrl`

* Username: `pi`
* Password: `clusterctrl`

> Note: if you want to change your password, utilize `sudo raspi-config` and go to `Change User Password`

![](./clusterhat-ssh.png)

## Turning on the PI Zeros

The ClusterHat module acts as a USB Hub, which is turned off by default. To help us, a helper program was included that allows us to turn them on. 

* Turn Raspberry PI Zeros On:  `sudo clusterhat on` 
* Turn Raspberry PI Zeros Off: `sudo clusterhat off`

By default, ClusterHat exposes its USB Hub interface over gateway `172.19.181.1` to connect to the PI Zeros we can then connect over IPs `172.19.181.1 - 172.19.181.4`. To make our lives a bit easier, we set up a config files for the SSH Names in `~/.ssh/config`:

```bash
Host p1
  HostName 172.19.181.1
  User pi

Host p2
  HostName 172.19.181.2
  User pi

Host p3
  HostName 172.19.181.3
  User pi

Host p4
  HostName 172.19.181.4
  User pi
```

By then adding a SSH Public Key to them we are able to more quickly login. So first create a SSH Key with:

```bash
ssh-keygen -t rsa -b 4093
```

And copy it over to the PI Zeros with:

```bash
cat ~/.ssh/id_rsa.pub | ssh pi@p1 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh pi@p2 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh pi@p3 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh pi@p4 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
```

We can now connect to our PI Zeros without having to enter a password anymore! 😀

![](./clusterhat-ssh-p1.png)

## Syncing the time

We are almost there, just before our last step we need to install the time syncing tool. To do this, run the following commands on your controller:

```bash
ssh pi@p1 "sudo apt-get install -y ntpdate" &
ssh pi@p2 "sudo apt-get install -y ntpdate" &
ssh pi@p3 "sudo apt-get install -y ntpdate" &
ssh pi@p4 "sudo apt-get install -y ntpdate" &
```

## Finalization

As a final step - yes we are there! - we need to adapt our `/etc/rc.local` script such that we automatically spin up the cluster on boot. To do that, add the following lines to your `/etc/rc.local` file:

```bash
# Start the PI Zeros
/sbin/clusterhat init
/sbin/clusterhat on

# Fix networking towards the PI Zeros
/sbin/iptables -I forward -j ACCEPT
```

## Conclusion

From the steps above we can conclude that it is SUPER easy to set up our own cluster! In just 1 hour I have 4 Raspberry PI Zeros running on my Clusterhat on the network. 

The sad part of this story however is that while this is an amazing architecture, it is based on older hardware (running ARMv6). Due to this, it is practically impossible to run demonstrations on this. Which is why I decided to abandon this cluster and move on. Currently I am looking into alternatives for a nice demo cluster, so let me know if you know something!

References
* https://medium.com/@dhuck/the-missing-clusterhat-tutorial-45ad2241d738
* https://github.com/AKYD/clusterhat-kubernetes