
The Bonsai Machine Teaching platform has been released! Promising an easy to use environment for end-to-end Reinforcement Learning projects, starting with simulator selection / integration to algorithm configuration and training.

Of course I had to instantly try this out! Seeing that I "stalked" this team endlessly for over a year 😉

Let's try using the platform for our simple ["CartPole" example as covered before.](/post/2018/6/10/ai-ml-rl-openai-cartpole).

## Understanding Brains and Inkling

### Introduction

Bonsai utilizes the concept called "Brains". This concept is quite similar to my previously detailed [digital twin](/post/2020/4/15/coding-iot-digital-twin) and [reinforcement learning combination](/post/2020/4/15/coding-ai-rl-iot-twin-rl) posts, where they create an Inkling program that goes over the State, Action and Reward definitions.

Next to these definitions, there is also **meta information** added such as the configuration of the environment, which simulation environment to utilize, when the terminal state is reached, ...

At the end of this file, a block is added that summarizes all of this together, called a **concept** with **curriculum**.

### Creating a Brain for CartPole

When we dive into the example for the "Stellar Cartpole" problem, we can find a beautiful example worked out for us that can train the cartpole environment.

To go a bit more in detail, let's break this up in the respective State, Action and Reward definition as well as Concept creation.

**State (state received from simulator):**

```python
# Full Simulator State
type SimState {
    cart_position: number,
    cart_velocity: number,
    pole_angle: number,
    pole_angular_velocity: number
}
```

**Action (actions to take):**

```python
type SimAction {
    command: number<-1 .. 1> # Left = -1, None = 0, Right = 1
}
```

**Reward:**

In Cartpole we want it to stay stable. Stable can be defined in multiple ways, but to give 2 examples:
* Getting the Cartpole as stable as possible (e.g. go for the longest number of iterations, taking the rotation angle into account as a negative reward)
* Making the Cartpole survive (e.g. go for the longest number of iterations - shown below)

```python
function BalancePoleReward(State: SimState) {
    # Reward it for not falling.
    return 1.0
}
```

**Concept / Curriculum:**

```python
# Define a concept graph with a single concept
graph (input: SimState): SimAction {
    concept BalancePole(input): SimAction {
        curriculum {
            # The source of training for this concept is a simulator
            # that takes an action as an input and outputs a state.
            source simulator (Action: SimAction): SimState {
                # Automatically launch the simulator with this
                # registered package name.
                package "Cartpole"
            }

            # The objective of training is expressed as a goal with two
            # subgoals: don't let the pole fall over, and don't move
            # the cart off the track.
            goal (State: SimState) {
                avoid `Fall Over`:
                    Math.Abs(State.pole_angle) in Goal.RangeAbove(MaxPoleAngle)
                avoid `Out Of Range`:
                    Math.Abs(State.cart_position) in Goal.RangeAbove(TrackLength / 2)
            }

            training {
                # Limit the number of iterations per episode to 120. The default
                # is 1000, which makes it much tougher to succeed.
                EpisodeIterationLimit: 120
            }
        }
    }
}
```

All of this is thrown into an easy to use interface:

![./learn-inkling-overview.png](./learn-inkling-overview.png)

## Training our Brain

Once the Brain has defined, it's as easy as clicking the "Start Training" button as displayed above. This will kick of the learning platform that will automatically spin up the underlying training environment cluster and simulator to start maximizing our reward. There is no need to know anything about Reinforcement Learning algorithms to actually train the model! 😉

> What I love here is how clear they made it that the simulator is our "World" and that the agent is taking actions and receiving observations.

![./training-start-waiting.png](./training-start-waiting.png)

After a couple of minutes of training we are starting to see the reward increase beautifully, reaching the same reward as our [previous Cartpole Example description which utilizes Q-Learning algorithm](/post/2018/6/10/ai-ml-rl-openai-cartpole) after ~1hr.

![./training-done.png](./training-done.png)

> A thing I noticed is that training seems to go slower compared to testing it on myself. This is due to the usage of Deep Reinforcement Learning! The state action discritization is using a Neural Net rather than a fixed state array, thus on these small environments taking a longer time. When we would run this on a large-world example with millions of states, we would mainly look at a Neural Network.

## Testing the Brain

Once the Brain is trained, we can start an assessment that will launch the CartPole environment and visualize it, showing how well our brain is performing. 

> What I love here is that we are able to see different charts, illustrating what is happening in the observation space. 

![./training-done.png](./training-done-2.png)

## Conclusion

All of the above might look easy, but this is because the Bonsai platform makes it easy! It's no easy feat towards such a beautiful simulation integration as well as automated (and abstracted away) training. 

Therefor, a big well done to the entire Bonsai team for making this such an amazing platform, I can't wait to see how it will evolve over the coming years.

Reinforcement Learning with the Bonsai Platform


After explaining the end-to-end concept of [creating a Digital Twin Reinforcement Learning environment](/post/2020/4/15/coding-ai-rl-iot-twin-rl) I wanted to go into a deeper explanation of how the first part of this can be done.

This is something I have been working on for the last couple of months and decided to call ["Roadwork RL"](https://github.com/Xaviergeerinck/Roadwork-RL). It has also been open-sourced for others to try, so I would love to hear your feedback around this! 

## Requirements

As in any project, there are some requirements, and a lot of reshaping over the months to make this as easy as possible, satisfying the requirements I had put upfront. 

The requirements I decided to use for this project are the following:

* Allow people new to the Reinforcement Learning ecosystem to **get started within 5 minutes** (after satisfying prerequisites)
* Allow **cross-language development**
* Allow **easy cloud deployment** through Kubernetes clusters

> **Note:** you can find the open-sourced code here [https://github.com/Xaviergeerinck/Roadwork-RL](https://github.com/Xaviergeerinck/Roadwork-RL)

## Solution

Once requirements have been created, I typically create an architecture to include the different design decisions that have to made. For these requirements, cross-language compatibility is the most difficult one.

As detailed before, Reinforcement Learning consists out of Simulators that are being transformed through actions taken by an agent. Seeing that Simulators have their own unique integration piece often done by Subject Matter Experts and Reinforcement Learning engines have their own unique Algorithms done by Data Scientists, it's interesting to utilize a split between simulator (server) and algorithm (client) through a high-throughput protocol.

For this architecture I decided to utilize the gRPC protocol. This does add some technical complexity compared to HTTP, but is well worth the trade-off due to the gained speed in this project.

After keeping all of the above in mind, I came up with the following architecture detailing the example of the OpenAI Gym and Minecraft RL environment, that are being integrated through an "abstraction" layer specific for that simulator. The SDKs on the language side are then able to communicate with this abstraction layer and execute actions / receive rewards from the simulation environment.

Putting all of this into a diagram results in the following:

![./rl-platform.png](./rl-platform.png)

## Demonstration

Executing all of the above on the created platform is simple and can be done through a few commands:

```bash
# Start OpenAI Gym Server
# Note: this will start the server and keep on running
#       this is a Kubernetes "deployment"
sudo ./Scripts/build.sh Servers/openai/ roadwork/rw-server-openai
./Scripts/start.sh rw-server-openai

# Start Python Experiment - CartPole
# Note: this will execute the experiment and stop afterwards
#       this is a Kubernetes "pod"
sudo ./Scripts/build.sh Clients/python/experiments/cartpole roadwork/rw-client-python-cartpole
./Scripts/start-client.sh rw-client-python-cartpole
```

The above will create a **deployment** in the Kubernetes cluster **starting our OpenAI Gym Server**. This server will listen on gRPC messages coming in from our Dapr environment. After this, a client is spun up as a **pod** that will execute a Reinforcement Learning algorithm that solves the Cartpole environment.

After a few minutes of training, our algorithm is done and we an see the following as output:

![./experiment-run.png](./experiment-run.png)

Together with this, video is generated through the OpenAI Server environment. This video is saved on the server cluster and can be fetched through `./Scripts/get-output-server.sh rw-server-openai /mnt/f/test`. Which will show us this:

<iframe width="420" height="315" src="https://www.youtube.com/embed/k3O2Rn48aOI"></iframe>

## Future Work

The created implementation framework is far from complete, a lot of integrations still have to be written for different simulation environments (e.g. [ML Agents](https://github.com/Unity-Technologies/ml-agents), [Minecraft](https://marlo.readthedocs.io/en/latest/), ...) while keeping the original code of these environments running (showing the easy integration potential).

Next to that, I would also like this framework to act as a kind of "HPC" cluster. Where experiments can be submitted into a queue and that can be executed when capacity is available (seeing the long running nature of these experiments). 

I personally believe that most of the technical foundations are available right now, such that future improvements can be made quite easily and experiments can be ran in the state it is today.

## Conclusion

Roadwork RL is a framework specifically created to solve the hard and high entry barrier for Reinforcement Learning. I hope that by creating this framework, more people get interested in this amazing field.

Roadwork RL is definitely a work in progress and has room for growth. Please do share your feedback below, or reach out to me through Social Media for more information / contributing.

Roadwork RL - A Multi-Language Reinforcement Learning Environment


Looking at the history in manufacturing, we saw that many manufacturing companies were deploying statistical models on-edge (e.g. based on parameters such as temperature, pressure and speed, do we send an alert or not), having the main impact on their production volume. If they can then squeeze out this extra bit of improvement, they would thus be able to optimize this process and have a growth in production volume (and net earnings accompanied with it).

While working with large manufacturing enterprises, the trend of seeing them move to the Industrial Internet of Things (IIoT) is quite visible. This can largely be contributed to the fact that Artificial Intelligence models are slowly starting to replace the previously created statistical model due to their benefits such as understanding as well as easiness to train. We just have to feed it data, and it will figure trends out by itself (once we define an optimization parameter). - Very roughly sketched.

Next to the training of Artificial Intelligence models for the Internet of Things, we can also see the creation of a new concept called **Digital Twins**. But what is this? What will it allow us to do? Let's dig a bit deeper into this concept.

> **Note:** I will be going deeper on the topic of Artificial Intelligence and Digital Twins in a later post

## Digital Twin Definition

We can summarize the concept of a Digital Twin in one sentence:

```
A Digital Twin is a virtual representation of a physical object
```

The above however might seem a bit "vague" when looking at it for the first time, it also does not clearly indicates the different benefits this might have. So as a lengthier description, a Digital Twin is the concept or **hierarchical representation** of the different sensors within an object or environment (such as a Wind Turbine, Building, Car, ...) that get recreated in a virtual environment so that we are able to illustrate how the different sensors impact each other. 

> Think of Digital Twin as a "State Representation" of our physical object.

### Simple Example

To illustrate the above with a simple example: Take for example a car and its motor, let's describe this in a simple way and in an advanced way showing the sensor changes.
  * **Simple:** When we push the gas pedal, it will accelerate the car.
  * **Advanced:** When we push the gas pedal, the throttle valve will be opened more, allowing more air to enter the engine. The Engine Control Unit (ECU) will notice this change of the throttle valve and will then increase the fuel rate. 

### Notion of Hierarchy

What we can see in the example above is the hierarchy between components. We have a gas pedal connected to a throttle valve which in its turn will be picked up by the ECU. There is thus a notion of "hierarchy" that we should incorporate in our Digital Twin. It should show us the interconnection state of what happens between the components when one of them triggers something.

E.g. when in our example above the ECU detects a change of the throttle valve, which other components are being affected by it? 

### Use Cases

Digital Twins of course have their specific use cases, when applying them correctly we can see them coming back in the following:

* Simulation Creation
* Real-time training
* Component dependency
* ...

But to make all of this a bit clearer, let's go through an End-to-End example that illustrates a Digital Twin being created and represented visually.

## Digital Twin - Windmill Example

For our End-to-End example, I will take the description of a Windmill. We will analyze this physically in terms of Sensor composition as well as how this would translate in the Digital Twin concept. 

### Physical Description

When looking at a windmill, we can decompose this in different physical parts:

* Gearbox
* Rotor
* Blade
* Tower

Each of these parts has different sensors connected to it that monitor the working of our Wind Turbine. [The picture below](https://www.te.com/content/dam/te-com/documents/sensors/global/TE_SensorSolutions_Wind-Turbines.pdf) illustrates the different components and their connected sensors.

![./wind-turbine.png](./wind-turbine.png)

### Virtual Description - Building our Twin

Since we now understand our physical object, let's transform this into a virtual object. Showing the relation between the different components, as well as **enriching** the Digital Twin with extra data that might be beneficial for our end purposes.

> **Note:** In my case, I added a Weather API as enriching data, as well as slimmed down the amount of sensors for illustration purposes.

When we map this graphically, we can find something like this:

![./wind-turbine-graph.svg](./wind-turbine-graph.svg)

Now of course for any system to understand this, we should map this to a kind of definition language. Doing this for our picture, we can achieve something that looks like the below.

```
// API
api weather {
  int windSpeed [0, 90] # in mph
  enum windOrientation { N, NE, E, SE, S, SW, W, NW }
}

// Gearbox
sensor gearboxOilLevel {
  bool isOk
}

sensor gearboxAccelerometer {
  float xAccel [-40.0, 40.0]
  float yAccel [-40.0, 40.0]
  float zAccel [-40.0, 40.0]
}

twinItem gearbox {
  gearboxOilLevel g1
  gearboxAccelerometer g2
}

// Stator
sensor statorBearingsTemperature {
  float [0, 1500] // Kelvin
}

sensor statorWindingsTemperature {
  float [0, 1500] // Kelvin
}

twinItem stator {
  statorBearingsTemperature s1
  statorWindingsTemperature s2
}

// Blade (= Rotor)
sensor bladeRotation {
  float rotation [0.0, 360.0]
}

sensor bladePressure {
  float bar [0, 1000]
}

sensor bladeSpeed {
  float RPM [0, 60]
}

twinItem blade {
  bladeRotation b1
  bladePressure b2
  bladeSpeed b3
}

// Composition
twinItem windTurbineSensors {
  gearbox
  stator
  blade
}

twin windTurbine {
  windTurbineSensors wt
  weather apiWeather
}
```

## Conclusion

I hope that by reading the article above I was able to explain the concept of Digital Twins in a bit more detail.

Please feel free to reach-out to me in case of questions.

Internet Of Things - Digital Twins Explained


One of the ideas I have been playing around with the last couple of months is **the combination of Digital Twins and Reinforcement Learning**. This is an experimental idea where I would love to hear your opinions about it (feel free to comment below, send me an email or reach out to me on Social Media such as Twitter / LinkedIn), and that will be refined over the coming months.

But before I can explain more in depth on what this means let's quickly go through the concepts of Digital Twins and Reinforcement Learning again:

* **Reinforcement Learning:** "Reinforcement Learning is a Machine Learning paradigm that automatically teaches agents to take actions on an environment, such that the *future reward* is maximized". (More: [Xavier Geerinck - RL Intro](/post/2018/5/1/ai-ml-rl-rl-intro))
* **Digital Twins:** "A Digital Twin is a virtual representation of a physical object" (More: [Xavier Geerinck - Digital Twin](/post/2020/4/15/coding-iot-digital-twin))

After this small refresher, let's dive into the concept!

## Concept

### Reinforcement Learning Challenges

Reinforcement Learning - just as any new concept - has its own unique set of challenges that come along with it. Some of these challenges make it extremly hard to implement these algorithms, which is what I would like to tackle with this new concept. To name some of the issues in the Reinforcement Learning ecosystem today:

* Integration with existing simulators is hard and requires the most time
* Deployment of the model in an edge environment
* Shaping of the reward (Reward Engineering)
* Hierarchical State Overview
* Subject matter expertise is required
* Hiring of Reinforcement Learning specialists

### Hypothesis Creation - Digital Twin and RL

Now when we look at the concept of Digital Twins, we can actually relate a Digital Twin as the "World" or "Environment" in Reinforcement Learning, which describes how the different sensors are coupled together. From this we can define an **hypothesis** that will help us understand the concept: **"If we shape the environment in our simulation and physical environment in the same way (as a Digital Twin), would this provide benefits such as improved integration speed and separated concerns?"**

I personally believe that the answer to the hypothesis above is "Yes we can", due to the following reasons:

**1. Reward shaping can be integrated into the "Digital Twin" definition language**

Looking at the example of our Windmill in the [previous post](/post/2020/4/15/coding-iot-digital-twin), an extension can be added to this that allows us to describe the reward which is required in Reinforcement Learning. For Example, when we want to train our algorithm to maximize the blade rotation and constrain the oilLevel to make sure it stays ok, we could imagine the following being created:

```
twin windTurbine {
  windTurbineSensors wt
  weather apiWeather
  reward {
    (wt.gearbox.g1.isOk ? 0 : -100)
    + (wt.blade.b3.RPM)
  }
}
```

This is actually very interesting for us! Seeing that **no prior knowledge of the environment physics is required to tune these parameters** and which is something that can be done without having a technical background (A Subject Matter Expert is thus able to write this as well). In our example, the algorithm will have to find out which action to take such that the blade rotation is maximized (e.g. do we have to engage our Yaw Motor to turn the blade around the tower? If yes, how much should we move?).

**An extra advantage is that we can make this model more complex by adding state information previously unknown.** This way we can for example add weather information, which could have a benefit for our algorithm.

**2. Subject Matter expertise is split off**

As discussed partly in point 1, since the actual integration between the **ingestion system** (events of physical and simulation environment) and **processing system** (reinforcement learning engine) is being split off, we allow a new role to purely focus on the Digital Twin definition. This is something a Subject Matter Expert should definitely be able to do, utilizing his knowledge of the existing system to focus on the reward engineering (which was previously known as a hard thing to do). A technology company can then focus on the processing system creation or ingestion system.

**3. Extra: Cross Language compatibility potential**

Since a layer is created that will focus on the translation of our events to the Digital Twin concept, **we will have the possibility to ease the integration between the processing system and integration layer** such that it becomes more accessible to other developers out there (currently mainly being constrained to Python developers).

Therefor I suggest utilizing a high-throughput technology such as gRPC to become language independent.

### Architecture

Translating all of the above into an architecture, an architecture should be created that acts as an intermediate layer between ingestion system (where events from physical and virtual environment are coming in) and the Reinforcement Learning system. This intermediate layer should have a metastore where the state representation can be created in an unique language, such that a Subject Matter Expert is able to create a state mapping without having knowledge of the underlying technical system.

Taking a shot on creating such an architecture, the following was found:

![./digital-twin-architecture](./digital-twin-architecture.png)

### Example

To illustrate the advantages of such a system, let's take the example of our [Windmill example](/post/2020/4/15/coding-iot-digital-twin) again. When business now comes to us with the following questions:

* *"When should we perform maintenance on the WindTurbine, such that downtime is minimal and thus economical impact is the least"*
* *"How and when should we turn the Turbine to catch the most wind?"*

We are now able to answer these through a self-learning algorithm that can steer itself without manual intervention. Achieving the real promise of AI in an understandable and easy to integrate way. Leveraging in-house knowledge from Subject Matter experts, without having to onboard new roles (e.g. data scientists).

## Conclusion

In this blog post I explained the benefits of combining the concepts of Digital Twins and Reinforcement Learning to solve core issues in the ecosystem. To quickly summarize this concept, the different layers are being created with their respective personas:

* 1 - Ingestion System (events of physical and simulation environment)
  * **Responsible:** Tech Company
  * **Role:** IoT Engineer
* 2 - Digital Twin Definition System
  * **Responsible:** Your Company
  * **Role:** Subject Matter Expert
* 3 - Processing System
  * **Responsible:** Tech Company
  * **Role:** Data Scientist

Currently this is an idea in development which I wanted to share for further refinement. Already in development as well is a technical proof-of-concept system that will be tackled in a future blog post. Looking forward to hear your comments on this idea!

A Multi-Language Reinforcement Learning Digital Twin Environment


In an earlier post, I went in more detail how you can utilize Dapr to create a Microservice that communicates over gRPC for quick Inter-Process Communication (IPC).

Now I would like to take this a step further and create a real microservice that allows us to perform account management.

## Pre-Requisites

* Dapr installed locally (check with `dapr --version`)
	* Note: you can update with powershell -Command "iwr -useb https://raw.githubusercontent.com/dapr/cli/master/install/install.ps1 | iex"
* Node.js
* CosmosDB
* [A Typescript Project](/post/2021/3/24/coding-javascript-typescript-getting-started) as explained in my earlier post [here](/post/2021/3/24/coding-javascript-typescript-getting-started)

> **Note:** You can update your Dapr version through one of the following
>  * **Linux:** `wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash`
>  * **PowerShell:** `powershell -Command "iwr -useb https://raw.githubusercontent.com/dapr/cli/master/install/install.ps1 | iex"`

## Project Scope / Requirements

**Scope:** Allow us to create users and let them login, returning a JWT token.

**Routes:**
* `/user/login` should return a JWT token
* `/user/register` should create an account

## Writing a Hello World route in Express and Typescript

Since Dapr takes care of state management, we are going to write a Microservice that automatically hooks up to this state management (in this case creating and finding our users). But first we have to install our dependencies, so let's install those:

```bash
# Dependencies Installation
npm i -S express body-parser isomorphic-fetch express-validator bcrypt uuid
npm i @types/express @types/uuid @types/bcrypt @types/isomorphic-fetch @types/express-validator --save-dev
```

Whereafter we can add the following code which spins up the server and initializes the route `/` that will respond with `Hello World`.

```javascript
import express = require('express');

const PORT = process.env.APP_HTTP_PORT || 9000;
const PORT_DAPR = process.env.DAPR_HTTP_PORT || 9500;
const app: express.Application = express();

app.get('/', (req, res) => res.send('Hello World'));
app.listen(PORT, () => console.log(`Server started at 0.0.0.0:${PORT}, Dapr at ${PORT_DAPR}`));
```

Starting this with Dapr on the app port `9000` can be done with:

```bash
dapr run --app-id node-api-user --app-port 9000 --port 9500 npm run start
```

> **Note:** Port `9500` will be used for calling Dapr (e.g. state management)

## Dapr State Management

### Setting up connection to CosmosDB

For our State Management backend we will utilize `CosmosDB`. To set this up we can refer to the [documentation](https://github.com/dapr/docs/blob/master/howto/setup-state-store/setup-azure-cosmosdb.md) which explains that we should adapt our local created `components/statestore.yaml` file. When doing this, we end up with this:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: state-cosmosdb
spec:
  type: state.azure.cosmosdb
  metadata:
  - name: url
    value: <YOUR_URL>
  - name: masterKey
    value: <YOUR_PRIMARY_KEY>
  - name: database
    value: <YOUR_DB_NAME>
  - name: collection
    value: <YOUR_COLLECTION_NAME>
```

> **Note:** For use in production you can create a `.yaml` file (e.g. `cosmos.yaml`) and deploy it with `kubectl apply -f cosmos.yaml`

After doing this, we are now able to access our state through the DAPR port which we selected during start-up. We can access this through the environment variable `DAPR_HTTP_PORT`. 

### Example State Call

We can now store and receive states through http requests. To do this I create a `sample.http` file which includes the lines I want to send. When then installing the **VSCode Extension: REST Client**, I am able to utilize **CTRL + ALT + R** or **CTRL + SHIFT + P; > Rest Client: Send Request** to send a request.

![./rest-client.png](./rest-client.png)

My `sample.http` file looks like this:

```
POST http://localhost:9500/v1.0/state/state-cosmosdb

[
  {
    "id": "e86f44ee-ed09-4a71-afd7-3d6bbf782b39",
    "name": "Xavier Geerinck"
  }
]

###

GET http://localhost:9500/v1.0/state/state-cosmosdb/e86f44ee-ed09-4a71-afd7-3d6bbf782b39
```

Whereafter when Running the 2 queries we get the following results (CTRL + ALT + R):

![./dapr-state-post.png](./dapr-state-post.png)

![./dapr-state-get.png](./dapr-state-get.png)


## Writing our API Routes with Express

We can now write our API routes with the concepts learned above. When we implement this, we get the following end-result:

```javascript
import express = require('express');
import { v4 as uuidv4 } from 'uuid';
const bodyParser = require('body-parser');
const { check, validationResult } = require('express-validator');
const bcrypt = require('bcrypt');
const fetch = require('isomorphic-fetch');

const PORT = process.env.APP_HTTP_PORT || 9000;
const PORT_DAPR = process.env.DAPR_HTTP_PORT || 9500;
const DAPR_URL = `http://localhost:${PORT_DAPR}/v1.0`;
const DAPR_URL_STATE = `${DAPR_URL}/state/state-cosmosdb`;

const app: express.Application = express();

app.use(bodyParser.json());

app.listen(PORT, () => console.log(`Server started at 0.0.0.0:${PORT}, Dapr at ${PORT_DAPR}`));

app.post('/user/register', [
  check('firstName').isAlpha(),
  check('lastName').isAlpha(),
  check('email').isEmail(),
  check('password').isLength({ min: 6 })
], async (req: express.Request, res: express.Response) => {
  const errors = validationResult(req);

  if (!errors.isEmpty()) {
    return res.status(422).json({ errors: errors.array() });
  }

  try {
    // First encrypt password
    const bcryptSalt = await bcrypt.genSalt(10);
    req.body.password = await bcrypt.hash(req.body.password, bcryptSalt);

    // Save User
    const userId = uuidv4();
    console.log(`Creating user with Id: ${userId}`);
    const fetchRes = await fetch(`${DAPR_URL_STATE}`, {
      method: 'POST',
      body: JSON.stringify([{
        key: req.body.email,
        value: {
          id: userId,
          ...req.body
        }
      }]),
      headers: {
        "Content-Type": "application/json"
      }
    });

    // Handle Response
    if (!fetchRes.ok) {
      throw "Failed to register user";
    }

    return res.status(200).send({ success: true, user: {
      id: userId,
      ...req.body
    }});
  } catch (e) {
    return res.status(500).send({ message: e });
  }
});

app.post('/user/login', [
  check('email').isEmail(),
  check('password').isLength({ min: 6 })
], async (req: express.Request, res: express.Response) => {
  const errors = validationResult(req);

  if (!errors.isEmpty()) {
    return res.status(422).json({ errors: errors.array() });
  }
  
  try {
    // Find existing user
    const fetchRes = await fetch(`${DAPR_URL_STATE}/${req.body.email}`);

    // Handle Response
    if (!fetchRes.ok) {
      throw "Failed to find user";
    }

    const fetchResUser = await fetchRes.json();

    // Check password
    const bcryptResult = await bcrypt.compare(req.body.password, fetchResUser.password);
    
    if (!bcryptResult) {
      throw "Incorrect password";
    }

    return res.status(200).send(fetchResUser);
  } catch (e) {
    return res.status(500).send({ message: e });
  }
});
```


![./result.png](./result.png)

Dapr - Creating a User Login/Register Microservice


Due to the current COVID crisis I had some time to start revamping my blog, focusing on achieving features that were not possible before. All while still keeping the costs to a minimum (or even free). Here I would like to explain how I achieved just this, with the focus on adding the following features:

* Newsletter subscription serverless
* Easy previewing of post writing in-browser
* Easier custom theme control
* SEO

> **Note:** This post will not go into the details on custom code written, but will rather go into the specific plugins utilized, major problems encountered as well as how certain migration tracks were tackled.

## Current Blog Architecture

The current blog system is utilizing [Jekyll that is compiling the written markdown pages on each deployment step](/deploying-gh-pages-with-azure-pipelines). Now even though this system has served me well, there is - as in any project - room for improvement. But important is that the new system includes all the features I had before:

* Markdown Posts
* Markdown Metadata (also called frontmatter)
* Code Syntax highlighting
* Commenting
* Custom pages (e.g. Custom project pages)

So let's get started!

## Used Technology

Over the past few years I have come to love certain technologies that made it swift for me to create scripts, servers and even blog technologies while keeping the new languages I have to learn to a minimum. The following tools are the ones I will utilize for the new Blog system:

* [React.js](https://reactjs.org/)
	* It is just a heavenly good frontend framework
	* Disadvantage is server rendering / frontend rendering, but there are tools to help us with this.
* [Zeit.co](https://zeit.co)
	* Free for small sites
	* Production deployment to zeit.co is one simple command `now --prod`
* [Gatsby](https://www.gatsbyjs.org/)
    * Easy to use plugins
    * File drivers for "querying" files as a data source
    * Image optimization plugins
* [TailwindCSS](https://tailwindcss.com/)
    * Easy to use CSS Framework
    * Allows us to utilize custom CSS through classNames rather than custom css files.

> Besides the 2 above, I had to make the decision between [Gatsby](https://www.gatsbyjs.org/) and [Next.js](https://nextjs.org/). While both suited my needs, it was easier to work with Gatsby due to the included file drivers required to query my markdown files. 

## Planned Structure Changes

Currently the blog utilizes the following structure:
* Pages are under the root `/` domain
* Projects are under `/projects/:slug`
* Images are all in the `/assets` folder 
* Dates are in frontmatter configuration as well as file such as `YYYY-MM-DD-file-name.md`

```
content/
├── assets
│   └── covers/
│       ├── cover.png
│       └── image.png
└── blog
    └── <category>
        └── YYYY-MM-DD-<post>.md
```

The above however is definitely not a best practice. Blog posts should have their own specific url category and images should be co-located in the same directory, therefor I chose the following structure for the blog posts:

```
content/
├── assets
└── blog
    └── <category>
        └── <post>
            ├──  index.md
            └── image.png
```

This was done mainly manually together with some regex replacements to ensure a smooth process. The folder structure was changed through the following script:

```javascript
const fs = require('fs').promises;
const path = require('path');

async function walk(dir) {
    let files = await fs.readdir(dir);
    files = await Promise.all(files.map(async file => {
        const filePath = path.join(dir, file);
        const stats = await fs.stat(filePath);
        if (stats.isDirectory()) return walk(filePath);
        else if(stats.isFile()) return filePath;
    }));

    return files.reduce((all, folderContents) => all.concat(folderContents), []);
}

async function start() {
    let articles = await walk('./content/blog-old');
    articles.forEach(async (article) => {
        let re = new RegExp(/([0-9a-zA-Z-\.\\]+)[\\]+([0-9a-zA-Z-\.]+)/g);
        let results = re.exec(article);

        let file = results[2];
        let path = results[1];
        
        // Create dir
        let newDir = file.replace('.md', '').replace(/[0-9]{4}-[0-9]{2}-[0-9]{2}-/, '');
        await fs.mkdir(`./content/blog/${path}/${newDir}`, { recursive: true });

        // Create file
        let fileContent = await fs.readFile(`./${path}/${file}`);
        await fs.writeFile(`./content/blog/${path}/${newDir}/index.md`, fileContent);
    })
}

start();
```

## Full Gatsby Blog Structure

For the Gatsby set-up I chose the following file structure:

```
content/
├── static/
├── assets/
├── blog/
│   └── <category>/
│       └── <post>/
│           ├── index.md
│           └── image.png
│   projects/
│     └── <project>/
│         ├── index.md
│         └── image.png
├── config/
│   ├── authors.json
│   └── tags.json
├── src/
├── gatsby-browser.js
├── gatsby-config.js
└── gatsby-node.js
```

Where gatsby is customizeded through the following files:

* `gatsby-config.js` contains the configuration files and modules:
* `gatsby-node.js` contains the creation of static pages for:
    * Blog Pagination
    * Blog Pages
    * Tag Pages
    * Project Pages
* `gatsby-browser.js` contains the custom css files

## CI/CD With GitHub Actions

Next to setting up the blog on Gatsby, I have also decided to utilize GitHub actions for building it. By using this simple YAML script, I could create a `.github/workflows/main.yml` file that instantly deploys the blog upon push.

```yaml
name: Deploy Website
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: amondnet/now-deployment@v2
        with:
          zeit-token: ${{ secrets.ZEIT_TOKEN }} # Required
          now-args: '--prod' #Optional
          now-org-id: ${{ secrets.ORG_ID}}  #Required
          now-project-id: ${{ secrets.PROJECT_ID}} #Required 
          working-directory: ./
```

Revamping my Blog with NextJS (2020)


For most businesses out there it are hard times due to the recent crisis, causing a lot of pressure on optimizing OPEX costs as much as possible. That's why I would love to help all of you out there with some quick tips and tricks on how you can keep your spendings to a minimum on the Azure platform, or how you are able to gain quick benefits.

This list is for sure far of complete, but I will keep adding more along the way. Always feel free to comment below so I can add your tips and tricks to the list as well.

# Tips and Tricks - General

Let's start of with some general guidance when optimizing workloads:

## Shutdown/Resume vs Scale-Down/Scale-Up

One of the - by far easiest - ways of optimising your workload is to start looking into Shutting Down and Resuming or Scaling Down and Scaling Up your workload. However, when doing this we have to keep some points into account that might influence our decision here. Here you can find a list of advantages and disadvantages that each choice entails. 

||Shutdown/Resume|Scale-Down/Scale-Up|
|-|-|-|
|**Advantages**|• **Savings are more substantial** than scaling-down since compute will be turned off completely|• Very **easy to realize gain** (almost no engineering effort needed), especially on Development, Staging and Testing environments|
|**Disadvantages**|• **Engineering Effort is larger** compared to scale-down/scale-up<br />• **Requires automation code** to ensure successful start-up of the components|• **Requires upfront knowledge of current utilization rate** (might be more difficult to calculate)<br />• **Cost savings are not maximized** (though compensation happens due to low engineering cost)|

As a **summary** I would suggest to do the following when performing this excercise:
* What is the dependency graph of your architecture components?
* Is the trade-off engineering effort vs cost gains worth it?

To **realize the decision** made above, create a script that is able to automate the decision above, utilizing either the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest?WT.mc_id=DT-MVP-5004263) or [PowerShell Module](https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?WT.mc_id=DT-MVP-5004263) together with [Azure Logic Apps](https://docs.microsoft.com/en-us/azure/logic-apps/?WT.mc_id=DT-MVP-5004263) or [Azure Automation](https://docs.microsoft.com/en-us/azure/automation/?WT.mc_id=DT-MVP-5004263)

> Note: an example of Starting/Stopping VMs during off-hours can be found here: [https://docs.microsoft.com/en-us/azure/automation/automation-solution-vm-management](https://docs.microsoft.com/en-us/azure/automation/automation-solution-vm-management?WT.mc_id=DT-MVP-5004263)

# Tips and Tricks - Service Dependent

|Service|Tips|
|-|-|
|**Azure CosmosDB**|• Utilize [AutoPilot](https://docs.microsoft.com/en-us/azure/cosmos-db/provision-throughput-autopilot?WT.mc_id=DT-MVP-5004263) if possible<br />• If multiple containers are used, try to enable [shared throughput](https://docs.microsoft.com/en-us/azure/cosmos-db/set-throughput?WT.mc_id=DT-MVP-5004263)<br />• [Optimize your query performance](https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-query-metrics#best-practices-for-query-performance?WT.mc_id=DT-MVP-5004263)
|**Azure Databricks**|• When running automated jobs, utilize the **"Data Engineering"** tier! (Don't utilize the clusters page in Databricks, use an On-Demand cluster through the Jobs page or [Azure Data Factory](https://docs.microsoft.com/en-us/azure/data-factory/solution-template-databricks-notebook?WT.mc_id=DP-MVP-5004263))
|**Azure Kubernetes Service (AKS)**|• Turn down the development, staging or testing nodes towards 1 where possible.|
|**Virtual Machines (VM)**|• If you know your workload, you can easily apply [reserved instances](https://azure.microsoft.com/en-us/pricing/reserved-vm-instances/?WT.mc_id=DT-MVP-5004263) where possible for a 1Y or 3Y commitment|
|**SQL DB**|• If you have multiple databases that have a varying load, it could be interesting to utilize [Elastic Pools](https://docs.microsoft.com/en-us/azure/sql-database/sql-database-elastic-pool?WT.mc_id=DP-MVP-5004263)|

Azure Optimization Tips and Tricks


A couple of months ago it struck me that my personal knowledge in AI could be improved quite a bit. That's why I took the time to brush up on the different concepts ranging from data gathering until the evaluation of a deployed model.

While brushing up my knowledge, I also thought on how I could share this back to the community as well as have a quick overview of everything I learned with pointers and tips of where I should look at to go more in depth on a topic. Making me end up in creating the following battle cards that I hope are useful for everyone to print and use as a kind of "Cheat Sheets".

As you might notice, there is no sklearn being utilized in steps 2. This is because I wanted to make a clear distinction between a **Data Engineer** and a **Data Scientist**, with the following reasoning:

- **Data Engineer:** They often utilize Spark, so we want to utilize the strength and scale-out capabilities of spark, without completely relying on the head node. Therefor I utilize Pandas examples as much as possible, which can be scaled through the [Koalas](https://github.com/databricks/koalas) platform.
- **Data Scientist:** They often use different libraries, with sklearn being one of them.

> **Note:** I realize that these are far from perfect, but I want them to be. If you encounter any remarks, please post them below and I will look at incorporating them :)

Printable Version:

- [Data Gathering](/battlecards/AI%20-%20Data%20Gathering.pdf)
- [Data Cleaning, Preparation and Modification](/battlecards/AI%20-%20Data%20Exploration.pdf)
- [Model Training and Tuning](/battlecards/AI%20-%20Model%20Preparation%20and%20Training.pdf)
- [Model Evaluation](/battlecards/AI%20-%20Model%20Evaluation%20and%20Classification.pdf)
- [One Pager](/battlecards/Battlecards.pdf)

## Step 1 - Data Gathering

![./ai-data-gathering.svg](./ai-data-gathering.svg)

## Step 2 - Data Cleaning, Preparation and Modification

![./ai-data-exploration.svg](./ai-data-exploration.svg)

## Step 3 - Model Training and Tuning

![./ai-model-prep-and-train.svg](./ai-model-prep-and-train.svg)

## Step 4 - Model Evaluation

![./ai-model-evaluation-and-classification.svg](./ai-model-evaluation-and-classification.svg)

AI Battlecards - End to End Process for building and evaluating AI models


The cloud war is heating up and one of the main challenges is the fear of lock-in created by these providers. Helping the raise of Kubernetes as a provider, since this allows customers to quickly move between one and the other (a deployment is a deployment, no matter the cloud you run it on.).

Looking at the above, a new Runtime was created that allows customers to create applications that still allow the goodness of these cloud platforms (PaaS components), while developing their Microservices. Which is why **[Dapr](https://dapr.io)** was born.

Dapr stands for Distributed Application Runtime and provides building blocks to create a resilient, stateless or stateful microservice that can run on the cloud and edge, while still being language independent. Backed by HTTP or gRPC depending on the performance needs that you have.

## Getting Started - Installing Dapr

Dapr was amazing to get started with! In just 5 minutes I had my first microservice running by following the [documentation](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md). To setup Dapr, we can simply run:

**Windows - PowerShell**

```bash
powershell -Command "iwr -useb https://raw.githubusercontent.com/dapr/cli/master/install/install.ps1 | iex"
dapr init
```

**Linux**

```bash
wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
dapr init
```

> **Important:** When running `dapr init`, dapr will take care of automatically creating your **state store** under the `components/` folder. We will now have 2 files called `messagebus.yaml` and `statestore.yaml` that we can use to communicate with.

## Getting Started - Running Python gRPC Example

For low-latency requiring application, gRPC is a smart choice. It's often used when we require inter process communication between applications, when normal HTTP overhead is not required (e.g. simulators that we want to hook up together).

Luckily Dapr allows both HTTP and gRPC connectors, making our lives easier! When looking at the [gRPC Documentation](https://github.com/dapr/docs/tree/master/howto/create-grpc-app) we just need to specify that we are utilizing gRPC through the command line (or Kubernetes YAML file).

Therefor we can get started by creating a Python example called **server.py**:

```python
from dapr import dapr_pb2 as messages
from dapr import dapr_pb2_grpc as services
import grpc
from google.protobuf.any_pb2 import Any

channel = grpc.insecure_channel('localhost:50001')
client = services.DaprStub(channel)

# Set the state
stateStoreName="my-state-store" # Note: see components/statestore.yaml at the metadata.name key and configure correctly
stateReq = messages.StateRequest(key="my-first-state", value=Any(value="Hello World".encode('utf-8')))
stateEnvelope = messages.SaveStateEnvelope(storeName=stateStoreName, requests=[stateReq])
client.SaveState(stateEnvelope)

# Close the channel
channel.close()
```

Which we can then start in the Dapr runtime through:

```bash
# Start our server.py in Dapr on HTTP Port 50002 and gRPC Port 50001
dapr run --app-id python-grpc --port 50002 --protocol grpc --grpc-port=50001 python server.py
```

Instantly showing the output stating that it started and saved our state configuration.

```bash
PS F:\dapr> dapr run --app-id python-grpc --port 50002 --protocol grpc --grpc-port=50001 python server.py
Starting Dapr with id python-grpc. HTTP Port: 50002. gRPC Port: 50001
You're up and running! Both Dapr and your app logs will appear here.

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="starting Dapr Runtime -- version 0.4.0 -- commit v0.3.0-rc.0-73-ga1e270f-dirty"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="log level set to: info"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="standalone mode configured"       

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="dapr id: python-grpc"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="mTLS is disabled. Skipping certificate request and tls validation"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="loaded component my-state-store (state.redis)"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="loaded component messagebus (pubsub.redis)"

== DAPR == 2020-02-17 21:41:29.863654 I | redis: connecting to localhost:6379

== DAPR == 2020-02-17 21:41:29.870653 I | redis: connected to localhost:6379 (localAddr: [::1]:56504, remAddr: [::1]:6379)

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="Initialized service discovery to standalone"

== DAPR == time="2020-02-17T21:41:29+01:00" level=warning msg="failed to init input bindings: 
app channel not initialized"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actor runtime started. actor idle 
timeout: 1h0m0s. actor scan interval: 30s"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actors: starting connection attempt to placement service at localhost:6050"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="http server is running on port 50002"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="gRPC server is running on port 50001"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="local service entry announced"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="dapr initialized. Status: Running. Init Elapsed 17.0031ms"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actors: established connection to 
placement service at localhost:6050"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actors: placement order received: 
lock"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actors: placement order received: 
update"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actors: placement tables updated"

== DAPR == time="2020-02-17T21:41:29+01:00" level=info msg="actors: placement order received: 
unlock"

== APP == Saved State
```

When now checking this with the following `curl` command, we are able to validate our state over the HTTP interface!

```bash
curl http://localhost:50002/v1.0/state/my-state-store/my-first-state
```

**Result:**

```bash
PS F:\dapr> curl http://localhost:50002/v1.0/state/my-state-store/my-first-state       
Hello World
```

And there we go, as simple as that we are now able to utilize Dapr with a state store to communicate through gRPC (which is normally something quite complex to do).