A couple of years ago, I decided to buy a Raspberry PI cluster with [Clusterhat](https://clusterhat.com/). In all that time however, I never too the time to configure it correctly to make it able to run Kubernetes. So now I decided to invest some time and get some learnings back from it!

> Note: Right now the ecosystem is utilizing ARMv7 mainly, while as the Pi Zeros are still running ARMv6. It will thus be an interesting task to get Kubernetes running with the Pi Zeros as nodes. Do note that supporting legacy hardware is an important skill that a lot of companies (especially Manufacturing ones) value a lot! Seeing that the devices they have running are often even older! 

So let's get started on how you can set this up! 

// @todo: insert picture clusterhat and pi zero
![]()

## Flashing the Images

We download the Images from https://clusterctrl.com/setup-software utilizing the CNAT version.

> The CNAT version has NAT configured (Network Address Translation) which will automatically share the Ethernet or Wifi on the controller with the Pi Zeros.

Download Links (for backup purposes):

* CNAT Lite Controller: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-CNAT.zip
* P1: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p1.zip
* P2: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p2.zip
* P3: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p3.zip
* P4: http://dist.8086.net/clusterctrl/buster/2020-12-02/2020-12-02-1-ClusterCTRL-armhf-lite-p4.zip

Once the images are downloaded, you can utilize a flashing tool to flash them to the SD Cards. For this I utilized the excellent tool called [Etcher](https://www.balena.io/etcher/) and flashed the different images for the controller and PI Zeros. 

![](./etcher.png)

After each image was flashed, I added a `SSH` file (without any content) to the SD Cards root.

> By adding the `SSH` file, we are able to SSH to the different Nodes.

![](./create-ssh-file.png)

Now the hardest part is done, and we can connect the Ethernet cable and boot up the Controller!

> Note: Make sure to utilize a Power Adapter instead of a USB Cable connected to the PC.

> Note 2: I connected the Zeros from Power Adapter back labeled 1, 2, 3 and 4

## Connecting to SSH

After waiting a couple of minutes, we should be able to access the Raspberry PI Controller. To make things easier and remove the need to connect a screen + keyboard, I used the tool [Advanced IP Scanner](https://www.advanced-ip-scanner.com/) find out the IP Address of the controller by quickly scanning the network.

![](./ip-scanner.png)

Through this way, I discovered that mine was running under `192.168.1.56`. So go ahead and log in on it then through `ssh pi@<your_ip>` with username `pi` and password `clusterctrl`

* Username: `pi`
* Password: `clusterctrl`

> Note: if you want to change your password, utilize `sudo raspi-config` and go to `Change User Password`

![](./clusterhat-ssh.png)

## Turning on the PI Zeros

The ClusterHat module acts as a USB Hub, which is turned off by default. To help us, a helper program was included that allows us to turn them on. 

* Turn Raspberry PI Zeros On: `sudo clusterhat on` 
* Turn Raspberry PI Zeros Off: `sudo clusterhat off`

By default, ClusterHat exposes its USB Hub interface over gateway `172.19.181.1` to connect to the PI Zeros we can then connect over IPs `172.19.181.1 - 172.19.181.4`. To make our lives a bit easier, we set up a config files for the SSH Names in `~/.ssh/config`:

```bash
Host p1
 HostName 172.19.181.1
 User pi

Host p2
 HostName 172.19.181.2
 User pi

Host p3
 HostName 172.19.181.3
 User pi

Host p4
 HostName 172.19.181.4
 User pi
```

By then adding a SSH Public Key to them we are able to more quickly login. So first create a SSH Key with:

```bash
ssh-keygen -t rsa -b 4093
```

And copy it over to the PI Zeros with:

```bash
cat ~/.ssh/id_rsa.pub | ssh pi@p1 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh pi@p2 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh pi@p3 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh pi@p4 -T "mkdir ~/.ssh && cat > ~/.ssh/authorized_keys"
```

We can now connect to our PI Zeros without having to enter a password anymore! 😀

![](./clusterhat-ssh-p1.png)

## Syncing the time

We are almost there, just before our last step we need to install the time syncing tool. To do this, run the following commands on your controller:

```bash
ssh pi@p1 "sudo apt-get install -y ntpdate" &
ssh pi@p2 "sudo apt-get install -y ntpdate" &
ssh pi@p3 "sudo apt-get install -y ntpdate" &
ssh pi@p4 "sudo apt-get install -y ntpdate" &
```

## Finalization

As a final step - yes we are there! - we need to adapt our `/etc/rc.local` script such that we automatically spin up the cluster on boot. To do that, add the following lines to your `/etc/rc.local` file:

```bash
# Start the PI Zeros
/sbin/clusterhat init
/sbin/clusterhat on

# Fix networking towards the PI Zeros
/sbin/iptables -I forward -j ACCEPT
```

## Conclusion

From the steps above we can conclude that it is SUPER easy to set up our own cluster! In just 1 hour I have 4 Raspberry PI Zeros running on my Clusterhat on the network. 

The sad part of this story however is that while this is an amazing architecture, it is based on older hardware (running ARMv6). Due to this, it is practically impossible to run demonstrations on this. Which is why I decided to abandon this cluster and move on. Currently I am looking into alternatives for a nice demo cluster, so let me know if you know something!

References
* https://medium.com/@dhuck/the-missing-clusterhat-tutorial-45ad2241d738
* https://github.com/AKYD/clusterhat-kubernetes

Creating a Raspberry PI Cluster (1 Raspberry Pi & 4 Pi Zeros) with Cluster Hat


Deepmind has [achieved a huge milestone](https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules) by publishing its latest paper around Reinforcement Learning in [Nature - 23/DEC/2020](https://www.nature.com/articles/s41586-020-03051-4). How they were able to train a Reinforcement Learning algorithm that masters Go, Chess, Shogi and Atari **without needing to be told the rules**.

## Introduction

I have always been interested in Artificial Intelligence. However, when looking at the landscape today, we see that it mainly consists of plain statistics, where we feed an algorithm a lot of data and we will eventually "generalize" well enough, such that it can be utilized on previously unknown datasets. While this has a huge impact on society and a lot of business value is extracted from it, my personal interest is more in the domain of Artificial General Intelligence (AGI) where a machine has the ability to learn an intellectual task without the intervention of a human. Or in a more simple sentence: 

>🤖 "We throw a task to the machine and it will figure out what to do and become good at it"

## Knowledge in Reinforcement Learning

In the figure below - created by Deepmind - we can see that a huge step was taken. Previously we were always required to have a certain knowledge around the domain (Human Data, Domain Knowledge or the Rules of the environment) such as in the AlphaGo engine which then evolved to only requiring to know the rules of the environment (AlphaGo Zero and AlphaZero) but we were never able to remove this requirement. **With MuZero we are now able to learn the rules of the games, removing the need for knowledge**

![](./rl-domains-knowledge-requirements.webp)

With this article, I mainly want to focus on applying the MuZero algorithm myself for the Lunar Lander environment. 

For more information around what MuZero exactly introduces and an in-depth coverage of it, I would recommend to check out the following resources:

* [MuZero Blog](https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-atari-without-rules)
* [MuZero YouTube Video](https://www.youtube.com/watch?v=L0A86LmH7Yw&ab_channel=JulianSchrittwieser)
* [MuZero Nature Article](https://www.nature.com/articles/s41586-020-03051-4)
* [MuZero Code](https://www.nature.com/articles/s41586-020-03051-4#code-availability)

## MuZero Architecture

As detailed in the MuZero Code overview, we can find the pseudocode in `pseuducode.py` in the [supplementary information](https://www.nature.com/articles/s41586-020-03051-4#MOESM1). 

The Pseudocode has the `def muzero()` that will start the algorithm for us. This will in its turn run 2 independent parts:

1. Network Training
2. Self Play data generation

```python
def muzero(config: MuZeroConfig):
  storage = SharedStorage()
  replay_buffer = ReplayBuffer(config)

  for _ in range(config.num_actors):
    launch_job(run_selfplay, config, storage, replay_buffer)

  train_network(config, storage, replay_buffer)

  return storage.latest_network()
```

The Pseudocode above will thus create **actors** that will take a snapshot of the latest network and make it available to the training job by writing it to a shared replay buffer.

To visualize this I created a simple diagram:

![](./muzero-architecture.png)

## MuZero Running

Now let's get started on running the MuZero algorithm ourselves! 

For my system I am using the following configuration:

* **Operating System:** Windows (Running WSL Ubuntu 20.04)
* **Python:** 3.8.6 64-bit

> ❗ I would recommend following [my previous article](/post/2021/1/5/coding-python-nvidia-wsl) that goes into more detail on how you can setup WSL with NVIDIA

Once all of this is working (which might take a while), we can start setting up our project. In this article, I have chosen to utilize the excellent repository written by [Werner Duvaud](https://www.linkedin.com/in/werner-duvaud/), seeing that he already implemented the algorithm on specific environments.

To run it yourself, feel free to utilize these commands:

```bash
# Clone repository
git clone https://github.com/werner-duvaud/muzero-general.git

# Open repository
cd muzero-general/

# Install PyTorch
# Note: you might need to customize this for your environment 
# https://pytorch.org/get-started/locally/
python -m pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# Install dependencies for the Gym Environment
sudo apt install cmake zlib1g-dev xorg-dev libgtk2.0-0 swig python-opengl xvfb

# Install Atari for the Gym Environment
python -m pip install gym[atari]
```

After the above has been executed, you should now be able to run the repository code through the commands below to start the training process.

```bash
python muzero.py
```

![](./muzero-running-1.png)

To view the training we can run Tensorboard within our windows environment with `tensorboard --logdir ./results --bind_all` which will show us our results:

![](./muzero-running-3-200000-1480.png) 

After training finishes, we can use the created model, resulting in the results below.

> Note: you can export the display with `export DISPLAY=$(ip route | awk '/^default/{print $3; exit}'):0` in WSL

![](./trained-lunar-lander.gif)

# Conclusion

Congratulations! We have just applied the MuZero algorithm and were able to train the Lunar Lander environment correctly 😎 of course this is not a representation of the real world, so you should definitely test this yourself! But it's a start! 

Seeing the interesting architecture of MuZero and the utilization of Actors, my next idea would be to try to utilize Dapr to start the out-scaling process. Hopefully I will be able to adopt some of the practices here and apply them in my recently released [Roadwork-RL](https://github.com/roadwork/roadwork-rl) library.

Let me know what you think!

Implementing Deepmind's MuZero Algorithm with Python


Running CUDA on the Windows Subsystem for Linux (WSL) is not a trivial task seeing that it requires the GPU to be available on a Virtual Machine. In this article I go in more detail how you can get your GPU working in WSL through something NVIDIA calls "Paravirtualization" 

> 💡 It is quite straightforward to utilize our Graphic Card, but at this moment of writing, a lot of dependencies are not out of the box for everyone.

# Prerequisites

* [A working WSL environment with Python installed](/post/2021/1/4/coding-python-windows-development-environment-setup)
* [Windows Insider Build >= 20150- Dev Channel](https://docs.microsoft.com/en-us/windows/win32/direct3d12/gpu-cuda-in-wsl?WT.mc_id=AI-MVP-5004263)
* WSL 2 Kernel >= 4.19.121 (check with `uname -r`)
* An NVIDIA Graphic Card

# Tool Installation

## Installing Docker on WSL

Once our WSL environment is setup, we can get started with setting up Docker. Installing Docker is quite trivial, but we should make sure **not** to utilize the apt packages, but the installer provided by docker. 

```bash
curl https://get.docker.com | sh
```

## Installing NVIDIA for Docker

Once Docker is installed, we need to install Nvidia their toolkit for it. Do this by running the commands below

```bash
# Enable Nvidia for Docker
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
curl -s -L https://nvidia.github.io/libnvidia-container/experimental/$distribution/libnvidia-container-experimental.list | sudo tee /etc/apt/sources.list.d/libnvidia-container-experimental.list
sudo apt update 
sudo apt install -y nvidia-docker2
sudo apt install nvidia-container-toolkit

# Finally restart the docker service
sudo service docker restart
```

# Testing

If the above was done correctly, we should now be able to test our graphic card by starting a benchmarking container:

```bash
sudo docker run --rm -it --gpus=all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -benchmark
```

Which should show something like the below:

![](./nvidia-nbody.png)

# Conclusion

It's quite trivial to run NVIDIA with its paravirtualization support, but we should make sure that we meet the prerequisites. 

Running CUDA on Windows Subsystem for Linux (WSL)


Setting up a Python Development environment on Windows might be a daunting task to do right. Especially since a lot of tools cause confusion or might not work out of the box.

If you have ever worked with Python before on windows, you will know that it might not be the best choice due to incompatible toolings, access violations, ...

In a [previous article](/post/2018/5/7/ai-ml-rl-running-openai-gym-on-windows-and-js), I have already explained how I set-up my environment to run the OpenAI Gym, but I wanted to revisit this for my new workflow to make it easier and more versatile.

So let's get started!

# Installing Linux on Windows through WSL

The first thing to do is to install the Windows Subsystem for Linux. Since many libraries in python are not working out of the box, it's best to always run everything through WSL.

The Windows Subsystem for Linux (WSL) was introduced not that long ago, but became more useful since its latest release (WSL 2) in [Windows 10 version 2004](https://devblogs.microsoft.com/commandline/wsl2-will-be-generally-available-in-windows-10-version-2004/?WT.mc_id=AI-MVP-5004263). 

> 💡 In more technical terms, the biggest change is that WSL 1 used syscall translations while WSL 2 is a native VM

By default, WSL installs straight on your C drive. However for disk space requirements it might be handier to install this on a separate disk (e.g. `E:\`).

Execute the following in PowerShell and adapt the drive location to install it to **the location of your choice**.

```powershell
# Navigate to the destination directory
cd E:\WSLTest

# Enable WSL
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

# Downloading Ubuntu 20.04
Invoke-WebRequest -Uri https://aka.ms/wslubuntu2004 -OutFile Ubuntu.appx -UseBasicParsing

# Unpack
move .\Ubuntu.appx .\Ubuntu.zip
Expand-Archive .\Ubuntu.zip

# Install
.\Ubuntu\ubuntu2004.exe

# Verify
wsl --list --all
```

# Installing Python and PyEnv

Once the Linux Subsystem is set-up, open it by clicking the `.exe` file in the created folder (e.g. `ubuntu2004.exe`). Easiest is to now create a shortcut of it on your taskbar. We should now see something like the screenshot below:

![](./wsl.png)

To manage our Python environment, I chose to utilize [`pyenv`](https://github.com/pyenv/pyenv) which allows us to manage the installed Python version.

> 💡 The reason we are using a Python Version Manager is because a lot of tools do not work in certain versions of Python (e.g. Reinforcement Learning often requires Python smaller or equal to 3.7)

So lets install PyEnv and Python 3.8.6!

## Installing PyEnv

```bash
# Update
sudo apt update -y

# Install dependencies
# https://github.com/pyenv/pyenv/wiki#suggested-build-environment
sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

# Clone pyenv
git clone https://github.com/pyenv/pyenv.git ~/.pyenv

# Configure the environment
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc

# Restart the shell (to enable pyenv)
exec "$SHELL"

# Verify installation
pyenv install --list
```

## Installing Python 3.8.6

Now PyEnv is installed, we can start managing our python installation.

```bash
# Show available versions
pyenv install --list

# Install version 3.8.6
pyenv install 3.8.6

# Verify installation
pyenv versions

# Configure the global python command to utilize 3.8.6
pyenv global 3.8.6

# Check version
python --version
```

# Configuring Python Tooling

## Opening our Code Editor - VSCode 

We are now set to start programming in Python! However, as in many cases we would like to fix our IDE (Integrated Development Environment = Code Editor) to work with the installed Python version.

Luckily for us, Microsoft ships Visual Studio Code together with its WSL images. So when we run `code .` we will open up a new VSCode instance straight on the path that we executed it from.

Let's thus create an `example` folder and run the command above in it to start-up VSCode:

```bash
mkdir example
cd example/
code .
```

Which will show

![](./wsl-vscode.png)

![](./wsl-vscode-2.png)

## Creating Jupyter Notebooks

Once VSCode is opened we can create Jupyter Notebooks by installing the Notebook Extension. So in VSCode, search after `jupyter` to find and install the Jupyter Notebook extension.

Then do the same for the `Python` extension.

![](./vscode-jupyter-2.png)

![](./vscode-jupyter-3.png)

If this is done, we are now able to create a Notebook and run a simple command in it!

![](./vscode-jupyter-4.png)

![](./vscode-jupyter-5.png)

![](./vscode-jupyter-6.png)

## Forwarding the Windows Display

Finally, it is interesting for graphical reasons to be able to output a Python GUI to windows when being rendered.

For this we can utilize a tool named `Xming`, so [download and install this](https://sourceforge.net/projects/xming/).

After installing Xming, go into the `Program Files` folder and execute `launch.exe` to configure it as follows.

![](./xming-1.png)

![](./xming-2.png)

![](./xming-3.png)

When it is configured, execute the following to add it to your environment.

```bash
# Export our display settings for XMing
export DISPLAY=$(ip route | awk '/^default/{print $3; exit}'):0
```

# Conclusion

Congratulations! You are now ready to start developing with Python and easily manage the versions. Feel free to let me know in the comments below if this was interesting for you and how it helped you!

How to set up a Python Development Environment Setup on Windows (with WSL)


Kubernetes is getting more popular everyday and it's no wonder why! When you are running applications on-premise or in-cloud, the possibility of having the applications in a portable way is a strong one! Removing the friction for scaling-out your application when you are ready for it, or even bursting scenarios.

## Use Case

For a use case I have been working on, I have the following requirements:

* 1 HTTP request = 1 Container 
* Instance can be up to 1GB of memory
* No pub/sub, but synchronous HTTP request/response!
* Requests execution time can take up to 60 seconds
* I do not want to utilize a pool, but have an auto-scaling system in place

## Possible solutions

As in any project, there are multiple paths towards a suitable solution. After researching this a bit, I came across the following tools that looked promising:

* [Knative](https://knative.dev/)
  * This felt like over-kill. It's quite complex to install (though there are tools such as `knctl` but it's outdated) and is quite heavy on the cluster it seems.
* [Keda](https://keda.sh/)
  * Super interesting auto scaler, but no HTTP based scaling supported (it's event driven) which I would need.
* [Fission](https://fission.io/)
  * Interesting! It however seems to keep images in memory (which would be 1GB in memory all the time for us) and auto-scaling is similar to Keda in this sense (it even supports KEDA)

Looking at the above, I was kind of disappointed since no solutions "seemed" to exist. But after a bit more of experimenting and exploration I came across a blog post by [Anirudh Garg](https://dev.to/anirudhgarg_99) explaining how you can hook up an Nginx Ingress controller to Keda!

Now the above does solves my issue completely, seeing that I want to transform an async request into a synchronous request (i.e. I want to to get my HTTP request, auto scale the cluster and then return the response). Thus a custom component will have to be written in my case. The blog post however goes into how KEDA can be utilized, which seems to be an excellent fit for me!

## Architecture

As always, the most important when creating a new application is to at least draw out a high-level architecture of how the idea will look like. It might sound like a boring and unnecessary job, but I think it helps me a lot in doing a thought excercise that makes me think things through more before wasting time on trying things that won't work. Even if it doesn't work out correctly, you atleast tried 😉

![Architecture](./architecture.png)

> **Note:** you can find the full source code utilised in this post here: [https://github.com/XavierGeerinck/PublicProjects/tree/master/JS/Dapr/AutoScalingHTTP](https://github.com/XavierGeerinck/PublicProjects/tree/master/JS/Dapr/AutoScalingHTTP)

## Solution (Gateway and Worker)

### Used Technologies

As for technologies, I decided to settle on the following:

* Dapr
* NodeJS
  * Express
* RabbitMQ
* Kubernetes
* KEDA

### Prerequisites

Before we get started, make sure the following is available:

* Helm
* Kubernetes cluster with `kubectl` mapped to it (**Note:** locally I just run `minikube start --cpus 2 --memory 2048` and I'm ready to go)

### Installing RabbitMQ

Once Dapr has been installed, we want to set up our queue. As shown in the architecture, we will have an inbound application that puts items on a queue. As soon as these items have been published, a worker will take them and process what it has to do (in our case, returning "Hello World" after a small delay).

To install RabbitMQ, run the following commands:

```bash
# ref: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq/#parameters
# Install repo
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Install RabbitMQ
helm install rabbitmq --set auth.username=admin,auth.password=admin,volumePermissions.enabled=true bitnami/rabbitmq
```

> **Note:** The default username is "user" and the default hostname is "rabbitmq.default.svc.cluster.local:5672" as dictated by the Kubernetes pod defaults of "SVC_NAME.NAMESPACE.cluster.local". The moniker then becomes `amqp://user:YOUR_PASSWORD@rabbitmq.default.svc.cluster.local:5672`.

> **Important:** When you created one before, make sure to delete the old PersistentVolumeClaim else the password won't work (`kubectl get pvc`)

### Installing Dapr

The first thing we want to do once we have a Kubernetes cluster ready is to deploy and [install Dapr](https://github.com/dapr/docs/blob/master/getting-started/environment-setup.md) on it. We can do this by running the following command:

```bash
# Add helm repo
helm repo add dapr https://dapr.github.io/helm-charts/
helm repo update

# Create namespace dapr-system
kubectl create namespace dapr-system

# Install latest dapr version
helm install dapr dapr/dapr --namespace dapr-system
```

### Installing Dapr Bindings for RabbitMQ

Once Dapr and RabbitMQ have been installed, we can create [bindings](https://github.com/dapr/docs/blob/master/reference/api/bindings_api.md) that will allow us to receive RabbitMQ events on a specific endpoint, as well as send events to RabbitMQ through calling a HTTP endpoint.

For this we create the following `.YAML` files and apply them on our kubernetes cluster with `kubectl apply -f binding-rabbitmq-in.yaml` and `kubectl apply -f binding-rabbitmq-out.yaml`

**binding-rabbitmq-in.yaml**

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: rabbitmq-worker-input
  namespace: default
spec:
  type: bindings.rabbitmq
  metadata:
  - name: host
    value: amqp://user:YOUR_PASSWORD@rabbitmq.default.svc.cluster.local:5672
  - name: queueName
    value: rabbitmq-worker-input
```

**binding-rabbitmq-out.yaml**

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: rabbitmq-worker-output
  namespace: default
spec:
  type: bindings.rabbitmq
  metadata:
  - name: host
    value: amqp://user:YOUR_PASSWORD@rabbitmq.default.svc.cluster.local:5672
  - name: queueName
    value: rabbitmq-worker-output
```

### Creating our Gateway

On to the hardest part! The gateway! If we take a look again at our architecture, we can see that 2 servers are required here. The reason being that we want to be able to divide external traffic (user request) from the internal Dapr traffic.

![](./architecture-gateway.png)

We thus create 2 files named `src/server-dapr.ts` and `src/server-external.ts` that will handle user traffic and internal traffic.

> **Note:** For the full source code, feel free to check the repository [https://github.com/XavierGeerinck/PublicProjects/tree/master/JS/Dapr/AutoScalingHTTP](https://github.com/XavierGeerinck/PublicProjects/tree/master/JS/Dapr/AutoScalingHTTP)

> **Note:** For testing purposes, it's easier to start-up the instance with `dapr run --app-id gateway --app-port 4000 --components-path ../components npm run start:dev` when running it in the `AutoScalingHTTP/Gateway` folder when running from the source code.

Once this is done, we can deploy it through the following Kubernetes YAML:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: d-dapr-autoscaling-http-gateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dapr-autoscaling-http-gateway
  template:
    metadata:
      labels:
        app: dapr-autoscaling-http-gateway
      annotations:
        dapr.io/enabled: "true" # Do we inject a sidecar to this deployment?
        dapr.io/id: "dapr-autoscaling-http-gateway" # Unique ID or Name for Dapr App (so we can communicate with it)
        dapr.io/port: "5001" # Port we are going to listen on for Dapr interactions (is app specific)
    spec:
      containers:
      - name: main # Simple name so we can reach it easily
        image: thebillkidy/dapr-autoscaling-http-gateway:latest
        imagePullPolicy: Always
        ports:
          - containerPort: 5000 # This port we will expose (external port)
        env:
        - name: DAPR_HOST
          value: "127.0.0.1"
        - name: DAPR_PORT
          value: "3500"
```

and run `kubectl apply -f deploy/k8s-gateway.yaml` with `kubectl logs -f deployment/d-dapr-autoscaling-http-gateway -c main` to view the logs

Now this is deployed, we can expose the service so we can access it externally:

```bash
# Production clusters
kubectl expose deployment d-dapr-autoscaling-http-gateway --type=LoadBalancer --name=svc-dapr-autoscaling-http-gateway --port=80 --target-port=5000

# Development (minikube) cluster
# when running on dev with minikube, we need to expose the svc like this through kubectl port-forward which will open on a random port
kubectl port-forward --address 0.0.0.0 deployment/d-dapr-autoscaling-http-gateway :5000
```

### Creating our Worker

The worker is a bit easier, in this case we will just listen to the incoming work through the input binding that we created. Once work comes in, we then do something (in this case a timeout) and then put it back on the queue! 😀

Kubernetes deployment YAML:

```yaml
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: d-dapr-autoscaling-http-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dapr-autoscaling-http-worker
  template:
    metadata:
      labels:
        app: dapr-autoscaling-http-worker
      annotations:
        dapr.io/enabled: "true" # Do we inject a sidecar to this deployment?
        dapr.io/id: "dapr-autoscaling-http-worker" # Unique ID or Name for Dapr App (so we can communicate with it)
        dapr.io/port: "3000" # Port we are going to listen on for Dapr interactions (is app specific)
    spec:
      containers:
      - name: main # Simple name so we can reach it easily
        image: thebillkidy/dapr-autoscaling-http-worker:v0.0.1
        imagePullPolicy: Always
        ports:
          - containerPort: 3000
        env:
        - name: DAPR_HOST
          value: "127.0.0.1"
        - name: DAPR_PORT
          value: "3500"
```

and run `kubectl apply -f deploy/k8s-worker.yaml`

## Demo Application

When we now go to the URL of the Gateway and we enter some query parameters (e.g. `http://172.28.28.243:38203/?name=Xavier%20Geerinck&timeout=10000`) we will see the following result:

![](./demo-1.png)

Showing what we wanted to achieve! A synchronous HTTP client that works over an Async Queue implementation for offloading the work to workers. But how do we now go and autoscale this? Well this is where KEDA comes in!

## Setting up Autoscaling with KEDA

[KEDA stands for "Kubernetes Event-Drive Autoscaling"](https://keda.sh/) which allows us to monitor certain resources and automatically scale up a deployment based on our needs. It's super easy to set-up and is super powerful in horizontal scalable solutions. So let's get started on configuring this for our set-up!

### Installing KEDA

The first thing we should do is to install KEDA. We can simply do this by adding the repository and installing it to our created namespace:

```bash
helm repo add kedacore https://kedacore.github.io/charts
helm repo update

# Install KEDA v2
kubectl create namespace keda
helm install keda kedacore/keda --version 2.0.0-rc --namespace keda
```

### Configuring KEDA

Once Keda is up and running, we need to configure it to autoscale based on the metrics from our nginx controller. To describe this, we can utilize a YAML configuration that details the different aspects that we want to watch and act upon.

**keda-rabbitmq-dapr-http-autoscale.yaml**

```yaml
# https://keda.sh/docs/2.0/scalers/rabbitmq-queue/
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: keda-dapr-autoscaling-http-worker
  namespace: default
spec:
  scaleTargetRef:
    name: d-dapr-autoscaling-http-worker
  triggers:
  - type: rabbitmq
    metadata:
      host: amqp://admin:admin@rabbitmq.default.svc.cluster.local:5672 # references a value of format amqp://guest:password@localhost:5672/vhost
      queueName: rabbitmq-worker-input
      queueLength: "2" # After how many do we scale up?
```

Once we created this, another `kubectl apply -f deploy/keda-rabbitmq-dapr-http-autoscale.yaml` will do the trick to configure the autoscaling! 

## Summary

In this article I showed you how you can utilise technologies such as Dapr and KEDA to make autoscaling workers a piece of cake on Kubernetes! In just a few simple steps we can achieve integrations and monitoring that would else take us a lot of infrastructure work.

I hope you enjoyed this and would love to see your opinions about it! Let me know how I can improve this article further!

Auto scaling a HTTP triggered application in Kubernetes using Keda

For a lot of infrastructure related blog posts I would like to utilize a very basic application that spins up a web server and allows us to access it. Once we access it we simple see a welcome message but with a delay that we can configure! 

> **Note:** Why a delay? Since we want to be able to test slow loading infrastructure! (load tests, scale tests, ...)

Or in other words, when I access: `http://localhost:3000/?name=Xavier%20Geerinck&delay=10000`

I get:

```
Welcome back, Xavier Geerinck

-- Page loaded in 10s
```

So let's build this with `Node.JS` and `Express`

## Creating our Application

### Prerequisites 

As always, we start off by creating a folder for our project and initializing the project. To initialize this, we can use:

```bash
npm init -y
npm i express --save
```

### Building the app code

Now for the application code it's quite straightforward, we utilize express to start a server and then add a basic route that takes an incoming `GET request` at the root route `/` and takes in query parameters `[ "name", "timeout" ]`.

```javascript
const express = require('express')
const app = express()
const port = parseInt(process.argv[2]) || 3000;

app.get('/', async (req, res) => {
 const hrstart = process.hrtime();
 console.log('Processing incoming request!');

 let { name, timeout } = req.query;
 name = name || "John Doe";
 
 if (timeout) {
 console.log(`- Need to wait ${timeout}ms`);
 await delay(timeout);
 }

 console.log(`- DONE`);

 const hrend = process.hrtime(hrstart);

 return res.send(`Welcome back, ${name} -- Page timeout: ${timeout}ms -- Page loaded in: ${hrend[0]}s ${hrend[1] / 1000000}ms`);
})

app.listen(port, () => {
 console.log(`Example app listening at http://localhost:${port}`)
})

const delay = async (timeout = 1000) => {
 return new Promise((resolve, reject) => setTimeout(resolve, timeout));
}
```

### Packaging it as a container

Whereafter we quickly create a Docker container out of it with the following Dockerfile

**Dockerfile**

```
FROM node:latest

WORKDIR /usr/src/app

# Install deps
RUN apt-get update

# Create Certificate
RUN apt-get install ca-certificates

# Install Package.json dependendencies
COPY package.json .
RUN npm install

# Copy Source Code
ADD . /usr/src/app

CMD [ "npm", "run", "start" ]
EXPOSE 3000
```

That we can then build with:

> Note: Since we often want to develop locally, we don't have to push the container. For that first load your minikube and then start building your container `eval $(minikube docker-env)` and validate with `docker images`

```bash
docker build -t local/myapp-helloworld .
```

### Running it on Kubernetes

Once it is build, we can now run it on our Kubernetes cluster. For that we create a Deployment YAML file:

**kubernetes.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: d-myapp-helloworld
spec:
 selector:
 matchLabels:
 app: myapp-helloworld
 replicas: 1
 template:
 metadata:
 labels:
 app: myapp-helloworld
 spec:
 containers:
 - name: container-myapp-helloworld
 image: local/myapp-helloworld:latest
 imagePullPolicy: Never # we compiled it locally
 ports:
 - containerPort: 8000
```

That we can apply with `kubectl apply -f kubernetes.yaml` and should now show the following after running `kubectl get pods -A`:

```bash
NAME READY STATUS RESTARTS AGE
d-myapp-helloworld-795f6f46b9-fh4pz 1/1 Running 0 2s
```

Creating a simple "Hello World" docker application with Express and NodeJS


[In a previous article](/post/2020/9/20/ai-ml-kubernetes-nginx-ingress-controller) I explained how we can set-up an Nginx Kubernetes Ingress Controller, but how can we now monitor this? This is what I would like to tackle in this article, on how we are able to utilize Prometheus and Grafana to start visualizing what is happening on our Ingress Controller.


## Prerequisites

After following [my previous article](/post/2020/9/20/ai-ml-kubernetes-nginx-ingress-controller) you will now have the following components running:
* An Nginx Ingress controller deployed in the `ingress-nginx` namespace
* A demo application that we can reach through a `ingress` route

If you do not have these 2 components running, I would recommend you to go check it out! 

## Setting up Prometheus

When we open the reference of [`ingress-nginx` online](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/monitoring.md) we can see that it should be quite straightforward to install prometheus.

Simply run the following to deploy and configure the Prometheus Server:

```bash
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/
```

Once this is done (~2 min) you will see some output of everything that was created:

```bash
serviceaccount/prometheus-server created
role.rbac.authorization.k8s.io/prometheus-server created
rolebinding.rbac.authorization.k8s.io/prometheus-server created
configmap/prometheus-configuration-hct76d4c56 created
service/prometheus-server created
deployment.apps/prometheus-server created
```

Now on to the "hard" part - reconfiguring our Nginx.

> Note: at the time of writing (20-SEP-2020) the documentation was incomplete, making me spend 5h on the issue of reconfiguring nginx...

## Reconfiguring Nginx

Reconfiguring Nginx to allow metrics to be sent to Prometheus might sound as an easy task, but actually it isn't... It appears that the official documentation is lacking on this point and that a [recent pull request](https://github.com/kubernetes/ingress-nginx/pull/6024) requires the [ServiceMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md) to be used.

Installing the ServiceMonitor is a good option, but when checking this on the [official repository](https://github.com/prometheus-operator/prometheus-operator) it shows that this is still in "Beta" so it's not a really a good option to configure this one.

Luckily for us we can always check the values defined in our [helm chart](https://github.com/kubernetes/ingress-nginx/blob/master/charts/ingress-nginx/values.yaml), which shows that we can still utilize the [prometheus annotations](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus).

Therefor we upgrade our Helm chart through the following command:

```bash
helm upgrade ingress-controller ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.metrics.enabled=true \
  --set-string controller.podAnnotations."prometheus\.io/scrape"="true" \
  --set-string controller.podAnnotations."prometheus\.io/port"="10254"
```

When we validate this through the following command

```bash
helm get values ingress-controller --namespace ingress-nginx
```

We can see that our values are now set as they should:

```bash
controller:
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
```

When I now open the Prometheus dashboard (remember: `kubectl get nodes` and get the external IP address, and open this with the prometheus port shown through `kubectl get svc -A`) and start typing `ng` it shows our metrics!

![](./prometheus.png)

## Installing Grafana

From here on, it's a smooth ride to the finish line! We can simply follow the [official documentation](https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/) and install grafana with:

```bash
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/
```

Once this is done, we do the same as for prometheus and open the grafana dashboard in our browser (`kubectl get nodes`; `kubectl get svc -A`) where we can use `admin:admin` as our credentials and run the following steps:

1. Click "Add data source"
2. Select "Prometheus"
3. Enter the details (note: I used `http://CLUSTER_IP_PROMETHEUS_SVC:9090`)
4. Left menu (hover over +) -> Dashboard
5. Click "Import"
6. Enter the copy pasted json from https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/grafana/dashboards/nginx.json
7. Click Import JSON
8. Select the Prometheus data source
9. Click "Import"

![](./grafana.png)

## Conclusion

In this article we learned how we can now start monitoring our earlier application its ingress controller. The goal of this is to in a next post be able to auto-scale the created infrastructure based on the incoming requests!

Monitoring the Kubernetes Nginx Ingress Controller with Prometheus and Grafana


Whenever you are creating an application that you want to expose to the outside world, it's always smart to control the flow towards the application behind it. That's why Kubernetes has something called `Kubernetes Ingress`. But what is it?

## Kubernetes Ingress

[Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) allows you to expose HTTP and HTTPS routes from outside the cluster to services within the cluster. The traffic routing is then controlled by rules defined in the ingress sources.

For this article, I will explain how you can get started on creating your own `Nginx Ingress Controller`. Of course this is not the only possibility, [so feel free to check other ingress controllers such as Istio, HAProxy, Traefik, ...](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/)

Some advantages of using an ingress controller:

* Rate limiting, Timeouts, ...
* Authentication
* Content based routing

## Sample Hello World application

Before we create our controller, let's get started on creating a simple demo application. The only thing our application will do is process the HTTP request, wait a couple of seconds and return a "Hello World" response.

### Creating our sample app

I decided to create this application in Node.js. So if you have `npm` and `node` installed, run the following commands:

```bash
npm init -y
npm i express --save
```

Whereafter you can create an `index.js` file with the following content:

```javascript
const express = require('express')
const app = express()
const port = 3000

app.get('/', async (req, res) => {
  console.log('Got request, waiting a bit');
  await delay(15 * 1000);
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`)
})

const delay = async (timeout = 1000) => {
  return new Promise((resolve, reject) => setTimeout(resolve, timeout));
}
```

### Packaging it as a container

Since everything is created in terms of application code, we can package it all up into a Docker container by creating a Dockerfile:

**Dockerfile**

```
FROM node:latest

WORKDIR /usr/src/app

# Install deps
RUN apt-get update

# Create Certificate
RUN apt-get install ca-certificates

# Install Package.json dependendencies
COPY package.json .
RUN npm install

# Copy Source Code
ADD . /usr/src/app

CMD [ "npm", "run", "start" ]
EXPOSE 3000
```

That we can build with (choose one for your use-case):

```bash
# Local build (for local use)
# Note: when using minikube, make sure to run `eval $(minikube docker-env)` to build images in minikube context
docker build -t local/node-sample-helloworld .

# Remote build (to push to docker repository)
docker build -t thebillkidy/node-sample-helloworld .
docker push thebillkidy/node-sample-helloworld
```

### Running it on Kubernetes

Once it is build, we can now run it on our Kubernetes cluster. For that we create a Deployment YAML file:

**kubernetes.yaml**

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: d-node-sample-helloworld
spec:
  selector:
    matchLabels:
      app: node-sample-helloworld
  replicas: 1
  template:
    metadata:
      labels:
        app: node-sample-helloworld
    spec:
      containers:
      - name: main
        image: thebillkidy/node-sample-helloworld:latest # if local, utilize local/node-sample-helloworld
        imagePullPolicy: Always # if local, utilize Never
        ports:
        - containerPort: 3000
```

That we can apply with `kubectl apply -f kubernetes.yaml` and should now show the following after running `kubectl get deployments -A`:

```bash
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
d-node-sample-helloworld   1/1     1            1           37s
```












































Kubernetes is getting more popular everyday and it's no wonder why! When you are running applications on-premise or in-cloud, the possibility of having the applications in a portable way is a strong one! Removing the friction for scaling-out your application when you are ready for it, or even bursting scenarios.

## Nginx Ingress Controller

We now have a simple Hello World application running, but it's only available internally! What we could do now is expose it through Kubernetes and a LoadBalancer, but let's actually utilize our Ingress Controller here! So let's get started creating this Ingress Controller.

### Installation

The first step that we should do is create the NGINX Ingress controller. For this we can follow these steps:

> Note: before the `stable/nginx-ingress` chart was utilized. But this is now deprecated!

```bash
# ref: https://github.com/kubernetes/ingress-nginx (repo)
# ref: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx (chart)

# 1. Create namespace
kubectl create namespace ingress-nginx

# 2. Add the repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# 3. Update the repo
helm repo update

# 4. Install nginx-ingress through Helm
helm install ingress-controller ingress-nginx/ingress-nginx --namespace ingress-nginx
```

Once we ran the above, we should now be able to access the ingress controller by loading the external IP (`kubectl -n ingress-nginx get svc`). 

> **Note:** when working with Minikube or others, we can utilize `kubectl port-forward svc/ingress-controller-ingress-nginx-controller --namespace ingress-nginx --address 0.0.0.0 8000:80` and access it on `http://localhost:8000`

We are now ready to expose our application!

### Exposing our Application

Once an ingress controller is created, we need to expose our application internally:
 
```bash
kubectl expose deployment d-node-sample-helloworld --name svc-node-sample-helloworld
```

and configure our Ingress controller to route traffic to it as defined in the [Kubernetes Ingress API](https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource). By creating a YAML file:

**ingress-node-sample-helloworld.yaml**

```yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-node-sample-helloworld
  annotations:
    # Target URI where the traffic must be redirected
    # More info: https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    # Uncomment the below to only allow traffic from this domain and route based on it
    # - host: my-host # your domain name with A record pointing to the nginx-ingress-controller IP
    - http:
        paths:
        - path: / # Everything on this path will be redirected to the rewrite-target
          backend:
            serviceName: svc-node-sample-helloworld # the exposed svc name and port
            servicePort: 3000
```

Which we apply with `kubectl apply -f ingress-node-sample-helloworld.yaml`

Now once this is applied, we should be able to execute a cURL request to access our application! So let's try this:

```bash
# Execute a GET request with the specified host and IP
# Note: the Host should match what is written in spec.rules.host
curl -k -X "GET" -H "Host: my-host" http://YOUR_IP
```

Or we can also open it in our browser and navigate to http://YOUR_IP

> **Note:** If this is not working, check `kubectl describe ing` and make sure that the configuration is correct

## Conclusion

In this article a demonstration was made on how you can set up your own ingress controller for Kubernetes. This is of course a small step in the entire chain of use cases, where most often you want to do more such as rate limiting, or even monitoring it.

[The next article](/post/infrastructure/kubernetes-nginx-ingress-controller-monitoring-prometheus) will explain more in-depth how you are able to start monitoring what we have just set-up through Prometheus and visualize all of it in Grafana.


Creating a Kubernetes Nginx Ingress Controller and create a rule to a sample application


For a new project I am working on, I want to utilize Dapr to integrate Kafka such that a REST endpoint is automatically called in my application.

Of course, we want our development environment to be as easy as possible and set-up in just a few minutes. Which is why I chose for the following:
* Minikube (for my Kubernetes wishes)
* Kafka through Helm
* Utilize: `dapr run ...` when testing my application for fast reboot.

So let's get started! 

## Dapr LOCAL installation

The entire purpose of our set-up is to run "difficult-to-setup" applications through Kubernetes, while keeping development easy through Dapr locally. Therefor we need to install Dapr locally and not in our Kubernetes cluster. 

```bash
wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
dapr init
```

> **Note:** For a more detailed setup guide, [follow this article](/post/2020/2/17/infrastructure-dapr-getting-started)

## Minikube Installation

First we start by installing Minikube through the following commands:

```bash
# Check if virtualization is supported (should NOT be empty)
grep -E --color 'vmx|svm' /proc/cpuinfo

# Download Minikube
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube

# Add it to your path
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

# Check installation
minikube status
```

Once that is done, we can **start up a Kubernetes cluster**

```bash
# Create a cluster with 4 cpus, 8GB RAM
minikube start --cpus=4 --memory=8196
```

## Installing Helm

A great package manager for us so we don't have to worry about anything anymore is `helm`. To install it we can run this quick command:

```bash
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
```

## Installing Kafka with Helm

Now we're ready for the real stuff! Let's get a kafka cluster installed on our Minikube:

### Creating Kafka

```bash
# Get Minikube IP
export MINIKUBE_IP=$(minikube ip)

# Install Kafka and run it on port 30090 external
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka bitnami/kafka --set externalAccess.enabled=true,externalAccess.service.type=NodePort,externalAccess.service.domain=127.0.0.1,externalAccess.service.nodePorts[0]='30090'
```

### Port Forwarding Kafka to Localhost

Networking wise, Kafka is running in an isolated network that we can't access from localhost. By creating an external access endpoint as shown above (helm install ... externalAccess.enabled=true) we can access this remotely. Now we just need to create a port-forward as shown below

```bash
# Expose kafka
# Note: we expose 9092 for initial connection and 30090 for consumer to broker discovery
kubectl port-forward svc/kafka-0-external 30090:9094 9092:9094
```

## Configuring Kafka bindings in Dapr

The hard part is done! We have `Kafka` set-up (which should have taken us around 5 minutes to do so) and now we can start running our applications. So let's configure our Input and Output bindings. But let's first have a small refresher on what those are:

* **Input Bindings:** Invokes a REST endpoint on your application
  * Example: if we configure `.metadata.name` to `kafka`, then Dapr will call `POST /kafka` on our application one a message appears in kafka
* **Output Bindings:** Invokes an external resource (e.g. you send to a dapr bindings endpoint, and it will send a message to Kafka)
  * Example: `curl -X POST http://localhost:3500/v1.0/bindings/kafka -d '{ "data": { "message": "Hi!" }, "operation": "create" }'`

With this in mind, create a file: `components/binding-kafka.yaml` with the following content:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka 
  namespace: default
spec:
  type: bindings.kafka
  metadata:
  - name: consumerGroup
    value: group1
  - name: authRequired
    value: "false"
  - name: brokers
    value: localhost:9092

  # Config Output Binding (invokes external resource)
  # This we can test by sending a CURL request
  # e.g. curl -X POST http://localhost:3500/v1.0/bindings/kafka -d '{ "data": { "message": "Hi!" }, "operation": "create" }'
  # note: start with `dapr run --port 3500` for the port
  - name: publishTopic # for output bindings (invokes external resource)
    value: stocks # comma separated string of topics for an input binding 

  # Config Input Binding (invokes app on endpoint from .metadata.name)
  - name: topics # for input bindings (trigger app on endpoint - see .metadata.name)
    value: stocks # comma separated string of topics for an input binding 
```

## Running

We're now ready to boot-up our application! Start your application as follows (where you application is listening on port 5000):

```bash
sudo dapr run --components-path ./components --port 3500 --app-port 5000 --app-id kafka -- dotnet run --urls=http://0.0.0.0:5000
```

If we now send an event through a cURL request to the **output binding**:

```bash
curl -X POST http://localhost:3500/v1.0/bindings/kafka -d '{ "data": { "message": "Hi!" }, "operation": "create" }'
```

We can see that events are coming in through the **input binding** if we log them in the console! 

![](./example.png)