
Autonomous Platforms are being deployed all around us, optimizing our daily processes and improving their efficiency and stability without the need of human intelligence. They learn by themself, iteratively improving themselves and staying ahead of the competition.

One of the biggest hurdles for Autonomous Platforms to be deployed is the access to a simulation environment that can be configured to the real-world use case and scaled to millions of instances to train on. Achieving 100s of iterations per second in training.

This is where the [Azure Project Bonsai platform](https://www.microsoft.com/en-us/ai/autonomous-systems-project-bonsai) comes into the picture, helping customers to teach their machines to achieve super-human optimization.

Through the following articles, I introduce you to how you can get started yourself with this amazing topic:

* [1. Platform Introduction & Overview](/post/ai/rl/bonsai/introduction) (+ [update!](/post/ai/rl/bonsai/introduction-revisiting))
* [2. Create and integrate your first "Brain" - CartPole Problem](/post/ai/rl/bonsai/brain-integration)
* 3 Create your own Simulation Environment from scratch - Self-Balancing Cube Problem [Coming Soon]
* 4 3D Model and Teach your own Robot to Walk - Custom Robotics Project [Coming Soon]

Learn how you can create an autonomous system through Machine Teaching with the Azure Project Bonsai platform.

Machine Teaching through Azure Project Bonsai - Series Overview

In a [previous article](/posts/infrastructure/azure-arc) I went in detail on how you can get started running your own Azure Arc setup and connect your cross-cloud and on-premises systems and Kubernetes clusters. In this article I would like to go in more detail how you are now able to utilize the previously created infrastructure and run Azure App Service on it! Bringing the ecosystem of Azure towards your multi-cloud environment.

## Overview

I'd love to explain the above through a graphic as illustrated below. Here you have several components:
* Multi-Cloud / Hybrid Cloud environment (illustrated as "Hosting" below)
* Azure Application Services with Azure Arc (the top block)

Making it clear that Arc is there to enable us to run Azure Application Services across our cloud environments, no matter the infrastructure layer beneath it. 

![](./arc-overview.png)

## Hands-On

After the above, let's actually do this ourselves! To deploy an Azure Service we need to have an Azure Arc enabled cluster or server available and execte the following steps:

1. Enable an Extension (e.g. App Service)
2. Create a Custom Location (our own "cloud" location!)
3. Create an App Service

### Enable an Extension (App Service)

First we start by enabling an extension on the target cluster. For this we open up the cluster in Azure Arc, navigate to "Extensions" and find the extension we want to install. In our case we will install the App Service extension.

![](./azure-arc-cluster-extensions.png)
![](./azure-arc-cluster-extensions-add-1.png)
![](./azure-arc-cluster-extensions-add-2.png)

Now we get to the details pane, where a public IP is required. This is the static IP created when you create the Kubernetes Cluster, to find this in GCP we can simply navigate to our GKE instance and locate the public IP endpoint.

![](./azure-arc-cluster-extensions-add-5-enter-details.png)
![](./azure-arc-cluster-extensions-add-3-gcp-public-ip-finding.png)
![](./azure-arc-cluster-extensions-add-4-gcp-public-ip-finding.png)

All details are now entered and a script will be generated for us. Execute this by copying this to a PowerShell instance of your choice that has access to the cluster and AZ CLI.

![](./azure-arc-cluster-extensions-add-6-run-script.png)
![](./azure-arc-cluster-extensions-add-7-copy-script-to-az-shell.png)

### Create a Custom Location

Since the Extension is now enabled, we are able to create a custom location. Navigate to Azure Arc "Custom Locations" and click the "Add" button.

![](./azure-arc-custom-location-overview.png)

This will open up the interface to add a custom location. Just navigate through this, selecting the settings you require.

![](./azure-arc-custom-location-create.png)
![](./azure-arc-custom-location-create-2.png)
![](./azure-arc-custom-location-create-3-created.png)

Once this has been created, we can verify that it has been created by navigating to a `kubectl` shell and checking the deployed pods. We will see a new namespace being created containing the resources.

![](./azure-arc-custom-location-create-4-created-gcp-overview.png)

### App Service Creation

We are almost there! So far we created a App Service enabled Kubernetes cluster on GCP and added the GCP Kubernetes cluster as a target location for Azure App Service instances. Finally we need to deploy App Service on it! 

Open Azure Arc and navigate towards "App Service" on the left, creating it in the previously created custom location with the default image.

![](./azure-arc-appsvc-1.png)
![](./azure-arc-appsvc-2.png)
![](./azure-arc-appsvc-3.png)
![](./azure-arc-appsvc-4.png)
![](./azure-arc-appsvc-5-deployed.png)

**Congratulations**, App Service is now up and running on your own instance! To verify it, open up the GKE console again and verify that an extra pod is running.

![](./azure-arc-appsvc-6-gcp-confirm.png)



## Conclusion

As shown in this article, it is trivial to deploy your own Azure Services across your existing hardware! This opens up a lot of scenarios where customers can connect their existing infrastructure running across multiple clouds and easily managing it from an easy to understand portal! Personally, I am a big fan of this solution and cannot wait for it to expand and improve more. Some of my personal learnings:

* Some errors exist and might be difficult to work with (e.g. I had a stamp issue as well as a Storage Class not existing, requiring some manual intervention - which can be automated). Tickets are being raised and the service is maturing more and more (it is still in preview!)
* I would love to see more automation happening. Currently a lot of the commands have to be ran manually, but it is prone to making mistakes.

In any case, if you have existing infrastructure and want a good control plane, Azure Arc is definitely a promising service to check out!

Learn how you can deploy Azure App Service across your Multi-Cloud environment on GCP through the Azure Arc control plane

Run Azure App Service on GCP through Azure Arc


Azure Arc was released a while ago, but got a great update in May 2021 during the [annual Build conference](https://azure.microsoft.com/en-gb/blog/build-cloudnative-applications-that-run-anywhere/?WT.mc_id=DT-MVP-5004263) allowing it to run Azure Application services across your Azure Arc enabled Kubernetes services.

## Introduction

In this article, I will explain how you can get started with Azure Arc. As **Prerequisites** I created the following on each cloud provider and on-premise infrastructure.

* Azure
    * Server
    * Kubernetes (AKS)
* Google GCP
    * Server
    * Kubernetes (GKE)
* On-Premises (VMWare)
    * Server
    * 3-Server Nodes combined to Kubernetes through K3S ([see this article on how you can set this up](/post/infrastructure/nvidia-jetson-k3s))

## Azure Arc

To summarize Azure Arc in my own words: "it's a management plane for your Kubernetes Clusters and Phyiscal servers that allows you to manage your machines no matter where they sit".

In Azure terms: "Azure Arc offers simplified management, faster app development, and consistent Azure services. Easily organize, govern, and secure Windows, Linux, SQL Server and Kubernetes clusters across data centers, the edge, and multicloud environments right from Azure."

Personally, I find this an amazing tool with a low cost to manage your entire datacenter and cloud estate! 

## Creating an Azure Arc connected Cloud

So let's get started and create our clusters and servers! Each cloud has their own specificalities, but I would love to show you how you can create them on each one of them. The overall process is quite similar:

* **Server**
    * Login an configure the Azure Arc agent

* **Kubernetes**
    * Fetch the kubectl access
    * Configure Azure Arc with `az cli`

### Azure Arc Connect Server

To connect a server, we navigate to Azure Arc in the Azure Portal 

![](./azure-arc-get-started.png)

In there we can click on Servers which will open up the overview of our servers. On the Left top we can now click "add" to add a server.

![](./azure-arc-servers-overview.png)

Seeing that the process is quite similar, I won't go into detail of how you should do this for the different cloud providers. To connect your server, open op the "Add" page and navigate through the steps while selecting your resource group operating system.

![](./azure-arc-servers-connect-3.png)
![](./azure-arc-servers-connect.png)
![](./azure-arc-servers-connect-4.png)

Once you have done that you need to install the agent. In our case I was running Linux as an operating system. At this step we will now see the commands we should run to get Arc up and running

> 💡 Do not forget to switch to root with `sudo su`

![](./azure-arc-servers-connect-5-install-agent.png)
![](./azure-arc-servers-connect-6-install-agent-final.png)
![](./azure-arc-servers-connect-7-configure-agent-complete.png)

Congratulations 🎉 you have now connected your server to Azure Arc!

### Azure Arc Connect Kubernetes

We already have single servers configured, but how can we configure our Kubernetes clusters? Well this is a bit more tricky. For this we can follow these steps:

1. Install Kubernetes (managed with AKS or GKE or through k3s for example)
2. Install & Configure Kubectl
3. Install Helm
4. Run AZ CLI

> 💡 You can make things easier (as I did) by utilizing the Azure Cloud Shell and configure `kubectl` in it. If you are working with GKE, just copy the `~/.kube/config` file content from GKE to Azure Cloud Shell and you can run `kubectl get pods -A` in your Azure Cloud Shell to connect to your GKE Kubernetes 😉

Showing this example on GKE (as Azure is straightforward with the shell), let's open up the Kubernetes pane and click "Add" again. We will now have to enter the subscription again, plus a cluster name (metadata) to name it.

![](./azure-arc-k8s-connect.png)
![](./azure-arc-k8s-connect-gcp-1.png)

Now we will be presented by the commands we need to execute to connect this Kubernetes cluster. Again, **remember to have kubectl configured** to connect to your cluster, then everything should be going smoothly!

![](./azure-arc-k8s-connect-gcp-2.png)

As explained before, I opened up the Kubectl configuration and copied the GKE config towards the Azure Cloud Shell one (to skip installation of AZ CLI on GCP):

![](./azure-arc-k8s-connect-gcp-3.png)
![](./azure-arc-k8s-connect-gcp-4.png)

Once this has been pasted I verify to ensure `kubectl get pods -A` is working and showing my GKE pods.

![](./azure-arc-k8s-connect-gcp-5.png)

Since that has been verified, I am now able to run the required commands and get cluster configured to Azure Arc.

![](./azure-arc-k8s-connect-gcp-6.png)
![](./azure-arc-k8s-connect-gcp-8.png)

We are now connected to Azure Arc, but let's for fun check on GKE to see our pods running (including the Azure Arc pods) in the Google Cloud Shell:

![](./azure-arc-k8s-connect-gcp-7-verify.png)


## Final Overview

Looking at our final overview, this is what we have created, a beautiful server park consisting out of our Kubernetes clusters and Servers!

![](./azure-arc-final-overview-servers.png)

![](./azure-arc-final-overview-k8s.png)

## Conclusion

In this article I went in more details on how you can get started with Azure Arc and connect your existing Servers and Kubernetes clusters, no matter where they are running! 

Learn how you can manage your own Multi-cloud environment with Azure Arc and set it up across Azure, Google GCP and On-Premise

Create a Multi-Cloud Infrastructure with Azure Arc


I recently acquired an Nvidia Jetson Nano 2GB Developer Kit and had some fun with it! In this series I present the overview of the articles I created:

* [Setting up an Nvidia Jetson Nano Node](/post/infrastructure/nvidia-jetson-setup)
* [Creating your own Nvidia Jetson Cluster](/post/infrastructure/nvidia-jetson-cluster)
* [Deploying Kubernetes on your Nvidia Jetson Nano Cluster](/post/infrastructure/nvidia-jetson-k3s)
* [Adding Dapr and Grafana to the Nvidia Jetson Nano Kubernetes Cluster](/post/infrastructure/nvidia-jetson-k3s-dapr-and-grafana)


Learn how to configure the Jetson Nano, Deploy a Kubernetes server and finally monitor it with Grafana

An Nvidia Jetson Nano Series


Since Kubernetes has now been installed, we can continue to run a demonstration! For this demonstration we are going to deploy the following:

1. [Dapr](https://dapr.io) as our Distributed Application Runtime
2. Deploy an Hello World Example
3. [Grafana](https://grafana.com)

So let's get started!

## Installing Dapr

To install Dapr we can utilize its `install.sh` script which we execute:

```bash
wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash
```

Whereafter we can run `dapr --version` to validate that it is installed:

```bash
# Output of dapr --version
CLI version: 1.2.0
Runtime version: n/a
```

Finally, we configure Dapr on our Kubernetes cluster by initializing it

```bash
dapr init -k
```

Which we again verify by running `dapr status -k` 

```bash
# Output of dapr status -k
NAME                   NAMESPACE    HEALTHY  STATUS   REPLICAS  VERSION  AGE  CREATED
dapr-sentry            dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-operator          dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-dashboard         dapr-system  True     Running  1         0.6.0    33s  2021-05-30 11:27.57
dapr-sidecar-injector  dapr-system  True     Running  1         1.2.0    33s  2021-05-30 11:27.57
dapr-placement-server  dapr-system  True     Running  1         1.2.0    32s  2021-05-30 11:27.58
```

## Installing a Demo Application

kubectl create deployment kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1

## Grafana Dashboard

Installing Grafana is a bit more complicated as it requires a store to gather our telemetry. For this we will be utilizing Prometheus.

### Install Prometheus

For configuring Prometheus, we utilize the commands below that will set it up with a persistent volume.

> 💡 To disable persistent volumes, append `--set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false` to the end of the `helm install` command.

```bash
# Create a dedicated namespace
kubectl create namespace dapr-monitoring

# Install Prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring

# Verify that it is running
kubectl get pods -n dapr-monitoring -w
```

### Install Grafana

Since we have our store set-up, we can now install our dashboard

> 💡 We can again disable persistent volumes by appending `--set persistence.enabled=false` to the `helm install` command

```bash
# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n dapr-monitoring

# Verify that it is running
kubectl get pods -n dapr-monitoring -w
```

### Grafana Admin Password

Once Grafana is up and running, gather its admin password by running the command below

```bash
kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

### Access Grafana

Now Grafana is up and running, we can access it by port forwarding its port ot our [local system](/nvidia-jetson-k3s#Installing-and-Configuring-Kubectl).

```bash
# Forward Grafana from our cluster to localhost
kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring

# Open browser at:
# http://localhost:8080/
```

We can now login with:

* **Username:** `admin`
* **Password:** See Grafana Admin Password (`kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo`)

### Configure Grafana with Prometheus

Finally we need to set-up our Grafana Dashboard. Luckily for us, Dapr provides some [out-of-the-box dashboards](https://github.com/dapr/dapr/tree/master/grafana)

To start, we load the dashboard and add our Prometheus Server

![](./grafana.png)

![](./grafana-prometheus.png)

As a URL we fetch the Dapr Prometheus Server url by viewing the services and finding the name of the deployment. Which will become `<DEPLOYMENT_NAME>.<NAMESPACE>`

```bash
kubectl get svc -n dapr-monitoring
```

![](./grafana-prometheus-server.png)

In our case this becomes `http://dapr-prom-prometheus-server.dapr-monitoring`

![](./grafana-configure.png)

Now we can `Save & Test` and go towads the import screen, where we import the dashboard that we copied from ]Dapr its GitHub page.](https://raw.githubusercontent.com/dapr/dapr/master/grafana/grafana-system-services-dashboard.json)

![](./grafana-import.png)
![](./grafana-import-2.png)

If everything is done correctly, we can now see our monitoring

![](./grafana-final.png)

Learn how you can install Dapr and Grafana on your own Kubernetes Cluster powered by Nvidia Jetson Nanos

Installing Grafana on your Kubernetes Cluster


We now have [a beautiful cluster assembled and running](/post/infrastructure/nvidia-jetson-cluster), but no software running on it yet. So let's get started and provision Kubernetes on top of it so we can manage our nodes easier!

## Introducing K3s

For our Kubernetes setup we want a lightweight Kubernetes installation seeing the required Edge computation. For this, we can utilize [K3S](https://k3s.io/) which is a distribution built specifically to run on IoT & Edge devices.

> Personally, the installation feels similar to how we installed and configured Docker Swarm before. Install it everywhere and join nodes to the master - quick and easy! 

## Installing K3s

Installing K3S is super trivial! There is however an easier way if you want to automate all of it, by utilizing the CLI and adding just 1 parameter.

### Master Node Setup

To configure our Master Node, we execute the following command that will pass the secret we define:

```bash
curl -sfL https://get.k3s.io | sh -s server --cluster-init --token "XAVIER_HIS_SECRET_TOKEN" --write-kubeconfig-mode 644
```

> 💡 As you may have noticed, we are utilizing `--write-kubeconfig-mode 644` to allow our cluster to be imported in rancher later.

### Worker Node Setup

Finally, we can add our Worker (or Agent) nodes by running the following line on each of the nodes:

```bash
curl -sfL https://get.k3s.io | sh -s agent --server https://MASTER_NODE_IP_ADDRESS:6443 --token "XAVIER_HIS_SECRET_TOKEN"
```

### Verification

When we now execute `sudo k3s kubectl get nodes -w` on our master node, we will see our nodes appearing.

![](./k3s-node-setup.png)

## Installing and Configuring Kubectl

> ⚠ Execute this on your local machine! (WSL in this case)

### Installing Kubectl

Since we now have a Kubernetes cluster running, it is time to configure it to run on our local system. For that I am utilizing the `WSL` or `Windows Subsystem for Linux` system as explained in a [previous article](https://xaviergeerinck.com/post/coding/python/windows-development-environment-setup).

To install WSL we execute the following:

```bash
# Download Kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Install it
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Verify
kubectl version --client
```

### Configuring Kubectl

Now this is installed, we can configure it towards our remote cluster:

```bash
# Copy over the Kubeconfig file to our local machine
scp xavier@xavier-nvidia-node-1.local:/etc/rancher/k3s/k3s.yaml nvidia-node-1-config.yaml

# Replace localhost with the location of the master node
sed -ie s/127.0.0.1/xavier-nvidia-node-1.local/g nvidia-node-1-config.yaml

# Verify that it works
kubectl --kubeconfig nvidia-node-1-config.yaml get pods -A

# Then copy it over to kubectl directory and activate it
# we can now run kubectl get pods -A
cp nvidia-node-1-config.yaml ~/.kube/config
```

## Troubleshooting

### Agent does not want to join

Sometimes an agent does not want to join, in that case try utilizing the IP address directly.

### Removing K3S

⚠ In case something happened, you can uninstall K3S with the following commands:

```bash
# Uninstall Server Node
/usr/local/bin/k3s-uninstall.sh

# Uninstall Agent Node
/usr/local/bin/k3s-agent-uninstall.sh
```

## Conclusion

We have now set-up our Nvidia Jetson nodes and are running a Kubernetes cluster on it! To keep this article light, I have decided to move the demonstration to a new article where you will learn to [run Dapr as your distributed application runtime as well as Grafana to monitor your nodes](/post/infrastructure/nvidia-jetson-k3s-dapr-and-grafana)

Learn how you can install Kubernetes on a small GPU Cluster for edge computation purposes

Installing Kubernetes on an Nvidia Jetson Cluster


[In my previous article](/post/infrastructure/nvidia-jetson-setup) I explained how we can get one single Nvidia Jetson Nano 2GB Node up and running. In this article I would love to go in more detail on how we can create a cluster.

## Requirements

To create my cluster, I utilized the materials below:

> **Note:** These requirements were created for 3 Nodes and receive internet through a Mobile Pocket router that can be put in Client mode to connect LAN over WiFi (very useful for portability).

* 3x [Jetson Nano](https://developer.nvidia.com/embedded/jetson-nano-2gb-developer-kit) ~195 Eur Total
* 1x [Switch](https://eu.dlink.com/be/nl/products/dgs-105-5-port-gigabit-ethernet-switch) ~19.95 Eur
* 1x [Standoffs](https://www.amazon.nl/gp/product/B0778JRXPS) ~8.99 Eur
* 1x [Mobile Router](https://www.tp-link.com/nl/home-networking/wifi-router/tl-wr902ac/) (it is end of life, but super cheap and still working!) ~34.15 Eur
* 4x 0.5m UTP Cables (I had those, but typically ~ 4eur for 4)
* 4x [0.3m USB C to USB A cables](https://www.blitzwolf.com/BlitzWolf-BW-TC13-PVC-3A-USB-Type-C-Charging-Data-Cable-1ft0.3m-22-AWG-Power-+-30-AWG-Data-Wire-Cores-p-276.html) ~19.68 Eur
* 1x [6 port USB Hub](https://m.blitzwolf.com/60W-QC3.0-6-Ports-Charger-Adapter-p-450.html) ~18.87 Eur
* 4x [128GB SD Card](https://www.samsung.com/be/memory-storage/memory-card/evo-plus-microsd-card-with-sd-adapter-128gb-mb-mc128ga-eu/) ~58Eur Total (minimum should be 32GB)

**Total Bill:** 358.64 Eur (or 163.64 Eur without the Nvidia Jetsons)

## While Assembling

To make efficient use of our time, we can flash the SD Cards with the Operating System as explained [before](/post/infrastructure/nvidia-jetson-setup). They take around 20 minutes each.

## Assembling

While the SD Cards are flashing, we can get started with assembly! This is quite straightforward, so mount the nodes on top of the switch and connect them through the UTP cables. Then take the Mobile Pocket Router and connect that to the first port of the switch.

![](./node-standoffs.jpg)

![](./nodes-standoffs-stacked.jpg)

![](./switch.jpg)

![](./nodes-ethernet.jpg)

Once this is done, we can provide power by connecting the nodes and the mobile Router to our USB Hub. Finally, power on the switch and the USB Hub to get started!

![](./cluster-assembled.jpg)

## Connecting to your Nvidia Jetson Nanos

Once the Nvidia Jetson Nanos have been set-up, you can quite easily detect them by installing [Angry IP Scanner](https://angryip.org/download/#windows) and scanning your home network. This will show the nodes and their respective IP Addresses.

![](./ip-scan.png)

## Configuring Windows Terminal

As mentioned earlier, you can easily login on all the nodes through a shortcut and entering the command below:

```bash
CMD /C wt.exe wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-1.local"; split-pane -H wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-2.local"; split-pane -V wsl -d "Ubuntu-20.04" -- bash -c "ssh xavier@xavier-nvidia-node-3.local"
```

## Conclusion

With a small bill we have built our own GPU Cluster that has a combined 384 Cores and 1.416 TFLOPS! (which is similar to 1 Jetson TX2)

![](./cluster-finalized.jpg)

Learn how you can create your own Nvidia Jetson Nano GPU Cluster

Creating an Nvidia Jetson Nano Portable Cluster


I wanted to analyze some data in Azure Synapse Analytics, Serverless SQL. This data was coming in from my IoT Hub that I created and was being routed towards an Azure Data Lake Storage account as small JSON files. However when trying to analyze this data, it appeared that IoT Hub shows the body as Base64 encoded and T-SQL couldn't analyze it.

## Problem Statement

Normally, to convert base64 towards a string, we can use a built-in method where it converts this as XML. The following SQL statement can then be used:

```sql
SELECT
    [JSON] = cast(cast('' as XML).value('xs:base64Binary(sql:column("StackOverflow.Base64JSON"))', 'varbinary(30)') as varchar(30))
FROM my_table
```

However, when executing this in Synapse Analytics, Serverless SQL we get the following error:

```
xml is not supported
```

## Solution

To solve this, we need to execute the following steps:

1. Create a User Database ([we don't have rights on the master database](https://docs.microsoft.com/en-gb/azure/synapse-analytics/sql/resources-self-help-sql-on-demand?WT.mc_id=DP-MVP-5004263#create-statement-is-not-supported-in-master-database))
2. Set the correct active Database
3. Create a [Base64 Decoder](https://gist.github.com/XavierGeerinck/354b7049ee12696ed92644acec362d77)
4. Load our JSON data
5. Decode our Column

Breaking this down

### 1. Create a User Database

```sql
CREATE DATABASE MY_DATABASE;
```

### 2. Setting the correct active database

```sql
USE MY_DATABASE;
```

### 3. Creating the Base64 Decoder

```sql
CREATE FUNCTION dbo.tvf_Base64Decoder( @encodedPayload VARCHAR(8000) )
RETURNS TABLE
AS
RETURN WITH cte8Rows AS 
    (   
        -- generating 4096 rows, here are the first 8
        SELECT  ones = 1
        FROM    
            (
                values  (1),(1),(1),(1)
                    ,   (1),(1),(1),(1)
            ) AS tens(ones)
    )
    ,   cte64Rows AS 
    (   
        -- generating 4096 rows, here are the first 64
        SELECT  ones = 1
        FROM    cte8Rows       AS eight1 
        CROSS   JOIN cte8Rows  AS eight2
    )
    ,   cteSextet as
    (   
        -- a Base64 encoded string is comprised of 4-byte sextets, create an iterator of starting offsets for each sextet
        SELECT  top ((len(@encodedPayload) / 4)) 
                setextOffset = ((ROW_NUMBER() OVER (ORDER BY (SELECT 1))) * 4) -3
        FROM    cte64Rows      AS sixtyfour1 
        CROSS   JOIN cte64Rows AS sixtyfour2
    )
    ,   cteMapping as
    (   
        -- Base64 mapping table
        SELECT  Base64.[value]
            ,   Base64.[char]
        FROM 
            (   
                values  ( 0, 'A'), ( 1, 'B'), ( 2, 'C'), ( 3, 'D'), ( 4, 'E'), ( 5, 'F'), ( 6, 'G'), ( 7, 'H'), ( 8, 'I'), ( 9, 'J'), (10, 'K'), (11, 'L'), (12, 'M'), (13, 'N'), (14, 'O'), (15, 'P')
                    ,   (16, 'Q'), (17, 'R'), (18, 'S'), (19, 'T'), (20, 'U'), (21, 'V'), (22, 'W'), (23, 'X'), (24, 'Y'), (25, 'Z'), (26, 'a'), (27, 'b'), (28, 'c'), (29, 'd'), (30, 'e'), (31, 'f')
                    ,   (32, 'g'), (33, 'h'), (34, 'i'), (35, 'j'), (36, 'k'), (37, 'l'), (38, 'm'), (39, 'n'), (40, 'o'), (41, 'p'), (42, 'q'), (43, 'r'), (44, 's'), (45, 't'), (46, 'u'), (47, 'v')
                    ,   (48, 'w'), (49, 'x'), (50, 'y'), (51, 'z'), (52, '0'), (53, '1'), (54, '2'), (55, '3'), (56, '4'), (57, '5'), (58, '6'), (59, '7'), (60, '8'), (61, '9'), (62, '+'), (63, '/')
            ) AS Base64([value], [char])
    )
    ,   cteDecode as
    (   
        -- the fun stuff! (see explaination above)
        SELECT  chars =	concat(
            char(
                cast(cast(((SELECT m.[value] FROM cteMapping AS m WHERE m.[char] = SUBSTRING(@encodedPayload, s.setextOffset   , 1) collate Latin1_General_CS_AS) *  4) AS BINARY(1)) AS TINYINT) 
                | 
                cast(cast(((SELECT m.[value] FROM cteMapping AS m WHERE m.[char] = SUBSTRING(@encodedPayload, s.setextOffset +1, 1) collate Latin1_General_CS_AS) / 16) AS BINARY(1)) AS TINYINT)
            )                  
            , char(
                cast(cast(((SELECT m.[value] FROM cteMapping AS m WHERE m.[char] = SUBSTRING(@encodedPayload, s.setextOffset +1, 1) collate Latin1_General_CS_AS) * 16) AS BINARY(1)) AS TINYINT) 
                | 
                cast(cast(((SELECT m.[value] FROM cteMapping AS m WHERE m.[char] = SUBSTRING(@encodedPayload, s.setextOffset +2, 1) collate Latin1_General_CS_AS) /  4) AS BINARY(1)) AS TINYINT)
            )
            , char(
                cast(cast(((SELECT m.[value] FROM cteMapping AS m WHERE m.[char] = SUBSTRING(@encodedPayload, s.setextOffset +2, 1) collate Latin1_General_CS_AS) * 64) AS BINARY(1)) AS TINYINT) 
                | 
                cast(cast(((SELECT m.[value] FROM cteMapping AS m WHERE m.[char] = SUBSTRING(@encodedPayload, s.setextOffset +3, 1) collate Latin1_General_CS_AS) /  1) AS BINARY(1)) AS TINYINT)
            )
        )
        FROM    cteSextet AS s
    )
    SELECT  decodedPayload = string_agg(chars, '')
    FROM    cteDecode;
```

### 4. Loading our JSON Data

```sql
SELECT
    JSON_VALUE(jsonContent, '$.Body') AS jsonBody
FROM
    OPENROWSET(
        BULK 'https://myadlsaccount.dfs.core.windows.net/myadlscontainer/myadlsfolder/*/*/*/*/*/*.json',
        FORMAT = 'CSV',
        FIELDQUOTE = '0x0b',
        FIELDTERMINATOR ='0x0b',
        ROWTERMINATOR = '0x0b'
    )
    WITH (
        jsonContent varchar(MAX)
    ) AS [result]
```

### 5. Decoding the Column

```sql
WITH decoder AS (
    SELECT
        JSON_VALUE(jsonContent, '$.Body') AS jsonBody
    FROM
        OPENROWSET(
            BULK 'https://myadlsaccount.dfs.core.windows.net/myadlscontainer/myadlsfolder/*/*/*/*/*/*.json',
            FORMAT = 'CSV',
            FIELDQUOTE = '0x0b',
            FIELDTERMINATOR ='0x0b',
            ROWTERMINATOR = '0x0b'
        )
        WITH (
            jsonContent varchar(MAX)
        ) AS [result]
)
SELECT 
    decoded = (SELECT decodedPayload FROM dbo.tvf_Base64Decoder(jsonBody))
FROM decoder
```

## Conclusion

When you now combine those 5 steps into one SQL file, you will get a decoded JSON string containing your IoT Hub Body!

> ⚠ I personally would recommend utilizing Spark, seeing that this method works, but is not the fastest!

Did you ever had to decode base64 (e.g. when it came from IoT Hub as encoded Body)? Learn how to decode this in Synapse Analytics Serverless SQL instantly!

Decode Base64 with Azure Synapse Analytics its Serverless SQL


Dapr is simply amazing, allowing you to interact with libraries that otherwise take a couple of hours to implement or even days depending, on your level of comfort with them. Next to that, when you would utilize libraries, you also need to take care of correct authentication, error handling, and more.

## Introduction

Dapr offers all of the above out of the box, removing the need to write code that interacts explicitly with each library. Instead, add a components file, and we are ready to go.

In my previous blog articles, I covered:
* [Creating an Account Microservice](/dapr/creating-account-microservice)
* [Creating an Email Microservice](/dapr/creating-email-microservice)
* [Integrating Dapr with IoT Edge](/dapr/iot-edge)
* [Adding a Kafka Binding](/dapr/kafka-bindings)

One disadvantage so far has been that I am primarily working in full-stack development, utilizing Javascript as my primary to-go-to language. On this side, Dapr has lacked so far its current SDK implementation only offers gRPC out of the box with documentation that could use some work.

This is of course understandable seeing the issue they are trying to solve is a pretty complicated one and creating SDKs for each one of them is not a trivial task. Which is why **I would love to help!**

## Introducing the Dapr-JS SDK

The past couple of months I have been working on a new Dapr JS SDK that is easy to use for Typescript developers! After a lot of work on this and utilizing it myself in production for one of my projects and developing for the [other](https://roadwork.io), I am now happy to announce that I am now confident enough that the public can use it!

**⚡*I am proud to announce the first release of the new Dapr JS SDK!*⚡**

> Note: at the current time of writing, this is still hosted on my Repository, but I am working with Dapr team to move this towards the official repository.

## Example - Invoking Across Microservices

A major point I want to focus on through this API is the easiness of use. Removing the need of going to the Dapr API reference documentation all the time, such that you as a developer, only have to focus on utilizing the SDK for your application.

As an example, we can take 2 Microservices `my-server` and `my-client`, where `my-client` wants to invoke a `/hello-world` endpoint on the `my-server` application. Through this SDK you can do the following:

### Server

Start with `dapr run --app-id my-server --app-port 5000 npm run start`

```javascript
import Dapr, { HttpMethod, HttpStatusCode, Req, Res } from "@roadwork/dapr-js-sdk/http";
const client = new Dapr("127.0.0.1", 3500, 4000);
await client.invoker.listen("method", async (req: Req, res: Res) => console.log(req), { method: HttpMethod.POST });
```

### Client

Start with `dapr run --app-id my-client --app-port 5000 npm run start`

```javascript
import Dapr, { HttpMethod, HttpStatusCode, Req, Res } from "@roadwork/dapr-js-sdk/http";
const client = new Dapr("127.0.0.1", 3500);
await client.invoker.invoke("my-server", "method", HttpMethod.POST, { hello: "world" });
```

### Demo

![](./demo.gif)

> Check out the [source code here](https://github.com/Roadwork/dapr-js-sdk/tree/main/examples/hello-world-distributed)

## Future

Of course, Rome was not built in a day, and neither is this SDK. Most of the APIs are covered, but they have to be maintained to stay up-to-date with the upstream Dapr references. Next to that, the current state focuses mainly on the HTTP interaction side of things, while Dapr also offers gRPC support. For me, this is a crucial must-have since it would make interactions way faster, but it should keep the current API definitions that were put in place.

To summarize the focus points:

* Maintain with upstream API Reference
* Implement gRPC under `@roadwork/dapr-js-sdk/grpc`
* Find additional contributors

If you'd love to help out, feel free to reach out to me on Twitter or Discord (Dapr Server)!

Learn how you can utilize the Dapr-JS SDK to interact with Dapr

Xavier Geerinck

Did you enjoy reading? Or do you want to stay up-to-date of new Articles?