
In Belgium we every year celebrate the coming of "Sinterklaas" on the 6th of December, a celebration for children where this mythologic figure delivers presents for all the children that behaved well that year. (see: https://en.wikipedia.org/wiki/Sinterklaas for more information). To follow this tradition, many companies hold internal events where children of the employees are invited to participate in all kinds of events. I was asked to participate in one of these events and come up with a workshop for children of all ranges - being divided in 2 groups (the very young ones, and the somewhat older ones).

## The Idea

The foundation of any idea starts with finding a base reasoning that will act as a guideline throughout the idea itself. Which in this case was that I want to spike the interest for technology in these childen, so that one day they might be interested in enrolling into technology studies 

> This is also often referred to as STEM (Science, Technology, Engineering and Mathematics). Which is an ongoing initiative world-wide to stimulate children chosing for these education areas

After some brain storming the idea popped up of creating a sort of game where the children would have to match "dolls" together with "stands" (that all almost look the same) through the use of technology. In this case the technology being a simple nfc-reader that would read the tags under these stands whereafter a message is send to an Event Hub to be able to visualize the messages as a visual on a website. To make this more attractive the "Minecraft" theme was chosen so that every kid would be able to recognize it and think of it as more fun. In this case, the dolls would be Minecraft themed and the stands would be decorated as "redstone ore" blocks.

## Purchasing and Assembling the needed material

So now we have the idea, what is needed to make this a reality?

* NFC Reader (In my case a RFID-RC522 reader)
* Raspberry PI
* Jumper Cables
* Breadboard (to be able to connect the reader to the raspberry pi)
* Lego for the stands (let's go back to being a kid again ;))
* Lego for the puppets

### Purchasing our Lego

After a quick shopping spree on our favorite Webshop together with a quick trip to the Lego store to purchase the material for the stands (a big thanks to the employees in there that helped me find all the material I needed) resulted in some Minecraft boxes and lego components to build everything we need:

**Puppets:**

![./lego-box-1.jpg](./lego-box-1.jpg)
![./lego-box-2.jpg](./lego-box-2.jpg)

**Stand Components:**

![./lego-purchase-1.jpg](./lego-purchase-1.jpg)
![./lego-purchase-2.jpg](./lego-purchase-2.jpg)
![./lego-purchase-3.jpg](./lego-purchase-3.jpg)

### Assembling our Lego Stand

Now we have our Lego, let's get started with building our stands for our Minecraft dolls. Here you can find the different steps that I used towards building those stands:

![./lego-stand-part-1.jpg](./lego-stand-part-1.jpg)
![./lego-stand-part-2.jpg](./lego-stand-part-2.jpg)
![./lego-stand-part-3.jpg](./lego-stand-part-3.jpg)
![./lego-stand-part-4.jpg](./lego-stand-part-4.jpg)
![./lego-stand-part-5.jpg](./lego-stand-part-5.jpg)

![./lego-stand-finished.jpg](./lego-stand-finished.jpg)

### Soldering our NFC Reader

After purchasing the RFID-255 reader a sharp eye will notice that it comes into a small package with nothing being assembled. 

![./nfc-tags-and-reader.jpg](./nfc-tags-and-reader.jpg)

A quick soldering trip later then resulted in something usable that we can easily stick on a breadboard

![./nfc-reader-soldering.jpg](./nfc-reader-soldering.jpg)
![./nfc-reader-soldering-done.jpg](./nfc-reader-soldering-done.jpg)
![./nfc-reader-breadboard.jpg](./nfc-reader-breadboard.jpg)

### Connecting the NFC Reader to our Raspberry PI

To connect our Raspberry PI reader to the RFID reader that we just soldered above, we need to know the pin layout. Luckily a lot of online documentation is able to provide us something as the layout shown below:

![./raspberry-pins.png](./raspberry-pins.png)

On here we can read our pin connections:

|RFID Pin|Raspberry PI Pin|
|-|-|
|MOSI|Pin 19 (GPIO10)|
|MISO|Pin 21 (GPIO9)|
|SCLK|Pin 23 (GPIO11)|
|SDA|Pin 24 (GPIO8)|
|Reset|Pin 22 (GPIO25)|
|Power|Pin 1 (3,3 V)|
|GND|Pin 6 (GND)|

Looking like this when we connect them:

![./raspberry-connected.jpg](./raspberry-connected.jpg)

### Enabling the SPI Interface on Raspberry PI

Before we are able to read out our RFID tag, we first need to enable the SPI interface on the Raspberry PI.

To do this we need to log in on the console of the Raspberry PI and run the `sudo raspi-config` command. This will open up an interface where we will select `5. Interfacing Options` -> `SPI` -> `Enable`.

Confirm that this is set by running `lsmod | grep spi` which should show `spi_bcm2835` if everything went correctly. The SPI interface is now enabled.

## Creating our code

Now everything is connected, the only thing left to do is write the actual code for 2 components that will do several things:

### NFC Reader Code

For our NFC Reader we will be using `Node.js` for the program code together with a library called `mfrc522-rpi` which will make our lives easier since it allows us to call functions that can easily read the tags for us.

#### 1. Detecting our NFC Tag

Detecting our NFC Tag can be done through the library mentioned earlier, then we can copy in the tutorial code as mentioned in the library README.md which looks like this:

```javascript
//# Init WiringPi with SPI Channel 0
mfrc522.initWiringPi(0);

//# This loop keeps checking for chips. If one is near it will get the UID and authenticate
console.log("scanning...");
console.log("Please put chip or keycard in the antenna inductive zone!");
console.log("Press Ctrl-C to stop.");

setInterval(function () { // Every 2 seconds check if a card is present
    //# reset card
    mfrc522.reset();

    //# Scan for cards
    let response = mfrc522.findCard();
    if (!response.status) {
        return;
    }

    console.log("Card detected, CardType: " + response.bitSize);

    //# Get the UID of the card
    response = mfrc522.getUid();
    if (!response.status) {
        console.log("UID Scan Error");
        return;
    }

    //# If we have the UID, continue
    const uid = response.data;
    console.log("Card read UID: %s %s %s %s", uid[0].toString(16), uid[1].toString(16), uid[2].toString(16), uid[3].toString(16));

    //# Select the scanned card
    const memoryCapacity = mfrc522.selectCard(uid);
    console.log("Card Memory Capacity: " + memoryCapacity);

    //# dump fifo buffer
    let cardMemory = {};
    for (let i = 0; i < 45; i++) {
        if (i != 44) {
            cardMemory[`block-${i}`] = mfrc522.getDataForBlock(i).toString().split(",").map(i => String.fromCharCode(i)).join(",").replace(/\\u[0-9a-zA-Z]+/, '');
        }

        console.log("Block: " + i + " Data: " + mfrc522.getDataForBlock(i));
    }
}, 2000);
```

#### 2. Extracting our NFC Tag Data

Since we do not want everything being extracted, we are just going to specify what we need for our tags. Looking into the result from the democode above shows us that our data is located in Block 7 and that it has a closing byte to show when all the data was delivered.

We can this add the following code to detect this:

```javascript
// Get out block (i = 7)
let blockNo = 7;
let blockClosingByte = 254;

let blockRaw = mfrc522.getDataForBlock(blockNo).toString().split(",");
let result = "";
let i = 0;

while (blockRaw[i] && blockRaw[i] != "0" && blockRaw[i] != blockClosingByte.toString() && i < 20) {
    i++;
    if (i == 0 || i == 1 || blockRaw[i] == blockClosingByte.toString()) {
        // Do nothing
    } else {
        result += String.fromCharCode(blockRaw[i]);
    }
}
```

#### 3. Sending our data to Event Hub

The last part to do now is to include our EventHub connection and send the message once a message has been detected. Through the awesome `azure-event-hubs` NPM package we are able to run the following code:

```javascript
const eventHubClient = require('azure-event-hubs').Client;
const client = eventHubClient.fromConnectionString(config.getConnectionString(), config.eventHub);
const sender = await client.createSender();

sender.send({
    name: process.argv[2] || result,
    raw: blockRaw
});
```

Which will configure our sender its connection string as defined in a `config.js` that we created and connect to the given eventHub. After that we will just send a message containing our block details that we extracted through the code in step 2.

### Website Code

#### 1. Creating the base website layout

For the layout of the website, I decided to start from the design presented by the original Minecraft website [https://minecraft.net/en-us/](https://minecraft.net/en-us/) which is very recognizable and easy to recreate using css and html, in just a few lines of codes we can get to this result:

![./website-start.png](./website-start.png)

#### 2. Getting the incoming data from Event Hub

Once the website base template is made, we need to hook it up to our EventHub. For this I reused my previous [work](/2018/5/8/coding-javascript-amqp-in-browser) that explains how you can easily hook your website to Event Hub and consume the events coming in through the AMQP protocol.

#### 3. Displaying our picture

Now we have our events coming in, we just have to add these few lines of code to process the events and update our HTML tree to show a picture:

```javascript
var decodedMessage = Utf8ArrayToStr(context.message.body.content);
var decodedMessages = decodedMessage.split('\n'); // Apparently multiple json messages per payload
var decodedMessage = JSON.parse(decodedMessages[0]);

console.log("Got Message: ");
console.log(decodedMessage);

document.querySelector("#content").innerHTML = "<img id=\"minecraft\" />";
document.querySelector("#minecraft").src = "/images/dolls/" + decodedMessage.name + ".png";
```

#### Result

When done successfully we are now able to put a doll on the NFC reader which will present us with the following screen:

![./website-doll.png](./website-doll.png)

Allowing the children to play around with the tags to match the different dolls to these tags.

If you want to replicate this, feel free to check out the code at: [https://github.com/Xaviergeerinck/PublicProjects](https://github.com/Xaviergeerinck/PublicProjects)

Building a Minecraft Puppet Scanner with Event Hub, a NFC 522 Reader and a Raspberry PI


One of the most common questions I get when talking with customers, is how they are able to set up a good big data architecture that will allow them to process all their existing data. With as an ultimate goal to perform advanced analytics and AI on top of it, to extract insights that will allow them to stay relevant in the ever faster evolving world of today.

To tackle this issue, I always first start by asking them what their understanding "Big Data" is, because one customer is not the other. One might think that Big Data is just the way they are able to process all their different excel files, while another might think that it is the holy grail for all their projects and intelligence. Well in this article, I want to explain you what Big Data means to me and provide you a thought process that can help you in defining your Big Data strategy for your organization.

Now what is Big Data? Gartner explains it through different V's: "Volume", "Variety", "Velocity", "Veracity", where if you have 2 of the first 3, then you are speaking about Big Data. To quote it in their own words:

> *Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.* - https://www.gartner.com/it-glossary/big-data

For me Big Data is however so much more. It is one thing to be able to speak about it and recognise when you are using, but it's another to implement it in such a way that your whole organisation is able to benefit from it. 

Let's tackle 3 of the most common pain points that I see while working with customers:

## 1. Creating a Centralized Data Lake

When talking with my customers, every time the same question comes up when wanting to start utilizing Big Data. The question of "Where are we going to store it?". But everytime this answer is quite simple: "As centralized as possible". Storing your data in a centralized place will allow you to maintain better control over it. This way you will be able to have a knowledge about your in flows, processing flows as well as your outgoing flows, giving you the power to draw complex analytics from all of this data.

But why can people overlook this then? Well because in the real world it is not as simple as it sounds. Many of the enterprises out there are tens of years old, having processes and structures in place that sometimes date back to these times. Just think about all the enterprises out there that still work in a vertical structure, where teams work in silos and where data belonging to these teams does not cross over to different teams. Resulting in a fragmented data landscape.

Due to this when setting up your centralized data repository, the first part is to perform change management on the organization. Can you go from a vertical structure, to a more horizontal structure where teams work together with each other, and consume the same data repository? More about this in point 3.

While the second part is, how do you correctly store the data coming in from all these different systems in all these different formats and data types. This is what we will look at in point 2.

If you would visualize this thought process, you might end up with something that looks like the following:

![./vision.png](./vision.png)

## 2. Utilizing your Data Lake through Landing Zones

So we have all this data coming in, in all their different formats and data types. At first sight, this might look like a bad thing; but actually it isn't. A Data Lake is created for just that, to store data in any kind of format - making it the perfect tool for all your historic needs. 

But how do we structure this data? Well, A good practice is to store the data coming in, in different "landing zones". Which are used to not only store your original raw data coming in from your BI systems, videos, ... but also to store your processed data - that is processed in such a way that every format and data type is normalized to the same standards that you are used to. Just think about normalizing data so it uses the same date format everywhere. These zones we often call the "raw" and "staging" zones. 

Linking these two zones together is just a matter of connecting the correct ETL processes, that pull the data from the "raw" zone, modify it in the way you want, and then puts it in your "staging" zone, ready for consumption by your reporting tools or data warehouses.

An interesting question you may now have is: "But aren't we losing performance if we are going from our original structured data sources towards a data lake, to then go back to a data warehouse"? This is indeed a interesting question that we can only answer by going back to the purpose of a data lake. A Data Lake it's goal is to have a *centralized data store*, where all our data is collected so that our *data scientists* are able to perform their jobs on the purest form of source data. But secondly, where our *ETL jobs* are running, constantly copying the data from the processed "staging" zone towards the target systems. Looking back at these different purposes, we are able to say that even though we might hit a small performance hit, it will be an easy system to control everything happening on our data, resulting in an improved analytics, reporting and even governance process.

## 3. Correct Internal Governance

As in all good things, there is still a hard thing for everything; this is no exception. The hardest part for creating a centralized data lake is the governance. Now you have have everything in place to pull your data from your different systems towards your data lake and you are able to process this data so that it is available for consumption. But how do you actually work with this across your different teams?

Different teams will work on the data in a data lake, making it necessary to have the correct access control roles in place that will define which users have access to which part of the data lake. Just think about different scenarios, such as a BI specialist who does not need access to the data lake, but rather needs access to aggregated views that we provide them. Or you want to work across regions that allow you to provide row level security on your data, so that the people in the different regions only see the data meant for them. These are all things you will have to think about when creating your Big Data architecture.

Often we see the following roles and how they are configured for these organizations:
* **Data Engineer:** Your data engineer is in charge of setting up ETL jobs to process data from one landing zone to the other, requiring full access to the data lake.
* **Data Scientist:** This role has access to the data lake in its purest form through the "raw" landing zone. By providing access to this, data scientists will be able to draw conclusions from your data and build AI models on top of these that could greatly benefit your organization.
* **BI Specialist:** These people often do not require access to the data itself, but rather to views. It could thus be beneficial to provide access to views created through a system such as *Hive* that allows your specialist to run simple SQL queries on top of your data, while still providing Row Level Security and everything else that you are used to in terms of security. (Note: Row Level Security is something that needs to be configured correctly, it might sound easy, but it can take a while to configure correctly on unstructured data)

> In case you wish extra security and nobody accessing the data lake, you could for example create a subset of your data lake through a simple ETL copy where the different roles are able to play and do everything they wish. With this subset being recreated every night.

## Example Architectures

So we defined 3 topics that we need to think about when designing a Big Data architecture, but how does one look in practice now? For this I created 2 architectures that I want to share with you.

The first one is a standard Big Data architecture, allowing you to see how we ingest data from our source systems into a data lake and start serving this. This kind of architecture is what I call a more "slower" architecture, because you might rely on polling systems that will poll every "x" seconds or minutes to see if new data is available. This will happen on the most left part where you get the data from your source systems, as well as on the most right part where data will be pushed towards the consumption layer. However for most use cases this could be fine, it all depends on the question: "How fast do I want data to be available in my consumption layer?".

![./architecture.png](./architecture.png)

If you however want your data to be accessible almost as soon as it is being put in the systems, then you need to look at more "event-driven" kind of systems. These will work with events that are triggered every time data is made available, rather than having to poll every few seconds or minutes.

![./architecture-pubsub.png](./architecture-pubsub.png)


Big Data, the why, how and what - A thought Process and Architecture


For a new project of mine, I wanted to get the different tweets of multiple people at once and save those as a CSV and JSON for further processing. But how can we do this easily in just a couple of lines?

As in every project, I start by assessing what will be the quickest way to resolve this problem (of course in my mind and based on experience). For this project, I decided that since I am going to be working with JSON a lot and web technologies (Twitter API) that [Node.js](https://nodejs.org/en/) is my best option and it should take me around 1h (or even less). More typed languages as C# or Java would require me to parse all the JSON code into objects, and that is too much work for me on this moment for the return I will get from it.

To get started, I started by looking what is already available. A quick search on the [npm](https://www.npmjs.com/) repository resulted in a package called [twitter](https://www.npmjs.com/package/twitter) where we just pass the API keys that you get from twitter (more here: [https://developer.twitter.com/en/apply-for-access.html](https://developer.twitter.com/en/apply-for-access.html)) to query their API. While looking at the API references for Twitter, I discovered the [statuses/user_timeline](https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html) endpoint allowing just what I want (getting tweets from users).

So I installed it by running `npm install twitter --save` and started writing my code resulting in the code that you can find at the end of this article.

> **Note:** As also mentioned in the code comments, pagination is NOT implemented so only 200 tweets per user are being shown.

What this code does is that it will call this API and get the tweets. But not only that, it will also map those tweets to a certain format that is easy to work with:

```javascript
{
    twitter_user: {
        tweets: [],
        metadata: []
    }
}
```

Which is important for further processing. Once these tweets are mapped to the correct format, I then save them as a `.json` file and as a `.csv` file. The `.csv` file however needs a correct translation which can be easily coded in a few lines like this:

```javascript
let csv = "user,user_followers,user_favourites,tweet,tweet_created_at";
Object.keys(result).forEach((user) => {
    result[user]['tweets'].forEach((tweet) => {
        csv += `\n${result[user]['metadata']['screen_name']},${result[user]['metadata']['followers']},${result[user]['metadata']['favourites']},${tweet['text'].replace(/,/g, '').replace(/\n/g, '')},${tweet['created_at']}`;
    });
});
```

> **Note:** the above also strips of ',' and '\n' to prevent parsing in our Excel to mess up due to the usage of a `.csv` file.

**index.js**

```javascript
const fs = require('fs');
const Twitter = require('twitter');
const config = require('./config');

const client = new Twitter(config.twitter);

// configure the tweets to fetch per user
// Note: we can get up to 3.200 tweets per user
// Note: we can fetch max 200 tweets as https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html notes
//       so pagination should be implemented @todo but not for now
const tweetsToFetch = 200;

const users = [ 
  'MSEurope',
  'XavierGeerinck',
];

const getTweets = (user) => new Promise((resolve, reject) => {
  const params = { 
    screen_name: user,
    count: tweetsToFetch
  };

  client.get('statuses/user_timeline', params, (err, tweets, res) => {
    if (err) {
      return reject(err);
    }

    return resolve(tweets);
  });
});

const start = async () => {
  const promises = [];
  const allTweets = {}; // Object since it's <user>: [tweets]
  let tweetCount = 0;

  // Get tweets
  users.forEach((user) => {
    promises.push(new Promise(async (resolve, reject) => {
      const tweets = await getTweets(user);
      allTweets[user] = tweets;
      tweetCount += tweets.length;
      resolve();
    }));
  });

  await Promise.all(promises);
  
  // Manipulate tweets, we want to store the text, date posted and metadata about the user
  let result = {};

  Object.keys(allTweets).forEach((user) => {
    // Can be empty, so skip
    if (!allTweets[user]) {
      return;
    }

    if (!result[user]) {
      result[user] = {};
    }

    result[user]['tweets'] = allTweets[user].map((tweet) => ({
      created_at: tweet.created_at,
      text: tweet.text
    }));
    
    result[user]['metadata'] = {
      id: allTweets[user][0]['user']['id_str'],
      name: allTweets[user][0]['user']['name'],
      screen_name: allTweets[user][0]['user']['screen_name'],
      description: allTweets[user][0]['user']['description'],
      followers: allTweets[user][0]['user']['followers_count'],
      favourites: allTweets[user][0]['user']['favourites_count'],
    }
  })

  // Save as JSON
  fs.writeFileSync('./results.json', JSON.stringify(result), 'utf8');

  // Also write as a CSV as in <user>,<user_followers>,<user_favourites>,<tweet>,<tweet_created_at>
  let csv = "user,user_followers,user_favourites,tweet,tweet_created_at";
  Object.keys(result).forEach((user) => {
    result[user]['tweets'].forEach((tweet) => {
      csv += `\n${result[user]['metadata']['screen_name']},${result[user]['metadata']['followers']},${result[user]['metadata']['favourites']},${tweet['text'].replace(/,/g, '').replace(/\n/g, '')},${tweet['created_at']}`;
    });
  });
  
  fs.writeFileSync('./results.csv', csv);

  console.log(`DONE, saved ${tweetCount} tweets`);
};

start();
```

**config.js**

```javascript
module.exports = {
  twitter: {
    consumer_key: '<your_consumer_key>',
    consumer_secret: '<your_consumer_secret>',
    access_token_key: '<your_access_token>',
    access_token_secret: '<your_access_token_secret>'
  }
}
```

Getting Tweets of multiple people on Twitter


When you work with the Internet of Things (IoT) or other real-time data sources, there is one things that keeps bothering you, and that's a real-time visualization dashboard. So in this post, I will explain how you can set up a pipeline that allows you to visualize through 2 popular visualization tools: Chronograf and Grafana.

> **Note:** This post will not cover setting up Grafana nor Chronograf due to the amount of work required. For more information about how to set these up, feel free to check this other post: [Visualize metrics with Grafana and InfluxDB](/visualize-metrics-with-grafana-and-influxdb)

## Architecture

Let's start by discussing what we will be creating and why we are creating it like this. The eventual architecture will be something that looks like this:

![./architecture.png](./architecture.png)

We can identify several components in here:

### [Stream] EventHub

**Description:** EventHub Allows us to ingest and distribute events from one component to the other.

**Why?** We use an EventHub because we will be processing a lot of IoT events, which is easier if we have one single point where all our data comes in and that we can pipe to other components. Note that we are not using an IoT Hub which can also manage our IoT Devices, because we will be sending our data from a custom built script so there is no need for device management.

**Alternatives?**

* Apache Kafka (#opensource, kind of like EventHub)
* IoT Hub (Includes device management and things such as configuration pushing)

### [Spark Engine] Databricks #opensource

**Description:** Databricks is a platform that allows you to process big data and streams through the use of the Spark platform.

**Why?** We want to be able to ingest our stream, modify it and stream it to our dashboard. A component such as this allows for a easy redistribution, while being able to modify our stream. We can do things such as multiple endpoint streaming, data windowing, ...

**Alternatives?**

* Azure Stream Analytics
* CosmosDB (We use the change-feed here)

### [Dashboard] Chronograf

**Description:** Chronograf is the interface component of InfluxData, created to make it easy to visualize and explore this data.

**Why?** For a real-time dashboard, we want to be able to see our events in less than 1 minute of delay, Chronograf does just this, it allows us to set a refresh interval of 5 seconds, which covers our need of showing this data near real-time. If we want an even lower refresh interval, we are looking towards socket solutions, which are quite hard to come by on this moment. Interesting to note here is that Chronograf requires us to use InfluxDB, which implicates a small lock-in being generated on this level. If you wish to avoid this, feel free to continue with Grafana as shown below.

**Alternatives?**

* PowerBI
* Kibana (ELK Stack)
* Grafana

### [Dashboard] Grafana

**Description:** Grafana is a tool for monitoring and analysing metrics through data sources such as Graphite, InfluxDB, Prometheus and many more.

**Why?** For a real-time dashboard, we want to be able to see our events in less than 1 minute of delay, Grafana does just this, it allows us to set a refresh interval of 5 seconds, which covers our need of showing this data near real-time. If we want an even lower refresh interval, we are looking towards socket solutions, which are quite hard to come by on this moment. The major difference with Chronograf is that Grafana integrates with more datasources on this moment, allowing us to evade a lock-in generated on this level.

**Alternatives?**

* PowerBI
* Kibana (ELK Stack)
* Chronograf

## Configuring our components

### InfluxDB & Grafana

To configure our InfluxDB, Grafana and Chronograf containers, feel free to check out my other post at [/howto-visualize-metrics-with-grafana-and-influxdb](/post/2018/9/3/big-data-visualize-metrics-with-grafana-and-influxdb)

### Event Hub

To configure our Event Hub check out my blog post ["Sending and Receiving events with Azure Event Hub"](https://medium.com/@xaviergeerinck/sending-and-receiving-events-with-azure-event-hub-f92af9ad7fa0).

Once you created an Event Hub create a consumer group and your Policy Key with the "Listen" permission. More information for this you can find [here](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-resource-manager-namespace-event-hub?WT.mc_id=DP-MVP-5004263)

Make sure to note down the following details
* PolicyName
* PolicyKey
* NamespaceName - This is the name of where all your EventHubs are located
* EventHubName - This is the name of the EventHub you created
* ConsumerGroupName - This is the consumer group that you created under your EventHub. The default name is: `$Default`

### DataBricks

For our DataBricks, set up an account through the [awesome documentation](https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal?WT.mc_id=AI-MVP-5004263)

Now we can log in on our Databricks portal and start creating our script that will ingest our data from EventHub. Start by creating your first notebook (I am using Scala here) and let's see what we need to put in it to connect to our event stream and process the data towards our visualization tools.

#### Connecting to EventHub

Before we can connect to EventHub, we need to add a Maven library. For this, click "Workspace" -> "Shared" -> Under the arrow click "Create" -> "Library". Now search after "azure-eventhubs-spark_2.11-2.3.2" and install and attach it to the cluster.

Once all this pre-setup is done, we can go to our notebook and connect to our EventHub with the following bit of code:

```scala
// Note: Before you can use these packages you need to add the Maven coordinate: com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.2
//       See https://docs.databricks.com/user-guide/libraries.html#maven-libraries for how to do so
//       More info: https://docs.databricks.com/spark/latest/structured-streaming/streaming-event-hubs.html
import org.apache.spark.eventhubs.{ ConnectionStringBuilder, EventHubsConf, EventPosition, EventHubsUtils }

// Connect to EventHub
val connectionString = ConnectionStringBuilder("Endpoint=<your_connection_string>")
  .setEventHubName("<your_eh_name>")
  .build();

val eventHubsConf = EventHubsConf(connectionString)
  .setStartingPosition(EventPosition.fromEndOfStream);

// eventHubs is a org.apache.spark.sql.DataFrame object
val eventHubs = spark.readStream
  .format("eventhubs")
  .options(eventHubsConf.toMap)
  .load();
```

If we now run this cell, we will see our variables initialized.

#### Performing ETL on our incoming EventHub data

To see the schema of the EventHub data coming in, we can utilize the printSchema method:

```scala
// Print schema of our stream
eventHubs.printSchema()
```

Here we see a field called `body` which contains our EventHub events data, but in a binary format. We thus have to parse this towards our original JSON format, together with correctly casting the variables.

```scala
import org.apache.spark.sql.types._ // https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/package-summary.html
import org.apache.spark.sql.functions._

// Our JSON Schema
val jsonSchema = new StructType()
  .add("sensor", StringType)
  .add("temperatureValue", StringType)
  .add("humidityValue", StringType)
  .add("createdAt", StringType)

// Convert our EventHub data, where the body contains our message and which we decode the JSON
val messages = eventHubs
  // Parse our columns from what EventHub gives us (which is the data we are sending, plus metadata such as offset, enqueueTime, ...)
  .withColumn("Offset", $"offset".cast(LongType))
  .withColumn("Time (readable)", $"enqueuedTime".cast(TimestampType))
  .withColumn("Timestamp", $"enqueuedTime".cast(LongType))
  .withColumn("Body", $"body".cast(StringType))
  // Select them so we can play with them
  .select("Offset", "Time (readable)", "Timestamp", "Body")
  // Parse the "Body" column as a JSON Schema which we defined above
  .select(from_json($"Body", jsonSchema) as "sensors")
  // Now select the values from our JSON Structure and cast them manually to avoid problems
  .select(
    $"sensors.sensor".cast("string"),
    $"sensors.createdAt".cast("timestamp"), 
    $"sensors.temperatureValue".cast("double") as "tempVal", 
    $"sensors.humidityValue".cast("double") as "humVal"
  )

// Print the schema to know what we are working with
messages.printSchema()
```

> Interesting to note here is the double casting being done, one in the JSON schema definition and one in the SQL select. This is done since when we do this in our JSON structure, we receive null values because of an incorrect parse.

As a debug method, we can now write our stream to memory and see if everything is coming in successfully:

```scala
// Display the stream
val messagesStream = messages.writeStream
  .outputMode("append")
  .format("memory")
  .option("truncate", false)
  .queryName("streamData")
  .start()

display(messages)
```

#### Sending our parsed events to InfluxDB

To connect to InfluxDB, we use a library called `Chronicler` which can be found at: [https://github.com/fsanaulla/chronicler](https://github.com/fsanaulla/chronicler) and [https://github.com/fsanaulla/chronicler-spark](https://github.com/fsanaulla/chronicler-spark).

We install it as we did before, by adding the libraries to our shared folder. The following libraries need to be installed:

```
com.github.fsanaulla:chronicler-akka-http
com.github.fsanaulla:chronicler-async-http
com.github.fsanaulla:chronicler-url-http
com.github.fsanaulla:chronicler-udp
com.github.fsanaulla:chronicler-macros
com.github.fsanaulla:chronicler-spark-structured-streaming
```

Once this is done, go back to your notebook and start adding your connection code:

```scala
// Create a connection to our InfluxDB
import com.github.fsanaulla.chronicler.async.Influx
import com.github.fsanaulla.chronicler.core.model.{InfluxConfig, InfluxCredentials}

import scala.util.{Success, Failure}
import scala.concurrent.ExecutionContext.Implicits.global

implicit lazy val influxConf: InfluxConfig = InfluxConfig("<host>", 8086, Some(InfluxCredentials("<user>", "<pass>")), gzipped = false)
```

Whereafter we can send our Events to InfluxDB by defining a writer that defines how we serialize our events in the tag and field format InfluxDB requires, and starting our stream with the `.saveToInflux` and `.start`.

```scala
// Save our stream to InfluxDB
//import com.github.fsanaulla.chronicler.spark.ds._ // DataFrame = DataSet[Row]
import com.github.fsanaulla.chronicler.spark.structured.streaming._
import com.github.fsanaulla.chronicler.macros.Macros
import com.github.fsanaulla.chronicler.macros.annotations.{field, tag}
import com.github.fsanaulla.chronicler.core.model.InfluxWriter
import com.github.fsanaulla.chronicler.core.model.Point

//case class Entity(@tag sensor: String, @field createdAt: Int, @field tempVal: Double, @field humVal: Double)
//implicit val wr: InfluxWriter[Entity] = Macros.writer[Entity]

implicit val wr: InfluxWriter[Row] = new InfluxWriter[Row] {
   override def write(o: Row): String = {
     val sb = StringBuilder.newBuilder
     
     // Query looks like: <tags> <fields> <timestamp RFC3339>
     sb.append(s"sensor=${o(0)}")
       .append(" ") 
       .append("tempVal=")
       .append(o(2))
       .append(",")
       .append("humVal=")
       .append(o(3))
     
     sb.toString()
   }
}

// Create our stream
val stream = messages
  .writeStream // Create a WriteStream
  .saveToInflux("myMetrics", "myValues")
  .start()
```

#### Creating our dashboards

We now have our stream running that is sending everything to our InfluxDB. Once we manipulate our dashboards a little bit, we can set them up like this:

##### Grafana

![./grafana-setup-4-humVal.png](./grafana-setup-4-humVal.png)

![./grafana-setup-5-tempVal.png](./grafana-setup-5-tempVal.png)

##### Chronograf

![./chronograf-setup-3-tempVal.png](./chronograf-setup-3-tempVal.png)

![./chronograf-setup-4-humVal.png](./chronograf-setup-4-humVal.png)

## Conclusion

Once all of this is done, we can view our completed dashboard and have a result that looks like this:

![./grafana-setup-6-dashboard.png](./grafana-setup-6-dashboard.png)

![./chronograf-setup-5-dashboard.png](./chronograf-setup-5-dashboard.png)

## EventHub Sender Script

As a last point, I would like to include the script that I used to send mock data towards my EventHub:

```javascript
const eventHubClient = require('azure-event-hubs').Client;
const config = require('./config.js');

// Init Client
const client = eventHubClient.fromConnectionString(config.getConnectionString(), config.eventHub);

// Create a sender
client.createSender()
.then((tx) => {
  setInterval(() => {
    const val = {
      temperature_sensor: {
        id: `sensor_${Math.floor((Math.random() * 10) + 1)}`,
        value: ((Math.random() * 41) - 10).toFixed(2),
        createdAt: new Date()
      },
      temp_value: ((Math.random() * 41) - 10).toFixed(2), 
    };

    tx.send(val);
    console.log(`Sent: ${JSON.stringify(val)}`);
  }, 1000);
})
```

with the config containing

```javascript
const config = {
  eventHubUrl: "<your_url>", // no protocol included
  eventHubSharedAccessKeyName: "RootManageSharedAccessKey",
  eventHubSharedAccessKey: "<your_key>",
  // This is the EventHub name (go to eventhub -> Event Hubs -> <Names>)
  eventHub: "<your_eh>",
  eventHubPath: "<your_eh_namespace>",
};

module.exports = {
  ...config,
  getConnectionString: () => `Endpoint=sb://${config.eventHubUrl}/;SharedAccessKeyName=${config.eventHubSharedAccessKeyName};SharedAccessKey=${config.eventHubSharedAccessKey}`
}
```

Building a real-time streaming dashboard with Spark, Grafana, Chronograf and InfluxDB


Visualizing is an important part of working with data. Here I will walk you step by step on how you can create a working Visualization dashboard with Grafana and Chronograf which queries events through InfluxDB.

This post is part of the "Real Time Streaming Dashboard" series which can be found here: [/real-time-streaming-dashboard](/post/2018/9/3/big-data-real-time-streaming-dashboard)

## [Time Series DB] Creating InfluxDB

The easiest way to create an application without too much operational cost, is by deploying it as a container instance situated in the PaaS operating model. Here I am using Azure Container Instances, which allows me to deploy a container within mere seconds after configuration.

To do this, go to the marketplace by searching for "Container Instance" and click on "create". Now use the details as shown below to configure our InfluxDB container.

![./container-influxdb-1.png](./container-influxdb-1.png)

![./container-influxdb-2.png](./container-influxdb-2.png)

![./container-influxdb-3.png](./container-influxdb-3.png)

If everything went ok, you will now be able to access InfluxDB on your FQDN:8086 endpoint through: `http://<FQDN>:8086` which will show an error message `404 page not found`, which means we did it well and that InfluxDB is running.

## [Visualize] Create Grafana

Just as with our InfluxDB container, we will also be creating a Grafana container that will host our Grafana application. For this we use the details below.

> Since Grafana is open-source, feel free to check the code here: [https://github.com/grafana/grafana](https://github.com/grafana/grafana)

![./container-grafana-1.png](./container-grafana-1.png)

![./container-grafana-2.png](./container-grafana-2.png)

![./container-grafana-3.png](./container-grafana-3.png)

If everything went ok, you will now be able to go to your `http://<FQDN>:3000` and be presented with the Grafana login screen.

> Note: The default login details are `"admin"/"admin"`, to change those, use the environment variables as described in: [http://docs.grafana.org/installation/docker/](http://docs.grafana.org/installation/docker/)

Continue by logging in to the Grafana login screen, whereafter you will be presented by a setup screen, showing a green buttin called "Add Data Source". Click it, to configure your first data source which will link to our previously created InfluxDB container.

![./grafana-setup-1.png](./grafana-setup-1.png)

![./grafana-setup-2-influxdb.png](./grafana-setup-2-influxdb.png)

Once you clicked "Save & Test", go back to our main page by entering our URL `http://<FQDN>:3000/` again to create your first Dashboard. Go Aheard by clicking the "New Dashboard" button, to start visualizing your InfluxDB data.

![./grafana-setup-3.png](./grafana-setup-3.png)

After playing around a little bit, you should have something that can look like this:

![./grafana.png](./grafana.png)

## [Visualize] Create Chronograf

Another way of creating real-time dashboards is through a tool called "Chronograf". We can kind of compare Chronograf to the eco-system of the ELK Stack (Elasticsearch, Logstash and Kibana), but where Chronograf is created by the people behind InfluxDB.

To start, we again create a container instance with the details below:

![./container-chronograf-1.png](./container-chronograf-1.png)

![./container-chronograf-2.png](./container-chronograf-2.png)

![./container-chronograf-3.png](./container-chronograf-3.png)

whereafter we will be able to navigate to the setup page located at: `http://<FQDN>:8888` and be presented by a beautiful setup screen. On here, fill in your details towards your InfluxDB and click the "Add Connection" button to continue towards your home screen.


![./chronograf-setup-1.png](./chronograf-setup-1.png)

![./chronograf-setup-2.png](./chronograf-setup-2.png)


You can now go to the dashboard tab and start creating your dashboard, eventually ending up with something that can look like this:

![./chronograf.png](./chronograf.png)

How-To Visualize Metrics with Grafana and Chronograf by using InfluxDB


Today marks my 1st anniversary of working as a Cloud Solution Architect at Microsoft, and wow what an exciting time it has been! But before I go deeper into my experiences and lessons learned, I first would like to start by thanking some people without whom all of this would never have been possible.

My first thanks goes to Ron Pooters, who was my mentor since the beginning, guiding me through all the tips, tricks and steps of working at a company as large as this one, without whom I would've never been able to ramp up in a role as exciting as this one. Even though he has become a manager now (in fact he has become my manager!) he still appears as the first time I met him, an awesome guy striving and fighting to bring up the best in everyone and help them achieve a greater purpose. 

Secondly I would like to thank my first Manager and lead of the Customer Success Unit - Thomas Kessler. He was the one that believed in me before I even started, giving me the chance to start in this awesome architect role where I was able to grow and prove my worth. My second fiscal in - even though I had a struggling year, this believe is still there and I was allowed to continue in this fascinating role. Next to all of this he guided me through the different procedures and growth process a young graduate as me needed, to be able to show my worth to the company. 

Through the year I had much more support from the different colleagues around me (to call a few: Wesley, Nathan, Karim, Tine, Gitte, Nills, ...), who I am also eternal grateful towards, since without the support of these awesome people, I would've never been able to go from being seen as a "rookie" who came straight out of University in this field to the more "evolved" kind of person (the road is still long) where I am right now. My colleagues allowed me to feel part of the team, motivating me and giving me the chance to prove myself.

A good practice that is being done within Microsoft is reflection. What did we do last year, and how can we improve on this? and what I would like to do with this post is share my most important lessons of my first year, with the fresh and naive mind a student has (or had).

## 1. Planning

Every job you start takes time to master, do not just expect everything to become clear from your first week. You can however accelerate things, by **taking the time to understand your role description inside-out and creating a battle plan**, know what is expected from you and make a mapping from it so that you know where your focus points are and what you will be challenged on. Just think of this as creating your own personal *Go-To-Market strategy*, without one you are like a fish on dry land, hopping around to find water, your source of life.

One way that I have learned to accomplish this throughout the year, is to **create Key Performance Indicators (KPIs) for myself**. Allowing me to measure what I do, and react on things when I lose focus. In a customer facing role for example, a KPI could be *the number of engagements you spotted in that week*? or even *the number of weekly customer meetings you had*? but other KPIs could just as well be *the time you spent on training yourself*. Because in the world of today, everything is moving at the speed of light and we have to try and stay up to date with everything happening around us.

## 2. Execution

Now you have your Battle Plan and KPIs set up, but how do you actually execute? An approach that works for me is to understand that even if you have those KPIs in place, that you tend to forget about them. One way to remediate this is to put a meeting in the beginning of my week - even if it's just for 15minutes - as **a reminder for myself to be aware of** these KPIs and **measure them**.

Next to that there are a couple of other things that work for me in executing these plans and allowing me to spend more time on the subjects that increase my impact in the end:

* **When you encounter something that could be automated, automate it!** This doesn't have to be big, but it helps reducing your work in the longer term (just make sure that you know how much time you will spend automating it versus the time it will gain you, a basic calculation of investment vs ROI).
* **Plan your battle of the rocks:** moving a big rock takes time and effort, why not instead plan a few smaller rocks representing the achievements you want to get within a certain timeframe? By focusing on a few smaller ones, you will be able to gain more energy and motivation when you complete each one of them and be less likely to fail when tackling a rock that is too big for you at this moment in time.

## 3. Personal Growth

A last point that I want to tackle are my personal learnings in the area of "Personal Growth". I can for sure say from experience that this has been the most difficult area by far for me as a person, not only because I was going from a "student" mindset towards an "enterprise" mindset, or that I am a *ENTJ* type of person. But also because of that you start working with people far more senior than yourself (which is a good thing, that you will have to learn to appreciate and utilize!).

Something I have noticed around me, is that for a junior profile as mine, it's good to start off humble and build up your experience. A nice quote I got from my managers here is that: "Gray Hairs Count", showing the importance of it. **To illustrate some of my few key takeaways in how I try show humbleness**:

* **Put others first:** Know your own self-worth, and rather than spreading it around, just show interests in others and ask about their job/day/... This will not only gain you a lot of respect (because others come first), but it will also improve your networking by it, getting to know everyone around you.
* **Listen:** Instead of charging into a conversation and start talking about how you want to get things done, try to listen. Never dominate the conversation, but come in with facts, numbers and your voice when it really counts. This will save you a lot of energy in the long-run and also gain you respect. Since really, who wants to be known as the "know it all"?
* **Say "thank you":** People like it when they feel appreciated. Therefor take the time to show how much you appreciate their work.
* **Speak your mind, but politely:** Instead of starting everything sentence with "I think", "We should do..." why not start the sentence with: "Is it a good idea to ..."? By just changing this little nuance in your sentence, people will feel less offended.

Another approach to show this and be less likely fall into the trap of coming over too direct is by paying attention to **a few key points that you should try to avoid** while at work:

* Gossip
* Judging
* Excuses
* Negativity / Complaining
* Exaggeration
* Facts versus Opinions

I hope by sharing that I am able to bring over some of my struggles that i had during my first year at a big corporate as this one. What were the struggles you had? Feel free to share them in the comments below and start the discussion! :)

Lessons learned after being at Microsoft for 1 year


For a new fun Project, I wanted to create an Artificial Intelligence model that will allow me to predict the comments and likes of new pictures that a user can post on Instagram based on their own historical data.

Luckily thanks to **[ML.NET](https://www.microsoft.com/net/learn/apps/machine-learning-and-ai/ml-dotnet?WT.mc_id=AI-MVP-5004263)** which was released during build 2018, we are able to do this easily through a simple to create *pipeline*.

What will we be creating? Something like this architecture:

![./architecture.png](./architecture.png)

As in every Machine Learning projects, we have to do a few simple steps to go from data towards our eventual prediction:

1. Download / Prepare our data -> `Step1.DataPrep` folder
2. Train our Algorithm based on the data from step 1 -> `Step2.DataTraining` folder
3. Consume our model from step 2 -> `Step3.DataPrediction` folder

Resulting in the following folder structure:

```bash
root/
    Step1.DataPrep/ # NodeJS project so `npm init` to create a package.json
        index.js
        config/index.js
        API/CognitiveServiceVision.js
        data/{*.jpg,*.json} # Contains our pictures and metadata
        output/result.csv # Contains our CSV created from our pictures
    Step2.DataTraining/ # .NET Core solution -> dotnet new console -o Step2.DataTraining
    Step3.DataPrediction/ # .NET Core solution -> dotnet new console -o Step3.DataPrediction
```

## Step 1 - Downloading & Preparing our data

For Instagram data we could write our own scraper, or go the easy way and download one from the internet. In my case, I looked on Github and found a nice project called [InstaLooter](https://github.com/althonos/InstaLooter) which is written in python and allows us to download instagram pictures + metadata from any user without being authenticated. 

```bash
# Install InstaLooter
pip install --user instalooter --pre

# Run InstaLooter in our Step1.DataPrep/data folder and download pictures from the <instagram> account
instalooter user instagram -d
```

After all these pictures are downloaded, you will be able to start preparing them towards a `output/result.csv` file through a simple node.js script (see below). This script will just go through each image listed in the `data/` directory and send them to the [Azure Cognitive Service - Computer Vision API](https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/?WT.mc_id=AI-MVP-5004263) to retrieve the tags (which we Rate limit through our [Throttling script](/post/2018/8/22/coding-javascript-throttling-api-calls)).

> **Note:** The Instagram API is limited to 1.000 calls, so you will be able to download 1.000 pictures whereafter InstaLooter will time-out.

**index.js**

```javascript
const fs = require('fs');
const delay = require('delay');
const config = require('./config');
const API = require('./API/CognitiveServiceVision');
const dataFolder = './data/';
const outputFolder = './output/';

const api = new API(config.thirdParty.cognitiveServices.imageAnalyze.apiKey);

// Get list of images in path with ascending time
let files = fs.readdirSync(dataFolder).filter(i => i.endsWith('.jpg')).sort();

// Go through files, getting the metadata
let csvResult = "shortcode;url;likes;comments;tags;\n"; // init buffer
let count = 0;
let skipped = 0;

// Process each image, getting the tags through Vision Cognitive Service, and checking the json
// Note: Out of order will appear here!
const start = async () => {
    // Go through promises, but throttled
    const limit = 10; // 10 at a time

    // files = files.slice(0, 50);

    while (files.length > 0) {
        console.log(`[Progress] Processing ${limit} of the ${files.length} images`);
        let currentFiles = files.slice(0, limit);
        files = files.slice(limit, files.length);

         // Prepare promises
         let promises = [];
         currentFiles.forEach(i => {
            promises.push(new Promise(async (resolve, reject) => {
                try {
                    // Get tags
                    const imageBinary = await api.readImageByPath(dataFolder + i);
                    const res = await api.analyzeImage(imageBinary);
                    const tags = res.tags.map(t => t.name).join(",");
            
                    // Get JSON content
                    const file = require(`${dataFolder}${i.replace('.jpg', '.json')}`);
                    csvResult += `${file.shortcode};${file.display_url};${file.edge_media_preview_like.count};${file.edge_media_to_comment.count};${tags};\n`;
        
                    count++;
                } catch (e) {
                    skipped++;
                }
        
                // Update progress
                console.log(`[Progress] ${count + skipped}/${files.length}`);

                return resolve();
            }));
        })
    
        console.log('awaiting promises');
        await Promise.all(promises);
        console.log(`Sleeping for 1000ms`);
        await delay(1000);
    }

    // Write result
    fs.writeFileSync(outputFolder + 'result.csv', csvResult);

    console.log(`Processed ${count + skipped} files, success: ${count} skipped: ${skipped} and saved to ${outputFolder}result.csv`);
}

start();
```


**API/CognitiveServiceVision.js**

```javascript
const fetch = require('node-fetch');
const fs = require('fs');
const util = require('util');

const readFile = util.promisify(fs.readFile);

// Based on https://github.com/Microsoft/BotFramework-Samples/blob/master/StackOverflow-Bot/StackBot/lib/bingsearchclient.js
class CognitiveServiceAnalyzeImage {
  constructor(apiKey, opts) {
    if (!apiKey) throw new Error('apiKey is required');

    this.apiKey = apiKey;
    this.baseUrl = "https://westeurope.api.cognitive.microsoft.com/vision/v2.0/";
  }

  /**
   * An util to help you get the binary data of an image
   * @param {string} imagePath 
   */
  async readImageByPath(imagePath) {
    const imageBinary = await readFile(imagePath);
    const fileData = imageBinary.toString('hex');
    const result = [];
    for (let i = 0; i < fileData.length; i += 2) {
      result.push(parseInt(`${fileData[i]}${fileData[i + 1]}`, 16))
    }
    return new Buffer(result);
  }

  async analyzeImageByUrl(imageURL, opts) {
    const url = this.baseUrl + "analyze/" + "?"
      + "returnFaceId=true"
      + "&visualFeatures=Tags,Description,Faces,Color"
      + "&returnFaceLandmarks=false"
      + "&returnFaceAttributes=age,gender,headPose,smile,facialHair,glasses,emotion,hair,makeup,occlusion,accessories,blur,exposure,noise";

    try {
      const response = await fetch(url, {
        method: 'POST',
        headers: {
          "Content-Type": "application/json",
          "Ocp-Apim-Subscription-Key": this.apiKey
        },
        body: JSON.stringify({ url: imageURL })
      });

      const json = await response.json();
      return Promise.resolve(json);
    } catch (e) {
      return Promise.reject(e);
    }
  }

  async analyzeImage(imageBinary, opts) {
    const url = this.baseUrl + "analyze/" + "?"
      + "returnFaceId=true"
      + "&visualFeatures=Tags,Description,Faces,Color"
      + "&returnFaceLandmarks=false"
      + "&returnFaceAttributes=age,gender,headPose,smile,facialHair,glasses,emotion,hair,makeup,occlusion,accessories,blur,exposure,noise";

    try {
      const response = await fetch(url, {
        method: 'POST',
        headers: {
          "Content-Type": "application/octet-stream",
          "Ocp-Apim-Subscription-Key": this.apiKey
        },
        body: imageBinary
      });

      const json = await response.json();
      return Promise.resolve(json);
    } catch (e) {
      return Promise.reject(e);
    }
  }
}

module.exports = CognitiveServiceAnalyzeImage;
```

When we now look into the `output/result.csv` file after running `node index.js`, we will see something like the below, showing that we have our data as we want it for the next step.

```csv
shortcode;url;likes;comments;tags;
BWAaIRoA-P7;https://instagram.fbru2-1.fna.fbcdn.net/vp/c5778b6ba5ec4397f23885d0ddc399ec/5C01F7C2/t51.2885-15/e35/19534728_388790801522556_2083772609937276928_n.jpg;1087963;4176;sky,outdoor,water,beach,nature,shore,sunset,clouds,promontory,rock,day,sandy;
BV-mVxyBlvv;https://instagram.fbru2-1.fna.fbcdn.net/vp/1a9647a992a0245b6c3ae652804a5013/5BFB7E57/t51.2885-15/e35/19623518_280508059085489_4094393732026073088_n.jpg;875193;3090;water,outdoor,green,sport,sunset,setting;
...
```

## Step 2 - Training our AI Model

Because we are using ML.NET, our model is straightforward. We will consume our CSV File and parse it by using a `TextLoader` which will detect our schema from the `.csv` header.

After that we split our `Tags` column on the character `,` and create a `OneHotVector` with it. This will take all the unique occurrences of our different tags (example: indoor,person,sun,sea,...) and will set a 1 if it occurs in that row. Giving us the following table for the tags given:

||sky|outdoor|water|beach|nature|
|-|-|-|-|-|-|
|Row #1|1|0|0|0|0|
|Row #2|0|1|1|0|0|
|Row #3|0|1|1|0|1|
|Row #4|0|0|1|0|1|

Now we train with this `OneHotVector` as our `Feature`, and tags as a `Label`, through a `StochasticDualCoordinateAscentRegressor` as our trainer, saving the model at the end as a simple `.zip` file.

To run this, just use `dotnet run` and wait for it to be done, you will see something as in:

```bash
PS E:\<MyPath>\Step2.DataTraining> dotnet run
=================== Training Model Likes ===================
Automatically adding a MinMax normalization transform, use 'norm=Warn' or 'norm=No' to turn this behavior off.
Using 2 threads to train.
Automatically choosing a check frequency of 2.
Auto-tuning parameters: maxIterations = 2114.
Auto-tuning parameters: L2 = 0.001.
Auto-tuning parameters: L1Threshold (L1/L2) = 0.25.
Using best model from iteration 74.
Not training a calibrator because it is not needed.
=================== Training Model Comments ===================
Automatically adding a MinMax normalization transform, use 'norm=Warn' or 'norm=No' to turn this behavior off.
Using 2 threads to train.
Automatically choosing a check frequency of 2.
Auto-tuning parameters: maxIterations = 2114.
Auto-tuning parameters: L2 = 0.001.
Auto-tuning parameters: L1Threshold (L1/L2) = 0.25.
Using best model from iteration 48.
Not training a calibrator because it is not needed.
```

### Step 2 - Code 

**Program.cs**

```csharp
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Models;
using Microsoft.ML.Trainers;
using Microsoft.ML.Transforms;
using Microsoft.ML.Runtime.Api;
using Step2.DataTraining.Models;

namespace Step2.DataTraining
{
    public class Program
    {
        static string dataPath = "../Step1.DataPrep/output/result.csv";
        static void Main(string[] args)
        {
            // Features: is our input (tags is our input -> based on what do we predict?) 
            // Labels: is our output (comments is our output -> what do we want to predict?)
            Console.WriteLine("=================== Training Model Likes ===================");
            CreateModelLikes().Wait();
            Console.WriteLine("=================== Training Model Comments ===================");
            CreateModelComments().Wait();
        }

        public static async Task CreateModelLikes()
        {
            // 1. Load in our data which has the header on the top and we use a csv so ';'
            var pipeline = new LearningPipeline();
            pipeline.Add(new TextLoader(dataPath).CreateFrom<InstagramData>(useHeader: true, separator: ';'));

            // 2. Prepare our Label (what do we want to predict?)
            //      ColumnCopier: Duplicates columns from the dataset 
            //          -> we need to do this since Label is the column used and ours is called LabelLikes
            pipeline.Add(new ColumnCopier(("LabelLikes", "Label")));

            // 3. Prepare our features (based on what do we predict?)
            // Process Tags in a OneHotVector
            //      WordTokenizer: splits the text into words using the separator characters
            //      CategoricalOneHotVectorizer: We create a table of the different tags and set a 1 where the tag appears in our dataset
            pipeline.Add(new WordTokenizer("Tags") { TermSeparators = "," });
            pipeline.Add(new CategoricalOneHotVectorizer("Tags"){ OutputKind = CategoricalTransformOutputKind.Bag } );

            // Prepare Features to train with, in this case with the "Tags" of our instagram pictures
            //      ColumnConcatenator: Concatenates one or more columns of the same item type
            var features = new string[] { "Tags" };
            pipeline.Add(new ColumnConcatenator("Features", "Tags"));

            // 4. Train our model
            pipeline.Add(new StochasticDualCoordinateAscentRegressor());
            PredictionModel<InstagramData, PredictedLikes> model = pipeline.Train<InstagramData, PredictedLikes>();

            // 5. Save the trained model
            await model.WriteAsync("model_likes.zip");
        }

        // ==================================================================
        // Do the same for CreateModelComments..., changing Likes to Comments
        // Omitted for readability
        // ==================================================================
    }
}
```

**Models/InstagramData.cs**

```csharp
// Imports...

namespace Step2.DataTraining.Models
{
    // input for prediction operations
    // - First 4 properties are inputs/features used to predict the label
    // - Label is what you are predicting, and is only set when training
    public class InstagramData
    {
        [Column(ordinal: "0")]
        public float Shortcode { get; set; }

        [Column(ordinal: "1")]
        public float URL { get; set; }

        // We set name: "Label" since we will predict this
        [Column(ordinal: "2", name: "LabelLikes")]
        public float LabelLikes { get; set; }

        [Column(ordinal: "3", name: "LabelComments")]
        public float LabelComments { get; set; }

        [Column(ordinal: "4", name: "Tags")]
        public string Tags { get; set; }
    }
}
```

**Models/PredictedComments.cs**

```csharp
// Imports...

namespace Step2.DataTraining.Models
{
    public class PredictedComments
    {
        [ColumnName("Score")]
        public float Score { get; set; }
    }
}
```

**Models/PredictedLikes.cs**

```csharp
// Imports...

namespace Step2.DataTraining.Models
{
    public class PredictedLikes
    {
        [ColumnName("Score")]
        public float Score { get; set; }
    }
}
```

## Step 3 - Consuming our AI Model

Our model is now trained and created, now we just have to consume it. For this we just load in the zip files through `ML.NET` and call the `.Predict` function on the model. To predict likes and comments based on Tags, we load in tags through the CLI and the `args` parameter in our Main method, resulting in:

```bash
PS E:\<MyPath>\Step3.DataPrediction> dotnet run indoor,sun
Predicted likes: 670824.3
Predicted comments: 5451.721
```

Which is close to the data that we used to train with (looking on sight and comparing values).

> **Note:** We are training with just 700 lines of data, production models have millions of lines of these data and will result in better predictions. Data is always key in AI algorithms.

### Step 3 - Code

```csharp
using System;
using System.Threading.Tasks;
using Microsoft.ML;
using Microsoft.ML.Runtime.Api;
using Step3.DataPrediction.Models;

namespace Step3.DataPrediction
{
    class Program
    {
        static readonly string modelPathComments = "../Step2.DataTraining/model_comments.zip";
        static readonly string modelPathLikes = "../Step2.DataTraining/model_likes.zip";

        static void Main(string[] args)
        {
            var tags = args[0];
            PredictLikes(tags).Wait();
            PredictComments(tags).Wait();
        }

        public async static Task PredictLikes(string tags)
        {
            var model = await PredictionModel.ReadAsync<InstagramData, PredictedLikes>(modelPathLikes);

            // Predict
            var prediction = model.Predict(new InstagramData
            {
                Tags = tags,
            });

            Console.WriteLine($"Predicted likes: {prediction.Score}");
        }

        public async static Task PredictComments(string tags)
        {
            var model = await PredictionModel.ReadAsync<InstagramData, PredictedComments>(modelPathComments);

            // Predict
            var prediction = model.Predict(new InstagramData
            {
                Tags = tags,
            });

            Console.WriteLine($"Predicted comments: {prediction.Score}");
        }
    }
}
```

I hope you enjoyed this article, let me know in the comments below on the results you achieved. For the final source code, feel free to check the repository at: [https://github.com/Xaviergeerinck/blog-instagram-correlator](https://github.com/Xaviergeerinck/blog-instagram-correlator)

Predicting Instagram Pictures Likes and Comments with .NET Core and ML.NET


A common thing in Node.js is to create a REST API consumer that will perform a REST call. Due to the rise of many REST APIs, you might find yourself in the situation that you want to execute 1000s of calls (for example: Azure Cognitive Services).

Now due to the limit of these services, you will often encounter being throttled and need to end up rewriting your project to work around these limitations. But how do we do this in Node.js?

Take the following example, where we are not throttling 1000 calls:

> In the example below, we simulate real REST calls with a wait between each, showing how they will be out of order!

```javascript
const delay = require('delay');

const start = async () => {
    let promises = [];

    for (let i = 1; i <= 1000; i++) {
        promises.push(new Promise(async (resolve, reject) => {
            // Delay between 0 and 500ms to simulate REST call
            await delay(Math.random() * 500);
            console.log(`Executing ${i}`);
        }))
    }
}

start();
```

But now we executed all of those 1000 calls instantly, what if we have a rate limit of X calls per second? If we start executing these promises (which will not execute all at the same time due to CPU core limitations / thread limitations - especially in Node.JS) we will hit this rate limit quite quickly.

We thus need to write a wrapper around this code above that will throttle it for us. A good basic idea here is to collect all our URLs into an array, loop over this array and do a Promise.all await at the end to await execution. How would this look in code?

> Note: Make sure to run `npm install delay` before running the code below.

```javascript
const delay = require('delay');

// Init our calls array -> in REST this would be URLs
let calls = [];
for (let i = 1; i <= 1000; i++) {
    calls.push(async (cb) => {
        // Delay between 0 and 500ms to simulate REST call
        await delay(Math.random() * 500);
        console.log(`Executing ${i}`);
        cb();
    });
}

// Total Call count for statistics
const totalCalls = calls.length;

// Set rate limit call count and time between those calls
const rateLimitCount = 10;
const rateLimitTime = 1000;

// Our Throttler
const start = async () => {
    // Go over them in a throttled way
    while (calls.length > 0) {
        console.log(`[${Math.abs(calls.length - totalCalls)}/${totalCalls}] Performing`);
        let callsToExecute = calls.slice(0, rateLimitCount);
        calls = calls.slice(rateLimitCount, calls.length);

        // Wrap our call in a promise and wait till cb is called
        // Note: We can also implement the promise earlier, but easier here for compatibility with older code
        let promises = [];
        callsToExecute.forEach((i) => promises.push(new Promise((resolve, reject) => i(resolve))));

        // Wait for all our alls to be done
        await Promise.all(promises);

        // Throttle time
        await delay(rateLimitTime);
    }

    console.log('Done!');
}

start();
```

The code above might look complex, but the only important part is the part within our `start()` method, which takes care of throttling all our calls.

Since we now made the basic Proof Of Concept, it is easy to transform this into a generic class that we can re-use throughout our different projects:

```javascript
const delay = require('delay');

/**
 * Example usage:
 * 
 * const start = async () => {
 *     // Create our throttler
 *     const throttler = new Throttler(calls, 10, 1000);
 *     await throttler.execute();
 * }
 *
 * start();
 *
 */
class Throttler {
    /**
     * 
     * @param {array} calls array of functions to be executed in batches of rateLimitCount every rateLimitCount
     * 
     * Example of how to create this array:
     * 
     * calls.push((isDone) => {
     *    isDone(); // Is our call done executing and did we process the result?
     * });
     *
     * @param {integer} rateLimitCount how many calls do we perform each time?
     * @param {integer} rateLimitDelay What is the delay to wait between each batch of calls?
     */
    constructor(calls = [], rateLimitCount = 10, rateLimitDelay = 1000) {
        this.calls = calls;
        this.callsCount = calls.length;
        this.rateLimitDelay = rateLimitDelay;
        this.rateLimitCount = rateLimitCount;
    }

    async execute() {
        // Go over them in a throttled way
        while (this.calls.length > 0) {
            console.log(`[${Math.abs(this.calls.length - this.callsCount)}/${this.callsCount}] Performing`);
            let callsToExecute = this.calls.slice(0, this.rateLimitCount);
            this.calls = this.calls.slice(this.rateLimitCount, this.calls.length);

            // Wrap our call in a promise and wait till cb is called
            // Note: We can also implement the promise earlier, but easier here for compatibility with older code
            let promises = [];
            callsToExecute.forEach((callIsDoneCallback) => promises.push(new Promise((resolve, reject) => callIsDoneCallback(resolve))));

            // Wait for all our alls to be done
            await Promise.all(promises);

            // Throttle time @todo: this is not the exact time but an artificial time, we actually need the time between latest call and finish.
            await delay(this.rateLimitDelay);
        }
    }
}

module.exports = Throttler;
```

Feel free to let me know in the comments below if you have problems or suggestions.


Creating a throttled REST API Consumer


While you are working with compiled code, you often find yourself in the situation that every bit of performance count (just think about GPU intensive operations or even game engines). For this reason, we often turn back to coding C++ coding to be as low-level as possible (seeing that writing pure ASM is just too hard).

But wouldn't it be nice if we were able to write C++ code and interface that through our Dotnet Core application? Well this is the case! Through something called **Native Interoperability** ([https://docs.microsoft.com/en-us/dotnet/standard/native-interop](https://docs.microsoft.com/en-us/dotnet/standard/native-interop?WT.mc_id=DT-MVP-5004263)) allowing us to interface with our C++ code through the *`DllImport` attribute found in the `System.Runtime.InteropServices` namespace.

Let's start by creating a small example project that makes use of this and will print a simple *"Hello World"* from our C# code, executed in the C++ part.

1. Create a folder called "Hello-World-Project"
2. In "Hello-World-Project" create a new Directory called `cpp`
3. In "Hello-World-Project" create a new console dotnet application with the output directory name `dotnet` (forgot how? check: [getting-started-with-dotnet-core](/getting-started-with-dotnet-core))

## Building our C++ Library

Since I like building my projects on WSL (Windows Subsystem for Linux), I utilize C++ compilation through `make`. Therefore I need to use `CMake` to compile my C++ project. To compile our C++ project with cmake, start by creating a `CMakeLists.txt` file. To this file add the following content:

```bash
# Configure our Required Version
cmake_minimum_required(VERSION 3.0)

# Define the name of our project
project(hello-cpp)

# Define the output library to be a shared file (.so or .dll) and output it as "(lib)hello-cpp.so"
add_library(hello-cpp SHARED HelloWorld.cpp HelloWorld.h)
```

We can now configure our project but still need our project files that actually define our "Hello World" method. For this create 2 files called `HelloWorld.cpp` and `HelloWorld.h` with the following content:

**HelloWorld.cpp**

```cpp
#include "HelloWorld.h"
#include <iostream>

void PrintHelloWorld() {
    std::cout << "Hello World!" << std::endl;
}
```

**HelloWorld.h**

```cpp
#ifdef __cplusplus
extern "C" {
#endif

#ifdef _WIN32
#  ifdef MODULE_API_EXPORTS
#    define MODULE_API __declspec(dllexport)
#  else
#    define MODULE_API __declspec(dllimport)
#  endif
#else
#  define MODULE_API
#endif

MODULE_API void PrintHelloWorld();

#ifdef __cplusplus
}
#endif
```

We can see that our `.cpp` file is quite straightforward, defining the method body as written in the Header file. What we however notice is that a lot of code was added to our `.h` file in the form of a `Macro` identified by `#define`. This is needed because Windows uses the `__declspec(dllexport)` function, while as Unix systems don't use this (more information: [https://docs.microsoft.com/en-us/cpp/cpp/declspec](https://docs.microsoft.com/en-us/cpp/cpp/declspec?WT.mc_id=DT-MVP-5004263)). The macro does nothing more than making our life easier, so we just have to write `MODULE_API` in front of our methods to export them and make them available in our C# code.

As a last step, we just have to build our library so that we are able to use it in our C# code, execute the following in the cpp folder:

```bash
# Configure the MakeFile
cmake .

# Build with all cores available (Mac OS)
make -j$(sysctl hw.ncpu | cut -d: -f2)

# Build with all cores available (Linux)
make -j$(grep -c ^processor /proc/cpuinfo)
```

> Note: For windows, using cmake is a bit more complicated. You can however get towards using it by installing it through `choco install cmake` so that you can run cmake like that.

## Building our C# Dotnet Core Application

Now move to your `dotnet` directory, and add the following code to your `Program.cs`:

```C#
using System;
using System.Runtime.InteropServices;

namespace CppBind
{
    class Program
    {
        [DllImport(@"../cpp/libhello-cpp.so")]
        public static extern void PrintHelloWorld();

        static void Main(string[] args)
        {
            PrintHelloWorld();
        }
    }
}
```

What we do here, is that we use the `DllImport` attribute to include our `.so` library that was created earlier. By defining this method in the upper part of our class as an extern, we're then able to call this in other functions as we did in our `Main` function here.

Eventually running this with `dotnet run` will result in the following beautiful output:

![./binding-cpp-with-dotnet-core-1.png](./binding-cpp-with-dotnet-core-1.png)