Introduction

For the last few days, I am playing with my Azure IoT Dev Kit MXChip. In this article, we are going to see how we can set up an Azure Function as an Output job topology of an Azure Stream Analytics job. Isn’t that sound interesting? In our previous articles, we have already seen what is an Azure Stream Analytics Job and How can we create in by using the portal and Visual Studio. If you haven’t read those articles, I strongly recommend you to read. Let’s jump on to this article now.

Background

As I have mentioned earlier, in this article we will be, 

1. using our existing Azure Stream Analytics job.
2. creating a new Azure Function App.
3. setting up the newly created Azure function as an output job topology of the stream analytics job.
4. monitoring the data coming to the Azure Function from the stream analytics job.

Play with Azure Function

Yeah, we are going to play with it. Let’s go and create one then.

Creating an Azure Function

To create an Azure Function application, you need to login to your Azure portal and click on the Create a resource icon, and then you can search for the ” Function App”. 

In the next screen, provide the following information. 

1. App Name
2. Subscription
3. Resource Group
4. OS
5. Hosting plan
6. Location 
7. Runtime stack
8. Storage
9. Application Insights

Here the consumption plan hosting plan allows you to pay per execution, and the App service plan allows you to have a predefined capacity. For the runtime stack, we will use .NET, however, you are free to use anything you wish. 

Once you have created the same, you should be able to see it under the Function Apps section. 

Creating an Azure Function Solution and Function

Now let’s go to our Visual Studio and create a new solution for our Azure Function. 

Now you can right click on your newly created project and add a new HttpTrigger Function. We will keep the Access Rights to Anonymous for now. I have named my function as “GetData”. For now, let’s just get the data from our Stream Analytics job and just check the length. 

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;

namespace ml.IoTPlatform.AzureFunctions
{
    public static class GetData
    {
        [FunctionName("GetData")]
        public static async Task<HttpResponseMessage> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post")]HttpRequestMessage req,
            ILogger log)
        {
            log.LogInformation($"GetData function triggered with Uri {req.RequestUri}");

            string content = await req.Content.ReadAsStringAsync();
            log.LogInformation($"String content is {content}");
            dynamic data = JsonConvert.DeserializeObject(content);

            log.LogInformation($"Data count is {data?.Count}");

            if (data?.ToString()?.Length > 262144)
            {
                return new HttpResponseMessage(HttpStatusCode.RequestEntityTooLarge);
            }

            return req.CreateResponse(HttpStatusCode.OK, "Success");
        }
    }
}

As you can see we are not doing nothing much for now, we are just receiving the data as HttpRequestMessage and we are reading the content as req.Content.ReadAsStringAsync() and then deserialize the object. If you are not doing this step, you may get an error as “No MediaTypeFormatter is available to read an object of type ‘Object’ from content with media type ‘application/octet-stream’.“

We are also checking the entity length, and if it is too large we are sending a HttpResponseMessage with status code 413. 

Publish the Azure Function App

To publish your Azure Function app, just right click on your project and click Publish and then set up your publish target by choosing the existing Azure Function App, remember we have created on earlier? Once you publish the same, you can go into your Function App and see your Function. You can also test the same with some dummy data. 

There are probabilities to get an error as “Web Deploy cannot modify the file on the destination because it is locked by an external process” when you try to publish your Function App from Visual Studio, while your Function App is running, to fix this you can see my answer here. 

Azure Stream Analytics Job

Let’s go back to our Azure Stream Analytics now as we have already configured our Azure Function App successfully. 

Configure Azure Function Output

In my previous article, we had created an Azure Stream Analytics job solution using Visual Studio, let’s open that solution now and configure the new output for Azure Function. 

While configuring the Azure Function Output, please make sure that you are selecting the existing azure function app.

Update the Script

We should also do some changes in our Script.asaql file to support our newly created output. 

WITH BasicOutput AS 
(
SELECT    
    messageId,
    deviceId,
    temperature,
    humidity,
    pressure,
    pointInfo,
    IoTHub,
    MAX(EventEnqueuedUtcTime) AS EventEnqueuedUtcTime,
    EventProcessedUtcTime,
    PartitionId    
FROM
    Input TIMESTAMP By EventEnqueuedUtcTime
    GROUP BY TUMBLINGWINDOW(second, 10), 
    messageId, 
    deviceId,
    temperature,
    humidity,
    pressure,
    pointInfo,
    IoTHub,
    EventEnqueuedUtcTime,
    EventProcessedUtcTime,
    PartitionId
)

SELECT * INTO SQLServerOutput FROM BasicOutput
SELECT * INTO AzureFunctionOutput FROM BasicOutput

Updating the TLS Version

Once that is done, just click the button Submit to Azure, if you have any doubts in this section, read my previous posts on this topic. Now let’s log in to the portal again and see all the outputs, inputs, and the query is been published or not.

Cool!. Well done, it seems like it is published. Now if you click on the AzureFunctionOutput, you may get a warning as “Please make sure that the Minimum TLS version is set to 1.0 on your Azure Functions before you start your ASA job“. I would rather treat this as an error instead of a warning because without making this changes our Azure Stream Analytics job will not write to our Azure Function. So this is very important, I spent many hours in this and finally found this was the root cause of my issue, you can see my answer about this here.

So just go to your Azure Function App and click on Platform Features -> SSL -> Minimum TLS Version

There is a saying that developers don’t care about warning but only the errors, in some cases it is true. Hm, I was just kidding. 

Output

Once you are done everything mentioned, you are good to go and start your Stream Analytics job, please make sure that your MXChip is connected to a power source so that the device can start sending the data. 

Checking the SQL Server Output

Now let’s login to our SQL Server Database and run the below query to make sure that we are getting the data from the device.

SELECT TOP (1000) [Id]
      ,[messageId]
      ,[deviceId]
      ,[temperature]
      ,[humidity]
      ,[pressure]
      ,[pointInfo]
      ,[IoTHub]
      ,[EventEnqueuedUtcTime]
      ,[EventProcessedUtcTime]
      ,[PartitionId]
  FROM [dbo].[StreamData] order by [EventEnqueuedUtcTime] desc

Checking Azure Function Output 

To check the Azure Function Output, we can go back to our Azure Function and click on the Function and use the Monitor option.

Please be noted that you can always check your Azure Stream Analytics job Activity Log if you found something is not working. 

Conclusion

In this article, we have learned how to,

1. work with an Azure Stream Analytics job
2. create an Azure Function App
3. create Azure Function App solution in Visual Studio
4. write an HttpTrigger function and publish the same to the Azure Function App
5. set up the Azure Function App as an output job topology of Azure Stream Analytics job
6. Use the created package in another solution

In our next article, we will see how you can send this Azure Function Output data to an Azure SignalR service and then get the same data in an Angular Application. I can’t wait to write my next article. 

Your turn. What do you think?

Thanks a lot for reading. I will come back with another post on the same topic very soon. Did I miss anything that you may think which is needed? Could you find this post as useful? Kindly do not forget to share me your feedback.

You can always see my IoT articles here. 

Kindest Regards
Sibeesh Venu

Introduction

Creating an Azure Stream Analytics job and working with them is always a fun. And what if there is a tool which helps you create a stream solution in Visual Studio so that you don’t want to go over the browser and do some manual button clicks? That’s where the Azure Data Lake and Stream Analytics Tool extension come into the play. In my previous post, we have already discussed what is an Azure Stream Analytics job and how can we work with the same, if you haven’t read the same, please read it. Now let’s go and use the extension.

Background

Recently, I started working with Azure IoT Device Kit, which is a wonderful micro controller board, it can send a lot of information like temperature, humidity to the Azure cloud, as it has a lot of inbuilt sensors. Now in this article, we will create a Visual Studio solution for our Stream Analytics Job so that the same can be moved to a source control and can be easily managed. 

Please be noted that the Extension we are going to use is not yet supported with Visual Studio 2019, and if you try to create the same in VS2019, you will get an error as the version of Visual Studio is not supported

The extension is not supported in VS2019

Setting up the Environment

I am assuming that you have a valid Azure Subscription and you have access to create the resouces in it. 

Modify Visual Studio with Required Workloads

Let’s go to our Visual Studio installer and modify the workloads now.

Once the workloads are modified, make sure that the extension is available in your Extensions, you can do that by going to Tools -> Extension menu.

Creating a New Stream Analytics Project

Once the Extension is enabled, you should be able to create a new Azure Stream Analytics Application.

The new project will be containing the below files.

1. Input.json
2. Output.json
3. JobConfig.json
4. Script.asaql

To configure your subscription, please make sure that you have added your Subscription in the Server Explorer.

The Input.json file is the replica of your Input job topology, double click on the file will give you the configuration page to configure the details. 

The Output.json file is your output job topology, you can have as many outputs you need. In my case it is just one, SQL Server Database. 

You can always configure your job using the JobConfig.json file. When you configure the Job, you need to be sure about the values you are providing and what are the needs of them. 

Data Locale is your locale input. The option Output Error Handling is for handling the situation when the events fail to be written to the output, you can select either Drop or Retry. The Late Arrival Tolerance Window is the timeframe which you can wait for the event to reach the IoT hub, the time difference is between the event timestamp and the system time. 

And the Script.asaql is the file where you need to add your custom query which gets data from the input and send it to the output. 

SELECT
    messageId,
    deviceId,
    temperature,
    humidity,
    pressure,
    pointInfo,
    IoTHub,
    EventEnqueuedUtcTime,
    EventProcessedUtcTime,
    PartitionId
INTO
    streamoutputs
FROM
    streaminputs

Once everything is done, you are ready to submit the same to the Azure. You can either create a new job or use the existing one. 

When you submit, you can see that a new Stream Analytics view will get opened and the job will be starting automatically. You can always see the blobs created under your container by going to the cloud explorer. 

Now just right click on your solution and select “Add solution to Source Control and then push the same to your git repository. Once you have added the solution to the source control, your team members can easily update the Input and Output configuration and have a history of the same. 

Conclusion

In this article, we have learned how to,

1. set up the Visual Studio for using Data Lake and Stream Analytics tool
2. use the Data Lake and Stream Analytics tool
3. configure the Input for Stream Analytics
4. configure the Stream Analytics Job
5. Use the created package in another solution

Your turn. What do you think?

Thanks a lot for reading. I will come back with another post on the same topic very soon. Did I miss anything that you may think which is needed? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu

Introduction

The capability of an Azure Stream Analytics Job is a lot, here in this post we are going to discuss a few of them. An Azure Stream Analytics is basically an engine which processes the events. These events are coming from the devices we have configured, it can be an Azure IoT Dev Kit (MXChip) or a Raspberry Pi and many more. The stream analytics job has two vital parts

• Input source
• Output source

The input source is the source of your streaming data, in my case, it is my IoT Hub. And the output source is the output what you are configuring. I had configured the output to save the data to an Azure SQL database. Let’s just stop the introduction part now and start creating our own Stream Analytics. 

Background

I recently got my MXChip (Azure Iot Dev Kit) and I was surprised with the capabilities that device can do. It has a lot of sensors within the device, like temperature, humidity, pressure, magnetometer, security etc. Then I thought it is time to play with the same. So the basic idea here was to,

1. Configure the device to send the data to the IoT Hub
2. Select the IoT Hub as a stream input
3. Send the output to an SQL Server database

In this article, we are going to concentrate on how to create a Stream Analytics Job and how you can configure the same to save the stream data to the SQL Server database. 

Prerequisites

To do the wonderful things, we always need some prerequisites.  

1. Azure Subscription
2. MXChip Azure IoT Dev Kit 
3. An active IoT Hubows Driver Kit (WDK) 10
4. IoT Core ADK Add-Ons
5. Windows 10 IoT Core Packages
6. The Raspberry Pi BSP
7. Custom FFU image we have created

Creating the Azure Stream Analytics Job

Login to your Azure Portal and click on the Create a resource, and then search for the “Stream Analytics job”. 

Once you clicked on the Create button, it is time to specify the details of your job.

1. Job Name
2. Subscription
3. Resource Group
4. Location
5. Hosting Environment

I would strongly recommend you to select the same resource group of your IoT Hub for the Stream Analytics Job as well so that you can easily delete the resources when there are not needed. Once the deployment is successful you can go to the resource overview and see the details.

Configure Inputs

In the left menu, you can see a section called Job topology, that’s where we are going to work. Basically, we will be setting the Inputs and Outputs and then we will be writing a query which can take the inputs and send the values to the configured output. Click on the Inputs label and click on Add stream input and then select the IoT Hub.

In the next screen, you will have options to select the existing IoT hub and to create a new IoT Hub. As I have already created an IoT hub, I would select the existing one.

Please be noted that you are allowed to use special characters in the Input alias field, but if you use such, please make sure to include the same inside [] in the query, which we will be creating later. 

About the special characters in Input alias field

Once you are successfully configured the Inputs, then we can go ahead and configure the outputs.

Configure Outputs

Click on the Outputs from the Job topology section and click Add, and then select SQL Database.

You can either create a new Database or select the one you had already created. I used the existing database and table. 

Configure the Query

Once you click the label Query on the left pan, you will be given an editor where you can write your queries. I am using the below query. 

SELECT
    messageId,
    deviceId,
    temperature,
    humidity,
    pressure,
    pointInfo,
    IoTHub,
    EventEnqueuedUtcTime,
    EventProcessedUtcTime,
    PartitionId
INTO
    streamoutputs
FROM
    streaminputs

As you can see that I am just selecting the fields I may need and saving the same to our stream outputs. You can always select all the fields by using the select * query, but the problem with that is, you will have to set up the table columns in the same order of the stream data. Otherwise, you may get an error as below.

Encountered error trying to write 1 event(s): Failed to locate column ‘IoTHub’ at position 6 in the output event

Stream analytics query error

If there are any errors, you can see that in the Output details.

Run the Stream Analytics Job and See the Data in the Database

As we have already done the initial set up, we can now start our Stream Analytics Job, please make sure that the IoT Hub is running and the device is sending data to the IoT Hub. If everything is working as expected, you will be able to see the data in the SQL server database. You can either connect your MXChip device to the network and test this or use the custom simulator app. 

If you are using the Simulator console application, make sure that you are giving the device id, key and the IoT hub uri correctly, otherwise you will get an unauthorized error as explained here. 

Test the Stream Analytics Job Inside the Portal

You also have an option to test the functionality in the portal itself. The only thing you will have to do is to prepare the sample input data. I have prepared the sample JSON data as follows. 

[
  {
    "deviceId": "test-device",
    "humidity": 77.699449415178719,
    "pointInfo": "This is a normal message.",
    "temperature": 32.506656929620846
  },
  {
    "deviceId": "test-device",
    "temperature": 52.506656929620846,
    "humidity": 17.699449415178719,
    "pointInfo": "This is a normal message."
  },
  {
    "deviceId": "test-device",
    "temperature": 42.506656929620846,
    "humidity": 57.699449415178719,
    "pointInfo": "This is a normal message."
  }
]

Now we can go to the Query section and upload the sample data file for our inputs. 

In the next window, you can select the JSON option and upload your JSON file. 

Click the Test button, and now you should be able to see the output as below.

Conclusion

Wow!. Now we have learned,

• What is Azure Stream Analytics Job
• how to create Azure Stream Analytics Job
• how to add Inputs to the Azure Stream Analytics
• how to add Outputs to the Azure Stream Analytics
• how to add custom Query in Azure Stream Analytics
• how to Test the Stream Analytics Query with sample data

You can always ready my IoT articles here.

Your turn. What do you think?

Thanks a lot for reading. Did I miss anything that you may think which is needed in this article? Could you find this post as useful? Kindly do not forget to share me your feedback.

Kindest Regards
Sibeesh Venu