Tutorial 2. Azure Lab: Azure Stream Analytics

2.1 Using Azure’s Stream Analytics with Twitter

First, you need to sign up for the Azure Free Trial. Go to this site (you have to enter credit card information) and sign up for the trial: https://account.windowsazure.com/SignUp.

You can follow the lab online or here; I’ll try to give more pictures and in depth explanation about what is going on. Here is where you’ll find the lab online: https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-twitter-sentiment-analysis-trends/.

Now, head to https://manage.windowsazure.com/ and we’ll start the lab.

2.1.1 Getting Twitter OAuth Keys and Downloading the Twitter Client

Luckily, we’ve already retrieved our OAuth keys. The Twitter Client we’ll be using to conduct the lab experiment can be downloaded by clicking here. You can download the source code for the application here.

2.1.2 Creating an Event Hub and Consumer Group

An Event Hub is a data transfer solution provided by the Service Bus. It creates a common hub in the cloud where events are published and then can be read on a very large scale.

In the Azure Portal click New > App Services > Service Bus > Event Hub > Quick Create
- Give the hub a name and a namespace that are easy to remember.
Create the New Event Hub
To create a new Consumer Group click your namespace name (highlighted in yellow)
Click “Event Hubs” (purple) > “Create Consumer Group” (red)
Click the name of your Event Hub > “Configure” (brown) tab > Create new shared access policy with manage permissions (purple) > Save (green) > Dashboard tab (yellow)
Click “Connection Information” at the bottom of the screen > Copy the “Connection String”

2.1.3 Configuring the Twitter Client Application

If you haven’t yet downloaded the TwitterClient application you can download it here. Now, we’ll be configuring the application to interface with the Service Bus that we just created.

Unzip the Twitter Client Application.
Open the TwitterClient.exe.config file. Notepad or a similar application works fine; do not use Microsoft Word or other applications similar to that.
Input your Twitter OAuth keys. It should look like this:
Input your “Connection Information” from the Event Hub you created.

This is where some errors could occur. The “Connection String” you copied should look something like this (but all in one line):

You have to paste this inside the “EventHubConnectionString” value’s quotes:

Then, you have to enter the actual name of your Event Hub NOT the name of your policy, so mine was simply “twitterhub”. Here is what mine looked like in the end:

Run “TwitterClient.exe” to see if you entered the information correctly. If all went well this is what you should see when you run the program:

*Possible Errors:
– If the console opens and then closes immediately, you entered the Twitter OAuth keys incorrectly.
– If the console opens, but nothing happens, you entered the Event Hub information incorrectly.

2.2 Stream Analytics

Now, we must set up stream analytics to analyze the data that we’re sending out.

2.2.1 Creating the Stream Analytics job

In the Azure Portal click New > Data Services > Stream Analytics > Quick Create. Input all the necessary information just as I do.
Select your newly created Stream Analytics Job.

2.2.2 Specifying Job Input

Click Inputs in the top menu. Then Add Input.
Select “Data Stream”, then continue.
Select “Event Hub”, then continue.
Enter the following values on the third page, then continue:
- Input Alias: Select a friendly name. You will use this later in the query.
- Subscription: “Use Event Hub from Current Subscription”
- Choose a Namespace: Choose your namespace. Mine was “TwittterHubRyanBury-ns (Central US)”.
- Choose an Eventhub: Choose your Eventhub. Mine was “twitterhub”.
- Event Hub Policy Name: Choose the policy we made earlier. Mine was “TwitterHubPolicy”.
- Choose a Consumer Group: Choose the consumer group we made earlier. Mine was “twitterhubgroup”.
Enter the following:
- Event Serializer Format: JSON
- Encoding: UTF8
Click the check to add the source and test the connection with the event hub.

2.2.3 Specifying Job Query

First, we need to receive some sample data to run some test queries with.

Run the “TwitterClient.exe” and let it run for at least 5 minutes before you go any further. This allows some events to collect within the Event Hub.
Select INPUTS after selecting your Stream Analytics job.
Select SAMPLE DATA at the bottom of the page.
The default selections should be adequate for collecting enough data.
After running click the DETAILS Then click the link to download the .JSON file with your data. Save this file in an easily accessible place; we will use it next.

We’ll run a couple of different queries to get familiar with the process. If you would like to learn more about how to run these queries, the reference is here.

Select your newly created Stream Analytics job. I selected “twitterstreamanalytics”.
Click QUERY at the top of the job’s page.
In the editor replace the query with this:
Click TEST under the query.
Select the sample .JSON file we created earlier.
Click the check button to view the results. They might be quite large.

Now we’ll count the number of tweets per topic every 5 seconds:

Change the query to:
Click RERUN under the query editor to view the results.

2.2.4 Finding Trending Topics

For this query we need a rather large dataset so you may have to go back to where we collected sample data and retrieve a larger set. Here is how you’ll find trending topics:

Change the query to:
Click Test and make sure you select a dataset large enough to find some trending topics. With my count (the number bold and red) at 20 I only found one trending topic. If I changed it to 15 I received 7 results.

2.2.5 Find Sentiment Statistics

Here we’ll collect various statistical measures such as number of mentions, average, minimum, maximum, and standard deviation of sentiment score for each topic.

Change the query to:
RERUN the query to view the results.
This is the query we’ll be using so click SAVE at the bottom of the screen.

2.2.6 Create output sink

Now we’ve done everything we have to except specifying the output of our application. You could push (store) the results to an SQL Database, Table Store, or Event Hub, but we’ll be pushing them to an Azure Blob storage.

Select the Storage Account we created earlier. Mine was named “twitterstreamstorage1”. If you never created one click NEW > DATA SERVICES > STORAGE > QUICK CREATE > and follow the on-screen instructions.
In the Storage Account click CONTAINERS at the top of the page and add a container.
Create a name for your container and set the access to “Public Blob”.

2.2.7 Specify Job Output

In your Stream Analytics job, click OUTPUT at the top of the page and add an output.
Select “Blob Storage”.
Choose the following values:
- Output Alias: Enter a friendly name for the output.
- Subscription: This should be selected for you to “Use Storage Account from Current Subscription”.
- Storage Account: This should also be selected for you to the name of your storage account.
- Container: Select your container if it isn’t already selected.
- Filename Prefix: Type in a simple file prefix. Example: “module1”.
Click the right button.
Specify these values:
- Event Serializer Format: CSV (CSV is easier to use with Power BI, you could select JSON)
- Encoding: UTF8
Click the check button to create the output.

2.2.8 Starting the Job

Navigate to the dashboard of your streaming job and click START at the bottom of the screen.
Make sure that Job Start Time is selected and click the checkmark. The job will start running.

* Make sure that your “TwitterClient.exe” is running before you start the twitterstream otherwise nothing will happen as the job is running!

If all went well your dashboard should have some graphs like this indicating that events are being processed:

Continue with Tutorial 3 …

Tutorial 3 will go over viewing your output and organizing the results in a meaningful way. It will also give some extra activities to try using the Stream Analytics.