Real Time Connected Vehicle Networking with HDInsight and Apache Storm

Building Client Apps with C# and Xamarin
Why Xamarin ? Build Once works universally across platforms on billions of devices..

Basic Hardware –OBD-II Port Connectivity
An ODB-II Elm 327 type interface (connected to
USB, Bluetooth or wi-fi) on your smart phone or
tablet connected to the diagnostic port below your
steering wheel, or broadband in your newer
vehicle. Newer vehicles now have optional
broadband in-vehicle.

Visual Studio Community Edition
Free IDE (Integrated Development Environment) for PCs Integrates with Xamarin for iOS,
Android and Mac Development.. Xamarin is free for students and affordable for Indy Developers

Polling Data From OBD-II
Your Xamarin Studio or Visual Studio project will need you to install an OBD manager to
communicate with your vehicle’s network of diagnostic codes.. I suggest installing the OBD-II
manager from Microsoft’s Coding for Fun Channel 9 Project.. You can integrate this easily in your
app using either Xamarin Studio or Visual Studio’s NuGet Package Manager also available from
the NuGet gallery..

Communicating with the OBD-II Port
OBD-II stands for On-Board Diagnostics. Connecting to this port allows us to query for
different types of data from the car. We can get the current speed, RPMs, fuel level,
etc. for display in the Instrument Cluster and other locations.
For the vehicle, because only one application can open and communicate with a serial
port at one time, we created a WCF service that polls the OBD-II data from the car and
GPS and returns it to our application when it queries the service.
For the OBD library a manual connection will be used to poll different values at
different intervals. For values critical to driving the car—like RPM, speed, etc, it is
necessary to poll for the values as quickly as the vehicle can return them. With other
values which aren’t critical to driving the car—like the fuel level, engine coolant
temperature, can be polled at a 1-2 second interval. For GPS, we can create a code to
read the GPS and create a LocationChanged event, which would fire when the GPS
values changed.
Rather than creating a new serial port connection for every WCF request for OBD data,
it’s important to use a singleton service that is instantiated when the service first runs.
Only one object in the WCF service that represents the last OBD and GPS data
returned, which is obtained by the continual reading of the latest OBD data using the
OBD library as described above. Calls to a WCF service ReadMeasurement method
didn’t actually compute anything, but instead serializes the last saved data and
returned it via the WCF service.

Instantiating the OBD Service
_service = new ObdService();
_service.ObdMeasurementChanged += service_ObdMeasurementChanged;
_service.Start(new TimeSpan(0, 0, 0, 0, 1000), localhost, Protocol.Http,
false);
void service_ObdMeasurementChanged(object sender,
ObdMeasurementChangedEventArgs e)
{
Debug.Writeline("MPH=” + e.Measurement.MilesPerHour);
}
This code creates a new ObdService class and signs up for an event when the measurement has changed. The Start method does the
following:
• Set the interval that you want to poll the ObdService, in this case every second (to update the instrument cluster).
• Determines what IP address the service is hosted at (localhost), the protocol (HTTP or TCP), and whether to send “demo mode” data.
So we can have some sample data for testing.
To capture the car telemetry data like MPH, RPM, engine load, and throttle (accelerator) position, as well as location data (latitude, longitude,
altitude, and course), we used a SQL database with a flat Entity Framework model.
• The primary key, the ObdMeasurementID is a GUID that is returned via the ObdService.
• The database logger subscribes to the ObdMeasurementChanged event and receives a new reading at the time interval set in the Start()
method.

public override string RowKey
{
get { return new DateTimeOffset(TimeStamp).ToEndOfDays(); }
set { }
}
public static class DateTimeExtensions
{ public static string ToEndOfDays(this DateTimeOffset source)
{ TimeSpan timeUntilTheEnd = DateTimeOffset.MaxValue.Subtract(source);
return timeUntilTheEnd.Ticks.ToString();
}
public static DateTimeOffset FromEndOfDays(this String daysToEnd)
{ TimeSpan timeFromTheEnd = newTimeSpan(Int64.Parse(daysToEnd));
DateTimeOffset source = DateTimeOffset.MaxValue.Date.Subtract(timeFromTheEnd);
return source;
}
}
The Windows Azure data model uses Azure Table Services instead of SQL Server. The data mapping is essentially the same since both have a
flat schema.
For Azure Table Storage, in addition to the schema above, you also need a partition key and a row key. For the partition key, we used a
custom TripID (GUID) to represent a Trip. When the car is turned on/off a new TripID is created. That way we could group all measurements
for that particular trip and do calculations based on that trip, like the average miles per gallon, distance traveled, fastest speed, etc. For the
row key, we used a DateTimeOffset and a custom extension method, ToEndOfDays() that provides a unique numerical string (since Azure's
row key is a string type) that subtracts the time from the DateTime.Max value. The result is that the earlier a DateTime value, the larger the
number.
Example:
Time=5/11/2012 9:14:09 AM, EndOfDays=2520655479509478223 //larger
Time=5/11/2012 9:14:11 AM, EndOfDays=2520655479482804811 //smaller
Since they are ordered in reverse order, with the most recent date/time being the first row, we can write an efficient query to pull just the
first row to get the current latitude/longitude without needing to scan the entire table for the last measurement.

public ActionResult PostData()
{
try
{
StreamReader incomingData = new StreamReader(HttpContext.Request.InputStream);
string data = incomingData.ReadToEnd();
JavaScriptSerializer oSerializer = new JavaScriptSerializer();
List<MeasurementForTransfer> measurements;
measurements = oSerializer.Deserialize(data, typeof(List<MeasurementForTransfer>)) as
List<MeasurementForTransfer>;
if (measurements != null) {
CloudBlob blob = _blob.UploadStringToIncoming(data);
_queue.PushMessageToPostQueue(blob.Uri.ToString());
return new HttpStatusCodeResult(200);
}
}}
To upload data to Azure, we used a timer-based background uploader that would check to see if there was an internet connection, and then
filter and upload all of the local SQL rows that had not been submitted to Azure using the Submitted boolean database field. On the Azure
side, we used an ASP.NET MVC controller to submit data. The controller deserializes the data into a List<MeasurementForTransfer> type, it
adds the data to a blob, and adds the blob to a queue.
A worker role (or many) will then read items off the queue and the new OBD measurement rows are placed into Azure Table Storage.

Xamarin already has App Store Apps for OBD-II
Xamarin is already popular among App Store Developers
Matt O’Connor from OCTech, and creator of OBD Fusion, shared his app
that wirelessly reads onboard vehicle diagnostics from a car. The app
enables car enthusiasts to create their own dashboards and access much of
the same diagnostic information a master mechanic would use with
specialized automotive equipment to diagnose and tune an automobile.
With OBD Fusion, users can calculate fuel economy, graph data, and also
test emissions in real-time.
Matt originally created his app in C# as a desktop application. Utilizing
Xamarin, he was able to convert it to iOS and Android with over 90% code-
reuse, and by using app templates he’s created native smartphone and
tablet user experiences. In addition to the automobile’s sensor data, he’s
also integrated sensors on the mobile device, including the GPS and
accelerometer to measure position and vehicle acceleration.
https://www.youtube.com/watch?v=Sd3N8zrD7qs

Client App (Tablet Dashboards)
Starting Point
http://obd.codeplex.com/

Creating a OBD-II Client Dashboard
Creating a dashboard is a lot like
game programming, to move the
needle updates at reasonable speed
the code needs to be setup as a
game loop as you are seeing here in
C Sharp for their XAML interface..
To do this in Xamarin, we can use
monogame.

Creating a OBD-II Client Dashboard
using System;
using System.Windows.Controls;
using System.Windows;
using System.Windows.Media;
using System.ComponentModel;
using System.Windows.Media.Animation;
namespace Coding4Fun.Obd.InstrumentCluster.Classes
{
public class Needle : UserControl
{
private const double SLOWNESS = 5;
Storyboard uxSB_GameLoop;
FrameworkElement uxNeedle;
public RotateTransform Needle_RotateTransform = new RotateTransform();
public TransformGroup tg = new TransformGroup();
private double desiredAngle;
public Needle()
{
Loaded += new System.Windows.RoutedEventHandler(Needle_Loaded);
Initialized += new EventHandler(Needle_Initialized);
}
void Needle_Initialized(object sender, EventArgs e)
{
uxNeedle = (FrameworkElement)FindName("uxNeedle");
uxSB_GameLoop = (Storyboard)TryFindResource("uxSB_GameLoop");
}
void Needle_Loaded(object sender, System.Windows.RoutedEventArgs e)
{
if (uxNeedle != null) uxNeedle.RenderTransform = Needle_RotateTransform;
if (uxSB_GameLoop != null)
{
uxSB_GameLoop.Completed += new
EventHandler(uxSB_GameLoop_Completed);
uxSB_GameLoop.Begin();
}}
void uxSB_GameLoop_Completed(object sender, EventArgs e)
{
Needle_RotateTransform.Angle += (desiredAngle -
Needle_RotateTransform.Angle) / Math.Max(SLOWNESS, 1);
uxSB_GameLoop.Begin();
}
public void UpdateNeedle()
{
desiredAngle = Maximum == Minimum ? Maximum : MinAngle + (MaxAngle -
MinAngle) * (Math.Min(Math.Max(Value, Minimum), Maximum) - Minimum) /
(Maximum - Minimum);
if (uxSB_GameLoop == null)
{
Needle_RotateTransform.Angle = desiredAngle;
}
}
private static void OnValuesChanged(DependencyObject d,
DependencyPropertyChangedEventArgs e)
{
Needle ln = (Needle)d;
ln.UpdateNeedle();
}
[Category("Common Properties")]
public double Value
{
get { return (double)GetValue(ValueProperty); }
set { SetValue(ValueProperty, value); }
}
public static readonly DependencyProperty ValueProperty =
DependencyProperty.Register("Value", typeof(double), typeof(Needle), new
PropertyMetadata(new Double(), new PropertyChangedCallback(OnValuesChanged)));
public double Minimum
{
get { return (double)GetValue(MinimumProperty); }
set { SetValue(MinimumProperty, value); }
}
public static readonly DependencyProperty MinimumProperty =
DependencyProperty.Register("Minimum", typeof(double), typeof(Needle), new
public double Maximum
{
get { return (double)GetValue(MaximumProperty); }
set { SetValue(MaximumProperty, value); }
}
public static readonly DependencyProperty MaximumProperty =
DependencyProperty.Register("Maximum", typeof(double), typeof(Needle), new
public double MinAngle
{
get { return (double)GetValue(MinAngleProperty); }
set { SetValue(MinAngleProperty, value); }
}
public static readonly DependencyProperty MinAngleProperty =
DependencyProperty.Register("MinAngle", typeof(double), typeof(Needle), new
public double MaxAngle
{
get { return (double)GetValue(MaxAngleProperty); }
set { SetValue(MaxAngleProperty, value); }
}
public static readonly DependencyProperty MaxAngleProperty =
DependencyProperty.Register("MaxAngle", typeof(double), typeof(Needle), new
}}

Real-time Sensor Data Networks with HDInsight and Apache Storm
Creating a sensor data network that we can use with our Xamarin App

Apache Storm and Real-Time Sensor Networks
Data Capture in Real-Time

What you need to get started
1. An Azure subscription with HDInsight and Apache
Storm (in beta) Active
2. Visual Studio with the Microsoft Azure SDK for .NET
3. Java and JDK
4. Maven
5. Git
6. Source Code for the Sample Project
http://azure.microsoft.com/en-
us/documentation/articles/hdinsight-storm-sensor-
data-analysis/
https://github.com/Blackmist/hdinsight-eventhub-
example
http://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-sensor-data-analysis/

Creating the Dashboard
The dashboard is used to display near-real time sensor information. In this case, the dashboard
is an ASP.NET application hosted in an Azure Website. The application's primary purpose is to
serve as a SignalR hub that receives information from the Storm topology as it processes
messages.
The website also contains a static index.html file, which also connects to SignalR, and uses D3.js
to graph the data transmitted by the Storm topology.
NOTE: While you could also use raw WebSockets instead of SignalR, WebSockets does not
provide a built-in scaling mechanism if you need to scale out the web site. SignalR can be scaled
using Azure Service Bus (http://www.asp.net/signalr/overview/performance/scaleout-with-
windows-azure-service-bus).
For an example of using a Storm topology to communicate with a Python website using raw
WebSockets, see the Storm Tweet Sentiment D3 Visualization project.

The dashboard is used to display near-real time sensor information. In this case, the dashboard is an ASP.NET
application hosted in an Azure Website. The application's primary purpose is to serve as a SignalR hub that receives
information from the Storm topology as it processes messages.
The website also contains a static index.html file, which also connects to SignalR, and uses D3.js to graph the data
transmitted by the Storm topology.
NOTE: While you could also use raw WebSockets instead of SignalR, WebSockets does not provide a built-in scaling
mechanism if you need to scale out the web site. SignalR can be scaled using Azure Service Bus
(http://www.asp.net/signalr/overview/performance/scaleout-with-windows-azure-service-bus).
For an example of using a Storm topology to communicate with a Python website using raw WebSockets, see the
Storm Tweet Sentiment D3 Visualization project.
1.In Visual Studio, create a new C# application using the ASP.NET Web Application project template. Name the new
application Dashboard.
2.In the New ASP.NET Project window, select the Empty application template. In the Windows Azure section, select
Host in the cloud and Web site. Finally, click Ok
NOTE: If prompted, sign in to your Azure subscription.
3.In the Configure Windows Azure Site dialog, enter a Site name and Region for your web site, then click OK. This will
create the Azure Website that will host the dashboard.
4.In Solution Explorer, right-click the project and then select Add | SignalR Hub Class (v2). Name the class DashHub.cs
and add it to the project. This will contain the SignalR hub that is used to communicate data between HDInsight and
the dashboard web page.
DashHub.cs

Startup.cs

Next Steps
• In Solution Explorer, right-click
the project and then click Add
| HTML Page. Name the new
page index.html. This page will
contain the realtime
dashboard for this project. It
will receive information from
DashHub and display a graph
using D3.js.
• In Solution Explorer, right-click
on index.html and select Set as
Start Page.

Next Steps
• In Solution Explorer, right-click the project and click Add | HTML Page.
• Name the new page test.html. This page can be used to test DashHub and the dashboard
by sending and receiving messages. Save All for the project.
• In Solution Explorer, right-click on the Dashboard project and select Publish. Select the
website you created for this project, then click Publish.
• Once the site has been published, a web page should open displaying a moving timeline.
Test the dashboard
To verify that SignalR is working, and that the dashboard will display graph lines for data sent to
SignalR, open a new browser window to the test.html page on this website. For example,
http://sensordash.azurewebsites.net/test.html.
The dashboard expects JSON formatted data, with a device id and temperature value. For
example {"device":0, "temperature":80}. Enter some test values on the test.html page, using
device IDs 0 through 9, while the dashboard is open in another page. Note that the lines for
each device ID are drawn using a different color.

Configuring the Azure Event Hub

Sending Messages to the Event Hub

Program.cs http://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-sensor-data-analysis/

Program.cs
For now, you will receive a warning on lines that
reference the Event class. Ignore these for now.
In the Program.cs file, set the value of the following
variables at the beginning of the file to the
corresponding values retrieved from your Event Hub in
the Azure Management Portal.

Program.cs

Setting up your
cluster and
virtual network
Creating your network..

Setting up your cluster and virtual
network

Enabling
Real-Time
Communication

Enabling
Real-Time
Communication
This tells Maven to do the following
when building the project:
Include the /conf/Config.properties
resource file. This file will be created
later, but it will contain configuration
information for connecting to Azure
Event Hub.
Include the /conf/hbase-site.xml
resource file. This file will be created
later, but it will contain information on
how to connect to HBase
Use the maven-compiler-plugin to
compile the application.
Use the maven-shade-plugin to build an
uberjar or fat jar, which contains this
project and any required dependencies.
Use the exec-maven-plugin, which
allows you to run the application locally
without a Hadoop cluster.

Adding
Configuration
Files
Scaffolding

Adding Configuration Files

Topology
Dataflows
http://azure.
microsoft.com
/en-
us/documenta
tion/articles/h
dinsight-
storm-sensor-
data-analysis/

Topology
Dataflows

Topology
Dataflows
http://azure.microsoft.com/en-us/documentation/articles/hdinsight-
storm-sensor-data-analysis/

Topology
Dataflows
Summary
You have now learned how to use Storm to read data from
Event Hub, store data in HBase, and display information from
Storm on an external dashboard using SignalR and D3.js.
•For more information on Apache Storm, see
https://storm.incubator.apache.org/
•For more information on HBase with HDInsight, see the HBase
with HDInsight Overview
•For more information on SignalR, see ASP.NET SignalR
•For more information on D3.js, see D3.js - Data Driven
Documents
•For information on creating topologies in .NET, see Develop
streaming data processing applications with SCP.NET and C# on
Storm in HDInsight
http://azure.microsoft.com/en-us/documentation/articles/hdinsight-
storm-sensor-data-analysis/

SendEvents
Pushing Data to
Azure HDInsight
and Apache Storm
Our Example running pushing
temperature data

CONNECTING TO REAL-TIME VEHICLE DATA AND MODELING IN THE LAB

Azure Mobile Services for Unity 3D
The Azure Mobile Services have a free tier that includes up to 500 devices as well as 500k API calls and you can also
use the free tier for up to 10 services. This means that you can test out a few things without having to pay for it.
Azure Mobile Services
Azure Mobile Services is a part of Azure that allows access to a database and has connection and sample code to talk
to any mobile system that is out there. This will provide you with the code or library to do the connection to Android,
iOS, Windows 8, Windows Phone, HTML5/JavaScript, or even the Xamarin libraries. To get started, if you do not have
any Mobile Services defined yet you can click on the Mobile Services tab on the left and then the Create a New
Mobile Service link next to it. You can also click on the New option on the bottom the of the screen and select
Compute -> Mobile Service -> Create.
From here, you will get a popup to fill in to finish up the creation. The first field is the name of the Mobile
Service. This name will also be the address for the service. It must be unique. For this example, I named mine
“unityleaderboard”. The next field is the database to use as a back end for the service. You can choice from “Use an
existing SQL database“, “Create a free 20 MB SQL database“, or “Create a new SQL database instance“.

So up to now we have the Mobile Service setup, but
there is no data yet. Go into your new Mobile Service
and then click on the Data link at the top. You can now
add a new Table to the database that was setup earlier.

The next step is to add the data fields to the new leaderboard table. This will allow
us to save the data for the UserName and the Score that is saved. This is going to
be a basic sample and not an optimized database, so I will be adding the UserName
as just a string field to the table. If this was a bigger system supporting multiple
games, I would probably make a Player table with all of the players info there and
then a leaderboard table that cross referenced that player table. Since this is just a
quick and simple leaderboard for a single game, keeping the raw text in the table is
not that bad. The Score field is going to be added as a Number so that we do not
have to change the numbers of the score into a text field back and forth. After
clicking on the table name, you will see and can click on the Columns link to get to
add new columns. To add a new column, use the Add Column link at the bottom of
the page.

Bitrave Mobile Services Plugin API
https://github.com/bitrave/azure-mobile-services-for-unity3d

Put the plugin binaries in your Assets/Plugins folder. These get built into an Output folder in the
root of the solution in the right folder structure. And it’s as simple as…
var data = new LevelSaveData() { SaveData = "some data here", Id = 1 };
var azure = new AzureMobileServices(_azureEndPoint, _applicationKey);
azure.Update<LevelSaveData>(data);
or
var azure = new AzureMobileServices(_azureEndPoint, _applicationKey);
azure.Lookup<LevelSaveData>(1, azureResponse =>
{
if (azureResponse.Status == AzureResponseStatus.Success)
{ var ourObject = azureReponse.ResponseData; }}
Data comes back via callbacks and response objects. Unity doesn’t support await/async, but when
it does it will move to that model.

API
Initialize
initialiation is just as simple as you’d expect.
var service = new AzureMobileServices(“url”, “token”);
Insert
Insert an item into your Azure database in a single line of code from Unity.
service.Insert<ToDoItem>(myItem);
Update
Update items in the Azure databsae with just one line of code from Unity.
service.Update<ToDoItem>(myItem);
Delete
Remove items from the Azure database in 1 line of code from Unity.
service.Delete<ToDoItem>(myItem);
Query
Query items in your Azure Mobile Services from Unity.
service.Where<ToDoItem>(p => p.Category == “Exercise”, azureResponse =>{List<ToDoItem> exerciseItems = azureRepsonse.ResponseData;
NOTE: await / async will be available when supported by Unity. Until then we are using callbacks.
Lookup
Lookup items in your Azure Mobile Services from Unity.
service.Lookup<ToDoItem>(myItem, azureResponse =>{ToDoItem myToDoItem = azureResponse.ResponseData;
Login
On supported platforms, LoginAsync can be called for authenticated services.
azure.LoginAsync(AuthenticationProvider.Facebook, loginResponse => {var token = loginResponse.ResponseData.MobileServiceAuthenticationToken;});

API
Initialize
initialiation is just as simple as you’d expect.
var service = new AzureMobileServices(“url”, “token”);
Insert
Insert an item into your Azure database in a single line of code from Unity.
service.Insert<ToDoItem>(myItem);
Update
Update items in the Azure databsae with just one line of code from Unity.
service.Update<ToDoItem>(myItem);
Delete
Remove items from the Azure database in 1 line of code from Unity.
service.Delete<ToDoItem>(myItem);
Query
Query items in your Azure Mobile Services from Unity.
service.Where<ToDoItem>(p => p.Category == “Exercise”, azureResponse =>{List<ToDoItem> exerciseItems = azureRepsonse.ResponseData;
Lookup
Lookup items in your Azure Mobile Services from Unity.
service.Lookup<ToDoItem>(myItem, azureResponse =>{ToDoItem myToDoItem = azureResponse.ResponseData;
Login
On supported platforms, LoginAsync can be called for authenticated services.
azure.LoginAsync(AuthenticationProvider.Facebook, loginResponse => {var token =
loginResponse.ResponseData.MobileServiceAuthenticationToken;});

MapNav Plug-In Unity 3D
https://www.assetstore.unity3d.com/en/#!/content/13153

Prime 31 Plug-in for Unity
Sign-in to Azure portal. If you don’t have an Azure account yet game developers can register for
the Cloud GameDev Offer. Create Azure Mobile Service

Get the Plug-in
https://prime31.com/plugins

Get the Plug-ins you need to install
https://prime31.com/plugins

Create a New Project and Install the Plug-in

Paste in your Azure Connection Strings

Target Windows Store In your program settings
• “Add Current”
scene
• Select “Windows
Store” and “Switch
Platform”
• Select C# Solution
and SDK “8.1”

Select Player Settings
Under “Metro
Unprocessed Plugins” set:
Size: 1
Element 0:
P31MetroAzure.dll
Click Build

Launch Visual Studio Open Windows Store Build

Open Package.AppXManifest.xml
Add Internet Capabilities to Manifest

Open Configuration Manager Target Current Hardware

Build and Run
You will see your items in Azure

Connecting to Vehicle Data Near Real-Time in the Cloud
The WCC Connected Pontiac G6
Modeled and Connected to Real-
Time Vehicle Data..

Real Time Connected Vehicle Networking with HDInsight and Apache Storm

More Related Content

Viewers also liked (20)

Similar to Real Time Connected Vehicle Networking with HDInsight and Apache Storm (20)

More from Our Community Exchange LLC (10)

Recently uploaded (20)

Real Time Connected Vehicle Networking with HDInsight and Apache Storm