Data Visualization for Big Data: Experience from the Front Line

Data Visualization for
Big Data
Rosa Romero Gómez, Ph.D
rosaromerogomez.com
@87rromero
Experiences from the Front Line

[Georgia Tech campus, Klaus Advanced Computing building, May 27th 2016]

Data Visualization
Why? What? How?

The greatest value of a picture is
when it forces us to notice what we
never expected to see
[John W. Tukey. (1981) Exploratory Data Analysis]

Let me put you a
simple example…

[Sample data sets recreated from Francis J. Anscombe (1973). Graphs in statistical analysis.
Source: Andy Kirk. (2012) Data visualization: A successful design process]

[Source: http://commons. wikimedia.org/wiki/File:Anscombe%27s_quartet_3.svg]

Data visualization addresses…
…Information Scalability
…Visual Scalability
…Human Scalability

Data visualization addresses…
…Human Scalability
• It enhances the recognition of patterns
• It increases our efficiency to explore large datasets
• It supports decisions
• It expands our working memory to solve problems

Data visualization is the use of interactive
visual representations of data to amplify
cognition
[Stuart Card. (2008) Information visualization]

[The Starry Night. (1889) Vincent Van Gogh. Source: https://en.wikipedia.org/wiki/The_Starry_Night#/
media/File:Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg ]

[Source: http://elpais.com/elpais/2016/10/28/media/1477669343_348572.html]

Why are we doing this
visualization project?
Even more important…

Case Study:
Visualization of the IPv4 address space
for network threat investigation

Network Threat
Analyst
Computer Network Data Collection Point
Get to know the context…
User
CMD Tools
Websites
Logs
Physical &
Task
Context
Technical Context

Step 1: Identify relevant visualization tasks
•Find suspicious IPs blocks
•Find domain names associated with specific IPs
•Examine the presence of domain names on blacklists
•Examine the relation of domain names with malware
•Identify the geographical location of IPs
•Identify the ownership of domain names
•Find suspicious Autonomous Systems

Data Visualization for Big Data: Experience from the Front Line

The more accessible your visualization,
the greater your audience and your impact
[Scott Murray. (2013) Interactive Data Visualization for the Web]
Step 2: Choose a library

•Functionality: Does it support the visualizations I
need?
•License: open source or commercial?
•Active support and development
•Browser compatibility
•Dependencies (e.g. React.js)

Building a
visualization
with charting
libraries such
as Chart.js,
Tableau…

Building a
visualization
with D3.js

•D3 is not really a “visualization library”; it does not
draw visualizations
•D3 = “Data-Driven Documents”; it associates data with
DOM elements and manages the results
•D3.js provides with tools such as layout, scales,
shapes that you can use to build visualizations

Step 3: Data transformations
{"date":"20160408","qname":"*.3rdandmonster.com.","qtype":1,"rdata":
{"string":"66.96.161.142"},"ttl":null,"authority_ips":"216.239.36.109","count":1,"hours":
1048576,"source":"gt","sensor":"active-dns"}

{"date":"20160408","qname":"*.aavxxnbm.org.","qtype":1,"rdata":{"string":"213.184.126.162"},"ttl":{"int":
604800},"authority_ips":"213.184.126.162","count":10,"hours":5543209,"source":"gt","sensor":"active-dns"}

{"date":"20160408","qname":"*.aenhfat.info.","qtype":1,"rdata":{"string":"213.184.126.162"},"ttl":{"int":

{"date":"20160408","qname":"*.agzksjhrmf.info.","qtype":1,"rdata":{"string":"213.184.126.162"},"ttl":{"int":
[Fragment of Active DNS resolution queries in deserialized Avro format - JSON format,
https://www.activednsproject.org]
Pre-processed data Domain Name
IP address

Guided by the Visual Information-Seeking Mantra:
“Overview first,
Zoom and Filter, and then
Details-on-Demand”
[Shneiderman. (1996) The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations]

{
"date": "dateValue",
"children": [{
"name": “/8Name",
"size": “numberOfIPs/8",
"color": “numberOfBlacklistedDomainNames/8",
"children":
[
{
"name": "/16Name",
"size": "numberOfIPs/16",
"color": "numberOfBlacklistedDomainNamesper/16",
"children": [
….
]
}
….
]
}
Nested JSON
format template
(JSON file per day)
Nested IPs in the following format:
/8 >> /16 >> /24 >> /32
Visual variables

{
"date": "dateValue",
"children": [{
"name": “Continent",
"size": “numberOfIPsContinent",
"color": “numberOfBlacklistedDomainNamesperContinent",
"children":
[
{
"name": "Country",
"size": "numberOfIPscOuntry",
"color": "numberOfBlacklistedDomainNamesperCountry",
"children": [
….
]
}
….
]
}
Nested JSON
format template
(JSON file per day)
Alternative nesting options:
Continent >> Country >> State >> City

> JSON files of 70
Mb
Nested JSON
format template
(JSON file per day)
Triple hierarchy!!!

Split into
IPhierarchy.json
GeographicalHierarchy.json
AS.json
Nested JSON
format template
(JSON file per day)

Step 5: User Experience Breadcrumbs User-adjustable
visual settings

[Astrolavos Team during S&P 2017 deadline, November 11th 2016.
Source: https://twitter.com/mAntonakakis?lang=es]

Data Visualization for Big Data: Experience from the Front Line

More Related Content

What's hot (20)

Similar to Data Visualization for Big Data: Experience from the Front Line (20)

Recently uploaded (20)

Data Visualization for Big Data: Experience from the Front Line