Big Data Scotland 2017

Welcome to
Big Data Scotland 2017
#datascot

Mark Stephen
BBC Scotland
@bbcscotland
#scotdata

Ray Bugg
DIGIT
@digitfyi
#scotdata

www.digit.fyi
50,000 Monthly Page Views
30,000 Unique Visitors
Monthly News, Views,
Opinion, Insight

Our Next Event
DT2018
3rd Annual Digital
Transformation Conference
www.digifutures.co.uk

Kate Goldman
KBG Solutions
@digitfyi
#scotdata

New Foundations for a
Data-Driven Organisation

The Time is Now
 The shift to data driven process is underway.
 Burgeoning data availability, open source, more affordable technology,
and consumer demand for betterment on every front suggests it might be prime
time for analytics teams.
 New and advanced analytics technologies on the rise: predictive and prescriptive
analytics, decision management software, smart machines, event stream
processing applications and operational intelligence platforms.
 As a result, the demand for analytics has moved outside specialty IT-based
communities and is now being headed up largely by business units as well.

Where’s the Beef?
 Building Blocks
 Fundamentally, building out the underlying data architecture as well as data
collection or generation capabilities.
 Switch from legacy data systems to a more nimble and exible architecture to store
and harness big data.
 Digitize operations more fully in order to capture more data from customer
interactions, supply chains, equipment, and internal processes.
 But… Data is not information, information is not knowledge,
knowledge is not understanding, understanding is not wisdom.
 Secondary (but no less important), is to find the answers to the
innumerable business opportunities your organisation faces in
your data.

So Here is What you Need to Do. Now.

5. Silo Busting to Let the Data Flow
 Silos, created because of structural issues, political factors, growth challenges,
or vendor lock-in can create significant challenges to creating insight and
advantage from data.
 The pressure is on, but incremental practical, methodical approaches are best for
those without the luxury of building from scratch.
 Find a good target.
 Use engagement practices to identify high-value opportunities.
 Analyze business needs, choose a problem where data could provide a tangible benefit and
value… perhaps in enhancing sales or preemptive incident response.
 Draw in the data from around the organization and invest in these use cases first. This is not
a proof of concept — you should do these earlier as a way of identifying opportunities — but
a banner project that can drive subsequent investments.
 Tie the integration to its application, so you get value early.
 Every Progressive Step, moves towards integration. (But beware planning fallacy!)

Dan Fiehn
Markerstudy Group
@danfiehn
#scotdata

DATA INNOVATION
The Gap Smart Automation Smart Learning Smart Devices

Device Obsession

Exceptional Experience

Connect Anywhere

Systems Unable to Cope

Capability to Change

Predicting Issues

Intelligent Services

Autonomous Teams

Award Winning
AUTOMATION

+34%Machine Beats Man

1. Model
Production
2. Data
Enrichment
3. Scenario
Simulations
4. Model
Deployment
5. AI
Aggregation
6.
Actionable
Insights
Intelligence Engine

IncreaseReduction
Customer Retention
Sweet
Spot

Insurance in a TRiCE

For the first time in history,
greater insight into our
driver behaviour.

ARE YOU DRIVING
DATA
INNOVATION?

DATAINNOVATION
uk.linkedin.com/in/danfiehn
@danfiehn
No algorithms were hurt in the m
GAP AUTOMATION
LEARNING DEVICES

Malachy Devlin
Clyde Space
@ClydeSpace
#scotdata

Advancing your Data Analytics Capabilities
Malachy Devlin
Big Data - Edinburgh | 7th December 2017

Company Journey – Initial Data
0
1
2
3
4
5
6
May-16 Sep-17 Feb-19 Jun-20 Oct-21 Mar-23
Company Revenue
0
20
40
60
80
100
120
140
May-16 Sep-17 Feb-19 Jun-20 Oct-21 Mar-23
Company Growth

Data Query Language - Relational Data
EQL

Relational Data
SQL1. Migrate to Databases
• Data
2. Migrate to MRP/ERP
• Data
• Business Logic
Manufacture
Customers
Finance
Quality
HR
Supply Chain
Management

Manufacturing/Entreprise Resource Planning
Manufacture
Customers
Finance
Quality
HR
Supply Chain
Management
Business Project
•IT is an enabler
Data Matured?
•Ready for extract/Transform/Load (ETL)
Right Software for Right Requirements
• Different companies, different formula
Business Process Upgrade
•Don’t implement bad processes
Training
•Tools and Process
Team - authority & understanding
• Budget, Resource, Technology
• Processes, Data, Requirements
Going Live is not the end game!

MRP/ERP - Real Time Business
• End to End records
• Critical for some markets
• Single point of truth
• Portfolio not Project view
• Accelerate intelligence
• 2 week to 1 minute in SC
• 15 minute to 15 seconds
production status
• Materials/Labour
variability
• More managed data
recorded, reduced effort
• Order to customer receipt
<18hours
• Partner Integration
• Port Merger Integrations
Agility Accuracy
TraceabilityInformed

The traditional space industry
Copyright © 2017 Clyde Space Ltd. All rights reserved.

The future of nanosatellite technology

Quality innovation

SmallSat market
Clyde Space Commercial in ConfidenceCopyright © 2017 Clyde Space Ltd. All rights reserved.

Where do satellites fit in?
Satellites are used to provide the main network connectivity for a localised area
which in turn relies on localised network (PAN or LAN) to distribute this
connectivity between devices within this area.

IoT from Space

Earth observation applications
SeaHawk
▪ Pair of 3U CubeSats flying multispectral imagers
▪ x30 smaller than existing SeaWIFS asset
▪ Putting Moore’s Law into space
▪ Revisit rate for IOD of 7 days
▪ Downlinked via X-band to NASA NEN

Earth observation applications
PICASSO
▪ ESA and BISA mission
▪ Hyperspectral imager supplied by VTT
▪ Ozone monitoring mission
▪ Single 3U IOD

IPP Mission: FireSat

Quantum applications

Clyde Space nanosatellite solutions

Launch and deployment

Ground control
Clyde Space ground station solution:
▪ Based at our headquarters in Glasgow, Scotland
▪ Further sites established in ME and US
▪ SDR based architecture
▪ Fully automated
▪ Working to develop partnerships with recognised and emerging
ground station solutions globally

SmallSat excellence

Clyde Space Cubesats

Meeting and generating demand

enquiries@clyde.space

Questions & Discussion
#scotdata

Refreshments & Networking
Please check rear of
badges for breakouts
#scotdata

How to make an impact
with your data:
Going from Boring to Beautiful
Louis Archer
Tableau
@louisarcher

BUSINESS CULTURE IS
THE PROBLEM

Collaboration Iteration
Training

“Data visualisation
is a language. It’s
a means to convey
an opinion, an
argument.”
Kim Rees – Founding Partner,
Periscopic

Design by Tableau; inspired by CTT Wireless Dashboard from

Pleasurable experiences:
The three levels of processing

Pleasurable experiences:
Visceral
Behavioural
Reflective

Don Norman’s Pleasurable experiences:
Visceral
Behavioural
Reflective

Tableau Research:
Eye-tracking

http://tabsoft.co/designmonth #VisualDesignTricks @acotgreave

How to make an impact with your data?
The three levels of processing:
Visceral
Behavioural
Reflective

“Impact” is NOT just about
beautiful/functional
dashboards

The interesting thing was, we thought we were doing well, and then
we discovered there was this big negative cost. It was like, ‘Oh my
God.' Suddenly you go and say, 'Okay, I've discovered a new aspect
of engine cost that we hadn't realized.‘
Suddenly you're going, “Bang, bang, bang, two minutes in Tableau”
and you can see the average per month, the average per day, and it's
like, “Oh, wow—we can do this slightly differently.'
Within two days, I'd literally re-worked the whole instruction, sent it
out to people, and off we went. As a result, it’s been a very
significant difference in terms of U.S. dollars.
Jonathan Capper
Production Planning Manager

Bang, bang,
bang, two
minutes…
Jonathan Capper
Production Planning Manager

Why?
Why?
Why?
Data visualisation
Known unknowns
Predefined answers only
Visual analytics
Unknown unknowns
Instant answers to new questions
Bang, bang, bang, two minutes…

We help see and understand datapeople

Welcome Back to
#datascot

Matthieu Poyade
The Glasgow School of Art
@GSofASimVis
#datascot

Data Visualisation
Dr. Matthieu Poyade

Data Visualisation
Graphical communication process which
empowers one to gain understanding and
insight into data.
“Visualization offers a method
for seeing the unseen” - McCornick (1987)
Data Visualisation

“People are generally better persuaded by the reasons
which they have themselves discovered than by those
which have come into the mind of others.”
Pascal (1623 – 1662)

Pragmatic Approach to Visualisation
• Data Communication
– Effectively and efficiently make the user understand
• Visual Efficiency
– Perceptually efficient, make maximum use of the visual
channels
• Data is given
– Visualization is not about generating data, although
sometimes data requires pre-processing

Looking at History… London’s Cholera
outbreak - Dr John Snow (1854)

Looking at history…
The London Underground through time

Looking at History…
The Glasgow Subway through time

We are drowning in a sea of data…The Flood of Data

A picture is worth a thousand words!!

How can Data Visualisation help to
envision decision making impact
Challenger Disaster
January 28, 1986

• Engineers knew there was a problem:
Failure of ring joint at low launching
temperatures
• Technical data presented through 13
Charts (notes, tables & diagrams) to
management officers recommended
not to launch
• Data were presented poorly and
didn’t enable the correlation
between cooler temperatures and an
increased chance of damage

Avoidable Tragedy?
?
“Bad slides or bad engineering don’t kill
people: bad decisions do” (Tufte)

How VR/AR can enhance Data
Visualisation for Businesses?

AR and VR in Data
Visualisation

Steven Faull
Aggreko
@stevenfaull
#datascot

D E C E M B E R 1 7
Steven Faull
Head of Software & Analytics
@stevenfaull
From Reactive to Proactive to Predictive:
Using Data to Drive Customer Benefit

174
Reactive
The way things used to be…
• Service scheduling
• Customer calls for support
• Technicians have no visibility of faults
• Spare parts?
• Low customer service level

175
Proactive
The way it is today…
• Telemetry-enabled fleet
• The ROC
• Visibility of every alert and alarm globally
• Highly proactive
• Enhanced reliability & customer service

176
Innovation
Examples…
• Enterprise Social Networking
• Geo-spatial Sales application
• Cloud technology
• Micro-services architecture on Service Fabric
• Big Data Analytics

178
Predictive
The future…using data to enable:
• Near-time data analytics
• Predictive alerting to the ROC
• ‘Just-in-Time’ servicing
• Zero breakdowns
• ‘Best-in-Class’ customer service

182
Approach
1. Build a great team

183
Approach
2. Empower for innovation and exploration

184
Approach
3. Find a great sponsor

185
Approach
4. Sell the vision through stories

186
Approach
5. Collaborate, collaborate, collaborate

187
Approach
5. Collaborate, collaborate, collaborate
6. Agile delivery

188
What’s Next?
• Reliability
• Market Intelligence
• Service Intervals
• Condition Based Service
• Product Development
• Procurement
• Audit
• Sales Effectiveness

Thank You189
Steven Faull
Head of Software & Analytics
@stevenfaull

Sarah Forbes
Peterson
@srf1980
#datascot

GUIDING PRINCIPLE 1
Full supply chain visibility for everyone
GUIDING PRINCIPLE 2
Predict the future state based on data
GUIDING PRINCIPLE 3
Enable ‘next best step’ decision-making or
management by exception to flourish
GUIDING PRINCIPLE 4
Enforcing a single source of truth encourages
collaboration across the entire supply chain
GUIDING PRINCIPLE 5
Provide a real-time trigger for smarter
communication

Scott Krueger
Skyscanner
@skrueg
#datascot

From Producers To Consumers
Data Engineering @ Skyscanner
Scott Krueger
Principal Data Engineer

Skyscanner growth: interesting challenges

But wait - we have been designing during growth

Perfection is in the eye of the data holder
Data systems old and
new
have constraints.
It is a fact of life.
Accept it, find them, and
change things for what
you need.

Scale – often the first business driver?

Organise your data organisation
Conway's law (1967): Technical system
determines org structure
ProcessIngest
Unified
Log
Transport
Archive
Science /
Report /
Analytics

(Re)Organise for growth
Process
Ingest
Unified
Log
Transport
Archive
Science /
Report /
Analytics

The Data Organisation
As a Business Strategy
Educate To Empower
The problem with Data Engineers and Data
Scientists
is that they’re Data Engineers and Data
Scientists....
(and no one else is)
"I'm not going to find that
out for you - you are"

Operational Strategy
Total Ownership
"You build it, you run it." -
Werner Vogels
This brings developers into contact with the day-to-day operation
of their software. It also brings them into day-to-day contact with
the customer. This customer feedback loop is essential for
improving the quality of the service."

Product Evolution - Data Assets into
Meaningful Information
Operational
Diagnostics
Your organisation
and strategy
Business
KPIs
Financial
Reporting
Just what do I do with 4TB's of Data Events Per Day?
Customer
Products
Partners
and Suppliers
Your
customers
(t)

The Future
* Barrier to entry gets lower and lower
* Data literacy and engineering up-skilling continues
* SQL is back
* Machine Learning / Auto-code generation
* 5G

Wrap Up
Unifying our data into a single platform…
…frees up energy
…to be invested into meaningful data products
…that allow us to innovate faster
…to better serve the world's travellers

Questions & Discussion
#datascot

TAKING DATA SEARCH, DISCOVERY AND ANALYSIS TO
THE NEXT LEVEL
DAVID RIVETT
C H I E F O P E R A T I N G O F F I C E R

Agenda
 Unstructured Data
 Information worker challenges
 The power of precision search
 When you don’t know where to start
 Data Science & AI
 Feral Data & GDPR

“Unstructured” Big Data
In every organisation data is “relatively” Big
 Organisational memory
 Staff turnover
 Personal and shared drives
 Email
 Document Management
 Business Operational Systems

Today’s challenges…
According to a McKinsey report,
“employees spend 1.8 hours every day - 9.3 hours per week, on
average - searching and gathering information.
Put another way, businesses hire 5 employees but only 4 show
up to work; the fifth is off searching for answers, but not
contributing any value.”
Source: Time Searching for Information.

Turning Data into Knowledge
WHOLE
DOCUMENTS
A SINGLE
SENTENCE
CONTENT BY
PARAGRAPH
DOCUMENTS
BY SECTION

The Chilcot Report
 58 separate PDF files.

SPARKVS FLINK
STEFAN PAPP,VDSG
BERNHARD ORTNER,THINKBIGANALYTICS

Stefan Papp
Seems himself as Data Evangelist who focuses on data since 2010. He is passionate
about how data can transform how we live and work and keen to explore all data
with an open mindset and in an agile manner.
He is a member of theVienna Data Science Group and has been working across all
industries and consults various companies on data strategies. He also works in
projects as data architects
Bernhard Ortner
He is a Senior Data Engineer at Think Big Analytics,A Teradata Company. He has
solid experience implementing several big data technologies across industries like
telecommunications, finance, energy and government. In his experience, he has
covered the whole spectrum in big data including visualization, data ingestion,
fusion and integration.

TERADATA PORTFOLIO
>
High-
impact
business
outcomes
Analytic Solutions
Analytics Business Consulting
Business Value Framework
Data Science
Business Solutions
Strategy and Roadmaps
Design and Implementation
Ecosystem Architecture
Managed Services
Architecture Expertise
Public Cloud,
Private Cloud,
Managed Cloud,
Hybrid Cloud,
On-Premises
Teradata Database,
Teradata Aster®
Analytics, Hadoop
Teradata
QueryGrid™, Presto,
Listener™, Unity,
AppCenter
Technology
Solutions
• ~1,400 + Customers in 77 Countries
• ~10,000 Employees including
~5,000 Consultants
• Market Cap: U.S. $4 Billion+
• World’s Most Ethical Companies –
Ethisphere Institute
• Fortune: Top 10 U.S. Software Company
• The leader in Gartner Magic Quadrant
Data Warehouse and Data Management
Solutions for Analytics
• The Forrester Wave™:
Big Data Hadoop-Optimized Systems
In-Memory Database Platforms
Teradata at-a-glance
© 2017 Teradata

VIENNA DATA SCIENCE GROUP
Mission
• Nonprofit association which aims to promote knowledge about data science methods/big
data/AI techniques.
Diverse members
• Academics, professionals,students and all other Data Science enthusiasts
Different fields
• Mathematics, physics,econometrics,electrical engineering,medical science,finance,real estate,
computer science and social sciences

Traditional Data:
• ERPs, CRM Databases
• Highly structured
0
5
10
15
20
25
30
35
40
2006
2007
2008
2009
2010
2011
2012
2013
2014
1015
2016
2017
2018
2019
2020
Zettabytes
Schema on Write Schema on Read
“New Data” :
• Human or Machine
Generated
• “unstructured”

SPARK – “THE SMART PHONE FOR BIG DATA”
What do 4 Gen. Processing Engines have in
common with the iPhone?

Administration
• Power BI uses Azure Active
Directory to authenticate users
who login to the Power BI service
User Friendly / Self-Serve
• Uses a graphical user interface
built aroundself-serve
Security & Privacy
• Currently complies with both EU-
U.S. Privacy Shield and EU Model
Clauses.
• Committed to GDPR compliance
Cross Platform
Functionality & Embedding
• Cloud and desktop versions
• iOS and Andriod appsavailable
• Can embed Power BI code within
websites
Sources & Connections
• Wide list of our of the box
connectors including Excel, SQL,
MySQL, and widely used APIs (i.e.
Google Analytics)
Managing, Cleaning &
Transforming
• Self-serve ETL process
• Flexible to perform more
advanced queries, i.e. mergingand
cleaning files
Storing & Querying
• Does not automatically store raw
data however there is an optionto
save data within the report
• Can refresh data from source
automatically
Reports
• Allows users to build visual reports
• Can export the reportsto
PowerPoint
Dashboards
• Elements from reports can be
reproduced within dashboards
• Best viewed on devices
Visualisation
• Out of the box visuals provided
• Online library of custom visuals
• Ability to create own visuals using
R
Analytics
• Functionality similar to Excel
• Has natural language processing
built in
Collaboration
• Able to share with selected people
or open access
• Creates an associated
conversation stream

Good
Data
Sufficient
Observations
Representative
Timely

ANSWER
Before Spark After Spark After Flink
Program-API Map Reduce Spark Core Flink API
Abstraction Pig, Cascading Spark Core Flink API
Streaming Apache Storm Spark Streaming Flink API
Machine Learning Mahout Spark ML FlinkML
Graph Engine Giraph Spark GraphFrames Gelly
SQL Hive Spark SQL TableAPI

SPARK ARCHITECTURE
• APIs: Scala, Java, Python or R
• Core Modules Custom Modules
H2O

DISTRIBUTED DATA SETS
• Data Containers to keep data in-memory
• RDDs: untyped
• DataFrames / DataSets: typed / object-oriented
• Processing with data sharing instead of shared nothing
• complex, multi-pass analytics (e.g. ML, graph)
• interactive ad-hoc queries
• real-time stream processing
Query
Input
Query
Query

STREAMING IN DETAIL
• Data is micro-batched, i.e. dividied into n RDDs
• For each RDD a task is launched and destroyed once it’s completed
• A task is a function you apply, e.g. filter, map, …

ANALYTIC
• MLlib: Out-of-box ML library
• Continuously new algorithms are added
• Sparkling water: port of H2O for spark  add additional DS algorithms
• H2O: ML,AI library, for more advanced analytics
• Deep Learning
• Generalized Linear model
• PCA
• ….

SPARK - PIPELINES
• Concept: sequence of algorithms to process and learn from data
• A pipeline consists of
• Transformers:transforms a DataFrame into another one, e.g. ETL Steps
• Estimators: trains a model on a DataFrame, output is the trained model

SPARK PROGRAM
• Step 1: create a distributed data set
• Step II: apply filter, models,…
• Step III: get result
val training = spark.read.format(“jdbc”).option(“…“).toDF(”col1", ”col2")
val lr = new LogisticRegression()
lr.setMaxIter(10) .setRegParam(0.01)
val pipeline = new Pipeline() .setStages(Array(lr))
val model = pipeline.fit(training)
pipeline.write.overwrite().save(„name“)
model.transform(testData) .select(„col1“, „col2“).collect().foreach(print…)
https://spark.apache.org/docs/2.2.0/ml-pipeline.html

• There is also a different approach….
... a micro-batch is a collection of streaming events…
But you can also use pure streaming

0 4,000,000 8,000,000 12,000,000 16,000,000
Storm
Flink
Flink (10 GigE)
Throughput: msgs/sec
10 GigE end-to-end
15m msgs/sec
SURVIVAL OF THE FASTEST….

275
Streaming: continuous processing on data that is
continuously produced
Sources
Message
Broker
Stream
processor
collect publish/subscribe analyze Serve & store

CONNECTORS AND ADAPTERS
DataBases
…
TwitterFeeds
SensorData
DataPlatforms
Feeds
Applications
…

SOURCE -> PROCESS -> SINK278
Source
Transformation
Transformation
Sink
val lines: DataStream[String] = env.addSource(new FlinkKafkaConsumer09(…))
val events: DataStream[Event] = lines.map((line) => parse(line))
val stats: DataStream[Statistic] = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.apply(new MyAggregationFunction())
stats.addSink(new RollingSink(path))
Source
[1]
map()
[1]
keyBy()/
window()/
apply()
[1]
Sink
[1]
Source
[2]
map()
[2]
keyBy()/
window()/
apply()
[2]
Streaming
Dataflow

279
Streaming
Source
Streaming
Source
Streaming
Source
Consumer
Forward events immediately
to pub/sub bus
Stream
Processor
Process at event time &
update serving layer
Message
Broker
Low latency
High throughput
Windowing / Out
of order events
State handling
Fault tolerance
and correctness

Huge Database
Application Application
Client Server
NoSQL
Micro Services
Continuous Streaming
RDMBS
SECOND USE CASE: CONTINUOUS STREAMING

DOYOU HAVE CASESTHAT REQUIRETO PROCESS MORE
THAN ONE MILLION MESSAGES (PER SECOND)?
Fraud Prevention
CEP – Complex Event Processing
Spam Prevention
Network anomaly detection
Predictive Maintenance
Deep Packet Inspection
File Processing
CRM Import
BATCH
STREAM
ERP Data
Social Media
Internet of
Everything
Image Recognition

DOYOUR OPERATIONAL SYSTEMS DELIVER
STREAMING DATA?

REAL TIME ANALYTICS
• Anomaly Detection: detect outliers (= observation that deviates tremendous from other)
• Distance or Density based => clustering
• Spark is used to train the models, but it’s too slow for production
• Flink natively support streams, but has less algorithms than spark

SPARK & FLINK AT A GLANCE
Bernhard Ortner, Sr. Data Engineer
bernhard.ortner@thinkbiganalytics.com
Mobile +43 664
Contact:
Stefan Papp, Data Architect
stefan.papp@icloud.com
Mobile: +43 699 10209453

Precision Search
- The Internet Experience

Precision Search
– The Desktop Experience

Precision Search
- The ultimate experience

Use Cases
 Research
 Detailed Analysis & Review
 Litigation
 Mergers & Acquisitions
 Investigative Journalism
 Better informed decisions
 Faster more confident results and process

Where to start?
“You don’t know what you don’t know!”
“To really get answers out of your data
you need the right questions!”

Data Science & AI
 Topic Clustering and Categorisation
 Semantic Enhancement
 Dictionary Enhancement
 Pattern Matching
 Sentiment
 Inherent Structure

Abuse
60%
Neglect
25%
Disfunctional
Family
10%
Disability
5%
S AMP LE P RE DI CTI O N
Train Model using real case
documents
Model learns topics based
on training data
Model can now identify
topics in case documents
Model then predicts the
probability of topics
present in document
Topic Clustering

Surfacing Knowledge -
Enhancement

Feral Data
 What does this mean for GDPR?

Feral Data
 Pattern matching PII
 Bank, Credit card numbers
 Passport numbers
 Driving License details
 Address, postcode, email
 …

Thank You!
More questions?
Please visit us at our stand
Take the Nalytics challenge!

Big Data Scotland 2017

More Related Content

What's hot (20)

Similar to Big Data Scotland 2017 (20)

More from Ray Bugg (20)

Recently uploaded (20)

Big Data Scotland 2017