OSDC 2017 - Christos Erotocritou - Apache ignite in-memory data fabric

© 2017 The Apache Software Foundation. Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are trademarks of The Apache Software Foundation.
In-Memory Computing Platform
Presenter: Christos Erotocritou
@ChrisErotocritu

Contents
• GridGain & Apache Ignite Project
• Ignite In-Memory Computing Platform
• Introduction to Clustering
• In-Memory Data Grid & SQL Grid
• Compute grid, Service Grid & Streaming
• Hadoop & Spark Integration
• Q & A

© 2017 GridGain Systems, Inc.
What is Apache Ignite
High-performance distributed in-memory platform
for computing and transacting on large-scale data
sets in near real-time.

Apache Ignite Project - Recap
• 2007: First version of GridGain (compute grid)

• Oct. 2014: GridGain contributes Ignite to ASF
• Aug. 2015: Ignite is the second fastest
project to graduate after Spark
• Today vs. Feb. 2016:
• 28 more contributors: 88+ contributors
• Huge development momentum -
Estimated 248 years of effort since the
first commit in February, 2014 vs 192 year
last Feb. [Openhub]
• 200k more SLOC and 2.5k more
commits: 900k+ SLOC & more than
18.5k commits
February 2017

Customer Use Cases
Automated Trading Systems 
Real time analysis of trading positions & market risk.
High volume transactions, ultra low latencies.

Financial Services 
Fraud Detection, Risk Analysis, Insurance rating and
modelling.

Online & Mobile Advertising 
Real time decisions, geo-targeting & retail trafﬁc
information.

Big Data Analytics 
Customer 360 view, real-time analysis of KPIs, up-to-
the-second operational BI.
Online Gaming 
Real-time back-ends for mobile and massively parallel
games.

SaaS Platforms & Apps 
High performance next-generation architectures for
Software as a Service Application vendors.

Travel & E-Commerce 
High performance next-generation architectures for
online hotel booking.

• What is GridGain Enterprise Edition?
• Is a binary build of Apache Ignite™ created by GridGain
• Added enterprise features for enterprise deployments
• Earlier features and bug fixes by a few weeks
• Heavily tested

What In-Memory Capabilities are Supported?
‣ HPC
‣ Machine learning
‣ Risk analysis
‣ Grid computing
‣ HA API Services
‣ Scalable
Middleware
‣ Web-session
clustering
‣ Distributed caching
‣ In-Memory SQL
‣ Real-time
Analytics
‣ Big Data
‣ Monitoring tools
‣ Big Data
‣ Realtime Analytics
‣ Batch processing
‣ Distributed In-
Memory File
System
‣ Node2Node &
Topic-based
Messaging
‣ Fault Tolerance
‣ Multiple backups
‣ Cluster groups
‣ Auto Rebalancing
‣ Complex event
processing
‣ Event driven
design
‣ Distributed queues
‣ Atomic variables
‣ Dist. Semaphore

Introduction to Clustering

– Distributed Key-Value Store
– Fault Tolerance and Scalability
– SQL Queries (ANSI 99)
– ACID Transactions
– In-Memory Indexes
– RDBMS / NoSQL Integration
– 100% JCache Compliant (JSR 107)
High-level Architecture
On-Heap
Off-Heap
On-Heap
Off-Heap
On-Heap
Off-Heap
Ignite In-Memory Computing Platform
SQL Cache Trans. Compute
DB

Definitions and Terminology
An Ignite cluster is a group of Ignite
nodes working together to accomplish
tasks like distributed compute and caching
An Ignite node is a single Ignite process
running in a JVM
Many Ignite nodes can live on one physical
server or JVM
Ignite nodes can be Clients or Servers
Server/VM/ContainerServer/VM/Container
JVMJVM
…IGNITE
IGNITE
IGNITE
JVM
IGNITE

Shared-nothing clustering involves
multiple identical nodes forming a
cluster with no single master or
coordinator
All nodes in a shared-nothing cluster
run the exact same process
Nodes communicate using message
passing
Ignite Clients & Servers
Server / VM
JVM
IGNITE
Server / VM
JVM
IGNITE
Server / VM
JVM
IGNITE
Server / VM
JVM
IGNITE

An Ignite node can be started as a client
or a server.

Server nodes participate in caching and
computations. Client nodes can also
participate in computations.

Client nodes are used for IgniteAPI
operations from the client side such as
cache operations, transactions, and data
streaming.
Ignite Clients & Servers
CLIENT CLIENT CLIENT
SERVER
SERVER
SERVER
SERVER

In-Memory Data Grid & SQL Grid

Data Grid: Cache modes & Horizontal Scaling
Replicated Cache Partitioned Cache & Near Cache
JVM 1
A
D
Primary
Backup
JVM 2
B
C
Primary
Backup
JVM 3
D
B
Primary
Backup
JVM 4
C
A
Primary
Backup
Client JVM
Remote Client
A
Near Cache
B B
A
CC
JVM 1
A
Primary
Backup
JVM 2
B
Primary
Backup
JVM 3
D
Primary
Backup
JVM 4
C
Primary
Backup
B D C D C A
A B C B A D

Data Grid: Cache modes & Horizontal Scaling
Replicated Cache Partitioned Cache
Client JVM
JVM 1
Local Client
A
D
Primary
Backup
JVM 2
B
C
Primary
Backup
JVM 3
D
B
Primary
Backup
JVM 4
C
A
Primary
Backup
A
Remote Client
A
Near Cache
B B
A
B
C
CC
Client JVM
JVM 1
Local Client
A
Primary
Backup
JVM 2
B
Primary
Backup
JVM 3
D
Primary
Backup
JVM 4
C
Primary
Backup
A
Remote Client
A
Near Cache
B B
A
B
C
CC
B D C D C A
A B C B A D

• Unlimited Vertical Scale
• Avoid Java Garbage Collection
Pauses
• Small On-Heap Footprint
• Configurable eviction policies
• Off-Heap Indexes
• Full RAM Utilisation
• Simple Configuration
Data Grid: Off-Heap Memory

• Multiple (up to 32) Data Centres
• Complex Replication
Technologies
• Active-Active & Active-Passive
• Smart Conflict Resolution
• Durable Persistent Queues
• Automatic Throttling
• GridGain Enterprise
Data Grid: DC Replication
DC1 DC2
DC3
Active
Bi-Directional
Replication
Transactional or
Eventually
Consistent

Data Grid: External Persistence
• Read-through & Write-through
• Support for Write-behind
• Configurable eviction policies
• DB schema mapping wizard:
• Generates all the XML configuration and
Java POJOs
Data
Data
Ignite Cache
Nodes
Write
Through
Read
Through
DB
Ignite
Clients
External
Persistent
Store

Data Grid: Cache APIs
• Predicate-based Scan Queries
• Text Queries based on Lucene indexing
• Query configuration using annotations, Spring
XML or simple Java code
• SQL Queries: Automatic Group By,
Aggregations, Sorting, Cross-Cache Joins,
Unions
• Memcached (PHP, Java, Python, Ruby)
• HTTP REST API
• JDBC & ODBC

• ANSI-99 SQL
• In-Memory Indexes (On and Off-
Heap)
• Automatic Group By,
Aggregations, Sorting
• Cross-Cache Joins, Unions
• Use local H2 engine
Data Grid: SQL Support (ANSI 99)

Data Grid: Transactions
• Fully ACID
• Support for Transactional & Atomic
• Cross-cache transactions
• Optimistic and Pessimistic
concurrency modes with multiple
isolation levels
• Deadlock protection
• JTA Integration

Distributed Java Structures
• Distributed Map (cache)
• Distributed Set
• Distributed Queue
• CountDownLatch
• AtomicLong
• AtomicSequence
• AtomicReference
• Distributed ExecutorService

Continuous Queries
• Execute a query and get
notified on data changes
captured in the filter
• Remote filter to evaluate event
and local listener to receive
notification
• Guarantees exactly once
delivery of an event

• Create chains of event processors & transform an object through various states
• Synchronous or asynchronous execution of remote filters & listeners with thread control
Payment Validator Payment Verifier Payment Processor
Ignite Cache
Event Processing using Ignite

In-Memory Compute Grid & the rest

• Branching Pipelines
• Sliding Windows for CEP/
Continuous Query
• JMS, Kafka, MQTT, Flume,
Camel data streamer
integrations
Streaming and CEP
Event 1 Event 2 Event 3 Event 4
Event Window
Evicted
Event
Incoming
Event
Ignite
Streamers
Data
Data
Ignite
Clients
Ignite Cache Nodes
SQL
1. Process Streamed Data
in Parallel on all Nodes
2. Process SQL Queries in
Parallel on all Nodes

Client-Server vs. Affinity Colocation
1
2
4
3 Data 1
Job 1
2
3
Data 2
Job 2
Processing
Node 1
Processing
Node 2
Client
Node
Data
Node 1
Data
Node 2
Processing
Node 1
1
3
4
Data 1
Data 2
2
2
1. Initial Request
2. Fetch data from remote nodes
3. Process entire data-set
4. Return to client
1. Initial Request
2. Co-locating processing with data
3. Return partial result
4. Reduce & return to client

• Direct API for MapReduce
• Cron-like Task Scheduling
• State Checkpoints
• Load Balancing
• Round-robin
• Random & weighted
• Automatic Failover
• Per-node Shared State
• Zero Deployment
• Distributed class loading
In-Memory Compute Grid
C1
C = C1+C2+C3
R = R1+R2+R3
C2
C3
R3
R2
R1
C
R
In T/3

• Resilience - Build an in-
memory resilient service layer
between your client application
and the grid
• Shielding- Only expose
application APIs and not direct
grid APIs
• Continuations - Call services
internally via compute tasks to
create service chains
In-Memory Service Grid
On-Heap
Off-Heap
On-Heap
Off-Heap
On-Heap
Off-Heap
Ignite In-Memory Computing Platform
SQL Cache Trans. Compute
Service A Service B Service C Service D
DB DB DB

Hadoop & Spark Integration

• Ignite In-Memory File System (IGFS)
– Hadoop-compliant
– Easy to Install
– On-Heap and Off-Heap
– Caching Layer for HDFS
– Write-through and Read-through
HDFS
– Performance Boost
IGFS: In-Memory File System
MR HIVE PIG
In-Memory MapReduce
IGFS
HDFS
IGFS
YARN
}Any
Hadoop
Distro

Hadoop Accelerator: Map Reduce
• In-Memory Performance
• Zero Code Change
• Use existing MR code
• Use existing Hive queries
• No Name Node
• No Network Noise
• In-Process Data Colocation
• Eager Push Scheduling
User
Application
Hadoop
Client
Ignite
Client
Hadoop
Jobtracker
Hadoop
Name Node
Hadoop
Tasktracker
Hadoop
Tasktracker
Ignite
Data Node
(IGFS)
Ignite
Data Node
(IGFS)
Hadoop
Data Node
(HDFS)
Hadoop
Data Node
(HDFS)
Ignite Path
Hadoop Path

• IgniteRDD
– Share RDD across jobs on the
host
– Share RDD across jobs in the
application
– Share RDD globally
• Faster SQL
– In-Memory Indexes
– SQL on top of Shared RDD
Spark & Ignite Integration
Spark Application
Spark Worker
Spark
Job
Spark
Job
Ignite Node
Yarn Mesos Docker HDFS
Server
Spark Worker
Spark
Job
Spark
Job
Ignite Node
Server
Spark Worker
Spark
Job
Spark
Job
Ignite Node
Server
In-Memory Shared RDDs / IGFS

• Docker
• Amazon AWS
• Azure Marketplace
• Google Cloud
• Apache JClouds
• Mesos
• YARN
• Apache Karaf (OSGi)
Cloud Deployment

Thank You!
www.gridgain.com
@gridgain
#gridgain
Thank you for joining us. Follow the conversation.
Author: Christos Erotocritou

OSDC 2017 - Christos Erotocritou - Apache ignite in-memory data fabric

More Related Content

What's hot (20)

Similar to OSDC 2017 - Christos Erotocritou - Apache ignite in-memory data fabric (20)

Recently uploaded (20)

OSDC 2017 - Christos Erotocritou - Apache ignite in-memory data fabric