Transforming your application with Elasticsearch

TRANSFORMING
YOUR APPLICATION WITH
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com

TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Brian Ritchie
CTO for XeoHealth Corporation
22 years of technology experience developing
software and managing technology teams in various
industries.
Author of RavenDB High Performance published by
PACKT in 2013.
Pursuing a Masters in Computer Science from
Georgia Tech.
@brian_ritchie

WELCOME
INTRODUCTION TO ELASTIC SEARCH
BRING YOUR DATA
SECURITY CONSIDERATIONS
PUTTING IT ALL TOGETHER

LET’S GET STARTED
With the story of your application…

IT WAS AMAZING…
It even had the new application smell.

BUT THEN CAME THE USERS, THE DATA, AND THE
NEW REQUIREMENTS….

NOW YOUR APPLICATION IS LEAVING YOU STRANDED
ON THE SIDE OF THE ROAD…

SO, WHY IS YOUR APPLICATION MELTING?
Too Many Reads,
Too Many Writes
= Contention

IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?

NO WAY! IT’S TIME TO PIMP YOUR RIDE!

INTRODUCING

INTRODUCING
Elasticsearch is a search engine based
on Lucene. It provides a lightening fast,
distributed, multitenant-capable full-text
search engine with a HTTP RESTful
interface.
Elasticsearch is released as open source
under the terms of the Apache License.
It can be deployed locally or hosted in the
cloud.

INTRODUCING
Well known for
being part of the
Stack
The Elastic StackNote: this was rebranded
used to analyze log data
generated by your application

WHY
FAST
Document
Centric
Shard/
Partition
Index
Everything
Extremely Fast. Optimized for search. Flexible Schema.
Distributed
Cluster
FOR YOUR APPLICATION

WHY
SCALABLE “We can support clusters of massive scale. Well
into the 100s of terabytes. One of our largest
customers is currently at 750TB and should be in
excess of 1PB by the end of the year. “ - Scalefastr
“Today there are 60+ Elasticsearch clusters and
2000+ nodes. The daily ingestion reaches 18 billion
documents, and daily search requests reach 3.5
billion. ” - eBay
FOR YOUR APPLICATION

YOUR USERS ON

That sounds awesome!!
But, how does Elastic Search fit
into my data infrastructure?
BRINGING YOUR DATA INTO

Utilizing a CQRS inspired pattern
BRINGING YOUR DATA INTO ELASTIC SEARCH
Command/Query Responsibility Segregation (CQRS) is the idea that
you can use a different model to update information than the model
you use to read information
Persistent View Model
Transactional Data

OPTION 1: Add Elastic search into your database pipeline
cluster
Kafka
ActiveMQ
RabbitMQ
….

clusterKafka
ActiveMQ
RabbitMQ
….
OPTION 2: Replicate database updates to Elastic Search

cluster
OPTION 3: Batch your database updates on a schedule

Structuring your data for search
DATA MODELING
Conceptually, indexes are like databases.
• SQL => Databases => Tables => Columns/Rows
• Elasticsearch => Indices => Types => Documents with Properties
Loans Payments Inventory

Defining indexes
DATA MODELING:
Like databases, but more powerful. Queries can easily span indexes bringing
data together as needed.
Use indexes for:
• Security – use indexes to support
multi-tenant system
loan_hud loan_fdic
• Partitioning – time or other
data
log_20180501 log_20180502

Defining documents
DATA MODELING:
{
"FirstName": "Jonathan",
"Address": "15 Wanamassa Point Road ",
"City": "Jacksonville",
"State": "FL",
"Children": [
{ "Name": "Michael", "Age": 10 },
{ "Name": "Jennifer", "Age": 8 },
{ "Name": "Samantha", "Age": 5 },
{ "Name": "Elena", "Age": 2 }
]
}
Instead of decomposing
data into separate entities:
Design documents based on what needs to
be searched and displayed together:

Teach Elastic Search about specific types
DATA MODELING:
PUT _template/file_name_mapping
{
"template": ”loan_*",
"settings": { "number_of_shards": 5 },
"mappings": {
"_default_": {
"dynamic_templates": [ {
"filename_string_not_analyzed": {
"match": "edi_filename",
"mapping": {
"index": "not_analyzed",
"type": "string” }
} } ]
} }
}

Giving the right users access to the right data
SECURITY CONSIDERATIONS
cluster
Search Guard
Custom
.NET Core / Java / etc.
Elastic Stack
Security

The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to
maintain, report, and publicly disclose loan-level information about mortgages.
Searching CFPB loan records w/ Elastic Search

107 million
loan records
var record = csv.GetRecord<dynamic>();
var loan = Loan.FromFileLoan(record);
var index_name = string.Format("loan_{0}",
loan.agency_abbr.ToLower());
var response = client.Index<Loan>(
new IndexRequest<Loan>(loan, index_name));
Loading CFPB loan records into Elastic Search
{
"county": "Maricopa County",
"respondent_id": "0000451965",
"lien_status": "Not applicable",
"sequence_number": "0945293",
"applicant_income": 132000,
"preapproval": "Not applicable",
"applicant_race": "White",
"msamd": "Phoenix, Mesa, Scottsdale - AZ",
….

Using Kibana to search and visualize your data

Using Elastic Search to optimize your user experience
cluster
Single Page
Application
(SPA)
Angular /
React / etc.
JWT
Query
Request
REST API
.NET Core /
Java / etc.
Add security,
logging, etc.

Custom application using Angular, PrimeNG, .NET Core, and Elastic Search

BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
QUESTIONS?
TRANSFORMING
YOUR APPLICATIONS WITH

Transforming your application with Elasticsearch

More Related Content

What's hot (19)

Similar to Transforming your application with Elasticsearch (20)

More from Brian Ritchie (7)

Recently uploaded (20)

Transforming your application with Elasticsearch