Cache is King - RubyHACK 2019

@molly_struve
Cache is King
Get the Most Bang for Your Buck From Ruby

@molly_struve
Site Reliability Engineer

@molly_struve
Adding Indexes
Using SELECT statements
Batch Processing

@molly_struve
Elasticsearch::Transport::Errors::GatewayTimeout 504
{
"statusCode": 200,
"took": "100ms"
}

@molly_struve
Quantity of Datastore Hits

@molly_struve
The average company has...
60 thousand
assets
24 million
vulnerabilities?

@molly_struve
MySQL
Elasticsearch
Cluster

@molly_struve
MySQL
Elasticsearch
Cluster
ActiveModelSerializers

@molly_struve
module Beehive
module Serializers
class Vulnerability < ActiveModel::Serializer
attributes :id, :client_id, :created_at, :updated_at,
:priority, :details, :notes, :asset_id,
:solution_id, :owner_id, :ticket_id
end
end
end

@molly_struve
11 hours and counting...

@molly_struve
(1.6ms)
(0.9ms)
(4.1ms)(5.2ms)
(5.2ms)
(1.3ms)
(3.1ms)
(2.9ms)
(2.2ms)
(4.9ms)
(6.0ms)
(0.3ms)
(1.6ms)
(0.9ms)
(2.2ms)
(3.0ms)
(2.1ms)
(1.3ms)
(2.1ms)
(8.1ms)
(1.4ms)

@molly_struve
Bulk Serialization

@molly_struve
class BulkVulnerabilityCache
attr_accessor :vulnerabilities, :client, :vulnerability_ids
def initialize(vulns, client)
self.vulnerabilities = vulns
self.vulnerability_ids = vulns.map(&:id)
self.client = client
end
# MySQL Lookups
end

@molly_struve
module Serializers
class Vulnerability
attr_accessor :vulnerability, :cache
def initialize(vuln, bulk_cache)
self.cache = bulk_cache
self.vulnerability = vuln
end
end
end
self.cache = bulk_cache

@molly_struve
class Vulnerability
has_many :custom_fields
end

@molly_struve
CustomField.where(:vulnerability_id => vuln.id)
cache.fetch('custom_fields', vuln.id)

@molly_struve
The Result...
(pry)> vulns = Vulnerability.limit(300);
(pry)> Benchmark.realtime { vulns.each(&:serialize) }
=> 6.022452222998254
(pry)> Benchmark.realtime do
> BulkVulnerability.new(vulns, [], client).serialize
> end
=> 0.7267019419959979

@molly_struve
Decrease in database hits
Individual Serialization:
Bulk Serialization:
2,100
7

@molly_struve
1k vulns 1k vulns 1k vulns
Vulnerability Batches

@molly_struve
1k vulns 1k vulns 1k vulns
Vulnerability Batches
7k 7

@molly_struve
MySQL Queries
Bulk Serialization
Deployed

@molly_struve
Bulk Serialization
Deployed
RDS CPU Utilization

@molly_struve
Elasticsearch
Cluster
+
Redis
MySQL
Vulnerabilities

@molly_struve
Redis.get
Client 1
Index
Client 2
Index
Client 3 & 4
Index

@molly_struve
indexing_hashes = vulnerability_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end

@molly_struve
(pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”)
DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234"
DEBUG -- : [Redis] call_time=1.07 ms
GET

@molly_struve
client_indexes = Hash.new do |h, client_id|
h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”)
end

@molly_struve
indexing_hashes = vuln_hashes.map do |hash|
{
:_index => Redis.get(“elasticsearch_index_#{client_id}”)
:_type => hash[:doc_type],
:_id => hash[:id],
:data => hash[:data]
}
end
client_indexes[hash[:client_id]],

@molly_struve
1 + 1 + 1
Client 1 Client 2 Client 3
1k 1k 1k

@molly_struve
65% job
speed up

@molly_struve
Process
in Bulk
Hash
Cache

@molly_struve
Sharded Databases
CLIENT 1
CLIENT 2
CLIENT 3

@molly_struve
Asset.with_shard(client_id).find(1)

@molly_struve
{
'client_123' => 'shard_123',
'client_456' => 'shard_456',
'client_789' => 'shard_789'
}
Sharding Configuration

@molly_struve
Sharding Configuration Size
20 bytes 1kb 13kb

@molly_struve
ActiveRecord::Base.connection

@molly_struve
(pry)> ActiveRecord::Base.connection
=> #<Octopus::Proxy:0x000055b38c697d10
@proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8

@molly_struve
module Octopus
class Proxy
attr_accessor :proxy_config
delegate :current_shard, :current_shard=,
:current_slave_group, :current_slave_group=,
:shard_names, :shards_for_group, :shards, :sharded,
:config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false
end
end

@molly_struve
Process
in Bulk
Framework
Cache
Hash
Cache

@molly_struve
Avoid making datastore hits you don’t
need

@molly_struve
User.where(:id => user_ids).each do |user|
# Lots of user processing
end

@molly_struve
(pry)> User.where(:id => [])
User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0
=> []

@molly_struve
return unless user_ids.any?
end

@molly_struve
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
> 10_000.times do
> next unless ids.any?
> User.where(:id => [])
> end
> end
=> 0.0006368421018123627

@molly_struve
> 10_000.times { User.where(:id => []) }
> end
=> 0.5508159045130014
“Ruby is slow”Hitting the database is slow!

@molly_struve
end
users = User.where(:id => user_ids).active.short.single

@molly_struve
(pry)> User.where(:id => []).active.tall.single
User Load (0.7ms) SELECT ùsers`.* FROM ùsers` WHERE 1=0 AND
ùsers`.àctive` = 1 AND ùsers`.`short` = 0 AND ùsers`.`single` = 1
=> []
(pry)> User.none.active.tall.single
=> []
.none in action...

@molly_struve
Logging
pry(main)> Rails.logger.level = 0
$ redis-cli monitor > commands-redis-2018-10-01.txt
pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

@molly_struve
Preventing useless
datastore hits

@molly_struve
Report
Elasticsearch
MySQL Redis

@molly_struve
(pry)> Report.blank_reports.count
=> 10805
(pry)> Report.active.count
=> 25842
(pry)> Report.average_asset_count
=> 1657
Investigating Existing Reports

@molly_struve
return if report.asset_count.zero?

@molly_struve
Process
in Bulk
Framework
Cache
Database
Guards
Hash
Cache

@molly_struve
Resque Workers
Redis

@molly_struve
45 workers
45 workers
45 workers

@molly_struve
70 workers
70 workers
70 workers

@molly_struve
Redis Requests
70 workers
100k
200k

@molly_struve
Resque
Throttling

@molly_struve
Redis Requests
100k
200k

@molly_struve
Redis Network Traffic
48MB
16MB

@molly_struve
Process
in Bulk
Framework
Cache
Database
Guards
Remove
Datastore
Hits
Hash
Cache

@molly_struve
Every
datastore hit
COUNTS

@molly_struve
Contact
https://www.linkedin.com/in/mollystruve/
https://github.com/mstruve
@molly_struve
molly.struve@gmail.com

Cache is King - RubyHACK 2019

More Related Content

What's hot (20)

Similar to Cache is King - RubyHACK 2019 (20)

More from Molly Struve (10)

Recently uploaded (20)

Cache is King - RubyHACK 2019