JavaOne-2013: Save Scarce Resources by Managing Terabytes of Objects off-heap or Even on Disk

Use your current wisely
Harvey Raja
Oracle
Chris Neal
Pegasus

§  Senior Engineer
§  Oracle Coherence
Introduction
Harvey Raja

§  Systems Architect
§  Pegasus Solutions
Introduction
Chris Neal

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.5 5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
The following is intended to outline our general
product direction. It is intended
for information purposes only, and may not be
incorporated into any contract.
It is not a commitment to deliver any material,
code, or functionality, and should not be relied
upon in making purchasing decisions. The
development, release, and timing of any features
or functionality described for Oracle s products
remains at the sole discretion of Oracle.

What’s all this about?
§  Big Data / Big Memory on a transistor diet
§  Applications and conceived concerns
§  Object Profiling
§  Elastic Data
§  Improvements

Heap me up
§  JVM manages our objects
§  Understands Live Data
–  References
–  Free Lists
new Object()

Heap me up
§  Two distinct regions of data locality
–  Young Generation
–  Old Generation
§  Allows conscious distinction between:
–  Long living objects
–  Short lived objects

Heap me up
§  All memory allocations are against the
same resource
§  Why would it be any different?
§  Provides means to access
–  off-heap memory
–  File Descriptors a.k.a. any resource

Heap me up
JVM
100%
0%
0%

Applications & Their Objects
§  Every application has very different uses of objects
–  Size
–  Scope
§  Structures / Containers
–  Structures {Containers}
–  Containers {Structures}

Application Object Profiles
§  Ye olde faithful… Pet Store
§  Short-lived objects
–  Search Results
§  Long-living objects
–  Popular items
JEE Pet Store
JEE Pet Store
JEE Pet Store
JEE Pet Store
Pet Store

Application Object Profiles
§  Foreign Exchange Position Keeping
§  Aggregate Trades Values per
currency pair
§  Some currencies are a ‘busier’
§  Currencies may have varying SLAs
FX Position Keeping
T
T
T
T
USD-GBP
USD-EUR
T
T T
T
T
T
T
T
T
T
T
T

What we know about applications?
§  The application understands more about each object than the JVM
–  Frequency
–  Size
§  Keeping everything in RAM is
possible but is it efficient?
§  Huge Leap between Object on heap &
stored in DB
-  Recency
-  Custom characteristics of the object

Can the JVM just make the right choice?
§  The JVM would have to span multiple devices
§  Non-Heap must be serialized
§  Applications are diverse therefore to make generic decisions on object
usage would likely lead to false-positives
§  Down to the application or a layer above the JVM allowing users to
define resource assignment policies

Municipality
§  The application deems its own usage of each resource
§  JVM provides primitives to load & store to these devices
§  May be useful to have an API that performs this storage appropriate to
the device:
–  Routing stores to appropriate device
–  Handle concerns of multiple applications on the same JVM

CPU Architectures
§  As latency increases so does
capacity
§  Data fetched as required by instructions
§  Data is demoted as well as writes to
shared data rippling through the caches

CPU Architectures
Speed Capacity
Registry 1ns 0.00
L1 Cache 2-5ns 2x32KB
L2 Cache 5ns 256KB
L3 Cache 20ns 8MB
Main Memory 20-60ns 16GB
Nehalem2GHZ
processor
Mayfair
Kensington
Camden
Wimbledon
Manchester
§  Perhaps we should charge our objects rental premiums?

Blank Canvas
£600pcm
£1800pcm
£2400pcm
£100000pcm
£100
Per TimeUnit
£500
Per TimeUnit
£250
Per TimeUnit
£1000
Per TimeUnit

How do you choose the right property?
§  How do you select the right property:
–  How often are you in the office?
–  How long does it take you to get into the
office?
–  How much space do you occupy in the
office?
–  How do you get to the office?
–  MFU
–  Device latency
–  Object size
–  Every device has its own
quirks

Translated to memory
USD -> GBP
USD-> JMD
Heap RAM
(off heap)
Disk
GBP -> MUR
Usage
Pet store
example
Pedigree Chum
Orijen
Dog Ugg boots

§  Objects held on heap are:
–  Structures (Containers)
–  Containers (Structures)
§  Similar to a file descriptor, each object has its own metadata:
–  Access Time
–  Modified Time
§  With ((Map) Containers) we already have a location to store
metadata
-  Size
-  Touch Count

CPU
Heap
NIO
Flash
Mechanical Disk
§  Each object has metadata
§  Some policy can manage these
objects
§  Demoted & promoted to the
various media types
§  Big jump from heap to NIO

Brainstorm Summary
§  Cherish your high commodity investments
§  Reduce the regularity of going to a highly contended foreign
resource
§  Would be ideal to have objects float between high latency resources
using a telepathic API
–  Having some metadata could drive our decision for data locality
§  Map provides a nice abstraction for objects that should ripple

Challenges
§  Serialization cost
§  Generally interactions are performed against the object form
§  False Positives
§  Device type peculiarities
§  A handle to the object (key) and metadata must be held on heap

Elastic Data
§  A Feature in Coherence
§  Store binary key and value objects
in RAM or Disk
§  Overflow from RAM to Disk
§  RAM can be configured as NIO
store(byte[] key, byte[] val)
load(byte[] key)
erase(byte[] key)
RAM
Flash

Elastic Data
§  Its simply writing a number of bytes to some stream?
§  How do you maintain handles?
§  Need a pointer to the written data?
§  How about updates, seek & replace?
Easy Peasy

Elastic Data
§  Require a compact structure to hold handles (keys) to device pointers
§  Journal writes to the file system
§  Consistent API regardless of write to RAM, NIO, Flash or Mechanical
Disk
§  Buffer writes with thread dedicated to writing
Implementation Details

Elastic Data
FlashJournal
RAMJournal
Collector
Overﬂow
Preparer
Writer
Binary Key Pointer
00110101001
11001001011
...
...
Serialize
110101
LFU
Collector
Store Index
Object deemed
unworthy of Heap
store(key, value)Pointer returned
Create

Elastic Data
FlashJournal
RAMJournal
Collector
Overﬂow
Preparer
Writer
Binary Key Pointer
00110101001
11001001011
...
...
Collector
Store Index
Deserialize
110101
Deserialized object
returned
load(pointer)binary value
The binary
key provides
the physical
location of
the stored
item
Read

Elastic Data
Elastic Data
100%
100%
100%
Utilization

Elastic Data
§  RAM Journal can be used with NIO ByteBuffers
–  Memory managed using same mechanisms between RAM & Flash
§  Consider device specifics prior to use and design components /
interactions accordingly
§  Multi-threaded clients
§  Flash vs Mechanical
More Features

Elastic Data
§  Several platters & reading / writing heads
§  Faraday & Lorentz
§  Seek time + Rotational Latency = L
§  Can not get to the same speeds as the disk controllers
Mechanical Media

Elastic Data
§  NAND gates
§  MLC
§  Write in pages erase in blocks
§  No Seek Time or Rotational Latency
Flash Media

Elastic Data
JVM
handle *
handle *
handle *
handle *
handle *
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
0111011100110110101
0111010110100110101
handle *(disk pointer)
HARRIET *
HARVEY *
HILARY *
HILTON *
Handles are
stored in
process
Number of handles
constrains amount of
data that can be
stored
Key Management

Elastic Data
H
AR
RIET VEY
IL
ARY TON
HARRIET
HARVEY
HILARY
HILTON
1 2 4 8Tickets:
§  Data structure to hold handles
is a Binary Radix Tree
–  allows sharing
of common denominators
§  Handles (keys) are stored in
serialized form
§  Benefit increases as common bytes increase and less
heap memory is used

Elastic Data
§  Writes are journalled
§  Erase is a logical removal
§  Update = erase + write
§  Avoids seek time or cascading pointer
changes
write write
write
APPEND
APPEND
APPEND
erase
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
010111001
101111010
100111010
100111100
File 1
File 2
File 3

Elastic Data
§  Require a Journal Garbage Collector
–  Reclaim unreferenced memory
§  Evacuation process for each file
§  Eviction logically removes from Journal File
§  May enter an exhaustive mode
–  Synonymous to Full GC
1001110
100111100
010111001
101111010
100111101
110011000
1001110
100111100
010111001
101111010
100111101
110011000
1001110
100111100
010111001
101111010
100111101
110011000
1001110
100111100
010111001
101111010
100111101
110011000
1001110
100111100
010111001
101111010
100111101
110011000
1001110
100111100
010111001
101111010
100111101
110011000
1001110
100111100
010111001
101111010
100111101
110011000
Journal
Collector
sort
evacuate
(exhaustive-
mode)

Elastic Data
§  Dedicated threads to unblock writes
–  Tuned to device type
§  Client write appears to be as fast as
heap write
§  Overwhelming number of writes will
result in push back to the client
Preparer Buffer
Writer

Who am I?
§  Chris Neal, Systems Architect
§  Started with Pegasus in 1994
§  Worked with Coherence since 3.3.1 in 2007
§  Participate in Coherence CAB

Who is Pegasus Solutions?
§  Founded in 1988
§  Provide technology and services to hotels and travel distributors
§  Three main service areas:
–  Representation Services
–  Distribution Services
–  Central Reservation Services (CRS)

Distribution Services
–  Connects hotel systems with distribution partners.
–  100,000 hotels connected to all major distributors (Expedia, Orbitz, Hotwire, Travelocity, etc)
–  Cheaper and easier than a direct connect
–  If you book a hotel online, chances are your transaction goes through Pegasus.
–  Pegasus processes roughly 8 billion transactions per month… sustained ~3000TPS @200ms
latency or less.

Why do we need a cache?
§  In travel agent days, “Look to book” ratios were 3:1
§  At internet scale, they are roughly 4200:1
–  Travel aggregators like Sidestep, Kayak, Mobissimo, etc burst this to >100,000:1
§  Looks are the most expensive transaction from systems processing perspective.
§  We make no money on “looks”, so saving resources by not processing these
transactions is important to both Pegasus and the downstream hotel systems.
§  …Hello Coherence….

Distribution Services and Coherence
Physical deployment:
–  Production cluster consists of 6 servers, 144GB RAM each.
–  Each server runs 3 storage enabled JVMs
–  Servers run Solaris x64
–  Each Hotspot JVM is 32GB
–  Using CMS collector, and having no GC pauses
–  The vast majority of the storage space is for AvailibilityCache (22GB) to service the “looks”
Client Applications
–  ~120 storage disabled clients either in containers or stand alone
–  Backing store is so large that NearCache is disabled

The problem…
§  The cache is too small. Empty to full in ~3 hours.
§  Evicting valuable, usable data
§  Cache hit rate is too low

The Challenge….
§  Increase the cache size on the current servers from 20M to 200M
§  Spend as little money as possible
§  Do it by EOY

The Process…
§  “spend as little as possible” means adding servers to reach 200M is not
possible, which means no more RAM.
§  Enter ElasticData
§  In terms of $, RAM > SSD > SATA, but is SATA fast enough?

iozone
§  Reads and writes a file to a filesystem as fast as it can
§  Compare SSD to SATA with regards to throughput
§  We know SSD is faster, but will SATA do?
www.iozone.org - Filesystem benchmark tool

SSD benchmark results

SATA benchmark results, part 1

SATA benchmark results, part 2

Elastic Data Hardware configuration

What we observe
§  At production volumes in RAM:
–  Avg get/put times ~2ms
§  At production volumes on SATA:
–  Avg put times ~3ms
–  Avg get times ~10ms
Benchmark data through the application

Hurdles We Have Overcome
§  Configuring the heap size:
–  Enough room to store keys for 1.6TB of objects, so that Full GCs do not
occur
–  Enough room to store puts() while partitions are being evicted
–  64GB (up to 72GB with G1)
§  Eviction killing throughput. Eviction process was reading the values from disk
at it evicted (for indexes and listeners). Behavior was changed with
BlindCompactSerializationCache.
§  Stopping a JVM: Transferring full partitions. Instead, drop the data, then
transfer (DropContentPartitionListener)
§  Starting a JVM: Rebalancing partitions to the new JVM.
(DropContentPartitionListener)

Configuring Coherence for Elastic Data
§  Operational overrides:
<journaling-config>
<ramjournal-manager>
<minimum-load-factor>.4</minimum-load-factor>
<maximum-size>8GB</maximum-size>
</ramjournal-manager>
<flashjournal-manager>
<minimum-load-factor>.7</minimum-load-factor>


<maximum-file-size>3690MB</maximum-file-size>
<collector-timeout>30m</collector-timeout>

<high-journal-size>1600GB</high-journal-size>
</flashjournal-manager>
</journaling-config>

Configuring Coherence for Elastic Data
§  Cache-config.xml
<distributed-scheme>
<backup-count>0</backup-count>
<partition-listener>
<class-name>com.tangosol.net.partition.DropContentPartitionListener</class-name>
</partition-listener>
<backing-map-scheme>
<ramjournal-scheme>
<class-name>com.tangosol.net.cache.BlindCompactSerializationCache</class-name>
<high-units>1600KB</high-units>
<low-units>1400KB</low-units>
<unit-calculator>Binary</unit-calculator>
<unit-factor>1048576</unit-factor>
</ramjournal-scheme>
</backing-map-scheme>
</distributed-scheme>

Garbage Collection settings
argv[12]: -XX:+UseG1GC
argv[13]: -XX:MaxGCPauseMillis=800
argv[14]: -XX:ConcGCThreads=10
argv[15]: -XX:ParallelGCThreads=10
argv[16]: -XX:InitiatingHeapOccupancyPercent=25
argv[17]: -XX:NewRatio=16

Did we meet our goals
§  Goal: 500M cached objects
–  Actual: 1.6B cached objects
§  Goal: Spend as little as possible
–  Actual: Spent 1400/machine (8640 total), 84x more objects
§  Goal: Do it by EOY
–  Actual: On track for production release before EOY
So far, so good…

Graphic Section Divider

JavaOne-2013: Save Scarce Resources by Managing Terabytes of Objects off-heap or Even on Disk

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to JavaOne-2013: Save Scarce Resources by Managing Terabytes of Objects off-heap or Even on Disk (20)

Recently uploaded (20)

JavaOne-2013: Save Scarce Resources by Managing Terabytes of Objects off-heap or Even on Disk