HPE NonStop GTUG Berlin - 'Yuma' Workshop

Yuma POC “phase 2” @ comForte
Thomas Burg, March 2016
Results
Product visions

> What is InfiniBand and
why should you care
> The comForte Yuma
POC phase 2 results
> A business case and
technical vision for
„Hybrid NonStop“
> The comForte Yuma
product vision
Agenda

3
“full roundtrip time”
= 0.4 milli seconds
= 400 micro seconds
= 2500 TPS
165 min to
move a
TeraByte
Current
speed NSX
over 1 Gb
Ethernet
Typical speeds for TCP/IP networking (*)

4
= 11 micro seconds
= 90.000 TPS
= 34 x faster
3 min to move
a TeraByte
= 55 times
faster
Awesome
Speed of IB
Moving to InfiniBand, comparing to 1 Gbit TCP IP over
Ethernet…
(*) How fast is fast enough? Comparing Apples to ….

5
When do I need to move data really really fast …
BigData
(Duh)
Stock
Exchanges
Telco
NonStop
Hybrid
Discussion
to follow

> Context
> Goals
> Results
> Moving data
> Moving files
> Other observations
The comForte Yuma
(a.k.a. NSADI)
POC Phase 2

comForte – better always on
Phase 2 of comForte Yuma POC - Context
>Now all on comForte Hardware
> comForte owned and operated NS3 X1
> HPE ProLiant, RHEL Linux, Mellanox IB card
> Only a single InfiniBand cable, no Switch on
“Linux end” of connection
>Still with plenty of help from HPE folks
> Direct contact with key developers
> Direct contact with HPE product management
Thank you much HPE!
HPE
ProLiant

Phase 2 of comForte Yuma POC - Context
>comForte Resources for Phase 2
> comForte: Thomas Burg, various folks in sys
admin NonStop and Linux
> Gemini: Richard Pope, Dave Cikra
>Gemini Communications, Inc.
> www.geminic.com
> No direct sales
> Several ‘comm’ products over the decades, some
of them sold by comForte now

Phase 2 of comForte Yuma POC - Goals
>Compare InfiniBand with 1 Gbit TCP/IP
> Like all NS3 X1, comForte system does not have 10 Gbit Ethernet
> Hence 10 Gbit could not be measured
> Compare 1 Gbit Ethernet with InfiniBand
>Re-measure some key data points for ‘moving of data’:
> Latency and throughput for ‘typical’ packet sizes
> Maximum throughput using ‘optimal’ packet sizes
>Can we do ‘FTP over InfiniBand’ and if so, how fast?

Phase 2 of comForte Yuma POC - Disclaimer
>It has been a tight race to GTUG
> The comForte NS3 X1 system was delivered in October 2015
> The Linux system was set up in January 2016
> The missing InfiniBand cable was ordered in February 2016
> InfiniBand was up and running in March 2016
>Please treat all number as preliminary. Things should only
get better, but all numbers are the result of a POC, rather
than benchmarks of a finished product

The comForte Yuma POC Phase 2
Moving Data

Moving data – model used
>For “POC Phase 1” (TBC Nov 2015) we used ‘echo’ approach
> Send some bytes of data
> Send same packet size back
>For “POC Phase 2” (GTUG April 2016) we used ‘one way’ approach
> Send some bytes of data
> Send small packet (“acknowledgement”) back
>Both models occur in real life, but we felt ‘one way’ is more common

Moving data 16 KBytes – results
>‘one way’ approach (see prior slide)
> 16 KBytes = 16384 bytes data, 20 bytes “ack”
> Data moves from NonStop to Linux
Transport over Latency
(microseconds)
MegaBytes/s
TCP/IP 1 Gbit Ethernet 374 43
InfiniBand 11 1413
InfiniBand gain x 34 x 32

Moving data optimum packet size – results
>‘one way’ approach
> ‘Optimal’ packet size chosen for InfiniBand and TCP/IP, “ack” still 20 bytes
> Data moves from NonStop to Linux
Transport over Packet size Chunk of data moved
to measure real time
[in GigaBytes]
Real time
elapsed
[in seconds]
Throughput
[in MegaBytes/s]
TCP/IP 1 Gbit Ethernet 262144 10 97 102
InfiniBand 2097152 1024
[one TeraByte]
176 5734
InfiniBand gain x 55
> time to move one TeraByte over TCP/IP 1 Gbit Ethernet extrapolates to 9900 seconds

Moving files from NonStop to Linux

‘FTP’ over InfiniBand - introduction
>During POC phase 1 comForte and Gemini managed to connect
NonStop FTPSERV with Linux open source FTP client
> No modifications to NonStop FTPSERV (!). Used comForte “TCP/IP to InfiniBand
intercept framework” (see next slide)
> Converted Linux open source FTP client to rsockets
>FTP protocol is NOT ‘InfiniBand’ friendly
>During POC phase 2 we focused on speed measurements, hence
we wrote test programs with direct file I/O on both ends

comForte FTPSERV over IB POC (done for TBC 2015)
>This worked, but
it needed some
‘tricks’
>Performance was
good, but not
faster than 10
Gbit Ethernet,
about 300 MB/s
>Works for Telnet
as well 
HP NonStop Linux (Red Hat)
FTPSERV
Guardian
Open Source FTP
Client, ported to
rsockets
comForte
TCP/IP
Intercept
Library
comForte IB
Daemon
(OSS 64bit
PUT)
InfiniBandIPC
rsockets
NonStop file
system
Linux file
system

‘FTP’ over InfiniBand – changes for Phase 2 of POC
>No longer use FTP protocol at all
>Have comForte code on both ends
>Full control, no extra IPC between Guardian and OSS layer

comForte ‘FTP’ over Infiniband April 2016
HP NonStop Linux (Red Hat)
comForte InfiniBand file server
OSS, 64bit PUT
InfiniBand
(rsockets)
NonStop file
system
Linux file
system
comForte InfiniBand file client
C, Native Linux

FTP over TCP/IP, 1 Gbit Ethernet
> Single file read maxes out @ about 150 MByte/s [used test program for this]
> TCP/IP maxes out @ about 128 MByte/s
> FTP file transfers based on number of parallel transfers for a 1 GigaByte file from NonStop to Linux

‘FTP’ over Infiniband – POC results
> InfiniBand has no real limit here  [it is about 6 GByte/s]
> ‘FTP’ file transfers based on number of parallel transfers, same file, but now over InfiniBand:
> Already moved from 111 MByte/s to 410 MByte/s  Nearly four times faster
> Limitations of file transfer speed are now:
> How effectively can we “scale out” File I/O read operation
> This was measured on a two CPU NS3 X1

Moving data from NonStop to Linux
– testing the limits on Linux and InfiniBand
> Use ‘FTP over InfiniBand’ POC framework
> Do *not* do file read on NonStop, use test data created in memory
> Send data to Linux, flush to disk
> This measures
> Disk write speed on Linux
> How well current comForte POC FTP over InfiniBand file server and client scale

Moving data from NonStop to Linux
– testing the limits on Linux and InfiniBand
> Scales up nicely on a two CPU system with a single InfiniBand cable

What to make of ‘FTP over IB’ results
> comForte can move data real fast from NonStop to Linux
> 6 GigaBytes per second seems doable on a fully scaled out NS7 X1
> This includes flushing the data to Linux Disk
> Potential use cases (???):
> Fast replacement for FTP
> Data replication
> Big data
> Backup

Other observations

Other observations during POC
> Setting up InfiniBand hardware on NonStop and Linux is new to sysadmin folks (both on Hardware and
Software level)
> InfiniBand rsockets interface is straightforward to code, both on NonStop and Linux
> InfiniBand Low level verb interface is NOT straightforward to code
> Did not get beyond very early POC code but making progress
> InfiniBand and rsockets are rock solid both on NonStop and RHEL Linux
> rsockets is only available from OSS PUT64 (not available under Guardian!). That’s why comForte built a
plug-compatible sockets DLL for Guardian socket apps (like CSL, FTPSERV, anything using TCP/IP under
Guardian)
> HPE NonStop InfiniBand team very competent and helpful

A business case and technical vision for
„Hybrid NonStop“

Cloud Business Case “Looking versus Booking”:
Many NonStop systems as of today
NonStop System
transactions coming
from “somewhere”
DATABASE
Server classes
encapsulating
business logic
Looking and Booking traffic (typical use case for multiple
NonStop customers in travel section):
Looking is stateless, 95+% of traffic
By nature of transaction, can be hosted in cloud or on
commodity platform
Booking is transactional
By nature of data, you don’t want to lose it and it also has
“state” (ACID) – run it on NonStop
Similar two-types-of-transaction logic applies to stock
exchanges, potentially other verticals (Base24 !?)

Cloud Business Case “Looking versus Booking”:
The high level requirement/vision
NonStop System
transactions coming
from “somewhere”
DATABASE
Server classes
encapsulating
business logic
Looking does not hit NonStop at all
… and is handled in the cloud (public or private)
… but how to move ‘state’ (database) to cloud???

Cloud Business Case “Looking versus Booking” – InfiniBand and NonStop Hybrid vision
NonStop System
Server classes
encapsulating
business logic Looking indeed handled in cloud
transactions do not hit NonStop (business tier knows it is
Looking and hence simply uses local DB copy)
Cloud tier sends Booking transactions to NonStop,
via Infiniband (again, business logic sees this is Booking, hence
switches to NonStop)
Fast replication via InfiniBand enables (one-way, “read only”,
near real-time) replication to multiple Linux boxes in parallel
with low latency and low CPU overhead
CLOUD
Web traffic – looking
and booking
CLOUD
Database,
Replicated into
cloud near
real-time
Cloud tier
business logic
DATABASE

> CSL/Infiniband
> Become *the*
company for IB-
enabling applications
and middleware
products
> Work with ISVs, end
usersThe comForte Yuma Product Vision

32
CSL/InfiniBand
>Covers “left half” of
InfiniBand Hybrid
vision
>Available very soon…

33
CSL/InfiniBand
> A very natural extension of the CSL product
> A new option CSL/InfiniBand
> First release will provide C/C++ API on Linux
> To be announced @ GTUG Berlin, again at TBC 2016
> EAP-ready October 2016
> Come to comForte presentation or talk to us to find out more

34
The broader comForte Yuma framework
> Can InfiniBand-enable *any* existing application on HPE NonStop
> Without application changes (!)
> Just like SecurData and CSL – it is a *framework*
> Existing application/middleware on NonStop
> InfiniBand will boost performance
> With comForte experience from POC and framework to be announced @ TBC 2016, Middleware/application
vendors can focus on their features, comForte takes care of InfiniBand details
> Customer/partner needs to do work on Linux himself
> Rather easy via rsockets approach
> comForte can provide proxy (speed to be confirmed)
> comForte can help
> comForte vision: Become THE player in “InfiniBand low level coding”

35
The broader comForte Yuma framework (contd.)
>Whom does this help moving from TCP/IP to InfiniBand?
> NonStop ISVs
> Software houses with their own applications
> NonStop users with their own applications
>Interested?
> Come talk to comForte

37
Summary, Q&A
>HPE NonStop is now InfiniBand enabled to
connect to HPE ProLiant Server running RHEL
>InfiniBand is extremely fast
>Now that HPE has created an environment
that we can build on with InfiniBand,
comForte has several products which can be
used in the “Hybrid space”
> CSL/InfiniBand
> InfiniBand enabling framework
> maRunga/InfiniBand (?)
>Time to start moving to Hybrid!?
= 11 micro seconds
= 90.000 TPS
= 34 x faster
3 min to move a
TeraByte
= 55 times faster
THANK YOU !
Questions?

HPE NonStop GTUG Berlin - 'Yuma' Workshop

More Related Content

What's hot (20)

Similar to HPE NonStop GTUG Berlin - 'Yuma' Workshop (20)

More from Thomas Burg (12)

Recently uploaded (20)

HPE NonStop GTUG Berlin - 'Yuma' Workshop