Joyent circa 2006 (Scale with Rails)

Scale with Rails
Real-life capacity and deployment planning for
small to large scale Rails applications

The Guys {Thanks}
David Young, Dean Allen, Jason Hoffman, Matt
Imbach, Michael Koziarski, Scott Barron, Chris
Morris, Luke Crawford, Justin French, Johan
Sörensen, Ben Myles, Marten Veldthuis, Bryan
Bell, Ryan Schwartz, Jan Isley, Florian Munz,
Filip Hajný, Daniel Crowell, Christopher Horrell,
Terrell Russell, Peter Watridge, Josh Roebuck

Today
• Please humor me => a 5 minute story with too
many slides about how I came to stand here in
front of you
• Somethings about the Ruby on Rails site
• An accounting what’s buzzing in my ears all the
time
• Some examples with numbers
• How to think about “scaling”
• To talk about where we’ve taken our
infrastructure

It started with Textpattern
The best CMS/blog thing written in PHP ever

We used basecamp
Ooooooh, it’s in Ruby? Reeeeeeeeeally.

So we thought let’s involve some
friends
• Dean Allen (Textpattern)
• Matt Mullenweg (Wordpress)
• Brad Choate (doing MT hosting + plugins)
• David Heinemeier Hansson (Instiki and “Rails”)
• Rickard of PunBB
• Allan => Textmate

What did they get me into?
5/2003 9/2005
5/2004
“it” started

Joyent circa 2006 (Scale with Rails)

Books are important
Not going to pay in my house.
Send those books out.

rubyonrails.com
~3500-4500 TBs/month (~10-15Mbps)

But sometimes it’s gotten “bad”
• Ruby on Rails
• Turbogears
• Textmate
• All have videos (”screencasts”)
• They’ve been in the same paragraph on
Slashdot
• Then we’ve pushed 150-200 Mbps
• Open Source is free?

Some of Joyent’s
We combined last year to be the
“perfect” company.

OK, So what?
• We found ourselves needing an infrastructure
for “diverse” things
• But fundamentally we’re talking about web
stuff, mail stuff, database stuff, storing and
moving around ﬁles stuff

And why are we here?
• One indication: because I get, on average, 2
emails/day asking the same questions
• That’s just “me”
• Support system and “sales” gets them as well

Examples from last week
• I have yet to ﬁnd any examples of websites that have heavy traffic and
stream media that run from a Ruby on Rails platform, can you suggest
any sites that will demonstrate that the ruby platform is stable and
reliable enough to use on a commercial level?
• We are concerned about the long-term viability of Ruby on Rails as a
development language/environment.
• How easily can a ruby site be converted to another language? (If for any
reason we were forced to abandon ruby at some point in the future or I
can’t ﬁnd someone to work with our code?).
• My company has some concerns on whether or not Ruby on Rails is the
right platform to deploy on if we have a very large scale app.
I could go on for a while.

So what is?
• A “large scale” application?
• Do many of us really have one? I mean a BIG
ONE?
• What is an “intensive” application?
• What is an “enterprise” application?

But we’ve been doing “it” with
• Perl
• PHP
• Python
• Java
• Does the language really really matter to the
system’s folks?

Ah-ha!
• System’s Folks you say?
• What are these “system administrators” that you
speak of?
• Does the network really matter?
• You mean there’s “networking people”?
• You mean it takes more than just having my
designer(s) learns The Rails?

If I had a nickel for every time ...
• “We’re going to need to scale this up to Flickr-
sized proportions!”
• “This could go very very fast!”
• “The market is HUGE”

Reality Check
• “OK”, I say
• “Not a problem”, I say
• “I can do that”, I say
• “For what you’re asking, I’ll need $325,000
tomorrow to start, it’ll take $18,500 a month to
run and you can expect that to go up along with
your growth.”
• “It’s a good rule of thumb to try and keep it
about 10% of your revenue once you’re going
because people ended being the most
expensive”

Oh. But we don’t have that kind
of money. And the app is free to
start.

What a minute!
• Why are you going on and on about this not
worry about scaling til you need to scale stuff?
• I can read this same thing on what’s their face’s
weblog.
• It’s not what I’m asking about!
• I need to know if I can “scale” with ruby!
• We’re not a start-up, why are you talking about
this from a “start-up’s” perspective?!
• I need a ...

A What?
• A framework for thinking about the entire
system?
• You mean it’s more than just the language?
• It’s more than about a readily available IDE?
• You mean one’s “development” framework is
just one small part?
• My system’s guys can scale anything.

And oh yes ... you should worry
still worry about scaling

Because what does “scaling” actually
mean?
• For a little software application company?
• Can you do the Start-up => Mid-sized, “small
caps” company transition without going out of
business?
• In a big company?
• Can you do what you gotta do without having
your program cut?
• Will your app run on a phone? Scale down.

First, let’s talk about limiting
factors

The slowest part ?
• The fast ethernet or gigabit network port
(assuming there’s more than one drive)
• Transactions of something/second

• 1 Gbps = 125 MB/sec (100 Mbps = 12.5 MB/sec)
• The question is can your OS and our CPUs
push it?
• And let’s say you can, just how much is a Gbps
in some kind of other thing?

We used to use FreeBSD
Moral of the story? Even restricted to a single processor with a
single core, Solaris Nevada Build 31 can push 60% of a gigabit
connection. FreeBSD can’t.

OK, Solaris Good.
What’s a 100 Mbps in normal web traffic?
• Textdrive.com => 122 KB and has 20 objects
(~125KB for an uncached page view).
• 125KB page => 100-1000 unique visitors per
second
• 20 objects per page, that is 2000 requests per
second that could pump out of that system.
• Maximum.

• What is the ability to do 2000 requests/second
then?
• ((2000 requests/sec)*(20 requests per page)) *
0.125MB per page = 12.5 MB/sec (100Mbps
constant).
• 86400 seconds/day on 100Mbps => 8,640,000
uniques in a day with 172,800,00 hits.

• $8000/month
• Give or take a couple of grand depending
How much does just a 100 Mbps
commit cost?

Can you do 2000 requests/second?
• Sure an Apache, Lighttpd or Litespeed can do
1000-15,000 static or proxy requests/second

How do you do 2000 requests/
second?
• Alistapart.com bursts to that with Zeldman’s
“Web 3.0” for over an hour
• 10 Lighttpds => 200 proxy requests/second
each
• 1 Lighttpd => 40 request/sec x 5 Rails-FCGI
each
• We cached everything but writing to MySQL for
comments and that kept it on a 3.2 Ghz P4 with
4GB of RAM

The Shared Hosting
• Is this odd heterogenous largely ruby “application”
• Users received 8,641,866 emails, they sent 5,860,769 emails and had
14,984,680 pieces of spam blocked. Mail through the system averages
300 emails/minute with bursts up to 60,000 emails/minute.
• Websites cumulatively did ~400,000,000 page views.

What is it running now?
• There’s 22 TBs of strongspace
• Fiber-attached EMC storage
• Migrated to Solaris “11”
• One big ZFS pool with LUNs
• Migrated down to one “hot” server: Dual 3.0
Ghz Xeon with 8GB of RAM
• Apache, Lighttpd, static FCGI

Fsck You
{if you think ZFS isn’t production}

Is it a web app or what?
~20 Mbps constant

“Logical” servers for the hosted
connector
1) Jumpstart/Boot & Administrative servers (x2 per setup)
2) DHCP/LDAP for server identification/authentication and control (x2)
3) DNS: DNSCache for outside resolution (DNSCache as resolver) and a private DNS system
4) DNS MySQL (x2, master/slave, innodb tables)
5) SPAM filtering servers (running DSPAM -> files to NFS store and tracking to postgresql) (x4)
6) SPAM database setup (running postgresql) (x2)
7) SPAM NFS store (x2 heads clustered)
8) SMTP gateway out (x2)
9) SMTP gateway in (x4 -> delivery to Maildir over NFS)
10) Mail NFS store (x2 heads cluster) *main user file store
11) IMAP proxy servers (x2)
12) IMAP servers (x6)
13) User LDAP servers (x2 with hitting postgresql DB backend)
14) User LDAP-postgresql db (x2)
15) User postgresql DB servers (x4)
16) User Web/Application servers (running with kernel SSL accelerators) (x6)
17) User File Storage (NFS dual cluster heads) (x2)
18) Joyent Organization Provisioning server

So how do you scale that “web”
application?

That beautiful Connector
application where only 2 of the
18 “things” is “rails”.

So most scale is how you scale
those “common” services

CPUs are largely idle & RAM RAM RAM
08:23:01 PM CPU %user %nice %sys %iowait %irq %soft %idle intr/s
08:23:01 PM all 2.64 0.00 0.36 2.81 0.02 0.14 94.02 1239.49
08:23:01 PM 0 6.86 0.00 1.18 4.72 0.10 0.69 86.45 68.94
08:23:01 PM 1 2.63 0.00 0.34 1.95 0.01 0.11 95.01 192.97
08:23:01 PM 2 2.78 0.00 0.43 7.41 0.02 0.11 89.29 197.21
08:23:01 PM 3 2.16 0.00 0.29 4.12 0.01 0.07 93.39 178.35
08:23:01 PM 4 1.67 0.00 0.17 1.09 0.01 0.04 97.04 117.29
08:23:01 PM 5 1.67 0.00 0.17 1.11 0.01 0.04 97.05 161.00
08:23:01 PM 6 1.67 0.00 0.16 1.04 0.01 0.04 97.13 161.86
08:23:01 PM 7 1.68 0.00 0.16 1.08 0.01 0.04 97.07 161.86
• “Busy” mysql database server
• ~240GB of data with ~150,000 users
• Swapping just a little little bit, so ...

So what kind of CPUs have we
moved into?

MB/CMP0/P0 0 1000 MHz SUNW,UltraSPARC-T1
Sun T1000’s Ultrasparc T1
32 logical 1.0 Ghz
Combined with many software instances
Zones
Best match between actual CPU and RAM
usage

It’s not really about high frequencies
If you buy intel, buy weak ones,
spend the money on RAM
Throughput!

Back to simple
Console server
One kind of switch
One kind of server
One kind of storage
One kind of interconnect
One kind of power plug
One kind of power strip

Well. We do have some load balancers
still.

There’s actually lots of different kinds
of plugs

What the “default” 5-15R gets you
Umm ... ok so there’s a real disconnect between
server power and datacenter “design”

I/O
Storage
PublicRemote Interconnects
Jumbo Frames
Own Switches

4 “old” San Diego Racks
Per Rack:
39 processors
39 logical CPUs
52 GB RAM
5280 watts
Total:
156 logical CPUs
208 GB RAM
21,120 watts

Yes that is what they look like

Become a new node
1152 logical 1Ghz CPUs
576 GB RAM
6,480 watts
31 amps @ 208V
2 x 20amp @ 208V L6-20R
2 x 24 plug 20amp/208V
----------------------------------
12 Enclosures
36 controllers
180 drives
90 TBs raw storage
77 TBs clustered storage
29.4 amps @ 208V
2 x 20amp @ 208V L6-20R
2 x 24 plug 20amp/208V
Per Rack:
39 processors
39 logical CPUs
52 GB RAM
5280 watts
Total:
156 logical CPUs
208 GB RAM
21,120 watts

Server + Storage
RAID5
7 TBs
RAID5
7 TBs
RAID5
7 TBs
RAIDZ ZFS Pool
iSCSI

This is going to be open for you
to use

DIY: How do you start?
20amp 208V AC power
L6-20R
Street price: ~$50,000 co-lo’ed

Why DIY? Or at least a visit?
Outsourcing!You can whois yourself?

Some final things
• Keep it simple even if you’re starting out big
• Keep it simple even if you’re starting out small
• Our “stacks” are currently:
• Solaris, Mongrel, Apache 2.2 event, kssl
• Solaris, FCGI, Apache 2.2 event, kssl
• Solaris, FCGI, Lighttpd, kssl
• Solaris, FCGI, Lighttpd, Big-IP
• Zones, zones, zones

Joyent circa 2006 (Scale with Rails)

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Joyent circa 2006 (Scale with Rails) (20)

More from bcantrill (17)

Recently uploaded (20)

Joyent circa 2006 (Scale with Rails)