![Also great for Munin
#!/bin/bash
str="includedir /etc/munin/munin-conf.d"
for addr in `/usr/bin/mco facts ipaddress | awk '{gsub("found", "");
print $1}' | grep "^[0-9]"`
do
fqdn=`/usr/bin/mco facts fqdn -F ipaddress=$addr | grep "^W" |
awk '{print $1}'`
str="$str
[$fqdn]
address $addr
use_node_name yes"
done
echo "$str" > /etc/munin/munin.conf
/usr/sbin/service munin-node restart
60](https://image.slidesharecdn.com/mcollective-130116194021-phpapp01/85/Rails-infrastructure-85-320.jpg)
Rails infrastructure
- 2. Topics Covered 2
- 3. Topics Covered • Lots of facepalm 2
- 4. Topics Covered • Lots of facepalm • Rackspace 2
- 5. Topics Covered • Lots of facepalm • Rackspace • Linux distribution choices 2
- 6. Topics Covered • Lots of facepalm • Rackspace • Linux distribution choices • Automation and Orchestration 2
- 7. Topics Covered • Lots of facepalm • Rackspace • Linux distribution choices • Automation and Orchestration • Logging 2
- 8. Edison Nation 3
- 9. Edison Nation • Distributed team (US/Canada/UK) 3
- 10. Edison Nation • Distributed team (US/Canada/UK) • Old (2009) and poorly maintained application 3
- 11. Edison Nation • Distributed team (US/Canada/UK) • Old (2009) and poorly maintained application • Rails 2.3 app 3
- 12. Edison Nation • Distributed team (US/Canada/UK) • Old (2009) and poorly maintained application • Rails 2.3 app • (previous) focus on churn 3
- 13. Edison Nation • Distributed team (US/Canada/UK) • Old (2009) and poorly maintained application • Rails 2.3 app • (previous) focus on churn • 3 Rails developers (+ 1 designer and an intern) 3
- 14. Edison Nation • Distributed team (US/Canada/UK) • Old (2009) and poorly maintained application • Rails 2.3 app • (previous) focus on churn • 3 Rails developers (+ 1 designer and an intern) • >100,000 members 3
- 15. Edison Nation • Distributed team (US/Canada/UK) • Old (2009) and poorly maintained application • Rails 2.3 app • (previous) focus on churn • 3 Rails developers (+ 1 designer and an intern) • >100,000 members • Little inhouse sysadmin experience 3
- 16. Additional Quirks 4
- 17. Additional Quirks • Used 1.8.7 since God does not play nicely with Ruby Enterprise Edition and we couldn’t use 1.9 because of Rails 2.3 4
- 18. Additional Quirks • Used 1.8.7 since God does not play nicely with Ruby Enterprise Edition and we couldn’t use 1.9 because of Rails 2.3 • Provisioning process was terribly slow 4
- 19. Additional Quirks • Used 1.8.7 since God does not play nicely with Ruby Enterprise Edition and we couldn’t use 1.9 because of Rails 2.3 • Provisioning process was terribly slow • Very little caching 4
- 20. Additional Quirks • Used 1.8.7 since God does not play nicely with Ruby Enterprise Edition and we couldn’t use 1.9 because of Rails 2.3 • Provisioning process was terribly slow • Very little caching • Quite a lot of server generated JS 4
- 21. SURPRISE! 5
- 22. Featured on Nightline 6
- 23. Featured on Nightline • No warning (announced pretty late EST) 6
- 24. Featured on Nightline • No warning (announced pretty late EST) • No preparation time (engineers already signed off for the night) 6
- 25. Featured on Nightline • No warning (announced pretty late EST) • No preparation time (engineers already signed off for the night) • Couldn’t provision servers to deal with the traffic spike in time (and we would have needed a lot of them) 6
- 26. 7
- 27. Load balancer recorded 3000 concurrent requests including assets or around 300 excluding assets 8
- 28. The Stack 9
- 29. Figuring out the bottlenecks 10
- 30. Nginx kept serving - though these were 502 errors 11
- 31. Post-mortem of the requests that did make it through made it look like the application servers were to blame 12
- 32. Database was under heavy load but by no means the bottleneck 13
- 33. Make better use of the application server pool 14
- 34. Got some quick wins in the code by caching more and moving jQuery to Google 15
- 35. <script src="// ajax.googleapis.com/ ajax/libs/jquery/1.6.2/ jquery.min.js"></script> 16
- 36. Get rid of any server generated JS 17
- 37. Pretty much re-trained myself to be a systems administrator 18
- 38. Completely re-think the way we do Operations 19
- 39. What components make up a solid multi-server setup? 20
- 40. Load balancing 21
- 41. TLS SNI Extension 22
- 42. Theoretically only have two load balancers for ALL domains 23
- 43. Simplified SSL Nginx config server { listen 443; server_name www.edisonnation.com; ssl on; ssl_certificate /path/to/cert/en.com.cert; ssl_certificate_key /path/to/cert/en.com.key; } server { listen 443; server_name www.edisonnation.vn; ssl on; ssl_certificate /path/to/cert/en.vn.cert; ssl_certificate_key /path/to/cert/en.vn.key; } 24
- 44. Windows XP + Internet Explorer 25
- 45. Windows XP • Internet Explorer 6-8 on Windows XP would not work compared to modern OS + Browser combinations • Ignores the server name for HTTPS • Will give you an invalid SSL certificate error when browsing 26
- 46. Rackspace (v2) Load Balancer 27
- 47. Rackspace Load Balancer • SSL termination at the Load Balancer • No need to serve HTTPS traffic from Nginx any more - X-Forwarded-Proto tells Rails if page is supposed to be encrypted • Less processing required here • Less complexity managing certificates and Nginx configs 28
- 48. Split up the application servers 29
- 49. Move Nginx to it’s own machine and reverse proxy back to Unicorn app servers 30
- 50. New stack 31
- 51. Switch Unicorn to use TCP sockets rather than Unix 32
- 52. Linux 33
- 53. Debian Squeeze 34
- 54. Why Debian? 35
- 55. Why Debian? • Pick the most stable distribution 35
- 56. Why Debian? • Pick the most stable distribution • Debian is pretty stable, plus you can use Lucid Lynx packages for anything that you need which is cutting edge 35
- 57. Why Debian? • Pick the most stable distribution • Debian is pretty stable, plus you can use Lucid Lynx packages for anything that you need which is cutting edge • However, God requires you to use a custom kernel before it will work properly http://bugs.debian.org/cgi-bin/ bugreport.cgi?bug=609004 35
- 58. Ubuntu LTS also viable as a choice as is any RHEL 36
- 59. Basically, anything where the packages aren’t crazy and support is still there (not Arch/Fedora/ Ubuntu) 37
- 60. Packaging 38
- 61. We don’t image servers (but may start doing so) 39
- 62. Provisioning tools should be able to build a server on any hardware 40
- 63. Never build from source 41
- 64. Never build from source • Either package yourself or get from a reliable source 41
- 65. Never build from source • Either package yourself or get from a reliable source • Ditch RVM (though they now have binary rubies - anyone tried?) 41
- 66. Never build from source • Either package yourself or get from a reliable source • Ditch RVM (though they now have binary rubies - anyone tried?) • Check out Brightbox Next Generation Ubuntu packages http://wiki.brightbox.co.uk/docs:ruby-ng 41
- 67. Pin everything else Package: * Pin: release a=squeeze-backports Pin-Priority: 200 Package: puppet Pin: release a=squeeze-backports Pin-Priority: 900 Package: puppet-common Pin: release a=squeeze-backports Pin-Priority: 900 42
- 68. Server build time decreased from 45 minutes to < 15 minutes 43
- 69. How do we provision servers? 44
- 70. A small bash script + Puppet 45
- 71. Bash script does basic pinning and installs essential packages (Ruby + Emacs + Puppet + puppet-el) 46
- 72. Works very well since we use Hetzner EX4S’s for non-critical systems 47
- 73. Hetzner + (Xen/OpenVZ) == FANTASTIC 48
- 74. (See me at the end if you want to talk about provisioning some more) 49
- 75. Managing Puppet 50
- 76. Always running Puppet rather than run on demand 51
- 77. Encourage developers to document infrastructure changes 52
- 78. Still unsure about how to go about Puppet testing 53
- 79. Campfire reporting 54
- 80. Orchestration 55
- 81. MCollective 56
- 82. STOMP server connects all of our servers together 57
- 83. MCollective executes Remote Procedure Calls 58
- 84. Great for pushing out urgent Puppet updates 59
- 85. Also great for Munin #!/bin/bash str="includedir /etc/munin/munin-conf.d" for addr in `/usr/bin/mco facts ipaddress | awk '{gsub("found", ""); print $1}' | grep "^[0-9]"` do fqdn=`/usr/bin/mco facts fqdn -F ipaddress=$addr | grep "^W" | awk '{print $1}'` str="$str [$fqdn] address $addr use_node_name yes" done echo "$str" > /etc/munin/munin.conf /usr/sbin/service munin-node restart 60
- 86. No longer have to manually maintain Munin 61
- 87. Can be used for other painful tasks - such as making sure packages are up to date on all the servers 62
- 88. RPC libraries are written in Ruby 63
- 89. Service management 64
- 90. M/Monit 65
- 91. Not free - however, extremely worthwhile. Can hook into shell scripts 66
- 92. Log management 67
- 93. Graylog2 68
- 94. Java JAR with a Rails frontend and Elasticsearch + Mongo backend 69
- 95. Deals with exception management 70
- 96. Can do analytics on logs 71
- 97. Specify streams of logs (i.e 404 errors) 72
- 98. No longer have to juggle lots of files which exist on different machines 73
- 99. A little tricky to set-up 74
- 100. Use the gelf-rb gem sparingly in your Rails app and NOT as your main logger 75
- 101. Found out, that the log requests were not threaded 76
- 102. For us, gelf-rb ONLY sends exception notifications 77
- 103. Introducing Logstashd 78
- 104. Written by the awesome Jordan Sissel (FPM) 79
- 105. Nginx doesn’t support sending to Graylog straight out 80
- 106. Logstashd acts as a log tailing and transporting mechanism 81
- 107. Runs in its own process - so threading doesnt matter so much 82
- 108. Whats left? 83
- 109. Upgrade to Rails 3 84
- 110. Great benefits with Rails 3 such as Dalli for memcached failovers and Lograge 85
- 111. Oh yeah - assets pipeline! 86
- 112. Implement read slaves for backups 87
- 113. Make Jenkins do our deployment 88
- 114. Better caching solutions - maybe Varnish / conditional GET 89
- 115. Re-implement TLS SNI once Windows XP security updates stop 90
- 116. Handle large spikes better 91
- 117. Autoscaling? 92
- 118. Using AWS as an additional cloud failover 93
- 119. Hybrid Dedicated and Cloud for production 94