Burn down the silos! Helping dev and ops gel on high availability websites
0 likes1,638 views
The document discusses a DevOps case study focused on applying technologies for a fundraising website, emphasizing the need for consistency, repeatability, and visibility in deployment and configuration management. It highlights tools like Puppet and Capistrano for automating deployments and managing complex application environments while detailing the challenges and solutions around configuration drift and metrics collection. The case study illustrates the importance of collaboration between development and operations teams and incorporates real-world examples of deployment strategies and monitoring practices.
![class apache2 {
package { "apache2":
ensure => installed
}
service { "apache2":
enable => true,
ensure => running,
require => [ Package["apache2"] ]
}
}](https://image.slidesharecdn.com/burn-down-the-silos-110130055415-phpapp02/85/Burn-down-the-silos-Helping-dev-and-ops-gel-on-high-availability-websites-27-320.jpg)
![class capistrano::user {
group { "deploy":
gid => 499
}
user { "deploy":
uid => 499,
gid => 499,
home => "/home/deploy",
shell => "/bin/bash",
require => Group["deploy"]
}
file { "/home/deploy/.ssh/authorized_keys":
source => "puppet:///modules/capistrano/authorized_keys",
mode => 644,
owner => "deploy",
group => "deploy",
require => [ User["deploy"] ]
}
}](https://image.slidesharecdn.com/burn-down-the-silos-110130055415-phpapp02/85/Burn-down-the-silos-Helping-dev-and-ops-gel-on-high-availability-websites-63-320.jpg)
![define capistrano::site {
include capistrano::user
file { "/srv/$name":
ensure => directory,
owner => deploy,
group => deploy,
require => [ User["deploy"] ]
}
file { "/srv/$name/releases":
ensure => directory,
owner => deploy,
group => deploy,
require => [ File["/srv/$name"] ]
}
...](https://image.slidesharecdn.com/burn-down-the-silos-110130055415-phpapp02/85/Burn-down-the-silos-Helping-dev-and-ops-gel-on-high-availability-websites-64-320.jpg)
![sudo mmm_control show # blocks under high IO
echo -en “shownquitn” | nc 127.1 9988 # instant
socket = ::TCPSocket.new("127.0.0.1", 9988)
socket.print("shownquitn")
output = socket.read.split("n")
hosts = output.map do |line|
parts = line.scan(/nasty regex/).flatten
{ "hostname" => parts[0],
"address" => parts[1],
"mode" => parts[2],
"state" => parts[3],
"role" => parts[5],
"role_address" => parts[6] }
end](https://image.slidesharecdn.com/burn-down-the-silos-110130055415-phpapp02/85/Burn-down-the-silos-Helping-dev-and-ops-gel-on-high-availability-websites-108-320.jpg)
Burn down the silos! Helping dev and ops gel on high availability websites
- 1. Burn Down The Silos Lindsay Holmwood
- 2. DevOps
- 3. Case study
- 6. Strong siloisation Devs + Ops in di"erent companies
- 7. 100% uptime
- 9. 3 Concepts
- 10. Consistency
- 11. Repeatability
- 12. Visibility
- 14. Consistency
- 17. or across multiple environments
- 20. testing
- 22. Puppet language library client/server
- 23. package { "apache2": ensure => installed } service { "apache2": enable => true, ensure => running }
- 24. class apache2 { package { "apache2": ensure => installed } service { "apache2": enable => true, ensure => running } }
- 25. Puppet work#ow write apply debug
- 26. class apache2 { package { "apache2": ensure => installed } service { "apache2": enable => true, ensure => running } }
- 27. class apache2 { package { "apache2": ensure => installed } service { "apache2": enable => true, ensure => running, require => [ Package["apache2"] ] } }
- 31. VMware snapshots
- 35. “roles”
- 36. define app_server($collectd_destination, $logging_destination) { server { $fqdn: logging_destination => $logging_destination } include apache2 include mysql::client collectd::client { $fqdn: collection_destination => $collectd_destination } if $environment == "production" { include production::only::module } }
- 37. node "app-01.stage.charity.com" { app_server { $fqdn: collectd_destination => "stats-01.stage.charity.com", logging_destination => "log-01.stage.charity.com" } } node "app-01.production.charity.com" { app_server { $fqdn: collectd_destination => "stats-01.production.charity.com", logging_destination => "log-01.production.charity.com" } }
- 38. node /app-d+.stage.charity.com/ { app_server { $fqdn: collectd_destination => "stats-01.stage.charity.com", logging_destination => "log-01.stage.charity.com" } } node /app-d+.production.charity.com/ { app_server { $fqdn: collectd_destination => "stats-01.production.charity.com", logging_destination => "log-01.production.charity.com" } }
- 39. 30 minute builds
- 40. Consistency
- 42. Repeatability
- 44. automate, to remove human error
- 45. increase speed by shortening feedback loops
- 50. Capistrano
- 52. Simple, powerful, can blow your legs o"
- 53. Not a substitute for con!guration management
- 54. railsless-deploy http://bit.ly/i56ra9
- 56. Great for PHP
- 57. capistrano-multistage (part of capistrano-ext) http://bit.ly/i2moIp
- 58. # config/deploy.rb set :user, "deploy" set :application, "charity.com" set :keep_releases, 10 set :deploy_to, "/srv/#{application}" set :stages, %w(uat staging production)
- 59. # config/deploy/stage.rb role :app, "app-01.stage.charity.com", "app-02.stage.charity.com" role :static, "static-01.stage.charity.com”
- 60. # config/deploy/production.rb role :app, "app-01.prod.charity.com", "app-02.prod.charity.com", "app-03.prod.charity.com" role :static, "static-01.prod.charity.com”
- 61. cap staging deploy # deploy to staging # test cap production deploy # deploy to production
- 63. class capistrano::user { group { "deploy": gid => 499 } user { "deploy": uid => 499, gid => 499, home => "/home/deploy", shell => "/bin/bash", require => Group["deploy"] } file { "/home/deploy/.ssh/authorized_keys": source => "puppet:///modules/capistrano/authorized_keys", mode => 644, owner => "deploy", group => "deploy", require => [ User["deploy"] ] } }
- 64. define capistrano::site { include capistrano::user file { "/srv/$name": ensure => directory, owner => deploy, group => deploy, require => [ User["deploy"] ] } file { "/srv/$name/releases": ensure => directory, owner => deploy, group => deploy, require => [ File["/srv/$name"] ] } ...
- 65. ... file { "/srv/$name/shared/log": ensure => directory, owner => www-data, group => www-data, } file { "/etc/$name": source => "puppet:///modules/capistrano/etc/$name", recurse => true, mode => "644", owner => root, group => root } }
- 66. define app_server($collectd_destination, $logging_destination) { include apache2 include mysql::client capistrano::site { “charity.com”: } }
- 67. Deploying to a new app server is as easy as:
- 68. # config/deploy/production.rb role :app, "app-01.prod.charity.com", "app-02.prod.charity.com", "app-03.prod.charity.com" role :static, "static-01.prod.charity.com”
- 69. # config/deploy/production.rb role :app, "app-01.prod.charity.com", "app-02.prod.charity.com", "app-03.prod.charity.com", "app-04.prod.charity.com" role :static, "static-01.prod.charity.com”
- 70. or...
- 71. # deploy/production.rb role :app, "app-01.prod.charity.com", "app-02.prod.charity.com", "app-03.prod.charity.com", "app-04.prod.charity.com" role :static, "static-01.prod.charity.com”
- 72. # deploy/production.rb role :app, *(1..4).map do |n| "app-%.2d.prod.charity.com" % n end role :static, "static-01.prod.charity.com”
- 74. git-svn mirror
- 75. 182MB * 20 == PAIN
- 76. remote_cache
- 77. bad with svn tags
- 78. git-svn + cron
- 79. fast clones commit access 21st century tech
- 80. Repeatability
- 82. Visibility
- 83. one eye on the past
- 84. one eye on the future
- 87. code changes
- 88. monitoring
- 89. reports
- 92. collectd
- 93. lightweight statistics collection daemon
- 94. platform for collecting time series data
- 95. plugin based
- 96. network aware
- 97. well de!ned APIs
- 98. curl_json
- 99. <Plugin curl_json> <URL "http://localhost:5984/_stats"> Instance "httpd" <Key "httpd/requests/count"> Type "http_requests" </Key> <Key "httpd_status_codes/*/count"> Type "http_response_codes" </Key> </URL> </Plugin>
- 100. /metrics
- 102. code changes
- 103. application & con!g management
- 105. Your new best friend
- 106. monitoring
- 107. sudo mmm_control show # blocks under high IO echo -en “shownquitn” | nc 127.1 9988 # instant
- 108. sudo mmm_control show # blocks under high IO echo -en “shownquitn” | nc 127.1 9988 # instant socket = ::TCPSocket.new("127.0.0.1", 9988) socket.print("shownquitn") output = socket.read.split("n") hosts = output.map do |line| parts = line.scan(/nasty regex/).flatten { "hostname" => parts[0], "address" => parts[1], "mode" => parts[2], "state" => parts[3], "role" => parts[5], "role_address" => parts[6] } end
- 110. reports
- 111. mk-query-digest & logrotate
- 112. # prerotate SLOWLOG_FILENAME="/var/log/mysql/mysql-slow.log" OPTIONS="--report --group-by distill --order-by Query_time:max --timeline --report-format query_report,profile" DATE="$(date +%Y-%m-%dT%H:%M:%S%z)" REPORT_FILENAME="/tmp/$(hostname)-mysql-slow-query-report-$DATE" mk-query-digest $SLOWLOG_FILENAME $OPTIONS > $REPORT_FILENAME SUBJECT="MySQL Slow Queries Report for $(hostname) as of $DATE" RECIPIENTS="developers@charity.com,ops@bulletproof.net" cat $REPORT_FILENAME | nail -n -E -s "$SUBJECT" "$RECIPIENTS"
- 114. Retrospectives
- 115. Slave explosion
- 116. Background:
- 119. Background: MySQL replication MMM 2 masters + 4 slaves
- 120. REPLICATION_FAIL on one slave
- 121. Down to 3 nodes
- 123. REPLICATION_DELAY on another slave
- 124. Down to 2 nodes
- 126. Swapping like mad Half the memory allocated
- 127. Shutdown, upgrade, boot
- 128. Visibility
- 129. metric collection
- 130. Consistency
- 133. Soft launch
- 135. Con!g parses + loads
- 136. Add con!g dump url
- 137. Visibility
- 138. curl_json
- 139. Redeploy
- 140. ...
- 141. Typo
- 142. 2 reviewers of con!g management
- 143. both in ops team
- 144. Visibility
- 145. code changes
- 146. Your new best friend
- 147. reviewer diversity devs should have visibility of ops changes
- 149. Data consistency
- 150. New release
- 151. Database migrations
- 152. Release promotion
- 153. uat stage production
- 154. uat ! stage production
- 155. uat ! stage ! production
- 156. uat ! stage ! production "
- 157. uat ! stage ! production "
- 158. CREATE TABLE foo should have been CREATE TABLE IF NOT EXIST foo
- 159. Consistency
- 160. uat stage production
- 161. uat stage production
- 162. uat stage production
- 163. Repeatability
- 165. Consistency
- 168. or across multiple environments
- 169. Repeatability
- 171. automate, to remove human error
- 172. increase speed by shortening feedback loops
- 173. Visibility
- 174. one eye on the past
- 175. one eye on the future
- 176. Communicate!
- 177. Thank you! Credits: http://www.#ickr.com/photos/48722974@N07/4682302824/ http://www.#ickr.com/photos/lyza/4144764381/ http://www.#ickr.com/photos/acediscovery/3030548744/ http://www.#ickr.com/photos/matchew/424026531/ http://www.#ickr.com/photos/andrew_wertheimer/5268407700/ http://www.#ickr.com/photos/mrwoodnz/4289893182/ http://www.#ickr.com/photos/azrasta/4528604334/ http://www.#ickr.com/photos/myprofe/4396178084/ http://www.#ickr.com/photos/boliston/2351083198/ http://www.#ickr.com/photos/nnova/4834954885/ http://www.#ickr.com/photos/brianwestcott/1497708345/ http://www.#ickr.com/photos/pjern/2150874047/ http://www.#ickr.com/photos/brunogirin/73014722/ http://www.#ickr.com/photos/rubodewig/5161937181/ http://www.#ickr.com/photos/eole/4500783172/ http://www.#ickr.com/photos/rutty/460520720/ http://www.#ickr.com/photos/jacockshaw/1811056252/ http://www.#ickr.com/photos/sarah_lincoln/4740037328/ http://www.#ickr.com/photos/jenny-pics/2719309611/ http://www.#ickr.com/photos/shindotv/3835365695/ http://www.#ickr.com/photos/ldsykora/2414497811/ http://www.#ickr.com/photos/thalamus/306881919/ http://www.#ickr.com/photos/listed_crime/1342164481/ http://www.#ickr.com/photos/traviscrawford/323366600/ http://www.#ickr.com/photos/localsurfer/369116556/ http://www.#ickr.com/photos/webtreatsetc/5303216304/