"Building a Resilient Cloud Infrastructure. From Scratch." - Cloud East, 28 July 2012
0 likes570 views
The document outlines the process by which Brightbox built its own cloud infrastructure from scratch. It describes how Brightbox previously hosted Ruby applications but faced challenges with unwieldy systems. This led them to realize an opportunity to build their own cloud infrastructure that would better suit their needs and those of their customers. The design process involved requirements gathering, consulting experts initially, extensive testing and experimentation. Elements designed included the network architecture, application architecture and hardware selection. The infrastructure was then built out through steps like network provisioning, hardware installation, software development and documentation. The cloud was launched in private beta in 2010 and continued to iterate on improvements.
"Building a Resilient Cloud Infrastructure. From Scratch." - Cloud East, 28 July 2012
- 1. Building a Resilient Cloud Infrastructure. From Scratch. by Jeremy Jarvis Co-founder
- 4. Overview • Preface: Who, what etc. @jeremyjarvis
- 5. Overview • Preface: Who, what etc. • Chapter 1: Feeling the pain @jeremyjarvis
- 6. Overview • Preface: Who, what etc. • Chapter 1: Feeling the pain • Chapter 2: Realising an opportunity @jeremyjarvis
- 7. Overview • Preface: Who, what etc. • Chapter 1: Feeling the pain • Chapter 2: Realising an opportunity • Chapter 3: Design phase @jeremyjarvis
- 8. Overview • Preface: Who, what etc. • Chapter 1: Feeling the pain • Chapter 2: Realising an opportunity • Chapter 3: Design phase • Chapter 4: Let’s build this thing! @jeremyjarvis
- 9. Overview • Preface: Who, what etc. • Chapter 1: Feeling the pain • Chapter 2: Realising an opportunity • Chapter 3: Design phase • Chapter 4: Let’s build this thing! • Chapter 5: It’s alive! @jeremyjarvis
- 10. Overview • Preface: Who, what etc. • Chapter 1: Feeling the pain • Chapter 2: Realising an opportunity • Chapter 3: Design phase • Chapter 4: Let’s build this thing! • Chapter 5: It’s alive! • Epilogue: Some conclusions/lessons @jeremyjarvis
- 12. Preface. • Agile cloud infrastructure service @jeremyjarvis
- 13. Preface. • Agile cloud infrastructure service • “Multi-zone” (datacentre) architecture @jeremyjarvis
- 14. Preface. • Agile cloud infrastructure service • “Multi-zone” (datacentre) architecture • UK-based (HQ in Leeds, DCs in Manchester) @jeremyjarvis
- 15. Preface. • Agile cloud infrastructure service • “Multi-zone” (datacentre) architecture • UK-based (HQ in Leeds, DCs in Manchester) • Small team (still only 10 in total) @jeremyjarvis
- 16. Preface. • Agile cloud infrastructure service • “Multi-zone” (datacentre) architecture • UK-based (HQ in Leeds, DCs in Manchester) • Small team (still only 10 in total) • Distributed team (Mainly around Leeds) @jeremyjarvis
- 17. Preface. • Agile cloud infrastructure service • “Multi-zone” (datacentre) architecture • UK-based (HQ in Leeds, DCs in Manchester) • Small team (still only 10 in total) • Distributed team (Mainly around Leeds) • Developer/DevOps focused (it’s who we are) @jeremyjarvis
- 18. Preface. • Agile cloud infrastructure service • “Multi-zone” (datacentre) architecture • UK-based (HQ in Leeds, DCs in Manchester) • Small team (still only 10 in total) • Distributed team (Mainly around Leeds) • Developer/DevOps focused (it’s who we are) • Profitable (for 4 yrs, built from revenue) @jeremyjarvis
- 19. Chapter 1: Feeling the Pain @jeremyjarvis
- 20. Chapter 1: Feeling the Pain @jeremyjarvis
- 21. Chapter 1: Feeling the Pain • Launched Ruby-specific hosting service (Sep 2007) @jeremyjarvis
- 22. Chapter 1: Feeling the Pain • Launched Ruby-specific hosting service (Sep 2007) • Acquisition interest (no thanks!) @jeremyjarvis
- 23. Chapter 1: Feeling the Pain • Launched Ruby-specific hosting service (Sep 2007) • Acquisition interest (no thanks!) • Built a good reputation, hosted apps for some large customers @jeremyjarvis
- 24. Chapter 1: Feeling the Pain • Launched Ruby-specific hosting service (Sep 2007) • Acquisition interest (no thanks!) • Built a good reputation, hosted apps for some large customers • Systems stable, but became unwieldy and hard for us to manage + we wanted to expand offering @jeremyjarvis
- 25. Chapter 1: Feeling the Pain • Launched Ruby-specific hosting service (Sep 2007) • Acquisition interest (no thanks!) • Built a good reputation, hosted apps for some large customers • Systems stable, but became unwieldy and hard for us to manage + we wanted to expand offering • Began looking at existing options, nothing fitted our needs (e.g Eucalyptus) @jeremyjarvis
- 26. Chapter 1: Feeling the Pain • Launched Ruby-specific hosting service (Sep 2007) • Acquisition interest (no thanks!) • Built a good reputation, hosted apps for some large customers • Systems stable, but became unwieldy and hard for us to manage + we wanted to expand offering • Began looking at existing options, nothing fitted our needs (e.g Eucalyptus) • Decided to build our own “cloud” (Brightbox NG) @jeremyjarvis
- 27. Chapter 2: Realising an opportunity @jeremyjarvis
- 28. Chapter 2: Realising an opportunity @jeremyjarvis
- 29. Chapter 2: Realising an opportunity • Limited options in EU/UK @jeremyjarvis
- 30. Chapter 2: Realising an opportunity • Limited options in EU/UK • If we’re building this for ourselves why not sell it? @jeremyjarvis
- 31. Chapter 2: Realising an opportunity • Limited options in EU/UK • If we’re building this for ourselves why not sell it? • Generated lots of internal debate @jeremyjarvis
- 32. Chapter 2: Realising an opportunity • Limited options in EU/UK • If we’re building this for ourselves why not sell it? • Generated lots of internal debate • Lack of flexibility with PaaS services @jeremyjarvis
- 33. Chapter 2: Realising an opportunity • Limited options in EU/UK • If we’re building this for ourselves why not sell it? • Generated lots of internal debate • Lack of flexibility with PaaS services • We’re good at this stuff + we have awesome team @jeremyjarvis
- 34. Chapter 2: Realising an opportunity • Limited options in EU/UK • If we’re building this for ourselves why not sell it? • Generated lots of internal debate • Lack of flexibility with PaaS services • We’re good at this stuff + we have awesome team • Decided to shift development focus @jeremyjarvis
- 35. Chapter 3: Design phase @jeremyjarvis
- 36. Chapter 3: Design phase @jeremyjarvis
- 37. Chapter 3: Design phase • Requirements: @jeremyjarvis
- 38. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) @jeremyjarvis
- 39. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) @jeremyjarvis
- 40. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) – Agile network layer (Load Balancing, Cloud IP, Firewall) @jeremyjarvis
- 41. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) – Agile network layer (Load Balancing, Cloud IP, Firewall) – Distributed (no SPOF) @jeremyjarvis
- 42. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) – Agile network layer (Load Balancing, Cloud IP, Firewall) – Distributed (no SPOF) – Modular (easy to grow, JIT) @jeremyjarvis
- 43. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) – Agile network layer (Load Balancing, Cloud IP, Firewall) – Distributed (no SPOF) – Modular (easy to grow, JIT) – Future-proof (IPv6) @jeremyjarvis
- 44. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) – Agile network layer (Load Balancing, Cloud IP, Firewall) – Distributed (no SPOF) – Modular (easy to grow, JIT) – Future-proof (IPv6) – Programmable (API, CLI) @jeremyjarvis
- 45. Chapter 3: Design phase • Requirements: – Geographic redundancy (separate datacentres) – Resilient network (multiple connections) – Agile network layer (Load Balancing, Cloud IP, Firewall) – Distributed (no SPOF) – Modular (easy to grow, JIT) – Future-proof (IPv6) – Programmable (API, CLI) – Open (easy to get stuff in and out) @jeremyjarvis
- 46. Chapter 3: Design phase @jeremyjarvis
- 47. Chapter 3: Design phase • Process: @jeremyjarvis
- 48. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. @jeremyjarvis
- 49. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. – We’re on our own! @jeremyjarvis
- 50. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. – We’re on our own! – Hands-on, R&D style, lots of testing, experimentation, iteration @jeremyjarvis
- 51. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. – We’re on our own! – Hands-on, R&D style, lots of testing, experimentation, iteration – Investigated competition (what’s good/bad) @jeremyjarvis
- 52. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. – We’re on our own! – Hands-on, R&D style, lots of testing, experimentation, iteration – Investigated competition (what’s good/bad) – Access to plenty of kit -set up mini-clouds to hack @jeremyjarvis
- 53. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. – We’re on our own! – Hands-on, R&D style, lots of testing, experimentation, iteration – Investigated competition (what’s good/bad) – Access to plenty of kit -set up mini-clouds to hack – John and Neil worked very closely on architecture (daily calls) @jeremyjarvis
- 54. Chapter 3: Design phase • Process: – Consulted, at first. Network infrastructure gurus. – We’re on our own! – Hands-on, R&D style, lots of testing, experimentation, iteration – Investigated competition (what’s good/bad) – Access to plenty of kit -set up mini-clouds to hack – John and Neil worked very closely on architecture (daily calls) – Don’t reinvent the wheel (Rubyists!) @jeremyjarvis
- 55. Chapter 3: Design phase @jeremyjarvis
- 56. Chapter 3: Design phase • Elements: @jeremyjarvis
- 57. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) @jeremyjarvis
- 58. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) @jeremyjarvis
- 59. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) – Hardware selection: @jeremyjarvis
- 60. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) – Hardware selection: • Border routers (Cisco) @jeremyjarvis
- 61. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) – Hardware selection: • Border routers (Cisco) • Switches (Cisco +) @jeremyjarvis
- 62. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) – Hardware selection: • Border routers (Cisco) • Switches (Cisco +) • Host servers (Dell) @jeremyjarvis
- 63. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) – Hardware selection: • Border routers (Cisco) • Switches (Cisco +) • Host servers (Dell) • Standardised rack design (modular) @jeremyjarvis
- 64. Chapter 3: Design phase • Elements: – Network architecture (proof-of-concept) – Application/software architecture (evolving, iterative) – Hardware selection: • Border routers (Cisco) • Switches (Cisco +) • Host servers (Dell) • Standardised rack design (modular) – Datacentre selection (Proximity, Independent, Competing) @jeremyjarvis
- 65. Chapter 4: Let’s build this thing! @jeremyjarvis
- 66. Chapter 4: Let’s build this thing! @jeremyjarvis
- 67. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) @jeremyjarvis
- 68. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) • Buying (negotiating) @jeremyjarvis
- 69. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) • Buying (negotiating) • Installing kit (Routers, Switches, Servers) @jeremyjarvis
- 70. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) • Buying (negotiating) • Installing kit (Routers, Switches, Servers) • Use datacentre staff where possible (racking/stacking) @jeremyjarvis
- 71. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) • Buying (negotiating) • Installing kit (Routers, Switches, Servers) • Use datacentre staff where possible (racking/stacking) • Software development (several applications, iterative) @jeremyjarvis
- 72. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) • Buying (negotiating) • Installing kit (Routers, Switches, Servers) • Use datacentre staff where possible (racking/stacking) • Software development (several applications, iterative) • Writing configs (again, iterative - infra as code) @jeremyjarvis
- 73. Chapter 4: Let’s build this thing! • Network provisioning (Transit, Metro links, RIPE membership) • Buying (negotiating) • Installing kit (Routers, Switches, Servers) • Use datacentre staff where possible (racking/stacking) • Software development (several applications, iterative) • Writing configs (again, iterative - infra as code) • Documentation (wiki, changelogs, code comments) @jeremyjarvis
- 74. Chapter 5: It’s alive! @jeremyjarvis
- 75. Chapter 5: It’s alive! @jeremyjarvis
- 76. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) @jeremyjarvis
- 77. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) @jeremyjarvis
- 78. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) • New features (Load balancing, Firewall etc) @jeremyjarvis
- 79. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) • New features (Load balancing, Firewall etc) • Billing data (distributed stats collection) @jeremyjarvis
- 80. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) • New features (Load balancing, Firewall etc) • Billing data (distributed stats collection) • “Enterprise” billing software *sad face* @jeremyjarvis
- 81. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) • New features (Load balancing, Firewall etc) • Billing data (distributed stats collection) • “Enterprise” billing software *sad face* • Oct 2011 - General availability (no change really, just £ £) @jeremyjarvis
- 82. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) • New features (Load balancing, Firewall etc) • Billing data (distributed stats collection) • “Enterprise” billing software *sad face* • Oct 2011 - General availability (no change really, just £ £) • Product development (more and better features) @jeremyjarvis
- 83. Chapter 5: It’s alive! • Nov 2010 - Launched private beta (700 users) • Iteration (fix bottlenecks, improve resilience) • New features (Load balancing, Firewall etc) • Billing data (distributed stats collection) • “Enterprise” billing software *sad face* • Oct 2011 - General availability (no change really, just £ £) • Product development (more and better features) • Marketing (getting the word out, communication) @jeremyjarvis
- 84. Epilogue: What did we learn? @jeremyjarvis
- 85. Epilogue: What did we learn? @jeremyjarvis
- 86. Epilogue: What did we learn? • Building stuff is hard (but can lead to competitive advantage) @jeremyjarvis
- 87. Epilogue: What did we learn? • Building stuff is hard (but can lead to competitive advantage) • Be your own customers (understand market, use products) @jeremyjarvis
- 88. Epilogue: What did we learn? • Building stuff is hard (but can lead to competitive advantage) • Be your own customers (understand market, use products) • Don’t *over*estimate competition (look behind the mask) @jeremyjarvis
- 89. Epilogue: What did we learn? • Building stuff is hard (but can lead to competitive advantage) • Be your own customers (understand market, use products) • Don’t *over*estimate competition (look behind the mask) • Learn good negotiation (clue: it’s not a battle) @jeremyjarvis
- 90. Epilogue: What did we learn? • Building stuff is hard (but can lead to competitive advantage) • Be your own customers (understand market, use products) • Don’t *over*estimate competition (look behind the mask) • Learn good negotiation (clue: it’s not a battle) • All about the launch (could have timed/co-ordinated things better for more “oomph”) @jeremyjarvis
- 91. Epilogue: What did we learn? • Building stuff is hard (but can lead to competitive advantage) • Be your own customers (understand market, use products) • Don’t *over*estimate competition (look behind the mask) • Learn good negotiation (clue: it’s not a battle) • All about the launch (could have timed/co-ordinated things better for more “oomph”) • Momentum is important (PR, morale) @jeremyjarvis