Virtualization at Gilt - Rangarajan Radhakrishnan
1 like12,536 views
Gilt uses virtualization including LXC containers and CloudStack to improve efficiency and reliability. They aim to automate provisioning, use immutable containers, and avoid repeated config enforcement. Currently they use LXC containers provisioned through CloudStack and their in-house Galactica system, which integrates with services like Puppet, LDAP, DNS and load balancers. They are testing provisioning LXC containers on different Linux distros and configurations to optimize performance.
Virtualization at Gilt - Rangarajan Radhakrishnan
- 1. O’REILLY Virtualization at Gilt Rangarajan Radhakrishnan, ranga@gilt.com Infrastructure Engineering Jun 6, 2013
- 2. O’REILLY What is Gilt Groupe? > Founded in 2007, Gilt provides a curated, personalized experience featuring products by today’ s top designer labels, at up to 60% off retail > Products across four stores: Men, Women, Home, and Baby & Kids
- 3. O’REILLY Gilt Tech: > The engine behind the Gilt experience Infrastructure Eng: > Tasked with improving efficiency and reliability
- 4. O’REILLY Tech at Gilt Offices: NYC, Dublin and Portland Infra Eng: NYC and Dublin Architecture: NYC, Portland and Dublin Engineering: NYC, Portland and Dublin
- 5. O’REILLY Gilt Tech History > 2007: Small Debian footprint with Ruby on Rails > 2008: Migrated to OpenSolaris (from RoR to Java services) > 2010: Migrated to CentOS (from Java services to Scala services) => =>
- 6. O’REILLY Current Technology at Gilt
- 7. O’REILLY Technology > Databases: PostgreSQL, MongoDB > Search: Solr, Splunk > Frameworks: Play > Languages: Scala, Java, Ruby, Node.js > Config mgmt: Puppet > Continuous integration: Jenkins > Messaging: RabbitMQ
- 8. O’REILLY Virtualization Goals > Build on what is available > Avoid a one-size-fits-all approach > Offer seamless integration with existing application stack > Use lightweight containers that offer great performance
- 9. O’REILLY Additional Virtualization Goals > Automate provisioning as much as possible > Feature immutable containers -- avoid uncertainties as a result of changes over time > Avoid repeated enforcement of config management policies
- 10. O’REILLY Virtualization History at Gilt: The Past > Bare metal: Debian (2007 - 2008) > OpenSolaris zones (2008 - 2010) > Migration to CentOS (201
- 11. O’REILLY Virtualization History at Gilt: The Present > Xen VMs (since 2010) > LXC (since 2012): strong performance, low overhead > EC2 (off and on since 2008), with varying degree of prod/non-prod use (mostly spillover traffic) > Apache CloudStack (since 2013): to increase utilization and maximize efficiency
- 12. O’REILLY Why CloudStack for Virtualization? > open source > support for multiple hypervisors (KVM, Xenserver and, recently, LXC) > great web UI and Rest API > increasingly active community > committer in-house
- 13. O’REILLY Virtualization at Gilt with Cloudstack and Galactica > CloudStack primarily for use with LXC containers > CloudStack integration with Galactica (in-house) > Galactica implements a meta-data model > Galactica integrates with Puppet/LDAP etc.
- 14. O’REILLY Galactica ... > is a family of services > allocates containers (via CloudStack) > allows for provisioning of other services > implements a meta-data model for: >> HW/SW provisioning/management >> User access provisioning/management >> IP address/DNS provisioning/management
- 15. O’REILLY > maintains desired state of services > provides DNS updates for services, especially when provisioning or decommissioning > updates load balancer config > integrates with monitoring system/audit > integrates with ioncannon (automated and continuous deployment) Galactica also:
- 16. O’REILLY Why LXC? > lightweight: minimal performance overhead > tested in staging with good results > proof of concept with live traffic showed good performance > significant work to support LXC in cloudstack "LXC will be implemented as a hypervisor in CloudStack and will be a first class citizen similar to other hypervisors ..." - Phong Nguyen, Gilt (CloudStack committer)
- 17. O’REILLY Other Key Pieces > config management system: Puppet 3.1 (for first-time run on container, then turned off) > Cobbler 2.2.3 for kickstarting bare metal servers (with both Ubuntu and CentOS profiles) > Jenkins: used for continuous integration and testing
- 18. O’REILLY Recent and Ongoing Work > Testing provisioning LXC containers with different libvirt versions >> Up to 1.0.5 on CentOS 6.3 and 0.9.8 on Ubuntu > Testing entire application stack on CentOS 6.4 LXC container on Ubuntu host >> Trying to avoid libvirt issues that came up with CentOS host
- 19. O’REILLY Recent and Ongoing Work (cont.) > Testing entire application stack on Ubuntu 12.04 LTS LXC container on Ubuntu host >> Challenge: requires different packaging mechanism > Provisioning LXC with and without libvirt to identify issues with libvirt
- 20. O’REILLY For More Info: > Gilt Tech blog: http://tech.gilt.com > Gilt Twitter: @gilttech > Gilt Facebook: facebook.com/gilttech > Work for us: gilt.com/careers/tech Thanks to: > CloudStack community > Gilt Tech team > NYCDevOps