DOES14 - David Ashman - Blackboard Learn - Keep Your Head in the Clouds
Download as PPTX, PDF2 likes1,356 views
Blackboard, a provider of educational technology software, transitioned from an on-premise model to a cloud-based model through DevOps practices. They faced issues like long development cycles, poor communication between teams, and high failure rates during testing. By adopting automation, using cloud infrastructure, and shifting their culture, Blackboard was able to reduce lead times, improve feedback loops, and have developers take ownership of production issues. This included fully automating their testing pipeline, standardizing deployments across environments, and gaining executive support for cultural changes.
DOES14 - David Ashman - Blackboard Learn - Keep Your Head in the Clouds
- 1. Keep Your Head in the Clouds Blackboard’s Transition from Enterprise to Cloud Software Through DevOps David Ashman Chief Architect, Cloud Architecture @davidbashman
- 7. ● The industry leader in educational technology and services ● 17 years young ● Privately held ● $450M revenue at time of going private ● Headquarters in Washington, DC
- 9. Blackboard Learn ● 17 year old codebase with roots in Perl ● Millions of lines of Java code (Hybrid Java/Perl for many years) ● 7 development/operations offices, worldwide ● 700 people between development, testing and operations (All products) ● ~3000 virtual machines across 1000 clients in hosting ● ~8PB of content and data storage
- 10. We are a horse.
- 11. And like so many of you... ● 6+ month lead times ● Technical debt ● High update failure rates ● Poor communication paths from development to operations ● Poor feedback loops from operations to development
- 12. But through the art of DevOps...
- 13. We are a horse.
- 14. We are a horse. ^
- 15. What changed? ● Automation ● Cloud Infrastructure ● Culture
- 16. Automation
- 17. # of code commits
- 20. 24-36 Hours
- 21. Mike McGarr
- 23. # of code commits
- 24. # of code commits
- 25. # of code commits
- 29. But wait... There’s more! Functional Acceptance Testing
- 30. Value 5 minutes Create Ticket 4 hours 15 minutes 1 hour 1 hour Commit/ Build/ Integrate 5 minutes Problem! Code Test Suite Analyze Failed Tests 5 minutes 36 hours 24 hours 6 hours (value) 70 hours total time = 9% Efficient Waste (wait) 15 minutes Assign Ticket 6 hours
- 31. Development Testing (2-5 days) Commit Ticket Project Management 3-6+ days
- 32. 100s of Failures 60% script issues (invalid tests, out of sync with functionality) 30% data/environment issues (data left in the database or filesystem) 7% pre-existing issues 3% newly discovered issues
- 33. Problem
- 34. The Problem ● Elongated testing cycles (3+ months) ● Longer time to market ● Reduced visibility to our development team ● Long delays between coding and fixing ● Too much noise distracting us from product improvements
- 35. Test Automation ● TDD ● Fully automated acceptance testing pipeline ● < 30m acceptance test feedback ● New testing approaches (Jasmine, Protractor, RESTAssured) 6+ month lead time 1-2 week lead time
- 37. Product Development Operations != != Development Test Production
- 39. Deployment Environments ● Snowflakes - No two environments were alike ● Completely different deployment methods o Manual development builds o Automated installs in testing o Gold master images in production ● Completely different deployment architectures o Windows development machines o Linux VMs in testing o Linux clusters in production
- 44. Learn in the Cloud ● Everything, from the ground up, is automated. ● Orchestration in code - stored in SCC, executed in ALL deployment environments. ● On demand provisioning for developers. o Still develop on their laptops, but can now test in a real deployment environment.
- 45. BbCloud ● OpenStack in our data centers ● Abstract cloud fabric to gain benefits in both clouds ● Centralized, standardized Chef automation ● Increased visibility for development o Monitoring, APM, statsd, centralized logging
- 46. Culture
- 48. “QA is responsible for defining the testing strategy.” “QA is responsible for checking quality when a feature is done.” “Unit testing is not enough to verify a feature.”
- 50. Executive Buy-in
- 53. What we’ve achieved ● Development teams deploying their production environments. ● Developers solving operational issues in production. ● Open feedback loops on operational issues. ● Data-driven decisions based on that feedback.
- 55. I might be able to help... ● Understand the impact that cloud computing can have on DevOps even in an “enterprise” company. ● Understand the cost models comparing cloud computing and traditional hosting. ● Frame your pitch to the executive team to make material changes in your company culture.
- 56. I’d like to learn more about... ● Effective testing strategies for traditionally manual tests (UI, etc). ● Applying DevOps strategies for shipped-to-premise software.
Editor's Notes
- #3: So who am I. I build things for Blackboard.
- #4: And I think a lot. Though, usually I do this with clothes on… especially at the office.
- #5: Sometimes I manage people
- #6: But really, I build things for Blackboard.
- #7: And just generally try to make things better. And this presentation is about just that – making things better at Blackboard through DevOps.
- #8: First a little bit about Blackboard: Went private in 2011
- #9: Extensive portfolio of teaching and learning, communications and analytics products and services. These products serve all stages of education from K through16 and on into adult continuing education. But today I’m going to focus on our flagship Learn platform.
- #11: Backstory - Gene Kim (the conference creator) describes companies as unicorns (“fictionally” perfect companies like Netflix, Ebay, Etsy, Facebook, etc) and horses (everyone else). He postulates that DevOps is even more valuable to the horses than the unicorns of the world.
- #12: Describes the common enterprise issues that we face like so many other companies at the conference. Creates connection with others at the conference.
- #15: Draws focus towards our improvements as a company.
- #18: For the product, we wanted to increase our release cadence and fix issues faster in the field. To do this, we took on modularizing our product.
- #19: Tried to contain the growth by introducing tools to improve the build performance and visibility.
- #20: Ultimately, we were building a monolith. And with that came: Poor product quality Slower release times More instability Slower developer productivity
- #21: The machine was getting more and more complex and error prone. And the larger the product got, the longer the lead times got. 24-36 Hour integration feedback
- #22: Enter Mike McGarr. Though no longer at Blackboard, he was hired by Steve Feldman, another former blackboarder here this week, to really kickstart DevOps.
- #23: Mike brought a different mindset to our team. He started to push the team to think about new ideas.
- #24: For the product, we wanted to increase our release cadence and fix issues faster in the field. To do this, we took on modularizing our product.
- #25: For the product, we wanted to increase our release cadence and fix issues faster in the field. To do this, we took on modularizing our product.
- #26: This made for some impressive improvements in code modularity and in combination with the updates to our build process, better feedback to the developers. This graph shows the shift from our previous monolithic codebase to our modular codebase that can be built in pieces.
- #27: He replaced the antiquated build tools. He started to introduce automation tools like Chef and Puppet to our integration test infrastructure. And this wasn’t just replacing our current build with a new one. He was introducing tools that allowed developers to do their own operations work. He was taking us towards DevOps.
- #34: Old acceptance tests *heavily* skewed towards GUI tests.
- #38: Differentiation between test deployment environments creates inconsistent testing results. “It worked on my machine” syndrome.
- #41: Chef began to take hold, but we had two completely separate efforts with minimal knowledge sharing.
- #42: The promise of cloud computing was our next stage.
- #43: I got the opportunity to build a super team of development and operations engineers to take on an shadow project - to take our flagship product, Blackboard Learn, into the cloud.
- #48: As an organization, we were very siloed. Every team just packaged up a deliverable and passed it over to the next team. Dev built a product and handed it off to QA for testing. Washed it’s hands and moved on. QA handed the GA build to operations and washed it’s hands. Operations was left on it’s own to make it work with very little feedback to the development team. There was no shared responsibility.
- #51: Key to cultural change at Blackboard was executive buy in. Once the team saw that the leadership wanted devops, the tide shifted.
- #52: Single biggest change was bringing the groups together into one organization. We put people in leadership positions that cared about development and operations in one. We placed development groups into the infrastructure and IT teams. We integrated application operations teams back into the development teams.
- #53: And we automated all the things! No, you can’t automate culture changes. This takes human capital.
- #55: And most importantly, we are finally talking.
- #56: I was honored to be invited to speak along side some of the leaders in DevOps. I hope that I can offer something back to the community here. «