Version Control meets Database Control

Editor's Notes

• #5: Founded in 2008, launched in 2010Jedidiah Yueh, President and CEOFounded Avamar in 1999, sold to EMC in 2006, VP Product Mgmt at EMCAvamar: >$1B revenue, 150 Employees: HQ in Menlo Park, SF, Boston, DC, London, NY and AtlantaGrowing 250% annually – 130+ customers including 100 Fortune1000 Customers
• #6: Founded in 2008Part of the Extreme Group which has about 180 IT professionals consultants
• #7: The topic of today webinar is how you can have your database development under version control principles and methods to catch up with your application development and how you can utilize the database virtualization in order to clone your database for parallel development
• #8: From the business point of view, they would like to have their system uptodate with no downtime.However researches show that:* 80% of outages, impact mission critical services* From the 80%, 50% are due to human errors* And In addition 40% of changes fail and require rollbackThis sums into many hours in which the systems are down and of course reflect the business.80%of outages impacting mission-critical services caused by people and process issues thru 2015, with the majority of those outages (50%+) caused by change/configuration/release integration and hand-off issues (Gartner RAS Core Research Note G00208328 Ronni J. Colville, George Spafford [October 27, 2010] – Strategic Planning Assumption(s) “Top Seven Considerations for Configuration Management for Virtual and Cloud )
• #9: From the technical point of view, Developers, DBA and basically no one like to repeat tasks over and over again.And we would like to have a system that will remember the changes we made 6 months ago and to allow us to focus on development tasks and not on overhead tasks and definitely not to repeat ourselves – this is why we want automation
• #10: In software development you cannot talk on automation and not to mention Agile & DevOps.So how we deal with the risk of changes, we release smaller and more focused changes while we believe they are easier to managed.We do it often, so we must have some automation (because a. we don’t want to repeat ourselves and b. we don’t want errors in this critical process)This is called Agile, but so far we were doing staff internally.With DevOps we can release the laser-focus changes to the operations often with confident.
• #11: No one today think on developing software without having any version control solution that will manage the changes, keep track on the history.But when considering the version control, a small but critical tier in the application is left aside – the database
• #12: Database is also part of the application.Code can be in several ways, schema structure, business code written in PL/SQL and reference data in lookup tables.The main reasons why database code is not native to traditional file based version control are that database is a central resource, objects within the database cannot being dropped and created as done to source code and as done in deployment to the executables.This brings us to a big challenge, how can we branch a database or clone it easily in order to be more agile? With this in mind, I’ll let Kyle answer.
• #13: You might be familiar with this cycle that we’ve seen in the industry:Where IT departments budgets are being constrainedWhen IT budgets are constrained one of the first targets is reducing storageAs storage budgets are reduced the ability to provision database copies and development environments goes downAs development environments become constrained, projects start to hit delays. As projects are delayed The applications that the business depend on to generate revenue to pay for IT budgets are delayedWhich reduces revenue as the business cannot access new applications Which in turn puts more pressure on the IT budget.It becomes a viscous circle
• #14: There is saying in the industry that we want “good, cheap, fast: choose two”Meaning we want to build applications quickly, ie fast, we want those applications to have good functionality and we want those applications to be \cheap to buildBut we can’t have all three.
• #15: I don’t knowIf these situations ring a bell at your organization orif you can imagine some of these situations But here are some of the issues we at Delphix are seeing in the industry with the companies we are talking to.Let’s look at the 5 points in more detail
• #16: We talked to Presbyterian HealthcareAnd they told us that they spend 96% of their QA cycle time building the QA environmentAnd only 4% actually running the QA suiteThis happens for every QA suitemeaningFor every dollar spent on QA there was only 4 cents of actual QA value Meaning 96% cost is spent infrastructure time and overhead
• #17: Because of the time required to set up QA environmentsThe actual QA tests suites lag behind the end of a sprint or code freezeMeaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increasesAnd the more time that goes by after the introduction of a bug into the codeThe more dependent is written on top of the bug Increasing the amount of code rework required after the bug is finally foundIn his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm Introduce the computer world to the idea that the longer one delays fixing a bug in the application design lifescyleThe more expensive it is to to fix that bug and these cost rise exponentially the laterThe bug is address in the cycle
• #18: Not sure if you’ve run into this but I have personally experience the followingWhen I was talking to one group at Ebay, in that development group they Shared a single copy of the production database between the developers on that team.What this sharing of a single copy of production meant, is that whenever a Developer wanted to modified that database, they had to submit their changes to codeReview and that code review took 1 to 2 weeks.I don’t know about you, but that kind of delay would stifle my motivationAnd I have direct experience with the kind of disgruntlement it can cause.When I was last a DBA, all schema changes went through me.It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided byThey developers to go to an EAV schema. Or entity attribute value schemaWhich mean that developers could add new fields without consulting me and without stepping on each others feat.It also mean that SQL code as unreadable and performance was atrocious.Besides creating developer frustration, sharing a database also makes refreshing the data difficult as it takes a while to refresh the full copyAnd it takes even longer to coordinate a time when everyone stops using the copy to make the refreshAll this means is that the copy rarely gets refreshed and the data gets old and unreliable
• #19: To circumvent the problems of sharing a single copy of productionMany shops we talk to create subsets.One company we talked to , RBS spends 50% of time copying databases have to subset because not enough storagesubsetting process constantly needs fixing modificationNow What happens when developers use subsets -- ****** -----
• #20: Stubhub (ebay) estimates that 20% of there production bugs arise from testing onSubsets instead of full database copies.
• #21: The biggest and most pervasive problem we see is slow build times.In order to set up an database copy for a development environmentsRequires submitting a request to management who has to review itThen if the request is granted, it is passed to the DBA who has to coordinate with the Sysadmin who has to coordinate with the storage admin.In such a situation it makes sense that copying a large database would take a long timeBut even when we talk to someone who uses netapp storage snapshots like Electronics Art, they said even using storage snapshot sit took2-4 days to get a database clone copy due to the coordination between DBA, sys admin and storage adminAt many of the customers we talk to provisioning a database clone copy takes weeks or months.One large global bank quotes us as taking typically 6 months to provision a database clone copy environment.Requirements: self service for app teamsRequirements: end-to-end automationMetrics: # people, process, time for deliverySo far we have talked about the weight of infrastructure on app delivery. Of course, to control and manage that infrastructure, firms layer on a large set of bureaucratic processes, change control, approvals, procurement, governance, etc etc. So the operational and organizational hurdles then create an even bigger drag on IT and app development.Here’s an example from one banking customer.Once the app developer puts in a request for a new development environment, there’s at least a week long wait for management approvals. Then project DBA work with the sysadmin and storage groups for capacity. If more capacity needs to be allocated, it’s 3 more days. If more needs to be purchased, weeks or months. If a copy of production data is needed, the process needs to wait on a production DBA, who might be busy with production issues. Recovering the database to a specific point in time and configuration can also take days.It is very common for two weeks to pass between a developer request and a ready environment. The process can be repeated for multiple environments, for data refreshes, and for integration across multiple systems.With Delphix, turns stop signs into green lights. Provisioning, refresh, rollback, and data integration happen nearly instantly and do not trigger approvals from production systems or require additional storage. That is why KLA is able to deliver 5 times the output from its SAP teams…Without Delphix, it’s impossible for organizations to implement the level of agile processes they desire. The management of data, and the bureaucracy of data management, slows things down too much.
• #22: Due to the constraints of building clone copy database environments one ends up in the “culture of no”Where developers stop asking for a copy of a production database because the answer is “no”If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
• #25: The problem is getting the right data to the right people at the right time
• #29: As Vmware takes a single set of hardware and provisions many virtual machinesDelphix takes a set of datafiles and provisions many virtual database clones
• #30: In the physical database world, 3 clones take up 3x the storage.In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compression
• #31: Software installs an any x86 hardware uses any storage supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX support SQL Server
• #33: EMC,Netapp, Fujitsu, Or newer flash storage likeViolin, Pure Storage, Fusion IO etc
• #34: Delphix does a one time only copy of the source database onto Delphix
• #39: Quote from a customer “Delphix GUI is what Oracle Enterprise Manager would look like if Apple had designed it”Delphix inter face is user friendly, polished and easy to use
• #40: Source Syncing* Initial backup once onlyContinual forever change collection Purging of old data Storage DxFSShare blocks snap shots , unlimited, storage agnosticCompression , 1/3 typically, compress on block boundaries. Overhead for compression is basically undetectable Share data in memory, super caching*Self Service AutomationVirtual database provisioning, rollback, refresh*, branching*, tagging*Mount files over NFSInit.ora, SID, database name, database unique nameSecurity on who can see which source databases, how many clones they can make and how much storage they can use
• #46: Presbyterian when from 10 hour builds to 10 minute buildsTotal Investment in Test Environment: $2M/year10 QA engineersDBA, storage team dedicated to support testingApp, Oracle server, storage, backupsRestore load competes with backup jobsRequirements: fast data refresh, rollbackData delivery takes 480 out of 500 minute test cycle (4% value)$.04/$1.00 actual testing vs. setup
• #48: For example Stubhub went from 5 copies of production in development to 120Giving each developer their own copy
• #49: Stubhub estimated a 20% reduction in bugs that made it to production
• #51: ExamlplesMacys 4000 hours/year cloning to 8 hours/yearKLA-Tencor over doubled project output, like taking 100 person team and making it a 200 person teamHealth Dialogue reduced storage from 720TB tos 8TB
• #52: Thanks Kyle, I'm sure that everyone will agree the opportunities with Delphix are very excited, the idea of spin up the databases very quickly it is exciting.
• #54: DBmaestro TeamWork is a solutions that enables you to take control your database development from the very beginning using version control.It is integrated with the existing ALM tools you already have.All changes are stored in a version control repository which is connected to the impact analysis module. More than just compare & sync.With automation in mind, you can complete your DevOps or Continuous Delivery or Continuous Deployment processes and if there is a slight change that something should not be promoted, TeamWork will make sure you know about it.
• #55: The challenges we’ve seen when we speak with people that there are:Many delay in developmentMany silos in the organization and within the development teamMany organizations still miss the due date of releasesThere are errors in productions