DevOps tools for winning agility

Editor's Notes

• #4: Setting off on the DevOps journey can be unnerving. If you only came here to learn how to use DevOps for testing, you’re only part way there. DevOps often overlooks testing, (dev/ops, yes?) but it needs to have a complete Development cycle solution. What’s it going to take to implement DevOps across the enterprise? As many who have gone before you, that means you have to plan anywhere from a small team to possibly hundreds or more resources, applications, data sources and hardware. How many scripts and tools will be required to make your pilot projects work? Consider those to support infrastructure, manage deployments, even orchestrate release pipelines or the DevOps toolchain. How will you take this on without having DevOps consume your teams completely? This is what we’ll discuss here and what tools can ease the transition.
• #5: Deployment can be simple if you’re deploying a single application to a single environment that only plans on changing on a regular interval or include a small portion of your business. What do you do when it comprises of large data sources and complex organizations with constant change? This is the norm for today’s enterprise applications, which deploy multiple times to different environments, even hourly. Deployment has traditionally meant writing and testing scripts, customizing and configuring them for different applications and environments, and updating them every time something changes… in a very manual way, but that isn’t scalable to DevOps. Introducing the DevOps model- A more modern and scalable approach is to automatically generate standardized deployment plans based on a model of your application and environment. Think of it like a “template”, with different baselines and sizes for enivornments to use. Relying on a model means that your deployment plans are more likely to execute in a repeatable manner across the enterprise, even with extensive custom scripting, as it’s transparent due to tools utilized. When deployment plans are generated automatically, development teams can spend less time writing and maintaining scripts and more time innovating.
• #7: The larger and more complex your environments are, the more benefit you’ll see from advanced deployment approaches such as blue-green deployments, canary deployments, and rolling releases. Implementing these patterns with a provisioning, configuration management, or Continuous Integration tool requires extensive manual work and scripting, which means more complicated maintenance… forever. Take advantage of the pre-built DevOps best practices in an Application Release Application (ARA)tool, rather than trying to script, in other words- don’t reinvent the wheel! When you have built-in support for sophisticated release and deployment approaches, adapting a deployment for a particular situation is as easy as selecting a different pattern. As teams mature in their Continuous Delivery adoption, they can experiment with the patterns that are best for their goals, while still benefitting from standardized processes and pipelines.
• #8: Keep in mind that there are many terms used for the concepts on this slide. I’ve chosen the most common ones, but depending on the choice in Agile and DevOps methodology, the words may change, but the goal is the same. DevOps is a software engineering practice that aims to bring Development and Operations teams closer together to achieve shorter development cycles and more frequent, more reliable software deployments. As companies venture off into galaxies unknown, the DevOps journey usually brings big changes to a company’s culture, processes, and tools. Ultimately, DevOps is about delivering products, features, and services to customers and to the market faster—and automation of the complete application lifecycle plays a key role. Reliably promoting software through a delivery pipeline, step by step and from environment to environment, typically requires a toolchain of myriad software tools designed to carry out a variety of specialized delivery functions, from Continuous Integration to configuration management, from build to Application Release Automation… and often many more. Application Release Automation (ARA) orchestrates the overall build, test, provisioning, configuration management, and deployment tools in your software delivery pipeline to automate the release process, starting with the initial idea of a feature and ending with the ultimate delivery of that feature to the user. ARA also encompasses automating the deployment of applications across multiple environments in the pipeline, from Development to Testing to Staging to Production.
• #9: Build automation is the process of automating the creation of a software build and the associated processes including: compiling computer source code into binary code, packaging binary code, and running automated tests.
• #10: Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life. A DBA’s desire for low risk and stability assists here as we desire routine that results in expected outcomes.
• #11: Continuous delivery (CD) is a software engineering approach in which teams produce software ... incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.
• #12: This is another area that introduces risk, so DBAs can be very adverse to it, but the focus on a single feature, helps minimize the impact and often isolate any issues.
• #13: Release Orchestration is the use of tools like XLRelease which manage software releases from the development stage to the actual software release itself.
• #14: OK, yours may currently may not be the same. We need to talk about how you can become aligned with everyone else’s goals. It doesn’t mean you have to give up your first database. You can be part of the goals of the company and still protect the data, all of the data and the database. Enterprises have many DevOps tasks that can’t be automated and require manual intervention, and scripts are inherently not suited for many of these activities, but this is why we incorporate tools into the solution to begin with. You will be required to have the insight and management skill to audit your end-to-end development process and delivery pipeline to include all manual and automated processes, automating anything that isn’t already and should be. You will need to help identify and build out a shared, purpose-built ARA platform that everyone can utilize as part of your Continuous Delivery solution.
• #15: The concept was first coined just a few years ago by a Senior VP Platform Engineer, Dave McCrory. It was an open discussion aimed at understanding how data impacted the way technology changed when connected with network, software and compute. He discusses the basic understanding that there’s a limit in “the speed with which information can get from memory (where data is stored) to computing (where data is acted upon) is the limiting factor in computing speed.” called the Von Newmann Bottleneck. These are essential concepts that I believe all DBAs and Developers should understand, as data gravity impacts all of us. Its the reason for many enhancements to database, network and compute power. Its the reason optimization specialists are in such demand. Other roles such as backup, monitoring and error handling can be automated, but the more that we drive logic into programs, nothing is as good as true skill in optimization when it comes to eliminating much of data gravity issues. Less data, less weight- it’s as simple as that.
• #16: In computing, virtualization means to create a virtual version of a device or resource, such as a server, storage device, network or even a database. The framework divides the resource into one or more execution environments. For data, this can result in a golden copy or source that is used for a centralized location and removal of duplicated data. For read and writes, having unique data for that given copy, while duplicates are kept to singular. So why is this so important to me? I was a database administrator before becoming DevOps and I continually see data as the biggest hurdle for most successes.
• #17: Point out the engine and size after we’ve compressed and de-duplicated. Note that each of the VDBs will take approximately 5-10G vs. 1TB to offer a FULL read/write copy of the production system It will do so in just a matter of minutes. That this can also be done for the application tier!
• #21: Using any storage and only fraction of space Syncs with native or third-party database/snapshot backups Can maintain weeks of data changes. Managed just like any full copy of a database Users can instantly provision a read/write virtual copy of a database Can be used for replication, ETL, mirroring, and change data capture (CDC).
• #22: How do we “rewind” data and code changes now? Why should the DBA rewind changes made in dev and test? Why should you be the one to do this in test? Virtualization removes this. The Virtual databases are read and write, so even maintenance tasks, like DBCC’s can be offloaded to one. Ability to version control, not just the meta data, but the user data!
• #23: This is the time to be eliminated.
• #24: DBA has to commandeer a database for patch testing. This has to be performed for EACH environment, 100’s or 1000’s of databases! Most are not synchronized with production, different outcomes when released to production. Bugs occurring in one, not another!
• #25: Over 80% of time is waiting for RDBMS, (relational databases) or other data source/test data to be refreshed. Developers and Testers are waiting for data to do their primary functions. This allows for faster and less costly migrations to the cloud, too.
• #26: Package software into standardized units for development, shipment and deployment. A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
• #27: A data pod is a set of virtual data environments and controls built then delivered to users for self-service data consumption. It allows for self-management without the need for DBAs to manage standard processing, automate rebuilds and even remove need for backout scripts when development, testing and promotion goes wrong.
• #28: We refer to a container as a template in our product. Note that a data pod can be moved here or to the cloud…
• #29: This is a cornerstone to developers and testers, so as DBAs, we know the pain when a developer comes to us to flashback a database and before that, recover or logically recover, (import or datapump) independent objects. What is The developer/tester could do this for themselves?
• #30: If you rely on custom, handwritten scripts, you won’t have automated rollback when things go wrong—leaving you scrambling to resolve the failure and restore your application to a usable state- rarely if ever is the application the only thing that’s changed- external files and even more complex- data sources have changed and need to be rolledback as well. You could script this and it often is just this once you get to the release- just means the number of scripts you’re dealing with multiplies exponentially, and the overhead to make sure those rollbacks scripts are all properly implemented increases dramatically. Depending on where the failure occurs… your rollback script might fail, too. This is where self-service portals to eliminate the need in non-prod environments can make a real difference, removing the need from the development/testing cycle until the test of the actual release/rollback.
• #31: This is the interface for Developers and testers- they can bookmark before important tasks or rewind to any point in the process. They can bookmark and branch for full development/testing needs.
• #32: This may appear to be a traffic disaster of changes, but for developers with Agile experience, a “sprint” looks just like this. You have different sprints that are quick runs and merges where developers are working separately on code that must merge successfully at the correct intersection and be deployed. Versioning with source control is displayed at the top, using Virtual images. You can see each iteration of the sprints. In the middle section is the branches of that occur during the development process. A virtual can be spun from a virtual, which means that it’s easier for developers to work from the work another developer has produced. Stopping points and release via a clone is simply minutes vs. hours or days.
• #33: The maturity of the DevOps environment will decide how silo’d or how blended the role you’ll have in DevOps vs. your standard role as a DBA.
• #34: Methods provide a format or guide to work from. Hybrid approaches often implement best. Collaboration methods ensure that communication continues when team members return to their desks Deployment tools help with documenting and lessons learned Build tools help with automation and orchestration
• #35: Scrum focuses on features, bug fixes and backlog debt. Serves very large teams, including those 800+ Lean’s goal is to eliminate all waste, over demand on resources and ability to deliver faster and more effectively each time. XP is one of the most controversial due to the ability to deliver even to large companies every couple minutes if required.. Very disciplined approach. Crystal is often known under Crystal Clear, Yellow Orange and others.
• #36: What does a sprint look like?
• #37: These are just a few, but the ones we’ll commonly see as DBAs These are some of the most common software products that you’ll be using as a DevOps DBA.
• #42: Ant is another java based built tool that’s part of Apache open-source project. Similar to Make and written in XML.
• #43: This Groovy script executes another script, making it valuable in environments that already have a number of mature scripts in place that should be reused in automation.
• #44: This is our plugin- that’s how important we find these tools that we’ve built it into Delphix….
• #46: Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.. It groups containers that make up an application into logical units for easy management and discovery. Originally by Google and now run by Cloud Native Computing Foundation. Name stands for pilot. Kubernetes is being used in a number of vendor solutions including Mesosphere for its DC/OS product[12], Rancher Labs for its container management platform[13], Pivotal for its PKS product[14], Red Hat for its OpenShift product,[15][16] CoreOS for its Tectonic product, Mirantis for its Mirantis Cloud Platform[17], and IBM for its IBM Cloud Container Service [18][19] and IBM Cloud Private product.[20][21][22] Oracle joined the Cloud Native Computing Foundation as a platinum member on September 13, 2017. Oracle open sourced a Kubernetes installer for Oracle Cloud Infrastructure and released Kubernetes on Oracle Linux.[23]
• #47: Any cloud, application and a huge number of plugins, this product is focused on automation and from a DevOps tier. Allows for lab automation, CI/CD, Cloud Transformation, Network transformation and… Cyber Ranges: Reduce the effort required to achieve comprehensive security compliances to make cyber training more accessible for companies. This product also deploys virtually, which means a reduction in the impact to cloud costs, (storage and sprawl)
• #48: Jira isn’t just for bugs, (that’s what DBAs think, anyway… )
• #49: The dashboard is clear and concise, displaying the success and failure rates of test cases and progression.
• #52: DevOps automation, scale, legacy retirements Weaker on the Security, as it still discusses accessing data vs. masking data, but that’s still a common problem with most test data. Full integration with JIRA, Confluence, Crowd, LDAP, Continuous Integration tools like Jenkins, Bamboo and Automation tools like Selenium, QTP/UFT etc.
• #53: Test Data, Test Case and Test Script Generation Test Management It has orchestrators and automator for scripting. This is a very test-focused tool.
• #54: Script against any UI, Document with one-click, Execute in parallel, and host your test environment with in-house private cloud streamline test automation to do regression testing Just-in-Time Any applications, desktop automation with Cross-Platform execution, Early Test Automation right at the design phase
• #56: Automation Release Application Tools are Enhanced for testing why?
• #58: The status of a release—and the chance that it’s going to fail—isn’t just a technical concern. All stakeholders need tools that will take their “must have” requirements into scope. Trying to hand-code and maintain scripts is highly error-prone, especially as you scale. Any scripts used should focus on a tool that can eliminate human intervention from them and release with a secondary push file. Many tools today can serve unlimited programming languages and plugins for different DevOps tools. Use these to determine where new tools should be introduced with the focus on longevity.
• #61: Main points are: Collection of data on EU citizens, including IP Addresses, cookies, URL and user names online are all subject to GDPR. The citizen has a right to have proof of why you require their data, have it validated, audited, corrected if it isn’t correct And the right to be forgotten. Fines are up to 4% annual revenue or 20 million euro.
• #62: As development teams are empowered to deploy applications as part of their Continuous Delivery process, a simple approach to user management and granular access control becomes more important to ensure that code that doesn’t belong in Production doesn’t end up there. If access control is defined in code, then the burden of configuring and maintaining it falls to technical staff. Public companies and organizations in regulated industries are often specifically required to ensure that access control is carefully managed and that separation of duties is enforced.
• #63: Or does it shift the problem toward authentication and authorization?
• #67: Two books at the bottom…they’re homework