White Paper: Scalable Lifecycle Management via Perforce
0 likes344 views
Perforce is used at NYSE Euronext for scalable lifecycle management across their growing infrastructure of thousands of servers. It provides traceability from production binaries back to source code, facilitates automated deployments of dozens of releases per hour across servers, and enables fast rollbacks. Perforce maintains an accurate "meta-map" of each server's configuration and is used to detect any unauthorized changes to production environments.
White Paper: Scalable Lifecycle Management via Perforce
- 1. Scalable lifecycle management via Perforce Adam Breashears - Release Automation Tools & Services Valerie Hendrickson - Release Management Abstract Managing complex, dynamic production environments where speed & accuracy of deployment intersect with a requirement of deep information about the state of each environment (current and historical) and a guarantee of application integrity is a hallmark of high-transaction low-latency environments. The same qualities that make Perforce a fantastic source code repository, the attention to protecting and tracking a company’s source and intellectual property, overlap to a large degree with the requirements of maintaining a critical production environment. Introduction1 NYSE Euronext faces a wide range of challenges stemming from its leading market position globally, its recent mergers which created a global exchange and financial service provider, with offices and data centers across two continents bound together and connected to its customer base by a global private network the Secure Financial Transactions Infrastructure (SFTI) created and maintained by NYSE Euronext. In addition to synergistic challenges created from the various mergers, the server infrastructure maintained by the Release Management team at NYSE Euronext has expanded from 300 (as the start-up Archipelago) to 9,000 servers (at peak during a data center migration), with a target of 6,000 globally by EOY 2011, in a corporate culture which prizes increases in scalability and efficiency (alongside an indifference to extremes in variance), not increases in headcount. Continuous corporate growth that is fueled by both organic and M&A activity has necessitated a process where variant processes can be made common at certain systemic points. Hardware and software migrations take time and involve organizational risks, but 'synergies' require that they be managed as a single 'environment' on both side of the life-cycle, whether it be radically different styles of SDLC entering into the production environment or interactions with said production environment. There's no room for 1:1 solutions to development styles and environments. Therefore, virtual consistency must be high, and certain functions must look the same for example software installs/rollbacks. In addition to virtual consistency, a common prerequisite to our environment is the ability to have absolute traceability path into the production environment, binary to source, configs/scripts to baselines, and historical definitions of all changes. Every script run in a production environment must be available for review at a later time and of course, all of this has to happen with no human driven interaction or time delay to the ability to execute. Perforce at NYSE Euronext is much more than a source code repository2), it is the foundation of information transfer and validation between any software artifact that goes between production and non- production. As such, it is under high load 24/7 and is considered a mission critical resource for IT. What is the full scope of Life Cycle management? There are two related concepts at NYSE Euronext in terms of Life Cycle management - Software Life 1 This should be read in conjunction with the presentation presented at the 2011 Perforce conference. 2 Perforce fulfills that function , along with many others to various groups within the company, however here we'll be limiting the scope of discussion to its role in Production focused Life Cycle management
- 2. Cycle Management (controlling how software is used in production throughout its usable life cycle) and the Software Development Life Cycle3 (SDLC) and both utilize Perforce as a critical component. Specifically within the scope of this white paper we'll be focusing on the final cycles of the SDLC and the production component of software utilization. Some specific challenges that needed to be addressed (in no order of importance): • Detecting unauthorized use/deployment • Detecting unauthorized change (config/script/binary) • Making sure software still exists on appropriate systems • Creating common patterns of usage for unique systems • Having full traceability from binary on machine X in production to source files in development • Be able to deploy dozens of releases to thousands of servers in under an hour • Verify explicit patterns of tested software configuration • Rapidly adapt (intraday) to shifting asset states by the hundreds (new/deprecate old/changes) • Be able to absolutely verify the software level of all assets real-time • Remove any possible source of inaccuracy (humans) from the release/install process How does Perforce help us accomplish this task? Perforce specific features The key features that Perforce offers its development community also coincide perfectly with the needs of a secure production environment, scalability, fast data operations, designed to accommodate global user communities/solutions, Easy administration/support, flexible/fast/Intuitive branching, an affordable pricing model and world-class technical support4. In addition to the 'high-level' value adds of the Perforce design, some more specific points include: • Maintaining integrity of data objects along pipelines (by atomic change lists) • Easy & secure integrations between secure dev/qa/production sections of the repository (as a result of the branching system) • Detailed trace of data objects as they route along pipelines for audit requirements (as a result of the embedded versioning mechanism) • Detailed trace of Perforce's own data objects, such as the client specs that represent meta-data around production systems (spec depot) • Protection of production meta-data objects from tampering (triggers) • Secure sections within each repository for different levels of access (dev/qa/prod) combined with triggers to explicitly limit access to appropriate users & domains • With all software objects defined by change list and pulled from a central source, Quality validation against known configurations can be explicitly defined and baselined for risk against component specific (as opposed to release) level rollbacks. • The production repository is a meta-map of all production machines due to the easy yet robust flexibility of client specs (with spec components pulling from different repositories for 3rd party software packages, Release objects, scripts, configs, etc) 3 SDLC - creating or altering systems, and the models and methodologies used to develop these systems 4 Reference Perforce website (www.perforce.com)
- 3. • Production machines are compared to baseline each night for possible change and additional of extra components (p4 diff/p4 add) • Production rollbacks occur within seconds across entire machine farms and are limited only by dependencies on other applications (atomic changelists with release naming conventions in the description guarantee precise conditions can be easily duplicated during a rollback). As an example of scalability of this method, while keeping the same general level of release management resources (who have to manage not only Perforce, but our release/issue tracking system, our build tool, our custom automation and provide user & branching/depot support), we've gone from approximately 100 releases per human (over a server population of 300) in 2007 to 3000 releases per human (over a server population of 6000 with a spike at the end of 2010 of 8,500 due to data center migrations) in 2011. Implementation of Perforce as a Life Cycle management tool Release Installation Process All production machines are set up in Perforce with a 'meta-map' of what their software configuration should be in production. The meta-map is a locked client spec, drawing from multiple sections of the repository; with a particular syntax in its description that we can use to map the client to its production host and purpose (for automation purposes it helps to have more data than just the 'host'). Where multiple user accounts are required for applications, multiple client specs are created and locked to their owner only (by depot permission settings). While the locked down client specs allow us to prevent changes to the meta-map in production, new spec creation mapping to a production section is restricted by triggers, only Release Management can create new meta-maps for production. Each line of software has its own sections for development, quality assurance and production, with each being integrated to the next through the p4 integrate command set. The descriptions of each have syntax that define what level of software they represent in our release tracking tool, which also stores the change list number corresponding to that release. The dual storage of release/changelist in each system allows for easy access, depending on 'which way' the user is trying to get to a particular piece of data - with the servers as the starting point (you'd look at the p4 have with the changelist description) or the end point (where you'd look at the release tracking tool, identify the change list number and then ask to install that number on a particular asset). Combined with a source of reference data (our Environment Mapping Database) we can automatically create client-specs (and meta-maps) as needed depending on business, software, and machine/OS type requested. The end result of this allows for very specific & rapid environment configurations in multiple environments and across multiple branches (in QA) as client-spec behavior has an attribute that when you change the meta-map of the spec, it removes all of the old synced files and replaces with the new ones, allowing groups to swap between completely different development efforts in minutes without having to work about binary overlap of differing functional levels. Automated Production Deployment For deployment outside of local environments, multiple proxies are setup to reduce latency of files to the local environment, but by far the most resource intensive component of the installation is the update to the 'Have' table in Perforce (currently at 30 gig and rising), one of the most important components of a front end Perforce server in this configuration is RAM (and local disk for the database - running release installations in parallel generates high levels of database activity that can exceed SAN capacity). Since Perforce requires a 'pull' methodology, there's a built in level of 'assurance' that we can't
- 4. 'accidentally' trigger an install at an inappropriate time. The 'install event' is a series of cascading scripts (automatically generated each night before the installs) that specify what each client should be from the client and under the watchful eye of our operations division. Once a sync has been completed and the Have table updated (Perforce doesn't update the Have table unless the file transfer is successful), release rollbacks are as simple as using the same scripted methodology to sync to an earlier changelist, which removes all file versions submitted after that version, removing the fear of having an untested release configuration in production (for example, version 7 of X tested with version 5 of Y, but the requested rollback scenario only wants a rollback of X). Rollbacks to N- 1 releases are auto-generated and included with the release package, but rollbacks to earlier releases are easily accommodated and take minutes to set-up and deploy. Should multiple systems require rollbacks and there be confusion over the desired release levels, rollbacks using a variant of p4 sync @05/01/2011 are always possible as well (although fortunately we've never had to do this). Once the software objects have been integrated into the production staging area, release times are literally the time for any pre-install scripts to run5, a few milliseconds of database time, the raw speed of an ftp transfer to the target system plus the time of any post-install scripts (in Perforce and a point of variance depending on system design). At the end of the install, all information results from scripted activity is stored in a text file and submitted to Perforce for later use. Post-Install Once objects have been installed into the production environment, Perforce becomes the 'root' of our application environment that we use to watch for change. Unlike applications that watch for file system change, Perforce allows us to differentiate between known changes and unknown changes by doing a P4 diff against the 'have' version currently on the client. Run after shutdown and before we start for the day, the diffs give us an excellent insight into application and configuration integrity before the beginning of a trading day, and quickly highlight any changes made during the day based on decisions by operational staff so those changes can be incorporated back into the baseline upon confirmation. After each diff is run an email that groups all of the results by project area is sent out (sorted by those projects with failures at the top) which links to an interface allowing current and historical perusal of diff data, with links to breakdowns of the server farms and the actual diffs per machine. This allows us a quick method of seeing the impacted business areas, the role of the servers effected and the timeliness desired for follow-up and resolution. With complete historical knowledge of software configurations, troubleshooting production issues becomes much more focused, as every piece of information around how the system was setup at that point in time is stored within Perforce and can be easily replicated in a lab environment or reviewed in detail. Operationally, the meta-map allows us to check the state of all software and configurations on a production machine without actually going to the machine, which creates not only a very convenient and easy to use method of checking any machine maintained by Perforce but also a much safer method of giving visibility into machine specific setups without giving everyone access to secure areas. 5 Stored in Perforce and designed to validate various criteria per project, generically refer to as 'release readiness checks' s consisting of steps ranging from checking disk space to validating user accounts and directory permissions.
- 5. The information stored within the meta-map6 is used throughout the company in anything involving the production environment, the visualization tools using Perforce data to view machines related to software projects and roles, and the historical information about software levels on that box for use in scheduling application deployments for phased roll-outs of new software. Furthermore, it will be used on ongoing projects building management consoles that will dynamically manage and monitor resources from a point of view related to the business unit and software project and not from the machine perspective. Conclusion Perforce adds value far beyond is initial role as a source code repository for NYSE Euronext, it is a fast flexible information rich robust 'root' of our software production environment. As we increasingly look to expand globally and our global resource pools find themselves working in far flung environments, Perforce will continually provide a key common structural component that allows us to continue to scale by growing our automation platforms and not our resource pools. 6 The combination of data from the diffs, baseline, client spec, change and have tables.