Agile mk-journal-issue-002
1 like351 views
The document summarizes a discussion at the January 2013 Agile:MK user group meeting about a project to develop a cloud-based computing grid platform called MG-ALFA Compute for Azure. A team of 4 developers used agile methods to develop the platform over 2 years. Key aspects included keeping the process simple, frequent design sessions, test-driven development, pair programming, and prioritizing the minimum required features. The platform successfully executed thousands of complex actuarial modeling jobs in the cloud.
Agile mk-journal-issue-002
- 1. Agile:MK December 2012 Journal January 2013 ISSUE 02 A special interest group for professionals from the Milton Keynes area who are interested in learning, sharing and encouraging the use of agile methodologies such as Scrum, Extreme Programming & Lean Thinking. Agile: “Getting the Business Involved” Open Space Session, facilitated by Iain McKenna, Certified Scrum Trainer Monday 25th February 2013 6pm – 8pm Sponsored by
- 2. Agile:MK Journal January 2013 Agile:MK User Group Happy New Year and welcome to the second edition of the Agile:MK Journal. The Agile User Group in Milton Keynes meets once a month to share and learn about agile methods through real-world experiences. Each month Steve Garnett we’ll publish a brief overview of the discussions Founder Agile:MK and insights of the group in this journal. To register your interest, If you’d like to join the group so you can please join the attend, the sessions, and be involved in the LinkedIn Group at http://tinyurl.com/ discussions face to face, to learn or share agile-mk-liug your experiences, please join us on LinkedIn. Agile: “Getting the Business Involved” Open space discussion workshop Facilitator: Ian McKenna Monday 25th February 2013 Profile Facilitator: Iain McKenna, CST Jurys Inn Hotel, Midsummer Boulevard, Milton Keynes, MK9 2HP A Certified Scrum Trainer with over 30 years’ experience in software engineering and over 10 years’ experience of helping teams use Scrum, XP and other agile and lean frameworks to deliver amazing software products in organisations including United Health, Cisco, AOL, BskyB, The Open University and the BBC. Agile:MK Journal, design & produced by endjin www.endjin.com
- 3. December 2012 The State of the Art The adoption of Agile methods and practices within the UK has been on-going for close to 20 years. Scrum has become a more predominant method over the last decade with a high growth curve over the past 5 years. However, this high growth of adoption has not come without cost. Agile has suffered from a lack of understanding, some governance structure, then it is likely that strong poor implementations, and a proliferation of terms oversight will be enforced. The creation of steering and complexity that means that Agile is increasingly groups, status reports, detailed business cases, and becoming polarised from one of its core other artefacts to facilitate distanced decision-making principles: Simplicity. may be required to see a project to completion. This can lead to waiting for decisions and delays in progress. When a mature company or organisation adopts agile Can this governance be distributed? I.e. Can more of principles, values and methods, it necessarily has to the control of how a project is run, who works on it, undergo significant changes in structures, processes, and its decision making be given to the project team governance, culture, and technologies. without recourse to a governing body? For a company to value its individuals and personal Another example. There are issues with the team’s interactions OVER processes and tools, working understanding of the business domain required to software OVER comprehensive documentation, create software because they are located off-shore customer collaboration OVER contract negotiation away from business subject matter experts? So, can and responding to change OVER following a plan, the software development be brought on-shore to the mind-set, governance and culture of the the business? Can the business people be seconded organisation has to change. to the development team? Or is there little movement for change and therefore we may have to implement At this fundamental level of change lies the crux of Proxy Product Owners or a team of Product Owners the problem. (the ‘middle-men’) to represent the business subject matter experts? What is the company’s propensity for change? Because agile teams are constantly trying to improve How UK companies answer this question is shaping cycle times and remove impediments, they are often how the UK Agile community is developing. looking at problems at their root cause, and finding ways to help the team move on. If the company has a high propensity for change then it is possible to get to the root cause and remove it. However, more commonly, the root cause problem is embedded in the organisation at a level that is very difficult for a team to remove. Work-arounds and sub-optimal solutions are adopted. These sub-optimal solutions have led to a proliferation of “agile” patterns and solutions that are making Agile a complex beast when its essence is simplicity. Propensity for Change & Complexity of Method The Case Study below was the main subject of the When adopting agile, we are essentially embarking January session and we learned through our on a continuous improvement activity and the more discussions that an essential part of adopting agile a company is able to change, the closer to root cause methods effectively and creating hyper-productive removal an agile team can reach. teams is making organisational change. For example, if a company has a rigid or centralised MG-Alfa (you’ll hear more about them later) recognised
- 4. Agile:MK Journal the project was important. They hired good people, comply with the new Solvency II regulations, but was co-located the team, gave the team autonomy over also seen as a valuable exercise in its own right. As their working practices, environments, processes part of the wider actuarial project, an on-demand, and architecture and let the team grow. The team cloud-based grid computing platform was also to be maintained agile principles and values and flourished developed to cost-effectively execute the complex because of the key decisions made earlier on about models being created. how to run an agile project. After some proof of concepts work during 2010, an I believe this foresight was key in this project’s success initial team of 2 senior developers began developing the grid and were joined in early 2011 by 2 more senior developers, including myself. With just one An Agile Cloud Based Project - change of personnel in late 2011, we became a highly Milliman MG-ALFA Compute for Azure effective team. by Ian Shimmings, Associate at Endjin The problem we had to solve was how to execute This article is based on a short presentation and the large number of jobs generated by the actuarial discussion to the Agile:MK user group, which was, models in a timely, reliable manner. As a rough guide, in turn, based on nearly 2 years as a senior developer a quarterly complete model recalculation, as required on the project. for Solvency II, may consist of thousands of jobs, each with typically 1,000 tasks which may each take between Summary 20 minutes and 4 hours on a single core machine. A computing grid platform for Windows Azure, called This equates to many hundreds of years computing MG-ALFA Compute for Azure, was developed and which needed to be complete within 3 to 4 days. It also launched at the end of 2011 followed by multiple needed to provide for the needs of ad-hoc job runs. enhancement releases. It is used to execute the complex actuarial models created by actuaries using High level requirements of the system included: the MG-ALFA core model generation tool. It has been tested with more than 45,000 compute nodes in a • Jobs must never be lost single grid, across multiple Microsoft data centres. It is in highly successful production use with Phoenix • Jobs must complete if possible Life who frequently run multiple grids with many thousands of cores. It is also now being used by • Problems that occur in production other existing MG-ALFA customers as a Software should be traceable as a Service to simply and cost-effectively run their actuarial models. MG-ALFA Compute for Azure • Independent of model changes and versioning has become one of the largest Azure installations outside Microsoft. • Be able to control and scale the grid to maximise cost-efficiency The grid platform was developed by a highly focussed team of just 4 co-located senior developers who took The chosen solution was to build the computing full responsibility for all analysis, testing and life-cycle platform based on the Microsoft Azure Compute management, as well as actual development. We used a platform using worker/web roles, storage and the simple, but very effective, agile process allied with good management API. Azure queues are used to development practices, that evolved with the project. transmit control and status information with blob storage holding data. As well as the main job execution This team at times reached what may be described and grid control systems, there are custom diagnostics as ‘hyper-productivity’ where significant functionality and an event driven information service used to drive was created extremely quickly to very high levels of web, mobile web and Windows 8/RT user interfaces. functional and software design quality. Agile Process Introduction The development process we started with was a simple In September 2010, Milliman MG-ALFA was chosen by agile one involving weekly iterations. At the end of Phoenix Group to simplify, rationalise and streamline each week we would review what was done, have a their actuarial projection systems and models (http:// retrospective to improve how we worked, and plan for uk.milliman.com/news-events/press/pdfs/uk-based- the following week. Our tracking consisted solely of phoenix-group.pdf). This was triggered by the need to cards and a simple three column Kanban style board.
- 5. December 2012 The cards consisted of MMFs (Minimum Marketable • If it can go wrong, (with thousands of cores running) Features), Stories and Tasks. Where possible we would it will go wrong! only work on a single MMF at a time. o Processes must be resilient and recoverable As the months went by the process began to evolve until we moved to a continuous development style o Persist data securely and often to aid recoverability where we no longer worked in weekly units but simply selected MMFs from the frequently re-prioritised and o Write lots of diagnostics, even when running live, modified list of MMFs originally chosen during release as repeating error conditions can be difficult and/ planning. We only created Story cards where a large or expensive MMF warranted it and rarely wrote Task cards as that was left to the developers working on the feature. A • The internet is slow and unreliable retrospective or re-planning session would be held occasionally if considered necessary, but changes to o Retry (almost) everything the required MMFs and the way we worked typically happened by agreement as and when one of us felt o Expect out of order and repeated events there was a need. • Keep in mind how the system will be managed This process evolution helped us be, as efficient as in production we could be while keeping the process itself as little more than a reminder of what we should be doing, o Any tool useful during development and testing developing the application. We specifically avoided may be useful during support and should be kept naming the process we used (Scrum or Kanban or anything else) as that makes it a thing in its own right, o Consider alerting on error, but reserve such errors frequently a hindrance, rather than a help. for worst case scenarios Other process features key to maintaining application Testing large, distributed applications thoroughly for quality included: the cloud can be challenging due to its remoteness, the number of processes and the time and cost • Being rigorous about developing the minimum involved. This makes development in the first place features required at the time of high quality code even more important than some other scenarios. • Frequently holding team design sessions whenever required, at least one for most MMFs • Rapid feedback on quality particularly important • Mandatory TDD (Test Driven Development) o Good unit test coverage • Mandatory pair programming o Effective local ‘implementations’ of the cloud environment important • A complete CI (Continuous Integration) environment, including deployment to an integration environment o Automated local end-to-end tests are vital for quick verification of the ‘wiring up’ of the • A fully automated build, test, package, deployment application developed an maintained by the development team o Automated deployment and in cloud end-to- Developing & Testing Large, end tests Distributed Systems in the Cloud All the best development practices, as well as high o Key integration and UI tests quality, well structured code, are as important when developing for the cloud as for any other kind of • Carefully scheduled full scale tests due to the development. However, like developing for any time and expense involved specific environment, it is important to understand the features presented and constraints imposed o Automated job submission when working in the cloud and/or developing large scale distributed applications. o Manual UI testing
- 6. Agile:MK Journal o Chaos Monkey testing – random server Companies outage, network failure, full memory or disk,etc. Milliman is among the world’s largest independent actuarial and consulting firms. Founded in Seattle, • Continuous Integration test runs that exercise all Washington in 1947, they are among the world’s tests in order of time taken largest providers of actuarial and related products and services. Consulting practices are owned and Learnings managed by principals. MG-ALFA is both a practice I enjoyed my time developing the Milliman MG-ALFA within Milliman and the name of the powerful Compute for Azure application. There were many financial modelling tool they produce. challenges and I certainly learned a lot. Below are just a few highlights: Phoenix Group Holdings has a Premium Listing on the London Stock Exchange and is a member of the • Agile is not about named processes, the arguments FTSE 250 index. The Group is a closed life assurance between proponents and sticking to the ‘rules’. It fund consolidator that specialises in the management is about using any simple template and constantly and acquisition of closed life and pension funds, and evolving as the team grows in experience operates primarily in the United Kingdom. Measured by and confidence total assets, the Group is the largest UK consolidator of closed life assurance funds. The Group has over • High quality software developers are required 6 million policyholders and assets of £71.6 billion as and are one of the most important factors for at 30th June 2012. a successful software development project Links • A small, highly skilled, agile team can • Milliman – www.milliman.com deliver massively • Phoenix Group – www.thephoenixgroup.com • Each team is unique, construct and modify as required • Endjin – www.endjin.com • Microsoft case study – www.microsoft.com/ • Once a good team has formed do what you can casestudies/Case_Study_Detail.aspx?casestudy to keep it together id=710000001018 • Good developers can (and are willing to) analyse, • Milliman cloud computing – www.milliman.com/ test, configure builds and deployments, as well expertise/life-financial/products-tools/mg-alfa/ as develop cloud-computing • A good developer without specific skills is generally • Arapsys – www.arapsys.com better than vice-versa • Developing for the cloud at Netflix – techblog.netflix. • Only build what is required now: com/2010/12/5-lessons-weve-learned-using-aws. html o Though don’t procrastinate and use as an excuse to ‘bite the bullet’ and make the big changes when needed • Making a practice mandatory (TDD, pairing, etc.) is the best way to REALLY learn it o Prevents reverting to type when under pressure o Once learned make optional to get the best results • Building and testing for large, distributed applications in the cloud is different…
- 7. Agile: MK Bookshop December 2012 The Machine that Changed the World By Womack, Jones & Roos At the time of its release, this book threw a bright spotlight on how the Japanese automobile industry was out-stripping the American automobile industry through the use of what is now termed “lean thinking”. The prelude to Lean Thinking, we are told the story of the automobile industry from Henry Ford through to modern day, highlighting the varying philosophies and methods of manufacturing. In taking the historical view, the book provides a clear illustration of the difference between traditional and lean practices. The parallels and connotations with our current software industry leap out from the page, and illustrate how and why some companies are repeating the mistakes of the past in their approach to adopting agile and lean practices. Crucial Conversations Fearless Change The Lean Startup by Patterson, Grenny, McMillan, Switzler by Manns and Rising by Eric Ries Organizational Patterns of Agile Essential Scrum Agile Software Development: Software Development by Kenny Rubin The Cooperative Game by Coplien & Harrison by Alistair Cockburn
- 8. ripplerock offers a set of services and tools that enable our customers to dramatically improve their software development capability RippleRock formed in 2009 with the mission to assist customers drive dramatic improvements in their software development capability. We apply our expertise across the full lifecycle; facilitating organisational transformation to enable Agile practices, all the way down to improving engineering practices within the teams. The team at RippleRock have a strong track record in the Agile space, some of this through experience gained while at the centre of Conchango’s Agile Practice and Scrum for Team System tools group. As a specialist in this area we are able to offer access to the most experienced Agile coaches, trainers and consultants with the particular mix of skills required to work with people, process, organisations and tools. Ripple Rock Ltd is registered in England and Wales No. 0708916 Ripple Rock LLC is registered in the United States of America. No. 27-1180168 www.ripple-rock.com