Old Tech to Shiny New Tech: Strategies on Upgrading Your Code Without a Big Bang

Editor's Notes

• #3: https://github.com/ThatConference/that-branding/tree/master/Speaker Slides
• #4: Old Tech to Shiny New Tech “No Big Bang” Strategies By Cristina Ruth @Cristina__Ruth @ Twitter (2 underscores) CristinaRuth @ LinkedIn Color Theme #881635
• #5: So we want to turn an old tech to shiny new tech. Let’s take a look at this house, our legacy system. Has old tech, has technical debt, hard to maintain.
• #6: And here’s our shiny new system we really want to get to. Latest technologies and capabilities! Easier to maintain! Minimal tech debt!
• #7: So how do we usually do this? Create a New Branch. Develop. Probably for several months. Test. Release. Finally!
• #8: But there are challenges with this. There are, what I call, “Rabbit holes” of forgotten functionalities. “How did this work?” “Is there any documentation on this?” Other functionalities get broken even if you did not touch them. The length of time it takes to get this completely done. The “big bang” when it’s time to go to production. Crossing fingers. “Hope this works”. Since it’s such a huge change, if there are critical issues and you’d have to undo your change, what then? Hopefully you can undo it within minutes. But realistically, it would probably take hours.
• #9: In this talk, we’ll talk through The 4 main challenges (problems) I’ve noticed with these types of changes. The 5 strategies I’ve encountered that helps address these challenges.. We’ll then walk through 5 steps to apply these strategies. And finally, we’ll wrap it up with a recap. After the talk, I am hosting an open space, so please feel free to come with any questions, or if you’d like to discuss further, or if you have different strategies yourself! I’d love to hear from you!
• #10: Hi! I’m Cristina Ruth and I’m a Senior Business System Consultant at CUNA Mutual Group. I am responsible for ensuring our self-service website for our customers stays up and running, keep it maintained and work with our business customers on making updates to deliver improved value to our customers. 1 website on the surface, but really comprised of 40 total “apps”. Each one with their own CI/CD pipeline, release pipelines, etc. I’ll upload my slides on slideshare.com by tomorrow and tweet on Twitter when it’s available. Please note that my Twitter contains TWO underscores.
• #11: When we’re making huge changes like these, I’ve noticed that there really are 4 main challenges that we encounter.
• #12: 1. Size of Change Bigger Changes = Bigger Risk / Bigger Bang Rollback capability is limited. Confidence Level of change success diminishes.
• #13: #2 User Impact Breaking Changes = High Impact Ideally, we would want no impact, or very minimal interruptions, if it cannot be avoided.
• #14: #3 Resources Trio = Time, People, Money Usually Limited. If only you are the only one available to do the upgrade, it will likely take you months to completely finish the work. Tech upgrades are also usually difficult to attach to a business value, and therefore, don’t usually get the needed funding to get completed.
• #15: #4 Technical Debt. Tightly Coupled Logic. Hard to test code. Hard to update – things break when you do.
• #16: These challenges are not present only when you’re switching technologies. They apply to ANY major changes you make. These problems apply to any major changes you do to your code. Switching infrastructure (database switching, API/service version upgrades). Code rewrites. Major functionality updates. Dependency version upgrades. Major new functionality with new infrastructure pieces (services, database changes, etc).
• #17: So how can we have an easier time addressing these challenges? What strategies can we use to help us mitigate these risks? We’ll talk through 5 strategies we use today in our team, and we’ll go over each one of them. They are: Backwards-compatibility Toggles Small Chunks Greenfield Revisit Requirements
• #18: #1 Backwards-Compatibility
• #19: The concept of avoiding breaking changes for users who use your application, despite the code changes you made that introduced a “new” way of doing things. Sometimes, the “new” way is not user-facing, but just in how the “engine” under the hood works. For example, you are changing the schema of your database behind your application. You wouldn’t want to delete a column because any applications who use that column will then start breaking. This also helps should we need to backout our change.
• #20: Let’s take a look at an example. An API/web service just released a new major version. What if they told you they’re releasing next month and that your application will stop working if you stay on the old version? Wouldn’t fly. With this strategy, the developers would have a retirement timeline wherein: Support both the “new” and “old” way. Give users time to shift to the “new”. Retire “old” after some time. Control versions via input or parameters or pathing.
• #21: Consider your application’s users and the impact of this on them, and the effort it could take for them to shift to your new version. Or, if no action is required on them, just make sure your changes will still support the older version (i.e. the new database schema need to be compatible with both code versions) But also, take note that the duality of Old + New = technical debt. Ensure to retire it to prevent future confusion, and for easier maintainability.
• #22: With backwards-compatibility, we ensure that impact on our users is minimal, if any. No breaking changes since that results in unhappy users, and probably unhappy bosses/colleagues. If user action is required, we need to give them time to shift, but not too long because during that time, we are in a duality and we’d like to shift away from the duality as soon as we can.
• #23: Strategy # 2 – Toggles Toggling is turning things on or off. Heard of feature toggles? Same concept, basically.
• #24: Reduce risk. Allows increments. Deploy small changes as “off”. Turn “on” to test and turn it off again. Allows rollbacks. Turn off if critical bugs are found. Release early to production, and turn on specific date with high confidence that it WORKS.
• #25: To effectively toggle, you’d need a system/spot to store/read them. Some options: Config Files (CI/CD) Tokenize and use release variables to control the toggle per environment. Database Similar to above, but controlled via database table values. Configuration System e.g. Content Management Systems Intrastructure/Network It depends on your change, but typically, the ones above that do not require a code deploy are preferred. Code Deploys = App Restarts = Blips for your users. Unless you have an amazing deployment system wherein your code deploys don’t cause your app to “blip” 
• #26: We’ve heard of the feature toggles, but I’ve used many non feature-toggle types, and they grow in size and also in complexity. Function – toggle a function Component – toggle a component Dependency – toggle a dependency, like a web service, or database Server – toggle a server via network
• #27: Toggles give us many benefits but they come at a cost. Similar to backwards-compatibility, we have a duality present, and with duality comes technical debt because there is no “one” source. I think of them as “branches”, but these branches co-exist in master, but the branch management/overhead still applies. Any updates to the functionality I have to make sure both toggles get. Also, once the change is done and stable, there would be no more use for the toggle. They need cleanup as well. Remove the toggle, remove the old code. Not doing so will cause future confusion of which one is right and also give your code “bloat” of unused code being in there. Try to use them only as needed, recommended for major changes/risks.
• #28: So we can toggle a function, dependency, server, etc. But there’s another “layer” of toggles we can apply to each of these. A global toggle is where the toggle is available to everyone once turned on. A phased toggle is controlling the impact of the toggle to a specific set o fusers. Use phased toggles for very, very major changes wherein you would ideally like to test it on a subset of users. If you have thousands of users, and you are not sure a functionality would be well-accepted, a phased toggle could be a way to introduce it slowly and test and learn from. Increase the impact size over time until the decision is made to release to all users.
• #29: We talked through the gotchas of toggles. The larger the toggle (function vs server) means increased complexity. Larger toggles = more code “duplicated”, and the more code to manage (and maintain – they would both need updates) while you have that toggle present. Keep it simple, the simplest one that meets your needs.
• #30: With toggles, we have a duality present so we need to ensure that we test both cases where the toggle is on and off. The benefit this gives us is higher confidence level should we need to back out, but if we don’t test the “off” version, then that confidence level goes away.
• #31: Toggles work best with backwards-compatibility in mind. Always asking how the changes would be if the toggle is off AND testing it will help ensure your application runs as smoothly as possible.
• #32: Toggles enables us to deploy and test in small chunks and therefore in increments, because we are able to turn things off until we are ready to “go”. Toggles also help us mitigate critical bugs, and we can sleep easier knowing that all we need to do is toggle the thing “off” should we run into major issues. Our business customers/stakeholders feel better also, and we’ll gain a higher confidence-level from them.
• #33: Strategy #3 – Small Chunks With toggles, we were able to deploy small chunks of code and keep them toggled off until they are fully ready to be “released”. Similarly, we can chunk out our work in smaller chunks to enable us to “release” portions of our code faster and with lower risk since we’re releasing smaller pieces.
• #34: Chunking is hard and I still struggle with it, but it helps to look at vertical slices first. Think features or functionalities. Once you’ve identified one, then think through the dependencies (database, API/web service, etc) it needs. These would be your “chunks”. Ask “What do we need to have done before this feature works?” Another thing is that you may be looking at a functionality. But it’s really a BIG one. It may be worth re-looking at and ask if it really can be comprised of chunks. A recent example of this is we were implementing a multi factor authentication feature into our site. We had to update the login form and provide a way for users to sign up for factors and maintain them after. “What needs to come first?” helps identify the chunks here. The chunks are then 1. the login form 2. the sign up for factors and 3. maintenance. Without the login form, the sign up for factors would not have any way of being “presented” to the user. Without the sign up, there would be no maintenance. So we split these 3 into 3 chunks. Each one is a major chunk by itself. In addition to the smaller pieces, releasing in small chunks also help when resources are limited because the amount of work (scope) to be done is smaller. Chunking also reduces the duality faster since the “ready” pieces can be cleaned up shortly thereafter.
• #35: With chunks, we help reduce change size and thus, lowers our risk at release time. It also helps keep the team morale higher because things are getting “done”.
• #36: Strategy # 4 – Greenfield on Legacy I’ve personally struggled trying to make major changes to a codebase that has so much technical debt. This is a strategy I recently discovered, and it’s basically another type of “toggle” but on a project level, where the old code full of technical debt is physically separated from a new project that, hopefully, will have zero to little technical debt. Can utilize editor add-ons to automatically enforce rules that the new code will never reference the old code, but old code can reference the new code. It’s great because it allows more “greenfield” feel. Use with toggling and small chunk strategies to make it even more effective!
• #37: As we build our new “greenfield” code on top of our legacy systems, it is optimal to avoid any technical debt as much as possible. Otherwise, future maintainers of our “greenfield” code will run into the same issues we are now and we don’t want them to hunt us down, do we?  This topic is massive and could be a full session itself (or multiples even) but here are a few key things to keep that technical debt from piling up. Dependency injection extracts dependencies as parameters passed to the constructor, so an object does not initiate any other objects it’s dependent on. Interfaces define an object’s properties and methods and allows objects to be mocked (fake duplicate) via a mocking framework (i.e. Moq). With dependency injection, interfaces and mocking present, unit tests are easy to write. Unit tests can act as guardrails and automated checks and are self-documenting. A future developer who does not know the codebase can trip these up and there’s no prior “documentation” that needs to be read that they probably won’t even remember anyway. And lastly, documentation. I KNOW! It’s not fun, but think about those rabbit holes you had to dig through and scratching your head on why something was written the way it was. I’m not saying document everything. There is a balance. Try to capture key decision points to help capture reasonings on why that route was taken.
• #38: When technical debt is present, and lots of it, it may help to use the greenfield strategy to help bring that “new” feel back into the project and removing limitations that existing technical debt may bring.
• #39: As we work through our small chunks or components or features, or as we dive down the rabbit hole of “why does this work like this?”, it may be worth the effort to revisit the requirements and see if that functionality is still needed. Business rules/processes may have changed. Engage with your business users (or your co-workers) to get the question “Do we still need this?” answered. Similar to “small chunks”, this helps reduce scope and therefore, can result in less work that needs to be done, and even better, the work can be done in a faster timeline.
• #40: Revisiting requirements can help reduce the amount of work that needs to be done, and therefore reduce the change size. Also helps when resources are limited because the team would have a lighter workload on their shoulders.
• #41: With these 5 strategies in mind, let’s put it all together and try to apply it in an example scenario.
• #42: Let’s take a website that has a modular architecture, runs on a content management system, and is on WebForms technology. We want to upgrade to MVC because that will give our system access to better capabilities that our business customers would like to use. But how do we do it safely, and with zero to minimal impact to our users and release something every sprint? Think about it for a couple of seconds and we’ll walk through how we could apply the strategies we just discussed on getting this converted to MVC.
• #43: Step 1. Analyze the challenges. Which challenges are we faced with on this task? We have all 4. We have 40+ components to tackle. (big change) These components have technical debt. It’s hard to update the code without breaking something else. We have thousands of users and would need zero to minimal impact on them. We have limited people to get this done, but can take as long as we want.
• #44: Step 2. Choose a strategy. Or more. We have all 4 challenges so we’ll use all 5 strategies combined. We will focus our small chunks on components, and re-assess the requirements for each. We’ll then use a toggle to “release” the new component version. We’ll also store the new code in a new project to physically separate the old code from the new low-technical-debt code.
• #45: Step 3. Identify small chunks. With toggles, we have dualities in place but is there another duality we could do? Could we have a “hybrid” where MVC components co-live with WebForms components on a page? From research, yes, we can, so our small chunks are: Enable “hybrid” on platform. Create a new MVC project. (greenfield) Next, we want to vertical slice, so we try to find the most important component. We consult with our business customers, and identify component A. We then ask what foundational pieces does this component need? Which pieces need to be in place first before this piece is considered complete? User information, web service/API integration, database integration? These are our next small chunks, along with the toggling and cleanup. Integrate with Web Service A. Convert Component A to MVC. Ensure Component A has Unit Tests. (or do this with above bullet point via Test Driven Development (TDD)) Swap Component A WebForms with Component A MVC. Mark Component A WebForms Deprecated. (backwards-compatibility) Retire Component A WebForms when Component A MVC is stable for 1-2 sprints.
• #46: Step 4. Execute and release our identified small chunks. We are trying to avoid a “big bang”. We’ve identified and planned some work, all of them can go live by themselves because we have a toggle in place to be turned on only when the component is ready. And because we are vertical slicing, we CAN release the component when all the chunks are ready. So let’s stop planning, and start executing and release the component when ready! If each chunk turns out bigger, chunk them out further. Develop. Test. Release. Again, we are empowered with the toggle because we can easily turn it off after a release if major bugs are found.
• #47: Step 5. Rinse and repeat. Keep identifying vertical slices and their chunks and executing and releasing until the conversion is complete. As you do this, remember your previously completed vertical slices and try to clean up their toggles and old code too! You may need to do a final cleanup once this is all done to ensure no old code is lingering around.
• #48: As we execute and release and keep iterating through our steps, and releasing something every sprint, new code will accumulate in production and the old code will be retired. We, of course will have extra to manage because of the dualities, but what we get in return is safer, higher-confidence, smaller releases. During the upgrade, our codebase, because of the duality, will have a “technical debt” with the toggles. Which is why it’s important to keep things as small as possible and take the toggles out as soon as you can.
• #49: So we’ve talked about a lot of things. Let’s recap what we’ve covered.
• #50: When switching technologies or making any major changes, the 4 challenges are, 1. size of change, 2. impact on users, 3. resources, and 4. technical debt. The bigger the size of the change, the more complexity, the bigger the risk, the bigger the “bang”, the more unreliable your “backout” plan is, and your confidence level of a successful change diminishes. Any changes we make we do not want to break anything for our users, or very minimal interruptions if unavoidable. Resources are sometimes limited, whether it’s time, people or money or all. Technical debt exists. Code breaks when changes are made. Hard to test. Scary to even touch!
• #51: To help address these 4 challenges, we’re armed with 5 strategies. Backwards compatibility is ensuring nothing breaks for the user with your change, and if you are releasing a new version, that the old version is still supported, at least for some time, before you deprecate it. This also makes your backout plan easier should you need to undo your changes. We’ll have a duality for a while here but with the benefit that users are happy with us. We just need to ensure that we cleanup this duality in a reasonable amount of time. Toggles allow us to turn things on or off and can be applied to many different scales. From small things like functions or even just variables, to large scale like an entire server. There are also global vs phased toggles to help “phase” things / enable your business customers to pilot changes to selected users. One of our favorite usages is to release all the necessary changes early, toggle it on to test, and then turn it off again, until the “real” release date. But we need to be careful using this because they require management, and also needs to be cleaned up when they aren’t needed anymore. Small chunks allow us to release smaller changes. Think dependencies or infrastructure pieces. Or, even chunking out the functionality to keep the change small. Keeping these small helps when you have limited resources so the scope is smaller and lower risk. Use with toggles to release smaller “pieces of the pie” to production without having to worry about releasing something that’s “not ready” to users. Greenfield helps address existing technical debt by creating a physical separation between the old code and the new code. Hopefully, the new code will have zero to minimal technical debt  Revisit requirements helps reduce the total size/scope and can even help save some work from having to be done. Especially useful when resources are limited.
• #52: So when making major changes, we then apply our 5 steps. Ask which challenges are present. From those challenges, choose a strategy or more. Identify a few small chunks to help you get started. Execute and release those small chunks. Repeat steps 3 & 4 until you are done! Ensure to cleanup 