Legacy: A Retrospective - Open West 2016

Editor's Notes

• #5: In 2013 I joined a small company based out of Plano, Texas which offers online continuing education for multiple professions within the medical industry. They record video of industry experts teaching on a topic write an exam to test the student’s knowledge, then work with individual state boards and accrediting agencies to get the content approved as a continuing educational unit. They can then sell the course to customers. They also resell access to the content to other companies through a subscription model.
• #6: (next) When I started at the company, the software was not maintainable. To start, a large portion of the admin system didn’t have any functionality written - there would be a form with input fields and buttons, but no actual processing for it, or only for some of the fields. The parts that did work didn’t allow our users to do half the things they needed to do, so they were constantly coming to us with spreadsheets to manually import data into MySQL, requests for simple data records to be updated or deleted, or questions about how the site worked. (next) None of these problems were documented in any way, so whenever a new employee started, they had to be trained to ignore the UI, because those forms didn’t really work. (next) The customer facing site worked, but needed a facelift and new features that we couldn’t add, because every time we tried, it would break another part of the site. At that time, our customers were essentially serving as our QA. (next) The application also had some complex business logic, like many businesses do, that dictated reasons for certain customer-facing functionality. These were not documented anywhere either, they basically just existed in the heads of the people who had worked there longest.
• #7: We knew something had to change. We identified our major goals. Our business goals were: (next) Add missing functionality fill in the gaps in the admin panel, automate tasks that were being done manually by IT. That would give us the ability to meet the next goal: (next) Add new features without breaking existing ones. Speaks for itself. (next) Avoid Downtime historically the site had a bit of a stability problem in general, but especially when we would try to do releases. A code update could take the site down for a bit, and that had to stop.
• #8: So, these informed our technical goals: Maintainable code. Who here has ever written a comment or commit message that contained the words “I’m sorry” or “not sure why this works” or “don’t touch this or it will break” Who’s ever being trying to fix a bug, and thought “who wrote this garbage?” and then who’s ever done a git blame on that code and seen your own name? When we write code that is not easy to understand or edit, we’re creating roadblocks for ourselves and our successors. It causes wasted time, frustration, and bugs. We wanted to be able to easily change the code without breaking other code, and without cursing each other’s names. So this means readable code with clear names as well as cleanly separated components to reduce and make obvious our dependencies. Quality Code: By quality code I’m talking about writing code that not only works, but works well. It doesn’t have to be micro-optimized, but it should at least be macro-optimized. This means things like not writing SQL queries in loops. Not using deprecated or insecure functions. Performant: runs fast enough can handle large sets of data, not just the limited data a developer might test with Documentation We set a goal to document our code not by adding comments. if we are meeting goal #1 of writing maintainable code, any of our developers should be able to look at the code and understand what it’s doing So, we comment WHY the code does that, because that's what might not be obvious now. Rapid Development We had a very short timeline to get everything done that we needed to. This means not spending time writing code that we could easily use a third party library for. Easy Deployment We wanted to have a one-button deployment process that would not cause downtime during the release. 0 Regressions Per Release We wanted to ensure we didn’t break anything that was documented, so we needed to set up more automated testing and add it to the release process.
• #9: Typically it's not a good plan of action to just throw away your existing code base - however, we basically just needed a CRUD - that's Create, Read, Update, Delete - for our database records. What we had didn't really work, so this was a good candidate for starting over.
• #11: Version Control - it doesn't matter which you use: Git, SVN - just use one. I think Git has proven to be the forerunner, but if you have a good reason to use a different service that's fine. Version control offers you a way to easily track and record each change you make to the software It also allows you to easily revert changes if something breaks - you don't have to drop everything and start trying to fix the bug in production, you can just revert to the previous stable version, and take your time calmly fixing the bug locally - and be sure to add tests for the bug! One of the best benefits of version control is branching - this allows developers to each be working on as many tasks as they need without interfering with each others code or causing confusion Version control really works best when you make an effort to make small changes and pull requests, and keep master updated continuously. I used to be a really big fan of "gitflow", which is a development workflow for managing various feature branches, release branches and hotfix branches. However, after using it for several years and then switching to a workflow focused on keeping short branches and small changesets, I've found the latter to be much more effective in the long term.
• #13: When you're working with legacy code it can be very hard to add unit tests. If the code is not testable, how do you know you're not going to break something when you refactor it to make it testable? We handled this by writing Behavior tests in Gerkhin. These human readable automated test files serve as documentation that explain in plain English how your application works and why. What this means is we can write out descriptions of how the site works and then use an automated testing tool to test that the site actually works that way. Once we have tests in place that ensure the user experience doesn't break, we can begin to refactor the legacy code, and we can add unit tests, to ensure our new smaller, modernized components don't break.
• #17: While writing unit tests for legacy code can be very tricky, writing behavior tests for existing code is not that difficult. The typical BDD and TDD workflow involves writing tests first, having them fail, then writing the code to pass the test. When we have code already written, our process will be a little different. We'll start by writing the feature file, as usual. Then we'll run Behat and see if our test passes. If it does, great. It's a good idea now to check that the steps do fail if the code is broken. So I like to double check that the test works by going through each step and changing something small in the code to break it - just commenting out or changing an important line of the relevant code is usually enough. Then you can revert that change and have a passing test that you know will catch regressions. If the test fails, then either we described the feature incorrectly, made a mistake in the test code, or the feature is actually broken. So, double check these three things.
• #18: You'll also want to have a code style guide. Having the debate of tabs vs spaces, or brackets on the same line or new line, over and over again because there is no standard in your office not only slows things down, it also makes clutter in your commits. While there are some ways to filter out whitespace from your diffs, you can't filter out everything. By choosing a standard and setting up our IDE to automatically apply the style to our code upon save, we were able to stop wasting time arguing about things that didn't matter, and easily see the real changes in the code. It also helps prevent animosity on the team over what are honestly stupid arguments. When we first started working on the project, another developer and myself kept differing on the spelling of a certain word, convinced we were each right, and changing each other's code to reflect the spelling we preferred. It was so annoying to see my spelling changed, and I would always go change it back, and then the other developer would change it back too. After a few times of this happening, we realized the spelling of one word was the least important part of the code changes, and worked around it.By the way, we were both right. The word had two acceptable spellings. We originally had a sort of custom one just based off of what we liked, but we eventually switched to PSR-2. The point is not to pick the "right" standard, but just to have one.
• #19: Quality is a lot more important than style, which is why I've called this "Rules" rather than "guidelines". These rules exist to make sure that your new code isn't becoming "legacy" code as you write it - meaning that you're not writing new code that is too coupled with other code to be tested, or has too high N-path complexity or cyclomatic complexity, or any number of other rules that the community commonly agrees on as code smells. So, create a set of specific rules about code quality. It can't be vague like "low n-path", it should be a specific measurement. N-Path of 200 or less. There are a lot of different ways you can get these measurements on your projects. There are of continuous integration services that allow you to calculate these numbers, and there are others that calculate a score based on their own metrics. The biggest problem I've had with all of them is actually two problems: it's usually hard to calculate the measurements only for new code, not the entire project secondly, the feedback happens too late. We set up scrutinizer ci, which I really love. Whenever we push a branch, it does a build and calculates a bunch of metrics. It then hooks into our hipchat, and announces the developer, the branch, and how many "issues" they caused or fixed. This helps because it does only count new issues they caused in that announcement - but there's still a timing problem. By the time they get this feedback, they've already moved on to the next task, and are focused on something else. It's really hard to enforce having developers fix these problems, even when we can all agree they are important.
• #20: So we added PHP Mess Detector to our project. This tool can be run locally, on specific files, so our developers can get instant feedback. So we have our rules for php mess detector for our project, and every developer should have PHPMD. But how do we make sure they remember to run the tool, and fix the code? We added a git hook, which provides immediate feedback because it won't let us commit code that violates these rules. We also added a git pre-commit hook that runs mess detector before commit, and blocks the commit if the rules are broken. Obviously, this hook can be bypassed if necessary, however with the other processes in place it shouldn't be necessary
• #21: We set up a process for code review. Again, this is something that there are numerous tools that you can use for automation and enforcement, but the key is doing it, every time. Depending on the size of your team, you can decide you need a certain number of approvals before merging. CR works best on small PRs - the more code there is to review at once, the harder it will be. When doing code review, you're not concerned with things like formatting and style - camelcase vs snake case or indenting. You're looking for long term maintainability and design quality. Is this code readable - do you as the reviewer understand what this code does and why, without having to ask? Code Comments should explain *why* something is happening, not *what* is happening. Issues that you can't catch via automated tools.
• #22: We found it extremely helpful to put an emphasis on continuing education for ourselves as engineers.
• #23: Is your application: a database abstraction layer? An email sending tool? A PDF Generator? Or is it an application that serves some business purpose that happens to also need to use these things? Don't grab the first library you find. Research the options and determine which one to use based on important criteria: At Grovo, we also undertook a similar project to rewrite the legacy version of our software - and in the new version we decided to use UUIDs - universally unique identifiers. So for the PHP parts of the codebase, we pulled in the most popular PHP UUID library, written and maintained by Ben Ramsey. A few weeks into my work at Grovo, I was tweeting about test quality and coverage, and Ben contacted me asking if I could help improve the coverage and quality of the UUID library tests. I'd have been happy to do this even if we weren't using it at Grovo, but even more so since we were. I spent a few days reviewing the tests, adding coverage to some of the hard-to-test areas, and improved the quality and maintainability of some tests. So the library went from 89% coverage to 93%, and is now around 96%, but more importantly, a lot of the tests were much stronger and would be easier to debug when they fail. Don't forget to contribute back!
• #24: As I mentioned earlier, our first focus was re-writing the admin panel since it just didn't work. Most admin actions were based around the CRUD actions, so it was really easy for us to pull in a third party library for this. We used this admin panel generator called Sonata, which is a bundle for the Symfony framework. Sonata also offered a lot of extra functionality beyond just CRUD such as custom blocks that you can use to display important information.
• #25: With our admin panel written and functioning, our next step was to start working on our API.While we did have some clients asking for an API, we also needed to write the API for internal use, so that we could have one place for the business logic for both the customer facing site and our admin panel. Having the API would also enable us to iteratively refactor the frontend site without needing to rewrite all of it at once. Meaning as we touched parts of the old framework’s functionality to add new features, we wrote those features into the API so they could be fully tested, We then removed the code in the original controllers and replaced it with API calls. Our behavior tests helped ensure the frontend didn’t lose any functionality.
• #26: The API is an area where we did really mess up. Remember when I talked about not writing code that doesn't have to do with your business logic? The business logic of your product is the one thing that makes your product unique. It's the thing people actually care about. It's the thing that is going to make your company money! Is anyone familiar with a company called Salesforce? Salesforce is a "Customer Relationship Management" platform - a tool for managing customer information for integrating sales, marketing and customer support. How much of Salesforce's revenue do you think comes from their APIs?
• #27: 50 % I'm going to assume we're all familiar with eBay
• #28: Salesforce reportedly generates 50 percent of its revenues through APIs, eBay nearly 60 percent and Expedia a whopping 90 percent.
• #29: Salesforce reportedly generates 50 percent of its revenues through APIs, eBay nearly 60 percent and Expedia a whopping 90 percent.
• #30: Your API is the most important gateway to the combination of the business logic that powers your unique software, and the non business specific actions and tools like sending email and accessing data. Once you write the APIs, you can build multiple clients on top of it - responsive web frontend, native mobile apps, third parties can consume it. So make your API a first class citizen
• #32: So, as I said we started this project back in May of 2013, and I'm happy to announce that we did meet our business goals. New Symfony based Admin was launched after about 9 months (thanks developers!) Had a few bugs, took about 3 more months to be considered stable Over past year, close to 0 regressions (Thanks Behat!) Very limited downtime (Thanks Amazon, Elastic Beanstalk, Aurora!)
• #33: We also met our technical goals for the most part. Maintainable Code: While we started off a little rocky, we have gotten much better about writing code that doesn't prompt anyone to curse or yell "what is this doing?" We do still have that code from the original application, or from our first few months while we were learning, but because of our quality tools, we are constantly modernizing all of it. Quality Code: The site is less buggy and is faster. Even as our IT team has tripled in size, our code remains high quality thanks to the tools we've put in place - and a good hiring process. Documentation: We have a huge library of behat tests that document the site, as well as good comments in the code. We engage in peer code review sessions often and encourage each other to write good comments and good code. Our behat files are also available through an open source tool I wrote that takes the .feature files and turns them into an interactive html site for our admin users to use as a site manual Rapid Development: We add new features so often that we typically do 1-2 new releases a week. We have to plan several weeks ahead with management and business users to keep up with the speed at which our developers are churning out new quality code Easy Deployment: I actually tweeted the other day about how awesome our deployment process it, I hit the deploy command and then remembered I had left my breakfast in the office kitchen, went off to go heat it up, started a conversation with a coworker, and never once had to think about that release. It all just worked. Zero Regressions Per Release: We very rarely have regressions or new bugs, but when we do, it's much easier to fix them now than it ever was.