Evolving an Application Architecture

Editor's Notes

• #3: First, I want to thank Coco for inviting me. Second, I want to ask her what she was thinking inviting me here. I don’t consider myself a JavaScript developer. In fact, I have spent most of my career in C and C++ land developing CAD tools for electronics. This is basically something like AutoDesk, but for designing electronics. I left that about 4 year ago, founded a company, got funded, then decided I valued sleep more than the company. So here I am now, relatively new to the web landscape. In fact, you might all know more about JavaScript than me. However, I can probably claim one title. So, lets do this by a show of hands. Who here was writing JavaScript in 2015? 2010? 2005? 2000? 1995? Well, I wrote my first JavaScript around 1997, when it was also known as dynamic HTML. Enough about me. If I’m not an expert JS developer, what am I going to talk to you about?
• #4: OpsStack – the web application we have been developing at ChinaNetCloud. I know none of you are here to listen to a sales pitch, and I promise you wont get that from me. My goal for this talk is convince you that application architecture should be suited the particular circumstances and not your personal preferences. I hope to convince you by sharing with you how the front end of our web application architecture has evolved over the last year, and most importantly, the decisions that influenced this evolution. But to do that, I think it helps a lot for you to have some basic knowledge about the company and application.
• #5: OpsStack is a product of ChinaNetCloud. It’s primary business has been to help companies that don’t have lots of expertise run their website by providing operations as a service. As an operations company, our customers regularly ask us to do work on their website, which means they create tickets. Long before I joined the ChinaNetCloud, it had a customer portal where customers could login, create tickets and view a history of work we had done on their website. About a week after joining the company, I was given the task to rewrite the website and have it done in a few weeks. This project eventually became known as OpsStack. That’s all I’m going to tell you about the product, and that brings me to my first lesson.
• #6: Your application architecture is determined by your most important requirements. I do know this is a JavaScript meetup, so what I’m about to say verges on blasphemy. It was already decided before I joined that we would do this re-write in PHP using the Laravel framework. Those were our first requirements. We wanted to show something fast to quickly validate – think fail fast. Although the project was new, it needed to connect to many legacy systems. If we weren’t going to be able to use Laravel or we were going to find fundamental issues where it wouldn’t work with the legacy systems, we wanted to know before we invested lots of time. On the front end, the only objective was to replace a website where customers would login, there was one page where they could create tickets, and everything else was simply viewing data from a database. There was no long term thoughts about what it might become. In addition, we had a small team of PHP developers, none of whom had ever used Laravel and as I found out later, had very little JavaScript experience. Taking all those things together, they pretty precisely defined what the website could and could not look like.
• #7: The answer was the simplest possible Laravel application. Starting from scratch, we were able to duplicate most of the functionality of the existing site in a couple weeks. That was good. There were few reasons for this. First, the team didn’t need to learn a new language. In fact, because the original site was also written in PHP, we could reuse pieces of code from it. This architecture was also very easy setup and test. Yes, setting up gulp, webpack or just plain node scripts is pretty easy, but 0 setup is even easier. If you want to make a change and see the effect, you simply reload the page. If you want to debug a problem, it is easy to run the entire code in a debugger or print out extra data to the browser. Deployment was also trivially easy – basically a git pull. As a site that simply displayed data, this architecture worked very well, and I don’t think there is anything inherently wrong with this design. In fact, it isn’t that different from many CMS sites. There were lots of fancy things we could use, and we thought about using a JS framework, but there was no clear problem that it would solve. If you are going to introduce something more complex, So we continued to expand functionality, but mostly stayed within the same model of displaying data from other systems. But then, something changed. We started getting requests for more interactive functionality.
• #8: In reality, this wasn’t entirely a surprise and there was no immediate cause for alarm. We simply added more jQuery and more jQuery plugins. This again worked reasonable well, but it also became increasingly clear that we were going to want something more interactive on the front end, but the architecture made it too easy to create a poor user experience and too much work to make a good user experience. In addition, I was pretty dissatisfied with our code re-use on the front end. We used all sorts of tricks to improve re-use on the font end, but there were limits to how far and easily this could go. I was also unhappy with overall cohesion, and the difficulty in writing automated tests. In fact, as far as I know, there are no tools that let you write style checks for Laravel’s blade templates. All combined, as the requirements continued to change, it became clear that we should at least consider a using a JS framework to resolve these issues. Where to start?
• #9: Lets do another survey. I’ve listed 6 common JS frameworks here. My question for you is – which framework do you think is the best one for my project? As the architect, I need to make a recommendation about which one I think is the best one for our project. If you can, ask around. But in the end, you are the person responsible for your project. Choosing the wrong technology gives no one else to blame but yourself. That decision shouldn’t be based on emotion. It needs to be data-based.
• #10: Right now, JavaScript is probably in the midst of the innovation product life cycle phase an incredible explosion of frameworks and platforms, but this will end and we will eventually see standardization on a few technologies. It is very easy to get caught up in hype about a new technology, only to find that in a few years, it has been abandoned and the community has moved one. As much as possible, you want to remove your emotion from evaluation. I looked at a number of different factors coming. Using Laravel for the backend was non-negotiable, which quickly eliminated a few choices such as Meteor. From there, I looked at a number of different attributes. Some of these might be a bit of a surprise – why would we care about LinkedIn hits. The answer is it tells you something about how easy it will be to hire developers with that experience. Similarly with the number of published books. Published books is a very good sign of adoption. We also looked at Angular’s breaking transition from version 1 to version 2 as risk. Python 3 was first released in 2008 and still many systems ship with only Python 2. Silverlight – even had a contractual agreement from Microsoft, until they changed their strategy Eventually, we selected to write our applications in React, which brings us to architecture number 2 (alpha).
• #11: Using React was going to require substantial changes to our application – not simply that we would write most of the front end in JavaScript. It was going to fundamentally change how we can deploy our application code. It meant we would need to actually build the application. If you are going to invest the time and effort to make that work, then you want to make sure that you are going to use it. To avoid that work until we had certainty that we were going to use React, our initial setup was to embed React and JSX directly in the PHP blade templates. This meant including the babel.js library on the page, but it allowed us to validate quickly with minimal investment. The answer was yes to React, but certainly no to using it in this form. Let’s review the scenario. We have a application that is being actively used and it needs to keep moving forward. You have also decided that you are going to make a major architectural change. How do you make this change?
• #12: There are a few ways you can approach this. You can create a separate branch and make your changes in that branch until have everything converted and then merge your code back. This is almost doomed to fail because your few developers are not going to be able to develop faster than the rest of the team. You can convince management to have the team drop everything and focus on the re-architecture. But you need management fully committed to the re-architecture or they will get impatient waiting to see something working and eventually declare the project a failure. Your goal is to make the smallest possible change so that you can get your code into production as soon as possible, without causing any regressions. This might mean a big if statement deep in your application code to switch between the two architectures. Normally, we try to avoid this kind of architecture, but in my opinion, this is preferable to the alternatives.
• #13: Which brings us to architecture version 2. We end up with a small piece that represents our new React code. Over time, that React piece grows in size while the old piece shrinks.
• #14: Perhaps the most important thing here is that we didn’t need any long refactor branch and minimal risk of introducing regressions because the application was in a working state as we were making the architectural change. This is truly a gradual and progressive change. Eventually we will get far enough along that we may decide to migrate all remaining content in a single step. But this will be far easier than if we did that initially because we will have built up the controls and library components we need to do that efficiently.
• #15: It was also about this time that I nearly had a mutiny on my hands. This is something you many need to prepare yourself for. In the middle of this re-architecture, the developers were afraid. React was a big change to how they previously had written code. It was in a language that was relatively new to them. And to top it all off, it was with ES6 + JSX syntax. Management was also worried – our velocity had dropped dramatically. Just before National Holiday this year, the team had an impromptu sit-down. The question on their minds was – should we be continuing with React. A majority on the team felt it was simply too hard. Others were interested in the opportunity to learn something new. Management had a similar sit-down, even proposing a date by which we would abandon the effort. It was pretty hard to tell the team to trust me that we would get through it. There was at least one night where I went home wondering if I had made the right choice. Remember how I talked about before about making decisions for the right choices. That helps a lot here. I am pretty confident that if we had taken another approach, one where we didn’t have a site that worked throughout the change or where we had a long period of branching, we probably would have abandoned the effort before is was completed. In the end, we have succeeded and are fully on react now.
• #16: Lets get back on track. And talk a look at our applications structure today. In fact, our current architecture looks a lot like the following. We have a series of applications, one per page. Given this history of the application, this shouldn’t be a surprise. Changing to React and a single page application in one stop would be a more ambitious change than simply converting the pages to React. This diagram is still a bit inaccurate because each React application is hosted within a wrapper Laravel template.
• #17: This avoided re-writing a bunch of code and we can also share the same menu system as the pages that have not yet been converted. This overall made it much faster to migrate to the new architecture because we didn’t have to touch those pieces.
• #18: Ok, it is audience participation time. I’ve told you about my philosophy and guidelines for decision making. And I’ve told you a bit about the project – that we don’t have lots of interaction, but growing. We don’t have a strong team of JS developers – but are learning. We have a strong need to produce things quickly. With that in mind, there are a few common design patterns in React. Can you guess which pattern represents the bulk of our application?
• #19: If you guessed we manage state by putting it in the top-level component, then you guessed correctly. We are currently consistently managing component state in a top level component that we call an app and it owns the state. There are exceptions, but I consider each of those to be a bug and we are slowly and progressively removing those so that our application follows this common pattern. As it turns out, I do have a bigger plan for our architecture. In fact, my long term vision is to enable us to become a single page application. Since we cannot stop what we are doing in to make that transition, the transition needs to be gradual. We will introduce this change gradually by implementing common error reporting over the next month and use that to start migrating the application to have a common global state. This will probably be in Redux, but that decision has yet to be made because we have not yet evaluated alternatives. And that change will eventually allow us to migrate towards a single page application.