Building JavaScript

Editor's Notes

• #2: Influence comes from conversations with lead web architects in our local area: Brad Rodgers – Lead developer over Adobe Analytics front end management portal Dave Smith – HireVue Engineering Director Nick Seegmiller – Lead developer over mobile and web applications at Vivent Rich Snapp – Needle Developer ADP Dealer Services (actually not in our area) Ryan Day – AnglarJs contractor
• #3: Elevator Speech Mimic Production: Developers are able to identify concatenation and minification issues before code is checked in. The build config is shared amongst all environments Prepares us for AngularJs 2 and ES6 Released, versioned library packages: Treat our shared libriaries like 3rd parties On check-in, build, package and version shared libraries deploying to local package repository like git. This speeds up the build process for the Spa and also removes versioning within TFS. Part of the build process can also incorporate the automation of generating documentation and demos. See how AngularStrap and UI-Boostrap and achieving this from their source code. Utilize bower package manager to keep dependencies current from a local artifact repository: DevOps team is working on a local artifact repository that will house third party libraries and in-house libraries. Reduce Spa loading time: Use a module system and an intelligent dependency map bundling tool, like webpack. Making our Web Application Server Agnostic and run on Akamai NetStorage: Instead of loading 100+ servers with our web application files, we can upload them to one location Avoids the second hop to our F5 and web farm in our current architecture Brings us closer to a true micro services SOA architecture; separating the front end from the middle tier. Allows us to use redirection only for api calls. Don’t have to perform a browser redirect during login.
• #4: Akamai NetStorage is a managed product that provides petabytes of storage space in our data centers all over the world. Automatic content replication and enhanced delivery performance. Customers upload content to the closest NetStorage data center using any of the supported methods: FTP, SFTP, FTPS, SCP, RSYNC, RSYNC over SSH, WGET, NetStorage HTTP API, File Manager (web UI available on the Akamai Luna Control Center), Aspera Upload Acceleration, CMS shell, etc. NetStorage automatically replicates content to the other geographically distributed data centers configured to host that content. Content is delivered to end users worldwide with the best performance thanks to the Global Akamai Intelligent Platform http://www.akamai.com/html/solutions/netstorage.html http://www.akamai.com/dl/product_briefs/product-brief-netstorage.pdf
• #5: Some of these concepts can be applied to our old PM and EHR but don’t let their current structure effect or restrict what can be done in the new PM front end web application. AngularJS 2.0 has just moved from Alpha to Developer Preview (04/05/2015)You can also use a pollyfill to run AngularJs 2 code if you want to write it in ES5. Some of the items I am going to review have already been addressed with the UX Steering Committee. The rest have been added to the UX Steering Committee discussion list and haven’t been addressed yet. One of the items that are currently in discussion is weather we should use Gulp, Grunt or Npm. The current verdict is Gulp. This slide deck is not intended to determine which task runner we should be using, but rather how the task runner should be utilized. Visual Studios 2015 will be replacing most of Web Essentials functionality with Gulp and Grunt integration. Visual Studios 2013 Web Essentials already supports Gulp and Grunt integration.
• #6: Lets take a look at our current developer processes and deployment.
• #7: Aptimize did the job. It was a quick way to increase our performance without having to make much changes in our code base. But the JavaScript world has evolved and moved on.
• #8: In our current PM, we have 500+ spas. Each popup acts like its own Spa. Each downloads its one waxed.js and one waxed.css file. Even if the previous spa had most of the same code, it is downloaded again.
• #9: We have all seen the video Sheridan Richey did on Akamai and Aptimize. https://youtu.be/VLrGWpQd8yc I would like to re-evaluate the concept when it comes to baking cookies and the management and distribution of the ingredients. Yes we are baking the cookies at the client end using the browser, but I would argue that instead of using ingredients locally, we are delivering to the user the pre-packaged Nestle cookie mix. We just have to add water on the client end and we have bake the cookies. The problem with this scenario is the Nestle cookie mix has an expiration date. The chocolate chips within the mix can last for a year, but the eggs can only last for a couple weeks. When the eggs expire, we have to discard the entire bag and go back to the supermarket to get a new Nestle cookie mix bag. If the supermarket is out, because they have also expired, we have to wait for the manufacturer to mix up the ingredients again and package up the bags and ship a new crate to the market.
• #10: Lets now take what we have learned and take the next step.
• #11: Disclaimer: This slide deck is not intended to determine which task runner we will use. Webpack can be used as a task runner, but it can also be used it as a plugin within gulp, grunt, npm at this current state in time. The UX Steering Committee is leaning toward using gulp for its task runner, but hasn’t yet considered webpack as its bundler. To better set the context on how the front end world has evolved over the past 5 years, please watch the video below.  Peter talks about the pros and con of a SPA and the solutions Instagram implemented to mitigate that con. Talks about the pros and cons of a SPA: https://youtu.be/VkTCL6Nqm6Y?t=7m56s Why bundling everything into one file is not the best approach: https://youtu.be/VkTCL6Nqm6Y?t=9m39s Instagram solution - webpack: https://youtu.be/VkTCL6Nqm6Y?t=25m43s Webpack takes modules with dependencies and generates static assets representing those modules.http://webpack.github.io/docs/what-is-webpack.html Since Peter’s talk back in July 2014, Gulp and Grunt have integrated webpack plugins for the bundling process.  This allows you to use the task runner like Gulp or Grunt and still use the intelligent dependency mapping of webpack. A 2.5MB gzip bundled file is too big. It causes issues with mobile devices.
• #12: The WebClinical EHR currently is building AngularJS code into a packaged build. It is waxing its bundled files using Asp.Net’s built in versioning. David Neilson has already created a AngluarJs seed with integrated gulp tasks. This gulp task is similar to what Milos has implemented in his Benchmark AngularJs app. gulp-html2js: converts AngularJS templates to JavaScript Both of these examples are statically bundling the js and css into a vendor bundle, a shared bundle and a app bundle.
• #13: Each project contains these files and structures. Not everything is checked into TFS.
• #15: It is time that we allow the evolution of our dev. process when it comes to front end development. Note that evolution isn’t immediate…
• #16: gulp-angular-filesort: organizes your modules in the correct order gulp-inject: a javascript, stylesheet and webcomponent injection; i.e. inject file references into your index.html gulp-css-base64: transform all resources found in a CSS (those within a url( ... ) declaration) into base64-encoded data URI strings gulp-uncss: remove unused css selectors gulp-webpack: takes modules with dependencies and generates static assets representing those modules gulp-cachebust: generates checksums and renames references to files gulp-cache-bust: append a query string to your assets to bust that cache gulp-newer / gulp-changed: pass through newer source files only. This addresses the current workflow we use were we only patch out files that have changed. Granted we will be patching out bundles instead of individual files, but it does reduce the amount of files modified in production.
• #17: It would be very beneficial to catch minification, concatenation and other build hiccups prior to a developer checking in that code into TFS. Different gulp tasks can be defined with arguments. gulp buildruns all subtasks including minification, concatination, linting, unit tests, etc. gulp build –env=maindeploys the build with custom tasks specific to main; i.e. distribution folder location and revs the package.when deployed to production, also can disable certain debugging tools built into AngularJs. gulp devruns all tasks including gulp-watch or gulp-live-reload which rebuilds on any changes to files; could omit sourcemaps. gulp dev-debugruns only the critical tasks like injecting the <script> tags in the index.html; used for fast builds for iterative code changes. gulp dev –minify=falseomits the minification step
• #18: There is on gulpfile.js per project. The gulpfile.js is checked into TFS. It contains tasks that can be altered by parameters allow for environment specific operations. This gives the power back to the dev. to make sure that the code is built properly and unit tested in each environment. The build config is also used when building the library out to a local repository like Artifactory.
• #19: Why manually managing glofs and glogs is bad: Easy to break your app through manual dependency management. It's difficult to integrate tools like traceur or babel. Managing <script> tags manually is painful and error prone. See Dave Smith’s (HireVue lead developer) presentation on this very topic:http://slides.com/djsmith/how-we-made-a-mess-of-our-angular-app#/17/1 Gulp bundling and gulp-inject solve this problem gulp-webpack goes a step further and creates an intelligent dependency map using require() Implementing a watch providers other benefits: Automatically runs the build after a file is changed Runs karma unit test automatically Can alert the developer of lint issues
• #20: Utilize gulp-watch / gulp-livereload / BrowserSync in the developers workflow on his / her workstation. Takes over process of running Karma in a separate console and jshint. Replicate production build environment Forces running unit tests and linting because the developer can’t view the web page without running the build. Integrates with Visual Studios and Webstorm gulp-watch: https://www.npmjs.com/package/gulp-watch gulp-livereload: https://www.npmjs.com/package/gulp-livereload BrowserLink: http://vswebessentials.com/features/browserlink BrowserSync: http://www.browsersync.io
• #21: For demos, please get with Brady Clifford… These have not been checked in.
• #22: By omitting certain gulp processes, like minification, sourcemaps, etc., the build process will increase tremendously. Utilizing multiple Spas allows us to modularize the code base so we are only developing and running builds under feature based domains. Instead of a 1000+ files Spa, we can reduce that down to 100+. Each Spa would share packaged bundles but would also contain its own app bundles. If a developer wants to make changes and test multiple spas at once, they can utilize the gulp-environment and karma-environment plugins that allow for multi-spa development. Adobe Analytics has over 200+ files and 15+ routes in each of their Spas. They utilize a watch and have the build time down to 200 ms. Email from my friend, Brad Rodgers (lead developer for their front end web applications): Hey Brady,   We have a watch script that watches for any changes and then automatically re-compiles our JavaScript bundles. It’s actually surprisingly fast! Whenever I make a change a file is automatically uploaded to the server and then recompiles my concatenated files in about 100-200 milliseconds. It takes longer to switch to the browser and refresh then it does to recompile the file. We have implemented livereload and use it but made it so it's optional. Personally, I find it easier to just refresh on my own so that I don’t lose something I was working on before.    And just so that you know we have more than one SPA because we have a HUGE application with a ton of code. So we have tried to organize our SPA’s into logical building blocks that work together. But just know that our SPA’s are not small. Some have up to 10 routes within them and they still compile in about 200 milliseconds. I don’t think you have anything to worry about using gulp/npm/livereload. From my experience it’s amazing! They are still thinking about dotNet/Java land which takes a lot longer to recompile and refresh.    NOTE: Our watch script does not minify our files because that process takes a lot longer. We have another gulp process that minifies, internationalizes, and any other heavier process for our production environment. Also make sure you use gulp or node watch instead of grunt because grunt is super slow and not optimized.    Hope that helps, Brad
• #23: Disclaimer: The ng360 team has not yet finalized how we will be architecting the multi-spa concept. The second step in optimizing the build process is to separate your code out into domains, multiple Spas. This really moves us more toward the micro services architecture. We could start partitioning out these Spas by mimic the Com+ components in our middle tier. By isolating each feature into its own domain, this reduces the amount of code that needs to be built for each gulp build process; thus reducing the cycles. In the multi-spa modal, each spa depends on core binaries and assets. This would include: foo-ui styling foo-core services menu, tab and toolbar components master layout html partials 3rd party package libraries Building on top of that are additional packaged libraries that are downloaded when needed. An example is Claim Status in the Coding Spa. The js, css and html is not downloaded from the server until the user access that route. This is done using either CommonJs, RequireJs or webpack.
• #24: The last item to address to reduce our build times: Pre-building our shared JavaScript and css libraries will reduce the amount cycles for a task runner.
• #26: Version & Package in-house shared components Released, versioned library packages become statically housed and served into the build process from an artifact repository tool. Treat these shared libraries like a vender; 3rd party library. Vendor Folder: Each Spa has its on localized vendor folder. The vendor folder is not checked into TFS. A bower.json file contains a list of all needed dependencies for the project. The packages do not contain versions. Versions are controlled instead by the bower.json file. During build after a checking, task runner “compiles” the assets and revs it and uploads it to our local build repository (like Git or Artifactory). Developers do not reference the raw TFS code, but rather the local build repository using a package manager like Nuget, npm or bower. Eliminates the need to manage html template relativity (html2js) For larger components, give them their own package. Like foo-grid, etc.
• #27: Issue: managing long ../../../ relative paths Issue: karma.config Glof management nightmare Solution: all vendor libraries are downloaded to the local project using a package manager like Bower and Nuget. Issue: reference issues with shared html templates. Solution: since the library has already been built, html is already converted to JavaScript and injected into the directive code.
• #28: Get with Russell Webster and DevOps for more details on the implementation and usage of the local package repository. Note that it will be used not only for front end pacakges but also backend; i.e. NuGet packages. Main takeaways: Versioning is managed in the package.json for each project. The version is rev. for each checking, built and saved in the repository The developer will reference the local repository instead of TFS for shared and 3rd party libraries. Packaged libraries can contain multiple components or just a single component. Depends on its rev iteration and complexity. See UI-Bootstrap and AngularStrap source code for examples on building up a library package of n number of components. Issue: how do I test and QA a shared library component when it doesn’t have a place to call home?Solution: following the UI-Bootstrap and AnglarStrap solutions, demos are created for both documentation purposes and QA. Issue: what if I want to work on both the shared library component and a web application?Solution: you can build the shared library component locally on your box and temporarily point your web application to the local packaged. gulp-environment plugin has the ability to build and watch multiple projects. Issue: what if there is a bug in an older version. How do we fix that version and redeploy?Solution: from the rev. in TFS, a branch is made to fix the issue in the older version and it is than built, deployed to repository and the branch is than merged.
• #29: After implementing a task runner in our development process, we can start to take advantage of the more advanced tools. Disclaimer: evolution takes time
• #30: Our Spas will by nature have more binaries to download. We are taking 500+ Spas down to 10+. The code has to go somewhere…
• #32: For the one Spa, we have a full course meal to serve instead of just baking cookies. Instead of our web farm being a manufacture, it becomes a distribution center. The manufacturing happens before the web application is pushed to production. By intelligently bundling our packages, we can take more advantage of caching on the browser and only make trips to the supermarket when certain ingredients expires. By passing bundled ingredients, we can actual do more than just add water. By intelligently bundling, we can first serve the appetizer, reducing the time it takes to serve up the Spa. After the appetizer is served, comes the main course, downloading Bundle A only when it is needed. Finally, if you want dessert, you can request a cookie. If you don’t want dessert, Bundle B is not downloaded. This “Code Splitting” or lazy loading of binaries can be done using RequireJs, Webpack or other tools. I recommend webpack. See later slides for more details.
• #35: CSS and other assets are also part of the dependency graph.
• #36: By implementing a task runner and taking some steps short-term, we can ease into a module system where we bundle and lazy load our dependencies to reduce the unnecessary trips to the family market. We are already utilizing a module system with AngluarJS. Lets go a step further and bundle not in one giant file, but optimize our bundles. We can manly maintain a dependency graph using a Glof and Glog within our task runner, or we can adopt an automated dependency mapper like gulp-webpack.
• #37: gulp-webpack is used as a plugin within build process. It creates the dependency graph using the require() within each JavaScript file. It optimizes the bundles into modularized packages contain similar dependencies. It is ran with the other gulp tasks and plugins. You can also use webpack as a task runner. What is webpack: http://webpack.github.io/docs/what-is-webpack.html When implemented into the developers workflow, it is usually not ran as a gulp dev task; when the developer is developing. It is instead ran in environment builds and when the developer wants to make sure his changes will work in production but test it on his own box. Why webpack is awesome: https://github.com/petehunt/webpack-howto https://medium.com/@tomchentw/why-webpack-is-awesome-9691044b6b8e
• #38: For big web apps it’s not efficient to put all code into a single file, especially if some blocks of code are only required under some circumstances. webpack has a feature to split your codebase into “chunks” which are loaded on demand. Some other bundlers call them “layers”, “rollups”, or “fragments”. This feature is called “code splitting”. http://webpack.github.io/docs/code-splitting.html It’s an opt-in feature. You can define split points in your code base. webpack takes care of the dependencies, output files and runtime stuff. To clarify a common misunderstanding: Code Splitting is not just about extracting common code into a shared chunk. The more notable feature is that Code Splitting can be used to split code into an on demand loaded chunk. This can keep the initial download small and downloads code on demand when requested by the application. Another common library that allows for lazy loading of binaries is RequireJs. Richard Snap with Needle gave a presentation at the AngularJs meet up back in January demonstrating these concepts: http://www.slideshare.net/rsnapp/lazy-angular-w-webpack https://www.youtube.com/watch?v=U4LYo0SiJJw Other Resources: https://github.com/snapwich/lazy-angular-demo https://github.com/snapwich/lazy-angular https://github.com/ocombe/ocLazyLoad
• #39: Once we have shifted our build process for our web applications sooner in our development process, we can make our front-end server agnostic and decouple it from our middle tier servers. Disclaimer: evolution takes time
• #40: If our front-end can be server agnostic, it can be served by IIS, node.js, Apache and even Akamai NetStorage. Serving our static front end content from Akamai NetStorage would give us the following benefits: Take the 2 hope process of getting updated cached files from our web farm down to 1 hope. Akamai to the client. Take the 100+ IIS servers that host our web applications down to 1. Reducing issues with synchronization and deployment times. Decouples the front end from the middle teir allowing for a more natural micro services architecture Provides a door for Pops to move their middle tier com+ components to the cloud; AWS or Azure, leaving the SQL servers in our current data center. Redirection at login no longer has to take place in the browser; currently a headache in multiple levels. Instead, the redirection only occurs with Ajax calls back to processrequest.asp. The API reports back to the first call informing the client where to place all future requests.
• #42: Overall, I would like to see us first modify our development workflow / process and implement a gulp-watcher on our development machines. Second, improve our bundling by implementing gulp-webpack And finally, go server agnostic with our front-end web application with Akami NetStorage.
• #46: If we ever want to use our Akami NetStorage as a CDN for individual packages, we could utilize the gulp-cdn plugin that allows up to change <script> and <stylesheet> tags dependent upon environments.