Dgeni documentation generator

Editor's Notes

• #2: Hello ngEurope. It is a real pleasure to be speaking here today.
• #3: My name is Pete Bacon Darwin. I am the London annexe of the core AngularJS team for about the last year or so. Sometimes known as that guy with a funny surname! There is probably a whole talk in itself about how to work with the Angular team from across the Atlantic. But today I am going to stick to talking about Dgeni.
• #4: Here are the highlights of this talk. By the end I hope you might be inspired enough to go try using dgeni yourself. But first some questions for you. I'd like to start today with a bit of a quiz. You might want to get out your laptops or tablets or phones…
• #5: Let's start with AngularJS: Who can tell me the syntax of the method on the $sce service I need to use if I want to declare a piece of HTML to be trusted so that I can bind to it safely in a template.
• #6: https://docs.angularjs.org/api/ng/service/$sce#trustAsHtml
• #7: OK so AngularJS is easy, eh? That is why we are here after all. How about a Protractor question? How do I access the second row of this repeater?
• #8: http://angular.github.io/protractor/#/api?view=ProtractorBy.prototype.repeater
• #9: Finally, for those mobile developers out there. If I am using the Ionic Framework what is the name of the directive used to show a pull down widget that is used to refresh the currently displayed content.
• #10: How did we answer those questions? Did you get out a printed book on the topic? - I know of a good AngularJS one written by my friend Pawel Kozlowski! I bet some of you may have taken a look at the source code Or perhaps you just knew all the methods and components for all three projects off the top of your head? Probably the majority of you looked at their documentation websites, right?
• #11: I think we can all agree that online documentation is a good thing. Not only does it offer a place for new users to get up to speed with a project but for many users of a project it can be the focal point of their development process. I know that even after working exclusively on AngularJS for over two years I still rely on the AngularJS docs for one thing or another almost every day. And it seems that so do you. According to our analytics the AngularJS websites get almost 3 million hits per month!
• #12: But documentation is a bit like bread! Unless you are really desperate, no bread is better than stale bread! This is a case where you definitely don't want to be dogfooding your docs The key to having fresh docs is finding a way to ensure that the docs get updated when the code gets updated.
• #13: So how do we manage this in AngularJS? What we do is keep the docs as close to the code as possible and integrate the process of updating the docs into the code development process.
• #14: We took a tip from the JSDoc toolset which encourages us to store our docs inside tagged comments in the code itself. This means that when reviewing code changes it is easy to see whether the docs have been updated correctly too. It also means that the docs get version controlled in exactly the same way as the code. This makes it much easier to create correctly versioned sets of docs. But how do we convert these tagged comments into a usable website?
• #15: JSDoc, as a tool, is very generic and does a pretty good job of guessing how your code fits together - a mighty task given the dynamic nature of JS. But for the AngularJS project, we have a number of specific concepts such as directives filters services providers These are not well catered for by vanilla JSDOC tags. So the Angular team built their own node.js tool, known as ngdocs. Unfortunately as the complexity of our docs application grew so did the ngdocs codebase. In over 1800 LOC the main monolithic ngdocs.js file was over 1200 LOC! At the end of 2013, I was trying to remove colons from filenames. This would enable AngularJS to be built on Windows. But this behaviour was spread out throughout the ngdocs code and it became apparent that a major refactoring was needed.
• #16: The result of that refactoring is now the project called dgeni. Dgeni is a versatile little tool that can be configured to do just about any kind of documentation generation you can imagine. From the start I set out to apply good design principles; many of which are similar to those that make AngularJS such a joy to work with. Dgeni is built from small, cohesive, decoupled components, which can be swapped in and out, and configured for each documentation scenario. It makes extensive use of dependency injection The code base is comprehensively tested - the main tool itself and most of the fundamental packages have 100% coverage.
• #17: Dgeni as a reusable tool is still pretty immature but there are some significant project using it to generate their docs. Here are four of the most well known What is fascinating is that each project has very different requirements and documentation generation strategies but they all use dgeni and the amount of code required to configure them is quite small.
• #18: Angularjs - generates a partial HTML file for each documentation page. These partials are loaded into an AngularJS application via ngInclude
• #19: Protractor - generates a single JSON object containing all the API docs. This data is filtered and displayed in an AngularJS application by a Controller.They also incorporate docs that have to be parsed from the webdriver project which they don't control.
• #20: Angular Material - generates mini Angular apps for each widget. These are displayed in iframes in a yet another AngularJS container application
• #21: Ionic Framework - have a very different strategy It generates markdown files that are processed and served up by Jekyll So we have seen that dgeni can be used in many different scenarios but how does it work?
• #22: There are a very small number of concepts in dgeni. If you have spent some time with Angular then you will find them quite familiar Let's take a look at them...
• #23: In dgeni the fundamental idea is that documents are generated by a series of processors. Each processor takes a collection of docs objects and returns a collection of doc objects. The processors are run one after the other, with each processor having a chance to add to, remove from or modify the docs collection. In the most basic dgeni configuration you would probably have processors that would: read in some source files, creating a doc for each JSDoc comment found extract the JSDoc tags from each doc assigning their value to properties on the doc apply some template to the doc to render some output content for the doc write a file for each doc containing the rendered content Processors can be configured before they are run with information specific to your project. For instance, which source files to read, what tags to extract and so on.
• #24: Throughout the system, dgeni uses Vojta's dependency injection library to wire up all our components. If you understand dependency injection in AngularJS then this will be very familiar. We can register processors and services as injectable components but also, similar to modules in AngularJS, we can register config blocks that can be injected with components. This allows us to configure those components before the processing begins. We will look at processors and services in a moment but first we need to understand packages.
• #25: Packages in dgeni are similar to AngularJS modules. They are containers for services, components and config blocks. You can see from this code snippet that creating a package and registering components is fairly straightforward. We are defining a docPackage package and registering a service, a processor and then configuring the templateFinder service. Since dgeni is a node.js application we can simply require the node module that contains the component, which keeps your project files tidy. Notice that we can specify a package dependency, in this case on the jsdoc package. This means that this package will have access to all the components registered in that package. This is an important part of dgeni which enables us to reuse and customize other people's packages, saving us time.
• #26: Services, like in AngularJS, are singleton objects that can be injected into other components. Just like in AngularJS services that are dependency injected can be overridden which makes it much easier to customize project setup and to test components in isolation. You provide a factory function that returns the actual service. If the factory function has a name dgeni will use this for the name of the service. The factory function is injectable so dgeni would try to fill any parameters that the factory function declares from it injector. Since we are going to use this in a package by requiring it as a node module we export the factory function in the module.exports property.
• #27: Processors, as we have seen, are the workhorse of dgeni. In fact without processors dgeni does nothing at all. The dgeni tool itself doesn't have any processors at all! The factory function is injectable and name of the factory function is used for the name of the processor. We return the actual processor object from the factory function. This is somewhat analogous to how we might define a directive in AngularJS. In this example we are defining a filterNgDocsProcessor, which is used in the AngularJS docs to filter out JSDoc comments that do not contain the @ngdoc tag. You can see we are asking to have the log service injected. Processor objects must have a $process method, which takes the previous docs collection and returns the new docs. Processor objects can also contain additional optional properties, such as $runBefore and $runAfter, which specify where in the pipeline this processor should appear.
• #28: So putting all of this together: We define a package, possibly depending upon other packages; register components and config blocks on the package; create an instance of dgeni, passing in the packages to load; then call generate(). If a processor returns a promise to a set of docs rather than the docs themselves then the pipeline becomes asynchronous. Because of this the call to generate also returns a promise, to which we can attach handlers.
• #29: As I said earlier dgeni itself doesn't provide any processors. On its own it does nothing at all. We need to define processors in packages to get dgeni to do useful work. There are a lot of common tasks that need to appear in most document generation projects. Many of these are provided for in the dgeni-packages project. The main packages in dgeni-packages are shown here. base : contains things like reading and writing files, abstract document rendering and computing ids and paths jsdoc : depends upon the base package and adds the ability to parse and extract information from jsdoc tags nunjucks : implements document rendering built upon Nunjucks, a JavaScript rendering engine. ngdoc : depends upon both the jsdoc and nunjucks packages and adds various processing specific to AngularJS code. examples : provides support for the runnable examples that you can see in the AngularJS docs. Of course you are free to depend upon, use and configure any of these packages in your own projects or ignore them and write your own processors instead. The packages in dgeni-packages were originally developed to support the angular.js project rather than angular apps in general. Today I would like to show you a work in progress; a prototype of the basis for a turn-key documentation package for angularjs applications.
• #30: The new package is called angularjs and builds upon the base, jsdoc and nunjucks packages. Let's look at the angularjs package... Now let's have a look at documenting an example app with this package: You can see that we have a src folder that contains the application. I have marked up the AngularJS modules, controllers, directives and so on with minimal doc comments. Now let's look at the dgeni setup: We create a package for this project called 'dgeni-example', which depends upon angularjs, jsdoc and nunjucks. We configure the readFilesProcessor and writeFilesProcessor with where to get and store the files. We configure the templateFinder with where and how to find the Nunjucks templates and fix up the templateEngine binding to prevent them conflicting with Angular. There is a gulp task to generate the docs, when we run it you can see it running through each of the processors. The output files are written to the build folder. In there we can see that there are now a bunch of HTML files. The layout is pretty raw right now but you can see that the basic contents of what is found in the code is there. The most interesting part of this process is the extractAngularModulesProcessor which actually parses the code of your application and makes sensible guesses about the modules and components. This means that you comments can be pretty minimal. If we look again at the
• #31: So basically what we have is a set of processors that create docs for each module and component in an angular app. With these objects in place the world is your oyster: You could provide your own set of templates You could add in additional processors to compute other properties or docs: perhaps you could generate a table of contents doc. Many docs apps that we have seen use a container application in which the docs are loaded, via partials or JSON for instance. You could build your own container app that consumed these docs.
• #32: So here is a challenge for today. Try forking this dgeni-angular project and develop a more cool set of templates I'll merge the best sets of templates into this project with full credits to the author - and I'll even buy them a drink tonight! Grab me around the conference if you want to have a chat about how to do this.
• #33: I hope that this talk has given you something to think about when it comes to documenting your projects. Do go away and try installing dgeni and dgeni-packages. Have a play with documenting your projects and don't hesitate to get in touch with me if you need help or have ideas.
• #34: Finally I'd like to say a big thank you to all the people that have so far contributed to dgeni. In particular Stephane Reynard who has tirelessly translated all the dgeni documentation files into French.
• #36: It is worth taking a quick look at the nunjucks package, which implements a rendering engine built with Nunjucks. This processor will combine Nunjucks templates with each document to generate some output content, which will be stored in the renderedContent property on the document. There are some clever services, such as the templateFinder, which are able to compute which template to use for a given document based on the properties in the document. Also notice that since the binding syntax for Nunjucks conflicts with the AngularJS binding syntax (double curlies) the ngdocs package reconfigures the nunjucks engine to use {$ $} instead.