Lessons and tips from writing my first open source terraform module.pptx

Editor's Notes

• #2: Hi, I’m Anthony Scata and I’m going to talk about some of my experience, lessons, coding tips and tricks while on my journey to write my first terraform open source module. The talk will consist of some technical and non technical aspects and all going well show some code as well but no live demos.
• #5: As a good consultant i cannot start a presentation without talking about where I work
• #10: These are some of the more softer lessons i learnt whey building the module that i think are worth mentioning before we move into some code and detail. It is open these softer skills that are less appreciated and talked about but have just as much, if not more of an impact on your work.
• #11: First thing is easy to say but if you are thinking about it, just do it Even though as a consultant I review terraform code daily, write code, take open source modules and modify them, I still felt afraid. Felt like there was a stigma or a really high bar for getting something into the public. That if it wasn’t good enough it would reflect poorly on me. Also a perception that people that write these public modules are the rockstars of our industry, knowing more or better than the rest of us. But this isn't true, they are just like us, trying to solve problems and making that available for us. I am not trying to downplay their role or expertise. I feel like it is normal for most of us but it doesn’t need to be, people are more likely to be grateful for the effort and what you do rather than pointing fingers so there really is no need to be afraid.
• #12: That leads into the next issue of over doing it, as engineers we tend to over engineer the solution, making the swiss army knife of modules, looking for every corner case. Again this probably reflects into what i mentioned before, part procrastination, part worry, part perfectionist which can lead to getting nothing done. I won’t lie, this was me, working and working and never completing it till i got to the point where i said, enough is enough and its better to be out in the open, for people to use and benefit from.And once it is out there the benefit of a great open source community is that people will help with these edge cases. PRs, raise issues, forked repositories. There will also be cases you cannot anticipate so let others find them and help make the necessary changes. Be comfortable with the fact there will be issues, there will be errors and this isn’t a bad thing, look at any project or piece of software, so don’t try and make your solutions everything to everyone otherwise they may need up being nothing to nobody.
• #13: It can be different to build a module or solution for an wider open source community rather than internally, but it really shouldn’t be. Often we are able to avoid some of the more difficult aspects of modules like secrets, hard coding or making use of less secure methods that are not acceptable or usable by others. I like to keep this in mind when writing any code, how could this be used by others, at a different client or in the public domain. Scenarios like this may include, use secrets manager for pulling secrets, ensuring secrets are not stored within the state, reference the object not the secret itself. And this rolls into documentation, add enough so that users know what it does, why and how it does this and how to seed the environment if necessary to make it work. You want that initial experience to be easy, streamlined, not a guessing game that puts consumers on the backfoot and more likely to misuse or not use the modules.
• #14: All this being said it is likely to take longer than you think, which goes with the point aboves But don’t let that discourage you. A lot of the time we think issues or problems in software are limited to our use cases but in reality this is the exception not the rule. Its more likely that somebody else has a similar issue to you, looking for a solution who would be grateful for your contribution so why not showcase your work, your solution, so other can use it an be inspired, so they don’t need to reinvent the wheel.
• #15: A team member at digio posted this article in our slack a weeks ago from GitHub that brings in so many good points. Sure I wrote a module but i also comment back to issues, i raise issues when errors occur in opensource github project. These are good ways to get started on an open source journey, build a bit of confidence and see how others do it. Its great experience for when its your project and generally help others out.
• #16: Now some of the more technical tips and tricks that i learnt along the way. Most of these make use of newer terraform concepts and features which are making it easier to write reusable code
• #17: This was a technique i picked up from another module and is possible after a few recent changes to Terraform. From 0.14 as an experimental feature and added in 1.3.0 the use of optional variables has made it easy for modules that have a large number of input variables to define them within an object. This helps to group them all together, rather than defining all 30+ variable blocks, we can compress these into at least half the size. Without the optional keyword in maps or objects users need to pass in all of the values which makes their code larger than it needs to be and possibly more complex. The less the end consumer needs to define the better. You can see how we make use of the variable, as an object. The side effect is you don’t have as many descriptions to explain every variable but you can use but these can be helped with variable validation and HEREdocs with multi line.
• #18: As of 0.13.0 you have the ability for variable validation. To check the contents of a variable for example is within a certain number range, or matches a regex or is a valid json string. The idea being that sometimes a plan does not catch these incompatibilities due to the provider, we only find them when its running the apply which is likely too late. Lets do this before the plan to ensure we have a consistent and working environment. One advantage of the variable map and optionals as mentioned before is that we can check multiple variable values, for example the min is less than the max and the desired in somewhere in between. If the variables are defined separately this cannot be done
• #19: This may seem minor but it really helps others who are viewing or changing your code. Some resources may use 10 or 20+ attributes and it may be hard to comprehend what is being used. Sorting the attributes alphabetically makes it easy for others to look and see where its places and then how its used. Reducing the cognitive load of making change and decisions, where does this go, should i put this here helps. This includes the resources themselves. Although Terraform does not run in a sequential order it helps for us humans to again comprehend change and find resources.
• #20: We have all seen code with hundreds or thousands of lines and though, oh god, not this file again. This adds extra stress and cognitive load to changes. You are much better off splitting the files for a higher level resource type, possible autoscaling, cloudwatch and then add a locals specific to that set of resources into the file. Keeping the resources somewhat contained helps to facilitate change. This may sound contradictory to before in terms of logic ordering and it does depend on how many resources you are creating but anything more than 10 resources per file starts to ger unwieldy.
• #21: Null resource check Rebuilding the zip for lambda
• #22: You can make use of inline templatefile or file function instead of the need to create a data resource and use the rendered output to the necessary resource. I find this easier to view and use. Less clicking and looking around back and forwards. Also here we can use a template within a template if necessary
• #23: I want my code to be well documented and for those interested in it to look if necessary, key word being necessary. Terraform docs provides the ability to automatically generate resource, variable, input, provider and other docs based on the code. This means less looking at the code if you are new to the module and provides a better snapshot. Now I can see if I this works with the aws provider version i need for another module, does it use a resource type that my organisation does not allow but more importantly which input variables I need to supply, why and how. With the validation mentioned earlier and the docs a consumer shouldn’t need to view the code to see how a variable will be used making it easier to use for less experience engineers.
• #24: The use of locals makes it easier to reuse strings or data without having to hard code it in multiple places. Sometimes people make these variables as defaults which can be messy as it gives consumers the ability to change them. Utilise locals where possible to reduce duplication of magic strings, once, twice, three times extra into a local. Locals can also be used to remove the complexity of how a variable is computed into a separate file. Resource definitions can be large enough, let alone when you add in a join, compact, concat, split, tostring, try. Move this out, make it a local and reference it when needed.
• #25: Try and infer variables where possible rather than having the consumer pass them in. For example the caller account, no need to pass in an account id, we are deploying into this account, just grab the ID with a data source, same with region. This reduces the duplication of variables and the possibility where the consumer changes region and forgets to update the variable.
• #26: As with anything, architecture diagram or documentation as a whole is important. Nothing says this is a well maintained piece of code like documentation which is factual and thorough. Diagrams can really help pain a picture of what will be deployed. Again, why have consumers extra data out by looking at code when they can see it from a high level. Coupled with examples of working code makes it easier for people to try. You want to lower the barrier to entry for any piece of software and you may need them yourself as they provide a good guide. And lastly, example why in the documentation decisions were made. At times we focus on what was done but not the motivation or limitation as to why. The how can mostly be seen from the code, we can reason about this, the why is more abstract and less obvious. We use this cloudwatch setting because, this is set to negative value so that. This helps the next person out who thinks, why was this done, let me change it to something else that makes more sense to me, only to find themselves in the same situation and rabbit hole you did. Be kind to your future self and engineers.
• #28: Lastly, a small note that I am growing my moustache to help raise awareness for mens mental health, suicide prevention and raising money for prostate and testicular cancer research.
• #29: Thanks you for listening to my presentation and hope that you gain something useful for your Terraform and Infrastructure as Code journey.