Is Technical Debt the right metaphor for Continuous Update - AllDayDevOps 2022
Download as PPTX, PDF0 likes39 views
The document discusses technical debt and its relation to continuous updates in software management, emphasizing the importance of frequent updates to maintain system integrity. It defines technical debt, compares it with concepts like depreciation and inflation, and questions whether it's the right metaphor for ongoing software maintenance. The content also presents insights into various platforms and operating systems, including their release and patch schedules.
![T R A C K : C U L T U R A L T R A N S F O R M A T I O N
Depreciation is […] the decrease
in the value of assets and the
method used to reallocate, or
"write down" the cost of a
tangible asset (such as
equipment) over its useful life
span. — Wikipedia
Technical Depreciation?
Unintended reduction in value of
a software product over time,
independent of source code
changes.](https://image.slidesharecdn.com/istechnicaldebttherightmetaphorforcontinuousupdate-alldaydevops2022-221112082158-fd7afe87/85/Is-Technical-Debt-the-right-metaphor-for-Continuous-Update-AllDayDevOps-2022-15-320.jpg)
![T R A C K : C U L T U R A L T R A N S F O R M A T I O N
Inflation is a general increase in
the prices of goods and services in
an economy […] corresponds to a
reduction in the purchasing
power of money. — Wikipedia
or Technical Inflation?
Unintended reduction in value of
a software product over time,
independent of source code
changes.](https://image.slidesharecdn.com/istechnicaldebttherightmetaphorforcontinuousupdate-alldaydevops2022-221112082158-fd7afe87/85/Is-Technical-Debt-the-right-metaphor-for-Continuous-Update-AllDayDevOps-2022-16-320.jpg)
![T R A C K : C U L T U R A L T R A N S F O R M A T I O N
Reverse indexes
Library → Binaries [SCA tool]
O.S. API → Binaries [SAST tool]
Binary → Pipelines [artifact store]
Pipeline → Repo(s) [pipeline tool]
Bill of Materials
on steroids
SBOM
Pipeline
Binaries
Production
Library
Git Repo CVE](https://image.slidesharecdn.com/istechnicaldebttherightmetaphorforcontinuousupdate-alldaydevops2022-221112082158-fd7afe87/85/Is-Technical-Debt-the-right-metaphor-for-Continuous-Update-AllDayDevOps-2022-23-320.jpg)
Is Technical Debt the right metaphor for Continuous Update - AllDayDevOps 2022
- 1. T R A C K : C U L T U R A L T R A N S F O R M A T I O N N O V E M B E R 1 0 , 2 0 2 2 Giulio Vian, Unum Is Technical Debt the right metaphor for Continuous Update?
- 2. T R A C K : C U L T U R A L T R A N S F O R M A T I O N New site launches, Success!
- 3. T R A C K : C U L T U R A L T R A N S F O R M A T I O N then…
- 4. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Was it Technical Debt fault? Image source: Max Pixel
- 5. T R A C K : C U L T U R A L T R A N S F O R M A T I O N How often do I need to update? Is it Technical Debt? What is Continuous Update? Agenda
- 6. T R A C K : C U L T U R A L T R A N S F O R M A T I O N How often do I need to update?
- 7. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Operating System patches Application stack patches Libraries updates and patches Update what?
- 8. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Operating Systems and Images Platform Released every Patched every (avg.) Alpine 6 months 52.2 days (7.5 weeks) Ubuntu 2 years (LTS) 6 months 21.8 days Amazon Linux 2 years (LTS) 3 months 21.7 days Windows Server 3 years 6 months monthly Patch Tuesday
- 9. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Application Platforms Platform Released every Patched every (avg.) Chrome 1 month 14 days Node.JS 30 months (LTS) 6 months (non-LTS) 25 days Go 6 months Two major releases supported. 26 days MongoDB 30 months 5 weeks .NET 3 years (LTS) 18 months 6 weeks Java 3 years (LTS) 6 months 12 weeks
- 10. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Libraries Source: Sonatype
- 11. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Docker: hidden dependency Source: Snyk
- 12. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Is it Technical Debt?
- 13. T R A C K : C U L T U R A L T R A N S F O R M A T I O N • something, especially money, that is owed to someone else, or the state of owing something — Cambridge Dictionary • Debt is an obligation that requires one party, the debtor, to pay money or other agreed-upon value to another party, the creditor. — Wikipedia • Debt is something, usually money, borrowed by one party from another. Debt is used by many corporations and individuals to make large purchases that they could not afford under normal circumstances. A debt arrangement gives the borrowing party permission to borrow money under the condition that it is to be paid back at a later date, usually with interest. — investopedia What is debt?
- 14. T R A C K : C U L T U R A L T R A N S F O R M A T I O N «With borrowed money you can do something sooner than you might otherwise, but then until you pay back that money you'll be paying interest. I thought borrowing money was a good idea, I thought that rushing software out the door to get some experience with it was a good idea, but that of course, you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.» Ward Cunningham, 2009 Technical Debt
- 15. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Depreciation is […] the decrease in the value of assets and the method used to reallocate, or "write down" the cost of a tangible asset (such as equipment) over its useful life span. — Wikipedia Technical Depreciation? Unintended reduction in value of a software product over time, independent of source code changes.
- 16. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Inflation is a general increase in the prices of goods and services in an economy […] corresponds to a reduction in the purchasing power of money. — Wikipedia or Technical Inflation? Unintended reduction in value of a software product over time, independent of source code changes.
- 17. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Maintenance costs to Release more often or increase of Operational Costs?
- 18. T R A C K : C U L T U R A L T R A N S F O R M A T I O N What is Continuous Update?
- 19. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Necessity of frequently updating a system, independently of source code changes*. Continuous Update
- 20. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Who manage the layers? Application Run-time OS image libraries Image (Ops?) Self-contained (Dev?)
- 21. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Who manage the layers? Application Run-time OS image libraries Container
- 22. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Always & everything Image by Hypock_
- 23. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Reverse indexes Library → Binaries [SCA tool] O.S. API → Binaries [SAST tool] Binary → Pipelines [artifact store] Pipeline → Repo(s) [pipeline tool] Bill of Materials on steroids SBOM Pipeline Binaries Production Library Git Repo CVE
- 24. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Testing, Resources, oh my!
- 25. T R A C K : C U L T U R A L T R A N S F O R M A T I O N Is Technical Debt the right metaphor for Continuous Update?
- 26. T R A C K : C U L T U R A L T R A N S F O R M A T I O N No, we must rebuild Production frequently (Continuous Update) and it is not our fault (aka Technical Debt)
- 27. T R A C K : C U L T U R A L T R A N S F O R M A T I O N https://www.linkedin.com/in/giuliovian @giulio_vian
- 28. T R A C K : C U L T U R A L T R A N S F O R M A T I O N https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021 https://blog.chromium.org/2021/03/speeding-up-release-cycle.html https://nodejs.org/en/about/releases/ https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/process/release_cycle.md https://support.google.com/chrome/a/answer/6220366 https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core https://docs.fedoraproject.org/en-US/releases/lifecycle/ https://www.oracle.com/java/technologies/java-se-support-roadmap.html https://kubernetes.io/releases/release/ https://www.mongodb.com/support-policy/software https://heartbleed.com/ Why Every Business Is a Software Business — Watts S. Humphrey Informit, Feb 22, 2002 http://www.informit.com/articles/article.aspx?p=25491 https://en.wikipedia.org/wiki/Watts_Humphrey https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021 https://www.shopify.com/enterprise/global-ecommerce-statistics References (1/3)
- 29. T R A C K : C U L T U R A L T R A N S F O R M A T I O N https://blog.cloudflare.com/popular-domains-year-in-review-2021/ https://radar.cloudflare.com/year-in-review-2021 https://snyk.io/blog/net-open-source-security-insights/ https://www.contrastsecurity.com/the-state-of-the-oss-report-2021 https://octoverse.github.com/static/github-octoverse-2020-security-report.pdf https://www.soa.org/globalassets/assets/files/resources/research-report/2020/quantification-cyber-risk.pdf https://www.soa.org/globalassets/assets/files/resources/research-report/2020/exposure-measures-cyber-insurance.pdf https://www.csis.org/programs/strategic-technologies-program/significant-cyber-incidents https://www.verizon.com/business/resources/reports/dbir/ https://www.accenture.com/us-en/insights/security/cost-cybercrime-study https://www.ibm.com/security/data-breach https://go.snyk.io/SoOSS-Report-2020.html https://www.amazon.co.uk/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339 https://www.sciencedirect.com/science/article/abs/pii/0164121279900220 https://daverupert.com/2020/11/technical-debt-as-a-lack-of-understanding/ References (2/3)
- 30. T R A C K : C U L T U R A L T R A N S F O R M A T I O N https://wiki.owasp.org/images/b/bd/Software_Composition_Analysis_OWASP_Stammtisch_-_Stanislav_Sivak.pdf https://googleprojectzero.blogspot.com/ https://googleprojectzero.blogspot.com/2022/04/the-more-you-know-more-you-know-you.html https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V14.md https://dotnet.microsoft.com/en-us/download/dotnet/3.1 https://docs.mongodb.com/upcoming/release-notes/5.0/ https://www.devsecops.org/ https://github.com/golang/go/wiki/Go-Release-Cycle https://go.dev/doc/devel/release https://libraries.io/data https://github.com/devopsenterprise/2021-virtual-us/blob/main/Bryan%20Finster%20-%20DOES%202021%20-%20Misuse%20and%20Abuse%20DORA%20Metrics.pdf https://trumpwhitehouse.archives.gov/wp-content/uploads/2018/02/The-Cost-of-Malicious-Cyber-Activity-to-the-U.S.-Economy.pdf http://wiki.c2.com/?WardExplainsDebtMetaphor References (3/3)
- 31. T R A C K : C U L T U R A L T R A N S F O R M A T I O N
Editor's Notes
- #2: Good Morning, my name is Giulio Vian and today I will talk about a change transversal across the entire IT industry. The title “Is Technical Debt the right metaphor for Continuous Update?” may seem obscure now, but I promise it will resonate before I finish the talk.
- #3: Many of us lived a similar adventure. Working on a green-field project and the go-live day arrives. Everyone celebrates the success, Application is online, users coming in flocks. Start thinking of the next steps, new features, new ideas.
- #4: But no, your plans are ruined: you must drop everything and work on updating the code to prevent an attack. It was not your fault, you scanned the source with all kind of tools, you attacked the site, you hired an expert in pen-testing… all useless. A flaw was found in a 3rd party library and now you have to rebuild and redeploy the entire application. I’m dramatizing here, but it is not impossible or even rare. Is becoming more and more frequent.
- #5: So, you did your best, you took no shortcuts, no explicit technical debt. What happened then? I hope to convince you that we entered a new era where we must take in account the entire surrounding environment (also known as the jungle). In the past, with the slow pace of change, we were able to reckon this work of Continuous Updating as Technical Debt, but today, we must bring it to the light and put a different label on it.
- #6: I organized the content in three blocks: in the first section I’ll try to demonstrate how fast we must updated our systems and applications; in the second part, I propose some definitions for Continuous Updating that helps in discussion with management and planning; finally, in the third section, I will hint at the engineering beyond Continuous Updating, what current tools offer and their limits.
- #7: Let’s start our journey exploring the current landscape of updates and patches.
- #8: A simple, but not simplistic, description of any execution environment considers three layers: Operating System (Linux distro, docker base image, Windows Server), the Application stack or run‑time (NodeJS, JRE, .NET runtime, C DLLs/SO independently distributed), and Libraries. Some languages, like Go, embed the run‑time in the final executable: the run-time version is tied to the compiler/linker version in this case. Each layer has its own independent source of updates and patches (Canonical, Microsoft, RedHat, Oracle, Amazon, …). Depending on your architecture and deployment mechanism, each layer can be updated independently of the other or they are bundled together in a single deployment unit (immutable infrastructure scenario).
- #9: I spent some time researching how frequently each layer receives an update. Analysing public repositories and sources, I came out with this data: most operating systems are patched between 3 and 4 weeks. Source https://github.com/docker-library/official-images git log --pretty=format:"%ci %h %s" --since="2019-01-01" -- library/alpine Ubuntu release cycle | Ubuntu Amazon Linux release cadence - Amazon Linux 2022
- #10: Moving to the next layer, application run‑times, we can observe more spread. We must not be deceived by major or minor releases: the frequency of patches is substantially higher. It ranges from the bi‑weekly patching of the most used client run‑time, that is Google Chrome, to Java SDK which average to almost three months between patch releases. Someone may wonder why I included MongoDB in this list: from a practical perspective a modern application relies on many components, databases, caches, queues, messaging systems, etcetera. Their updates reflect on application security and behaviour, so I included Mongo as a representative. .NET Core 3.1 3.1.0 December 3, 2019 3.1.22 December 14, 2021 got 22 patch releases in 3 years i.e. every 45 days/6 weeks Node v14 (Fermium) Active LTS start 2020-10-27 v14.15.0 2022-02-01, Version 14.19.0 total 19 releases in 463 days or 66 weeks i.e. every 24.4 days JDK 11 Java SE 11 (LTS)September 25, 2018 11.0.13+8 (GA), October 19th 2021 total 13 releases(updates) in 1121 days i.e. every 12.3 weeks or 86.2 days Go 1.16 released 2021-02-16 go1.16.14 (released 2022-02-10) total 14 updates in 360 days i.e. 26 days go1 (released 2012-03-28) -> go1.17 (released 2021-08-16) 17 major releases in 3429 days or 490 weeks MongoDB 5.0 5.0.0 - Jul 13, 2021 5.0.6 - January 31, 2022 total 6 releases in 203 days or 29 weeks i.e. every 4.8 weeks
- #11: The next layer includes all the libraries used by the application. A modern trend is the increased dependency on 3rd party libraries, and in particular on open source libraries. This chart from Sonatype illustrates the huge increase registered by major OSS repositories: in one year, from 2020 to 2021, the absolute number of downloads increased from a minimum! of 50% to over 90%, depending on the language platform. ISO 5230 – OpenChain Standard The standard defines the key requirements of a quality open source license compliance program, which builds trust between organisations exchanging software solutions composed of open source software. This ‘trust’ is founded on the fact that an organisation’s conformant program indicates to others that it has been designed to achieve license compliance for the open source software it shares.
- #12: The net result is a constant shift of the stack, the need to protect the entire software stack and the chain that produces it. Thus the need to update software frequently.
- #13: Let’s move to the second section and consider what technical debt is.
- #14: We all know that Technical Debt is a metaphor: we use the word debt to better communicate with a non‑technical audience. The listener may be an expert with a Master in Economics or have a simpler, common-sense, idea. I put here three definitions of debt. I think there are three common elements across the definitions, and these three elements are key to understand Technical Debt. They are: will, principal (capital), and interest. One does not enter debt without agreeing and accepting it, it is an act of will. He borrows an exact amount of money, the principal, for a duration, more or less defined. Part of the contract is the interest to pay as time passes. Debt can be re-negotiated, for example delaying payments.
- #16: Ward Cunningham used debt as an explanation for a technical problem. The technical debt metaphor matches the three core elements described before. The engineering team knows that a design is sub-optimal, and which solution would be preferrable. Will element. They have an estimate for the future solution and the temporary solution, and the temporary solution is substantially cheaper to implement, otherwise choosing to take the short cut would be irrational. This is the capital element. Finally, the interest: the more one delays paying back the debt, the bigger the amount.
- #17: Continuous Updating is not refactoring: it is a simple adaptation to the environment. No new domain knowledge or user feedback requiring rearchitecting or refactoring. So, what analogy should we use to explain it? I do not see the three elements characterizing debt, thus I am suggesting other terms borrowed (pun intended) from economics. My first proposal is the term depreciation. It captures the fact the investment in software is not a constant value but decreases over time. The limit of this analogy is that depreciation is an accounting and fiscal technique, disconnected from external events.
- #18: Another word might be inflation. This metaphor has an advantage to depreciation, because catches the external elements. The ratio of software value changes is not constant, there can be periods with high inflation (a technology leap, new attack techniques) and period of calm.
- #19: The last way to describe the need to update constantly is an increase of operational costs to keep the lights on. Cost to rebuild, to track, store, and deploy. In my opinion, all three ideas are more effective to explain the Continuous Update phenomenon compared to the concept of technical debt.
- #20: We reached the third part where we analyse what Continuous Update means in practice.
- #21: It is not Continuous Delivery or Continuous Deployment. The necessity of frequently updating a system, independently of source code changes. Necessity, we must update to prevent attacks. Frequently, because we probably have to update every day, if we average across operating system patches, run‑time patches, and libraries’ patches. Independently, means even when there are no functional changes, we rebuild and redeploy to production. Source code changes excluding the portion of build and deploy scripts which identify the version of dependencies.
- #22: In practice, we can be “lucky” and have a monolith running on VMs. IT Operations is responsible for updating to OS and maybe the run‑time.
- #23: If we consider a micro‑service architecture, each service packaged in a Docker image, we must redeploy the whole image, no matter which layer is patched. This is an interesting downside of containers. It applies anywhere immutable infrastructure is implemented.
- #24: The simplest way to implement Continuous Update is to rebuild and redeploy all your application every single day, blindly. But if we use the other senses, we’ll find a better way, won’t we? Image source: Matt Murdock/Daredevil drawing I did in hopes of the reboot :) : Daredevil (reddit.com)
- #25: Coupling Software Bill of Materials (SBOMs) with Configuration Management Items, we can easily locate which component requires a rebuild and redeploy as soon as our security team receive the news of a security patch at all levels. The next logical steps to automate the process is automatically editing all references to the patched component, spanning build scripts (packages.json, pom.xml, .csproj), Dockerfiles, Ansible/Puppet/Chef, to Terraform/CloudFormation/ARM templates. The last step is to kick-off the pipeline to build and deploy the impacted components. Note that in this scenario the normal approval process may fail short (do you want to approve dozens of deployments?).
- #26: Consider what is required to automate the process in full. That is the flip side of the coin. We need a strong tests harness around the application to guarantee that a patch does not break any core functionality. If you have a big portfolio of components based on the same technology, think of Java, a single library upgrade can start a build storm with hundreds of builds queueing for hours or worse, killing production performance due to the straining of a massive number of concurrent deployments. Another side‑effect might be the increased usage of storage for intermediary artifacts, especially if immutable infrastructure pattern is in place. Current image technology is not optimized for frequent and small changes to images. Finally, with the increasing adoption of safe languages like Rust, might be that in a few years we won’t see security patches anymore. Maybe.
- #27: Back to the original question that gives the title to this speech “Is Technical Debt the right metaphor for Continuous Update?”.
- #28: I think that my answer is a clear ‘No’. Technical Debt is a great metaphor for some kind of the problems we face while delivering software. We borrow time whenever we judge better taking a shortcut than going the long way. Short cuts make long delays (LOTR p.88) came the long way around (Doctor Who, ep. 11 s. 9, Heaven Sent) Nowadays we face a new phenomenon: perfectly working software weakens and needs continuous maintenance. The speed of decay is noticeable, much more similar to inflation: our money value less and less as time goes by.
- #29: My name is Giulio Vian, I am Italian, work in Ireland as a Principal Engineer for Unum, a Fortune 500 insurance company. I post my thoughts on DevOps and random topics in my blog, LinkedIn, and other media.
- #30: To prepare this talk I research a number of sources which are listed in the next three slides. I won’t apologize for the font because it is not meant to be read now: please download the deck from the conference or my own site.
- #33: Thanks once more for staying to the end and hope you enjoyed the content.