![on: [push
]
jobs
:
build
:
runs-on: ubuntu-lates
t
steps
:
- uses: actions/checkout@v
1
- name: Set up JDK 1.
8
uses: actions/setup-java@v
1
with
:
java-version: 1.
8
- name: Build with Gradl
e
run: ./gradle
w
- name: Archive buil
d
uses: actions/upload-artifact@v
2
with
:
name: build-ja
r
path: build/libs/*.jar
SAMPLE GITHUB WORKFLOW FILE](https://image.slidesharecdn.com/realworlddevops-220422132900/85/Real-World-DevOps-20-Practical-Developers-Tips-for-Tightening-Your-Operational-Feedback-Loops-14-320.jpg)
![<appender name="SLACK" class="com.github.maricn.logback.SlackAppender"
>
<webhookUri>${SLACK_INCOMING_WEB_HOOK}</webhookUri
>
<layout class="ch.qos.logback.classic.PatternLayout"
>
<pattern>%-4relative [%thread] %-5level %class - %msg%n</pattern
>
</layout
>
<username>${APP_NAME}@${APP_SPACE}</username
>
<iconEmoji>:rage:</iconEmoji
>
</appender
>
<appender name="ASYNC_SLACK" class="ch.qos.logback.classic.AsyncAppender"
>
<appender-ref ref="SLACK" /
>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter"
>
<level>ERROR</level
>
</filter
>
</appender>
SAMPLE LOGBACK.XML (ERRORS -> TEAMS)](https://image.slidesharecdn.com/realworlddevops-220422132900/85/Real-World-DevOps-20-Practical-Developers-Tips-for-Tightening-Your-Operational-Feedback-Loops-45-320.jpg)
Real-World DevOps — 20 Practical Developers Tips for Tightening Your Operational Feedback Loops
- 1. REAL-WORLD DEVOPS 20 PRACTICAL DEVELOPERS TIPS FOR TIGHTENING YOUR OPERATIONAL FEEDBACK LOOPS Victor Szoltysek victor_szoltysek@mac.com
- 2. DISCLAIMER ▸ Some steps will seem obvious ▸ Some steps will seem controversial ▸ Focus is on bootstrapping (scrappiness) - 80% of the value, for 20% of the cost ▸ But items become even more important at larger scale ! ▸ Focus is on value and velocity and not technologies, or exact de fi nitions ▸ Battle Tested Strategies ! ▸ Call to Action - pick one to try on your next / current engagement
- 3. BEWARE THOSE WHO PROMOTE THE SOLUTION WITHOUT DEFINING THE PROBLEM Wise Developer
- 4. WORKED FINE IN DEV, NOW IT’S AN OPS PROBLEM Sadistic Developer
- 6. Operations Developers API CALLS DevOps (No actual team) JIRA TICKETS A P I C A L L S
- 7. TIP #1 - ADOPT A “YOU BUILD IT, YOU RUN IT” MENTALITY ▸ 2 Pizza Teams (popularized by AWS) ▸ Self Suf fi cient / Skin-in-the-Game / No lobbing over the fence ▸ Prefer API’s interaction (fast) instead of Tickets (slow) ▸ i.e. CI/CD and Cloud Tools ▸ Devs are responsible for End-to-End solution (from an app site) ▸ Operations is responsible for non-app stuff with SLAs , an guard rails (automated) ▸ No separate bug fi x, release engineering, deployment, qa, etc teams
- 8. I HAVEN’T COMMITTED CODE IN OVER A WEEK AND AT THIS POINT I’M AFRAID TO TRY. Wary Developer
- 10. TIME SINCE LAST: TIME TO: BAD !! (WASTEFUL) GOOD (EFFICIENT) FIND KEYS SAW KEYS FIX / RESOLVE THINGS MERGE, DEPLOY, TEST, ETC
- 11. TIP #2 - TIGHTEN FEEDBACK LOOPS BY DOING THINGS MORE SOONER ▸ If it’s painful , do it more often ! ▸ Build ▸ Deploy ▸ Integrate (i.e. code) ▸ Breaking Things (but react quicker)
- 12. GETTING A BUILD TOGETHER IS ERROR- PRONE, TEDIOUS, AND TAKES A LONG TIME Frustrated Tester / Developer
- 13. TIP #3 - AUTOMATICALLY BUILD ON EACH COMMIT ▸ Setup Basic CI on Day #1 ▸ Build, Test, Package, and Save Artifact ▸ Modern CI - GitHub Actions / Bitbucket Pipelines ▸ CI Scripts as part of SCM ▸ Just place in correct directory and MAGIC ! ▸ Developers OWN CI !
- 14. on: [push ] jobs : build : runs-on: ubuntu-lates t steps : - uses: actions/checkout@v 1 - name: Set up JDK 1. 8 uses: actions/setup-java@v 1 with : java-version: 1. 8 - name: Build with Gradl e run: ./gradle w - name: Archive buil d uses: actions/upload-artifact@v 2 with : name: build-ja r path: build/libs/*.jar SAMPLE GITHUB WORKFLOW FILE
- 15. WHAT’S IN THIS BUILD? Frustrated Tester / Developer
- 16. TIP #4 - VERSION BUILD ARTIFACTS ▸ Human Readable Symantec Versioning ▸ i.e. Major.Minor.Build# ▸ Include: ▸ GIT Hash + Branch ▸ Build Time + Build Number ▸ Use CI Environment Variables - BUILD_NUMBER (Jenkins) , GITHUB_RUN_NUMBER (GitHub) ▸ Spring Boot Actuator + BuildInfo + Git Properties Plugin (automatic endpoint creation) ▸ /actuator/info endpoint (REST) ▸ Show PM’s / QA’s / non-technical people how to check ▸ Include issue # in commit message
- 17. { "build": { "version": “1.2.12-hotfix1” , "name": "Sample App" , "time": "2022-03-06T09:11:53.380Z" , } "git": { "branch": "main" , "commit": { "id": "5404bdf" , "time": "2022-03-06T10:34:16Z " } } } SAMPLE /INFO ENDPOINT OUTPUT
- 18. plugins { id 'org.springframework.boot' version '2.3.1.RELEASE ' id 'io.spring.dependency-management' version '1.0.9.RELEASE ' id "com.gorylenko.gradle-git-properties" version "2.2.2 " id 'java ' } version = "1.0.${System.env.BUILD_NUMBER ?: System.env.GITHUB_RUN_NUMBER ?: '0-SNAPSHOT'}" //Use Jenkins BUILD_NUMBER env or GitHub Actions RUN_NUMBER if availabl e springBoot { buildInfo( ) } dependencies { implementation 'org.springframework.boot:spring-boot-starter-actuator ' } } SAMPLE GRADLE BUILD FILE
- 19. BUILD WON’T WORK ON MY MACHINE Frustrated Developer
- 20. TIP #5 - NOTIFY ON BROKEN BUILDS ▸ Automatic Noti fi cations On Broken Builds ▸ Fix Builds Immediately ▸ Allow people to roll back broken builds ▸ Don’t Commit on Broken Builds ▸ Don’t Commit a Broken Build at 5pm on a Friday and leave !
- 21. PEOPLE DON’T CHECK EMAILS. IS THERE AS BETTER WAY OF NOTIFYING PEOPLE IF THINGS ARE BROKEN ? Frustrated Developer
- 22. TIP #6 - USE INCOMING WEBHOOKS ▸ Slack or MS Teams ▸ Created a dedicated “Bot” Teams Channel ▸ curl -s -d "payload={"text":"Test Message"}" INSERT_YOUR_WEB_HOOK_URL_HERE
- 24. THE BUILD WORKED ON MY MACHINE, BUT ISN’T WORKING ON THE CI SERVER Frustrated Developer
- 25. TIP #7 - USE GRADLE WRAPPER ▸ i.e. run gradlew (bash/bat) instead of gradle itself ▸ Removes Gradle Dependency from CI (and dev machines) ▸ Ensures everyone builds with same version of Gradle ▸ Maven Wrapper exists as well
- 26. MY JAVA BUILD SCRIPTS ARE HARD TO READ AND TAKE FOREVER TO FINISH EVEN FOR SMALL CHANGES Impatient Developer
- 27. TIP #8 - USE GRADLE (INSTEAD OF MAVEN) ▸ Easier to Read ▸ Better Plugin Support ▸ 10x+ Speed Improvements
- 28. THE BUILD WORKED, BUT THE DEPLOY IS MISSING DEPENDENCIES Angry Developer
- 29. TIP #9 - CREATE FAT PACKAGES WITH ALL DEPENDENCIES BUNDLED ▸ i.e. “Uber / Fat Jars” ▸ Use same build artifact for all environments ▸ “How one programmer broke the internet by deleting a tiny piece of code” — see “Leftpad JS” outage
- 30. THE EXACT SAME BUILD WORKED LAST WEEK, BUT ISN’T WORKING TODAY Raging Developer
- 31. TIP #10 - USE FIXED DEPENDENCY VERSIONING ▸ Build should always be exactly the same ▸ (regardless of when it was run)
- 32. WHERE TO DEPLOY ? I DON’T WANT TO LEARN OPS / NETWORKING / ETC .. I JUST WANT MY JAR TO RUN Typical Developer
- 33. AWS COMPUTE OPTIONS ▸ EC2 ▸ EKS ▸ ECS ▸ Fargate ▸ Lambda ▸ App Runner ▸ Beanstalk ▸ LightSail ▸ Which is the least amount of work for a typical Cloud-Native App ?
- 34. TIP #11 - DEFAULT TO PAAS (PLATFORM AS A SERVICE) FOR DEPLOYMENTS ▸ Amazon Beanstalk / Heroku / PCF / Azure Websites / etc ▸ Ideal for bootstrapping Green Field + Cloud-Native Microservices ▸ Easy to migrate later .. ▸ Out of the box: ▸ Networking ▸ Scaling (vertical, horizontal, and auto) ▸ Self Healing
- 35. TIP #12 - SKIP CONTAINERS ▸ There’s no need (initially) for Docker fi les , Pod Files , Kubernetes , etc ▸ Also no need for Sonar / Maven Repo / Docker Repo initially either
- 36. TIP #13 - DEPLOY FREQUENTLY (VIA CI) ▸ Options: ▸ Nightly (UAT) ▸ Daily / or on Commit (Dev) ▸ Manual (via CI GUI) or On Commit Triggers ▸ Give Non-Technical People Access
- 37. TIP #14 - KEEP SCRIPTS AS PART OF SCM ▸ Tightens the feedback loop around initial testing ▸ Version Controlled ▸ You always have a fallback if CI is down ▸ i.e. for AWS Beanstalk .. use Gradle Beanstalk Plugin (no AWS CLI needed) ▸ #gradle deployUat
- 38. MANDATED PULL REQUESTS SLOW US DOWN, AND MOST TIME IS SPEND ON NON- VALUEADD NITPICKING Frustrated PM
- 39. TIP #15 - STOP MANDATING PULL-REQUESTS ▸ Rely on Builds / CI for Static Code Analysis, and general coding consistency ▸ Consider Pair Programming, Mob Code Reviews, or Manual (after the fact) reviews ▸ Allow PMs / QAs / UI to commit directly to SCM ▸ Track average Time Spent in Pull Request if you need data ..
- 40. MERGE HELL IS TIME CONSUMING. REGRESSIONS ARE COMMON, AND REFACTORING IS IMPOSSIBLE. Frustrated Developer
- 41. TIP #17 - FOLLOW TRUNK-BASED DEVELOPMENT ▸ Instead of Long Lived Feature Branching ▸ Developer that created the new feature is responsible for the merge — and NOT the new developer in a full time “merging” role. ▸ Encourage Developers to Pull / Merge from Master frequently
- 42. TIP #17 - COMMIT INCOMPLETE FEATURES ▸ Feature Toggles (implicit is usually suf fi cient) ▸ Facebook Model ▸ Commit Code to Trunk frequently (daily+)
- 43. RANDOM THINGS ARE BREAKING, BUT WE DON’T KNOW ABOUT THEM RIGHT AWAY. LOGS SHOW DOZENS OF DIFFERENT ERRORS. Furious Stake Holder
- 44. TIP #18 - NOTIFY ON UNKNOWN ERRORS ▸ Catch all unknown errors / exceptions and alert on them immediately ▸ i.e. into Bot Channel via Webhook ▸ Avoid the “Broken Window” fallacy ▸ Fix the errors, or change them to warning / lower priority ▸ Reduce the Signal to Noise Ratio
- 45. <appender name="SLACK" class="com.github.maricn.logback.SlackAppender" > <webhookUri>${SLACK_INCOMING_WEB_HOOK}</webhookUri > <layout class="ch.qos.logback.classic.PatternLayout" > <pattern>%-4relative [%thread] %-5level %class - %msg%n</pattern > </layout > <username>${APP_NAME}@${APP_SPACE}</username > <iconEmoji>:rage:</iconEmoji > </appender > <appender name="ASYNC_SLACK" class="ch.qos.logback.classic.AsyncAppender" > <appender-ref ref="SLACK" / > <filter class="ch.qos.logback.classic.filter.ThresholdFilter" > <level>ERROR</level > </filter > </appender> SAMPLE LOGBACK.XML (ERRORS -> TEAMS)
- 46. THE WEBPAGE WON’T LOAD, BUT NO- ONE SEEMS TO KNOW ABOUT IT. End-User with out-of-date Browser
- 47. TIP #19 - NOTIFY ON UNKNOWN FRONT-END ERRORS ▸ Especially important for SPA’s (i.e. ReactJS / Angular) , and page “snaps” due to incompatible versions and dynamic typing. ▸ Include User Agent ▸ Out of date browsers usage is prevalent (especially in the government). ▸ Scrappy solution — catch uncaught exceptions on the user side, and sent them to backend for alerting.
- 48. <script > window.onerror = function(message, url, lineno, colno, error) { $.post( { url: 'logging/client-error' , contentType: 'text/plain' , data: 'Source:' + url + ':' + lineno + ' Error:' + message + ' UserAgent:' + navigator.userAgen t }) ; } </script> QUICK AND DIRTY FRONTEND ALERTING JAVASCRIPT CODE
- 49. OUR APP RANDOMLY FAILS DUE TO INTERMITTENT 3RD PARTY OUTAGES WITH ZERO ACCOUNTABILITY Frustrated PM
- 50. TIP #20 - NOTIFY ON 3RD PARTY DEPENDENCY ERRORS ▸ i.e. 3rd Party API calls , databases, etc ▸ Communication Errors / Database Failures / Timeouts ▸ Automatically Message the 3rd Party Team directly and automatically ▸ i.e. email to PM or WebHook into their Teams / Slack Channel ▸ You’ll get Bonus Points for being able to know a Teams Service is down before they do
- 51. CALL TO ACTION - IMPLEMENT A CHANCE ON YOUR PROJECT ▸ Build Automation ▸ GitHub Actions / Bitbucket Pipelines ▸ Optional Pull Requests ▸ Error Alerting ▸ Less Branching / More Committing