DSR Microservices (Day 1, Part 2)
0 likes551 views
The document discusses various topics related to microservices including DevOps, continuous delivery, configuration management, monitoring, containers, virtual machines, inter-service communication using messaging queues, testing microservices, and chaos engineering. It provides examples of concepts like immutable infrastructure, service discovery, concurrency, and using the event log as a source of truth.
DSR Microservices (Day 1, Part 2)
- 2. What Microservices actually look like
- 3. DevOps
- 4. “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” John Allspaw, Velocity 2009
- 5. “...a cross-disciplinary community of practice dedicated to the study of building, evolving and operating rapidly-changing resilient systems at scale.” Jez Humble
- 11. “If you have more meetings than releases, you’re an enterprise company” Clifton Cunningham, CTO @ TES Global
- 12. Containers
- 13. Containers credit to Anne Currie
- 14. Bare Metal
- 15. Bare Metal ✔ Powerful ✔ Simple
- 16. Bare Metal ✔ Powerful ✔ Simple ❌ Brittle ❌ Inflexible
- 17. Bare Metal ✔ Powerful ✔ Simple ❌ Brittle ❌ Inflexible
- 18. Virtual Machines
- 19. Virtual Machines ✔ Flexible ✔ Networking ✔ Security
- 20. Virtual Machines ✔ Flexible ✔ Networking ✔ Security ❌ Overweight
- 21. Virtual Machines ✔ Flexible ✔ Networking ✔ Security ❌ Overweight
- 22. Virtual Machines ✔ Flexible ✔ Networking ✔ Security ❌ Overweight
- 23. Containers
- 25. Containers ✔ Lightweight ✔ Agile ❌ Untested ❌ Networking ❌ Security
- 26. Containers ✔ Lightweight ✔ Agile ❌ Untested ❌ Networking ❌ Security
- 27. Containers ✔ Lightweight ✔ Agile ❌ Untested ❌ Networking ❌ Security
- 29. Exercise
- 32. Immutable Infrastructure Don’t change, replace Requires automation
- 33. Immutable Infrastructure Don’t change, replace Requires automation Blue-Green
- 35. Docker Compose Manage multi-container apps Collection of dockerfiles → services version: '2' services: web: build: . ports: - "5000:5000" volumes: - .:/code - logvolume01:/var/log links: - redis redis: image: redis volumes: logvolume01: {}
- 36. Is Docker production ready?
- 37. Is Docker production ready?
- 38. Is Docker production ready? Not great at backwards compatibility… Requires extra tooling
- 42. “The development contracts are being performed concurrent with the production contracts” 2015 Lockheed Martin Annual Report
- 45. Ideas
- 46. Ideas Build
- 55. Build Test HTTP
- 56. Build Test Email
- 62. “Service Discovery is an anti-pattern”
- 63. “Service Discovery is an anti-pattern” Richard Rodger, CTO @ nearForm
- 64. Messaging!
- 68. Time To Live
- 69. mq.publish(msg, {ttl: 10}); Time To Live
- 70. mq.publish(msg, {ttl: 10}); function put(msg, startTime) { response = http.put(msg); if (response.code != 200) { if (getTime() - startTime < timeout) { put(msg, startTime); } } } startTime = getTime(); put(msg, startTime); Time To Live
- 71. mq.publish(msg, {ttl: 10}); function put(msg, startTime) { response = http.put(msg); if (response.code == 404) { // ??? } else if (response.code != 200) { if (getTime() - startTime < timeout) { put(msg, startTime); } } } startTime = getTime(); put(msg, startTime); Time To Live
- 72. mq.publish(msg, {ttl: 10}); function put(msg, startTime) { response = http.put(msg); if (response.code == 404) { // ??? } else if (response.code == 429) { // ??? } else if (response.code == 500) { // ??? } else if (response.code != 200) { if (getTime() - startTime < timeout) { put(msg, startTime); } } } startTime = getTime(); put(msg, startTime); Time To Live
- 73. mq.publish(msg, {ttl: 10}); function put(msg, startTime) { response = http.put(msg); if (response.code == 404) { // ??? } else if (response.code == 429) { // ??? } else if (response.code == 500) { // ??? } else if (response.code != 200) { if (getTime() - startTime < timeout) { timeSpent = getTime() - startTime; delay = Math.pow(timeSpent, 2); setTimeout(put(msg, startTime), delay); } } } startTime = getTime(); put(msg, startTime); Time To Live
- 74. mq.publish(msg, {ttl: 10}); function put(msg, startTime) { response = http.put(msg); if (response.code == 404) { // ??? } else if (response.code == 429) { // ??? } else if (response.code == 500) { // ??? } else if (response.code != 200) { if (getTime() - startTime < timeout) { timeSpent = getTime() - startTime; delay = Math.pow(timeSpent, 2); remainingTime = timeout - timeSpent; delay = Math.min(delay, remainingTime); setTimeout(put(msg, startTime), delay); } } } startTime = getTime(); put(msg, startTime); Time To Live
- 75. mq.publish(msg, {ttl: 10}); function put(msg, startTime) { response = http.put(msg); if (response.code == 404) { // ??? } else if (response.code == 429) { // ??? } else if (response.code == 500) { // ??? } else if (response.code != 200) { if (getTime() - startTime < timeout) { timeSpent = getTime() - startTime; fuzz = Math.floor(Math.random() * 10); delay = Math.pow(timeSpent, 2) + fuzz; remainingTime = timeout - timeSpent; delay = Math.min(delay, remainingTime); setTimeout(put(msg, startTime), delay); } } } startTime = getTime(); put(msg, startTime); Time To Live
- 77. // Best effort, fire and forget mq.publish(unimportantMessage, {qos: 0}); // At least once delivery mq.publish(idempotentMessage, {qos: 1}); // Exactly once delivery mq.publish(importantMessage, {qos: 2}); Quality of Service
- 78. // Best effort, fire and forget mq.publish(unimportantMessage, {qos: 0}); // At least once delivery mq.publish(idempotentMessage, {qos: 1}); // Exactly once delivery mq.publish(importantMessage, {qos: 2}); Quality of Service
- 79. // Best effort, fire and forget mq.publish(unimportantMessage, {qos: 0}); // At least once delivery mq.publish(idempotentMessage, {qos: 1}); // Exactly once delivery mq.publish(importantMessage, {qos: 2}); Quality of Service
- 80. Concurrency
- 81. Concurrency Upload text → Generate video → Post video
- 82. Concurrency Fast Slow Fast Upload text → Generate video → Post video
- 83. Concurrency Fast Slow Fast Upload text → Generate video → Post video
- 88. Concurrency Upload service Video generation service Posting service Text Video Video generation service Video generation service
- 89. Concurrency Upload service Video generation service Posting service Text Video Video generation service Video generation service LoadBalancer
- 90. Concurrency publish RabbitMQ subscribe Upload service Video generation service Video generation service Video generation service
- 91. Concurrency publish RabbitMQ subscribe Upload service Video generation service Video generation service Video generation service
- 92. Concurrency publish RabbitMQ subscribe Upload service Video generation service Video generation service Video generation service
- 93. Concurrency publish RabbitMQ subscribe Upload service Video generation service Video generation service Video generation service
- 94. Concurrency publish RabbitMQ subscribe Upload service Video generation service Video generation service Video generation service
- 95. “Publish everything!” Tom Livesey, Droplet
- 97. Exercise
- 99. Lots of data sources
- 100. Lots of data sources Lots of data consumers
- 101. Lots of data sources Lots of data consumers Lots of integrations
- 102. LinkedIn before
- 104. LinkedIn after
- 105. Up close
- 106. The log Append only Time ordered
- 107. Distributed logging
- 113. “If two identical, deterministic processes begin in the same state and get the same inputs in the same order, they will produce the same output and end in the same state.” State Machine Replication Principle, Jay Kreps, LinkedIn
- 115. Log as source of Truth
- 119. Initial population
- 120. person born person died person born person born Initial population
- 121. Population: 308,745,538 person born person died person born person born Initial population
- 126. 2 million writes s-1 (on 3 cheap machines) 10s of millions of writes s-1 (LinkedIn production)
- 127. 2 million writes s-1 (on 3 cheap machines) 10s of millions of writes s-1 (LinkedIn production) 7 transactions s-1 (Bitcoin network)
- 128. Log as source of Truth
- 129. Event Sourcing
- 130. “We don’t care if our production system crashes.” Valerii Vasylkov, William Hill
- 132. Data science. Analytics about the business Full-fledged stream processing framework Microservice like. Core business functions Build an app that consumes from Kafka directly
- 133. Data science. Analytics about the business Full-fledged stream processing framework Microservice like. Core business functions Build an app that consumes from Kafka directly
- 134. Data science. Analytics about the business Full-fledged stream processing framework Microservice like. Core business functions Build an app that consumes from Kafka directly
- 138. Monolithic testing
- 140. Monolithic testing App Test Test data Test data Test data
- 141. Monolithic testing App Test Test data Test data Test data Output ✔ Output ✔ Output ✔
- 142. Monolithic testing App Test Test data Test data Test data Output ✔ Output ✖ Output ✔
- 143. Monolithic testing App Test Test data Test data Test data Output ✔ Output ✔ Output ✔ Output ✔ Output ✔ Output ✔
- 144. Monolithic testing App Test Production App Test data Test data Test data Output ✔ Output ✔ Output ✔
- 149. Monitoring = Testing
- 154. Chaos Gorilla
- 155. Chaos Kong
- 156. Conformity Monkey
- 157. Security Monkey
- 158. Janitor Monkey
- 159. Latency Monkey
- 160. “Everything fails all the time” Werner Vogels, VP + CTO @ Amazon
- 161. Design for Failure
- 162. Self healing systems
- 165. “How big should my Microservice be?” Everyone
- 166. Domain Driven Design (DDD)
- 171. “Not all of a large system will be well designed” Eric Evans
- 172. “The first draft of anything is shit” Ernest Hemingway
- 173. “... big enough to fit in your head” Dan North
- 174. “Replaceable component architecture” Dan North
- 175. Monolith first?
- 176. 12factor.net
- 177. I. Codebase One codebase tracked in revision control, many deploys
- 178. I. Codebase One codebase tracked in revision control, many deploys Always use version control!
- 179. I. Codebase One codebase tracked in revision control, many deploys Always use version control! One app (microservice) per repo
- 180. I. Codebase One codebase tracked in revision control, many deploys Always use version control! One app (microservice) per repo Multiple deploys per app (local, dev, prod, etc.)
- 181. II. Dependencies Explicitly declare and isolate dependencies
- 182. II. Dependencies Explicitly declare and isolate dependencies Never assume something is pre-installed
- 183. II. Dependencies Explicitly declare and isolate dependencies Never assume something is pre-installed Package managers/repos make this easy! $ pip install -r requirements.txt $ npm install $ bundle install
- 184. II. Dependencies Explicitly declare and isolate dependencies Never assume something is pre-installed Package managers/repos make this easy! Isolate dependencies to prevent ‘leaking’ $ pip install -r requirements.txt $ npm install $ bundle install
- 185. II. Dependencies Explicitly declare and isolate dependencies Never assume something is pre-installed Package managers/repos make this easy! Isolate dependencies to prevent ‘leaking’ $ npm install
- 186. II. Dependencies Explicitly declare and isolate dependencies Never assume something is pre-installed Package managers/repos make this easy! Isolate dependencies to prevent ‘leaking’ (Aims to) prevent “it works on my machine” $ npm install
- 187. III. Config Store config in the environment
- 188. III. Config Store config in the environment Config changes between deploys (dev, prod, etc.)
- 189. III. Config Store config in the environment Config changes between deploys (dev, prod, etc.) Store config in environment not codebase
- 190. III. Config Store config in the environment Config changes between deploys (dev, prod, etc.) Store config in environment not codebase
- 191. III. Config Store config in the environment Config changes between deploys (dev, prod, etc.) Store config in environment not codebase Suggest environment vars, but other options...
- 192. III. Config Store config in the environment Config changes between deploys (dev, prod, etc.) Store config in environment not codebase Suggest environment vars, but other options... Supports multiple deploys from one codebase
- 193. III. Config Store config in the environment Config changes between deploys (dev, prod, etc.) Store config in environment not codebase Suggest environment vars, but other options... Supports multiple deploys from one codebase Makes Open Sourcing easier
- 194. IV. Backing services Treat backing services as attached resources
- 195. IV. Backing services Treat backing services as attached resources Backing services are any remote services
- 196. IV. Backing services Treat backing services as attached resources Backing services are any remote services Loosely coupled to a deploy
- 197. IV. Backing services Treat backing services as attached resources Backing services are any remote services Loosely coupled to a deploy Treated as local resources
- 198. IV. Backing services Treat backing services as attached resources Backing services are any remote services Loosely coupled to a deploy Treated as local resources Allows easy swapping out of services
- 199. V. Build, release, run Strictly separate build and run stages
- 200. V. Build, release, run Strictly separate build and run stages Build
- 201. V. Build, release, run Strictly separate build and run stages Build → Release
- 202. V. Build, release, run Strictly separate build and run stages Build → Release → Run
- 203. V. Build, release, run Strictly separate build and run stages Build → Release → Run Strict separation between the stages
- 204. V. Build, release, run Strictly separate build and run stages Build → Release → Run Strict separation between the stages One way (no tweaking the live server!)
- 205. VI. Processes Execute the app as one or more stateless processes
- 206. VI. Processes Execute the app as one or more stateless processes Apps run as processes (duh)
- 207. VI. Processes Execute the app as one or more stateless processes Apps run as processes (duh) No state: every node is independent
- 208. VI. Processes Execute the app as one or more stateless processes Apps run as processes (duh) No state: every node is independent Caching is fine, but not for future use
- 209. VI. Processes Execute the app as one or more stateless processes Apps run as processes (duh) No state: every node is independent Caching is fine, but not for future use State must be stored in backing services
- 210. VII. Port binding Export services via port binding
- 211. VII. Port binding Export services via port binding Expose endpoints http://localhost:5000/ test.mosquitto.org:1883 redis://redisbox:6379/0
- 212. VII. Port binding Export services via port binding Expose endpoints No runtime injection or native APIs http://localhost:5000/ test.mosquitto.org:1883 redis://redisbox:6379/0
- 213. VII. Port binding Export services via port binding Expose endpoints No runtime injection or native APIs HTTP or other protocols make things easy http://localhost:5000/ test.mosquitto.org:1883 redis://redisbox:6379/0
- 214. VIII. Concurrency Scale out via the process model
- 215. VIII. Concurrency Scale out via the process model Processes are the unit of scaling
- 216. VIII. Concurrency Scale out via the process model Processes are the unit of scaling Don’t daemonize!
- 217. VIII. Concurrency Scale out via the process model Processes are the unit of scaling Don’t daemonize! Take advantage of OS resource scaling
- 218. IX. Disposability Maximize robustness with fast startup and graceful shutdown
- 219. IX. Disposability Maximize robustness with fast startup and graceful shutdown Can be started or stopped quickly (Cattle not pets)
- 220. IX. Disposability Maximize robustness with fast startup and graceful shutdown Can be started or stopped quickly (Cattle not pets) Quick startup allows easy scaling up Graceful shutdown allows easy scaling down
- 221. IX. Disposability Maximize robustness with fast startup and graceful shutdown Can be started or stopped quickly (Cattle not pets) Quick startup allows easy scaling up Graceful shutdown allows easy scaling down Easier if you scale with independent processes...
- 222. X. Dev/prod parity Keep development, staging, and production as similar as possible
- 223. X. Dev/prod parity Keep development, staging, and production as similar as possible Keep time between environment deploys low
- 224. X. Dev/prod parity Keep development, staging, and production as similar as possible Keep time between environment deploys low Devs also handle deployment and maintenance (DevOps) Keep tooling between environments as similar as possible
- 225. X. Dev/prod parity Keep development, staging, and production as similar as possible Keep time between environment deploys low Devs also handle deployment and maintenance (DevOps) Keep tooling between environments as similar as possible If you can, try one environment...
- 226. XI. Logs Treat logs as event streams
- 227. XI. Logs Treat logs as event streams Don’t mess around writing to files, just use stdout
- 228. XI. Logs Treat logs as event streams Don’t mess around writing to files, just use stdout In dev, you can watch stdout In production, it can be routed to other stores (Splunk etc.)
- 229. XI. Logs Treat logs as event streams Don’t mess around writing to files, just use stdout In dev, you can watch stdout In production, it can be routed to other stores (Splunk etc.) Keeps aggregated logs time ordered (like Kafka)
- 230. XII. Admin processes Run admin/management tasks as one-off processes
- 231. XII. Admin processes Run admin/management tasks as one-off processes One-off admin tasks (DB migrations etc.) should be run from production environment
- 232. XII. Admin processes Run admin/management tasks as one-off processes One-off admin tasks (DB migrations etc.) should be run from production environment Don’t run scripts from local machine or directly on the DB
- 233. XII. Admin processes Run admin/management tasks as one-off processes One-off admin tasks (DB migrations etc.) should be run from production environment Don’t run scripts from local machine or directly on the DB Check in scripts to app repos to keep things in sync
- 234. One-off admin tasks (DB migrations etc.) should be run from production environment Don’t run scripts from local machine or directly on the DB Check in scripts to app repos to keep things in sync ssh into production box and run the task XII. Admin processes Run admin/management tasks as one-off processes
- 235. XII. Admin processes Run admin/management tasks as one-off processes One-off admin tasks (DB migrations etc.) should be run from production environment Don’t run scripts from local machine or directly on the DB Check in scripts to app repos to keep things in sync ssh into production box and run the task ...or don’t. Immutable Infrastructure!
- 236. 1990 2000 2010 2015 Orderleadtime(log(ish)) < 1 sec 1 hour 1 week 6 months Data center Virtual machines Commercial cloud (AWS) Docker FaaS 5 secs PaaS
- 237. Microservices or distributed systems?
- 240. Hardware lead times 1990 2000 2010 2015 Orderleadtime(log(ish)) < 1 sec 1 hour 1 week 6 months Data center Virtual machines Commercial cloud (AWS) Docker FaaS 5 secs PaaS
- 241. Hardware lead times 1990 2000 2010 2015 Orderleadtime(log(ish)) < 1 sec 1 hour 1 week 6 months Data center Virtual machines Commercial cloud (AWS) Docker FaaS 5 secs PaaS
- 242. Amazon Lambda Serverless ‘Serverless’ (not really) Function as a Service (FaaS) No persistence, no upkeep costs 200 ms startup time (Node.js)
- 243. What is the role of a manager?
- 244. Managers
- 246. Managers Provide guidance Keep things ‘safe’
- 247. Managers Provide guidance Keep things ‘safe’ Organise the team
- 248. Managers Provide guidance Keep things ‘safe’ Organise the team Enable team to work
- 249. Managers Microservices Provide guidance Keep things ‘safe’ Organise the team Enable team to work
- 250. Managers Microservices Provide guidance Keep things ‘safe’ Organise the team Enable team to work Explore new tech
- 251. Managers Microservices Provide guidance Keep things ‘safe’ Organise the team Enable team to work Explore new tech Experiment
- 252. Managers Microservices Provide guidance Keep things ‘safe’ Organise the team Enable team to work Explore new tech Experiment Enables self-organisation
- 253. Managers Microservices Provide guidance Keep things ‘safe’ Organise the team Enable team to work Explore new tech Experiment Enables self-organisation Tech empowers team
- 255. No managers (or other titles) Programmers take responsibility Expect/allow failure (remove fear) Work directly with customers Programmer Anarchy
- 256. Conclusion
- 257. Speed > efficiency DevOps Empower teams Embrace the distributed world
- 258. Speed > efficiency DevOps Empower teams Embrace the distributed world
- 259. Speed > efficiency DevOps Empower teams Embrace the distributed world
- 260. Speed > efficiency DevOps Empower teams Embrace the distributed world
- 261. “Microservices are hard. Some things you get for free, but you have to work for the good stuff. If you won’t put in the work, you shouldn’t be doing Microservices. (You should be doing that stuff anyway!)” Steve Upton
- 262. “Microservices are hard. Some things you get for free, but you have to work for the good stuff. If you won’t put in the work, you shouldn’t be doing Microservices. (You should be doing that stuff anyway!)” Steve Upton
- 263. “Microservices are hard. Some things you get for free, but you have to work for the good stuff. If you won’t put in the work, you shouldn’t be doing Microservices. (You should be doing that stuff anyway!)” Steve Upton
- 264. “Microservices are hard. Some things you get for free, but you have to work for the good stuff. If you won’t put in the work, you shouldn’t be doing Microservices. (You should be doing that stuff anyway!)” Steve Upton
- 265. “Microservices are hard. Some things you get for free, but you have to work for the good stuff. If you won’t put in the work, you shouldn’t be doing Microservices. (You should be doing that stuff anyway!)” Steve Upton
- 266. “Microservices are hard. Some things you get for free, but you have to work for the good stuff. If you won’t put in the work, you shouldn’t be doing Microservices. (You should be doing that stuff anyway!)” Steve Upton