Brendan O'Bra Scala By the Schuykill

Editor's Notes

• #2: Our journey to Microservices from a “classic” monolith About the strange interfaces between design patterns and operations
• #3: I took the original “Functional Progamming in Scala” in 2013 Despite that, everyone around me knows more than I do ;)
• #6: And so many more… sometimes with several repeats
• #7: Started in 2013 First production work for Sochi Olymics with Sports app First X1 work in 2014 Large scale use in 2015 Now deploys entire X1 backend infrastructure plus many other projects Will be open source at some point Currently supports “traditional VMs” (i.e. not docker)
• #8: Yes ,the deployment tool space is now somewhat crowded Why not
• #9: - A lot of apps seems to start as Play! Apps even if Spray is a better choice
• #12: A science experiment goes prod
• #13: That’s 67k LOC, and some of them are even good
• #15: One, bigass sbt-assembly jar. Large, opaque.SNAPSHOTs up the wazzoo. Deployed onto ”tradtional VMs”
• #16: - - - - Compilation errors, general brittleness - Whole days could be spent untying wet knots
• #17: Onboarding was a bitch, due to things like cloud configs, network access and image configuration
• #21: And also…. Multitenancy
• #23: Low coupling == agility Well defined BECAUSE they are small in scope Old school “contracts” make understanding who says what easier to determine DNS service? Hello World? Depends on your project Akka is , of course, perfect for microservices
• #26: On the Gumby team: All do development (write tests + write code) All do production support All get to smell the roses All get to smell the shit All have at least 2(Ops,Dev) inner voices present This allows (forces) us to confront operational AND development issues at the same time. This includes deployment, scaling and other “Enterprisey” things
• #28: They are BFFs because Devops (coupled with automation), makes being incremental more approachable – automate once, run many. This means that, not only is it repeatable, it’s also much more exercisable
• #29: Robots!
• #32: One tenant per vm In/our security model NO self service
• #33: One “grid” with common database Multitenant Self service Granular security control
• #35: We had a dependency graph that looks like this….
• #37: There are 13 of these damn things now ;) The interconnections are the story Job Control is Cluster Sharding, backed by akka-persistence-cassandra Each one of those is probably a build configuration
• #38: - Point out management overhear of circled items.
• #39: This archictecture is fairly complex. Each microservice is a JVM, and ultimately a docker container They are all very connected They need to be able to locate and talk to each other
• #40: Form a management perspective, the monolith is much more attractive – it’s one type of thing to manage, even if there are multiple of them
• #41: Dependencies and interconnections expand geometrically Aggregate answers to questions such as “How is my service performing” become much more complex to answer The use of tools such as Logging-As-A-Service become impossible to do without The sheer volume of data and number of nodes demands automation Automation AND Configuration AND CI expand to be one instance PER Microservice We realized that one runtime “thing” had become 11 (not including external dependencies). And we were also putting all customers onto “this thing” So, what do we do?
• #42: Spinal Tap Anyone? Microlith or “ulith”
• #43: OK.. So we have everything modularized. Let’s make it do something
• #44: Take these services
• #45: Modularize based on previous
• #46: We tried to use a module-per-project for interfaces, but the dependency graph soon became circular Bottleneck is that MANY things will now depend on common-messages module Makes it MUCH easier to write tests around common messaging ;) 3 – Because you are almost forced to anyways Being able to depend on hard versions becomes important to common module. This is another great place for an sbt plugin
• #47: In the end, it will all be packaged as one big thing, and needs to pass the “sbt-release” uberjar sniff test This will be painful and annoyingly “iterative”, but will produce a clean dependency graph Yes, this could be construed as heavy handed and “enterprisey”
• #48: -We favored ‘explicit dependency inject’ The factory (implementations are “known” to the “main method” that will use them (i.e. it’s dependent) Scala Props() work nicely for this.
• #51: And you end up with a dependency graph that looks like this
• #54: Submodules can be a pain, and devs have to remember to update them Delayed integration can be a pain
• #55: Intellij frequently pukes on this module layout and goes all ”red crazy”
• #56: Backed by cassandra, which is managed as ”pets” Things like off node logging and monitoring become indispensible
• #57: - A lot of apps seems to start as Play! Apps even if Spray is a better choice
• #59: With each microservice already mostly defined, each one can be peeled off , adding a “main” to each. The “peeled off” standalone microservices can then be incrementally deleted from the uLith, with the final architecture looking like a a “genuine” microsevices architecture
• #60: We ”needed” a singleton for JWT token (decided we did not want to deal with database stuff) We still love Akka tho Cassandra is great, but needs people who know WTF they are doing. Type sensitivity – unhandled message, and being susceptible to dupicate classnames in different packages Serialization w/respect to cluster sharding. What happens with upgrades?
• #61: 1 Akka in one jvm is not the same as akka in >1 JVM. REALLY learn to cluster 2 Embrace the idiosyncracies of the overlay nework. Leverage the dns 3 We were – eventually we developed SBT plugins to build images for different enviroments , and provide correct config 4 Implicit caused confusion (sort of as expected) for new folks – consider your audience 5 We really struggle with using maven/ivy (local) repos 6 it’s much better to understand and harominize your dependency graph. Exclude is the singleton of sbt ;) 7 We work with many Java folks, and a bunch of Go folks as well. 8 Vikas, Alex – it’s about quality, not quantity 9 Even doing nothing, everything has a shelf life 10 this stems from years of ‘mvn clean install’ providing default test. We’ve had SO many broken tests in CI because of this