Breaking the Monolith - Microservice Extraction at SoundCloud
3 likes969 views
This document summarizes SoundCloud's journey from a monolithic Rails application to a microservices architecture. It describes SoundCloud's initial architecture using Rails and MySQL. As traffic grew, they scaled vertically by adding caching, workers, and database optimizations. They eventually extracted services like messages into independent Scala/Finagle services for improved scalability, deployability and maintenance. The summary discusses the risks of integrating new services and retiring old components, highlighting the need for load testing, rollback plans, and handling stale clients during the migration.
![Migrating the Schema
Convenient Dependency Management with @Grapes
23
#!/usr/bin/env groovy
@Grapes([
@Grab(group='org.yaml', module='snakeyaml', version='1.12'),
@Grab(group='mysql', module='mysql-connector-java', version='5.1.24')
])
import groovy.sql.Sql
import org.yaml.snakeyaml.Yaml](https://image.slidesharecdn.com/bobtalk2015-150301081920-conversion-gate01/85/Breaking-the-Monolith-Microservice-Extraction-at-SoundCloud-23-320.jpg)
![Scala Joy = Options I
Good bye NullPointerException
30
val opt: Option[String] = params.get("id")
val id: Int = opt.map(id => id.toInt).getOrElse(10)](https://image.slidesharecdn.com/bobtalk2015-150301081920-conversion-gate01/85/Breaking-the-Monolith-Microservice-Extraction-at-SoundCloud-30-320.jpg)
![Scala Joy = Functional Goodness
Function arguments and references
33
delete("/playlist/:urn/likes")(destroy)
def destroy(request: Request) =
write(request, 200)(repo.deleteLike)
def write
(request: Request, statusCode: Int)
(f: (UserSession, Urn) => Future[Like]) = {
// ...
}](https://image.slidesharecdn.com/bobtalk2015-150301081920-conversion-gate01/85/Breaking-the-Monolith-Microservice-Extraction-at-SoundCloud-33-320.jpg)
![Futures
Instance API (excerpt)
36
class Future[A] {
def get(): A
def map[B](f: A => B): Future[B]
def flatMap[B](f: A => Future[B]]): Future[B]
def onSuccess(f : A => Unit): Future[A]
}](https://image.slidesharecdn.com/bobtalk2015-150301081920-conversion-gate01/85/Breaking-the-Monolith-Microservice-Extraction-at-SoundCloud-36-320.jpg)
: Future[A]
def exception[A](e: Throwable): Future[A]
def collect[A](fs : Seq[Future[A]]): Future[Seq[A]]
}](https://image.slidesharecdn.com/bobtalk2015-150301081920-conversion-gate01/85/Breaking-the-Monolith-Microservice-Extraction-at-SoundCloud-37-320.jpg)
Breaking the Monolith - Microservice Extraction at SoundCloud
- 1. Breaking the Monolith Microservice Extraction at SoundCloud
- 2. Soundcloud ● 11 hours uploaded every minute ● 150 million tracks ● 300 million users 2
- 3. History ● Rails 2.3 ● MySQL ● S3 3
- 5. Success! 5
- 6. History ● Rails 2.3 ● MySQL ● AWS ● Cassandra ● Hadoop ● SolR ● RabbitMQ https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture 6
- 7. Still not enough ● More servers ● Add caching layer ● Defer long running tasks to workers 7
- 8. Still not enough ● Optimize database schema ● Introduce read slaves ● Dedicated databases for some models 8
- 9. Monolith 9
- 10. Major pain points ● Testing, building and deploying ● Dependency hell ● “I’d rather not touch this” 10
- 11. Rails problems I - No service layer 11 <% Category.all.each do |cat| %> <li><%= cat.name %></li> <% end %>
- 12. Rails problems I - No service layer ⇒ Tight coupling with storage layer! 12 <% Category.all.each do |cat| %> <li><%= cat.name %></li> <% end %>
- 13. Rails problems II - Active Record Magic 13 class User < ActiveRecord::Base validates_length_of :username, :within => 2..64 before_save :encrypt_password, :accept_terms_of_use has_many :comments, :dependent => :destroy # ... end
- 14. Rails problems II - Active Record Magic ⇒ Easy to write, hard to maintain 14 class User < ActiveRecord::Base validates_length_of :username, :within => 2..64 before_save :encrypt_password, :accept_terms_of_use has_many :comments, :dependent => :destroy # ... end
- 15. Ruby Problems ● GIL ● Native extensions ● Dependency management can be painful 15
- 17. Extract features as Services ● Less painful to maintain ● Easy to replace ● Fun to build 17
- 18. An Example: Messages ● 200 million messages ● MySQL database on shared host ● Features: ○ embedded sounds ○ email notifications ○ spam detection 18
- 19. Migration Requirements ● New schema to support upcoming features ● Dedicated database ● Zero downtime 19
- 23. Migrating the Schema Convenient Dependency Management with @Grapes 23 #!/usr/bin/env groovy @Grapes([ @Grab(group='org.yaml', module='snakeyaml', version='1.12'), @Grab(group='mysql', module='mysql-connector-java', version='5.1.24') ]) import groovy.sql.Sql import org.yaml.snakeyaml.Yaml
- 24. Strategy ● Import all messages ● Setup cron job to get new messages ● Listen to events for updates 24
- 26. Creating a new service 26
- 27. Convo ● Scala ● Twitter Finagle ● Scalatra Framework 27
- 29. Scala ● Functional ● OOP ● Static but inferred typing 29
- 30. Scala Joy = Options I Good bye NullPointerException 30 val opt: Option[String] = params.get("id") val id: Int = opt.map(id => id.toInt).getOrElse(10)
- 31. Scala Joy = Options II Safe chaining with for comprehensions 31 for { id <- params.get("id") user <- users.lookup(id) count <- counts.forUser(user) } yield count
- 32. Scala Joy = Pattern Matching Expressive code with decomposition 32 Urn("soundcloud:users:20") match { case Urn(_, "tracks", _) => None, case Urn(_, "messages", "20") => None, case Urn(_, "users", id) => Some(id) }
- 33. Scala Joy = Functional Goodness Function arguments and references 33 delete("/playlist/:urn/likes")(destroy) def destroy(request: Request) = write(request, 200)(repo.deleteLike) def write (request: Request, statusCode: Int) (f: (UserSession, Urn) => Future[Like]) = { // ... }
- 34. Futures!
- 35. Finagle ● Twitter rpc library on top of Netty ● Support for multiple protocols ● Future composition 35
- 36. Futures Instance API (excerpt) 36 class Future[A] { def get(): A def map[B](f: A => B): Future[B] def flatMap[B](f: A => Future[B]]): Future[B] def onSuccess(f : A => Unit): Future[A] }
- 37. Futures Object API (excerpt) 37 object Future { def value[A](a: A): Future[A] def exception[A](e: Throwable): Future[A] def collect[A](fs : Seq[Future[A]]): Future[Seq[A]] }
- 38. Futures - Examples Multiple transformations - The ugly way 38 service.getUsers().flatMap { users => service.tracksFor(users).flatMap { tracks => asJson(tracks) } }.onSuccess(json => log(s"found $json"))
- 39. Futures - Example Multiple transformations - The nice way 39 val response = for { users <- service.getUsers() tracks <- service.tracksFor(users) json <- asJson(tracks) } yield json response.onSuccess(json => log(s"found $json"))
- 40. Scala Problems ● Implicit conversions ● Binary compatibility of libraries ● Tooling still not perfect 40
- 41. SBT
- 42. IntelliJ ● Code inspection ● Debugging ● SBT support 42
- 45. Integration Risks ● Service failure ● Data loss after rolling back ● Data loss caused by stale clients 45
- 46. Integration Risks ● Service failure → load testing, A/B testing ● Data loss after rolling back ● Data loss caused by stale clients 46
- 47. Integration Risks ● Service failure → load testing, A/B testing ● Data loss after rolling back → prepare scripts, practice ● Data loss caused by stale clients 47
- 48. Integration Risks ● Service failure → load testing, A/B testing ● Data loss after rolling back → prepare scripts, practice ● Data loss caused by stale clients → keep migration running 48
- 51. Convo ● 500 million requests per day ● 1000 qps during peak time ● 5 instances 51
- 52. Microservice Problems ● Event bus dependency ● Maintenance overhead ● Distributed tracing 52
- 53. Microservices → Not a silver bullet 53
- 55. Questions?
- 56. Images ● Slide 4,7 - Rails Logo http://en.wikipedia.org/wiki/File:Ruby_on_Rails.svg ● Slide 6,51 - Party Cat http://ghostexist.deviantart.com/art/Party-Cat-logo-287986071 ● Silde 7 - MySQL Logo http://blogwifi.fr/?p=9990 ● Slide 7 - Hadoop Cop https://svn.apache.org/repos/asf/hadoop/logos/out_rgb/hadoop-security-logo. jpg ● Slide 10 - Hello, My Name Is: http://commons.wikimedia.org/wiki/File:Hello_my_name_is_sticker. svg ● Slide 14 - Sad Panda: http://www.whatsupyasieve.com/2012/09/17/lockout-blues/sad-panda-2/ ● Slide 16 - Exit Sign: http://logo-kid.com/emergency-exit-sign-left.htm ● Slide 22 - Groovy Logo: http://groovy.codehaus.org/images/groovy-logo-medium.png ● Slide 20, 25, 44 - Book Page: http://daviddiazolivares.deviantart.com/art/Old-Book-Page- 345869530 ● Slide 34 - Back to the future: http://i.huffpost.com/gen/1369403/thumbs/o-BACK-TO-THE-FUTURE- facebook.jpg ● Slide 42 - Tommy Lee Jones: http://persephonemagazine.com/2014/04/friday-news-bites-airline- pranks-gabriel-garcia-marquez-pulitzers-more/film-title-no-country-for-old-men/ ● Slide 55 - That’s all folks: http://www.hd2wallpapers.com/view/thats_all_folks-1280x800.php 56