![So Certificates ...
• Some Certificates Weren’t[sic]
@royrapoport rsr@netflix.com
Friday, March 22, 13
Some certificates weren’t even SSL certificates -- we have certificates we get from a partner
that cannot be accessed via SSL, and for which the answer to the question “when does this
expire?” require scraping a web page.](https://image.slidesharecdn.com/20130321baylisasecuritymonkey-130322022532-phpapp02/85/SSL-Certificate-Expiration-and-Howler-Monkey-s-Inception-21-320.jpg)
SSL Certificate Expiration and Howler Monkey's Inception
- 1. SSL* Certificate Reporting BayLISA March 21st, 2013 @royrapoport rsr@netflix.com Friday, March 22, 13 This is the story of how we went from SSL certificates expiring without notice in production to deploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSL certificate expiration as a production-class issue.
- 2. SSL* Certificate Reporting BayLISA March 21st, 2013 @royrapoport rsr@netflix.com Friday, March 22, 13 This is the story of how we went from SSL certificates expiring without notice in production to deploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSL certificate expiration as a production-class issue.
- 3. Technology Overview @royrapoport rsr@netflix.com Friday, March 22, 13
- 4. Technology Overview • SoA, REST, Mostly Java @royrapoport rsr@netflix.com Friday, March 22, 13
- 5. Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: @royrapoport rsr@netflix.com Friday, March 22, 13
- 6. Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: @royrapoport rsr@netflix.com Friday, March 22, 13
- 7. Culture Overview @royrapoport rsr@netflix.com Friday, March 22, 13 We hire very smart people, give them all the context and situational awareness they want, and set them free. We design our environment, our systems, and our teams to be empowered to make decisions without requiring slow approval processes, cumbersome formal communication, or any other unnecessary friction.
- 8. Culture Overview • Freedom and Responsibility @royrapoport rsr@netflix.com Friday, March 22, 13 We hire very smart people, give them all the context and situational awareness they want, and set them free. We design our environment, our systems, and our teams to be empowered to make decisions without requiring slow approval processes, cumbersome formal communication, or any other unnecessary friction.
- 9. Culture Overview • Freedom and Responsibility • Distributed Operations @royrapoport rsr@netflix.com Friday, March 22, 13 We hire very smart people, give them all the context and situational awareness they want, and set them free. We design our environment, our systems, and our teams to be empowered to make decisions without requiring slow approval processes, cumbersome formal communication, or any other unnecessary friction.
- 10. Culture Overview • Freedom and Responsibility • Distributed Operations • Get out of the way of Developers @royrapoport rsr@netflix.com Friday, March 22, 13 We hire very smart people, give them all the context and situational awareness they want, and set them free. We design our environment, our systems, and our teams to be empowered to make decisions without requiring slow approval processes, cumbersome formal communication, or any other unnecessary friction.
- 11. So Certificates ... @royrapoport rsr@netflix.com Friday, March 22, 13
- 12. So Certificates ... • Dozens of Certificates @royrapoport rsr@netflix.com Friday, March 22, 13
- 13. So Certificates ... • Dozens of Certificates • Different kinds of places @royrapoport rsr@netflix.com Friday, March 22, 13
- 14. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private @royrapoport rsr@netflix.com Friday, March 22, 13
- 15. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB @royrapoport rsr@netflix.com Friday, March 22, 13
- 16. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs @royrapoport rsr@netflix.com Friday, March 22, 13
- 17. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 @royrapoport rsr@netflix.com Friday, March 22, 13
- 18. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 • Source Control @royrapoport rsr@netflix.com Friday, March 22, 13
- 19. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 • Source Control • EIPs @royrapoport rsr@netflix.com Friday, March 22, 13
- 20. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 • Source Control • EIPs • Totally Distributed Design @royrapoport rsr@netflix.com Friday, March 22, 13
- 21. So Certificates ... • Some Certificates Weren’t[sic] @royrapoport rsr@netflix.com Friday, March 22, 13 Some certificates weren’t even SSL certificates -- we have certificates we get from a partner that cannot be accessed via SSL, and for which the answer to the question “when does this expire?” require scraping a web page.
- 22. So Certificates ... @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 23. So Certificates ... • SSL Certificates expire @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 24. So Certificates ... • SSL Certificates expire • Millions of people can’t stream @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 25. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 26. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 27. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This • Excel worksheets @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 28. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This • Excel worksheets • Wiki documents @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 29. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This • Excel worksheets • Wiki documents • Events on public calendars @royrapoport rsr@netflix.com Friday, March 22, 13 (obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)
- 30. Let’s Do This Thing Cassandra Certificate @royrapoport rsr@netflix.com Friday, March 22, 13 Start with a very simple model -- a Certificate entity, which is really just a combination of name, expiration date, and a series of locations where we can find this. It’d be trivial to feed this thing from my todo list, if I wanted to (but given the state of my todo list, probably a bad idea)
- 31. Let’s Do This Thing ELB Cassandra Certificate @royrapoport rsr@netflix.com Friday, March 22, 13 Then start building location-aware spiders -- e.g. this spider that knows how to probe all our ELBs to see if they listen on 443 and gets their certificate if they do.
- 32. Let’s Do This Thing ELB Cassandra EC2 Instance Certificate @royrapoport rsr@netflix.com Friday, March 22, 13 Or this spider that knows how to talk to a specific kind of EC2 instance we have with some certificates.
- 33. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate @royrapoport rsr@netflix.com Friday, March 22, 13 etc ...
- 34. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem @royrapoport rsr@netflix.com Friday, March 22, 13
- 35. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem DNS @royrapoport rsr@netflix.com Friday, March 22, 13
- 36. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem DNS @royrapoport rsr@netflix.com Friday, March 22, 13 Once you have all this information, you can easily generate a web page showing certificates, where they are, and when they expire
- 37. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem DNS @royrapoport rsr@netflix.com Friday, March 22, 13 And send out emails, too -- once we built the capability for teams to subscribe to emails for a given certificate and specify how many days before expiration they should start getting notified
- 38. Since Then @royrapoport rsr@netflix.com Friday, March 22, 13 We validated the design by continuing to iterate on it -- recently, when building the DNS spider component, that work took only about 15 minutes to implement. We also expanded subscription capabilities so teams could subscribe to certificate expiration warnings based on certificate name regular expressions.
- 39. Since Then • No Production Emergencies due to SSL certificate expiration @royrapoport rsr@netflix.com Friday, March 22, 13 We validated the design by continuing to iterate on it -- recently, when building the DNS spider component, that work took only about 15 minutes to implement. We also expanded subscription capabilities so teams could subscribe to certificate expiration warnings based on certificate name regular expressions.
- 40. Since Then • No Production Emergencies due to SSL certificate expiration • Validated Design @royrapoport rsr@netflix.com Friday, March 22, 13 We validated the design by continuing to iterate on it -- recently, when building the DNS spider component, that work took only about 15 minutes to implement. We also expanded subscription capabilities so teams could subscribe to certificate expiration warnings based on certificate name regular expressions.
- 41. Since Then • No Production Emergencies due to SSL certificate expiration • Validated Design • Better Subscription Capabilities @royrapoport rsr@netflix.com Friday, March 22, 13 We validated the design by continuing to iterate on it -- recently, when building the DNS spider component, that work took only about 15 minutes to implement. We also expanded subscription capabilities so teams could subscribe to certificate expiration warnings based on certificate name regular expressions.
- 42. Soon ... @royrapoport rsr@netflix.com Friday, March 22, 13 We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.
- 43. Soon ... • Customized, automated alerting @royrapoport rsr@netflix.com Friday, March 22, 13 We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.
- 44. Soon ... • Customized, automated alerting • Automated renewal @royrapoport rsr@netflix.com Friday, March 22, 13 We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.
- 45. Soon ... • Customized, automated alerting • Automated renewal • Telling you a problem is about to happen: Good @royrapoport rsr@netflix.com Friday, March 22, 13 We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.
- 46. Soon ... • Customized, automated alerting • Automated renewal • Telling you a problem is about to happen: Good • Preventing the problem automatically: Priceless @royrapoport rsr@netflix.com Friday, March 22, 13 We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.
- 47. Soon ... • Customized, automated alerting • Automated renewal • Telling you a problem is about to happen: Good • Preventing the problem automatically: Priceless • Open Source @royrapoport rsr@netflix.com Friday, March 22, 13 We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.
- 48. Remember ... @royrapoport rsr@netflix.com Friday, March 22, 13
- 49. Remember ... • Be Lazy @royrapoport rsr@netflix.com Friday, March 22, 13
- 50. Remember ... • Be Lazy • Help Others Be Lazy @royrapoport rsr@netflix.com Friday, March 22, 13
- 51. Remember ... • Be Lazy • Help Others Be Lazy • Computers Are Better Than Humans @royrapoport rsr@netflix.com Friday, March 22, 13
- 52. Remember ... • Be Lazy • Help Others Be Lazy • Computers Are Better Than Humans • For some things @royrapoport rsr@netflix.com Friday, March 22, 13
- 53. Remember ... • Be Lazy • Help Others Be Lazy • Computers Are Better Than Humans • For some things • Don’t compete on their terms @royrapoport rsr@netflix.com Friday, March 22, 13
- 54. Questions? @royrapoport rsr@netflix.com Friday, March 22, 13