The Power of Real Time Dashboards In Agile Development: Visualize & AttackTarantino presentation

Editor's Notes

• #2: My talk is called “visualize and attack”. It’s the outcome of our journey into real-time production monitoring; coupled with some realizations we’ve had about the importance of shortening the feedback loop when designing operational dashboards. Real-time understanding of what your application is doing after it's been released to market is critical, and the only way to ensure you're capturing the right metrics is to see them on a dashboard right when you’re writing the code.
• #3: + This talk is a culmination of watching, listening and learning from giants that freely publish content... + Giants like netflix, linkedin, etsy, gilt + and really many of the things you find on infoq.com
• #4: + This talk is a culmination of watching, listening and learning from giants that freely publish content... + Giants like netflix, linkedin, etsy, gilt + and really many of the things you find on infoq.com
• #5: + This talk is a culmination of watching, listening and learning from giants that freely publish content... + Giants like netflix, linkedin, etsy, gilt + and really many of the things you find on infoq.com
• #6: + This talk is a culmination of watching, listening and learning from giants that freely publish content... + Giants like netflix, linkedin, etsy, gilt + and really many of the things you find on infoq.com
• #7: + This talk is a culmination of watching, listening and learning from giants that freely publish content... + Giants like netflix, linkedin, etsy, gilt + and really many of the things you find on infoq.com
• #8: + This talk is a culmination of watching, listening and learning from giants that freely publish content... + Giants like netflix, linkedin, etsy, gilt + and really many of the things you find on infoq.com
• #9: + This talk is also the culmination of Ray Fallon from 360 pounding the word metrics into my head for the past year. + that's not a grammatical typo, that's what I actually hear through his thick Irish accent.
• #10: sooooooooo
• #11: + Hi, I’m Mario Pareja, + This is me playing Aussie-rules Football. + If you’re looking for an excuse to wear extremely short shorts, + come talk to me afterwards, and I’ll set you up with a team.
• #12: + I work at a company called 360incentives + We’re modernizing how brands execute promotions and engage with their channel partners: + the retailers that sell their products; + the front-line salespeople who sell their products; + and at the end of the day, you and me, the consumers buying their products. + Back in the day, you bought stuff, mailed out a rebate and 8-years later got some money back. + We changed that. Our goal is to turn people into brand advocates.
• #13: + Previously, we were on Deloitte’s Fast 50 list of Technology Companies to Watch. + Unfortunately, we just found out last week that we’re no longer on that list.
• #14: + It turns out we’ve made it on a better list. + We’re ranked the 11th fastest growing technology company in Canada.
• #15: + Right-o, so this presentation is broken up into three main parts.
• #16: + Visualize in Production + I want you to realize the benefits of visualizing what your app is doing in production. + A lot has been said about this, so I won't spend too much time here. There’s plenty of material out there.
• #17: + Visualize in Development + I want you to realize the importance of having the exact same view you would have in production at development time. + Man, this has been a super interesting realization.
• #18: + Finally, we have attack + I want you to realize how easy it was for us at 360 to shorten the feedback cycle by scripting a production-like world on our laptops. + Then I want you to be able to build your own world. + 25 minutes is not long. + Most time on last bit - Docker, I want you to gain confidence around it + Some time on the first two sections, mostly because I want you to get REALLY REALLY pumped about doing this kind of thing. + When I think of the TINY level of investment required to create a little world, MAN, I'll NEVER accept anything less.
• #19: + Alright, walk with me through a little story. + Pretend we've built a cutting edge and successful training platform.
• #20: + We're capital A, Agile – whatever that means. + We've got a tight set of unit tests, acceptance test. + We've got some form of delivery pipeline, + and we're somewhat regularly delivering to production. + And then it hits us.
• #21: + We're capital A, Agile – whatever that means. + We've got a tight set of unit tests, acceptance test. + We've got some form of delivery pipeline, + and we're somewhat regularly delivering to production. + And then it hits us.
• #22: + We're capital A, Agile – whatever that means. + We've got a tight set of unit tests, acceptance test. + We've got some form of delivery pipeline, + and we're somewhat regularly delivering to production. + And then it hits us.
• #23: + We're capital A, Agile – whatever that means. + We've got a tight set of unit tests, acceptance test. + We've got some form of delivery pipeline, + and we're somewhat regularly delivering to production. + And then it hits us.
• #24: + We're capital A, Agile – whatever that means. + We've got a tight set of unit tests, acceptance test. + We've got some form of delivery pipeline, + and we're somewhat regularly delivering to production. + And then it hits us.
• #25: + First it's just an email - Bob couldn't complete his course. + Of course, Bob also happens to be the guy that killed an hour of our life last month because he was running IE6. + He says he spent the week working through the 24 lessons, but it wouldn’t complete course.
• #26: + A few more emails come in…
• #27: + Yeah, at this point you're thinking...O o
• #28: Yeah, it’s not really a talk unless you take a stab at Macaulay Culkin.
• #29: + What the hell happened to my app? + It turns out we missed cache-busting a javascript file so people with cached versions couldn't complete the lessons. + The crappy part: a bunch of people, put in a bunch of effort and didn't realize it wasn't paying off until they thought they were done. + How could we have seen this coming? + Not through contrived tests, but from knowing that the business value we implemented WAS NOT actually being delivered.
• #30: + Metrics + I’ve got this thought – mind you it isn’t fully formulated yet – that Metrics are akin to constantly-running acceptance tests telling you that business value IS being delivered. + Consider Bob's story. There wasn't an error on our server or even the client. There was an absence. + Absence of activity is tough to detect without monitoring trends.
• #31: + Tools, we’re leaning on open source, it could be more mature, but it works nonetheless. + Push metrics to statsd which aggregates and forwards them to Graphite. + Graphite efficiently stores time-series data, query & apply crazy interesting functions to it. + We use Grafana to create graphs from Graphite data and put together entire dashboards. + Folks, check these out.
• #32: + Tools, we’re leaning on open source, it could be more mature, but it works nonetheless. + Our services push metrics to statsd which aggregates and forwards them to Graphite. + Graphite efficiently stores time-series data, query & apply crazy interesting functions to it. + We use Grafana to create graphs from Graphite data and put together entire dashboards. + Folks, check these out.
• #33: + Tools: Statsd > Graphite > Grafana + Push metrics to statsd which aggregates and forwards them to Graphite. + Graphite efficiently stores time-series data, query & apply crazy interesting functions to it. + We use Grafana to create graphs from Graphite data and put together entire dashboards. + Folks, check these out.
• #34: + Tools: Statsd > Graphite > Grafana + Push metrics to statsd which aggregates and forwards them to Graphite. + Graphite efficiently stores time-series data, query & apply crazy interesting functions to it. + We use Grafana to create graphs from Graphite data and put together entire dashboards. + Folks, check these out – they’re pretty powerful.
• #35: + Lower Graphite learning curve, slick UI + Creating dashboards is sweet with drag-drop widgets, zoom in to specific times, etc..
• #36: + PRO-TIP: dashboard variables are cool. + We use them to control resolution so we can efficiently change to large time periods.
• #37: + Dashboards need to be managed like code, you want them consistent across environments. + Use source control, create a deployment pipeline, etc.. + For us, having new graphs show up in all environments is huge, so our dashboard nodes update on commit.
• #38: RECAP + It is critical to know that the business value you've implemented IS being delivered to market. + This is a developer task not some back-office "operations" team. + Keeping an eye on trends uncovers issues & opportunities you would have otherwise NOT seen. + Use graphs, meters, annotations, whatever you can get your hands on to see what is happening (good or bad). + There are free tools available to you now that you can use to capture & view metrics.
• #39: + This is a good one. + Dude, your dev environment needs to feel like, smell like, taste like and look like it would in production. + Of course, you want this for your app. + But we’ve also realized this is SUPER important for metrics. + Here's the deal.
• #40: + Capturing log entries and capturing metrics is dead simple – add a line of code here and there.
• #41: + Knowing what to capture, on the other hand, is hard. + We don't know IF the data we're capturing is going to be helpful until we visualize it. + We get this wrong all of the time! + What we think makes sense when we're typing away into our editor often makes zero sense when viewed otherwise. + Would you ship a new UI without looking at it? + OMG this can make your logs, SOOO much better.
• #42: + Visualize in Development is all about shortening the feedback cycle. + Think back, remember the old Word Perfect, Word Star DOS days? + Remember when WYSIWYG became norm, WOAH paper = screen. Mind blown. + These days, you wouldn’t expect anything less. + I’m super passionate about shorten feedback cycles. + Give me WYSIWYG operational dashboards, man. + I want to write some beautiful code, hit save and watch those blips show up on my local dashboards. + This, folks, is where we're heading, and why Docker is our ticket to get there. + BTW, we've stumbled across a bunch of old bugs by having our logs & metrics dashboards right by our side at development time. + You'll be running through unrelated scenarios and you'll notice something weird in the logs. + It's pretty powerful stuff.
• #43: Let's rethink how we develop systems by making production insight a first-class citizen starting right at the software construction phase.
• #44: RECAP + Capturing metrics is easy. + Capturing the right metrics is hard. + Give your self a short feedback-cycle for designing your operational dashboards. + You wouldn’t push a whack of UI changes without first opening them up in your browser, would you? + Don’t do it for your operational insights either.
• #45: + Alright, in order to discuss how we at 360 are tackling this, + I’ll need to give you a quick run-down of Docker.
• #46: + How many are familiar with Linux Containers, Solaris Zones, etc.? + How many are familiar with Docker? + I have very little time, and I want this to be VERY practical, rather then deeper understanding of Docker + I want you to walk out of here with the ability TO DO stuff + shortcut for comprehension (don't quote): docker concepts => Virtual Machine Concepts, so we can be on the same page. + Not covering Docker benefts, except they’re VERY lightweight and no-boot time. + Moving on, here’s a glossary of terms.
• #47: + A Docker image is kind of like a Virtual Hard Drive Snapshot – it’s IMMUTABLE.
• #48: + A Docker Container is like a REALLY fast virtual machine instance. + boot container from image + start and stop + Docker creates a writeable layer on-top of the image, so changes are held on to. + The biggest difference is that there is no boot time and you can run hundreds of them.
• #49: + A Docker Repository is a set of related Docker Images identified by tags. + For instance, the Ubuntu repository contains images tagged with (latest, 14.04, 12.04, etc.) + An image is basically nameless until tagged within a repo.
• #50: + A Docker Registry is a well-known place to store Repository Images. + You can host your own Private Registry. + Or you can use the Public Registry at hub.docker.com.
• #51: + Volumes are super useful. + They allow you to mount a directory on your host inside of a container. + For instance, /app in your container might map to a project in your home directory. + You can also map volumes across containers.
• #52: + Links connect containers. + For instance, say you link your app container to a Mongo container. + Docker will enable network access from your app to the ports exposed by the Mongo container. + It will also create a “mongo” hosts file entry so you can write “ping mongo” and it will resolve.
• #53: + A Dockerfile is a way of telling Docker how to create an image. + Dockerfiles mostly looks like glorified shell scripts. + There are other ways to create images, but this way is reproducible.
• #54: + Let’s talk about how we’ll leverage these individual pieces to build our own little worlds.
• #55: + The first step is finding the Docker images you need for your little world.
• #56: + The common scenario is that you head over to hub.docker.com and find one that meets your needs. + The alternative is that you build one. + Once you know the image name, you can fire up a container.
• #57: + Fire up a new container based on an image using the Docker run command. + It will reach out and download the image if necessary. + It will create a writeable layer on top of that image and set up network stacks, etc..
• #58: + The –d simply means run the container in the background.
• #59: + Naming your containers becomes critical for managing them later.
• #60: + Finally, we’ve need to tell Docker what Image to start from. (this would be an official image)
• #61: + To remove a container and destroy all of its data, execute docker rm.
• #62: + The –v tells docker to purge the layer that was put on top of the image for the purpose of writing. + You must do this if you’re building and tearing down worlds frequently. + Otherwise, you’ll fill up your hard drive pretty quick.
• #63: + The –f turns off the container abruptly - since we’re destroying the state, why wait?
• #64: + Don’t look directly at this command, it’ll hurt your eyes. + I included this because it’s super useful to get a container’s IP address. + Don’t worry, I’ll be publishing the slides.
• #65: + If you have special needs, you can create your own image. + I won’t go any deeper, just look at some of the scripts. + But basically, you need to define how the image should be built. + Build it. + Tag it including a version. Tag it again with a version of “latest”. + Push the image to either your Private or the Public registry.
• #66: + Here’s how we’re leveraging Docker and a bit of automation to shorten feedback cycles at 360. + First, some background: + When we started building out services, we were sitting around and realized we needed some names. + Naturally, we decide on Quentin Tarantino characters. + payment service > dollars > dollar > buck + Well if you’ve seen Kill Bill, you’ll know that Buck is a character. + A despicable guy, but a Tarantino character none the less. + And with that….
• #67: + Meet Tarantino, he's the directory of our little world + Tarantino is a bit of development environment shell scripting + He's got a one-liner install (be forewarned, it will install Docker) + He'll also pull down various Docker images + Once installed, we just call upon tarantino by executing the tt command. For instance...
• #68: + tt clone will pull down the git repositories we care about. + But wait, how's Tarantino know which repositories?
• #69: + It's literally hard-coded at the top of the script. + That's the thing, we're not interested in building an off the shelf solution here. + We've found tremendous value in being able to add operations that are specific to what WE need. + We're open sourcing an equivalent of Tarantino. + But it’s literally intended to be used as fully functional EXAMPLE, SO YOU CAN HIT THE GROUND RUNNING.
• #70: + tt create will generate and run all of our docker containers based on the images we've built. + This includes a bunch of infrastructure like: RabbitMQ, Kibana, Statsd, Graphite, Grafana, ElasticSearch. + It also includes our custom docker image for running all of our node services.
• #71: + tt destroy tears them all down and throws away any data. + This is critical when you’ve borked your graphs because you were emitting the wrong stats.
• #72: + tt data is our gateway into data generation + After recreating the world, I'll usually run tt data aray. + That'll generate data for our fictitious test user: Aldo Ray + So I mentioned Tarantino characters: this one here is totally the one I'm most proud of.
• #73: + I spent like an hour in the middle of the night trying to come up with a phoney name. + It turns out in one of his first films, + Tarantino himself had a scene where he phoned-up his girl-friend and pretended to be "Aldo Ray". + Thank the lords, I can go to sleep. (Buahaha, tt upgrade required an smaller font than previous examples, so I dropped this in between to avoid font shifting!)
• #74: + tt upgrade is worth mentioning because it makes it easy to fetch the latest version of tarantino and his world. + I won’t go into any of the tarantino commands, for now.
• #75: + I mentioned we have a single custom docker image for running all of our node services. When running a container for each service: + We mount /app inside the container to our source code repo in our home directory + We use “forever” to restart our service upon source changes + We link our services to our infrastructure containers for Metrics and Log capture. + As you can see, this container is completely optimized for development. + What it means is that we can make changes to the code in our working directory and see them immediately reflected on our graphs.
• #76: RECAP + Docker is awesome. It’ll let you create, run and destroy a bunch of isolated “virtual machines“ with ease. + We went through a REALLY quick crash-course on Docker. I used some leaky abstractions to help get us through the presentation, so at some point do yourself a favour and dive right into Docker. + We talked about how we glue a bunch of docker containers together using Tarantino so we can monitor our applications on our laptops just as we would in production. + I gave you a glimpse of how we inject our code into node containers to obtain quick feedback about changes in our code.
• #77: RECAP + So, what have we talked about today? A LOT. + It is critical to know that the business value you've implemented IS ACTUALLY being delivered after you've released to market. + Capturing the right metrics requires feedback just like any other development activity. + Docker gives us way to run our operational infrastructure on our laptops + so we can tighten the feedback loop when designing operational dashboards. + And make production insight a first-class citizen in our software development process. + Please, try creating your own little world. It’s much easier than you think.