PyData London 2015 - How We Turned EverythingMe Into a Data Driven Company
Download as PPTX, PDF4 likes1,231 views
The document describes how Everything.me transformed into a data-driven company by utilizing Amazon Redshift and re:dash for data access and visualization. It outlines the company's data infrastructure, query execution rates, and the collaborative use of BI tools across the organization. The initiative emphasizes making data accessible to all employees while improving data interaction through templates and alerts.
PyData London 2015 - How We Turned EverythingMe Into a Data Driven Company
- 1. HOW WE TURNED EVERYTHINGME INTO A DATA DRIVEN COMPANY
- 6. Requirements: 1. scalable 2. fast 3. easy to query (accessible)
- 7. Amazon Redshift “Petabyte scale; massively parallel Fully managed; zero admin…”
- 9. 8TB total size of data* 9 Nodes Cluster Loading new data every ~5 minutes ~1500 query executions per day
- 10. One main fact table: fact_events
- 11. Step #2 Give Everyone Access
- 12. BI TOOLS?
- 14. CSV Sharing
- 15. (hackathing!)
- 25. re:dash supports multiple data sources: PostgreSQL Redshift MySQL BigQuery MongoDB InfluxDB Graphite
- 26. Everyone at the company is using re:dash at some capacity. 8K queries created to date, with >130 dashboards.
- 27. Over 25* companies using it & 24 contributors. * Probably more. Not everyone reaches out to me.
- 28. Step #3 Improve
- 29. Giving everyone raw access is only the first step, not the end of the road.
- 30. Working with SQL all the time -> repeating yourself.
- 32. Alerts
- 35. • Give everyone at the company access to the data Make it easy (accessible) Avoid restrictions
Editor's Notes
- #4: We’re building a contextual Android launcher, that delivers “One Tap Happiness” to its users. Early on we decided that we want to empower all of our employees with access to all possible data related to the company and product.
- #5: I will start with a disclaimer we’re a bunch of geeks at EverythingMe. Even our CEO writes SQL queries. Not every solution that works for us, will work in every company, but I’m sure that there is something for everyone here and the principals apply.
- #6: We had three main tools: Splunk, Google Analytics & Flurry. Due to limits of the 3rd party tools, we had growing need to get our own numbers from our own data. Because we already had our events data in Splunk, we tried to use it but were limited, due to Splunk’s nature. It’s was great for Ops, but failed us in Analytics.
- #7: When we decided to look for a new tool, we wanted to enable everyone access to raw data. Therefore we needed a scalable solution, that will grow with our data, and will allow for running fast queries and most important — will be easy to query. Due to our previous experience with 3rd party tools we knew we want something we can run in-house. For various reasons, we looked mainly at Hadoop and Redshift.
- #8: Redshift is a columnar database that exposes PostgreSQL interface. We picked Redshift mainly due to the ease of querying it and the familiar interface (SQL). The quote is how Amazon sells Redshift. Most of it is correct. Zero admin is a fantasy, but it’s definitely much easier to admin than many other comparable options. It is scalable and massively parallel though. To get the most out of it, you really need to understand it.
- #9: Scribe is a server for aggregating log data that's streamed in real time from clients (Scribe is a deprecated project and these days we’re switching Scribe with nsq). We invest in tooling to bring all our data into a single repository (Redshift). It gives all the users equal access to everything. Google Spreadsheets is used to import small dimension tables.
- #10: We’re not dealing with real big data. Just with lots of rows.
- #11: ~200 columns table. We didn't need to "design" our data model in advance, everything was reaction to actual use. Making raw data available is key #1. Key #2 is Redshift’s ability to run queries on top of the raw data without much hiccup, something that wasn’t really possible with other tools.
- #13: Maybe it’s our engineering culture or the fact that Redshift was new at the time, but no existing tool could give us easy access to Redshift *full capabilities*.
- #14: Because the data was available, users started using their favorite SQL tool to access the data.
- #15: Soon they started sharing results with CSV files attached to emails. This sucks: Don’t know the source Don’t know when it was generated Can’t reproduce
- #16: Luckily shortly after deploying Redshift, we had one of our internal hackathons (“hackAthing”), where I decided to tackle the problem of CSV sharing.
- #17: The result was re:dash — open source project that we released over a year and a half ago. Written in Python. Of course. With Flask, Celery, and many other good libraries.
- #18: Write a query.
- #19: Visualize it.
- #20: The key for sharing is that every data set & a query have their own URL you can share. So people can review your results and the query used to generate them.
- #21: You can create your own fork. This is useful for further researching, changing, etc. But also useful for people less familiar with SQL or the data model, to be able to create their own things without much knowledge of SQL.
- #22: Collect results and visualizations into dashboards.
- #23: More features: filters.
- #24: Parameters.
- #25: Python as queries! It can be used to merge data sets, to apply simple transformation that are hard to do with sql, to bring data from external sources via API or to use Pandas and such.
- #27: Internal adoption.
- #28: External adoption.
- #36: The key thing is to give everyone access to the data. You tools might vary depends on your organization type, and the amount of data you have. But you need to start with making it accessible. Data analysts: the idea is not to put you out of job, but rather to free you to do the interesting work instead of generating the same reports.
- #37: Thank you very much. I’ll really appreciate your feedback.