Big Data LDN 2018: ENABLING DATA-DRIVEN DECISIONS WITH AUTOMATED INSIGHTS
- 1. Enabling Data-Driven Decisions with Automated Insights Charlotte Emms Data Insight Analyst @ Seenit charlotte@seenit.io
- 2. Charlotte Emms Data Insight Analyst @ Seenit - 4 years of experience in the Data Analytics field - Joined Seenit’s Product team in Nov ‘17
- 3. Introducing Seenit A video collaboration platform that enables organisations to co-create authentic and engaging content with their communities of customers, superfans, or employees.
- 5. A Brief History of Seenit Founded in January 2014 Over 200k uploads hit 4k+ monthly active users MVP used Couchbase deployed on GCP Still the same today! Seenit Capture v2.0 released in April 2018
- 6. Our Couchbase Journey • Seenit’s MVP in 2014 was powered by Couchbase 2.2 back when querying the data required MapReduce techniques • The introduction of N1QL transformed Seenit, allowing us to develop new features rapidly and analyse event- based audit data for product usage research • N1QL allowed Seenit to easily understand this data by hiring a Data Analyst proficient in SQL
- 7. Non-first (N1) Normal Form Query Language (QL) • It is based on ANSI 92 SQL • Its query engine is optimized for modern, highly parallel multi-core execution SQL-like Query Language • Expressive, familiar, and feature-rich language for querying, transforming, and manipulating JSON data N1QL extends SQL to handle data that is: • Nested: Contains nested objects, arrays • Heterogeneous: Schema-optional, non-uniform • Distributed: Partitioned across a cluster What is N1QL?
- 8. ...this is not a scalable approach for repeatable work! The Seenit Data Analysis Toolkit Display the results using Plotly - a personal favourite in terms of data display versatility and clarity Extract data for analysis using N1QL queries through the Couchbase Python SDK Manipulate the data in Jupyter Notebooks with Python, using analytics libraries like Pandas and Numpy
- 9. How can I share this data with the wider team?
- 10. “Let’s build a dashboard” - every data analyst joining a startup ever (probably)
- 11. Tell a different story with the data
- 12. How to Communicate Platform Usage and Make it Interesting There was a need for something more tangible. Something that can engage someone from any role in the business For example: How many uploads in a week? How far was our global reach this week? Which were the active projects this week & who was running them Have mobile users been interacting with the in-app-feed?
- 13. Seenit’s Regular, Fully Automated Platform Update Important overused pun
- 14. Three core sections ○ This week in numbers ○ This week in lists (what’s new, what’s active) ○ This week in fun facts, which includes the following: ■ most exotic upload location (furthest from the office) ■ most committed contributor (uploads to projects) ■ biggest crowd pleaser (likes from others) The “Engaging” Part
- 15. The Technical Journey Productionised ● Introduced more thorough HTML templating using Jinja2 ● Scheduled in Jenkins to run once a week ● Runs in Kubernetes by default - also set up to run in Docker Initial Idea / PoC ● Off the back of a conversation at stand-up ● Designed a process in Python through a Jupyter notebook ● Query data in Couchbase, manipulating and returning in tabular form for friendly email display ● Sent using my own gmail account MVP ● Found and wrestled with a nicer looking email template ● Introduced fun facts section ● Moved code to a formal Python project (separate scripts, classes, functions) using an executable file to trigger the email generation ● Sent using Sendgrid as we do for generic email notifications from the platform
- 16. Next steps
- 17. Another idea
- 18. DataBot A Slack Bot user to deliver bespoke analytics on request • Coded in Python and uses the RTM (Real Time Messaging) Slack API • Mention the name of the bot user to get a response • The bot uses the first word after the direct mention to decide the correct response - no AI/ML training or NLP methods needed Image
- 19. Image
- 20. Things I learnt - You don’t need a relational database structure to build automated data programs - You also don’t need a BI tool to communicate data insights (at first*) - Try new things even if they don’t go to plan - Learn new skills from teammates - Welcome feedback and keep iterating!
- 21. Questions