Making App Developers More Productive
1 like133 views
This document discusses how to build real-time analytics applications by leveraging object-relational mappers, Elasticsearch, and Rockset query lambdas. It describes constructing analytical APIs with ORMs, using Elasticsearch for real-time search functionality, and how Rockset enables writing SQL queries in lambda functions to power analytical APIs with millisecond query performance. The document concludes with advocating for Rockset to build real-time analytics applications as it allows teams to easily collaborate on query lambdas and requires less server-side code to create and maintain real-time analytical APIs.
Making App Developers More Productive
- 1. Making Application Devs More Productive with Real-time Analytics as an API Nadine Farah (Senior Developer Advocate at Rockset) nadine@rockset.com
- 2. Rockset: Built for Real-time Analytics in the Cloud 2 Self-service on semi-structured data Enable high query performance (Converged Index) Eliminate the need to manage clusters Turn powerful sql queries into real-time analytical APIs
- 3. Agenda ● Overview for building real-time analytics in 3 steps ● Object Relational Mapping (ORM) ● Elasticsearch ● Rockset Query Lambdas ● Building a Developer API Platform for Real-time Analytics ● Demo 3 3
- 4. 4
- 5. Build Real-time Analytics in 3 Major Steps 5 PostGreSQL, MongoDB, DynamoDB, etc... Elasticsearch, PostgreSQL, Data warehouses, etc... Examples of databases and data stores
- 6. Build Real-time Analytics in 3 Major Steps - contd 6
- 7. Construct Real-time Analytical APIs with ORMs (Object Relational Mappings)
- 8. Construct an Analytical Query with ORMs 8 ORM: u2 = Users.objects.select_related('city').get(id=2) SQL: SELECT u.user_id, u.title, y.city_id, y.city_name FROM users AS u INNER JOIN city AS y ON y.id = u.city_id WHERE u.user_id = 2
- 9. Construct a Real-time Analytical API with ORMs
- 10. Construct a Real-time Analytical API with ORMs- contd
- 11. ORMs for Analytical Queries: Considerations ● Great for getting started on an app that needs structured & relational data ● ORMs hide how many reads are happening behind the scenes, possibly stressing the database ● ORMs can cache, trading off memory for speed 11
- 12. 12 SQL: cursor.execute(""" SELECT products.*, purchases.number_purchases, reviews.average_rating FROM commons.products LEFT JOIN ( SELECT product_id, COUNT(*) as number_purchases FROM commons.purchases GROUP BY 1 ) purchases on products.id = purchases.product_id LEFT JOIN ( SELECT product_id, AVG(CAST(rating as int)) average_rating FROM commons.reviews GROUP BY 1 ) reviews on products.id = reviews.product_id WHERE revews.product_id= ‘%s’”””%id) Using Raw SQL to Write Analytical Queries ● Be vigilant about not using string interpolation
- 13. Construct Real-time Search APIs with Elasticsearch
- 14. 14 Elasticsearch for Real-time Search Functionality
- 15. Construct APIs with Elasticsearch 15
- 16. Joining Data in Elasticsearch vs. SQL 16 SQL: SELECT products.*, purchases.number_purchases, reviews.average_rating FROM commons.products LEFT JOIN ( SELECT product_id, COUNT(*) as number_purchases FROM commons.purchases GROUP BY 1 ) purchases on products.id = purchases.product_id LEFT JOIN ( SELECT product_id, AVG(CAST(rating as int)) average_rating FROM commons.reviews GROUP BY 1 ) reviews on products.id = reviews.product_id WHERE + whereClause Elasticsearch: ...(next slide)...
- 18. Application-side Joins in Elasticsearch: Considerations 18 ● Great for text search & log search on semi-structured data/less structured data ● Writing joins natively within your app increases complexity at the implementation level ● You can denormalize data to combine data from different models, but it comes at an expense
- 19. Construct Real-time Analytical APIs with Rockset Query Lambda
- 20. Build Real-time Analytics with Rockset 20 $ curl --request POST --url https://api.rs2.usw2.rockset.com/v1/orgs/self/ws/commons/lambdas/MyReco/versions/832f29e fdad4e57b/...
- 21. Real-time Analytical Queries in Rockset 21 SQL: SELECT products.*, purchases.number_purchases, reviews.average_rating FROM commons.products LEFT JOIN ( SELECT product_id, COUNT(*) as number_purchases FROM commons.purchases GROUP BY 1 ) purchases on products.id = purchases.product_id LEFT JOIN ( SELECT product_id, AVG(CAST(rating as int)) average_rating FROM commons.reviews GROUP BY 1 ) reviews on products.id = reviews.product_id WHERE + whereClause SQL in Rockset: TLDR no change! SELECT products.*, purchases.number_purchases, reviews.average_rating FROM commons.products LEFT JOIN ( SELECT product_id, COUNT(*) as number_purchases FROM commons.purchases GROUP BY 1 ) purchases on products.id = purchases.product_id LEFT JOIN ( SELECT product_id, AVG(CAST(rating as int)) average_rating FROM commons.reviews GROUP BY 1 ) reviews on products.id = reviews.product_id WHERE + whereClause
- 22. Real-time Analytics as an API 22
- 23. 23 Real-time Analytics with Query Lambdas: Considerations ● Used for search and analytics on semi-structured & structured data ● Rockset is cost efficient for 10’s of terabytes of data ● Rockset is optimized for cloud-only analytics ●
- 24. Building a Developer API Platform for Real-time Analytics ● Teams can easily collaborate and version Query Lambdas ● Millisecond Query performance out-of-the-box ● Less server-side code to create and maintain real-time analytical APIs 24
- 25. Live Demo
- 26. Using Rockset to Build Real-time Analytics 26 26