CARTO Cloud Native – An Introduction to the Spatial Extension for BigQuery
0 likes443 views
The document introduces the Carto Cloud Native spatial extension for BigQuery, detailing its user-defined functions and procedures for performing spatial analysis directly within a cloud data warehouse environment. Key features include serverless scalability, over 60 core functions, advanced spatial analytics, and tools like the BigQuery tiler for visualizing large datasets. Additionally, it highlights benefits like eliminating ETL processes and ensuring security compliance while planning for future functionalities in Snowflake.
![CARTO — Unlock the power of spatial analysis
Spatial indices = scalable spatial analysis
Enable to partition the space at different resolutions [viz]](https://image.slidesharecdn.com/20210616webinaranintroductiontothespatialextensionforbigquery-210617091518/85/CARTO-Cloud-Native-An-Introduction-to-the-Spatial-Extension-for-BigQuery-22-320.jpg)
CARTO Cloud Native – An Introduction to the Spatial Extension for BigQuery
- 1. CARTO Cloud Native – An introduction to the Spatial Extension for BigQuery Follow @CARTO on Twitter
- 2. CARTO — Unlock the power of spatial analysis Introductions Margara Tejera Product Manager at CARTO Miguel Álvarez Data Scientist at CARTO
- 3. CARTO — Unlock the power of spatial analysis Agenda ● Introduction to the Spatial Extension and vision ● Overview of the functions available in the Spatial Extension for BigQuery ● Demo: cluster, polygonize and spatial indexes ● Tiler + feature preview. ● Demo: solving a complete spatial analysis ● Spatial Extension for Snowflake ● Q&A
- 4. CARTO — Unlock the power of spatial analysis POLL 1 Data Analyst ………..……………………………………………………………. Data Scientist ………………….……….……………………………………….. Developer …..…………………………………...……………………………….. What’s your current role? Other …..…………………………………………………………………………….
- 5. CARTO — Unlock the power of spatial analysis Intro
- 6. CARTO — Unlock the power of spatial analysis CARTO is going cloud native
- 7. CARTO — Unlock the power of spatial analysis The CARTO Spatial Extension is a suite of user defined functions (UDFs) and procedures to perform spatial analysis natively within the cloud data warehouse environment. What’s the Spatial Extension?
- 8. CARTO — Unlock the power of spatial analysis
- 9. CARTO — Unlock the power of spatial analysis ● Scalable and serverless spatial analytics ● No more ETLs: everything happens within the data warehouse ● Security compliance! Some key benefits
- 11. CARTO — Unlock the power of spatial analysis Status in BigQuery
- 12. ● 60+ core functions ● 12 advanced functions and procedures * Full list in the Documentation
- 13. CARTO — Unlock the power of spatial analysis ● Spatial Extension: BigQuery project called bqcarto. ● Module: BigQuery dataset, e.g. h3, quadkey, tiler, etc. Organises the functionality of the Spatial Extension in categories. ● Function / procedure: BigQuery UDF functions and procedures.
- 14. CARTO — Unlock the power of spatial analysis POLL 2 Only CARTO ………………………………………………………………………. Only BigQuery ………………………….……………………………………….. Both CARTO & BigQuery…..……………………………………………….. Do you use CARTO and/or BigQuery ?
- 15. CARTO — Unlock the power of spatial analysis Examples & new CARTO Cloud Native
- 16. CARTO — Unlock the power of spatial analysis CARTO Cloud Native Workspace
- 18. CARTO — Unlock the power of spatial analysis Cluster and polygonize
- 19. CARTO — Unlock the power of spatial analysis Weather stations coverage using Voronoi diagrams New police stations based on Chicago Crime locations
- 20. CARTO — Unlock the power of spatial analysis Spatial Indexes H3 · Quadkey · S2
- 21. CARTO — Unlock the power of spatial analysis What are they?
- 22. CARTO — Unlock the power of spatial analysis Spatial indices = scalable spatial analysis Enable to partition the space at different resolutions [viz]
- 23. CARTO — Unlock the power of spatial analysis Spatial indices = scalable spatial analysis Expensive spatial operations like intersections become table JOINS
- 24. CARTO — Unlock the power of spatial analysis Example
- 25. CARTO — Unlock the power of spatial analysis Tiler
- 26. CARTO — Unlock the power of spatial analysis BigQuery Tiler is a solution to visualise massive datasets hosted in BigQuery through the transformation of the original data into vector tiles. What is it?
- 27. CARTO — Unlock the power of spatial analysis How it works ● Converts tables into tilesets: z, x, y, data (MVT), carto_partition ● Metadata stored in z/x/y -1/NULL/NULL (JSON) ● 4000 partitions using carto_partition: improve speed ● Uses procedures: reads query twice (get metadata, generate tileset) z x y carto_partition data 16 45340 24576 3605 H4sIAAAAAAAA/5Py52JPdt3eyCLEwM (…)
- 28. CARTO — Unlock the power of spatial analysis 1. World Population (Worldpop) 2. New American Atlas (CDRC) 3. Night time lights (Colorado School of Mines) Examples Publicly available in BigQuery project carto-do-public-tilesets
- 29. CARTO — Unlock the power of spatial analysis ● No configuration needed ● Smart tile memory management Feature preview: Smart Tiler ST_GENERATETILESET( input_sql STRING, output_table_name STRING, --OPTIONAL options)
- 30. CARTO — Unlock the power of spatial analysis CARTO offers a complete toolset 1. Create tilesets with BigQuery Tiler 2. Serve tiles through the Maps API 3. Visualise tilesets ○ In the Dashboard / Cloud Native Workspace ○ In custom apps through CARTO for deck.gl and CARTO for React
- 31. CARTO — Unlock the power of spatial analysis Point Aggregation Tileset ● Encodes the original features (points, lines, polygons) and their attributes. ● Only for points or data that can be converted to points. ● Encodes the aggregation of the points’ attributes. Simple Tileset
- 32. CARTO — Unlock the power of spatial analysis It’s time for a real world example!
- 33. CARTO — Unlock the power of spatial analysis Getting access
- 34. CARTO — Unlock the power of spatial analysis 1. Create BigQuery connection, OAuth type. 2. Choose any billing project 3. Check that you have access running this in BQ: bqcarto.tiler.VERSION()
- 35. CARTO — Unlock the power of spatial analysis Other data warehouses
- 36. CARTO — Unlock the power of spatial analysis Status in Snowflake ● Same functionality as in BigQuery, except Tiler and Data module. ● Installation through the Snowflake Data Marketplace. * Full list in the Documentation Coming soon!
- 37. CARTO — Unlock the power of spatial analysis Q&A
- 38. Thanks for listening! Any questions? Request a demo at CARTO.COM Miguel Álvarez Data Scientist at CARTO // malvarez@cartodb.com Margara Tejera Product Manager at CARTO // mtejera@cartodb.com