Building Data Products with BigQuery for PPC and SEO (SMX 2022)
2 likes779 views
The document outlines a presentation by Christopher Gutknecht on building data products in BigQuery, focusing on PPC and SEO use cases within a modern data stack. It emphasizes principles for robust data products, including structured data transformation, version control, and sufficient test coverage, while leveraging tools like DBT. The document also provides specific examples of use cases and encourages participation from potential team members.
Building Data Products with BigQuery for PPC and SEO (SMX 2022)
- 1. How To Build Data Products in BigQuery for PPC & SEO Christopher Gutknecht | @chrisgutknecht | Bergzeit
- 2. 1. Intro Our Plan: Build Data Products & Activate Data 3. PPC & SEO Use Cases 2. Product Principles
- 3. About Chris: Acquisition & Analytics at Bergzeit Digital Marketer Data Nerd Climber 1997 2008 2010 2022 Dad of 2 Online Store for Mountain Gear 145 M Revenue in FY 21/22 14 Countries, 5 Languages World-class, data-driven team 🔥 Hiring a PPC!
- 4. I’d like to Set Clear Expectations for This Session No BigQuery Intro Data Management-Talk What this session IS What it’s NOT No BigQuery tactics No ML Focus Google Cloud & dbt focus Data Product mindset 75% PPC, 25% SEO
- 5. Large-Scale PPC is Becoming the Science of Managing Data Pipelines RECAP FROM SMX 2019 Recap of My SMX 2019 Talk
- 6. Is Your Data Scattered Across Dozens of Islands?
- 7. If You’re Done With This… Get A Data Warehouse Connectors
- 8. Data Warehouse = The New Center of Gravity? Source: https://mikkeldengsoe.substack.com/p/future-of-the-data-warehouse
- 9. You Need Clean & Reliable Data for Data Products
- 10. Everybody has a Data Horror Story To Tell
- 11. Modern Data Stack: BigQuery & dbt at the core Data Products
- 12. dbt is a Programming Environment for SQL
- 13. Are there Alternatives to dbt? Not really No extra setup No control Saved queries Google Dataprep No SQL (no-code) Transformation & scheduling only Google Dataform Free for GCP Smaller ecosystem SQL framework
- 14. The dbt community is rapidly growing
- 15. Wait: What about Javascript and Python? Easy to get started Instantly ready Javascript Python Powerful libraries Leading data tools SQL Super scalable Ideal for production Centralized code Only for smaller data tasks Harder to centralize
- 16. 1. Intro Plan: Build Data Products & Activate Your Data 3. PPC & SEO Use Cases 2. Product Principles
- 17. Principles for Robust Data Products (with dbt)
- 18. #1: Structured Data Transformation
- 19. Models Break your Transformations into Small Steps Sources Exposures (Data Products) Stage Intermediate Marts
- 20. Use Jinja Macros to Template Your SQL Code Jinja-Macro Compiled SQL https://docs.getdbt.com/docs/building-a-dbt-project/jinja-macros
- 22. Only Write Production Code with Version Control
- 23. Use JIRA-Tickets to Create Feature Branches
- 24. #3: Sufficient Data Test Coverage
- 25. Typical Characteristics of Data Quality
- 26. Define Data Quality Through Automated Tests dbt core (5) dbt utils (15) dbt expectations (~60) custom tests (*)
- 27. #4: Deployment & Environments
- 28. Separate Environments for Test and Production Test = Paper cup Production = Porcelain
- 30. Keep Your Documentation Close To Your Code Documentation in .yml and .md Files Document all Sources, Models and Exposures Get documentation as HTML YAML File for Model Metadata Markdown File for Description SQL Files with Model code
- 31. Get Auto-Generated Project Documentation
- 32. Example Documentation for Source & Exposure Exposure (Output) Source (Input)
- 33. #6: Logs & Alerts
- 34. Production Updates and Tests are run as Jobs
- 35. Job Failures are logged as Alerts to Slack/Teams Example Alert Failure
- 36. Result: Our BigQuery-focused GCP Architecture CSV File Enrichment select * Transformation Ingestion Web APIs ML Model Dashboard Storage
- 37. 1. Intro Let’s Finally Dig Into Some Use Cases! 3. PPC & SEO Use Cases 2. Product Principles
- 38. PPC Use Case: No Consent, No Ads Conversions? ?
- 39. Our Simple Transaction Backend Tracking gclid Conversion value Timestamp orderIdentifier Example Python Code (see SMX 2020): https://gist.github.com/ChrisGutknecht
- 40. Identify New Conversions Missing in GA SQL transformation steps in dbt (called “DAGs”) Final output for Conversion Import
- 41. Yes We can! Upload Missing Ads Conversions
- 42. Combining Tag and Import Conversions in GAds Full conversion coverage, no modeling 👍
- 43. SEO Use Case: Sitemap Efficiency Report Elias Dabbas’ advertools Package and sitemap_to_df()
- 44. Combine Sitemap with Crawl and GSC Data SQL transformation Result of combined data (30 days)
- 45. Bonus: Spy on a Competitor Sitemap with 4 LOC
- 46. PPC Use Case: Sync Product Ratings to Shopping How do I get these (for free)?
- 47. Datainput: You Need A Simple Review Feed review-level data sku-level data: GTIN, sku, url
- 48. The Flow Diagram for the Shopping Ratings Sync
- 49. The CTR results are not conclusive (but, hey ⭐ )
- 50. Want to Sync Your Ratings? Test my Notebook! Notebook on Github: https://gist.github.com/ChrisGutknecht
- 51. Use Case: Custom Large-Scale Link Checker Didn’t work for us! (Errors, no control, little customisation, Java-based)
- 52. Multi-Threaded URL-Crawling with Advertools Elias Dabbas’ advertools and adv.crawl(url_list, …) Fast crawling Customised input
- 53. Filter Relevant Columns From Crawl Dataframe Elias Dabbas’ advertools and adv.crawl(url_list, …)
- 54. The Flow Diagram for Our Link Checker Notebook on Github: https://gist.github.com/ChrisGutknecht
- 55. PPC Use Case: Inventory-Based Campaigns
- 56. PPC Use Case: Inventory-Based Campaigns Turn product feed data into campaign structures
- 57. We went from This Complex Graph in Dataprep
- 58. To This SQL-based Transformation in dbt better control for complex transformations easier to extend and reuse parts of logic lower cost in long run automated testing and documentation
- 59. There Are More Data Products in the Lab… (Join Our Team!)
- 60. In The End: Do You Want to be an Architect?
- 61. Your Takeaways from this Session 1. What the modern data stack is and why it’s exciting 3. Which principles to apply for data products in production 2. Why dbt is the best tool for data warehouse transformation 4. A few interesting PPC and SEO use cases for you to try
- 62. Thanks for Your Time. Looking Forward To Questions! Chris Gutknecht | Teamlead A&O | Hiring a PPC!