ETL Practices for Better or Worse
1 like657 views
This document discusses ETL practices and opportunities for improving data integration processes. It presents ELT and RIT approaches to extract, load, and transform data in Hadoop/MPP systems for better performance and scalability. While data modeling is still important, the document questions how to balance normalization with ease of querying for analytics. Integration is noted as key to bringing value from distributed data sources, and challenges of unique identifiers and cross-referencing data are discussed. The document also emphasizes best practices like profiling, prototyping, deploying to sandboxes before production, and ensuring tools for performance monitoring, problem detection and education are in place.
ETL Practices for Better or Worse
- 1. ETL Practices for Better or Worse opportunities and challenges along with the rise of machines
- 2. Agenda 1. Bring the data together and then transform 2. Data modeling for pragmatism 3. Let the datasets talk to each other 4. Beef up sandbox and backfill 5. Performance tuning 6. Examples, questions and comments
- 3. ELT and RIT ● ELT = ○ Extract from Source ○ Load to HDFS/MPP ○ Transform and Integrate inside Hadoop/MPP ● RIT = ○ Replicate message/log to Queue (JMS/Kafka/CDC) ○ Stream from Queue and Ingest to HDFS/MPP ○ Transform and Integrate inside Hadoop/MPP
- 4. Why ELT and RIT (instead of ETL) ● Store related raw data together for better leverage ● Big queryable staging area is quite useful ● Reduce workload impact against source systems ● Write data cleansing and business logic in similar languages/scripting used by BI
- 5. Is Data Modeling still Important ● Do we still need to model the data in the era of NoSQL and Big Data? ● Shall we de-normalize/pre-join everything? ● Shall we use hierarchical JSON/XML and/or Key-Value pair for everything? ● Balance and trade-off: analytics, reusability, metadata-driven, size vs. easy-to-query
- 6. Is Data Modeling still Important ● Cluster all attributes and children objects to the tree structure (thinking in NoSQL way) ● Can we live without JOIN operator? ● Is mutable/updatable dataset still useful? ● Why not snapshot everything? Why SCD2? ● Is relational-model outdated? ● Model it in source or fix it in report? ● All or none: Index, Hash, and Full Scan?
- 7. Integration Brings the True Value ● Like the idea of SOA, be careful with DQ ● Data producers are loosely-coupled for the sake of scalability ● Integration and cross-reference is deferred to DW/BI layer, yet someone has to do it ● How can unique identifier help here? ● Replicate dim/ref/lkp and Federate tx/event
- 8. Profiling Prototype Deploy Backfill ● Profiling data to understand data ● Prototype with real data ● Enrich and harden the derived data in sandbox before deploying to production ● Be ready to backfill the data because it will happen (easier to produce or to consume?)
- 9. Self-service is Cool, but ● Strong automation tools must be built first ● Software can monitor and throttle ● If a user’s job gets killed, there needs to be enough info/clue to explain why and how to (try to) fix it ● Education and knowledge sharing is essential, is wiki page/runbook good enough
- 10. Performance Matters ● Good instrumentation/logging will pay off big time in performance tuning ● Can we run complex OLAP reports on top of operational metadata? Where needs the tuning the most? ● Swapping (spill to disk) can be a big issue ● Detect and kill the bad jobs early
- 11. Examples 1. data exploration in MPP/Hadoop instead of in source systems (don’t let brain wait) 2. web click stream backend transaction 3. replicate/synchronize rollup hierarchy (mapping lookup) to multiple data systems; then produce near-real-time aggregation in each system; finally federate the aggregates