Database Tips & Tricks: Strategies for Detecting and Handling Schema Drift
0 likes4,976 views
The document presents strategies for detecting and managing schema drift, which is defined as changes in source or target data definitions that can lead to poor data quality and frustration for data professionals. It discusses coping mechanisms for both source and destination schema drift, emphasizing the importance of designing resilient workflows that can adapt to changes using tools like FME. Additionally, it provides insights into automating workflows through schema validation and introduces various licensing options for FME products.
Database Tips & Tricks: Strategies for Detecting and Handling Schema Drift
- 1. Schema Drift Database Tips & Tricks: Strategies for Detecting and Handling Schema Dri
- 2. Presenters Brian Pont Support Manager, Customer Success Annabelle Glass Product Owner, Desktop Authoring Dale Lutz Co-Founder
- 3. Agenda: ● Part 1: Schema Drift Defined ● Part 2: Coping With Source Schema Drift ● Part 3: Managing Destination Schema Drift ● Part 4: Designing for Schema Drift ● Part 5: Automating Schema Resilient Workflows
- 5. noun a change in source or target data definition. Example: “And now they’ve added another column to my excel spreadsheet -- this schema drift madness has to stop” Schema Drift:
- 6. Challenge: If schema drift is not handled properly, it can result in brittle and error-prone data workflows which lead to: ● Poor data quality ● Time spent troubleshooting ● Frustration & headaches
- 7. Data professionals should design their workflows to detect & handle schema drift. Results: ● Resilient workflows that can manage any changes ● Set it and forget it! Solution:
- 8. FME Desktop FME Server FME Cloud Build & Run Workflows Automate Workflows Hosted FME Server FME offers a powerful toolset for detecting and handling schema drift. FME® Integration Platform Connect. Transform. Automate.
- 9. What is Schema? Schema is the data model, including: ● Feature type (e.g. table or sheet) name ● Attribute (column or field) names ● Attribute data types (e.g. integer, float, date) ● Optional: Permitted geometries (e.g. point, line)
- 10. Schema Drift Scenarios ● Externally supplied CSV file suddenly has renamed fields ● Target database table has had columns renamed or removed ● CAD file has data on new layers ● Online service restructures its JSON feed
- 11. Coping with Source Schema Drift 2
- 12. Source Schema Drift: Demonstration What could possibly go wrong?
- 13. Guard Strategy 1 Use AttributeValidators to guard against missing attributes
- 14. Guard Strategy 2 Validate the Schema Before Loading
- 15. Reaction Playbook ● Update Reader & Refresh Schema ● Trace to see where any impacts might be ● Repair invalid transformers
- 17. Destination Schema Drift: Demonstration What could possibly go wrong?
- 18. Guard Strategy 1 ● Don’t worry, be happy
- 19. Guard Strategy 2 ● Build Schema Validation into your workflows
- 21. Design Strategy 1 Go Dynamic: ● Enables you to automatically handle Schema Drift Benefits: ● Flexible ● Easy to maintain KB Link: Tutorial: Dynamic Workflows
- 22. Design Strategy 2 Use a Schema Mapper: ● Utilize schema design from a spreadsheet ● Enables Non-FME users Demos: ● SchemaMapper and existing table ● SchemaMapper and dynamic mode KB Link: Tutorial: SchemaMapper Transformer
- 24. noun In FME Server: graphically created workflows that selectively invoke one or more steps in response to an event or trigger Automations:
- 25. Design Strategy ● Use Schema Validation Workspaces ○ One for source ○ One for destination ● Glue them together with an Automation ● Notify if something has changed and stop ● Only run the data load if all is well
- 29. Automation Magic
- 30. We'll keep you on track. Reimagine your workflows in Automations to do a schema dri check Your workflow is reliable, resilient, robust.
- 31. "FME sets the standard for support and is the leader by far". - Brad Very helpful and went above and beyond to help find a solution to my request! A+" - Justin "Thank you! Answered all my questions - AGAIN. You folks always seem to have a way of doing that." - Ray FREE RESOURCES WE’RE ALL ABOUT FME Community +10,000 Global Users Knowledge & Training Learn FME Technical Support Report a Problem Support cases Live Chat When in doubt… ask us! Articles & tutorials Live online training On-demand resources Ask how-to questions Engage with Safers Submit ideas for FME
- 32. LICENSING OPTIONS Enterprise Subscription Individual Licenses ● Perpetual licenses + Annual Maintenance ● Designed for smaller deployments Cloud Deployment ● Subscription program ● Ideal for cross-department deployments ● Unlock enterprise automation without the hassle of hardware ● Flexible pricing options including hourly and annual subscription More info: safe.com/pricing
- 33. Thank You! Get a free trial of FME at safe.com/trial
- 34. CREDITS Special thanks to: - Presentation template by SlidesBoss.com - Photographs by Pexels