Don’t Bring Old Problems to Your New Cloud Data Warehouse
0 likes57 views
This document discusses best practices for migrating data from on-premises systems to the cloud. It recommends developing a multi-phase migration strategy, considering different approaches to migration, putting the right skills in place, and using data integration tools to improve data quality and analytics capabilities rather than just moving data. The case study of Crowley Maritime is presented, which faced challenges integrating legacy mainframe data into its cloud data warehouse until it used Precisely Connect to transform and replicate the data in real-time.
Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 1. Fern Halper Ph.D. VP and Senior Director, TDWI Advanced Analytics November 18, 2020 Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 2. SPONSORS 2
- 3. FERN HALPER VP, Senior Research Director for Advanced Analytics TDWI
- 4. What we will talk about today • How and why organizations are moving to the cloud • Best practices for your cloud data migration/integration strategy • Data integration solutions to avoid common problems of integrating legacy data to cloud data warehouses
- 5. THE WHY AND HOW OF CLOUD ADOPTION
- 6. 10% 12% 18% 18% 19% 23% 27% 30% 36% 38% 44% 49% 68% 86% 95% 21% 18% 17% 25% 31% 35% 25% 21% 36% 21% 27% 22% 13% 6% 3% 0% 20% 40% 60% 80% 100% 120% Audio data Video data Image still data Clickstream data Machine generated data (e.g., from sensors,… Real-time event streams Geospatial data External text data Semi-structured data Internal text data Time series data Log data Demographic data Transactional data Structured data WHAT KIND OF DATA ARE YOU CURRENTLY MANAGING? LOOKING TO MANAGE IN THE NEXT YEAR? Manage now Manage in next year (Source: TDWI 2020 survey, n=~100) • Data becoming more complex • Analytics is becoming more advanced • Past reports and dashboards • More self-service • Iterative, compute intensive
- 7. More organizations are looking to move to the cloud for analytics and other use cases Source: 2020 TDWI survey, n=~100 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Object storage in the cloud NoSQL DBMSs Spark Analytic DBMSs Data integration platforms in the cloud Data lake on premises Open source software (except ubiquitous… Containers (e.g., Docker) Non-relational DBMS Data lake in the cloud Data warehouse in the cloud Hadoop Analytics tools in the cloud Data integration platforms on premises Data warehouse on premises Analytics tools on premises Relational DBMS What data management and analytics tools does your organization currently use? Please select all that apply.
- 8. CLOUD BENEFITS Scalability/ Elasticity • Scales up/scale down New Data • Capture and analyze diverse cloud generated data Agility • Separates compute from storage • New innovations • Enables focus on value added activities New Analytics • Traditional DW often has trouble supporting AA • Cloud good for compute intensive analytics • Numerous benefits to cloud • COVID is accelerating change
- 9. What we see at TDWI • Different reasons for moving to the cloud • Move to a multi-platform environment • Different broad strategies for cloud adoption – Many adopting a cloud first strategy and for new projects – Others moving all but sensitive data workloads to the cloud – Still others utilizing cloud for advanced analytics workloads
- 10. Challenges • Talent • Data migration • Data governance • Building cloud strategies • Dealing with legacy systems
- 11. BEST PRACTICES FOR CLOUD MIGRATION
- 12. Develop a multi- phase strategy Consider approaches and plan Put talent in place Don’t just move: improve
- 13. Develop a multi- phase strategy • Develop the strategy – Segment the work into manageable pieces that minimizes risk – Involve different parts of the organization in the plan – Each piece has a technical goal that adds value – Devise a technology roadmap • Additionally – Avoid “big bang” projects – Align on platform
- 14. Consider approaches and plan • Approaches – Lift and shift – Minimal changes – Modernize? • Plan – Steps include assess, prioritize, architecture design, development, migration, test (iterate), parallel run, acceptance
- 15. Put skills in place • Talent requirements – Architects – Engineers – Cloud expertise • Additionally – Legacy systems • Solutions that can natively understand and access legacy data – Apps expertise for apps that are migrated – Consider consultants but make sure there is training before they leave
- 16. Don’t just move: improve! • Move – Consider data loss – Ensure integrity – Apps, processes, people will move too • Improve – Enhance data quality/trust – Enhance analytics – The experience – Architecture – Improve governance with a holistic plan
- 17. Summary Trends Best Practices • Develop a multi-phase strategy • Consider migration approaches and plan • Put the right talent in place • Don’t just move: improve! • Organizations using new data types for new analytics • More often, they are moving to cloud for scalability/flexibility to support new use cases • There are different use cases and broad strategic principles employed
- 18. THANK YOU!
- 19. Ashwin Ramachandran Senior Product Manager Precisely
- 20. Crowley Maritime Attaining enterprise data access with Precisely Connect 2020 Ashwin Ramachandran | Senior Product Manager
- 21. The global leader in data integrity Trust your data. Build your possibilities. Our data integrity software and data enrichment products deliver accuracy and consistency to power confident business decisions. Brands you trust, trust us Data leaders partner with us of the Fortune 100 90 Customers in more than 100 2,000 employees customers 12,000 countries Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 22. Connecting today’s infrastructure with tomorrow’s technology to unlock the potential of all your enterprise data. Extract, Transform, Load Data Replication via CDC High-performance ETL for Apache Spark, Cloud, Windows, Linux, Unix and Hadoop MapReduce Real-time database replication to streaming platforms, Cloud, databases and data warehouses Connect Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 23. Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse • Founded in 1892 • Global leader in Marine Solutions, Energy and Logistics Services • Over 5,000 employees worldwide • Provides services and staffing solutions for Shipping, LNG Distribution and Port Escort operations • Leading provider of logistics for U.S. Federal and NATO agencies
- 24. Business Mandate 1. Connect customer and shipment information that resides on multiple systems of record 2. Improve integration and replication of changes to meet real-time demand Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 25. IT Mandate 1. Modernize IT environments by moving data from on-prem systems to the cloud 2. Lower costs and dependency on mainframe system Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 26. Challenges • Legacy data formats (e.g., mainframe and IBM i) are not readily readable and consumable within cloud data warehouses – Binary data formats – REDEFINEs in data fields – Complex record structures • Making data movement between on- premises and the cloud data warehouse was slow and delayed • Complex integration architectures, made it harder to connect to the cloud data warehouse Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse
- 27. Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse +Connect
- 28. Precisely + TDWI: Don’t Bring Old Problems to Your New Cloud Data Warehouse Results 1 Integrate and replicate hundreds of z/OS tables to Snowflake 2 Power customer dashboards for “near real-time” reporting 3 Improve productivity with hourly dashboard updates for employees 4 Provide shipment information in a way that no other competitor can
- 31. CONTACT INFORMATION If you have further questions or comments: Fern Halper, TDWI fhalper@tdwi.org @fhalper Ashwin Ramachandran aramachandran@precisely.com tdwi.org
- 32. Thanks to Our Sponsors 3