Lessons Learned Migrating from IBM BigInsights to Hortonworks Data Platform
Download as PPTX, PDF2 likes627 views
The document summarizes lessons learned from migrating from IBM BigInsights to Hortonworks Data Platform (HDP). It discusses challenges with the IBM platform around interoperability, support, and compatibility. It then outlines the scope of work for the migration project, including moving batch jobs, scripts, data, and environments to HDP. Finally, it discusses lessons learned around securing access, ingest frameworks, stakeholder engagement, data migration, and operational maturity when transitioning platforms.
Lessons Learned Migrating from IBM BigInsights to Hortonworks Data Platform
- 1. Public Information Lessons Learned Migrating from IBM BigInsights to Hortonworks Data Platform Data Works Summit 2018 Lisa Coleman – Data and Analytics Infrastructure Robert Tucker – HDP Platform Administrator Mica Glover – HDP Platform Administrator May 2018
- 2. Public Information 2 OUR MISSION The mission of the association is to facilitate the financial security of its members, associates and their families through provision of a full range of highly competitive financial products and services; in so doing, USAA seeks to be the provider of choice for the military community. THE USAA STANDARD • Keep our membership and mission first • Live our core values: Service, Loyalty, Honesty, Integrity • Be authentic and build trust • Create conditions for people to succeed • Purposefully include diverse perspectives for superior results • Innovate and build for the future
- 3. Public Information 3 Challenges & Drivers Scope of Work Lessons Learned Agenda Public Information 3
- 4. Public Information 4 Platform Challenges and Drivers Interoperability Support Model Velocity of Change Compatibility IBM Strategy & Guidance In-place upgrades not possible without GPFS Stack integration – Platform Symphony, WLM, BigSQL Common BI tools in the industry not certified for connectivity via GPFS Limited ability to leverage Streaming Data Platforms as none were certified on our version of IBM BigInsights Difficult resolution process Evolution of documentation to support version upgrade Potential added expenses for professional services Inconsistency in stability of enhancements and break-fix releases Difficulty securing the proper resources for assistance Limits our ability to innovate and take advantage of new capabilities Slow adoption of new ODPi projects Public Information 4
- 5. Public Information 5 Apache Project Support Apache Project Usage Hortonworks 2.5 BigInsights 4.2 Required Capability Hadoop Hive Sqoop Ambari Spark Zeppelin Atlas Ranger Knox Tez Nifi Flume Kafka Pig Storm HBase Phoenix Solr Accumulo Core Components SQL on Hadoop Data Ingestion Cluster Management IoT, Streaming Data Science Governance Security Security Improve Hive Performance IoT Data Ingestion IoT, Streaming ETL IoT, Streaming NoSQL SQL Interface for Hbase Social Media / NLP NoSQL 2.7.3 1.2.1 1.4.6 2.4 1.6.2 AND 2.0* 0.6.0 0.7.0 0.6.0 0.9.0 0.7.0 1 1.5.2 0.10.0.1 1.2.0 1.0.1 1.1.2 4.7.0 5.2.2 1.7.0 2.7.2 1.2.1 1.4.6 2.2.0 1.6.1 N/A N/A 0.5.2 0.7.0 N/A N/A 1.6.0 0.9.0.1 0.15.0 N/A 1.2.0 4.6.1 5.5 N/A
- 6. Public Information 6 Hortonworks 2.5 BigInsights 4.2Capability Key Innovations Focus Operations Security / Governance Security Rolling Upgrade – Zero cluster downtime / Full cluster HA Express Upgrade – controlled cluster downtime Cluster preventative maintenance Role Based Access Control (RBAC) Basic Tag policy – Access and entitlements can be based on attributes Geo-based policy – Access policy based on location Time-based policy – Access policy based on time windows Prohibitions – Restrictions on combining two data sets which might be in compliance originally, but not when combined together Yes. Proven. GA multiple releases Yes. Proven. GA multiple releases Yes, Hortonworks SmartSense Yes, via Ranger Yes, via Atlas and Ranger Yes, via Atlas and Ranger Yes, via Atlas and Ranger Yes, via Atlas and Ranger No, equivalent offering New, Unproven No, equivalent offering Yes, via Ranger No, Atlas not part of distribution No, Atlas not part of distribution No, Atlas not part of distribution No, Atlas not part of distribution
- 7. Public Information 7 Scope of Work Batch Jobs & Scripts DDL replication Convert Posix to HDFS commands Covert Hive CLI to Beeline Convert existing Ingest Utilities Historic & Incremental Loads Re-evaluated all data assets Bulk Data loads required specialized configuration Data validation Hand-Off to Support Teams Phased Releases Planning with data support teams Parallel runs Execute phased turn-off Convert clients to use Knox and Hive Provision for HDP Readiness AD Integration Enable Kerberos Standard HDFS/Local filesystem layout Establish DB / FTP connections Transition Cut-Over & Retirement Data Migration Component Migration Environment Set Up Documentation Access provisioning Knowledge transfer Sign-off
- 8. Public Information 8 In-Scope Component Summary Hive Tables 4,700 Data Volume 500TB Env Files 233 Python Scripts 499 Linux Scripts 4,000Pig Scripts 243 Prod Jobs 7,356 Public Information 8
- 9. Public Information 9 Workload Transition Public Information 9
- 10. Public Information 10 Modified pathing/dir structure on HDFS Quality checks Networked additional set of nodes and leveraged HDFS client copy from local to move data (due to GPFS HDFS) Enterprise monitoring Code repository Automated code deploy Managed asset provisioning HA Services Lack of dedicated resources across enterprise required third party assistance Extensive knowledge transfer sessions to aid in transition Developed training plan Re-write all ingest utilities for HDFS Standardize metadata delivery to Atlas Standard asset request Adoption of data stewardship LDAP/AD integration Kerberos Ranger Knox – non Kerberos connectivity to Hive Lessons Learned Security and Access Ingest Framework Stakeholder Buy-In Table Data Migration Operational Maturity Transition from hive CLI to beeline Optimized file format (ORC/Parquet/Avro) Conversion of all scripts to acquire Kerberos ticket Transition from GPFS to HDFS Code Management
Editor's Notes
- #5: ODPi – open data platform initiative: goal was to contribute to open source collaboratively and provide interoperability across different distributions
- #8: Prework to evaluate all data prior to migration – removed the need to migrate some data •Delete any data you no longer need. •Delete duplicate data •Compress all data with Bzip2 •Identify which Hive tables do not need to be migrated •Delete any temporary tables •Understand who is consuming your data and how they are consuming it •Understand whether your project conforms to the Ingest Framework methodology HDFS client needed on BigInsights cluster Clients of SAS big consumers of BigSQL had to be converted Dedicated team to address a majority of the script conversions Post migration security model implementation required for some assets which did not have an assigned owner. KT – office hours, data consumption through Hive, search of available assets
- #11: Add Code section, e.g. transition from hive cli to beeline Add Ranger in Security section I removed code management section GPFS shielded USAA form having to implement Kerberos Memory management – JVMs, Spark Case mismatch between Unix accounts Code Migration Conversion of all scripts to acquire Kerberos tickets Hive cli to beeline GPFS to HDFS Python libraries Table Data Pathing changed Quality checks Networked additional set of nodes and leveraged HDFS client copy from local to move data