Hadoop Summit 2012 | A New Generation of Data Transfer Tools for Hadoop: Sqoop 2

Editor's Notes

• #2: Apache Sqoop was created to efficiently transfer bulk data between Hadoop and external structured datastores because databases are not easily accessible by Hadoop. The popularity of Sqoop in enterprise systems confirms that Sqoop does bulk transfer admirably. That said, to enhance its functionality, Sqoop needs to fulfill data integration use-cases as well as become easier to manage and operate.
• #4: It’s a Apache TLP now
• #6: Different connectors interpret these options differently. Some options are not understood for the same operation by different connectors, while some connectors have custom options that do not apply to others. Confusing for users and detrimental to effective use.Some connectors may support a certain data format while others don’tRequired to use common JDBC vocabulary (URL, database, table, etc.)Cryptic and contextual command line arguments can lead to error-prone connector matching, resulting in user errorsDue to tight coupling between data transfer and the serialization format, some connectors may support a certain data format that others don't (e.g. direct MySQL connector can't support sequence files)There are security concerns with openly shared credentialsBy requiring root privileges, local configuration and installation are not easy to manageDebugging the map job is limited to turning on the verbose flagConnectors are forced to follow the JDBC model and are required to use common JDBC vocabulary (URL, database, table, etc), regardless if it is applicable
• #7: Sqoop 1’s challenges will be addressed by Sqoop 2 Architecture Connector only focuses on connectivitySerialization, format conversion, Hive/HBase integration should be uniformly available via frameworkRepository Manager: creates/edits connectors, connections, jobsConnector Manager: register new connectors, enables/disables connectorsJob Manager: submit new jobs to MR, gets job progress, kills specific jobs
• #8: Kathleen to take over.Sqoop 2 is a work in progress and you are welcome to join our weekly conference calls discussing its’ design and implementation. Please see me afterwards for more details if interested. During our discussions we’ve identified three pain points for Sqoop 2 to address. Those of you who have used Sqoop will find those points apparent: ease of use, ease of extension, and security.
• #9: Pause for Q. Kathleen to take over.Sqoop 2 is a work in progress and you are welcome to join our weekly conference calls discussing its’ design and implementation. Please see me afterwards for more details if interested. During our discussions we’ve identified three pain points for Sqoop 2 to address. Those of you who have used Sqoop will find those points apparent: ease of use, ease of extension, and security.
• #10: There are 4 points we want to address with Sqoop 2’s ease of use.
• #11: Like other client-side tools, Sqoop 1 requires everything – connectors, root privileges, drivers, db connectivity – to be installed and configured locally.
• #12: Sqoop 2 will be a service and as such you can install once and then run everywhere. This means that connectors will be configured in one place, managed by the Admin role and run by the Operator role, which will be discussed in detail later. Likewise, JDBC drivers will be in one place and database connectivity will only be needed on the server.Sqoop as a web-based service, exposes the REST APIFront-ended by CLI and browserBack-ended by a metadata repositoryExample of document based system is couchbase.Sqoop 1 has something called a sqoopmetastore, which is similar to a repository for metadata but not quite. That said, the model of operation for Sqoop 1 and Sqoop 2 is very different: Sqoop 1 was a limited vocabulary tool while Sqoop 2 is more metadata driven. The design of Sqoop 2’s metadata repository is such that it can be replaced by other providers.
• #13: Sqoop 1 was intended for power users as evidenced by its CLI. Sqoop 2 has two modes: one for the power user and one for the newbie. Those new to Sqoop will appreciate the interactive UI, which walks you through import/export setup, eliminating redundant/incorrect options. Various connectors are added in one place, with connectors exposing necessary options to the Sqoop framework and with the user only required to provide info relevant to their use-case.Not bound by terminal, well documented return codes
• #14: In Sqoop 1, Hive and HBase require local installation. Currently Oozie launches Sqoop by bundling it and running it on the cluster, which is error-prone and difficult to debug.
• #15: With Sqoop 2, Hive and HBase integration happens not from client but from the backend. Hive does not need to be installed on Sqoop at all. What Sqoop will do is submit requests to the HiveServer over the wire. Exposing a REST API for operation and management will help Sqoop integrate better with external systems such as Oozie.Oozie and Sqoop will be decoupled: if you install a new Sqoop connector then don’t need to install it in Oozie also.Hive will not invoke anything in Sqoop, while Oozie does invoke Sqoop so the REST API does not benefit Hive in any way but it does benefit OozieWhich Hive/HBase server the data will be put into is the responsibility of the reduce phase which will have its own configuration and since both these systems have are on Hadoop - we don't need any added security besides passing down the Kerberos principal
• #16: 4 pts
• #17: Pause for questions.
• #19: Because Sqoop is heavily JDBC centric, it’s not easy to work with non relational dbCouchbase implementation required different interpretationInconsistencies between connectors
• #20: Two-phases: first, transfer; second, transform/integration with other componentsOption to opt-out of downstream processing (i.e. revert to Sqoop 1)Trade-off between ease of connector/tooling development vs faster performanceSeparating data transfer (Map) from data transform (Reduce) allows connectors to specializeConnectors benefit from a common framework of functionality – don’t have to worry about forward compabilityFunctionally, Sqoop 2 is a superset of Sqoop 1 but does it in a different way Too early in the design process to tell if the same CLI commands could be used but most likely not primarily because it is a fundamentally incompatible changeReduce phase limited to stream transformations (no aggregation to start with)
• #21: Former is running MySQL b/c specifying driver option prevents the MySQL connector from working i.e. would end up using generic JDBC connector
• #22: Based on the URL which is JDBC centric and something Sqoop 2 is moving away from in the connect string used to access the database, Sqoop attempts to predict which driver it should load.What are connectors?Plugin components based on Sqoop’s extension frameworkEfficiently transfer data between Hadoop and external storeMeant for optimized import/export or don’t support native JDBCBundled connectors: MySQL, PostgreSQL, Oracle, SQLServer, JDBCHigh-performance data transfer: Direct MySQL, Direct PostgreSQL
• #23: Cryptic and contextual command line arguments can lead to error-prone connector matching, resulting in user errors. Due to tight coupling between data transfer and the serialization format, some connectors may support a certain data format that others don't (e.g. direct MySQL connector can't support sequence files).
• #24: With the user making an explicit connector choice in Sqoop 2, it will be less error-prone and more predictable. Connectors are no longer forced to follow the JDBC model and are no longer required to use common JDBC vocabulary (URL, database, table, etc), regardless if it is applicable.
• #25: Common functionality will be abstracted out of connectors, holding them responsible only for data transport. The reduce phase will implement common functionality, ensuring that connectors benefit from future development of functionality.
• #27: Pause for questions. Bilung to take over.
• #30: No code generation, no compilation allows Sqoop to run where there are no compilers, which makes it more secure by preventing bad code from runningPreviously required direct access to Hive/HBaseMore secure because routed through Sqoop server rather than opening up access to all clients to perform jobs
• #32: Connection is only for external systems
• #34: No need to disable user in database