Web Services Hadoop Summit 2012
9 likes6,237 views
This document discusses web services in Hadoop, including RESTful APIs that provide programmatic access to Hadoop components like HDFS, HCatalog, and job submission/monitoring. It describes the design goals of WebHDFS including supporting HTTP, high performance, cross-version compatibility, and security. Examples are given of using curl and wget to interact with HDFS files via WebHDFS URLs. The HCatalog REST API is also summarized, which allows creating, querying and managing Hadoop metadata. Finally, future work is mentioned around improving job management and authentication.
![Example: wget
• Use wget to open the same file
$wget "http://namenode:50070/webhdfs/v1/user/
szetszwo/w.txt?op=OPEN" –O w.txt
Resolving ...
Connecting to ... connected.
HTTP request sent, awaiting response...
307 TEMPORARY_REDIRECT
Location: http://192.168.5.2:50075/webhdfs/v1/user/
szetszwo/w.txt?op=OPEN&offset=0 [following]
Architecting the Future of Big Data Page 14](https://image.slidesharecdn.com/webserviceshadoopsummit2012-120615140023-phpapp01/85/Web-Services-Hadoop-Summit-2012-14-320.jpg)
![Example: wget (2)
--2012-06-13 01:42:10-- http://192.168.5.2:50075/
webhdfs/v1/user/szetszwo/w.txt?op=OPEN&offset=0
Connecting to 192.168.5.2:50075... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21 [application/octet-stream]
Saving to: `w.txt'
100%[=================>] 21 --.-K/s in 0s
2012-06-13 01:42:10 (3.34 MB/s) - `w.txt' saved
[21/21]
Architecting the Future of Big Data Page 15](https://image.slidesharecdn.com/webserviceshadoopsummit2012-120615140023-phpapp01/85/Web-Services-Hadoop-Summit-2012-15-320.jpg)
![HCatalog REST API
Get a list of all tables in the default database:
GET
http://…/v1/ddl/database/default/table
Hadoop/
HCatalog
{
"tables": ["counted","processed",],
"database": "default"
}
Indicate user with URL parameter:
http://…/v1/ddl/database/default/table?user.name=gates
Actions authorized as indicated user
© Hortonworks 2012
Page 18](https://image.slidesharecdn.com/webserviceshadoopsummit2012-120615140023-phpapp01/85/Web-Services-Hadoop-Summit-2012-18-320.jpg)
![HCatalog REST API
Create new table “rawevents”
PUT
{"columns": [{ "name": "url", "type": "string" },
{ "name": "user", "type": "string"}],
"partitionedBy": [{ "name": "ds", "type": "string" }]}
http://…/v1/ddl/database/default/table/rawevents
Hadoop/
HCatalog
{
"table": "rawevents",
"database": "default”
}
© Hortonworks 2012
Page 19](https://image.slidesharecdn.com/webserviceshadoopsummit2012-120615140023-phpapp01/85/Web-Services-Hadoop-Summit-2012-19-320.jpg)
![HCatalog REST API
Describe table “rawevents”
GET
http://…/v1/ddl/database/default/table/rawevents
Hadoop/
HCatalog
{
"columns": [{"name": "url","type": "string"},
{"name": "user","type": "string"}],
"database": "default",
"table": "rawevents"
}
© Hortonworks 2012
Page 20](https://image.slidesharecdn.com/webserviceshadoopsummit2012-120615140023-phpapp01/85/Web-Services-Hadoop-Summit-2012-20-320.jpg)
Web Services Hadoop Summit 2012
- 1. Web Services in Hadoop Nicholas Sze and Alan F. Gates @szetszwo, @alanfgates Page 1
- 2. REST-ful API Front-door for Hadoop • Opens the door to languages other than Java • Thin clients via web services vs. fat-clients in gateway • Insulation from interface changes release to release HCatalog web interfaces MapReduce Pig Hive HCatalog External HDFS HBase Store © 2012 Hortonworks Page 2
- 3. Not Covered in this Talk • HttpFS (fka Hoop) – same API as WebHDFS but proxied • Stargate – REST API for HBase © 2012 Hortonworks Page 3
- 4. HDFS Clients • DFSClient: the native client – High performance (using RPC) – Java blinding • libhdfs: a C++ client interface – Using JNI => large overhead – Also Java blinding (require Hadoop installing) Architecting the Future of Big Data Page 4
- 5. HFTP • Designed for cross-version copying (DistCp) – High performance (using HTTP) – Read-only – The HTTP API is proprietary – Clients must use HftpFileSystem (hftp://) • WebHDFS is a rewrite of HFTP Architecting the Future of Big Data Page 5
- 6. Design Goals • Support a public HTTP API • Support Read and Write • High Performance • Cross-version • Security Architecting the Future of Big Data Page 6
- 7. WebHDFS features • HTTP REST API – Defines a public API – Permits non-Java client implementation – Support common tools like curl/wget • Wire Compatibility – The REST API will be maintained for wire compatibility – WebHDFS clients can talk to different Hadoop versions. Architecting the Future of Big Data Page 7
- 8. WebHDFS features (2) • A Complete HDFS Interface – Support all user operations – reading files – writing to files – mkdir, chmod, chown, mv, rm, … • High Performance – Using HTTP redirection to provide data locality – File read/write are redirected to the corresponding datanodes Architecting the Future of Big Data Page 8
- 9. WebHDFS features (3) • Secure Authentication – Same as Hadoop authentication: Kerberos (SPNEGO) and Hadoop delegation tokens – Support proxy users • A HDFS Built-in Component – WebHDFS is a first class built-in component of HDFS. – Run inside Namenodes and Datanodes • Apache Open Source – Available in Apache Hadoop 1.0 and above. Architecting the Future of Big Data Page 9
- 10. WebHDFS URI & URL • FileSystem scheme: webhdfs:// • FileSystem URI: webhdfs://<HOST>:<HTTP_PORT>/<PATH> • HTTP URL: http://<HOST>:<HTTP_PORT>/webhdfs/v1/<PATH>?op=.. – Path prefix: /webhdfs/v1 – Query: ?op=.. Architecting the Future of Big Data Page 10
- 11. URI/URL Examples • Suppose we have the following file hdfs://namenode:8020/user/szetszwo/w.txt • WebHDFS FileSystem URI webhdfs://namenode:50070/user/szetszwo/w.txt • WebHDFS HTTP URL http://namenode:50070/webhdfs/v1/user/ szetszwo/w.txt?op=.. • WebHDFS HTTP URL to open the file http://namenode:50070/webhdfs/v1/user/ szetszwo/w.txt?op=OPEN Architecting the Future of Big Data Page 11
- 12. Example: curl • Use curl to open a file $curl -i -L "http://namenode:50070/webhdfs/v1/user/ szetszwo/w.txt?op=OPEN" HTTP/1.1 307 TEMPORARY_REDIRECT Content-Type: application/octet-stream Location: http://192.168.5.2:50075/webhdfs/v1/user/ szetszwo/w.txt?op=OPEN&offset=0 Content-Length: 0 Server: Jetty(6.1.26) Architecting the Future of Big Data Page 12
- 13. Example: curl (2) HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 21 Server: Jetty(6.1.26) Hello, WebHDFS user! Architecting the Future of Big Data Page 13
- 14. Example: wget • Use wget to open the same file $wget "http://namenode:50070/webhdfs/v1/user/ szetszwo/w.txt?op=OPEN" –O w.txt Resolving ... Connecting to ... connected. HTTP request sent, awaiting response... 307 TEMPORARY_REDIRECT Location: http://192.168.5.2:50075/webhdfs/v1/user/ szetszwo/w.txt?op=OPEN&offset=0 [following] Architecting the Future of Big Data Page 14
- 15. Example: wget (2) --2012-06-13 01:42:10-- http://192.168.5.2:50075/ webhdfs/v1/user/szetszwo/w.txt?op=OPEN&offset=0 Connecting to 192.168.5.2:50075... connected. HTTP request sent, awaiting response... 200 OK Length: 21 [application/octet-stream] Saving to: `w.txt' 100%[=================>] 21 --.-K/s in 0s 2012-06-13 01:42:10 (3.34 MB/s) - `w.txt' saved [21/21] Architecting the Future of Big Data Page 15
- 16. Example: Firefox Architecting the Future of Big Data Page 16
- 17. HCatalog REST API • REST endpoints: databases, tables, partitions, columns, table properties • PUT to create/update, GET to list or describe, DELETE to drop • Uses JSON to describe metadata objects • Versioned, because we assume we will have to update it: http://hadoop.acme.com/templeton/v1/… • Runs in a Jetty server • Supports security – Authentication done via kerberos using SPNEGO • Included in HDP, runs on Thrift metastore server machine • Not yet checked in, but you can find the code on Apache’s JIRA HCATALOG-182 © 2012 Hortonworks Page 17
- 18. HCatalog REST API Get a list of all tables in the default database: GET http://…/v1/ddl/database/default/table Hadoop/ HCatalog { "tables": ["counted","processed",], "database": "default" } Indicate user with URL parameter: http://…/v1/ddl/database/default/table?user.name=gates Actions authorized as indicated user © Hortonworks 2012 Page 18
- 19. HCatalog REST API Create new table “rawevents” PUT {"columns": [{ "name": "url", "type": "string" }, { "name": "user", "type": "string"}], "partitionedBy": [{ "name": "ds", "type": "string" }]} http://…/v1/ddl/database/default/table/rawevents Hadoop/ HCatalog { "table": "rawevents", "database": "default” } © Hortonworks 2012 Page 19
- 20. HCatalog REST API Describe table “rawevents” GET http://…/v1/ddl/database/default/table/rawevents Hadoop/ HCatalog { "columns": [{"name": "url","type": "string"}, {"name": "user","type": "string"}], "database": "default", "table": "rawevents" } © Hortonworks 2012 Page 20
- 21. Job Management • Includes APIs to submit and monitor jobs • Any files needed for the job first uploaded to HDFS via WebHDFS – Pig and Hive scripts – Jars, Python scripts, or Ruby scripts for UDFs – Pig macros • Results from job stored to HDFS, can be retrieved via WebHDFS • User responsible for cleaning up output in HDFS • Job state information stored in ZooKeeper or HDFS © 2012 Hortonworks Page 21
- 22. Job Submission • Can submit MapReduce, Pig, and Hive jobs • POST parameters include – script to run or HDFS file containing script/jar to run – username to execute the job as – optionally an HDFS directory to write results to (defaults to user’s home directory) – optionally a URL to invoke GET on when job is done POST http://hadoop.acme.com/templeton/v1/pig Hadoop/ HCatalog {"id": "job_201111111311_0012",…} © 2012 Hortonworks Page 22
- 23. Find all Your Jobs • GET on queue returns all jobs belonging to the submitting user • Pig, Hive, and MapReduce jobs will be returned GET http://…/templeton/v1/queue?user.name=gates Hadoop/ HCatalog {"job_201111111311_0008", "job_201111111311_0012"} © 2012 Hortonworks Page 23
- 24. Get Status of a Job • Doing a GET on jobid gets you information about a particular job • Can be used to poll to see if job is finished • Used after job is finished to get job information • Doing a DELETE on jobid kills the job GET http://…/templeton/v1/queue/job_201111111311_0012 Hadoop/ HCatalog {…, "percentComplete": "100% complete", "exitValue": 0,… "completed": "done" } © 2012 Hortonworks Page 24
- 25. Future • Job management – Job management APIs don’t belong in HCatalog – Only there by historical accident – Need to move them out to MapReduce framework • Authentication needs more options than kerberos • Integration with Oozie • Need a directory service – Users should not need to connect to different servers for HDFS, HBase, HCatalog, Oozie, and job submission © 2012 Hortonworks Page 25