Cloudera Federal Forum 2014: Tracking Provenance in Hadoop Clusters
8 likes4,261 views
The document outlines the capabilities and architecture of Cloudera Navigator, focusing on features such as metadata discovery, data lineage tracing, and auditing access to datasets in Hadoop. It provides details on how to query metadata, manage data lineage through various systems, and utilize a REST API for interactions. Additionally, it discusses the architecture model for elements and relations within the system, and includes examples demonstrating the usage of the functionality.
![Sidecars.. (Colocation of associated metadata)
/user/root/customers/cust_demo
/user/root/customers/.cust_demo.navigator
Contents of .cust_demo.navigator
{
"properties" : {
"secret" : "true",
"retention" : "small"
},
"tags" : ["pci"]
}](https://image.slidesharecdn.com/trackingprovenanceinhadoopclusters-140211152937-phpapp02/85/Cloudera-Federal-Forum-2014-Tracking-Provenance-in-Hadoop-Clusters-12-320.jpg){kind=tracking}
![Lineage (Hive Query)
INSERT OVERWRITE TABLE machine_vendors
SELECT upper(trim(regexp_extract(ms.dmidecode,"System InformationntManufacturer: ([^n]+)",1))) AS manufacturer,upper
(trim(regexp_extract(ms.dmidecode,"System InformationntManufacturer: ([^n]+)ntProduct Name: ([^n]+)",2))) AS product,ca.
address_state,ca.customerKey,cm.clusterId,ms.machineName
FROM crm_accounts ca JOIN cluster_metadata cm
ON ca.customerKey = cm.customerKey JOIN machine_stats ms
ON cm.customerKey = ms.customerKey AND cm.clusterId = ms.clusterId AND cm.collectionTS = ms.collectionTS](https://image.slidesharecdn.com/trackingprovenanceinhadoopclusters-140211152937-phpapp02/85/Cloudera-Federal-Forum-2014-Tracking-Provenance-in-Hadoop-Clusters-13-320.jpg){kind=tracking}
![Discovery & Lineage (REST API)
●
Elements Resource
○
curl 'http://localhost:5150/api/v1/elements?query=originalName:job_&limit=100&offset=100'
[{
"identity" : "513bf7add8d5f56b7f0f34769707cb5f",
"originalName" : "job_1389320017591_0024_conf.xml",
"firstClassParentId" : null,
"name" : null,
"description" : null,
"tags" : null,
"properties" : null,
"fileSystemPath" : "/user/history/done/2014/01/31/000000/job_1389320017591_0024_conf.xml",
"category" : "FILE",
"size" : 139211,
"lastModified" : "1969-12-31T23:59:59.999Z",
"lastAccessed" : "2014-02-04T02:12:01.369Z",
"owner" : "root",
"group" : "hadoop",
"blockSize" : null,
"mimeType" : "application/octet-stream",
"replication" : null,
"deleted" : false,
"resType" : "HDFS",
"permission" : 432,
"resId" : "858e5548b4cd3457432eb491ee74729d",
"type" : "fselement"
}, ...]
○
○
curl ‘http://localhost:5150/api/v1/elements/f53ae3547a90b7519b44041db1898972’
curl -X PUT -H "Content-Type: application/json" -d '{"displayName":"test","descriptin":"describe me","tags":[]}' http://localhost:
5150/api/v1/elements/e5f94cd59a8ca6df96247ce88b6c9c28](https://image.slidesharecdn.com/trackingprovenanceinhadoopclusters-140211152937-phpapp02/85/Cloudera-Federal-Forum-2014-Tracking-Provenance-in-Hadoop-Clusters-23-320.jpg){kind=tracking}
![Discovery & Lineage (REST API)
●
Relations Resource
curl 'http://localhost:7187/api/v1/relations?elementIds=83f4cdcc37c379144fef22e3dbdf7c8c&types=PARENT_CHILD&depth=2'
[{
"identity" : “91540192d3dd727f912b3c0bb91cdd81”,
"type" : “PARENT_CHILD",
"parent" : [ {
"elementId" : "83f4cdcc37c379144fef22e3dbdf7c8c",
},"children" : [ {
"elementIds" : [ "6144fabee63641275c5577697f16266a" ],
}
"name" : null},...]
●
Interactive Resource
curl 'http://localhost:7187/api/v1/interactive/elements?query=originalName:test&limit=2'
{
"offset" : 0,
"totalMatched" : 2,
"limit" : 1,
"results" : [ {
"identity" : "9b7b9d95eb06ccf0b1b0cd1a39642889",
"category" : "DIRECTORY",... },
"facets" : { },
"qtime" : 10
}](https://image.slidesharecdn.com/trackingprovenanceinhadoopclusters-140211152937-phpapp02/85/Cloudera-Federal-Forum-2014-Tracking-Provenance-in-Hadoop-Clusters-24-320.jpg){kind=tracking}
Cloudera Federal Forum 2014: Tracking Provenance in Hadoop Clusters
- 1. Cloudera Navigator Headline Goes Here Speaker Name or Subhead Goes Here DO NOT USE PUBLICLY PRIOR TO 10/23/12
- 3. Capabilities ● Discovery ○ ○ ● Lineage ○ ○ ● Search through metadata to find data set/operation of interest. View schema, associated metadata etc. for a dataset Given a data set, trace back to the original source. Understand the impact of modifying a data set. Audit ○ ○ Generate report of access to a data set in Hadoop. Generate alert when a restricted data set is accessed.
- 4. Discovery & Lineage(Questions to be asked?) ● ● ● Ad-hoc or only predefined? Granularity? Analysis?
- 5. Discovery & Lineage (Supported Systems) ● ● ● ● ● ● ● HDFS Hive MR1 Oozie Pig YARN ...More coming...
- 11. Discovery (Search on associated metadata)
- 12. Sidecars.. (Colocation of associated metadata) /user/root/customers/cust_demo /user/root/customers/.cust_demo.navigator Contents of .cust_demo.navigator { "properties" : { "secret" : "true", "retention" : "small" }, "tags" : ["pci"] }
- 13. Lineage (Hive Query) INSERT OVERWRITE TABLE machine_vendors SELECT upper(trim(regexp_extract(ms.dmidecode,"System InformationntManufacturer: ([^n]+)",1))) AS manufacturer,upper (trim(regexp_extract(ms.dmidecode,"System InformationntManufacturer: ([^n]+)ntProduct Name: ([^n]+)",2))) AS product,ca. address_state,ca.customerKey,cm.clusterId,ms.machineName FROM crm_accounts ca JOIN cluster_metadata cm ON ca.customerKey = cm.customerKey JOIN machine_stats ms ON cm.customerKey = ms.customerKey AND cm.clusterId = ms.clusterId AND cm.collectionTS = ms.collectionTS
- 14. Lineage
- 18. Lineage (Pig Script) posts = LOAD 'stackoverflow/posts/posts.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage() AS (id:int, postTypeId:int, acceptedAnswerId:int, parentId:int, creationDate:chararray, score:int, viewCount:int, body:chararray, ownerUserId:chararray, lastEditorUserId:int, lastEditorDisplayName:chararray, lastEditDate:chararray, lastActivityDate:chararray, tile:chararray, tags:chararray, answerCount:int, commentCount:int, favoriteCount:int, closedDate: chararray, communityOwnedDate:chararray); comments = LOAD 'stackoverflow/comments/comments.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage() AS (id:int, postId:int, score:int, text:chararray, creationDate:chararray, userDisplayName:chararray, userId: int); joined_post_comments = JOIN posts by id, comments by postId; post_comments = FOREACH joined_post_comments GENERATE posts::id..posts::communityOwnedDate, comments::postId..comments::userId; grouped_comments = GROUP post_comments BY posts::id; comments_per_post = FOREACH grouped_comments GENERATE group as postId, post_comments.comments::text as comment; rmf stackoverflow/output/comments_per_post STORE comments_per_post INTO 'stackoverflow/output/comments_per_post' USING PigStorage();
- 19. Lineage (Pig)
- 20. Discovery & Lineage Architecture
- 21. Model ● ● Generic (Element, Relations) Element ○ ○ ○ Unique Identity Key-value pairs Tags (Operation, Operation Execution, FSElement, Table, Column…)
- 22. Model (Contd…) ● Relation ○ ○ ○ Unique Identity Two sets of related elements Relationship type (Parent Child Relation, Data Flow Relation, Control Flow Relation, Instance Of Relation, Alias Relation, Generic Relation)
- 23. Discovery & Lineage (REST API) ● Elements Resource ○ curl 'http://localhost:5150/api/v1/elements?query=originalName:job_&limit=100&offset=100' [{ "identity" : "513bf7add8d5f56b7f0f34769707cb5f", "originalName" : "job_1389320017591_0024_conf.xml", "firstClassParentId" : null, "name" : null, "description" : null, "tags" : null, "properties" : null, "fileSystemPath" : "/user/history/done/2014/01/31/000000/job_1389320017591_0024_conf.xml", "category" : "FILE", "size" : 139211, "lastModified" : "1969-12-31T23:59:59.999Z", "lastAccessed" : "2014-02-04T02:12:01.369Z", "owner" : "root", "group" : "hadoop", "blockSize" : null, "mimeType" : "application/octet-stream", "replication" : null, "deleted" : false, "resType" : "HDFS", "permission" : 432, "resId" : "858e5548b4cd3457432eb491ee74729d", "type" : "fselement" }, ...] ○ ○ curl ‘http://localhost:5150/api/v1/elements/f53ae3547a90b7519b44041db1898972’ curl -X PUT -H "Content-Type: application/json" -d '{"displayName":"test","descriptin":"describe me","tags":[]}' http://localhost: 5150/api/v1/elements/e5f94cd59a8ca6df96247ce88b6c9c28
- 24. Discovery & Lineage (REST API) ● Relations Resource curl 'http://localhost:7187/api/v1/relations?elementIds=83f4cdcc37c379144fef22e3dbdf7c8c&types=PARENT_CHILD&depth=2' [{ "identity" : “91540192d3dd727f912b3c0bb91cdd81”, "type" : “PARENT_CHILD", "parent" : [ { "elementId" : "83f4cdcc37c379144fef22e3dbdf7c8c", },"children" : [ { "elementIds" : [ "6144fabee63641275c5577697f16266a" ], } "name" : null},...] ● Interactive Resource curl 'http://localhost:7187/api/v1/interactive/elements?query=originalName:test&limit=2' { "offset" : 0, "totalMatched" : 2, "limit" : 1, "results" : [ { "identity" : "9b7b9d95eb06ccf0b1b0cd1a39642889", "category" : "DIRECTORY",... }, "facets" : { }, "qtime" : 10 }
- 27. Audit View
- 28. Audit Details ● User ○ ● Operation Information ○ ● Username, Impersonator, Ip Address Operation Type, Session Id, Query Id, Operation Text, Status, Time Object Information ○ ServiceName, Path (Different in different systems)