Apache Eagle in Action
Download as PPTX, PDF9 likes1,533 views
The document provides an overview of Apache Eagle, an open source distributed real-time monitoring and alerting engine for Hadoop from eBay. It discusses Eagle's architecture, which includes real-time data collection, a distributed policy engine using stream processing, scalable data storage and querying, and machine learning integration. The document also covers Eagle's ecosystem, how it can be used for security, and integration with tools like Ambari, Docker, and Ranger.
![Apache Eagle - Distributed Real-time Policy Engine
17
METADATA MANAGER
Distributed Streaming Cluster Environment
Real Time
Alerts
Alerts
Policy
Management
Policy
Dynamical Policy Deployment
Usability
• Powerful SQL-Like CEP CQL for
Policy Definition
• Dynamical Poilcy Metadata Lifecycle
Management (Deployment/Update)
• Easy-to-use Policy management and
Alert analytics UI
from metricStream[(name == 'ReplLag')
and (value > 1000)] select * insert into
outputStream;](https://image.slidesharecdn.com/apacheeagleinactionv0-151129150608-lva1-app6891/85/Apache-Eagle-in-Action-17-320.jpg)
![Apache Eagle - Distributed Real-time Policy Engine
18
Full-function Streaming CEP CQL: Siddhi on Storm by default
hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user,
count(timestamp) as aggValue group by user having aggValue >= 5 insert into outputStream;
• Filter
• Join
• Aggregation: Avg, Sum , Min, Max, etc
• Group by
• Having
• Stream handlers for window: TimeWindow, Batch Window, Length Window
• Conditions and Expressions: and, or, not, ==,!=, >=, >, <=, <, and arithmetic operations
• Pattern processing
• Sequence processing
• Event Tables: intergrate historical data in realtime processing
• SQL-Like Query: Query, Stream Definition and Query Plan compilation](https://image.slidesharecdn.com/apacheeagleinactionv0-151129150608-lva1-app6891/85/Apache-Eagle-in-Action-18-320.jpg)
)
HBase
Key Distribution Statistics
(Online/Offline)
Realtime Partition
Strategy
Key Statistics Cache
Async
Strategy
• Greedy (Online/Offline)
• PoTC
• PKG
• Hashing](https://image.slidesharecdn.com/apacheeagleinactionv0-151129150608-lva1-app6891/85/Apache-Eagle-in-Action-21-320.jpg)
![Apache Eagle - Scalable Data Storage and Query
26
• Entity Metadata on large-scale NoSQL
storage like HBase
• Full-function SQL-Like REST Query
• Optimized rowkey design for time-series
monitoring data
• HBase Coprocessor
• Secondary Index
@Table("alertdef")
@ColumnFamily("f")
@Prefix("alertdef")
@Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME)
@JsonIgnoreProperties(ignoreUnknown = true)
@TimeSeries(false)
@Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"})
@Indexes({
@Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true),
})
public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{
@Column("a")
private String desc;
@Column("b")
private String policyDef;
@Column("c")
private String dedupeDef;
query=
AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}](https://image.slidesharecdn.com/apacheeagleinactionv0-151129150608-lva1-app6891/85/Apache-Eagle-in-Action-26-320.jpg)
Apache Eagle in Action
- 1. Apache in Action Secure Hadoop in Real-time Hao Chen / 陈浩 / hao@apache.org / @haozch
- 2. Who is the Guy 2 Co-creator, Committer and PMC @ Apache Eagle hao@apache.org Hao Chen / 陈浩 Sr. Software Engineer @ eBay Cloud Service hchen9@ebay.com Speaker @ Hadoop Summit (SJC, SHA, BJ) ... http://people.apache.org/~hao
- 3. Agenda 3 • About Eagle • Architecture • Ecosystem • Q & A
- 4. What’s Apache Eagle 4 Apache Eagle is a distributed real-time monitoring and alerting engine for hadoop from eBay Open sourced as Apache Incubator Project on Oct 26th 2015 Secure Hadoop in Realtime a data activity monitoring solution to instantly identify access to sensitive data, recognize attacks/ malicious activity and block access in real time. See http://eagle.incubator.apache.org or http://goeagle.io
- 5. Apache Eagle History Donated to Apache Software Foundation (ASF) from eBay at Oct 26th, 2015 5 Dec 2013 Oct 23 2015 Oct 26 2015 Hadoop Eagle Project Initiative Apache Incubator eagle.incubator.apache.org Github Open Source github.com/apache/incubator-eagle Hadoop Eagle Production Release May 2014
- 6. Why build Apache Eagle 6 Eagle was initialized by end of 2013 for hadoop ecosystem monitoring as any existing tool like zabbix, ganglia can not handle the huge volume of metrics/logs generated by hadoop system in eBay. 2013/2014 10,000 nodes 150,000+ cores 170 PB 2000+ user 3000+ nodes 10,000+ cores 50+ PB 2012 2011 1000+ nodes 10,000+ cores 10+ PB 100+ nodes 1000 + cores 1 PB 2010 2009 50+ nodes 2007 1-10 nodes Hadoop Data • Security • Activity Hadoop Platform • Heath • Availability • Performance Hadoop @ eBay Inc
- 7. Apache Eagle @ eBay 7 7 CLUSTERS 7427 NODES 160 PB DATA 10 B+ EVENTS / DAY 500+ METRIC TYPES 50,000+ JOBS / DAY 50,000,000+ TASKS / DAY MONITOR PROCESS
- 8. Agenda 8 • About Eagle • Architecture • Ecosystem • Q & A
- 9. Apache Eagle Architecture Overview 9 Scalable Scales to monitor thousands of policies and billions of access events Machine Learning Create dynamic user profiles based on user behavior Real-time Generates alerts in real time and blocks users with malicious intent Extensible Eagle can be easily extended to monitor other data sources
- 10. Apache Eagle Architecture Overview 10 STREAM PROCESSING ENGINE DataCollector Kafka HDFS, Audit, Security METADATA MANAGER DATASTORES REMEDIATION ENGINE Apache Ranger MACHINE LEARNING MODULE Custom module Actionable Alerts Activities Actionable Alerts PolicyThresholdsUser properties MLThresholds Real Time Alert Dashboard Security Analyst Admin Console Security Engineer Insights Metadata Management MACHINE LEARNING TRAINING MODULE Policy Engine
- 11. Apache Eagle Architecture Features 11 • Real-time Data Collection • Distributed Policy Engine • Stream Processing DSL • Scalable Data Storage & Query • Machine Learning Intergration NOTE {NAME}-{NUMBER} like HDFS-6914 means open source project ticket id contributed by us
- 12. Apache Eagle - Data Collection 12 Decoupling with Message Bus • Apache Kafka: high-throughput distributed messaging • Partition: balance between logic and throughput Cross-Platform Integration • Community Kafka Client (18+) • Python/Go/C/C++/JAVA .. • Enhanced Log4j-kafka • KAFKA-2041: Extensible Partition Key • KAFKA-2077: Advanced Topic Selector
- 13. Apache Eagle - Data Collection 13 Availability: Filebeat + Logstash Logstash-1 Logstash-2 Logstash-… Logstash- Ligh-weight collector (golang) with daemon Logstash instances cluster Shuffle Grouping KafkaField Grouping Distributed Message Bus Resource consumption balance Message throughput balance ( LOGSTASH-179)
- 14. Storm Spout: Distributed crawling for hadoop job, node jmx and service logs, etc. Zookeeper: Centralized state management and distributed locking Apache Eagle - Data Collection 14 Scalability: Distributed Real-time Ingestion Zookeeper METRIC Centralized State Management JOB EVENT LOG
- 15. Apache Eagle - Distributed Real-time Policy Engine 15 METADATA MANAGER Distributed Streaming Cluster Environment AlertExecutor_{1} AlertExecutor_{2} … AlertExecutor_{N} Real Time Alerts Alerts Policy Management Policy Dynamical Policy Deployment Real-time Event Stream Stream_{1} Stream_{*} Dynamical Stream Schema Stream Processing Highlights • Real-time • Usability • Scalability • Extensibility • Metadata-driven
- 16. Apache Eagle - Distributed Real-time Policy Engine 16 METADATA MANAGER Real Time Alerts Alerts Policy Management Policy Event Stream (Kafka) Dynamical Stream Schema Dynamical Policy Deployment Real-time • Kafka-based Distributed Message Bus (Extensible) • Storm-based Real-time Execution Environment (Extensible) • Stream events are processed and alerts are evaluated during streaming
- 17. Apache Eagle - Distributed Real-time Policy Engine 17 METADATA MANAGER Distributed Streaming Cluster Environment Real Time Alerts Alerts Policy Management Policy Dynamical Policy Deployment Usability • Powerful SQL-Like CEP CQL for Policy Definition • Dynamical Poilcy Metadata Lifecycle Management (Deployment/Update) • Easy-to-use Policy management and Alert analytics UI from metricStream[(name == 'ReplLag') and (value > 1000)] select * insert into outputStream;
- 18. Apache Eagle - Distributed Real-time Policy Engine 18 Full-function Streaming CEP CQL: Siddhi on Storm by default hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue >= 5 insert into outputStream; • Filter • Join • Aggregation: Avg, Sum , Min, Max, etc • Group by • Having • Stream handlers for window: TimeWindow, Batch Window, Length Window • Conditions and Expressions: and, or, not, ==,!=, >=, >, <=, <, and arithmetic operations • Pattern processing • Sequence processing • Event Tables: intergrate historical data in realtime processing • SQL-Like Query: Query, Stream Definition and Query Plan compilation
- 19. Apache Eagle - Distributed Real-time Policy Engine 19 Distributed Streaming Cluster Environment AlertExecutor_{1} AlertExecutor_{2} … AlertExecutor_{N} Stream_{1} Stream_{*} Stream Processing Scalability: dynamic policy partition by {event} * {policy} • N Users with 3 partitions, M policies with 2 partitions, then 3*2 physical tasks • Physical partition + policy-level partition
- 20. Apache Eagle - Distributed Real-time Policy Engine 20 Distributed Streaming Partition Problem https://en.wikipedia.org/wiki/Partition_problem S = {3,1,1,2,2,1,1} S1 = {1,1,1,1,1} S2 = {2,2} S3 = {3}
- 21. Apache Eagle - Distributed Real-time Policy Engine 21 Distributed Streaming Partition Strategy groupBy[ GreedyStrategy ]((_.key1,_.key2 )) HBase Key Distribution Statistics (Online/Offline) Realtime Partition Strategy Key Statistics Cache Async Strategy • Greedy (Online/Offline) • PoTC • PKG • Hashing
- 22. Apache Eagle - Distributed Real-time Policy Engine 22 Distributed Real-time Policy Engine Siddhi CEP Policy Evaluator Machine Learning Policy Evaluator Extensibility • Support WSO2 Siddhi CEP as first class • Extensible policy engine implementation • Extensible policy lifecycle management Extensible Policy Evaluator public interface PolicyEvaluatorServiceProvider { public String getPolicyType(); // literal string to identify one type of policy public Class getPolicyEvaluator(); // get policy evaluator implementation public List getBindingModules(); // policy text with json format to object mapping } public interface PolicyEvaluator { public void evaluate(ValuesArray input) throws Exception; // evaluate input event public void onPolicyUpdate(AlertDefinitionAPIEntity newAlertDef);// policy update public void onPolicyDelete(); // invoked when policy is deleted } METADATA MANAGER Policy/Metadata
- 23. Apache Eagle - Distributed Real-time Policy Engine 23 Metadata-Driven • Stream Schema: AlertStreamSchemaEntity • Policy Definition: AlertDefinitionAPIEntity • Central metadata management • Dynamic metadata deployment @Table("alertdef") @ColumnFamily("f") @Prefix("alertdef") @Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME) @JsonIgnoreProperties(ignoreUnknown = true) @TimeSeries(false) @Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"}) @Indexes({ @Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true), }) public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{ @Column("a") private String desc; @Column("b") private String policyDef; @Column("c") private String dedupeDef; METADATA MANAGER Distributed Real-time Policy Engine Dynamic Metadata Loading
- 24. .flatMap(AuditLogTransformer) .groupBy(_.user) .flatMap(UserProfileAggregator); Apache Eagle - Fluent Stream Processing DSL 24 env.fromKafka (KafkaConfig) .alert.persistAndEmail val env = ExecutionEnvironment.getStorm() env.execute(); Distributed Streaming Cluster Environment AlertExecutor_{1} AlertExecutor_{2} … AlertExecutor_{N} Alerts Real-time Event Stream Stream_{1} Stream_{*} Stream Processing env.execute()
- 25. Apache Eagle - Fluent Stream Processing DSL 25 • Physical execution platform independent • Easily assemble data transformation, filtering, join and alerting DAG in fluent way • DAG rewrite and optimization • StreamUnionExpansion • StreamGroupbyExpansion • StreamNameExpansion • StreamAlertExpansion • StreamParallelismConfigExpansion trait StreamProducer{ filter flatMap map{1,2,3,4} groupBy streamUnion // stream join is hard, not implemented for storm alertWithConsumer } StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); env.newSource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1) .flatMap(new AuditLogTransformer()) .groupBy(0) .flatMap(new UserProfileAggregatorExecutor()); .alertWithConsumer(“userActivity“,”userProfileExecutor“) env.execute(); Optimizer 1. Development 2. Optimization 3. Compile to native app
- 26. Apache Eagle - Scalable Data Storage and Query 26 • Entity Metadata on large-scale NoSQL storage like HBase • Full-function SQL-Like REST Query • Optimized rowkey design for time-series monitoring data • HBase Coprocessor • Secondary Index @Table("alertdef") @ColumnFamily("f") @Prefix("alertdef") @Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME) @JsonIgnoreProperties(ignoreUnknown = true) @TimeSeries(false) @Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"}) @Indexes({ @Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true), }) public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{ @Column("a") private String desc; @Column("b") private String policyDef; @Column("c") private String dedupeDef; query= AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}
- 27. Uniform rowkey design • Metric • Entity • Log Rowkey ::= Prefix | Partition Keys | timestamp | tagName | tagValue | … Rowkey ::= Metric Name | Partition Keys | timestamp | tagName | tagValue | … Rowkey ::= Default Prefix | Partition Keys | timestamp | tagName | tagValue | … Rowkey ::= Log Type | Partition Keys | timestamp | tagName | tagValue | … Rowvalue ::= Log Content 27 Apache Eagle – Uniform HBase Rowkey Design
- 28. Apache Eagle - Machine Learning Intergration 28
- 29. 29 User Activity Profiling Offline: Determine bandwidth from training dataset the kernel density function parameters (KDE) Online: If a test data point lies outside the trained bandwidth, it is anomaly (Policy) PCs(Principle Components) in EVD (Eigenvalue Value Decomposition) Kernel Density Function Apache Eagle – User/System Activity Profiling
- 30. 30 Anomaly Metric Predictive Detection Offline: Analyzing and combining 500+ metrics together for causal anomaly detections (IG -> PCA -> GMM -> MCC) Online: Predictively alert for anomaly metrics Normal (Green) and Abnormal (Red) Data and Probability Distribution and Threshold Selection PCA (Principal Component Analysis) Apache Eagle - Anomaly Metric Predictive Detection Anomaly Metric Predictive Detection Case Study
- 31. Agenda • About Apache Eagle • Architecture • Ecosystem • Q & A 31
- 32. 32 Apps Security Hadoop Cloud Database Interface Web Portal REST Services Analytics Visualization Integration Ambari Docker Ranger Dataguise Eagle Framework Distributed real-time framework for efficiently developing highly scalable monitoring applications Eagle Apps Securiy / Hadoop / Cloud / Database Eagle Interface REST Service / Management UI / Customizable Analytics Visualization Eagle Integration Ambari / Docker / Ranger / Dataguise Open Source Community-driven and Cross-community cooperation Eagle Framework Apache Eagle Ecosystem
- 33. 33 Apache Eagle Ecosystem - Security How to Secure Hadoop in Realtime? • Apache Eagle • Apache Ranger • Apache Knox • Dataguise
- 34. 34 Apache Eagle Ecosystem - Hadoop Eagle in Apache Amabri: natively be part of hadoop ecosystem
- 35. 35 Apache Eagle Ecosystem - Docker Eagle in Docker: natively fly on Cloud/Container STORM KAFKA ZOOKEEPER HBASE HADOOP … Powered of git clone apache/incubator-eagle eagle-docker 15 + 1 = 1 docker pull apacheeagle
- 36. 36 Apache Eagle Ecosystem - Open Source If you want to go fast, go alone. If you want to go far, go together. -- African Proverb
- 37. Learn more about Apache Eagle 37 • EAGLE: USER PROFILE-BASED ANOMALY DETECTION IN HADOOP CLUSTER (IEEE) • EAGLE: DISTRIBUTED REALTIME MONITORING FRAMEWORK FOR HADOOP CLUSTER
- 38. Q & A apache/incubator-eagle @TheApacheEagle @ApacheEagle http://eagle.incubator.apache.org The slide is licensed under Creative Commons Attribution 4.0 International license.