Mt logging with_bam
Download as PPT, PDF0 likes400 views
The document describes the integration of WSO2 BAM with logging using Log4j, where logs are published to BAM, stored in Cassandra, and archived in HDFS daily. It details configuration settings for log publishers and analyzers, and outlines the advantages of the system, including asynchronous data publishing and high scalability. The setup provides tools for real-time and archived log event analysis for administrators.
![Stream Definition
streamId = dataPublisher.defineStream("{" + "'name':'log" + "." + tenantId +
"."
+ severName + "." + currDateStr + "'," + " 'version':'1.0.0',"
+ " 'nickName': 'Logs'," + " 'description': 'Logging Event',"
+ " 'metaData':[" + " {'name':'clientType','type':'STRING'}" + " ],"
+ " 'payloadData':[" + " {'name':'tenantID','type':'STRING'},"
+ " {'name':'serverName','type':'STRING'},"
+ " {'name':'appName','type':'STRING'},"
+ " {'name':'logTime','type':'LONG'},"
+ " {'name':'priority','type':'STRING'},"
+ " {'name':'message','type':'STRING'},"
+ " {'name':'logger','type':'STRING'},"
+ " {'name':'ip','type':'STRING'},"
+ " {'name':'instance','type':'STRING'},"
+ " {'name':'stacktrace','type':'STRING'}" + " ]" + "}");](https://image.slidesharecdn.com/mtloggingwithbam-121126041945-phpapp02/85/Mt-logging-with_bam-6-320.jpg)
Mt logging with_bam
- 1. MT-Logging with WSO2 BAM Amani Soysa Senior Software Engineer
- 2. How it Works?? Logs are published to BAM via Log4JAppender Published Logs are Stored in Cassandra (per tenant per day) When the day is over Logs are published to HDFS file system daily using BAM analyzers To view Logs :- Application Logs – Log events on web applications/services filter by the application name System Logs – View the system log of each product {With filtering categories} Log Viewers get logs from cassandra <<real time log>> and HDFS file system <<Archive logs>>
- 4. Log Publisher - Log4JAppender
- 5. Setting Up LogEventAppender log4j.rootLogger=INFO, LOGEVENT # LOGEVENT is set to be a LogEventAppender using a PatternLayout to send logs to LOGEVENT log4j.appender.LOGEVENT=org.wso2.carbon.logging.appender.LogEventAppender log4j.appender.LOGEVENT.url=tcp://localhost:7611 log4j.appender.LOGEVENT.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout log4j.appender.LOGEVENT.columnList=%T,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktrace log4j.appender.LOGEVENT.userName=admin log4j.appender.LOGEVENT.password=admin log4j.appender.LOGEVENT.processingLimit=1000 log4j.appender.LOGEVENT.maxTolerableConsecutiveFailure=20
- 6. Stream Definition streamId = dataPublisher.defineStream("{" + "'name':'log" + "." + tenantId + "." + severName + "." + currDateStr + "'," + " 'version':'1.0.0'," + " 'nickName': 'Logs'," + " 'description': 'Logging Event'," + " 'metaData':[" + " {'name':'clientType','type':'STRING'}" + " ]," + " 'payloadData':[" + " {'name':'tenantID','type':'STRING'}," + " {'name':'serverName','type':'STRING'}," + " {'name':'appName','type':'STRING'}," + " {'name':'logTime','type':'LONG'}," + " {'name':'priority','type':'STRING'}," + " {'name':'message','type':'STRING'}," + " {'name':'logger','type':'STRING'}," + " {'name':'ip','type':'STRING'}," + " {'name':'instance','type':'STRING'}," + " {'name':'stacktrace','type':'STRING'}" + " ]" + "}");
- 7. Log Analyzer
- 8. Logging-Config.xml Basic Information <isDataFromCassandra>true</isDataFromCassandra> Information Regarding Cassandra <cassandraHost>localhost:9160</cassandraHost> <userName>admin</userName> <password>admin</password> Information Regarding HDFS Server <archivedHost>hdfs://localhost:9000</archivedHost> <archivedHDFSPath>/stratos/logs</archivedHDFSPath>
- 9. Logging Summarizer Archive logs daily at a given time and store in hdfs file system
- 10. summarizer-config.xml Basic Information <isDataFromCassandra>true</isDataFromCassandra> <deleteColFamily>false</deleteColFamily> Information Regarding Cassandra <cassandraHost>localhost:9160</cassandraHost> <userName>admin</userName> <password>admin</password> Information Regarding BAM <publisherURL>localhost:7611</publisherURL> <userName>admin</userName> <password>admin</password> <cronExpression>0 0 1 ? * * *</cronExpression> Information Regarding HDFS Server <archivedHost>hdfs://localhost:9000</archivedHost> <archivedLogLocation>/stratos/logs</archivedLogLocation>
- 11. Advantages Asynchronous and None Blocking Data publishing Receives and Stores Log Events Cassandra Cluster which is high scalable and a big Data Repository Rich tools set for analytics Can be shared with CEP for real time Log Event analysis. Can provide Logging tool boxes and dashboards for system administrators using WSO2 BAM
- 12. Log Viewer Get Logs From Cassandra - <<Latest>> Get Archived Logs From HDFS File system
- 14. Log Viewer – Archived Logs
- 15. Log Viewer – Application Logs