Cloud Log Analysis and Visualization
22 likes4,762 views
The document presents a comprehensive overview of cloud-based logging and visualization techniques, emphasizing the use of visualization to address security challenges. Key components include data collection, processing, and visualization methods, as well as specific tools and APIs available for implementing these solutions. The author, Raffael Marty, shares insights on utilizing the cloud for log management and the importance of effective visualization in interpreting large sets of security data.
![Search
http://[domain].loggly.com/api/search?q=404
{
"data": [
{
"indexed": "2010-07-03T17:17:38.909Z",
"ip": "75.101.249.172",
"text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au]
[|domain] (DF)",
"inputname": "logglyweb",
"timestamp": "2010-07-03 10:17:38"
},
{
"indexed": "2010-07-03T17:17:37.879Z",
"ip": "75.101.249.172",
"text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au]
[|domain] (DF)",
"inputname": "logglyapp",
"timestamp": "2010-07-03 10:17:37"
},
...
Logging as a Service 30 (c) by Raffael Marty
Tuesday, July 6, 2010](https://image.slidesharecdn.com/rmll2010loganalysisandvis-100706075711-phpapp01/85/Cloud-Log-Analysis-and-Visualization-30-320.jpg)
![Parser
Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF)
Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF)
Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF)
(.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+):
(d+.d+.d+.d+).?(d*) [<>]
Regex / Parser (d+.d+.d+.d+).?(d*): (.*)
Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF)
Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF)
(CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF)
Logging as a Service 31 (c) by Raffael Marty
Tuesday, July 6, 2010](https://image.slidesharecdn.com/rmll2010loganalysisandvis-100706075711-phpapp01/85/Cloud-Log-Analysis-and-Visualization-31-320.jpg)
![Visualize
Parser AfterGlow Grapher
CSV file Graph file
digraph structs {
graph [label="AfterGlow 1.5.8", fontsize=8];
node [shape=ellipse, style=filled,
Configuration fontsize=10, width=1, height=1,
fixedsize=true];
edge [len=1.6];
color.source=“green” if ($fields[0] ne “d”)
"aaelenes" -> "Printing Resume" ;
cluster.target=regex_replace("(d+).")."/8" "abbe" -> "Information Encryption" ;
threshold.event=5 "aanna" -> "Patent Access" ;
size.target=$fields[1] "aatharuv" -> "Ping" ;
}
http://afterglow.sf.net
Logging as a Service 32 (c) by Raffael Marty
Tuesday, July 6, 2010](https://image.slidesharecdn.com/rmll2010loganalysisandvis-100706075711-phpapp01/85/Cloud-Log-Analysis-and-Visualization-32-320.jpg)
![<script type="text/javascript">
Google Vis Code
google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']});
google.setOnLoadCallback(call);
var trends = new Array();
function call() {
l!
a
$.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100",
n
type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx',
io
success: function(data) {
trends = data.data
drawChart();
c t
n
}
u
});
f
}
t
function drawChart() {
o
var data = new google.visualization.DataTable();
n
data.addColumn('string', 'Search');
data.addColumn('datetime', 'Date');
is
data.addColumn('number', 'Count');
e
data.addRows(trends);
od
var chart = new google.visualization.MotionChart(document.getElementById('chart_div'));
c
chart.draw(data, {width: 600, height:300, state:state});
is
var view = new google.visualization.DataView(data);
h
view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}]));
T
var table = new google.visualization.Table(document.getElementById('test_dataview'));
table.draw(view, {sortColumn: 1});
var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline'));
time.draw(timedata, {displayAnnotations: true});
}
</script>
Logging as a Service 35 (c) by Raffael Marty
Tuesday, July 6, 2010](https://image.slidesharecdn.com/rmll2010loganalysisandvis-100706075711-phpapp01/85/Cloud-Log-Analysis-and-Visualization-35-320.jpg)
Cloud Log Analysis and Visualization
- 1. Cloud-based Log Analysis and Visualization RMLL 2010, Bordeaux, France mobile-166 My syslog Raffael Marty - @zrlram Tuesday, July 6, 2010
- 2. Raffael (Raffy) Marty • Founder @ • Chief Security Strategist and Product Manager @ Splunk • Manager Solutions @ ArcSight • Intrusion Detection Research @ IBM Research • IT Security Consultant @ PriceWaterhouse Coopers Applied Security Visualization Publisher: Addison Wesley (August, 2008) ISBN: 0321510100 Logging as a Service 2 (c) by Raffael Marty Tuesday, July 6, 2010
- 3. Agenda •Introduction •Do it Yourself •Visualization •AfterGlow •Google Visualization API •InfoViz Process •Visualization Use-Cases •Visualization Tools •Visualization Resources •The Cloud •Loggly Logging as a Service 3 (c) by Raffael Marty Tuesday, July 6, 2010
- 4. Open Your Eyes Logging as a Service 4 (c) by Raffael Marty Tuesday, July 6, 2010
- 5. Security Is About Seeing Logging as a Service 5 (c) by Raffael Marty Tuesday, July 6, 2010
- 6. Goals - Learn how you can - use visualization to help solve security problems - leverage the cloud to build security visualization tools Logging as a Service 6 (c) by Raffael Marty Tuesday, July 6, 2010
- 7. Information Visualization? A picture is worth a thousand log records. Inspire Explore and Discover Answer a Pose a New Increase Communicate Support Question Question Efficiency Information Decisions Logging as a Service 7 (c) by Raffael Marty Tuesday, July 6, 2010
- 8. Visualization and The Cloud 8 Tuesday, July 6, 2010
- 9. InfoViz Process Collect Process Visualize •large-scale data collection •Your parsers •Visualization Tools •and processing •Standard formats •and Libraries Logging as a Service 9 (c) by Raffael Marty Tuesday, July 6, 2010
- 10. Collect 10 Tuesday, July 6, 2010
- 11. Log Management • Log Collection and Centralization • Log Storage • Log Filtering • Log Aggregation • Log Search and Extraction • Log Retention and Archiving Logging as a Service 11 (c) by Raffael Marty Tuesday, July 6, 2010
- 12. Process 12 Tuesday, July 6, 2010
- 13. Standard Formats • Multiple formats Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S 1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group "internet_access_in" Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc: 81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00 TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556 • Log Standards ‣ CEE (cee.mitre.org) ‣ SDEE ‣ WELF ‣ IDMEF ‣ CBE ‣ XDAS Logging as a Service 13 (c) by Raffael Marty Tuesday, July 6, 2010
- 14. Normalization • Parsers “To analyze or separate (input, for example) into more easily processed components.” (answers.com) • Generate a common output format for vis-tools (e.g., CSV) • For example ‣ Regex /(d{1,3}.d{1,3}.d{1,3}.d{1,3})/g ‣ http://secviz.org/content/parser-exchange Logging as a Service 14 (c) by Raffael Marty Tuesday, July 6, 2010
- 15. Visualize 15 Tuesday, July 6, 2010
- 16. Choose Your Poison Logging as a Service 16 (c) by Raffael Marty Tuesday, July 6, 2010
- 17. Reporting vs. Visualization • Reporting Libraries • Visualization Libraries - HighCharts - TheJIT - Flot - Graphael - Google Chart API - Protovis - Open Flash Chart - ProcessingJS - Flare JavaScript vs. Flash vs. XYZ Logging as a Service 17 (c) by Raffael Marty Tuesday, July 6, 2010
- 18. HighCharts • Click-Through • On load - near real-time updates • AJAX data input via JSON • Zoom http://www.highcharts.com/ Logging as a Service 18 (c) by Raffael Marty Tuesday, July 6, 2010
- 19. Google Visualization API http://code.google.com/apis/visualization/interactive_charts.html • JavaScript • Based on DataTables() • Many graphs • Playground - http://code.google.com/apis/ajax/playground Logging as a Service 19 (c) by Raffael Marty Tuesday, July 6, 2010
- 20. ProtoVis • JavaScript based visualization library • Charting • Treemaps • BoxPlots • Parallel Coordinates • etc. http://vis.stanford.edu/protovis/ Logging as a Service 20 (c) by Raffael Marty Tuesday, July 6, 2010
- 21. TheJIT http://thejit.org/ • JavaScript InfoVis Toolkit • Interactive • Link Graphs Logging as a Service 21 (c) by Raffael Marty Tuesday, July 6, 2010
- 22. Processing •Visualization library •Java based •Interactive (event handling) •Number of libraries to -draw in OpenGL -read XML files -write PDF files •Processing JS -JavaScript -HTML 5 Canvas http://processingjs.org/ -Web IDE http://processing.org/ Logging as a Service 22 (c) by Raffael Marty Tuesday, July 6, 2010
- 23. Building Your Own 23 Tuesday, July 6, 2010
- 24. Build Your Own AfterGlow Loggly Regexes Google Vis Logging as a Service 24 (c) by Raffael Marty Tuesday, July 6, 2010
- 25. Data Collection in the Cloud 25 Tuesday, July 6, 2010
- 26. The (public) Cloud What it is Types • multi-tenancy • SaaS - Software • elastic • PaaS - Platform • “infinite” resources • IaaS - Infrastructure • pay as you go Benefits • self provisioning • No installation • No elaborate configurations It’s not • No maintenance • private data center • Great scalability • virtualization • 7x24 availability Logging as a Service 26 (c) by Raffael Marty Tuesday, July 6, 2010
- 27. LaaS - Logging as a Service • All your data in one place • Loggly manages your data (index, store, archive, etc.) • Extremely fast search across all your data • Data source agnostic (no parsers) • Data management • access control • data segregation • data overview and summaries • API access Logging as a Service 27 (c) by Raffael Marty Tuesday, July 6, 2010
- 28. Loggly Architecture Loggly Data Sources Clients user interface mobile-166 My syslog Data collection API Data access Proxies Distributed Indexers and Search Machines indexing and processing Distributed data store Logging as a Service 28 (c) by Raffael Marty Tuesday, July 6, 2010
- 29. Loggly APIs • URL format: http://wiki.loggly.com/api-documentation http://<subdomain>.loggly.com/api/<resource> • RESTful API HTTP Based - Access through: /api/<resource> •GET - read - JSON, XML, JSONP output •POST - create • Authentication •PUT - update - Basic auth •DELETE - delete - oAuth http://loggly.loggly.com/api/search/?q=error syslog to: User: guest / Password: loggly logs.loggly.com:514 Logging as a Service 29 (c) by Raffael Marty Tuesday, July 6, 2010
- 30. Search http://[domain].loggly.com/api/search?q=404 { "data": [ { "indexed": "2010-07-03T17:17:38.909Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au] [|domain] (DF)", "inputname": "logglyweb", "timestamp": "2010-07-03 10:17:38" }, { "indexed": "2010-07-03T17:17:37.879Z", "ip": "75.101.249.172", "text": "Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au] [|domain] (DF)", "inputname": "logglyapp", "timestamp": "2010-07-03 10:17:37" }, ... Logging as a Service 30 (c) by Raffael Marty Tuesday, July 6, 2010
- 31. Parser Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF) Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF) Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF) (.*) rule ([-d]+/d+)(.*?): (pass|block) (in|out) on (w+): (d+.d+.d+.d+).?(d*) [<>] Regex / Parser (d+.d+.d+.d+).?(d*): (.*) Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF) Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF) (CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF) Logging as a Service 31 (c) by Raffael Marty Tuesday, July 6, 2010
- 32. Visualize Parser AfterGlow Grapher CSV file Graph file digraph structs { graph [label="AfterGlow 1.5.8", fontsize=8]; node [shape=ellipse, style=filled, Configuration fontsize=10, width=1, height=1, fixedsize=true]; edge [len=1.6]; color.source=“green” if ($fields[0] ne “d”) "aaelenes" -> "Printing Resume" ; cluster.target=regex_replace("(d+).")."/8" "abbe" -> "Information Encryption" ; threshold.event=5 "aanna" -> "Patent Access" ; size.target=$fields[1] "aatharuv" -> "Ping" ; } http://afterglow.sf.net Logging as a Service 32 (c) by Raffael Marty Tuesday, July 6, 2010
- 33. AfterGlow Cloud Grapher Loggly JSON CSV DOT Graph Logging as a Service 33 (c) by Raffael Marty Tuesday, July 6, 2010
- 34. Google Vis • JSON to Graphs • DataTable - used among all charts • Interactivity through events Logging as a Service 34 (c) by Raffael Marty Tuesday, July 6, 2010
- 35. <script type="text/javascript"> Google Vis Code google.load('visualization', '1', {'packages':['motionchart', 'table', 'annotatedtimeline']}); google.setOnLoadCallback(call); var trends = new Array(); function call() { l! a $.ajax({ url: "http://logdog.loggly.com/api/search/?q=404&facets=True&buckets=100", n type:'GET', dataType: 'jsonp', username: 'xxxxx', password: 'xxxxxx', io success: function(data) { trends = data.data drawChart(); c t n } u }); f } t function drawChart() { o var data = new google.visualization.DataTable(); n data.addColumn('string', 'Search'); data.addColumn('datetime', 'Date'); is data.addColumn('number', 'Count'); e data.addRows(trends); od var chart = new google.visualization.MotionChart(document.getElementById('chart_div')); c chart.draw(data, {width: 600, height:300, state:state}); is var view = new google.visualization.DataView(data); h view.setRows(view.getFilteredRows([{column: 1, minValue: new Date(2007, 0, 1)}])); T var table = new google.visualization.Table(document.getElementById('test_dataview')); table.draw(view, {sortColumn: 1}); var time = new google.visualization.AnnotatedTimeLine(document.getElementById('timeline')); time.draw(timedata, {displayAnnotations: true}); } </script> Logging as a Service 35 (c) by Raffael Marty Tuesday, July 6, 2010
- 36. Visualization Use-Cases 36 Tuesday, July 6, 2010
- 37. NetFlow Visualization • Treemap • Protovis.JS • Size: Amount • Brightness: Variance • Color: Sensor • Shows: Scans - bright spots • Thanks to Chris Horsley Logging as a Service 37 (c) by Raffael Marty Tuesday, July 6, 2010
- 38. Firewall Treemap Logging as a Service 38 (c) by Raffael Marty Tuesday, July 6, 2010
- 39. Firewall Log Port Source IP Destination IP Logging as a Service 39 (c) by Raffael Marty Tuesday, July 6, 2010
- 40. Visualization Resources 40 Tuesday, July 6, 2010
- 41. http://secviz.org Share, discuss, challenge, and learn about security visualization. • List: secviz.org/mailinglist • Twitter: @secviz Logging as a Service 41 (c) by Raffael Marty Tuesday, July 6, 2010
- 42. Applied Security Visualization • Bridging the gap between security and visualization • Hands-on, end to end examples • Data processing and analysis Chapters • Visualization • Compliance • Data Sources • Insider Threat • From Data to Graphs • Visualization Tools Addison Wesley (August, 2008) • Perimeter Threat ISBN: 0321510100 Logging as a Service 42 (c) by Raffael Marty Tuesday, July 6, 2010
- 43. Thank You! raffael.marty@loggly.com @zrlram 43 Tuesday, July 6, 2010