Big Data Analytics for Real-time Operational Intelligence with Your z/OS Data
1 like815 views
The document discusses the challenges of big data analytics on z/OS mainframes and introduces Ironstream, a tool that normalizes and streams critical mainframe data to Splunk for real-time analysis. It outlines the limitations of legacy technologies in mainframe monitoring and emphasizes the need for high-performance, low-cost solutions that can integrate various data sources without requiring mainframe expertise. Additionally, the document highlights use cases for IT operational analytics, service intelligence, and security monitoring using Ironstream integrated with Splunk.
Big Data Analytics for Real-time Operational Intelligence with Your z/OS Data
- 1. 1 Big Data Analytics for Real-time Operational Intelligence with Your z/OS Data “Splunking” Your z/OS Mainframe Ed Hallock Director, Product Management
- 2. Housekeeping • Webcast Audio: – Today’s webcast audio is streamed through your computer speakers. – If you need technical assistance with the web interface or audio, please reach out to us using the chat window. • Questions Welcome: – Submit your questions at any time during the presentation using the chat window. – We will answer them during our Q&A session following the presentations. • Recording and Slides: – This webcast is being recorded. You will receive an email following the webcast with a link to download both the recording and the slides. © 2016 Syncsort Incorporated
- 3. Agenda 3 Syncsort Confidential and Proprietary - do not copy or distribute Big Iron to Big Data Analytics Challenge “Splunking” Your Mainframe Data Introducing Ironstream® Ironstream for z/OS and Enterprise Security Ironstream for IT Operations Analytics Ironstream for IT Service Intelligence Ironstream apps on Splunkbase and the Ironstream Starter Edition Q&A
- 4. Big Iron to Big Data Analytics Challenge So many data sources – SMF, Syslog, Log4j web and application logs, RMF, RACF, USS files and standard datasets Volume of data – Millions of SMF records generated daily Format of data – Complex data structures (SMF) with headers, product sections, data sections, variable length and self-describing – EBCDIC not recognized outside of the mainframe world – Binary flags and fields Difficult to get the information in a timely manner – Not real-time, typically have to wait overnight for an offload 4 Syncsort Confidential and Proprietary - do not copy or distribute
- 5. What Has Been Done in the Past? Performance Monitors – Proactively analyze and manage z/OS operating systems, databases other z/OS sub- systems for optimal performance – Very good at detecting bottlenecks and other potential performance problems in z/OS, CICS, IMS, DB2, MQ, Storage, etc. – Most include historical reporting and trending facilities but that is typically limited to a subset of the data that the monitor collects Capacity Planning Tools – Next day, next week, next month reporting of offloaded SMF data Event Management Systems – Alert management 5 Syncsort Confidential and Proprietary - do not copy or distribute
- 6. Challenges with these Legacy Technologies Tend to have fixed displays with little room for customization on how an end-user can see data provided The interface(s) to these products have traditionally been closed and proprietary Limited view into security issues and threats Limited ability to monitor business services and provide service-level intelligence 6 Syncsort Confidential and Proprietary - do not copy or distribute They typically have a silo approach: a monitor for DB2, another monitor for CICS, etc. without any real correlation between the different pieces Require Subject Matter Experts (SMEs) with in depth technical knowledge of z/OS and its sub-systems in order to effectively use the products Most have evolved into very complex and resource intensive solutions in an attempt to cover ever aspect of the systems they monitor
- 7. What is Needed? High performance, low-cost, platform for collecting critical system information in real-time Normalization of the z/OS data so it can be used off platform analytics engines Full analytics, visualization, and customization with no limitations on what can be viewed Ability to easily combine information from different data sources and systems Address the SME challenge: use by network managers, security analysts, application analysts, enterprise architects without requiring mainframe access or expertise 7 Syncsort Confidential and Proprietary - do not copy or distribute
- 8. “Splunking” Your Mainframe Data into The Industry-Leading Platform For Machine Data Syncsort Confidential and Proprietary - do not copy or distribute Machine Data: Any Location, Type, Volume Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom Apps Messaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID On- Premises Private Cloud Public Cloud Platform Support (Apps / API / SDKs) Enterprise Scalability Universal Indexing Answer Any Question Developer Platform Report & analyze Custom dashboards Monitor & alert Ad hoc search Mainframe 8
- 9. Critical Mainframe Data Normalized and Streamed to Splunk with Ironstream® Log4jFile Load SYSLOG SYSLOGD logs security SMF 50+ types RMF Up to 50,000 values DB2SYSOUT Live/Stored SPOOL Data Alerts Network Components Ironstream API Application Data Assembler C COBOL REXX USS
- 10. Ironstream: Architectural Overview 10 Syncsort Confidential and Proprietary - do not copy or distribute Mainframe TCP/IP (SSL) Data Forwarder DCE IDT Ironstream DesktopData Collection Extension Data ForwarderData Forwarder DB2SYSOUT Live/Stored SPOOL Data Alerts Network Components Ironstream API Application Data Assembler C COBOL REXX USSLog4jFile Load z/OS SYSLOG SYSLOGD logs security SMF 50+ types RMF Up to 50,000 values Enterprise Security ACK
- 11. Primary Use Cases for z/OS Log Data 11 Syncsort Confidential and Proprietary - do not copy or distribute Security & Compliance IT Operational Analytics(ITOA) IT Service Intelligence(ITSI) RACF Intrusion Detection SMF Type 80 SyslogD Mainframe Application Operator logs for DB2, CICS, IMS, etc Related Mainframe Logs Syslog SMF DB2 Accounting Records CICS Accounting Records WebSphere Job / Step Accounting Records SMF Type 101 SMF Type 110 Log4j SMF Type 30
- 12. 12 Syncsort Confidential and Proprietary - do not copy or distribute IRONSTREAM Z/OS SECURITY & SPLUNK ENTERPRISE SECURITY
- 13. Security Issues You Can Monitor with Ironstream Intrusion Detection TSO logon tracking TSO account change activity FTP authentications and file transfers IP traffic analysis Network events 13 Syncsort Confidential and Proprietary - do not copy or distribute
- 14. Ironstream z/OS Security App All data sources collected by Ironstream are exposed in an application focused on z/OS security only This app shows z/OS mainframe security data and is NOT an enterprise-wide integrated view 14 Syncsort Confidential and Proprietary - do not copy or distribute
- 15. z/OS Security Dashboard 15 Syncsort Confidential and Proprietary - do not copy or distribute Intrusion Detection showing Port Scans and Denial of Service Attacks TCP/IP Network Traffic
- 16. z/OS Security Dashboard Syncsort Confidential and Proprietary - do not copy or distribute 16 Job Initiations TSO Account Activity TSO Lockouts FTP Session Activity FTP Transfer Activity
- 17. Ironstream z/OS Security & Splunk Enterprise Security App All collected data sources can also be mapped to Splunk CIM for Enterprise Security and automatically exposed in ES dashboards along with security information from other platforms – Requires the Ironstream TA for Splunk Enterprise Security to be installed – Provides an enterprise-wide, integrated view of security across all platforms via ES dashboards provided by Splunk 17 Syncsort Confidential and Proprietary - do not copy or distribute
- 18. Sample Intrusion Center Dashboard With Splunk Enterprise Security™ 18 Syncsort Confidential and Proprietary - do not copy or distribute Now shows z/OS® intrusions and anomalies along with events from other platforms
- 19. Sample Security Posture Dashboard With Splunk Enterprise Security™ 19 Syncsort Confidential and Proprietary - do not copy or distribute Now shows z/OS® intrusions and anomalies along with events from other platforms
- 20. 20 Syncsort Confidential and Proprietary - do not copy or distribute IT OPERATIONS ANALYTICS
- 21. What Can You with IT Operations Analytics? View RACF violations by type and user Look at message trends over time to determine potential security threats Monitor completion of critical batch JOBs Monitor CICS regions and transactions supporting critical business services Monitor DB2 database lock contention Monitor MQ connections and queues Define and monitor access to critical datasets Monitor all critical resources for a z/OS LPAR And much more!!! 21 Syncsort Confidential and Proprietary - do not copy or distribute
- 22. Operational Analytics: RACF Violations and Message Trends Data Source: SYSLOG Syncsort Confidential and Proprietary - do not copy or distribute 22 RACF Violations by type RACF Violations by user Trend message volumes today vs. same time last week and 2 weeks ago
- 23. Operational Analytics: Job Monitor for SLA Tracking Data Source: SMF Type 30 Syncsort Confidential and Proprietary - do not copy or distribute 23 Track JOB execution against defined service levels and identify JOBS that are at risk of non- compliance with service level agreement target Drill down to predecessor JOBS
- 24. Application Monitoring: DB2 Performance Data Source: SMF Type 100, 101, 102 Syncsort Confidential and Proprietary - do not copy or distribute 24 Logging Rate Uncommitted Records by Plan Lock State Escalations Lock Contention Unavailable Resources
- 25. Application Monitoring: CICS Transaction Analysis Data Source: SMF Type 110 Syncsort Confidential and Proprietary - do not copy or distribute 25 Transaction Rates CPU Usage by Transaction Transaction Response Time Transaction Failures
- 26. IT SERVICE INTELLIGENCE (ITSI)
- 27. Why is IT Service Intelligence Critical? Need to understand what critical IT services are dependent upon which IT resources and components What are the Key Performance Indicators (KPIs) for IT components comprising an IT service How is the performance of an IT resource or component affecting a critical IT service 27 Syncsort Confidential and Proprietary - do not copy or distribute
- 28. Ironstream Integration with Splunk ITSI KPIs provided for mainframe systems in Service Analyzer – CEC (Central Electronic Complex), i.e. “the box” – LPARs (logical partitions) – Critical services Glass Tables for visualization
- 29. Ironstream ITSI Glass Table for Mainframe
- 30. Ironstream ITSI Glass Table for Online Banking Service
- 31. Value Today for Organizations with a z/OS Mainframe 31 Less Complexity Collect mainframe data; correlate with data from other platforms; no mainframe expertise required Clearer Security Information Identify unauthorized mainframe access, other security risks Healthier IT Operations Real-time alerts identify problems in all key environments View latency, transactions per second, exceptions, etc. More Effective Problem-Resolution Management Real-time views to identify real or potential failures earlier View related 'surrounding' information to support triage repair or prevention Higher Operational Efficiency Enhanced event correlation across systems Staff resolves problems faster; can “do more with less” Eliminate Your Mainframe “Blind-Spot” Splunk + Syncsort Ironstream for Your 360ᵒ Enterprise View
- 32. Ironstream Apps Are Now On Splunk App Store (splunkbase) 32 Syncsort Confidential and Proprietary - do not copy or distribute https://splunkbase.splunk.com/ Search Syncsort
- 33. Ironstream Applications on splunkbase Syslog – RACF violations and message trends CICS Region Monitor – CICS Region Health Check – CICS Region transaction rates, response times, CPU usage, & failures MQ Monitor – Queue depths and response time – Message Get/Put rates and CPU use – Ability to filter by connection name and queue name 33 Syncsort Confidential and Proprietary - do not copy or distribute Additional information for each application is available via download on splunkbase, as well as via Product Documentation under Resources at www.Syncsort.com
- 34. Ironstream Applications on splunkbase System Performance Monitor – CEC MSU capacity alongside the 4-hour rolling average figures (4HRA) for each LPAR – z/OS system performance data including: • CPU utilization, memory and common storage utilization, Paging rates Dataset Analyzer – Critical datasets to be monitored are defined via a .CSV file in Splunk 34 Syncsort Confidential and Proprietary - do not copy or distribute Additional information for each application is available via download on splunkbase, as well as via Product Documentation under Resources at www.Syncsort.com
- 35. Get Ironstream® for SYSLOG for free 35 Syncsort Confidential and Proprietary - do not copy or distribute http://www.syncsort.com/en/TestDrive/Ironstream-Starter-Edition
- 36. 36 Syncsort Confidential and Proprietary - do not copy or distribute THANK YOU!