Experiences in Mainframe-to-Splunk Big Data Access
0 likes821 views
The October 2016 webcast detailed the experiences of four organizations that integrated mainframe data with big data platforms like Splunk for improved operational and security compliance. The presenters, including senior engineers Justin Eastman and David Friedman, outlined the challenges faced, such as the need for real-time monitoring and compliance with audit mandates, and how they implemented solutions using Syncsort Ironstream. The session included a Q&A segment, and attendees were encouraged to ask questions and engage with the presenters throughout the presentation.
Experiences in Mainframe-to-Splunk Big Data Access
- 1. Experiences in Mainframe-to- Splunk® Big Data Access: Learn What Your Peers are Doing October 2016
- 2. Housekeeping Webcast Audio: – Today’s webcast audio is streamed through your computer speakers. – If you need technical assistance with the web interface or audio, please reach out to us using the chat window. Questions Welcome: – Submit your questions at any time during the presentation using the chat window. – We will answer them during our Q&A session following the presentations. Recording and Slides: – This webcast is being recorded. You will receive an email following the webcast with a link to download both the recording and the slides. 2
- 3. Session Abstract and Speakers The requirement to add mainframe data to the stream of machine-to-machine or “log” data for operational and security/compliance purposes is real. This webinar details 4 organizations who faced these requirements and tells their individual stories as to what requirement/mandate they faced, what options they considered, and how they ultimately addressed it. There will be a live Q&A for participants to ask follow-up questions as to their stories and how they’re doing today. 3 etary - do not copy or distribute David Friedman, Senior Systems Engineer Steven Menges, Director, Product Management Justin Eastman, Senior Engineer
- 4. Big Data is No Longer a “Future” Syncsort Confidential and Proprietary - do not copy or distribute 4 DB2SYSOUT Live/Stored SPOOL Data Alerts Network Components Ironstream API Application Data Assembler C COBOL REXX USSLog4jFile Load SYSLOG SYSLOGD logs security SMF 50+ types RMF Up to 50,000 values
- 5. Mainframes Still Host the Most Critical Applications at Big Orgs 5 Syncsort Confidential and Proprietary - do not copy or distribute 71% Fortune 500 2.5 BillionBus. Transactions / day / per MF 23of Top 25 US Retailers of World’s Top Insurers10Top World Banks92 Source: IBM Organizations Overall2000+
- 6. Organization #1: Justin Eastman 6 Syncsort Confidential and Proprietary - do not copy or distribute Justin Eastman, Senior Engineer Reminder: Type in your questions at any time during the presentation using the chat window. We will answer them during our Q&A session following the presentations or afterward.
- 7. USE CASE: THE NEED/PROBLEM Incidents occurring in the organization would result in the need to turn on additional CPUs to recover from system being overloaded. There was a need to get visibility into the health of multiple sub-systems across different systems in order to monitor to the load to proactively react to these situations. No single tool or monitor gave visibility into all the different subsystems and across the entire organization. 7 Syncsort Confidential and Proprietary - do not copy or distribute
- 8. USE CASE ALTERNATIVES: IN-HOUSE, OTHER Continue with the human flare gun approach currently used in which multiple groups get involved elongating the mean time to resolution. Use existing monitors that require multiple sessions and SMEs to access and perform triage. Expand capacity to ensure that systems are not overloaded. Continue to rely on the customer to indicate when the services provided become less responsive. Look for a new solution to address their issues. 8 Syncsort Confidential and Proprietary - do not copy or distribute
- 9. USE CASE: SOLUTION AND RESULTS BIBD Solution to access z/OS log data in Splunk® for real-time monitoring of critical subsystem performance Creating single view into the health of all the systems and their corresponding subsystems Creating drill down dashboards that provide the KPIs to identify where the source of the issue resides to that the immediate source Resulted in a significant reduction of MTTR and a improve ability to detect problems before the customer does. 9 Syncsort Confidential and Proprietary - do not copy or distribute
- 10. Big Data Poll Syncsort Confidential and Proprietary - do not copy or distribute 10 Q1.Which Big Data analytics platforms does your company use today? o Hadoop o Splunk o Other Data Warehouse o Don’t Know (Check all that apply)
- 11. Organization #2: David Friedman 11 Syncsort Confidential and Proprietary - do not copy or distribute David Friedman, Senior Systems Engineer
- 12. USE CASE: THE NEED/PROBLEM 12 Syncsort Confidential and Proprietary - do not copy or distribute Customer had an audit and compliance mandate with approaching deadline. Using another product to manually retrieve information on a daily basis. Unable to monitor user log-on attempts, password changes, and access violations on their mainframe environment. Unable to obtain information in real-time.
- 13. USE CASE ALTERNATIVES: IN-HOUSE, OTHER Home-grown solution option explored; determined would not meet implementation deadline (and may not have satisfied requirement). POC bake-off (Syncsort Ironstream performed very well in a POC against competitive product). Validated the ability to replace the manual processes they were using with Ironstream. 13 Syncsort Confidential and Proprietary - do not copy or distribute
- 14. USE CASE: SOLUTION AND RESULTS Monitoring security activity on their mainframe applications to meet audit and compliance requirements outlined in regulation, including: – log-on attempts – password changes – user access violations – other security events Get the information in real-time (and eliminated manual processes previously accomplished using zSecure) Filtering enables selection of only the SMF records needed to produce desired results 14 Syncsort Confidential and Proprietary - do not copy or distribute Security Compliance
- 15. Organization #3: Justin Eastman 15 Syncsort Confidential and Proprietary - do not copy or distribute Justin Eastman, Senior Engineer
- 16. USE CASE: THE NEED/PROBLEM Security threats on the mainframe due to lack of visibility. Highly sensitive PHI (Protected Health Information) escaping as data was moved from the production to test environment despite having fences and an automated scrubbing process. Security information and event management (SIEM) solution required. 16 Syncsort Confidential and Proprietary - do not copy or distribute
- 17. USE CASE ALTERNATIVES: IN-HOUSE, OTHER Do nothing and wait for an audit, or even worse, a security exposure. Attempt to perform post-exposure forensics. Manually extract and process logs, SMF records, etc. and produce audit reports to demonstrate compliance. Do solution vendor search and utilize Gartner Magic Quadrant, etc. for enterprise-class SIEM. 17 Syncsort Confidential and Proprietary - do not copy or distribute
- 18. USE CASE: SOLUTION AND RESULTS SIEM Solution (Gartner SIEM Leader Splunk®) BIBD Solution to access z/OS log data in Splunk® for real-time alerts (Splunk’s chosen mainframe partner Ironstream) Combined solution for mainframe logs provides fast access to: Unusual data movements, amount of movements, and protocols being used How much of the data movement is compliant, non-compliant, or unknown Sources of inbound traffic relating to any anomalies 18 Syncsort Confidential and Proprietary - do not copy or distribute Organizational confidence in ability to audit data access compliance!
- 19. Big Iron to Big Data Poll Syncsort Confidential and Proprietary - do not copy or distribute 19 Q2. Is Mainframe “log” data going into your big data platform/repository? o Yes, it is being streamed into it today o Yes, it goes into it via periodic batch/other input method o No, but that data has been requested/is desired o No o Don’t Know
- 20. Organization #4: David Friedman 20 Syncsort Confidential and Proprietary - do not copy or distribute David Friedman, Senior Systems Engineer Reminder: Type in your questions at any time during the presentation using the chat window. We will answer them during our Q&A session following the presentations or afterward.
- 21. USE CASE: THE NEED/PROBLEM 21 Syncsort Confidential and Proprietary - do not copy or distribute Disbursed transaction information systems. Current tools provide partial solutions. Require comprehensive analytics across operation. Enterprise IT Operational Analytics (ITOA) dashboard desired.
- 22. USE CASE ALTERNATIVES: IN-HOUSE, OTHER Organization selected Splunk® Enterprise as their ITOA solution for distributed computing environment. Considered Syncsort to access mainframe logs and get comparable data from mainframe systems. Conducted thorough POC of Ironstream in conjunction with Splunk® After POC, they were able to quickly start deploying it as a comprehensive monitoring solution. 22 Syncsort Confidential and Proprietary - do not copy or distribute
- 23. USE CASE: SOLUTION AND RESULTS Complete picture of overall system health. Meaningful correlation of information from disparate sources for faster triage and shorter MTTR. Company now able to monitor entire IT infrastructure to detect potential issues before they become critical. 23 Syncsort Confidential and Proprietary - do not copy or distribute Reduce MTTR
- 24. Big Iron, Big Data and Big Iron to Big Data: Additional Use Cases? 24 Syncsort Confidential and Proprietary - do not copy or distribute 24 Syncsort Confidential and Proprietary - do not copy or distribute Security & Compliance (SIEM) • Access Control • Data Movement • Real-time Intrusion Detection • Others? IT Operations (ITOA) • Systems Performance and Tuning • Capacity Planning • Others? IT Service Intelligence? Other Monitoring & Analytics?
- 25. Big Iron MVPs: Always Important, Big Iron and Big Data Functions, Staff Now Critical 25Syncsort Confidential and Proprietary - do not copy or distribute “BMC Annual Mainframe Research Results 2015”1 Big Iron to Big Data Big Data
- 26. Syncsort Solutions for New and “Old” Requirements 26Syncsort Confidential and Proprietary - do not copy or distribute High-performance sort for z/OS® Best Sort for z Systems Offload Copy & SMS Compression and Sort work to zIIP processors Savings with zIIP Database Optimization Suites for IBM DB2® and CA IDMS™ Network Management z/OS® network management & security components Big Data integration with market- leading support for integration and access of mainframe and legacy data sources Data Access for Big Data Collect, transform and stream mainframe app and system log data in near real time to Splunk Enterprise Log Data Access for Big Data High-performance Big Data integration software – Linux/Unix/Windows; Hadoop & Spark; on premise and in the cloud Big Data Integration The most advanced sort features for Unix, Linux, and Windows platforms Best Sort for Distributed Platforms Faster application modernization with less hardware AppMod Big Iron Big Iron to Big Data Big Data Data Funnel Populate enterprise data lake at the push of a button Transparently migrate IMS to DB2 IMS and VSAM DB2 Migration Powerful new tools for your databases
- 27. Questions and More Information Additional Questions for David and Justin? For More Information: syncsort.com/ironstream blog.syncsort.com/ Try Ironstream for Free: syncsort.com/ironstreamstarteredition Comments/Other: Steven Menges: smenges@syncsort.com 27 Syncsort Confidential and Proprietary - do not copy or distribute