Big Data Security Analytics (BDSA) with Randy Franklin
Download as PPTX, PDF4 likes2,042 views
The document discusses big data security analytics and how HP addresses related challenges. It notes that big data analytics for security requires real-time analysis of high-volume, diverse data streams. While many big data solutions focus on batch analytics, security demands real-time correlation and detection of threats. The document outlines how HP's ArcSight platform collects, correlates, and analyzes security data from many sources in real-time. It also explains how HP uses Hadoop for long-term storage and analytics, and Autonomy for semantic analysis of unstructured data to enable predictive security.
Big Data Security Analytics (BDSA) with Randy Franklin
- 1. Sponsored by Top 5 Truths about Big Data Hype and Security Intelligence
- 2. Thanks to Made possible by www.hpenterprisesecurity.com SRIDHAR KARNAM HP ArcSight Product Marketing 2 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 3. Preview of Key Points 1. 2. 3. 4. 5. 3 There’s More to Big Data than “Big” The Real-Time Requirement for Big Data Security Analytics There’s More to Big Data Security Analytics than Big Data Technology The Trap of Data Silos within Big Data Repositories The 3 Vs of Big Data Aren’t New to Enterprise SIEM © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 4. 1 Velocity usually considered rate of new data to be stored Not analyzed But BDSA has a bigger velocity issue The type of questions being asked and the analytical techniques being used to answer them is what distinguishes Big Data from traditional data There’s More to Big Data thanData “Big” Big Data Is.. Velocity Data Science Cluster analysis Topological data analysis Machine learning Multi-linear subspace learning Data visualization Data Variety Put all data together; find relationships we didn’t know existed Variety – total record types Big data even with small volume 4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Data Volume Volume is only one dimension of “big” Record quantity better metric than byte About analysis or lots of information
- 5. 2 Big Data Security Analytics (BDSA) is a specialized application of the more general concept of Big Data. Most Big Data scenarios High velocity data aquisition The Real-Time Requirement for Big Data Human driven analysis Long Security Analyticsshelf life for conclusions drawn 3 types of velocity Insertion or append speed into Big Data repository Processing speed for queries upon data rest Analysis of events in real time Human driven analysis has a place in BDSA 5 Immediate tactical investigations in response to warning signs detected by automated correlation engines Forensic investigations Strategic research to tease out indicators of long-term, ongoing © Copyright 2012 Hewlett-Packard Development Company, attacks L.P. The information contained herein is subject to change without notice.
- 6. 2 But what about tactical, second-to-second monitoring? Core of security operation center work Analysis must be done automatically and in a streaming fashion The Real-Time Requirement for Big Data Current Big Data Security Analyticsa query, analyzetools tweak query, analyze Run results, results, repeat Not a streaming scenario in which a constantly updated tactical situation is plotted But real-time analytics require a purpose-built correlation engine Enterprise SIEM correlation engines 6 Designed to handle a constant stream in real time Maintain in memory a massive amount of partial pattern match © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. objects
- 7. 2 Trigger for tactical investigations Event feed The Real-Time Requirement for Big Data Security Analytics SIEM Big Data Real-Time Correlation Batch Analytics Context Criteria for better correlation rules 7 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Wide and deep trolling to identify ongoing attacks too low and slow to trigger SIEM alerts
- 8. BDSA requires 3 kinds of advanced skills 3 Big data Data Information platform science security There’s More to technology Security Analytics Big Data than Big Data Technology Still more of a concept and developer-level movement than a mature technology platform with available offthe-shelf solutions To detect cyber-attacks and internal malicious agents, analysts need to be more than data scientists To make any sense of Big Data, analysts using Big Data farms need to know how to use advanced analytics Must also be technical information security professionals that understand the organization’s IT infrastructure. Network security, host security, data protection, security event interpretation, and attack vectors 8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 9. 3 There’s More to Big Data Security Analytics than Big Data Technology 9 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 10. 4 Application A Application B Application C Application A The Trap of Data Silos within Big Data Repositories Point Solution for Monitoring Application B Point Solution for Monitoring Application B Application B Application C Point Solution for Monitoring Application B Big Data Repository Even after migrating from point solutions to Big Data, the same silos can persist. 10 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 11. 4 Example: consider usernames and email addresses If you are trying to track a user’s actions and communications through a variety of data, you must be cognizant of the fact that a given email address, such as jsmith@acme.com, could be one of the following: The Trap of Data Silos within Big Data Repositories Email sender Email recipient Actor in a audit log event (e.g., jsmith opened a file) Object of an action in an audit log event (e.g., Bob changed jsmith’s reset password) Subject of a memo Simply querying certain data can lead to extremely inaccurate results unless one of the following occurs: 11 The analyst filters the results manually after the query The analyst builds knowledge into the query about the structure or format of the various data queried to do the filtering The system understands the various formats and does the filtering © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. automatically
- 12. 4 Silos in Big Data is failure to deal with variety Being able to store all types of data and query it for keyword occurrences does not satisfy BDSA requirements. The Trap of Data Silos within Big Data Repositories Some enterprise SIEMs takes a more effective and pragmatic approach that embraces data variety Normalizing security events into a common event format Integrate non-event data sets into the correlation and analytics process. 12 Directory information IP reputation lists Geolocation data © Copyright 2012 Hewlett-Packard Development Company, social networkherein is subject to change without notice. L.P. The information contained feeds
- 13. 5 Big data architecture Enterprise SIEMs abandoned relational databases a long time ago Proprietary correlation and storage engines The 3 Vs of Big Data Aren’t New to Enterprise • Allow rapid SIEM storage and query of massive amounts of event data Real-time situational awareness Real-time analysis is a manifest requirement of security analytics Enterprise SIEMs analyze data as it arrives Combines • real-time, in-memory, event-log data • asset awareness, asset vulnerability • identity correlation Prioritize critical events and correlations to assist operating teams with immediate detection of threats 13 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. No data scientists required
- 14. Bottom line Hidden skill requirement of BDSA: data scientists Real-time requirement for security intelligence, often misunderstood in relation to Big Data Risk of data silos persisting in Big Data repositories Investing in a Big Data cluster that runs search and a schemaless database is only the beginning of building a BDSA practice An enterprise SIEM like HP ArcSight provides BDSA that is specialized for event data 14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 15. How HP Solves Big Data Security Analytics Problem? • • • • With CORR With Hadoop With Autonomy With HAVEn • Why HP? © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 16. Big data opportunities – won and lost Competitive Advantage in the Digital Universe Massive amounts of useful data are getting lost % of data that would be potentially useful IF tagged and analyzed 23% 3% 0.5% ¹Source: IDC The Digital Universe in 2020, December 2012 16 % of the Digital Universe that actually is being tagged and analyzed © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. % actually being tagged for Big Data Value (will grow to 33% by 2020)
- 17. HP ArcSight HP ArcSight Universal log management platform High-performance universal log management to consolidate machine data across IT Collect & correlate up to 100,000 events per second from 350+ connectors Collect, normalize, and categorize machine data such as logs, events, and flows from any device, any time, anywhere from any vendor Search over 2,000,000 events per second The unified machine data through filtering and parsing is enriched with rich metadata, which allows you to search machine data through simple text-based keywords without the need of domain expertise Store years’ worth of data The unified data is stored through high compression ratio in any of your existing storage formats, eliminating the need for expensive databases and DBAs Analytics & intelligence Built-in content packs, algorithms, rules, and the unified machine data help you deploy IT security, IT operations, IT GRC, and log analytics Collect, store, correlate, and analyze big data across IT 17 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 18. ArcSight CORRe for Big Data Security ArcSight has been dealing with Big Data since 2007 with CORR engine Volume • Cross-device, real-time correlation of data across IT • Long term archival at 10:1 compression ratio with ArcSight • Send it to Hadoop at over 100,000 EPS Volume Velocity Complexity • SmartConnectors collects logs, events, flows at over 100,000 EPS from almost any log generating source • Search data at over 2,000,000 EPS Variety • Collects machine generated data from 350+ distinct sources • Autonomy collects human generated data from 400+ distinct sources • © Copyright 2012 Hewlett-Packard Development Company, L.P. The information physical, virtual, and cloud 18 Collect from Hybrid network such as contained herein is subject to change without notice. Velocity Variety
- 19. Success Stories Beyond theory to practice: U.S. Department of Health and Human Services “HP solutions have helped us transform from a reactive to a proactive IT Operations function, and to align our priorities to match the business and drive business value, delivering 300% ROI in one year.” - Dan Galik, CISO Heartland Payment Systems “ ArcSight solution will give us a more comprehensive threat and risk management platform that optimally enables enterprise-wide visibility to identify illegal activity in progress and take prompt, preemptive action.” - Kris Herrin, CTO 19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 20. Security Intelligence ArcSight and Hadoop Storage Hadoop Analytics ESM/Logger Live/ Historical data Live (Real-time, cross-device correlation of security events) Historical (security intelligence) ESM/Logger Hadoop Live (Real-time analytics on unlimited data) Historical (Security analytics) 20 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 21. Sentiment Analysis ArcSight with Autonomy Meaning based security Predictive security – Moving from proactive security Answers critical questions: • • • 21 Where is our sensitive information? Who has access to it? Which systems store sensitive information? Do we have the right controls in place to protect sensitive information? © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 22. HAVEn – big data platform HAVEn Hadoop/ Autonomy HDFS IDOL Catalog massive volumes of distributed data Process and index all information Social media 22 Enterprise Video Audio Vertica nApps Security Email Analyze at extreme scale in real-time Texts Mobile Transactional data Collect & unify machine data Documents © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. IT/OT Powering HP Software + your apps Search engine Images hp.com/haven
- 23. How we help our customers? 5 minutes to generate IT GRC report Compliance packs generates IT GRC reports that otherwise would take 4 weeks 3 days to run an IT audit Search results yield audit-quality data that otherwise would take 6 weeks 10 minutes to fix an IT incident Full-text based searching and integration with HP portfolio detects and corrects IT incident that otherwise would take 8 hours 4 hours to respond to a breach Quick forensic tools enable instant response to a data breach that otherwise would take 24 days 2 days to fix a threat vulnerability ArcSight & TippingPoint solution builds threat immune that otherwise would take 3 weeks 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 24. HP Enterprise Security Momentum HP Security Technology HP Security SaaS markets we #1 In allin play #2 9 out of 10 Major banks 24 2.5B lines of code under SaaS subscription HP ESP Customers 10,000+Customers Managed 900+ Security Services 9 out of 10 10 of 10 Top software companies Top telecoms © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. New Products 35 Released in the last 12 months All Major Branches US Department of Defense
- 25. More Information: www.hp.com/go/ArcSight 25 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Editor's Notes
- #8: At the end of the day, you need both capabilities. SIEM’s real-time correlation provides constant situational awareness; the Big Data principles can be leveraged to do the following:Perform tactical drill-down investigations in response to tactical alerts from situational awareness.Provide context to tactical processing.Build more intelligent tactical-correlation rules, based on conclusions from long-term BDSA.Troll wide and deep to identify ongoing attacks that are too low and slow to trigger SIEM alerts.
- #10: BDSA is turning out to be the next evolution of SIEM. Winning SIEM providers are ones who do the following:Embed technical innovations from the Big Data developer field Integrate with Big Data platforms for two-way flow of security intelligenceBuild advanced data-science methods into their correlation and analysis engines so that security analysts don’t need to be data scientistsEnhance data visualization capabilities to help humans recognize hidden patterns and relations in security data
- #11: Thanks to the schemaless architecture of NoSQL databases and the ability to store unstructured data, one of the promising aspects of Big Data is the ability to query across a broad swatch of different kinds of information (i.e., variety). But ironically, after going to significant effort to deploy a Big Data platform and feed it a variety of data, organizations can quickly find themselves building silos within the Big Data repository. Silos explicitly defeat one of the key value propositions of Big Data.
- #12: This challenge is what leads analysts to build silos within Big Data repositories. To make sense of data and ensure the veracity of the analysis, these analysts begin to define views that purposefully select data from a narrow swath of all available data. This silo phenomena is already manifest in some products positioned as Big Data. In perusing the solutions built on top of the platform, one finds a preponderance of applications that focus on machine data from a single technology (e.g., Microsoft Exchange), thus limiting the analysis to the perspective of that one application. If all you need is analysis limited to a single component of your network (i.e., a silo), a good supply of monitoring applications for Exchange and other server products already exists. Organizations that invest in Big Data must ensure that the project stays true to its mandate, or else the organization will simply be maintaining the same data silo in its Big Data repository that was once found in a point solution
- #14: No silos Dumping terabytes of information into a completely schemaless, unstructured database allows cross data-source keyword searching. But in section 4, "The Trap of Data Silos within Big Data Repositories," we pointed out that organizations run the risk of creating silos within the very repository that is supposed to deliver wider visibility. Security-event data is well understood after more than a decade of analysis by the designers at HP ArcSight. And such data is better served with a normalized event schema that identifies a given action such as logon failure as the same event across all platforms and log sources regardless of format. By normalizing all events into one common event taxonomy, ArcSight Connectors decouple analysis from vendor selection. This unique architecture is supported out of the box across hundreds of commercial products as well as legacy systems.
- #16: Slide Objective: Lay out the following key points narrative:Key Points:Market drivers, trends and opportunitiesBig security for big data: HP’s solution for big dataSecurity intelligenceSecurity analyticsContext based SIEMSemantic analytics and concept searchingRoI, proof points, etcQ&ATransition: so first, a look at what’s new and different in the landscape
- #17: 99.5% of data is not tagged or analyzed. IDC predicts that 23% of the data is useful if tagged and analyzed. How much data are you analyzing today? Comprehensive monitoring and analysis is thus needed to extract value out of your dataSo how do you know if you have merely ‘a lot of information’ versus ‘Big Data’? If the information your organization is generating - or has access to but may or may not be capturing or analyzing
- #18: Imagine unifying the machine data across the IT in various formats from various vendors into a simple common format. With the unified tool you should be able to search for any information from any source without any domain expertise or through text-based searching. You can create reports, charts, and dashboards for compliance and regulations, perform quick forensic investigations or simply search through millions of events in seconds to quickly troubleshoot your IT.HP ArcSight Logger is a universal log management solution that unifies searching, reporting, alerting, and analysis across any type of enterprise log data making it unique in its ability to collect, analyze, and store massive amounts of data generated by modern networks. It supports multiple deployments such as an appliance, software, virtual machine, and within the cloud in both Windows® and Linux environment.HP’s approach to comprehensive log management solution is:Collect: Borderless collection of any data from any device in any format from 315+ distinct out-of-the-box loggenerating sourcesEnrich: While the data is being collected, filter and parse the data with rich metadata helping to unify the machine data across ITSearch: As the machine data is enriched during collection, you can simply search through millions of events in seconds on what you want through text-based keywords without any commands or domain expertiseStore: The unified data can be stored in any storage format that you have (NAS, DAS, SAN, etc) though high compression ratio of up to 10:1 eliminating the need for DBAs or expensive databasesAnalyze anything: the rich content built into Logger helps you to perform high-performance interactive searches, comprehensive drill-down reports, and real-time alerting to meet the needs of diverse teams to use machine data for IT Security, IT GRC, IT Operations, SIEM solution, and log analytics
- #19: Gartner in a recent report released in 2013 said that ArcSight has simplified the security intelligence and analytics through CORR engine.
- #20: Slide Objective: highlight the huge Time to Value improvements for a pair of real HP Customers.Key Points: HP’s Information Optimization solutions maximize Return on Information by accelerating Time to Value. With HP Autonomy, customers can analyze their unstructured (e.g. email, texts, video) and semi-structured (e.g. machine-generated) data. With HP Vertica, they can scale their structured data analysis to handle any dataset. When brought together they offer the only solution that bridges these two worlds. In addition, depending on the environment, HP can provide pre-packaged solutions in the form of Converged AppSystems solutions for SAP HANA and NetWeaver and Hadoop. The result is any customer can maximize top-line information value, minimize spend and optimize their Return on Information.
- #21: What is HAVEn: HAVEn is the #1 platform for big data in the industry.HAVEn stands for Hadoop Autonomy Vertica Enterprise Security and any n number of applicationsHAVEn is not a single product. It is a platform that consists of multiple components.As you see in the next slide we also have an HAVEn ecosystem around this platformHAVEn brings together everything you need to profit from big data; hardware, software and services. The 3 HAVEn platform components are connectors, applications, and engines.These are shipping already. We have 1000’s of customers using these components to build mission critical solutions.How does this all work together? As an example, one of the largest global banks does the followingWhen you call them, 3 things happen in parallel – your call gets logged into Hadoop for complianceYour call gets analyzed through autonomy for sentiment – to determine if the customer is happy or unhappy and this info is inserted into Vertica for real time analyticsSimultaneously, another thread gets other business info on this customer and merges it together to find if you are a profitable customerThis information along with other information is analyzed in Vertica in real time to determine how to effectively handle the customer. Should be be offered any promotion or discounts. Details on connectorsWe have 400 connectors from Autonomy and 300 from Arcsight that help you bring all kinds of data. With these many connectors, it is highly likely that you will be able to have off-the-shelf connector to your data.In addition each of the engine components (Autonomy, Vertica and Arcsight) also provide additional data connector frameworks and tools to help you write custom connectors .Additionally the HAVEn platform supports popular frameworks like Hadoop flume and Chukwa. And it is open to all ETL frameworks. Details on engines (For more details refer to individual product pages)Many HP customers use Hadoop or experimenting with it. HP believes in a open Hadoop strategy. HP has been shipping preconfigured Hadoop appliances and/or reference architectures with all major Hadoop vendors – Cloudera, Horton works and MapR. . What we are seeing is that Hadoop is great as a data store to bring in all kinds of data and for ETL, but customers are telling us that they want better engines. As an example Novartis switched from using Hadoop to Vertica and the processing went down from several hours to several seconds using Vertica. That meant rapid drug discovery. The impact saving livesAutonomy has the leading algorithms protected by tens of patents for human information processing - video, audio, text –ex in London Olympics, camera images captured in London were matched in real time to terrorist database. The impact – saving lives. It is one of a kind technology.Vertica is designed ground up in the last ten years in MIT. It was designed for the peta byte wave for blazing fast real time analytics on peta byte size sets. It is designed as analytics platform that supports standard SQL/JDBC/ODBC and R natively. But most importantly because it is designed for large data analytics you can do it at a fraction of what legacy systems cost. Arcsight has been the leader Security and Events Information mgmt. system on Gartner MQ for years. It is used by some of the largest organizations in the world. It has been proven to scale at a million events a second range. Details on applicationsWe have started modifying our existing application portfolio to use HAVEn. And we are building new applications that leverage power of HAVEnAs an example, HP has launched a new application for operation analytics which leverages the power of multiple HAVEn components.Many customers are already building applications that use multiple HAVEn components togetherTo help you get started we have lined up partners and SI’s that can help you build these solutions. Which brings us to the next point – the HAVEn ecosystem