Security and Audit for Big Data
6 likes3,711 views
The document discusses IBM Security Systems' approach to ensuring security and compliance in big data environments through tools like Infosphere Guardium. It emphasizes the need for real-time monitoring of data access, protecting sensitive data, and implementing robust security frameworks to prevent data breaches. Additionally, the document highlights the importance of compliance with various regulations and the capabilities of Guardium in automating security processes.
Security and Audit for Big Data
- 1. IBM Security Systems Security and Audit for Big Data Tina Chen Guardium Enablement chenti@us.ibm.com © 2013 IBM Corporation 1 © 2013 IBM Corporation
- 2. IBM Security Systems Please note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. 2 © 2013 IBM Corporation
- 3. IBM Security Systems Innovative technology changes everything 1 trillion connected objects 1 billion mobile workers Social business Bring your own IT Cloud and virtualization 3 © 2013 IBM Corporation
- 4. IBM Security Systems Compromises take weeks and months to discover & remediate Time span of events by percent of breaches 4 http://www.verizonbusiness.com/resources/reports/rp_data-breach-investigations-report-2012_en_xg.pdf?CMP=DMC-SMB_Z_ZZ_ZZ_Z_TV_N_Z038 © 2013 IBM Corporation
- 5. IBM Security Systems Four key drivers for data security “To mitigate business risk, you must proactively protect what matters — customer data, financial data and intellectual property — from both outside attackers and privileged insiders.” GLOBAL COMPLIANCE Aggressive New Regulations APTs DATA BREACHES 98% Stolen Records From Large Orgs2 CLOUD ADOPTION Compliance – PCI-DSS, HIPAA/HITECH, SOX, GLBA, Basel III and others – National Data Encryption Laws: UK Data Protection Act, EU Data Protection Directive, South Korean PIPA and others – Require encryption, separation of duties, privileged user controls Data Breaches – Valuable data is being targeted by sophisticated attackers - data breaches increasingly result – IP Protection, US Federal and State Data Protection laws, Data across Borders – Encryption + access controls limit risk, meet safe harbor requirements Enterprise Security #1 Inhibitor1 Cloud Adoption – Security the #1 concern BIG DATA Big Data = Big Risks Big Data is a Big Target 5 – Cloud efficiency and flexibility highly desired – Encryption + access control limits exposure to cloud admins and other new security risks – Large data sets inevitably include sensitive data – All data stores and report locations require protection From Vormetric 1. Global State of Information Security® Survey by PwC, CIO magazine, and CSO magazine – October 2012 2. Verizon Data Breach Investigation Report – March 2012 © 2013 IBM Corporation
- 6. IBM Security Systems The importance of monitoring Can you answer these questions? Big Data Platform Structured - Who is running map reduce jobs and what are those jobs accessing? - Is there a new job in the system that hasn’t been vetted? Unstructured - Is someone possibly trying to hack into the file system? Streaming Clients 6 Hadoop Cluster © 2013 IBM Corporation
- 7. IBM Security Systems 7 © 2013 IBM Corporation
- 8. IBM Security Systems Complement existing security with secure databases 8 © 2013 IBM Corporation
- 9. IBM Security Systems InfoSphere Guardium protects sensitive data in Hadoop environments and helps ensure compliance Introducing Hadoop Activity Monitoring o Protect your sensitive data with real time activity monitoring o Gain insights into data activity throughout the stack: Hive, MapReduce, HBase and HDFS o Detect unauthorized applications or users o Real time alerts reduce time to discovery for possible breach or infraction of compliance o Automate compliance and management tasks o Infrastructure in place to provide additional real-time controls over time 9 Monitor and Audit Hive Application MapReduce MapReduce Oozie Storage HBase HDFS © 2013 IBM Corporation
- 10. IBM Security Systems How it’s done InfoSphere Guardium monitors key Hadoop events: •Session and user information •HDFS operations – Commands (cat, tail, chmod, chown, expunge, etc), files, permissions •MapReduce jobs - Job, operations, permissions •Exceptions, such as authorization and access control failures •Hive/HBase queries , - Alter, count, create, drop, get, put, list,.. InfoSphere Guardium Collector Appliance InfoSphere Guardium Reports S-TAPs Clients 10 Hadoop Cluster Heavy lifting occurs on Guardium collector! Very low overhead on monitored nodes. Architecture supports separation of duties Sensitive data alert! © 2013 IBM Corporation
- 11. IBM Security Systems How it is done 11 © 2013 IBM Corporation
- 12. IBM Security Systems Capture and Parsing Overview S-TAP Guardium Collector Analysis engine Hadoop fs –mkdir …. Parse commands then log Namenode Hadoop fs –mkdir /user/data/sundari Sessions Hadoop fs –mkdir … Read Only Hardened Repository (no direct access) Objects /user/data/ sundari Joe Commands Hadoop commands Joe 12 Hadoop Client Hadoop fs – mkdir/user/data/sundari mkdirs © 2013 IBM Corporation
- 13. IBM Security Systems A recommended approach 1. Identify users and classes of users – “privileged” users, data scientists…Who is allowed to access sensitive data Validate with activity monitoring 2. Identify the applications, jobs, ad-hoc analysis Validate with activity monitoring 3. When possible identify and mask sensitive data before it enters the cluster and identify specific directory location in cluster for that data. Put tighter monitoring controls around that data. 4. Look at exceptions – permission exceptions, other operational errors 13 © 2013 IBM Corporation
- 14. IBM Security Systems Use cases Let’s do the following: • • • Log and/or alert access to sensitive files by “unauthorized or unknown” user Reporting on new jobs entering the system (identify new MapReduce jobs in the system) Exception reporting for permission errors on sensitive data And for each scenario, how to: Plan Monitor Automate 14 © 2013 IBM Corporation
- 15. IBM Security Systems Planning for sensitive data access and monitoring Plan o Do you have PCI or other sensitive data? Is sensitive data already identified in “source” systems? o How do you carry that sensitive identification over to Hadoop? o What are the internal and external compliance requirements for monitoring sensitive data access? o What is the plan for handling violations? Who need to be alerted and when? 15 © 2013 IBM Corporation
- 16. IBM Security Systems Where is sensitive data? Keep sensitive data localized, encrypted, and under monitoring control Plan Non-Sensitive data or known Sensitive data or unknown 16 Configuration files © 2013 IBM Corporation
- 17. IBM Security Systems Monitoring sensitive data Monitor Real-time security policies Default Hadoop Policy Your policy rules go here, such as sensitive data alerting… Flexible, granular rules…. 17 © 2013 IBM Corporation
- 18. IBM Security Systems Determine who is accessing sensitive data When? From where? Monitor Who? What? 18 © 2013 IBM Corporation
- 19. IBM Security Systems Alerts reduce time to discovery Automate Unauthorized access to sensitive data! Incorporate Data Events into QRadar unified view and real time analytics 19 © 2013 IBM Corporation
- 20. IBM Security Systems Planning for application access Plan What MapReduce jobs are being used? What kind of ad hoc analysis is allowed on the system? Have they been vetted for access to sensitive data? What is a normal pattern of activity? What process should the team use to communicate new deployments? Task Map Reduce (break task into small parts) (many results to a single result set) No human communication process is infallible… 20 © 2013 IBM Corporation
- 21. IBM Security Systems What applications are using the data? Monitor MapReduce reports …. Now, reduce the noise by filtering out authorized jobs…. 21 © 2013 IBM Corporation
- 22. IBM Security Systems What applications are using the data? Monitor Focus your resources on the unknown – unauthorized MapReduce jobs 22 © 2013 IBM Corporation
- 23. IBM Security Systems What applications are using the data? Audit process workflow and administrative automation Automate Should this job be approved? 23 © 2013 IBM Corporation
- 24. IBM Security Systems What applications are using the data? Audit process workflow and administrative automation Automate Business Owner approves or rejects new applications/jobs 24 © 2013 IBM Corporation
- 25. IBM Security Systems What applications are using the data? Audit process workflow and administrative automation Automate Business Owner approves or rejects new applications/jobs Information Security confirms Business Owner recommendation 25 © 2013 IBM Corporation
- 26. IBM Security Systems What applications are using the data? Audit process workflow and administrative automation Automate Business Owner approves or rejects new applications/jobs Guardium Admin adds authorized jobs to “authorized job list” Information Security confirms Business Owner recommendation 26 © 2013 IBM Corporation
- 27. IBM Security Systems What applications are using the data? Populate new vetted applications automatically 27 Automate © 2013 IBM Corporation
- 28. IBM Security Systems What applications are using the data? Populate new vetted applications automatically 28 Automate © 2013 IBM Corporation
- 29. IBM Security Systems What applications are using the data? Populate new vetted applications automatically 29 Automate © 2013 IBM Corporation
- 30. IBM Security Systems Alerting off the Exception reports Monitor Automate Table already exists IO exception 30 © 2013 IBM Corporation
- 31. IBM Security Systems Continuous database activity monitoring • PCI, SOX and HIPAA accelerators included with DAM (guidance, reports, and more) 31 © 2013 IBM Corporation
- 32. IBM Security Systems PCI – Data Access Report 32 © 2013 IBM Corporation
- 33. IBM Security Systems Streamline and simplify compliance processes for Hadoop InfoSphere Guardium Top Benefits InfoSphere Guardium Top Benefits Proven track record in data security Proven track record in data security Complete separation of duties Complete separation of duties Sensitive data monitoring to pass compliance audits Sensitive data monitoring to pass compliance audits Privileged user monitoring Privileged user monitoring Real-time alerting for abnormal/suspicious activity Real-time alerting for abnormal/suspicious activity Full forensics: Any activity – views, changes, updates… Full forensics: Any activity – views, changes, updates… Heterogeneous support – IBM, HortonWorks, Cloudera, Heterogeneous support – IBM, HortonWorks, Cloudera, Greenplum… Greenplum… Same platform for all databases in your enterprise Same platform for all databases in your enterprise 33 © 2013 IBM Corporation
- 34. IBM Security Systems 34 © 2013 IBM Corporation
- 35. IBM Security Systems IBM Securing all types of data… Data at Rest Data privacy for non-production environments: Optim Data Privacy (DP) Stored (Databases, File Servers, Big Data, Data Warehouses, Application Servers, Cloud/Virtual ..) Optim Test Data Management (TDM) Securing static data on repository : Guardium Data Encryption Data in Motion Data privacy for unstructured data (documents) – Guardium Data Redaction Data privacy for production environments: Over Network (SQL, HTTP, SSH, FTP, email,. …) – Guardium Data Activity Monitoring Configuration Ensuring database is configured and patched properly Repository Vulnerability (Database Configuration, Patch Level, OS Security, …) 35 – Guardium Vulnerability Assessment – QRadar/QVM © 2013 IBM Corporation
- 36. IBM Security Systems Data Masking for Data Privacy Mask confidential data to avoid data breach & meet privacy compliance Requirements Mask files Mask files Hadoop Database IMS VSAM More… Extract, mask & load Extract, mask & load Mask in-Hadoop Mask in-Hadoop using MapReduce using MapReduce Mask in-database Mask in-database • Protect confidential data while preserving analytics • Mask data anytime, anywhere • Mask data in Hadoop using MapReduce • Implement proven built-in masking algorithms • Support compliance with privacy regulations Benefits InfoSphere Optim Mask JASON MICHAELS JASON MICHAELS Before Masking 36 ROBERT SMITH ROBERT SMITH • Protect sensitive information from misuse and fraud • Prevent data breaches and associated fines • Achieve better information governance After Masking © 2013 IBM Corporation
- 37. IBM Security Systems Optim Data Masking implementation in Hadoop Optim Masking can also be executed in Hadoop for delimited files. Java application/interface for masking. • • • • MapReduce base classes and helpers. Distributed cache. Shared libraries. Use of masking in Reducers. Declarative specification of: • Metadata of data files • Masking rules OPTIM JASON MICHAELS JASON MICHAELS Before Masking 37 ROBERT SMITH ROBERT SMITH After Masking © 2013 IBM Corporation
- 38. IBM Security Systems Intelligence: A comprehensive portfolio of products and services 38 New in 2012 Products Services © 2013 IBM Corporation
- 39. IBM Security Systems All domains feed Security Intelligence Correlate new threats based on X-Force IP reputation feeds Hundreds of 3rd party information sources Guardium Identity and Access Management Database assets, rule logic and database activity information Identity context for all security domains w/ QRadar as the dashboard Tivoli Endpoint Manager Endpoint Management vulnerabilities enrich QRadar’s vulnerability database 39 IBM Security Network Intrusion Prevention System Flow data into QRadar turns NIPS devices into activity sensors AppScan Enterprise AppScan vulnerability results feed QRadar SIEM for improved asset risk assessment © 2013 IBM Corporation
- 40. IBM Security Systems Key Business Drivers for InfoSphere Guardium Continuously Monitor All Access too.. Prevent data breaches Assure data governance Reduce cost of compliance 40 © 2013 IBM Corporation
- 41. IBM Security Systems Extend real-time Data Activity Monitoring to also protect sensitive data in data warehouses, Big Data Environments and file shares DATA Big Data Environments InfoSphere BigInsights GreenPlum CouchDB Integration with LDAP, IAM, SIEM, TSM, Remedy, … 41 © 2013 IBM Corporation
- 42. IBM Security Systems Information and community InfoSphere Guardium YouTube Channel – includes overviews and technical demos developerWorks forum (very active) Guardium DAM User Group on Linked-In (very active) Community on developerWorks (includes content and links to a myriad of sources, articles, etc) New! InfoSphere Guardium Virtual User Group. Open, technical discussions with other users. Send a note to bamealm@us.ibm.com if interested. 42 © 2013 IBM Corporation
- 43. IBM Security Systems Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed or misappropriated or can result in damage to or misuse of your systems, including to attack others. No IT system or product should be considered completely secure and no single product or security measure can be completely effective in preventing improper access. IBM systems and products are designed to be part of a comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products or services to be most effective. IBM DOES NOT WARRANT THAT SYSTEMS AND PRODUCTS ARE IMMUNE FROM THE MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY. ibm.com/guardium © Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United © 2013 IBM Corporation 43 States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
- 44. IBM Security Systems Disclaimer Please Note: IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. 44 © 2013 IBM Corporation