Core intel
Download as PPTX, PDF0 likes108 views
This document discusses Core Intel, which is part of ING's cybercrime resilience program. It aims to improve capabilities for cybercrime prevention, detection, and response. Core Intel utilizes tools like Kafka, Spark, Elasticsearch and HDFS to collect, process, analyze and store large amounts of data like network logs and threat intelligence feeds. This allows for advanced analytics, matching, alerting and monitoring to detect cyber threats. The document outlines various configurations, optimizations and best practices for operating this big data pipeline and toolset at scale.
Core intel
- 1. Core Intel Krzysztof Adamski, Krzysztof Żmij On the bank secret service
- 2. Are security breaches common? https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/4 32412/bis-15-302-information_security_breaches_survey_2015-full-report.pdf
- 4. Core Intel is a part of ING Cyber Crime Resilience Programme to structurally improve the capabilities for the cybercrime • prevention • detection and the • response CoreIntel 4
- 5. • Measures against e-banking fraud, DDoS and Advanced Persistent Threats (APTs). • Threat intelligence allow to respond to, or even prevent, a cybercrime attack • (This kind of intelligence is available via internal and external parties and includes both open and closed communities) • Monitoring, detection and response to “spear phishing” • Detection/mitigation of infected ING systems’ • Baselining network traffic/anomaly detection • Response to incidents (knowledge, tools, IT environment) • Automated feeds, automated analysis and historical data analysis The reasoning 5
- 6. The world is not enough
- 7. So the challenge is…
- 8. Market leaders Benelux Growth markets Commercial Banking Challengers Most of our data is within Europe 8
- 9. Market leaders Benelux Growth markets Commercial Banking Challengers but we operate globally 9
- 10. Expect the unexpected to collect all the data 10
- 11. • What kind of data do we need? • Where is our data located? • How we can potentially capture it? • What are the legal implications? So there is a challenge to capture „all” the data 11
- 13. So what you would like to see is… Photo credit: edgarpierce via Foter.com / CC BY
- 14. …In fact it is slightly more complicated
- 15. All has its own purpose. Let’s see in details. 15 Photo credit: https://www.pexels.com/photo/dslr-camera-equipments-147462/
- 17. But tell how to capture that data 17 https://observer.viavisolutions.com/includes/popups/taps/tap-vs-span.php
- 18. Broker settings: Replication factor >= 3 min.insync.replicas = 2 unclean.leader.election.enable = false replica.lag.time.max.ms Producer settings: acks = all retries = Integer.MAX_VALUE max.block.ms = Long.MAX_VALUE block.on.buffer.full = true To have data in order max.in.flight.requests.per.connection = 1 Kafka producer configuration (as we don’t like losing data) 18
- 20. Time is crucial here 20 Photo credit: Cargo Cult via Foter.com / CC BY
- 21. But your business data more, so proceed with caution 21 Photo credit: https://www.pexels.com/photo/white-caution-cone-on-keyboard-211151/
- 22. • Network bandwidth control • quota.consumer.default • quota.producer.default Kafka mirror maker configuration 22
- 23. Secure data: listeners=SSL://host.name:port ssl.client.auth=required ssl.keystore.location ssl.keystore.password ssl.key.password ssl.truststore.location ssl.truststore.password Kafka mirror maker configuration 23 Secure data in transit
- 26. In memory data grid 26 val rddFromMap = sc.fromHazelcastMap("map-name-to-be-loaded")
- 27. Let’s find something in these logs 27 Photo credit: https://www.flickr.com/photos/65363769@N08/12726065645/in/pool-555784@N20/
- 28. Matching 28 Tornado - a Python web framework and asynchronous networking library - http://www.tornadoweb.org/ MessagePack – binary transport format http://msgpack.org/
- 29. • Automatically & continually match network logs <->threat intel • When new threat intel arrives, against full history network logs • When new network logs arrive, against full history threat intel • Alerts are shown in a hit dashboard • Dashboard is a web-based interfaces that provide flexible charts, querying, aggregation and browsing • Quality/relevance of an alert is subject to the quality of IoC feeds and completeness of internal log data. Hit, alerts and dashboards 29
- 30. Be smart with your tooling 30 Photo credit https://www.flickr.com/photos/12749546@N07/
- 31. and leverage e.g. elasticsearch templates 31
- 32. Data mapping: - doc_value - fielddata - fields Cluster settings to check: gateway.recover_after_nodes gateway.recover_after_master_nodes gateway.recover_after_data_nodes indices.recovery.max_bytes_per_sec indices.breaker.total.limit indices.breaker.fielddata.limit Elasticsearch configuration 32
- 33. For those who know how to use heavy equipment 33 Photo credit: News Collection & Public Distribution @techpearce2 via Foter.com / CC BY
- 34. Long data storage - HDFS 34
- 36. Core Intel allows users to perform advanced analytics on network logs using a set of powerful tools • Spark API to write code to process large data sets on a cluster • perform complex aggregations to collect interesting statistics • run large scale clustering algorithms with Spark’s MLLib • run graph analyses on network logs using Spark’s GraphX • transform and extract data for use in another system (which are better for specific analytics or visualization purposes) • Kafka, co you can write own Consumers and Producers to work with your data • to perform streaming analysis on your data • to implement your own alerting logic • Toolset • Programming languages: Scala, Java, Python • IDE’s: Eclipse / Scala IDE, IPython Notebook and R Studio Advanced analytics 36
- 37. How do we schedule the jobs 37
- 38. How to keep everything under control 38 Photo credit: https://www.flickr.com/photos/martijn141
- 39. Monitoring crucial points in your data pipeline 39
- 40. Something for smart guys 40 Photo credit: https://www.flickr.com/photos/jdhancock/5173498203/
- 41. Plenty of data to analyze 41
- 42. Upcoming challenges on the operations side 42
- 44. 44
- 45. Follow us to stay a step ahead ING.com YouTube.com/ING SlideShare.net/ING@ING_News LinkedIn.com/company/ING Flickr.com/INGGroupFacebook.com/ING