Logging at scale: doing more with less
3 likes563 views
The document discusses using Apache NiFi to improve logging capabilities at scale for a Security Operations Center (SOC). It provides an agenda that includes an introduction to the speaker, context on the SOC's logging needs, an overview of Apache NiFi, and a demonstration. The SOC generates terabytes of logging data per day across diverse tools and cloud services. It evaluated tools to better collect, process, enrich, and analyze logs. Apache NiFi was selected for its ability to flexibly route and transform data, integrate with existing systems, query diverse sources, scale horizontally, and provide security and data provenance.
![Macquarie Government
“Apache NiFi supports powerful and
scalable directed graphs of data
routing, transformation, and system
mediation logic.”
Open sourced by the National
Security Agency in 2014[1] and
submitted to The Apache Software
Foundation for on-going stewardship
[1] https://goo.gl/aZxCIC
| Logging at scale – Doing more with less
• User friendly interface
• Flexible
• Data Agnostic
• Inbuilt mechanisms to balance
between latency and throughput
• Fine grain control of delivery
guarantees (e.g. discard a flowfile once
it becomes too old to be relevant).
• “Secure”
• Data provenance (from where, to
where, changed by, etc.)
• Authorization Policies, TLS, Kerberos,
Encryption and a handful of other
features
• Designed for Extension
A bit about Apache NiFi16](https://image.slidesharecdn.com/fucsacsc-2017-170411161145/85/Logging-at-scale-doing-more-with-less-16-320.jpg)
Logging at scale: doing more with less
- 1. Logging at scale Doing more with less Presented by Andre Fucs de Miranda
- 2. Macquarie GovernmentMacquarie Government 1. A bit about me 2. A bit of context 3. Apache NiFi and the SOC 4. Demo 5. Questions The agenda. | Logging at scale – Doing more with less 2
- 3. Macquarie Government A bit about me. | Logging at scale – Doing more with less
- 4. Macquarie Government Manager @ Macquarie’s Security Operations Center 20 years working in information cyber security Apache NiFi committer and PMC member | Logging at scale – Doing more with less https://github.com/trixpan https://twitter.com/trixpan About me
- 5. Macquarie Government A bit of context | Logging at scale – Doing more with less
- 6. Macquarie Government A bit of context About Macquarie Government • 42% of Australian Government agencies are our customers • 3+ billion events per day; Our tool stack is diverse and busy: • We generate TBs of data per day. • Since 2015 we have been using “Big Data” (i.e. Hadoop ecosystem) for reporting and analytics. • We are constantly looking for ways to offer our customers with better insights over the threats targeting them. • We also felt that relying exclusively on traditional SIEM wasn’t enough anymore. | Logging at scale – Doing more with less 6
- 7. Macquarie Government A bit of context Could we leverage “big data” solutions to improve our SOC further? • Perhaps we could rationalise the way we collect and process log messages? • Perhaps we could do enrichment against a more diverse set of sources?? • What else? | Logging at scale – Doing more with less 7
- 8. Macquarie Government So we went and evaluated lots of tools and architectures looking to map things like: • Ability to integrate with SIEM pipelines natively • Ability to consume cloud services (IaaS, PaaS and Saas) • Ability to query odd stuff • Inbuilt Security • Ability to Scale out • How easy to maintain and extend | Logging at scale – Doing more with less 8 and many more… A bit of context All Apache project logos are trademarks of the ASF and the respective projects. Logstash is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. fluentd is trademark by Treasure Data
- 9. And the winner was…
- 10. Macquarie Government Sorry, there was a mistake… | Logging at scale – Doing more with less 10 All Apache project logos are trademarks of the ASF and the respective projects.
- 11. Macquarie Government Let’s talk tech. | Logging at scale – Doing more with less Apache NiFi and the SOC
- 12. Macquarie Government A bit about Apache NiFi – A brief Prologue When you are start shipping “data” seems like an “easy” task | Logging at scale – Doing more with less DC1 DB 12
- 13. Macquarie Government A bit about Apache NiFi – A brief Prologue But as the environment grows, complexity compounds… …but you keep adjusting your environment | Logging at scale – Doing more with less DC1 DB DC2 DB HQ ClientX AZ1 AZ2 13
- 14. Macquarie Government ‘til the point you suddenly realise your pipeline is missing a bit of cheese. Or worse… | Logging at scale – Doing more with less © Luca Nebuloni https://www.flickr.com/photos/nebulux/10708289086/ 14
- 15. Macquarie Government A bit about Apache NiFi – A brief Prologue | Logging at scale – Doing more with less Source: https://goo.gl/xKoavI 15
- 16. Macquarie Government “Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.” Open sourced by the National Security Agency in 2014[1] and submitted to The Apache Software Foundation for on-going stewardship [1] https://goo.gl/aZxCIC | Logging at scale – Doing more with less • User friendly interface • Flexible • Data Agnostic • Inbuilt mechanisms to balance between latency and throughput • Fine grain control of delivery guarantees (e.g. discard a flowfile once it becomes too old to be relevant). • “Secure” • Data provenance (from where, to where, changed by, etc.) • Authorization Policies, TLS, Kerberos, Encryption and a handful of other features • Designed for Extension A bit about Apache NiFi16
- 17. Macquarie Government A bit about Apache NiFi NiFi allows you easily move data between A and B (and B to A) in a controlled, secure and reliable way, while still allowing you to process and granularly apply logic to the data in motion. | Logging at scale – Doing more with less 17
- 18. Macquarie Government A bit about Apache NiFi A few examples on how NiFi capabilities help a SOC: • Rationalising the flows of data into your SIEM • Do you truly need your SIEM to be ingesting all your logs? • What happens when you run more than one SIEM (because it may well happen…)? • Enrich data against a diverse range of sources • ElasticSearch, REST APIs, DNS, Redis, Whois, GeoIP, SQL, MISP (via HTTP) • (Pull|push) data (from|to) a diverse set of platforms • Object based stores such as GCS or S3, FTP, SFTP, Mainframes via WebSphere MQ, Files, SQL and Syslog of course. | Logging at scale – Doing more with less 18
- 19. Macquarie Government Let’s take a closer look. | Logging at scale – Doing more with less
- 20. Macquarie Government | Logging at scale – Doing more with less DEMO Time
- 21. Macquarie Government Let’s talk. | Logging at scale – Doing more with less Andre Fucs de Miranda Macquarie Government amiranda@macquariegovernment.com 1800 004 943
- 22. Thank you.