MacSysAdmin Conference 2019 - Logging
0 likes187 views
Zentral.pro, founded in Q1 2019 and based in Germany, is a skilled team focused on professional services, research, and development for business and enterprise customers. The document outlines various event sources and types related to computing technology, security agents, and outputs from different systems while emphasizing the importance of event aggregation and log analysis for effective monitoring. It discusses how to connect sources, ship events to platforms like Elastic Stack, and improve collaborations between IT operations and security teams.
MacSysAdmin Conference 2019 - Logging
- 1. zentral.pro Logging About Needles in the Modern Haystack
- 2. • Founded in Q1 2019 • Based in Germany • Small, skilled team • Professional Services Research and Development • Business & Enterprises customers • B2B Partners zentral.pro Who are we ?
- 3. • Based in Hamburg, Germany • Zentral Pro Services (co-founder) • Started the Zentral open source Event Hub Project with Éric Falconnier (co-founder) • Zentral was first shown in public at MacSysAdmin 2015 Henry Stamerjohann Who am I zentral.pro
- 5. “A lot more events, from many more sources…” zentral.pro Logs & Events Landscape ‣ Computing / Technology
- 6. • Cloud Computing Platforms and SaaS • Linux (incl. ChromeOS) • Microsoft: • Azure, Intune, Windows 10 (new norm, great integrations) • Apple: • macOS, iOS, iPadOS, tvOS • Client Management & MDM Provider (well known challenge w/ integrations) zentral.pro Logs & Events Landscape ‣ Computing / Technology
- 7. “Where, when and what ? “ zentral.pro Logs & Events Landscape ‣ Computing / Technology
- 8. • Created by apps, systems, network and user activity • Event flow, time stamps, and Frequency • Common use: • Check-based fault detection • Log-based monitoring • Metrics-based monitoring • Collect telemetry data zentral.pro “Where, when and what ? “ Logs & Events Landscape ‣ Computing / Technology
- 10. OS • Installer • MDMclient • LaunchServices Software • Business apps • Other apps • Security Agents • Osquery • Santa • Xnumon zentral.pro On the endpoints Event sources and types ‣ Sources
- 11. zentral.pro On the endpoints Event sources and types ‣ Sources ‣ Security Agents: • OS • Multi Platform (Mac, Linux, Windows) • Cloud Native Foundation Project • Powerful Change Detection • SQL like view of the system Osquery Osquery Based on
- 12. zentral.pro On the endpoints Event sources and types ‣ Sources ‣ Security Agents: • Kernel extension • (soon) Security Extention • Binary Whitelisting / Blacklisting • TLS Server (Backend) • Dynamic Config • Local Log file Santa Google Santa Based on
- 13. zentral.pro On the endpoints Event sources and types ‣ Sources ‣ Security Agents: • Open BSM • Kernel extension • Log Information on • pid • path • ancestory • arguments • code-signing information • Trace activity (good/bad) Xnumon Based onXnumon
- 14. zentral.pro On the endpoints Event sources and types ‣ Sources ‣ Security Agents: System Extensions
- 15. zentral.pro On the endpoints Event sources and types ‣ Sources ‣ Security Agents: System Extensions
- 16. zentral.pro On the endpoints Event sources and types ‣ Sources ‣ Security Agents: • (New) Apple Endpoint Security System • (New) Apple Network Extension framework • Clients can Subscribe to Endpoint Security System and Network Extension • Option to make decisions • New version of Security and Firewall applications Based on System Extensions System Extensions
- 17. zentral.pro • Written to File • Written to local Database • Written to a Backend • Transferred by an Agent On the endpoints Event sources and types ‣ Outputs
- 18. zentral.pro On the endpoints Event sources and types ‣ Outputs /Library/Logs/… /var/log/… File based - the “classic” use case • mostly with not so well integrated apps • Text data in files (rotated) • Sometimes JSON (1 object per line) ‣ File based
- 19. zentral.pro On the endpoints Event sources and types ‣ Outputs ‣ File based
- 20. zentral.pro On the endpoints Event sources and types ‣ Outputs OS log facilities - for OS and well behaved / integrated apps • Apple Unified Logging • More structure • JSON output possible • Configurable persistence • Syslog (old in macOS) ‣ OS log facility
- 21. zentral.pro On the endpoints Event sources and types ‣ Outputs ‣ Unified Logging In-memory or persist into .tracev3 files
- 22. zentral.pro On the endpoints Event sources and types ‣ Outputs ‣ Unified Logging In-memory or persist into .tracev3 files
- 23. zentral.pro --predicate Filter element (subsystem type) --debug Details depth Formatting (json)--style On the endpoints Event sources and types ‣ Outputs ‣ Unified Logging
- 24. zentral.pro On the endpoints Event sources and types ‣ Outputs OS log facilities - for OS and well behaved / integrated apps • Apple Unified Logging • More structure • JSON output possible • Configurable persistence • Syslog (old in macOS) ‣ Unified Logging Howard Oakley @ Electriclight Company https://eclecticlight.co/ 2018/03/19/macos-unified-log-1- why-what-and-how/
- 25. • JSON payload posted on a HTTPS endpoint (Osquery, Santa,…) • Publish to Kafka (Osquery) • Other custom variants… zentral.pro On the endpoints Event sources and types ‣ Outputs ‣ Custom
- 26. zentral.pro Server / Cloud Event sources and types ‣ Sources Identity Provider • Sign-ins / Sign-in errors (AzureAD, Okta, …) Inventory • Computer check-in (Jamf Pro, WorkspaceOne, …) • Group changes (SimpleMDM, Jamf Pro, …) MDM (SaaS, open source MDM) • Configuration profile pushed • Device Enrollments Security providers • Malware detected/removed (Microsoft Defender ATP, Malwarebytes)
- 27. zentral.pro Server / Cloud Event sources and types ‣ Outputs /var/log/… • File based - for most of the logs • Text data in files (rotated) • Log archives • Service logs (systemd / journalctl)
- 28. zentral.pro Server / Cloud Event sources and types ‣ Outputs /var/log/… • File based - for most of the logs • Text data in files (rotated) • Log archives • Service logs (systemd / journalctl)
- 29. • API (Jamf Pro, Microsoft Graph SecurityAPI) • Webhooks (Jamf Pro, Okta, …) • Files on a server (i.e.Jamf Pro in custom deployment) • Blobs on a storage service • GUI + manual download • Events in a Message Broker (Azure Event Hubs) zentral.pro Server / Cloud Event sources and types ‣ Outputs
- 30. zentral.pro Server / Cloud Event sources and types ‣ Outputs ‣ Jamf Pro Search in browser or download
- 31. zentral.pro Server / Cloud Event sources and types ‣ Outputs ‣ Jamf Pro Search and grep (keywords, errors, …)
- 32. zentral.pro Server / Cloud Event sources and types ‣ Outputs ‣ Jamf Pro Search and grep (keywords, errors, …)
- 33. zentral.pro Event audit trail (sign-ins, edits or changes) Server / Cloud Event sources and types ‣ Outputs ‣ IDP - Okta
- 34. zentral.pro Authentications (export json, csv) Server / Cloud Event sources and types ‣ Outputs ‣ IDP - Duo
- 35. zentral.pro Sign-in Logs (export json, csv) Server / Cloud Event sources and types ‣ Outputs ‣ IDP - Azure AD
- 36. zentral.pro Server / Cloud Event sources and types ‣ Outputs ‣ IDP - Azure AD Sign-in Logs (export json, csv)
- 37. AV Activity / Remediation (export csv) Server / Cloud Event sources and types ‣ Outputs ‣ ATP Defender zentral.pro
- 38. • Build reports from CSV • Analyze/process JSON • Upload and repurpose event data • Share with other Teams • Store for Compliance (Backups) • Use to get support from a Vendor zentral.pro Server / Cloud Event sources and types ‣ Outputs ‣ Post processing
- 39. zentral.pro Ship and collect the events
- 40. zentral.pro Problems / Issues Ship and collect the events ‣ Reality • Many different sources • Many different formats • No single place where to look at events / search for events • Too many events
- 41. • Elastic Stack (formerly ELK Stack) • Splunk • Sumo Logic • Stackdriver • Zentral • et.al zentral.pro Existing Solutions Ship and collect the events
- 42. zentral.pro Existing Solutions Ship and collect the events ‣ Log Facilities ‣ Stackdriver Logging
- 43. zentral.pro Existing Solutions Ship and collect the events ‣ Log Facilities ‣ Stackdriver Logging
- 44. zentral.pro Existing Solutions Ship and collect the events ‣ Log Facilities ‣ Stackdriver Logging
- 45. • Collect file based logs (by agents) • Run agents directly • RPC / HTTPS Osquery events to Kolide or similar services • Unified logging to Elastic Stack on Mac endpoints (i.e. Filebeat) zentral.pro How to connect the sources Ship and collect the events ‣ Endpoints
- 46. zentral.pro How to connect the sources Ship and collect the events ‣ Endpoints ‣ Agents • Read local file based logs • Build-in Modules • Pre-filter, Normalize events • Ship to Elastic Stack (Kibana, Logstash) Based on • Open source code - Beats family • Elastic core component • filebeat.yml config file FileBeat (by Elastic) FileBeat
- 47. Ship and collect the events zentral.pro How to connect the sources ‣ Endpoints ‣ Agents Endpoint logs to ElasticStack Subsystem shipped to Elastic Stack
- 48. zentral.pro How to connect the sources Ship and collect the events ‣ Server / Cloud • Internal routing (Azure AD monitoring to Azure Sentinel) • Interconnect Services with Message Brokers (Azure Event Hubs connect to Sumo Logic) • Webhooks to push event data (Jamf, SimpleMDM) • API pulling data (Custom Apps for Reporting, Dashboards)
- 49. • Productive and Research Platform • Collect Events in parallel • Inventory (Jamf, Intune, Munki, et.all…) • Identity Providers (Okta, AzureAD) • Endpoint Agents (Santa, Osquery, Filebeat) • Normalize and attribute Event Data • Historic Data stored in Elastic Search • Connect with other Event Hubs (Azure Event Hub, SIEM Systems) zentral.pro How to connect the sources Ship and collect the events ‣ Dedicated Event Hub Zentral (Open Source)
- 50. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing • Binary Auditing with Xnumon • Inspect a Software install and launch • Look into the local log file (JSON) • See process logs, with SHA-256 and code sigining informtation • Ship the logs to Elastic Stack (w/ FileBeat) • Run a quick filtering in Zentral • See filtered Events in Kibana UI DEMO #1
- 51. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing
- 52. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing
- 53. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing
- 54. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing
- 55. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing
- 56. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 1 Binary Auditing
- 57. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing • Ship same log to a commercial SaaS • Look into events in the SaaS • Next level - interconnecting Event Hubs and normalized event stream • See Events filtered in a SIEM (Security Incident Event Management) DEMO #2
- 58. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 59. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 60. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 61. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 62. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 63. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 64. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 65. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 66. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 67. zentral.pro How to connect the sources Ship and collect the events ‣ Demo 2 Binary Auditing
- 68. zentral.pro How to connect the sources Ship and collect the events ‣ Server / Cloud ‣ Log analytics • Managed Platform • High volume capability • Cost based on volume • SumoLogic • Splunk • DataDog • Elastic Cloud et.al Commercial Log Analytics Benefits
- 69. zentral.pro How to connect the sources Ship and collect the events ‣ Server / Cloud ‣ EDR / SIEM Solutions • Managed Platform • High volume capability • ArcSite • Azure Sentinel • Chronicle Security • PaloAlto Cortex XDR • Q-Radar (IBM) et.al Commercial SIEM Benefits
- 71. • Better organize event aggregation • Consolidate data in Event Hubs • SIEM alerting, Machine Learning (too many sign-in errors, …) • Bring together the admins and the security engineers zentral.pro What can be improved ‣ Benefits / Next Level Conclusion
- 72. zentral.pro What can be improved Conclusion “Bring together the admins and the security engineers“ ‣ Benefits / Next Level
- 73. zentral.pro hi@zentral.pro zentral_io Support our open source development Q & A Thank you ! zentral.pro https://github.com/ zentralopensource/ MacSysAdmin-Conference-2019 https://int.zentral.pro https://www.patreon.com/zentral https://int.zentral.pro