Cloud firewall logging
Download as PPTX, PDF0 likes56 views
The document outlines the enhancements to Cloud Firewall (CFW) logging for Triton deployments, which now include detailed event logging of dropped packets and connections in compliance with Korean law. It describes various system changes, including the introduction of a new device (/dev/ipfev) for event generation and the implementation of a daemon (cfwlogd) for processing and logging these events in JSON format. Additionally, it discusses future developments and integration with Triton services for better log management and accessibility.
Cloud firewall logging
- 2. Overview ● Cloud Firewall (CFW) Logging records the following events: ○ Dropped inbound packets. ○ New TCP connections or UDP sessions. ● It does so in a manner that’s aggregatable for a Triton deployment. ● Specified in RFD 163, the changes needed were “full stack” (ick, pardon the cliché). ○ illumos and fwadm(1M), CFW log daemon, *API support. ○ This talk will proceed from the bottom up.
- 3. Motivation ● Korean law requires logging on inbound connections. ● Logging per-guest presents some complications: ○ ipfilter’s existing ipmon requires one file descriptor (FD) per VM if you wish to aggregate. ○ existing ipfilter already does most of the work we need with its “keep state” option. ● We want to be able to add “log” to a Triton-administered firewall rule.
- 4. Constraints ● As mentioned earlier, we really don’t want one FD per VM. ● Yet we need per-VM attributions of CFW Logging events. ● We need to make sure a post-CFW-Logging platform works on older Triton deployments. ● Or have newer Triton deployments understand older PIs won’t allow CFW-Logging.
- 5. illumos changes - /dev/ipfev ● All illumos changes are encapsulated as OS-7667. ● Creates a new global-zone-only device: /dev/ipfev ● /dev/ipfev is only openable by one process. ● Its being open is a check for performing event generation. ● Events are 88bytes. Listed below by purpose, not by layout: ○ Type, length, zone DID (unique) ○ IP packet information (src, dst, proto, ports) ○ ipfilter ruleid, CFW rule UUID ● Both Zone DID and rule UUID needed to be added to ipfilter’s internal state.
- 6. illumos changes - ipfilter rules ● ipfilter in SmartOS has two types of filter rules: ○ Traditional root@zone per-netstack ones (incl. global) ○ Global-zone-administered ones for non-global zones. ■ These are the ones we wish to monitor. ■ We now cache the zone DID in these rulebases. ● New ipf(4) extensions (we chose rule tags) needed for: ○ Do we care if this ipfilter rule is CFW-logged? ○ If so, what is the CFW rule UUID? ● CFW-logged rules need to keep state (for new connections) ○ ... keep state set-tag(cfwlog,uuid=a7d57476-4699-4d2a-b4ae-7af857fea3d5)
- 7. illumos changes - /dev/ipfev in context ● So every event generated by any VM’s ipfilter gets put on a global ring buffer. ● The single /dev/ipfev consumer reads off the ring buffer. COMPUTE NODE RING BUFFER VM/Zone VM/Zone VM/ZoneVM/Zone VM/Zone VM/ZoneVM/ZoneVM/Zone /dev/ipfev cfwlogd
- 8. smartos-live changes ● All smartos-live changes encapsulated as OS-7668. ● New fwadm versioning mechanism. (As of this, Version 2.) ● fwadm(1M) rule json now has new boolean: log. ● Existing log and uuid pass through to new ipf(4) rules mentioned earlier.
- 9. cfwlogd ● Needed a performant userland daemon capable of connecting to “/dev/ipfev” and sucking down events out of the ring buffer. ● The daemon needs to translate events from raw bytes represented by cfwev_t in the kernel into new line separated JSON formatted logs ● There’s been a recent interest in Rust from multiple engineering groups within Joyent.
- 10. cfwlogd ● Rust has a great FFI interface to interface with OS level concepts such as issuing ioctls. ● There’s a great serialization/deserialization crate (library) in rust called serde. ○ Gives us the ability to translate from raw bytes directly to JSON ● Rust offers safety and performance out of the box!
- 11. cfwlogd - gathering info ● We need to be able to get the customer uuid, the zone uuid, and the zone alias when we only have the zonedid from the kernel ○ We have something that does this already, it’s called vminfod and it provides access to this information in real time via an http stream! ● Match the information from vminfod with the event from the device ring buffer and use serde to serialize the log out to disk as newline separated JSON.
- 12. cfwlogd - data flow vminfod 127.0.0.1:9090 /dev/ipfev cfwlogd global zone kernel Customer 1 Zone 1 Alias: node-app1 Customer 2 Zone 2 Alias: web1 Customer 2 Zone 1 Alias: openvpn1 Customer 1 Zone 2 Alias: ruby-app2 ixgb0 - External Network ● Runs as an agent in the global zone. ● Drops as many privileges(5) as it can at startup after opening /dev/ipfev ● Talks to /dev/ipfev to suck down events out of the Kernel’s ring buffer ● Correlates the events with information gathered from vminfod in real time
- 13. cfwlogd - data flow continued vminfod Event Reader Thread Fanout Thread /dev/ipfev Vminfod Watcher Thread Signal Listener Thread Zone Logger Thread Zonedid: 1 Customer: 1 Alias: Redis Zone Logger Thread Zonedid: 2 Customer: 2 Alias: OpenVPN Zone Logger Thread Zonedid: 3 Customer: 3 Alias: Nginx CN Filesystem: /var/log/firewall/<customer>/<zone uuid>/current.log Main Thread SIGHUP SIGTERM SIG... cfwlogd
- 14. cfwlogd - setup ● Install/Modify/Check cfwlogd ○ sdcadm post-setup firewall-logger-agent ○ sdcadm experimental update firewall-logger- agent@<uuid> -C experimental ○ sdcadm health firewall-logger-agent ● If cfwlogd lands on a CN that doesn’t have “/dev/ipfev” it gracefully exits telling SMF that the service was successful but leaves no process running. ○ Also useful if a CN needs to get it’s PI rolled back.
- 15. cfwlogd - next steps ● Want to incorporate CMON metrics to alert operators that cfwlogd’s internal queues are maxed out and we are dropping events. (We try as hard as possible to remain running). ● Want more control over queue size per zone so that one bad actor doesn’t hurt everyone. (Serializing data and writing to disk is slower than event generation).
- 16. cfwlogd - next steps continued ● Would like to break out the vminfod client into a more generic crate that other consumers could use. ● Add support for any future event types such as “Ruleset change”. ● Explore a different model other than a thread per zone once zone density increases per CN?
- 17. Triton changes ● Need to build "something" to take the log files generated by cwflogd and put them into manta under a location the user can have access to, maybe use RBAC … ● ... we already got Hermes to do that for Triton services from the sdc zone. ● Should not add more stuff to the sdc zone. If possible, what we should do is to remove from there.
- 18. Triton changes: logarchiver ● New service/zone pair logarchiver, including its own Hermes instance, which is able to deploy its own agent logarchiver-agent that will run in parallel with hermes- agent. ● sdcadm post-setup logarchiver ● Same configuration than the sdc zone to talk to manta. ● Manta location for log files: /:customer_login/reports/firewall- logs/:year/:month/:day/:vm_uuid/:iso8601stamp .log.gz
- 19. Triton changes: Hermes ● Added ability to connect customers’ UUIDs with manta accounts using MAHI. ● Ability to deploy different agents FMRIs depending on the zone running hermes-proxy. ● Ability to delete some logs based in the same logsets we use to upload log files.
- 20. Triton changes: log (Boolean) ● fwrules have now a new attribute `log (Boolean)` which defaults to false when not present. ● AdminUI: Checkbox for log or not fwrules. ● CloudAPI/node-triton: Log column added to rules list. `-l|-- log` option added to rule creation/update. $ ./bin/triton -i fwrule ls SHORTID ENABLED GLOBAL LOG RULE 78d77145 true - true FROM any TO vm 3a2b9998... BLOCK tcp PORT 666
- 21. Triton changes: firewaller-agent ● ipf*.conf files modifications are not backwards compatible. ● requires a service that makes sure that each zone gets the rules which are compatible with the running system: firewaller-config-migration. ● vmadm and zones services depend on firewaller- config-migration
- 22. Triton changes: firewaller-agent ● There’s a # smartos_ipf_version <version> line into ipf*.conf files to determine which config is used. ● The configuration supported by a CN is at /etc/ipf/smartos_version, which consist of a single line containing a single integer. ● Rules tag fragments added/removed depending on versions set-tag(uuid=rule.uuid,cfwlog).
- 23. 23© 2019 Joyent. All rights reserved. Joyent Confidential Thank you for watching!