Dev confus.2020 compliance operator
- 1. Automating security compliance checks one scan at a time Compliance Operator Juan Antonio “Ozz” Osorio Robles Taco & beer lover… and engineer 1 Jakub Hrozek Senior Software Engineer
- 2. What we’ll talk about Agenda 2 Who are we? Why? What? How? Should we try it out? Was it easy? When? Why didn’t you do X? What’s next? Q&A
- 4. ▸ OpenShift Security & Compliance ▸ Previously in OpenStack About the presenters 4 ▸ Juan Antonio Osorio Robles - “Ozz” ▸ Mexican living in Finland Who are we?
- 5. ▸ OpenShift Security and Compliance ▸ Previously SSSD, FreeIPA About the presenters 5 ▸ Jakub Hrozek ▸ Czech living in Sweden Who are we?
- 7. About the team and the operator 7 Why? ▸ Compliance is crucial in highly regulated industries ▸ Reaching compliance at a cluster level is challenging ▸ Keeping compliance on a cluster level that’s a fast moving target is even MORE challenging ▸ Standards are written in lawyer-speak - someone needs to interpret them
- 8. About the team and the operator 8 Why? ▸ The ISC (Infrastructure Security & Compliance) team’s goal is to enable OpenShift in high security environments ･ A big part of this is helping OpenShift reach compliance to certain standards ▸ The team is turning the standards into security content ▸ We also develop tools to address any gaps we find and automate things as much as possible
- 10. What is this? 10 NIST-certified tool to scan and enforce security policies provided by the content. ContentOpenSCAP The compliance checks themselves are delivered through SCAP content, with a lifecycle independent from the operator or the OpenSCAP scanner. This is done as part of the ComplianceAsCode project compliance-operator The operator lets the administrator describe the desired compliance state of a cluster and provides them with an overview of gaps and ways to remediate them. Compliance Operator Automating compliance checks for OpenShift and CoreOS
- 11. The compliance-operator uses several custom resources to allow you to configure what you need to comply with and how in a declarative manner. 11 compliance-operator Custom Resources Profiles TailoredProfiles ProfileBundle ScanSettings ScanSetting Binding Compliance Suite Compliance Scan(s) Compliance Remediation(s) Compliance Check Result(s) Raw results
- 12. The compliance-operator uses several custom resources to allow you to configure what do you need to comply with and how in a declarative manner. 12 compliance-operator Custom Resources Profiles TailoredProfiles ScanSettings ScanSetting Binding Compliance Suite Compliance Remediation(s) Compliance Check Result(s) Raw results
- 13. What do you need to comply with? Select a policy or create a tailored one that fits your needs. What’s the organization’s policy on scanning and monitoring systems? 13 compliance-operator Figure out your policies Profiles TailoredProfiles ScanSettings ScanSetting Binding Compliance Suite Compliance Remediation(s) Compliance Check Result(s) Raw results
- 14. Profiles define the rules and variables that will be evaluated for a certain compliance standard. 14 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: Profile metadata: annotations: compliance.openshift.io/product: redhat_enterprise_linux_coreos_4 compliance.openshift.io/product-type: Node labels: compliance.openshift.io/profile-bundle: rhcos4 name: rhcos4-e8 namespace: openshift-compliance title: Australian Cyber Security Centre (ACSC) Essential Eight description: |- This profile contains configuration checks for Red Hat Enterprise Linux CoreOS that align to the Australian Cyber Security Centre (ACSC) Essential Eight. ... id: xccdf_org.ssgproject.content_profile_e8 rules: - rhcos4-accounts-no-uid-except-zero - rhcos4-audit-rules-dac-modification-chmod - rhcos4-audit-rules-dac-modification-chown ... Profiles
- 15. The available rules, which form the profiles, are also exposed as objects. This enables auditing what will be checked and how it could be fixed. 15 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: Rule metadata: annotations: compliance.openshift.io/rule: audit-rules-dac-modification-chmod control.compliance.openshift.io/NIST-800-53: AU-2(d);AU-12(c);CM-6(a) policies.open-cluster-management.io/controls: AU-2(d),AU-12(c),CM-6(a) policies.open-cluster-management.io/standards: NIST-800-53 labels: compliance.openshift.io/profile-bundle: rhcos4 name: rhcos4-audit-rules-dac-modification-chmod namespace: openshift-compliance title: Record Events that Modify the System's Discretionary Access Controls - chmod id: xccdf_org.ssgproject.content_rule_audit_rules_dac_modification_chmod ... Rules
- 16. The available rules, which form the profiles, are also exposed as objects. This enables auditing what will be checked and how it could be fixed. 16 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: Rule metadata: ... namespace: openshift-compliance ... description: '<code>auditd</code><code>augenrules</code><code>.rules</code><code>/etc/audit/rules.d</code><pre>-a always,exit -F arch=b32 -S chmod -F auid>=1000 -F auid!=unset -F key=perm_mod</pre><pre>-a always,exit -F arch=b64 -S chmod -F auid>=1000 -F auid!=unset -F key=perm_mod</pre><code>auditd</code><code>auditctl</code><code>/etc/audit/audit.rules</code><pre>-a always,exit -F arch=b32 -S chmod -F auid>=1000 -F auid!=unset -F key=perm_mod</pre><pre>-a always,exit -F arch=b64 -S chmod -F auid>=1000 -F auid!=unset -F key=perm_mod</pre>' rationale: The changing of file permissions could indicate that a user is attempting to
gain access to information that would otherwise be disallowed. Auditing DAC modifications
can facilitate the identification of patterns of abuse among both authorized and
unauthorized users. severity: medium warning: Note that these rules can be configured in a
number of ways while still achieving the desired effect. Here the system calls
have been placed independent of other system calls. Grouping these system
calls with others as identifying earlier in this guide is more efficient. ... Rules(continued)
- 17. The available rules, which form the profiles, are also exposed as objects. This enables auditing what will be checked and how it could be fixed. 17 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: Rule metadata: ... name: rhcos4-audit-rules-dac-modification-chmod ... - disruption: medium fixObject: apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:,-a%20always%2Cexit%20-F%20arch%3Db32%20-S%20chmod%20-F%20auid%3E%3D1000%20-F%20 auid%21%3Dunset%20-F%20key%3Dperm_mod%0A-a%20always%2Cexit%20-F%20arch%3Db64%20-S%20c hmod%20-F%20auid%3E%3D1000%20-F%20auid%21%3Dunset%20-F%20key%3Dperm_mod%0A filesystem: root mode: 420 path: /etc/audit/rules.d/75-chmod_dac_modification.rules Rules (continued)
- 18. Profiles are product dependent. For instance, there are different profiles for OCP (Platform checks) and RHCOS (Node checks). We (Red Hat) provide out of the box, ready to use profiles. 18 compliance-operator $ oc get profiles.compliance NAME AGE ocp4-e8 25m ocp4-moderate 25m ocp4-ncp 25m rhcos4-e8 25m rhcos4-moderate 25m rhcos4-ncp 25m Profiles
- 19. Allow for admins to tweak Profiles to fit their needs. 19 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: TailoredProfile metadata: name: rhcos4-e8-custom spec: extends: rhcos4-e8 title: My little profile (based on the e8 profile) disableRules: - name: ocp4-file-permissions-node-config rationale: This breaks X application. setValues: - name: ocp4-var-selinux-state rationale: trolling dwalsh value: permissive TailoredProfiles
- 20. Defines the operational policies on running scans. 20 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: ScanSetting metadata: name: default namespace: openshift-compliance rawResultStorage: rotation: 3 size: 1Gi scanTolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists schedule: ‘0 1 * * *’ roles: - worker - master ScanSettings
- 21. Tell the operator what you want 21 compliance-operator Specify your intent Profiles TailoredProfiles ScanSettings ScanSetting Binding Compliance Suite Compliance Remediation(s) Compliance Check Result(s) Raw results
- 22. Define the link between profiles and the operational policies (ScanSettings). With this one can say: This cluster should be evaluated for X Profile(s) with Y settings. 22 compliance-operator apiVersion: compliance.openshift.io/v1alpha1 kind: ScanSettingBinding metadata: name: my-companys-compliance-requirements profiles: # Node checks - name: rhcos4-e8-custom kind: TailoredProfile apiGroup: compliance.openshift.io/v1alpha1 # Platform checks - name: ocp4-e8 kind: Profile apiGroup: compliance.openshift.io/v1alpha1 settingsRef: name: default kind: ScanSetting apiGroup: compliance.openshift.io/v1alpha1 ScanSettingBindings
- 23. Monitor the status of the scans. 23 compliance-operator Keep track Profiles TailoredProfiles ScanSettings ScanSetting Binding Compliance Suite Compliance Remediation(s) Compliance Check Result(s) Raw results
- 24. Help you keep track on the state of the scans. Issue relevant events. 24 compliance-operator $ oc get compliancesuites NAME PHASE RESULT my-companys-compliance-requirements RUNNING NOT-AVAILABLE $ oc get events --field-selector involvedObject.kind=ComplianceSuite,involvedObject.name=my-companys-compliance-requirements LAST SEEN TYPE REASON OBJECT MESSAGE 23m Normal ResultAvailable compliancesuite/my-companys-compliance-requirements ComplianceSuite's result is: NON-COMPLIANT ComplianceSuites
- 25. View the state of compliance of the system. Audit possible remediations and apply them if necessary. Download raw results and provide them to your auditor. 25 compliance-operator Results Profiles TailoredProfiles ScanSettings ScanSetting Binding Compliance Suite Compliance Remediation(s) Compliance Check Result(s) Raw results
- 27. ProfileBundles 27 ▸ Built from the ComplianceAsCode/content project ･ Checklists and definitions for a product get packed in a datastream (in XML format) ▸ We package it in a container image ▸ The operator detects the ProfileBundle object and schedules a Deployment that will parse the XML ･ This generates the Profiles and Rules ･ These are meant for usability Where does the content come from?
- 28. ComplianceScans 28 ▸ A ComplianceScan represents a single scan ･ Needs all the parameters to run OpenSCAP: profile ID, image to get the content from, data stream file path ▸ When this object is detected, the operator schedules a pod on every applicable node. ･ Results are generated as ConfigMaps. ▸ A “result server” Deployment gathers raw results from the scan into a Persistent Volume ･ An ephemeral PKI is created to ensure these results get transmitted over (m)TLS ▸ An “aggregator” gathers the ConfigMaps and generates the results, remediations, and detects inconsistencies. How are scans done?
- 29. ComplianceScans 29 ▸ A Node scan verifies the OS configuration (e.g. RHCOS) ･ It’ll spawn a privileged pod per selected node with read access to the RHCOS host. ･ OpenSCAP is ran on a pod in each host ▸ A Platform scan verifies the OCP/Kubernetes configuration. ･ It’ll spawn one (non-privileged) pod ･ The pod has an init container (we call it api-resource-collector) that pre-fetches the kube objects. ･ The api-resource-collector parses the content, looks for the kube objects used and calls the kube API to fetch them and stage them in a shared volume ･ OpenSCAP is ran once and the pod has no access to the host (in offline mode). Platform or Node scan?
- 30. ComplianceSuites 30 ▸ A ComplianceSuite keeps track of several scans ･ The scans set in the suite object get created and then detected by the scan controller ▸ If a schedule is set for the suite, it’ll create a CronJob that annotates the scans to be re-run ▸ If auto applying remediations is done, it’ll wait for the scans to be ready, gather the remediations for the scans, and set the “apply” flag What are suites for?
- 31. ScanSettingBindings 31 ▸ Aims to improve usability by automating the parameter setting of suites. ▸ No need to know profile ids, content paths, what type of scan you need or even setting the appropriate container image, this is done for you. What are scan setting bindings for?
- 32. ScanSettingBindings 32 ▸ OpenSCAP outputs results in ARF format (a BIG XML file) ▸ These results are too big for ConfigMaps ･ There’s a 1MB limit due to etcd ▸ We store these results in a Persistent Volume Raw Results
- 33. Should we try it out? demo 33
- 34. Demo time 34 compliance-operator https://asciinema.org/a/352570 Let’s try it out!
- 36. Challenges 36 ▸ Raw results (ARF) are stored in a Persistent Volume ▸ Not all clouds have Persistent Volume implementations that support ReadWriteMany access mode ･ Some implementations support ReadWriteOnce only ▸ The result server addresses this by being the only pod that mounts the Persistent Volume ･ All scan pods send results to this server via HTTPs ･ We needed an ephemeral PKI to lock this down (it uses mTLS) Result server
- 37. Challenges 37 ▸ The cluster consists of potentially many nodes, but we still want to represent the scan with one result ▸ Nodes in OpenShift are grouped into MachineConfigPools which /should/ be identical ▸ ...but what if one node or several are different? Cluster result convergence
- 38. Challenges 38 ▸ The cluster consists of potentially many nodes, but we still want to represent the scan with one result ▸ Nodes in OpenShift are grouped into MachineConfigPools which /should/ be identical ▸ ...but what if one node or several are different? ▸ The check and the scan are marked as INCONSISTENT for better visibility ▸ The operator tries to find the most common state (=consensus) and flag the nodes that differ ▸ If possible, the operator still enables the admin to converge to the desired state Cluster result convergence
- 39. Challenges Content updates ▸ The remediations are stored as k8s objects in the content repository ▸ While you can patch an object, typically you’d need to replace the whole object payload (MachineConfig, ConfigMap, …) ▸ ..what if the defaults change or the remediation needs to be updated? ▸ The remediation object is flagged as OUTDATED ▸ The object would contain the current and the updated contents, the admin would review and apply the updated contents on their own 39
- 40. When? release 40
- 41. Compliance-operator release 41 OCP 4.6 The operator itself will be released with 4.6 and will be available in OperatorHub as an official Red Hat operator. The content will not be tied to the release, and we’ll be able to update it further even after 4.6 * Note: We require some fairly new kube APIs, so older releases won’t work
- 42. Why didn’t you do X? reasoning 42
- 43. Why didn’t you do X? FAQ reasoning 43
- 44. 44 ▸ Why OpenSCAP/ComplianceAsCode? ▸ Isn’t OpenSCAP mostly used in hardcore RHEL shops? ▸ Why didn’t you use OPA? ▸ Can I try this in OCP 4.3? ▸ What about RHEL? ▸ So… This makes us compliant, right? FAQ
- 46. 46 ▸ Content… content… content... ▸ Compliance-operator ♡ RHACM ･ We already issue events that RHACM can consume ･ More granular compliance results in RHACM ･ RHACM deploying compliance-operator by default What’s next?
- 48. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 48 Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you