The Future of Security and Productivity in Our Newly Remote World
0 likes275 views
The document discusses strategies for attacking and defending cloud-native infrastructure, focusing on security principles in Kubernetes and container environments. It emphasizes the importance of collaboration, automation, and testing throughout the application lifecycle, as well as the need for coding security policies and identifying vulnerabilities. Various tools and practices are mentioned for threat modeling, dynamic testing, and ensuring compliance with security standards within cloud-native environments.
![kubesec.io - example insecure pod
[
{
"object": "Pod/kubesec-demo.default",
"valid": true,
"message": "Passed with a score of 1 points",
"score": 1,
"scoring": {
"advise": [
{
"selector": "containers[] .securityContext .capabilities .drop",
"reason": "Reducing kernel capabilities available to a container limits its attack
surface"
},
{
"selector": ".spec .serviceAccountName",
"reason": "Service accounts restrict Kubernetes API access and should be configured
with least privilege"
},
{
"selector": "containers[] .resources .requests .cpu",
"reason": "Enforcing CPU requests aids a fair balancing of resources across the
cluster"
},
...](https://image.slidesharecdn.com/controlplane-attackinganddefendingcloudnativeinfrastructurewhitesourcewebinaraugust2020-200825175338/85/The-Future-of-Security-and-Productivity-in-Our-Newly-Remote-World-24-320.jpg)
![goss
command:
java -version:
title: Checking Java JRE version and architecture
exit-status: 0
stdout: []
stderr:
- /^openjdk version "1.8.*/
- /^OpenJDK 64-Bit Server VM .*/
timeout: 1000](https://image.slidesharecdn.com/controlplane-attackinganddefendingcloudnativeinfrastructurewhitesourcewebinaraugust2020-200825175338/85/The-Future-of-Security-and-Productivity-in-Our-Newly-Remote-World-27-320.jpg)
![Self Destructive SelfSubjectAccessReview
apiVersion: authorization.k8s.io/v1
kind: SelfSubjectAccessReview
metadata:
name: yaml-dos
namespace: default
data:
a: &a ["HONK","HONK","HONK","HONK","HONK","HONK","HONK","HONK","HONK"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e]
g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f]
h: &h [*g,*g,*g,*g,*g,*g,*g,*g,*g]
i: &i [*h,*h,*h,*h,*h,*h,*h,*h,*h]
j: &j [*i,*i,*i,*i,*i,*i,*i,*i,*i]
k: &k [*j,*j,*j,*j,*j,*j,*j,*j,*j]
l: &l [*l,*l,*l,*l,*l,*l,*l,*l,*l]](https://image.slidesharecdn.com/controlplane-attackinganddefendingcloudnativeinfrastructurewhitesourcewebinaraugust2020-200825175338/85/The-Future-of-Security-and-Productivity-in-Our-Newly-Remote-World-67-320.jpg)
The Future of Security and Productivity in Our Newly Remote World
- 1. Attacking and Defending Cloud Native Infrastructure WhiteSource Webinar, August 2020 @sublimino and @controlplaneio
- 2. I’m: - Andy - Dev-like - Sec-ish - Ops-y
- 3. I’m: - Andy - Dev-like - Sec-ish - Ops-y Course author: - SANS SEC584 - Cloud Native Security: Defending Containers and Kubernetes - O’Reilly - ControlPlane Trainer: - Hashicorp - Docker
- 6. ● Applications, infrastructure, security, and policy are all defined as code ● Everything is built as a type of software ● Similar controls can be applied to entire classes of software, containers, and systems (static analysis, composition scanning etc) Everything is Software
- 7. DoD Enterprise DevSecOps Reference Design Securing the entire application and infrastructure lifecycle ● Collaboration ● Automation ● Containerisation ● Testing SecOps
- 8. Third Party Code Risk and Supply Chain Security ● Build stages and their artefacts can be cryptographically signed to provide a chain of trust ● Software dependencies pulled into a secure build environment should be scanned for CVEs and their signatures verified where available
- 9. Defending ● Prevent network egress ● Isolate from the host's kernel ● Execute RUN commands a non-root user in container filesystem ● Run build process as a non-root user ○ or in a user namespace ● Share nothing non-essential
- 10. Kubernetes Attack Trees - Compromised Container Scenario
- 11. kubesim.io - K8S Hacking and Hardening Simulator ● infrastructure deployment ● cluster provisioning and workload configuration ● scenario runner with challenges, hints, and scoring ● raw command line experience ● open source core at https://github.com/kubernetes-simulator/simulator ●
- 12. What are we doing today? ● Burning some things ○ And trying to extinguish the flames ● For each attack ○ Intro ○ Demo ○ Remediate ○ DevSecOps-ify
- 13. Docker and Kube ● Kube doesn't love us ● Y so difficult? ● What's the problem?
- 14. Container Evolution ● Containers are awesome ● But they emerged from the primordial kernel soup ● We love Kube
- 16. SEC584 | Defending Cloud Native Infrastructure Kubernetes Logical Overview
- 17. SEC584 | Defending Cloud Native Infrastructure Where is Kubernetes vulnerable? ● Application workloads ● Workload configuration ● Cluster configuration - developer ● Cluster configuration - operations ● Cluster deployment http://nobleillustrations.blogspot.com/2012/08/kraken-rum-animations.html
- 18. SEC584 | Defending Cloud Native Infrastructure Attacking the Visible Horizon ● Self-namespace info in `/proc` ● Local filesystem, debug tools ● Mounted volumes from host ● All other pods, flux, helm ● Host Kernel, container runtime ● API server(s), serviceAccounts ● Node ● Datastores ● Registry ● Build server ● Supply chain ● Internet
- 19. LET'S GO
- 22. “Linting” Dockerfiles - Static Analysis ● https://github.com/projectatomic/dockerfile_lint - rule based 'linter' for Dockerfiles ● https://github.com/hadolint/hadolint - Dockerfile linter, validates inline bash with Shellcheck
- 24. kubesec.io - example insecure pod [ { "object": "Pod/kubesec-demo.default", "valid": true, "message": "Passed with a score of 1 points", "score": 1, "scoring": { "advise": [ { "selector": "containers[] .securityContext .capabilities .drop", "reason": "Reducing kernel capabilities available to a container limits its attack surface" }, { "selector": ".spec .serviceAccountName", "reason": "Service accounts restrict Kubernetes API access and should be configured with least privilege" }, { "selector": "containers[] .resources .requests .cpu", "reason": "Enforcing CPU requests aids a fair balancing of resources across the cluster" }, ...
- 25. Pod YAML - Isolation-Breaking Configs ● SecurityContexts: use for pods and containers ● Dangerous pod configurations; ○ Running as root (no user namespace support in Kubernetes) ○ Privileged (can perform root operations on the host) ○ No seccomp/AppArmor profile (unlike Docker, no default profile in Kubernetes) ○ Full RBAC (administrative access to cluster) ○ Mounting host volumes (resource contention, side-channel communication) ○ Sharing host pid, network, or IPC namespaces (can lead to escalation) ○ AllowPrivilegeEscalation (permits escalating to root inside the container) ○ Excess capabilities (violates least privilege) ○ No cgroups (unbounded resource consumption)
- 26. “Unit Testing” Containers - Dynamic Testing ● inspec ● serverspec ● goss
- 27. goss command: java -version: title: Checking Java JRE version and architecture exit-status: 0 stdout: [] stderr: - /^openjdk version "1.8.*/ - /^OpenJDK 64-Bit Server VM .*/ timeout: 1000
- 29. What is this? ● DirtyCOW. A copy-on-write vulnerability in the kernel from 2016 ● Allows a malicious user to gain root on the host from inside a container ● Was being exploited in the wild
- 30. "One of the sites I manage was compromised, and an exploit of this issue was uploaded and executed. A few years ago I started packet capturing all inbound HTTP traffic and was able to extract the exploit and test it out in a sandbox" http://www.v3.co.uk/v3-uk/news/2474845/linux-users-urged-to-prot ect-against-dirty-cow-security-flaw Dirty COW (2016)
- 31. Why is it bad? ● It pretty much hoses down your system and flushes the corpse ● Linux kernel since 2.6.22 (July 2007) ○ Fixed in 4.8.3, 4.7.9, 4.4.26 or newer (Oct 2016) ● All Docker versions ○ it’s a kernel bug: container syscalls hit host
- 32. Dirty COW (2016) Nondeterministic Live Demo (backup available: YouTube)
- 34. Security Contexts ● DAC (discretionary access control) ○ File system permissions, the basis of linux security ● Capabilities ○ Subdivides the full set of root capabilities into smaller buckets ○ Not perfect, but can limit the shape of the privilege that the user has ○ Beware CAP_SYS_ADMIN - the capability bucket ● Sandboxing ○ seccomp-bpf (user-supplied code running in kernel) ○ Filtering syscalls to reduce attack surface. Disallowed system calls get SIGKILLed ○ Seccomp is enabled by default in Docker, but NOT IN KUBERNETES ○ It should be mandated via a PodSecurityPolicy ● Mandatory Access Control ○ SELinux (RHEL/CentOS only) ○ AppArmor (Ubuntu, Debian and derivatives) - defaults on in Docker, off in Kubernetes
- 35. Docker’s Default seccomp Profile ● Docker uses a JSON DSL for seccomp profiles that compile down to eBPF (i.e. seccomp-bpf), and are run in the kernel ● Only whitelisted system calls are permitted ● Docker’s default seccomp whitelist blocks some dangerous system calls: ○ add_key, keyctl, request_key: Prevent containers from using the kernel keyring, which is not namespaced ○ clone, unshare: Deny cloning new namespaces. Also gated by CAP_SYS_ADMIN for CLONE_* flags, except CLONE_USERNS. ● “I specifically wanted to block cloning new user namespaces inside containers because they are notorious for being points of entry for kernel bugs” ● Seccomp security profiles for Docker https://blog.jessfraz.com/post/a-rant-on-usable-security/
- 36. Writing Effective Seccomp Profiles ● Dynamic ○ "Observational" or "learning" security systems will watch your application's behaviour at runtime in a pre-prod environment and generate a profile ○ This has limited value, as it requires a pre-prod system to demonstrate the complete set of behaviours it exhibits in production - this is a sadly unrealistic goal ○ Some distance can be made with this approach and a comprehensive test suite, but logging, crashing, and stress-related behaviours are unlikely to be comprehensively covered, resulting in potential production downtime ○ This approach can be used to inform the final policy ● Static ○ Laborious ○ Requires in-depth knowledge of the linux syscall interface
- 37. bypass container security controls
- 38. bypass container security controls
- 39. AppArmor bypass
- 40. Minimal PoC
- 41. /proc/self/exe
- 42. lesson: patch your hosts ● the kernel is the basis of container security ● containers doesn't really exist! ● as our reputable track hosts from Canonical's LXD team put it -- containers are a userspace fiction
- 43. containers are a userspace fiction ● there is no kernel representation of a container, it is an emergent property of a collection of stimuli and restrictions born from unintelligent design and years of evolution ● a lot like consciousness
- 44. <mind blown>
- 45. Security Test Suite! ● Testing is a Dark Art ● Anything can be a security test ● Arrange (set env) ● Act (perform test and capture result) ● Assert (does this match expectation?) ● Prove test fails as expected ● Beware acceptance testing and push that test as low as it can go
- 46. DevSecOps
- 47. testing: out of date versions ● well, this test is simple. What kernel version am I running? And what docker version? ● fancy security tooling will do this for you, OR you can use some basic tests for your node
- 48. goss ● goss is "go serverspec" ● it's not the ultimate test tool, but it's pretty good
- 50. TESTING IS COOL ● It’s how we “prove” we’re secure ● Against known quantities ● Test enough to install confidence ● “Goldilocks Test Suites” -- not too many, not too few
- 51. too easy
- 52. public clusters drive by pew pew pew
- 53. hunting clusters in the wild
- 55. binaryedge ● this platform has already portscanned the IPv4 address space for us ● indexed all the content ● and preemptively attacked it ● THIS MAY NOT BE LEGAL IN YOUR JURISDICTION ● you may not care ● but sadly, on stage and on camera, I do
- 56. binaryedge THIS MAY NOT BE LEGAL IN YOUR JURISDICTION ● this platform has already portscanned the IPv4 address space for us ● indexed all the content ● and preemptively attacked it
- 57. binaryedge
- 58. binaryedge
- 59. Here’s one I pwnd earlier
- 60. The Banner Banhammer ● let's test the API server and see if it's leaking ○ Use my nice nmap script https://gist.github.com/sublimino/c357379369808d0f77d3e2fe86fd4611
- 62. Watch It Burn ● unauthenticated client ● malicious websockets upgrade ● potential escalation of privilege
- 63. Here cometh the Lesson ● Tin foil hats are cool ○ DEFCON says so ○ Defence in depth isn’t going anywhere ● Don't run a Kubernetes API server endpoint on the public internet!!!1one
- 64. Kubernetes API Servers on the Public Internet
- 65. Billion Laughs
- 67. Self Destructive SelfSubjectAccessReview apiVersion: authorization.k8s.io/v1 kind: SelfSubjectAccessReview metadata: name: yaml-dos namespace: default data: a: &a ["HONK","HONK","HONK","HONK","HONK","HONK","HONK","HONK","HONK"] b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a] c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b] d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c] e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d] f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e] g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f] h: &h [*g,*g,*g,*g,*g,*g,*g,*g,*g] i: &i [*h,*h,*h,*h,*h,*h,*h,*h,*h] j: &j [*i,*i,*i,*i,*i,*i,*i,*i,*i] k: &k [*j,*j,*j,*j,*j,*j,*j,*j,*j] l: &l [*l,*l,*l,*l,*l,*l,*l,*l,*l]
- 68. # CVE-2019-11253 # https://github.com/kubernetes/kubernetes/issues/83253 # Shout out: @raesene for poc collab, @iancoldwater + @mauilion for # HONKing inspiration and other guidance. # Description: In Kubernetes 1.13 and below, the default configuration # is that system:anonymous can request a selfsubjectaccessreview # via mechanisms such as "kubectl auth can-i". This request can # include POSTed YAML, and just the act of trying to parse it causes # excessive memory usage by the API server. Anywhere from about 10 # to 100 concurrent requests of this nature can overwhelm the API # server's resources and cause it to become unresponsive to the point # that the worker nodes and user's running kubectl will believe the # control plane is offline. Since requests can last up to 60s by # default before the timeout kicks in, sustaining the attack only # requires between 10 and ~100 requests per minute.
- 69. # CVE-2019-11253 # Recommendation: Update Kubernetes to a release that includes YAML # parsing resource limits and limit direct, public access to API # servers. See the above GH issue for details. # https://github.com/kubernetes/kubernetes/issues/83253
- 70. HONK!
- 71. Testing testing 123 ● Arrange ● Act ● Assert For network infrastructure?
- 72. Network Policy Testing: nmap with Netassert.io k8s: # used for Kubernetes pods deployment: # only deployments currently supported test-frontend: # pod name, defaults to `default` namespace test-microservice: 80 # `test-microservice` is the DNS name of the target service test-database: -80 # test-frontend should not be able to access test-database port 80 new-namespace:test-microservice: # `new-namespace` is the namespace name test-database.new-namespace: 80 # longer DNS names can be used for other namespaces test-frontend.default: 80 default:test-database: test-frontend.default.svc.cluster.local: 80 # full DNS names can be used test-microservice.default.svc.cluster.local: -80 https://github.com/controlplaneio/netassert#configuration Network security testing with highly parallelised nmap https://github.com/controlplaneio/netassert
- 73. bats-core ● Test library for Bash ● TEST EVERYTHING EVERYWHERE ● https://github.com/bats-core/ bats-core/
- 74. bats-core
- 75. break out of service meshes
- 76. Istio Threats ● Insecure Control Plane Configuration ● User Misconfiguration ○ Avoid manual configuration ○ Regularly apply config defined in git (CI/Gitops) ● Compromised workload attacks Istio sidecar ○ Defence in depth: use dedicated Egress Gateway, K8S Network Policy & IaaS FW rules ● Init Containers Run Off-mesh ○ Unconstrained by istio security ○ Use K8s network policy ● PodSecurityPolicy blocks Istio init & sidecar ○ CNI Plugin ○ Sub-pod isolation proposals ● Misconfigured app container could run as privileged or use NET_ADMIN to exit the mesh
- 80. Lesson ● secure by default is difficult ● IDS ○ See: falco, $$$ offerings ● Want more? See: Security Rationale for Istio by Rowan Baker
- 81. evade detection
- 82. De-auditing K8S ● Reconfigure API server ● Blackhole traffic to remote logging endpoint ● DOS remote logging endpoint https://monzo.com/blog/we-built-network-isolation-for-1-500-services
- 83. hacking in a safe space
- 84. Simulator! ● https://github.com/kubernetes-simulator/simulator ● See “How to Train Your Red Team (for Cloud Native)” by ControlPlane
- 85. kubesim.io - K8S Hacking and Hardening Simulator ● infrastructure deployment ● cluster provisioning and workload configuration ● scenario runner with challenges, hints, and scoring ● raw command line experience ● open source core at https://github.com/kubernetes-simulator/simulator ●
- 86. K8S Breach Lessons ● Application workloads ○ container images (app, OS, deps, filesystem conf, user, baked-in secrets), malicious images ● Workload configuration ○ pod spec (privileged/PSP config, env vars, service accounts, file mounts, runAsUser, seccomp/selinux), runtime (docker, crio), priv workloads, identity, datastores, traffic sniffing/nmap ● Cluster configuration - developer ○ authn/RBAC, NetPol/networking ● Cluster configuration - operations ○ admission control, etcd encryption, control plane TLS, local/remote registry, dashboard/API, federated identity ● Cluster deployment ○ control plane config, user access, topology, transport & TLS, infrastructure/cloud, OS http://nobleillustrations.blogspot.com/2012/08/kraken-rum-animations.html
- 87. How to Train your Red Team (for Cloud Native) ● Hosted: kubesim.io ● Open source: https://github.com/kubernete s-simulator/simulator ● Attack Trees: https://github.com/cncf/finan cial-user-group/tree/master/p rojects/k8s-threat-model ● Training: https://control-plane.io/
- 89. Recap: Applications in Kubernetes ● Just applications, linux processes, memory, and filesystems ● More granular security profile ○ Easier to harden ○ Security boundary is the container process or pod, not the whole instance ○ Controlled networking environment
- 90. Layers of Security Testing ● Infrastructure: server hardening/conformance ● Supply chain: image validation, Kubernetes deployment YAML validation ● Runtime: application behaviour, telemetry, session handling, and networking
- 91. What's the Problem with Security Testing? ● What's the Problem with Security Testing? ○ False Positives ● Why? ○ Defensive security measures and nondeterministic test environment ● Mitigation ○ Reduce signal-to-noise ratio with targeted testing and retry budgets
- 92. Getting the House in Order ● Ensure that applications have a local testing story for developers ○ Build server verifies tests that developers are able to run locally ○ This proves out mocking and stubbing harnesses ● Once developer local testing is in place, testing can be as simple as standing up docker-compose with mocks and mock data ○ https://github.com/kubernetes/kompose can help migration ● Continuous Security (...ish) ○ A lot of tooling isn’t container-native ○ But continuous scanning leaves deeper exploration for humans
- 93. Prepare for the Unexpected
- 94. SEC584: Defending Cloud Native Infrastructure BETA Pricing: 50% Discount ($2100 USD) Live Online | 3-day beta October 12 - 14 with Andrew Martin & Eric Johnson https://www.sans.org/event/sec584-beta-one-2020