Cloud forensics putting the bits back together
1 like454 views
The document discusses forensic investigations of AWS EC2 instances and EBS volumes. It details the process the author took to launch EC2 instances with different EBS volume types, write and delete files, snapshot the volumes, and use forensic software to recover deleted files from the snapshots. The results showed that standard, gp2 and io1 volume types had the highest recovery rates of deleted files from snapshots, while sc1 and st1 volume types recovered fewer files and in some cases produced anomalously large PDF files. Maintaining chain of custody of forensic evidence and using separate AWS accounts was recommended to safeguard recovered data.
Cloud forensics putting the bits back together
- 1. Putting the Bits Back Together Forensic investigations in AWS EC2
- 2. $ whoami Brandon Sherman Senior Cloud Security Engineer at Twilio, Inc. Krav Maga instructor Has cloudy outlook Certified Solutions Architect, Professional
- 3. $ whoami Brandon Sherman Senior Cloud Security Engineer at Twilio, Inc. Krav Maga instructor Has cloudy outlook Certified Solutions Architect, Professional A picture, just in case you’ve forgotten what I look like already Or are in the back This is a better looking photo than the live version the people at the front get
- 4. ~/agenda Why? Background on AWS, EC2, and EBS Research, Methodology, and Findings Protecting your forensic data
- 5. – Twilio value “Start with why.”
- 6. $ cd ~/why? 💩 happens (citation needed)
- 7. ~/why? Compromises happen @TODO: fill in with the most recent breach
- 8. ~/why? Prevention is ideal Detection is a must If something bad happens, how can we determine how bad that something was?
- 9. ~/why? Forensic Investigation Investigate an event in the past by utilizing scientific techniques Post-event is not the time to be developing techniques
- 10. ~/why? Personally: What is hidden is much more interesting than what is visible Has an attacker… …deleted log files? …caused log files to roll over? …used a dropper which erased itself?
- 11. $ cd ~/the_cloud Super-high-level overview of the cloud $ ln -s ~/the_cloud /clouds/aws
- 12. ~/the_cloud/ This is not meant to be a introductory course in AWS Also not meant to be an advanced course in AWS Amazon Web Services is comprised of many, many, many, many, many, many, many services
- 14. ls ~/the_cloud/ Focusing today on EC2: Elastic Cloud Compute EBS: Elastic Block Storage IAM: Identity and Access Management S3: Simple Storage Service
- 16. $ cd ~/the_cloud/ec2 Elastic Cloud Compute Virtual machines on-demand Plus now, even a few bare-metal ones Varying capabilities, CPU, RAM, etc.
- 17. ~/the_cloud/ec2 Click button, get server Script with a loop, hit API, get lots of servers Default is shared tenancy, but you can get various degrees of dedicated hardware
- 18. ~/the_cloud/ec2 The starting point for an EC2 instance is an AMI Amazon Machine Image This contains the host OS, configuration, etc. Possible to create an AMI from a running instance— including configuration files, etc.
- 19. ~/the_cloud/ec2 Some instances have Instance Storage Some instances don’t Instance Storage is directly-connected disk Fast Ephemeral
- 20. $ cd ../ebs Elastic Block Storage Request a volume, specify its size, and where you want to attach it Kinda-sorta network attached storage that presents locally Various backing stores
- 21. ~/the_cloud/ebs standard gp2 io1 st1 sc1 Backing material HDD SSD SSD HDD HDD Sizes 1GB–1TB 1GB– 16TB 4GB– 16TB 500GB– 16TB 500GB– 16TB Max Throughput (volume) 90MBps 160MBps 500MBps 500MBps 250MBps Price $0.05/ GB/month $0.10/ GB/month $0.125/ GB/month $0.045/ GB/month $0.025/ GB/month AFR 0.1% – 0.2% 0.1% – 0.2% 0.1% – 0.2% 0.1% – 0.2% 0.1% – 0.2%
- 22. ~/the_cloud/ebs EBS has an AFR of <0.2% Commodity drive AFR is ~4%
- 23. ~/the_cloud/ebs EBS is 20x more durable than regular drives! How? Magic!
- 24. ~/the_cloud/ebs Recap: Disk storage which… …is network attached storage… …of various capabilities… …presenting as either NVMe or SATA… …with dynamic sizes… …but of static allocation… …which fail 1/20th as often as a single drive …and can be snapshotted via API call
- 25. $ cd ../iam Identity and Access Management aka permission controls in AWS Relatively fine-grained; who can perform which API calls on what resources We’ll get into this more later— it ties into how to protect the chain of custody
- 26. $ cd ../s3 EBS volumes can have a snapshot taken Does not have to be when the disk is detached, but be wary of inconsistencies when snapshotting an active volume Snapshots are stored as blobs in S3
- 27. ~/the_cloud/s3 Snapshots are immutable AMIs are special EBS snapshots “Blessed” to be usable as a boot volume Instance root volumes can be EBS or Ephemeral storage
- 28. Story Time! Alice Bob The Third Party The story, all names, characters, and incidents portrayed in this production are fictitious. No identification with actual persons (living or deceased), places, buildings, and products is intended or should be inferred.
- 29. Alice and Bob communicate with each other via the “WhatsMyGramBook” service “WhatsMyGramBook” uses one server, which was breached Were their communications accessed? What did the attacker do?
- 30. ~/the_cloud/questions Question One: If a snapshot of an EBS volume is taken, will that snapshot only contain in-use blocks, or are deleted blocks also included? Question Two: Does it matter what the original EBS volume type is? Has Amazon changed their implementation between versions? Question Three: Does the instance type matter? Does NVMe vs. SATA make a difference?
- 31. ~/research/process 1. Launch a selection of EC2 instances 2. Attach one of each EBS volume type to each class of instance 3. Write files 4. Delete files 5. Snapshot disks 6. Rehydrate snapshot to new disk 7. Look for files
- 37. WMGB sysadmin/founder/developer/person logs into the AWS Console and triggers a snapshot of the EBS volumes attached to their EC2 instance Now to see what happened on that volume
- 38. ./photorec PhotoRec is freely available software to look for deleted files https://www.cgsecurity.org/wiki/PhotoRec Looks at the raw blocks of a disk and compares data to known file signatures
- 39. ./scripts/forensics.rb With each snapshot, rehydrate to an EBS volume Attach the volume to an instance Run PhotoRec and look for deleted files
- 40. $ pry forensics.rb Launching new Investigate instance Waiting for i-04074842f4a3a8c10 to enter running state... Executing setup commands Waiting for command to be in finished, currently is Success Command execution successful! Output: Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 3276800 inodes, 13107200 blocks 655360 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2162163712 400 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done download: s3://forensics-software/testdisk-7.0/AUTHORS to ../../root/testdisk/AUTHORS download: s3://forensics-software/testdisk-7.0/INFO to ../../root/testdisk/INFO download: s3://forensics-software/testdisk-7.0/documentation.html to ../../root/testdisk/documentation.html download: s3://forensics-software/testdisk-7.0/fidentify.8 to ../../root/testdisk/fidentify.8 download: s3://forensics-software/testdisk-7.0/icons/48x48/apps/qphotorec.png to ../../root/testdisk/icons/48x48/ apps/qphotorec.png download: s3://forensics-software/testdisk-7.0/THANKS to ../../root/testdisk/THANKS download: s3://forensics-software/testdisk-7.0/NEWS to ../../root/testdisk/NEWS download: s3://forensics-software/testdisk-7.0/COPYING to ../../root/testdisk/COPYING download: s3://forensics-software/testdisk-7.0/Android.mk to ../../root/testdisk/Android.mk download: s3://forensics-software/testdisk-7.0/ChangeLog to ../../root/testdisk/ChangeLog download: s3://forensics-software/testdisk-7.0/icons/photorec.ico to ../../root/testdisk/icons/photorec.ico
- 41. Results!
- 42. Lots of files are returned! WhatsMyGramBook team is very happy they attended this talk If WhatsMyGramBook had their log files on the root EBS volume, they should expect to see many files recovered from the AMI Can be an overwhelming amount of returned files
- 43. AMIs are snapshots AMIs therefore contain deleted files Using PhotoRec on an EBS volume that was the startup volume can net large amounts of recovered files
- 45. Fair enough— I’ve told the end of the story before showing you the results How do we know it’s possible to recover all these files?
- 46. Pass Match the Hash
- 47. ~/scripts/compare.sh Frequently, more files are returned than originally seeded to disk This is due to PhotoRec guessing where files end and new files begin Especially with text files, which don’t have clearly defined “magic numbers” and EOF markers
- 49. ~/scripts/compare_bytes.sh Comparing the first n bytes is a reliable way to determine how many of the original files are recovered
- 50. $ grep -v -e '^/' analysis Examining 84 files in /recovery/ c5.large/gp2.1/ Recovered: 84 Matches:48/56 Examining 88 files in /recovery/ c5.large/io1.1/ Recovered: 88 Matches:52/56 Examining 37 files in /recovery/ c5.large/sc1.1/ Recovered: 37
- 51. ~/findings Source instance type has no detectable effect Recovery success varied based on source volume type Best recovery rates: Standard, gp2, io1 Less-good rates: sc1, st1
- 52. ~/findings Examining 84 files in /mnt/forensic_recovery/c5.large/gp2.1/ Matches: 48/56 (86%) Examining 88 files in /mnt/forensic_recovery/c5.large/io1.1/ Matches: 52/56 (93%) Examining 37 files in /mnt/forensic_recovery/c5.large/sc1.1/ Matches: 25/56 (47%) Examining 11 files in /mnt/forensic_recovery/c5.large/st1.1/ Matches: 11/56 (20%) Examining 86 files in /mnt/forensic_recovery/c5.large/standard.1/ Matches: 50/56 (89%)
- 53. ~/findings Weird artifacts Recovery of PDFs from sc1/st1 based drives resulted in massive files …but not other drive types
- 54. /dev/nvme1n1 493G 196G 272G 42% /mnt/forensic_recovery272G
- 55. $ find . -printf '%s %pn'|sort -nr|head -n 5 161053454336 ./m3.medium/st1.1/f209993256.pdf 24292724736 ./m5.large/st1.1/f428097024.pdf 21367603200 ./t2.nano/st1.1/f948187936.pdf 17745333 ./t2.nano/standard.1/f0475136.m4p 17745333 ./t2.nano/st1.1/f0393216.m4p
- 56. $ find . -printf '%s %pn'|sort -nr|head -n 5 161 GB ./m3.medium/st1.1/f209993256.pdf 24.30 GB ./m5.large/st1.1/f428097024.pdf 21.36 GB ./t2.nano/st1.1/f948187936.pdf 0.18 GB ./t2.nano/standard.1/f0475136.m4p 0.18 GB ./t2.nano/st1.1/f0393216.m4p
- 57. Not sure why— this could be a recovery issue of detecting the EOF marker further away from the start of the PDF But it only appeared when the source was an sc1/st1 volume
- 59. Chain of Custody Snapshots can be shared to other accounts If an attacker is in your account, they could delete the snapshots as you take them, causing data to be lost Copy to another, secured, account to keep them safe
- 62. $ sort -u ~/talk What does your threat model look like? Do you need high-quality forensics? Yes: Don’t use sc1/st1/ephemeral No: Understand the limitations
- 63. $ sort -u ~/talk Consider writing only to non-root EBS volumes Eliminates the large number of recoverable files deleted from the AMI Quite possibly too much noise to see the signal of deleted files
- 64. $ sort -u ~/talk Use multiple accounts A breach of a server could mean the breach of all your stuff The loss of an API key could mean the loss of all your stuff An account limits the blast radius Keep your forensics out of that blast radius
- 65. Thank you!
- 66. Few quick things… We’re hiring! If you are interested, or know someone who’s interested: https://boards.greenhouse.io/twilio We have a bug bounty program! https://bugcrowd.com/twilio SMS ‘shakacon’ to 213.27.SHAKA for my business card! 213.27.74252
- 67. head -n 5 /dev/random/ questions SMS ‘shakacon’ to 213.27.SHAKA for my business card! 213.277.4252