It's 10pm, Do You Know Where Your Access Keys Are?

It's 10pm, Do You Know
Where Your Access Keys Are?
Ken Johnson

Things to Mention
• Ask questions throughout presentation
• There will be no dedicated Q&A – so stick
around after and find me if you want to
chat
• This presentation will move fast. Slides will
be available so don’t worry about minutia.

Background/About
• Ken Johnson, CTO and Partner at nVisium
• Veteran, US Navy
• I speak about:
– DevOps (In)Security
– Exploiting Web Applications
– Coding and Coding + Security
– Node, Elixir, Python, Ruby, Go
– AWS (clearly)

Background/About
• I’m the CTO of a security company
• Naturally, I have some concerns as it
pertains to AWS
• My Concerns
– Risk Assessments (Compliance)
– Data security
– Reputation

Background/About
1. AWS used to be just a “thing” we had
2. Then it became a little more important
3. Then it became business-critical
4. Then I got worried…

Problem Statement
How can we prevent attacks?
How can we know if an attack is happening?
How can we recover if the worst case
scenario somehow happens?

My Plan
• Harden – Make it difficult to reach our
AWS environment
• Monitor – If our AWS environment is
breached, we need to know and alert
ourselves
• Restore – Have the ability to reconstruct
data/configs after a “hack”

AWS’s Plan
• Took the AWS Security Fundamentals Course
and…
– Fortunately, our strategy lines up with AWS
recommendations
– You are responsible for leveraging the tools AWS
provides (financially)
– Your configuration… that is on you
– https://aws.amazon.com/training/course-
descriptions/security-fundamentals/

AWS Hardening Basics
Making it difficult (for attackers) to
reach our environment

Hardening Checklist
1. Don’t Use The Root Account!
2. Disable Access Keys for Root Account
3. Multi-Factor Authentication
4. API + MFA
5. Strong Password Policy

Don’t Use Root Account
• Every AWS env has a root account, only
necessary to use for very specific
circumstances
• When these circumstances arise, notify
your team that the account will be used
• We will discuss why this is important when
we talk about CloudWatch metrics

Disable/Delete Root Account Access Keys
• Just delete them if they exist
– Disable the access keys in the event you are
unable to delete them completely for some
reason
• Make sure your admins have a
(verbal/written) policy that states “we don’t
create access keys for the root account”

MFA
• If credentials are stolen or guessed, we want a
second layer of protection
• You can use apps or hardware to do this
– Google Authenticator (Apps)
– Gemalto (Hardware)
• Find the full list of MFA devices here:
https://aws.amazon.com/iam/details/mfa/
• This is so ridiculously easy to do, everyone
should

MFA
Let’s demonstrate enabling MFA using a virtual
device (app) on an IAM account

MFA
Navigate to Identity & Access Management

MFA
Next, manage the MFA device

MFA
Lastly, use Google Authenticator to take a snapshot of the
QR code

MFA
• At this point, its worth mentioning that non-
administrators or those without IAM privileges
cannot enable MFA on their own account
• Why is this a problem? Well, they need to be
able to enable MFA on their own device… not
the administrator’s
• Fortunately, we have a solution!

MFA
• Okay so that wasn’t the easiest to read, so
here is the link:
http://docs.aws.amazon.com/IAM/latest/Us
erGuide/id_credentials_delegate-
permissions_examples.html#creds-
policies-mfa-console
• Basically this IAM policy allows a user to
manage their *OWN* MFA device

MFA (for Root Account)
• Need a shared MFA for root? TOTP!
• Recommend using something like
1password for teams, can share the TOTP
code:
https://support.1password.com/guides/mac/totp.html
https://www.youtube.com/watch?v=eZyb-ArMK9g

API + MFA
• You have the ability to place a restriction
where resources can only be interacted
with if the user has authenticated with
MFA
• This helps prevent (ab)use should
someone steal access keys or credentials

API + MFA
1. At a minimum, apply to administrator & power user
group policies… really any group that can do anything
of importance

API + MFA
This entry requires MFA for Web/API

API + MFA
• Truth be told, doing this can be painful at
first
• Things that used to work, might not (via
the API)
• Fortunately, we have some answers for
you
• Firstly, let’s discuss STS or SecurityToken
Service

API + MFA
• Leverage STS in order to interact with the
AWS API should this MFA restriction be
placed on resources (and it should  )
• Example of using STS:
https://gist.github.com/cktricky/127be4e431563a986f0f

API + MFA
Use this script to retrieve creds (from gist)

API + MFA
Use the creds to leverage tools like ec2-api-
tools
(-O <access key id>–W <secret> and –T <session token>)

API + MFA
And in case you don’t like Ruby…
https://github.com/jimbrowne/aws-sts-
helpers

API + MFA
• ElasticBeanstalk does not work with STS. Le
Terrible.
• However, there is a workaround, use
CodePipeline.
• Very simple process to setup but only works
with:
– GitHub
– AWS CodeCommit
– Amazon S3

API + MFA
• One final note of warning here, you may
see oddities/restrictions when you go to
use resources in the AWS web interface
AFTER having been logged in for a bit…
just reauthenticate

Password Policy
• Password policies are important because
historically people do not choose complex
passwords
• MFA should help, but we’re talking about a
layered approach
• Again, making our AWS environment
harder to reach

Hardening Recap
• Make credentials hard to guess
• If guessed or stolen, we still have MFA
• Remember MFA only protects against the
web and NOT the API… unless you
change your policies and use STS
• Root account is King, protect your King

Hardening Recap
• Things we did not (and won’t discuss)
– S3 bucket policies
– Security Group configurations
– SSH Key Management
– Encrypting Data (Volumes, S3 buckets)
• Trusted Advisor – Use it, because it
catches a lot of “low hanging fruit” style
issues

Hardening Recap
• Links to resources that discuss the items we’re not
covering:
– https://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing
_Security_Checklist.pdf
– http://aws-de-
media.s3.amazonaws.com/images/Produktblaetter/AWS-
Security-Check-List_eng.pdf
– http://www.slideshare.net/AmazonWebServices/masterclass-
advanced-security-best-practices
• Frankly you can’t throw a rock without hitting some
basic info regarding AWS Security Checklists

AWS Monitoring
Detecting malicious activity

AWS Monitoring
• Assuming hardening (prevention) has failed, how
would we know?
• Luckily, AWS provides several services which alert
to anomalies
• We will walk through examples of using these
services, but ultimately decide what is right for you
• Fair warning, some of these services will provide a
lot of noise

AWS Monitoring
4 important services:
1. CloudTrail
2. SNS
3. Config
4. CloudWatch

AWS Monitoring
• CloudTrail – Logs
• SNS – Notifications
• Config – Alerts for modifications &
noncompliance
• CloudWatch – Alerts for specific types of
behavior

AWS Monitoring
CloudTrail
Config
CloudWatch
SNS

AWS Monitoring (CloudTrail)
• CloudTrail is primarily used for log
collection
• Other services like CloudWatch, for
example, use those logs to filter relevant
data

Pretty easy, first turn it on..

Configure the log group

Allow the creation of an IAM role by CloudTrail

• At this point you are okay
• Start configuring CloudWatch/Config

AWS Monitoring (SNS)
• Fantastic offering, <3 it
– Examples of ways to be notified by SNS
• SMS
• Email
• JSON Post to your Application’s API endpoint

• Receive SMS/Email/Slack notifications for
important events
• ^ This is so you get immediate notifications
• You can have multiple subscribers, I’d
suggest you use that functionality
• Basic gist? Receive immediate updates for
things you want to see… immediately 

Create a topic

Create Subscription

Create SMS (or whatever, but in this case, SMS)

Example of creating email subscription… bottomline you
can have multiple ways of notifying people

AWS Monitoring (Config)
• Config:
– Alerts owners to changes or noncompliance
with regards to AWS resources
– Can either design custom Config rules or use
managed (pre-packaged) AWS Config rules

• Pre-packaged “Managed” AWS Rules
– CLOUD_TRAIL_ENABLED
– EIP_ATTACHED
– ENCRYPTED_VOLUMES
– INCOMING_SSH_DISABLED
– INSTANCES_IN_VPC
– REQUIRED_TAGS
– RESTRICTED_INCOMING_TRAFFIC
https://docs.aws.amazon.com/config/latest/developerguide/evaluate-
config_use-managed-rules.html

• Examples of things you can have alerts
set for:
– Change in Firewall (Security Group) ports
– Changes in VPC
– Any change… at all

Go to the Config service and choose resources to track

Or choose to track everything

Create a bucket, create an SNS topic (…we’ll discuss next)

Allow the role to be created and you’re all set!

AWS Monitoring (CloudWatch)
• We can be very particular here about what it is we
want to see
• Some very interesting things you can monitor
• Some examples:
– Billing Alerts (Important for detection of abuse
or mistakes)
– Track Root Account Usage
– Failed login attempts

AWS Monitoring (CloudWatch - Billing)
• Used to prevent abuse or mistakes from costing your
organization money
• Analyze and approximate your monthly spend
• Configure via CloudWatch
• Use SNS for instantaneous alerting

Navigate to billing & cost management; enable
billing alerts

Create an SNS topic

Subscribe to Topic

Navigate to CloudWatch -> Metrics -> Billing

Choose USD/EstimateCharges -> Create Alarm

Set price point, SNS topic, and create alarm

Exact steps to enable can be found here:
http://docs.aws.amazon.com/awsaccountbilli
ng/latest/aboutv2/free-tier-alarms.html

AWS Monitoring (CloudWatch – Root Login)
• Remember how I said don’t use the Root
account routinely?
• BUT… if this account is used, you should
know about it
• This is the reason you’ll want to notify
others (who receive SNS alerts) of the fact
you are about to use the account

Choose log group, create metric

Define Logs Metric Filter

Assign/Create Filter

Click “Create Alarm”

Define Alarm and you’re good…

Exact steps (with pics) exist here:
https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8
374D/How-to-Receive-Notifications-When-Your-AWS-
Account-s-Root-Access-Keys-Are-Used

AWS Monitoring (CloudWatch – Failed Logins)
• In the event someone is trying to break in,
let’s alert ourselves to this!
• Failed logins typically suggest either
someone forgot their password or…
someone is trying to guess yours

Navigate to Logs, click “Create Metric Filter”

Enter the relevant filter pattern, click create

Fill out filter/metric/metric-namespace info

Click “Create Alarm”

Fill in relevant details and click “Create
Alarm”

• Exact steps exist here:
http://docs.aws.amazon.com/awscloudtrail/la
test/userguide/cloudwatch-alarms-for-
cloudtrail.html#cloudwatch-alarms-for-
cloudtrail-signin

AWS Monitoring (CloudWatch) – Filter Patterns
• Create your own custom filter patterns,
here is a resource for that:
http://docs.aws.amazon.com/AmazonCloud
Watch/latest/DeveloperGuide/FilterAndPatte
rnSyntax.html

AWS + Splunk
• Splunk is a pretty great resource for monitoring
activity
• I’m fairly new to Splunk myself
• Two separate plugins:
Splunk App for AWS
https://splunkbase.splunk.com/app/1274/
Splunk Add-On
https://splunkbase.splunk.com/app/1876/

AWS + Splunk
• Examples of things you can view:
– Billing
– Topology
– Usage
– IAM Activity
– SSH Key Pair Activity
– User Activity
– Network ACL(s)
– VPC Activity
and a lot more…

AWS + Splunk
• Pretty Screenshot 1

AWS + Splunk

AWS + Splunk
• Add plugins (apps) to Splunk

AWS + Splunk
• Splunk will need an AWS account in order
to retrieve data
• Create account(s) for Splunk, grab the
necessary permission policy from here:
http://docs.splunk.com/Documentation/AddOns/r
eleased/AWS/ConfigureAWSpermissions

AWS + Splunk
• Add the newly created account(s) to Splunk
Add-on for AWS app - requires AWS access
token id/secret

AWS + Splunk
• Configure AWS App for Splunk, add account(s),
configure each input accordingly:

AWS + Splunk
• To view things like IAM Activity…
– Subscribe to a cloudtrail log via SNS
– Utilize SQS and subscribe SQS to an SNS
Topic

AWS Monitoring Recap
• Alert yourself when things change
• This will get noisy, find a way to filter that which is
important
– If it’s a high risk event, send an SMS/Slack/Email
blast
• At a minimum, alert yourself when odd things occur…
like:
– Billing increases past your normal spend
– When somebody authenticates as Root
– When someone has a login failure

AWS Monitoring Recap
• Interesting Quora thread:
– https://www.quora.com/My-AWS-account-was-hacked-and-I-
have-a-50-000-bill-how-can-I-reduce-the-amount-I-need-to-pay
• Highlights from the article:
– AWS has “a review board of sorts” to determine if you should be
refunded
– Bots are scouring GitHub searching for exposed access keys
– One of the more AWS-seasoned responders mentioned doing
part of what we discussed here today to avoid it
– A decent number of the people posting on this thread said “Yes,
happened to me too”

AWS Restoration & Recovery
Plan to fail, just don’t fail to plan

AWS Restoration & Recovery – Basic Incident
Response (IR)
• Understand who to contact if things go bad
• Understand how to communicate (ex:
“speak only over the phone”)
• Understand what information to parse
• Understand where your backups are
located and how they are secured

AWS Restoration & Recovery – Basic IR
• Do not USE AWS TO BACKUP YOUR
AWS
• Offsite backups (meaning, off AWS site)
• Common things to back-up:
– Databases/ Snapshots
– S3 Buckets
– EBS Volumes
– CloudFormation Templates

AWS Restoration & Recovery – Basic IR
• Resources:
– http://stackoverflow.com/questions/17087542/
backup-solutions-for-aws-ec2-instances
– https://github.com/Scalr/installer-ng
– http://www.n2ws.com/blog/3-ways-ec2-
windows-backup-and-recovery.html

Recap
• Makes your environment harder to reach… for
the bad guys
– Limit what stolen or “otherwise obtained”
access keys or credentials could be used to
do
– Prevent them being stolen in the first place
• Alert yourself to anomalies
• Have a plan for if things go bad
• Stay safe out there!

Contact Info
• My Info
• Twitter: @cktricky
• Email: ken@nvisium.com

It's 10pm, Do You Know Where Your Access Keys Are?

More Related Content

Similar to It's 10pm, Do You Know Where Your Access Keys Are? (20)

Recently uploaded (20)

It's 10pm, Do You Know Where Your Access Keys Are?

Editor's Notes