Aws meetup s3_plus
Download as PPTX, PDF0 likes95 views
The document provides an overview of Amazon S3, detailing its features such as object lock, events, batch operations, and the requester pays model. It discusses object retention management, notifications, batch operations configuration, and how access control permissions function. Additionally, it addresses usage scenarios, legal compliance options, and AWS services integration alongside practical instructions for setting up S3 functionalities.
![aws s3control create-job
--region us-west-2
--account-id acct-id
--operation '{"S3PutObjectTagging": { "TagSet":
[{"Key":"keyOne", "Value":"ValueOne"}] }}'
--manifest
'{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Buck
et","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::my_manifests/mani
fest.csv","ETag":"60e460c9d1046e73f7dde5043ac3ae85"}}'
--report '{"Bucket":"arn:aws:s3:::bucket-where-completion-
report-goes","Prefix":"final-reports",
"Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTas
ks"}'
--priority 42
--role-arn IAM-role
--client-request-token $(uuidgen)
--description "job Description"
--no-confirmation-required
EXAMPLE JOB](https://image.slidesharecdn.com/awsmeetups3plus-200123201302/85/Aws-meetup-s3_plus-26-320.jpg)
![POLICY EVALUATION
Identity-based
policy
Permissions
boundary
{
“Version”: “2012-10-17”,
“Statement:[
{
“Effect”: “Allow”,
“Action”: [
“logs:CreateLogGroup”,
“logs:CreateLogStream”,
“logs:PutLogEvents”
],
”Resource”: “arn:aws:logs:*:*:*”
},
{
“Effect”: “Allow”,
“Action”: [ s3:GetObject ],
“Resource”: “arn:aws:s3:::example/*”
}
]
}
{
“Version”: “2012-10-17”,
“Statement:[
{
“Effect”: “Allow”,
“Action”: [
“s3:GetObject”,
”s3:PutObject”,
],
“Resource”: “*”
}
]
}](https://image.slidesharecdn.com/awsmeetups3plus-200123201302/85/Aws-meetup-s3_plus-35-320.jpg)
![EXAMPLE POLICY
OAI Policy
{
“Version”: “2018-10-17”,
“Id”: “PrivatePolicyForCloudFront”,
“Statement:[
{
”Sid”: “1”,
“Effect”: “Allow”,
“Principal” : {
“arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity XXXXXXXXX
}
“Action”: “s3:GetObject”,
“Resource”: “arn:aws:s3:::testbucket/*”
}
]
}](https://image.slidesharecdn.com/awsmeetups3plus-200123201302/85/Aws-meetup-s3_plus-39-320.jpg)
Aws meetup s3_plus
- 1. AWS Atlanta Presents S3 PLUS
- 2. Sponsors
- 4. Agenda S3 Overview Object Lock / Legal Hold Events with S3 S3 Batch Operations Requester Pays Permission Boundaries Restricting S3 to CloudFront
- 5. A web-store not a file system Fast Access via APIs Economical Highly Available and Durable Highly Scalable Data Storage
- 6. Amazon S3 Namespaces Amazon S3 bucket bucket bucket object objectobject object objectobject Globally Unique Name Bucket Name + Object Name (key)
- 7. S3 Overview Summary Cost effective for a wide variety of use cases from cloud applications, content distribution, backup, archiving, disaster recovery & analytics S3 can be used alone or with other AWS Services or 3rd party tools & services S3 provides developers with secure, durable, highly scalable object storage
- 8. Object Lock / Legal Hold
- 9. Two ways to manage object retention Retention Period A Retention Period specifies a fixed period of time during which an object remains locked Legal Hold A Legal Hold provides the same protections as a retention period, but has no expiration date
- 10. You can view the object lock status of an Amazon S3 object version using the GET OBJECT or HEAD OBJECT commands. Both commands return the retention mode, RETAIN UNTIL DATE, and the legal hold status of the specified object. Viewing the Lock Info for an Object
- 11. FAQs Legal Hold is really for any situation where you are not sure how long you want your objects to stay immutable. This may be because you have active litigation or an upcoming external audit of your data or any other reason you want to keep your objects in a WORM state until the audit is complete. When should you use legal hold?
- 12. S3 object lock only works for buckets which have versioning enabled SETTING UP A BUCKET FOR OBJECT LOCK NOTE: You can only turn on object lock for new buckets
- 14. Navigate to S3 Object lock in the Advanced settings in the Properties Tab
- 15. S3 Events
- 16. S3 Events overview SNS Topic Lambda FunctionSQS Queue Incoming Objects Message InvocationMessage
- 17. Supported Destinations for S3 Events You must grant S3 permissions SNS Lambda SQS
- 19. Filtering by prefix or suffix of the file name (ie) .JPG Configuring Notifications with Filtering prefix HEAD OBJECT commands.
- 20. 1) Create the queue, or topic, or Lambda function if necessary START USING EVENT NOTIFICATIONS WITH S3 2) Grant S3 permission to publish to the target or invoke the Lambda function. For SNS or SQS, you do this by applying an appropriate policy to the topic or queue. For Lambda you must create and supply and IAM role, then associate it with the Lambda function. 3) Arrange for your application to be invoked in response to activity on the target, you have several options here 4) Set the bucket’s Notification Configuration to point to the target.
- 21. Each notification notification is delivered as a JSON object with the following fields: NOTIFICATION DETAILS • Region • Timestamp • Event Type • Request Actor Principal ID • Source IP of the request • Request ID • Host ID • Notification Configuration Destination ID • Bucket Name • Bucket ARN • Bucket Owner Principal ID • Object Key • Object Size • Object Etag • Object Version ID ( if versioning is enabled on the bucket)
- 22. • Delivery Latency - Notifications are delivered to the target in well under a second NOTIFICATION DETAILS • Cost - There is no charge for this feature. You will pay the usual messaging and execution charges for SQS, SNS, and Lambda • Regions- The bucket and the target must reside in the same AWS region • Event Types - You can configure one notification per event type per bucket • Delivery Reliability - S3 is designed to deliver notifications with a very high degree of reliability. It includes built-in backoff and retry mechanisms to deal with momentary issues that might affect the deliverability of messages to any of the three types of targets.
- 24. Types of Batch operations Restore Archive objects from Glacier Copy Objects between S3 buckets Replace object tag sets Modify access controls to sensitive data Modify object metadata & properties Invoke AWS Lambda Functions
- 25. The following is an example manifest in CSV format including Version IDs SPECIFYING A MANIFEST Mycoolbucket, objectkey1, PZ9ibn9D5lP6p298B7S9_ceqx1n5EJ0p Mycoolbucket, objectkey2, YY_ouuAJByNW1LRBfFMfxMge7XQWxMBF Mycoolbucket, objectkey3, jbo9_jhdPEyB4RrmOxWS0kU0EoNrU_oI Mycoolbucket, photos/catpics/jpegs/object4, 6EqlikJJxLTsHsnbZbSRffn24_eh5Ny4 Mycoolbucket, photos/jpgs/object5, imHf3FAiRsvBW_EHB8GOu.NHunHO1gVs Mycoolbucket, object%20key%20with%20space, 9HkPvDaZY5MVbMhn6TMn1YTb5ArQAo3w
- 26. aws s3control create-job --region us-west-2 --account-id acct-id --operation '{"S3PutObjectTagging": { "TagSet": [{"Key":"keyOne", "Value":"ValueOne"}] }}' --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Buck et","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::my_manifests/mani fest.csv","ETag":"60e460c9d1046e73f7dde5043ac3ae85"}}' --report '{"Bucket":"arn:aws:s3:::bucket-where-completion- report-goes","Prefix":"final-reports", "Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTas ks"}' --priority 42 --role-arn IAM-role --client-request-token $(uuidgen) --description "job Description" --no-confirmation-required EXAMPLE JOB
- 27. Understanding Job Status Status Description New A Job begins in a new state when you create it Preparing Amazon S3 is processing the manifest object and other job parameters Suspended The job requires confirmation, but you have not yet confirmed that you want to run it. Only jobs created using the console require confirmation. Ready Amazon S3 is ready to begin running the requested object operations Active Amazon S3 is executing the requested operations listed in the manifest. Pausing The job is transitioning to the Paused state Paused A job can become paused if you submit another job with a higher priority Cancelling The job is transitioning to the Cancelled state Cancelled The request was cancelled and was successful Failing The job is transitioning to the Failed state Failed The job has failed and is no longer running
- 28. Requester Pays
- 29. Requester Pays model can be used 2 ways By simply marking a bucket as Requester Pays, data owners can provide access to large data sets without incurring charges for data transfer or requests. 2 1 Requester Pays feature can be used in conjunction with Amazon DevPay. Content owners charge a markup for access to the data. The price can include a monthly fee, a markup on the data transfer costs, and a markup on the cost of each GET.
- 30. Amazon DevPay Amazon DevPay is not accepting new seller accounts at this time. Amazon DevPay is a simple-to-use online billing and account management service that makes it easy for businesses to sell applications that are built in, or run on top of, Amazon Web Services. It is designed to make running applications in the cloud and on demand easier for developers
- 31. REQUESTER PAYS EXAMPLE Available Data Set Hosting Account Requester Account S3 Bucket with Requester Pays permissions
- 32. FAQs AWS documentation talks about adding a special header - x-amz-request-payer, to make sure that you as the requester you understand that you are paying the network charges. Others say to make sure that you the user has the permissions AmazonS3FullAccess in order to be able to download from a requestor pays bucket How can I download from a S3 bucket marked as requester pays?
- 35. POLICY EVALUATION Identity-based policy Permissions boundary { “Version”: “2012-10-17”, “Statement:[ { “Effect”: “Allow”, “Action”: [ “logs:CreateLogGroup”, “logs:CreateLogStream”, “logs:PutLogEvents” ], ”Resource”: “arn:aws:logs:*:*:*” }, { “Effect”: “Allow”, “Action”: [ s3:GetObject ], “Resource”: “arn:aws:s3:::example/*” } ] } { “Version”: “2012-10-17”, “Statement:[ { “Effect”: “Allow”, “Action”: [ “s3:GetObject”, ”s3:PutObject”, ], “Resource”: “*” } ] }
- 36. Application owners giving roles for Amazon EC2 instances AWS Wide Use Cases Admins creating users for particular situations Developers creating roles for AWS Lambda Functions Restricting users and roles to a specific region
- 37. Restrict S3 to CloudFront
- 38. POLICY EVALUATION S3 Web Browser (http/s) Customers CloudFront Distribution - Assumed OAI Edge Locations Open Internet Allow OAI Deny Others
- 39. EXAMPLE POLICY OAI Policy { “Version”: “2018-10-17”, “Id”: “PrivatePolicyForCloudFront”, “Statement:[ { ”Sid”: “1”, “Effect”: “Allow”, “Principal” : { “arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity XXXXXXXXX } “Action”: “s3:GetObject”, “Resource”: “arn:aws:s3:::testbucket/*” } ] }
- 40. FAQs An OAI is a virtual Identity, a Cloud Front distribution can be configured to used it, so when accessing S3 the identity assumes the identiy. What is an OAI?
- 41. FAQs Yes, there are a number of tools available to look for for misconfigurations or open S3 buckets. Most are available for free on Github. S3 inspector S3Scanner BucketFinder Are a few that will uncover buckets and misconfigurations. Are there any tools that can help me find misconfigurations in my S3 buckets?
- 43. Hello Joshua
Editor's Notes
- #6: AWS Control Tower is the easiest way to set up and govern multiple accounts at scale
- #9: Amazon S3 Object Lock is an Amazon S3 feature that allows you to store objects using a write once, read many (WORM) model.
- #10: During this period, your object is WORM-protected and can’t be overwritten or deleted. You apply a retention period either in number of days or number of years with the minimum being 1-day and no maximum limit. Instead, a legal hold remains in place until you explicitly remove it.
- #12: Legal Hold works as an infinite retention period. Once applied it is not possible to delete any object until the hold is released manually. The hold can only be removed by users with special permissions.
- #13: If you want to turn on object lock for existing buckets then you have to contact support
- #14: With the advanced tab open you can turn on Object lock
- #15: Now we have a bucket with S3 Object lock turned on. What we’ve done so far doesn’t automatically lock objects that you put into the bucket. For that, we will set up a default retention mode and period on the bucket.
- #17: You can arrange for notifications to be issued to Simple Queue Service, or Simple Notification Service,when a new object is added to the bucket is added to the bucket or an object is overridden. Notifications can also be delivered to AWS lambda for processing by a Lambda function.
- #18: AWS Control Tower is the easiest way to set up and govern multiple accounts at scale
- #19: Object Created Events include: Put, Post, Copy and CompleteMultipartUpload Object Removed Events include: Delete and DeleteMarker Created
- #20: You can configure notifications to be filtered by the prefix and suffix of the key name of object.
- #21: From that point forward the events will be reliably delivered to the target as appropriate.
- #23: Here are a few things to keep in mind as you start to think about the best way to use these new notifications as part of your application
- #24: Requester Pays, works at the level of an S3 bucket. If the bucket’s owner flags it as Requester Pays, then all data transfer and request costs are paid by the party accessing the data.
- #26: If you overwrite an object with a new version while a job is running, and you didn't specify a version ID for that object, Amazon S3 performs the operation on the latest version of the object, and not the version that existed when you created the job.
- #27: AWS S3 Control command provides access to Amazon S3 control plane operations.
- #29: Requester Pays, works at the level of an S3 bucket. If the bucket’s owner flags it as Requester Pays, then all data transfer and request costs are paid by the party accessing the data.
- #30: AWS Control Tower is the easiest way to set up and govern multiple accounts at scale
- #31: Amazon DevPay removes the pain of having to create or manage your own order pipeline or billing system, which is traditionally a challenge for online subscription services or applications running on demand. It allows you to quickly sign up customers, automatically meter their usage of AWS services, have Amazon bill them based on the prices you set, and collect payments
- #32: By simply marking a bucket as Requester Pays, data owners can provide access to large data sets without incurring charges for data transfer or requests... Requesters use signed and specially flagged requests to identify themselves to AWS, paying for S3 GET requests and data transfer at the usual rates
- #33: There were stack overflow tips using the old s3cmd tools however the person with success had used this iam permission using the unified AWS cli tools
- #35: As your Organization grows, you might want to allow trusted employees to configure and manage IAM permissions to help your organization scale permission management and move workloads to AWS faster.
- #36: A permisssions boundry is an advanced feature that allows you to limit the maximum permissions that a principal can have. We can see here that even though the IAM policy was given Get and Put on all S3 objects, that the permissions boundry makes sure that they can only get things from the example bucket
- #37: First we’ll take a look at where you could use permission boundries account wide
- #39: By Default, when using Cloudfront with S3, CloudFront is optional, and S3 can be accessed directly. This can be changed by creating and Origin Access Identity (OAI) by using the OAI then access is removed from the S3 bucket by the users and then they must get the assets from CloudFront When the customers attempt to access the S3 bucket directly then they are denied because they have no permissions
- #41: There were stack overflow tips using the old s3cmd tools however the person with success had used this iam permission using the unified AWS cli tools
- #42: Misconfigured buckets leak sensitive data Keep in mind that threat actors use these tools as well and it’s better to use these yourself before they do and find sensitive data. There was a stat that 7% of all S3 buckets have unrestricted public access and 35% of all buckets are unencrypted.