AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트)
6 likes6,635 views
The document provides an overview of AWS Batch, a service designed to simplify batch job processing on AWS. It includes details on how to create job queues, compute environments, and submit jobs using JSON configuration files, while highlighting the benefits of resource optimization and cost management. Additionally, it contains links to case studies and resources for deploying batch workflows, particularly in genomics and other high-throughput environments.
![Job Depends on Array Job
“Job-B”
B:0
…
B:1
B:99
“Job-A”
$ aws batch submit-job –cli-input-json file://./Job-A.json
<Job-A.json>
{
"jobName": ”Job-A",
"jobQueue": "ProdQueue",
"jobDefinition": ”job-s3-list-validate:1",
}
$ aws batch submit-job –cli-input-json file://./Job-B.json
<Job-B.json>
{
"jobName": ”Job-B",
"jobQueue": "ProdQueue",
"jobDefinition": ”job-s3-CPU-intensive:1",
”containerOverride": { ”vcpus”: 32, “memory”: 4096 }
”arrayProperties": { “size”: 100 }
"dependsOn": [
{"jobId": "<job ID for Job A>" }
]
}
)3 ( ,
J
A
S B](https://image.slidesharecdn.com/awsbatchdeepdive-180829050328/85/AWS-Batch-AWS-27-320.jpg)
AWS Batch를 통한 손쉬운 일괄 처리 작업 관리하기 - 윤석찬 (AWS 테크에반젤리스트)
- 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2018
- 2. n and ff Predictable peaksWASTE O L S E L ( , ) , , T G
- 3. 1. https://aws.amazon.com/ko/solutions/case-studies/novartis/ 2. https://aws.amazon.com/blogs/aws/experiment-that-discovered-the-higgs-boson-uses-aws-to-probe-nature/ 3. https://www.slideshare.net/AmazonWebServices/bdt311-megarun-behind-the-156000-core-hpc-run-on-aws-and-experience-of- ondemand-clusters-for-manufacturing-production-workloads-aws-reinvent-2014 1 ,, N8D 6 7E ED R C2 1 6 H E D R 7E R C2 1 0 5 3 9 D D 9 3 7 R C2
- 4. • S3/EC2/Spot Fleet • Auto-Scaling • SNS/SQS/CloudWatch • S3/ECS • SQS/CloudWatch
- 6. t 12 a eW v Dm A So E m 1 6357 , r C B A - 53 76 SnS 187: SnSI p eW C B 21 06 6: B B
- 7. g P m , ( o h ) u d MW C A m s , / c il s il ( C l l a m S l t E e e C U v n L I E 2/ z ) L g P E
- 8. 21 ) )Batch Queue (2) Batch Queue (1) Batch Queue (0) priority ( Container Property Compute Resources DependsOn Container Property Container Property ( ) ) ( ) ( ) ( ) …( Define the Batch Job Create Compute Environment(s) Create Job Queue Submit Job Job Scheduling Monitor Job and Outputs
- 9. IAM Role for Batch Job Input Files Queue of Runnable Jobs S3 Events Trigger Lambda Function Submits Batch Job Compute Environments (ECS) Job Definition Batch Execution Application Image (ECR) Batch Scheduler Batch Job Input Batch Job Output Output Files
- 11. ( • B 2 CEA A 2 B aws batch submit-job --cli-input-json file://submit_job.json ( ) SUBMITTED PENDING RUNNABLE STARTING RUNNING SUCCEEDED FAILED
- 12. ()( ( ) • - C A , A aws batch register-job-definition --cli-input-json file://jdef.json
- 13. ) ( ( • , aws batch create-job-queue --cli-input-json file://job_queue.json
- 14. ) ( )( ( • . " " , O aws batch create-compute-environment --cli-input-json file://job_env.json
- 15. , !
- 18. 2
- 19. 2
- 21. $ aws batch create-compute-environment --compute-environment-name c4Spot50 --type MANAGED --state ENABLED --compute-resources type=SPOT,instanceTypes=c4,minvCpus=0,maxvCpus=10000,desiredvCpus=2500,bidpercentage=50,s ubnets=subnet-220c0e0a,subnet-1a95556d,subnet-978f6dce,instanceRole=ecsInstanceRole, securityGroupIds=secGrp01,secGrp02,secGrp03,spotIamFleetRole=arn:aws:iam::012345678910:ro le/aws-ec2-spot-fleet-role,serviceRole=arn:aws:iam::012345678910:role/AWSBatchServiceRole $ aws batch create-compute-environment --compute-environment-name C4-RI --type MANAGED -- state ENABLED --compute-resources type=EC2,instanceTypes=c4.2xlarge,minvCpus=0,maxvCpus=6000,desiredvCpus=1000,subnets=subn et-220c0e0a,instanceRole=ecsInstanceRole, securityGroupIds=secGrp01,spotIamFleetRole=arn:aws:iam::012345678910:role/aws-ec2-spot- fleet-role, serviceRole=arn:aws:iam::012345678910:role/AWSBatchServiceRole .2 12 1 12
- 22. 2 .
- 23. F A { ... "SubmitJob": { "Type": "Task", "Resource": ”arn…", "Next": "GetJobStatus" }, ... }
- 26. 3 . • M • • 1 • M • C • Job A Job C Job B:0 Job B:1 Job B:n … $ aws batch submit-job --job-name BigBatch --job- queue ProdQueue --job-definition monte-carlo:8 -- array-properties “size=10000” ... { "jobName": ”BigBatch", "jobId": "350f4655- 5d61-40f0-aa0b-03ad787db329” } Job Name: BigBatch Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329 Job Name: BigBatch Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:0 Job Name: BigBatch Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:1 Job Name: BigBatch Job ID: 350f4655-5d61-40f0-aa0b-03ad787db329:9999 …
- 27. Job Depends on Array Job “Job-B” B:0 … B:1 B:99 “Job-A” $ aws batch submit-job –cli-input-json file://./Job-A.json <Job-A.json> { "jobName": ”Job-A", "jobQueue": "ProdQueue", "jobDefinition": ”job-s3-list-validate:1", } $ aws batch submit-job –cli-input-json file://./Job-B.json <Job-B.json> { "jobName": ”Job-B", "jobQueue": "ProdQueue", "jobDefinition": ”job-s3-CPU-intensive:1", ”containerOverride": { ”vcpus”: 32, “memory”: 4096 } ”arrayProperties": { “size”: 100 } "dependsOn": [ {"jobId": "<job ID for Job A>" } ] } )3 ( , J A S B
- 28. Two Equally-Sized Array Jobs, a.k.a. “N-to-N” “Job-A” A:0 … A:1 A:2 A:3 A:9999 B:0 B:1 B:2 B:3 B:n “Job-B” “Job-A” “Job-C” C:0 … C:1 C:2 C:3 C:9999 D:0 D:1 D:2 D:3 D:9999 “Job-D” “Job-B” C is dependent on A and B D has an N_TO_N dependency on C
- 30. • E n • 1 - E u n • / E , 2 ,-2 n 13 -1 St • Mm a n • G E • P , 2 u • P h l UI W oi W , U C • zu d • r p 2 : A - 5 n • c r p 5B : A 5 t F
- 33. 9 2 E9CG9 89 , E B 9E 0- 9 8 4- Deletion E 6 96 .15E .15 G E 99 8 99 68 .15E .15 G E 99 8 99 9 9 9 8 9 8 9 8 9 9 9 9 .15E .15 G E .15E 0- -3 .15 G E 9 AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54
- 34. API Server (ECS) SWF Workflow Job Manager (ECS) Generate Variants Solve Variants start Workflow poll for Decision submit Batch Job poll for Activity submit Batch Job poll for Activity task Completed task Completed § : § 2 ) A: : § ( 2 ) A: : B ) AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54
- 35. Deploy an 8K HEVC pipeline using Amazon EC2 P3 instances with AWS Batch https://aws.amazon.com/ko/blogs/compute/deploy-an-8k-hevc-pipeline-using-amazon-ec2-p3- instances-with-aws-batch/
- 36. 1.2 MILLION JOBS SUBMITTED 300K INSTANCES CHURNED 2 TEAMS USE IN PRODUCTION • 5 D • , 0 2 c • 6 0 c 600 CPU YEARS SPENT ) ( 7 2 1 AWS re:Invent 2017: AWS Batch: Easy and Efficient Batch Computing on AWS (CMP323) https://www.youtube.com/watch?v=8dApnlJLY54
- 37. 비용 최적화 (Cost-optimized) • 인프라 설정 및 관리 운영 부담 최소화 • 언제나 원하는 시간에 따라 다양한 작업 실행 가능 • 별도 SW 구매 없이도 다양한 배치 작업에 맞는 작업 구성 관리 가능 비지니스 현장에서 발생하는 다양한 요구 사항에 대해 민첩하게 대응 가능한 배치 작업 아키텍처 제공 가능 자원 최적화 (Resource-optimized) • 예산이 한정되어 있는 경우 • 다중 업무에 따른 사내 작업 순위와 컴퓨팅 환경을 공유해야 하는 경우 • 기존 구매 자원 (Reserved Instance) 등의 활용을 못하고 있는 경우 RI 시간 최적화(Time-optimized) • 작업에 데드라인이 있는 경우, 작업 전체 시간을 줄이고 싶을 때 • 다양한 컴퓨팅 인스턴스와 지불 방식을 조합하여 빠르게 끝낼 수 있음
- 38. • / P ( T o S - a no r • A: . SL T w / A dr P s r fc • A: / W f He T h o S • / / : / / fc W b lt P u Ws g k • m .) / P p i CC B :C D EB B DBC A F G B R ! / D FMS P RC EF
- 39. • FFB H IA A F F • FFB H IA A A F C • : FFB H IA A F : FF : F DF • FFB A H IA A A D F F F G D:G H F F F • • B A AD F F B G F A - F :D F A FFB : F G A H H F : A • A BGF A: /A F D : AH FA F /+ FFB H IA A A: A BGF D F : BD A F • B . D : A F FFB H IA A A: A BGF B D : A H F
- 40. : : - / .! .