Long running aws lambda - Joel Schuweiler, Minneapolis
0 likes152 views
The document discusses running long-running tasks using AWS Lambda and ECS. It describes how Lambda has runtime and resource limits that prevent long running processes. It then outlines a method to use Lambda to trigger an ECS task to run the long-running process instead of running it directly in Lambda. Key aspects of the ECS task definition and IAM roles are also summarized.
![ECS Task Definition
{
"executionRoleArn": "arn:aws:iam::account_id:role/DockerContainerS3IAMRole",
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/aws/lambda/DatabaseBackups",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "DatabaseBackups"
}
},
"environment": [
{
"name": "DB_DUMP_DIR",
"value": "/opt/database/backups"
}
"image": "account_id.dkr.ecr.us-east-1.amazonaws.com/DatabaseBackups:latest",
"name": "DatabaseBackups"
}
],
"taskDefinitionArn": "arn:aws:ecs:us-east-1:account_id:task-definition/DatabaseBackups:6",
"family": "DatabaseBackups",
"requiresCompatibilities": [
"FARGATE"
]
}](https://image.slidesharecdn.com/longrunningawslambda-180611144209/85/Long-running-aws-lambda-Joel-Schuweiler-Minneapolis-6-320.jpg)
![The Lambda
import boto3
import os
def lambda_handler(event, context):
print('Starting lambda, request-id: ', context.aws_request_id)
print('ECS Cluster: ', os.environ['ecs_cluster'])
print('ECS Task Definition: ', os.environ['ecs_task_definition'])
print('ECS subnets: ', os.environ['ecs_subnet'])
ecs = boto3.client('ecs')
response = ecs.run_task(
cluster=os.environ['ecs_cluster'],
taskDefinition=os.environ['ecs_task_definition'],
launchType='FARGATE',
networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
os.environ['ecs_subnet'],
],
'assignPublicIp': 'ENABLED'
}
},
count=1
)](https://image.slidesharecdn.com/longrunningawslambda-180611144209/85/Long-running-aws-lambda-Joel-Schuweiler-Minneapolis-8-320.jpg)
Long running aws lambda - Joel Schuweiler, Minneapolis
- 2. AWS Lambda Limitations: ● 300 seconds run time ● 512MB disk space ● 3008 MB memory ● 1,024 file descriptors ● 1,024 Processes/Threads ● 6MB RequestResponse/synchronous body payload invocation ● 128K Event/asynchronous invocation body payload invocation
- 3. Limitless AWS Lambda ● No time limit ● No code run time requirements
- 4. Limitless AWS Lambda Lambda ● A trigger for that Function ● IAM Role ● A place to log output ● Ultimately launches ECS Two Cycles and their requirements ECS ● VPC ● Subnet ● Security Group ● IAM Role ● A place to log output ● Ultimately runs your code
- 5. ECS Basics ● Repositories ● Clusters ● Task Definition
- 6. ECS Task Definition { "executionRoleArn": "arn:aws:iam::account_id:role/DockerContainerS3IAMRole", "containerDefinitions": [ { "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/aws/lambda/DatabaseBackups", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "DatabaseBackups" } }, "environment": [ { "name": "DB_DUMP_DIR", "value": "/opt/database/backups" } "image": "account_id.dkr.ecr.us-east-1.amazonaws.com/DatabaseBackups:latest", "name": "DatabaseBackups" } ], "taskDefinitionArn": "arn:aws:ecs:us-east-1:account_id:task-definition/DatabaseBackups:6", "family": "DatabaseBackups", "requiresCompatibilities": [ "FARGATE" ] }
- 8. The Lambda import boto3 import os def lambda_handler(event, context): print('Starting lambda, request-id: ', context.aws_request_id) print('ECS Cluster: ', os.environ['ecs_cluster']) print('ECS Task Definition: ', os.environ['ecs_task_definition']) print('ECS subnets: ', os.environ['ecs_subnet']) ecs = boto3.client('ecs') response = ecs.run_task( cluster=os.environ['ecs_cluster'], taskDefinition=os.environ['ecs_task_definition'], launchType='FARGATE', networkConfiguration={ 'awsvpcConfiguration': { 'subnets': [ os.environ['ecs_subnet'], ], 'assignPublicIp': 'ENABLED' } }, count=1 )
- 9. Backing up a Database