![[SERVICE]
Flush 5
Daemon on
Log_Level Info
Coro_Stack_Size 8092
[INPUT]
Name tail
Path /var/log/someservice.log
Refresh_Interval 7
Tag app
[FILTER]
Name nest
Match *
Operation nest
Wildcard *
Nest_under event
[FILTER]
Name modify
Match *
Add index main_dev
Add source http:
Add sourcetype someservice
[OUTPUT]
Name splunk
Host *.splunkcloud.com
Port 443
tls On
tls.Verify Off
Splunk_Token SPLUNK_TOKEN_PLACEHOLDER
Splunk_Send_Raw On
Match *](https://image.slidesharecdn.com/shippinglogstosplunkfromacontainerinawshowto-191021135948/85/Shipping-logs-to-splunk-from-a-container-in-aws-howto-6-320.jpg)
![{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "app",
"image": “some-service:@build_number@",
"essential": true,
"portMappings": [
{
"hostPort": 8080,
"containerPort": 8080
}
],
"links": [
"fluentd"
],
"logConfiguration": {
"logDriver": "fluentd",
"options": {
"fluent-address": "fluentd:24224"
}
}
},
{
"name": "fluentd",
"image": “fluentd-logger:403323",
"essential": true
}
]
}
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "my-bucket",
"Key": "mydockercfg"
},
"Image": {
"Name": "quay.io/johndoe/private-image",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "1234"
}
],
"Volumes": [
{
"HostDirectory": "/var/app/mydb",
"ContainerDirectory": "/etc/mysql"
}
],
"Logging": "/var/log/nginx"
}](https://image.slidesharecdn.com/shippinglogstosplunkfromacontainerinawshowto-191021135948/85/Shipping-logs-to-splunk-from-a-container-in-aws-howto-9-320.jpg)
![ERROR: Encountered error starting new ECS task: {cancel
the command.
"failures": [
{
"reason": "ATTRIBUTE",
"arn": "arn:aws:ecs:ap-northeast-
1:000000000000:container-instance/00000000-0000-
0000-0000-000000000000"
}
],
"tasks": []
}
ERROR: Failed to start ECS task after retrying 2 times.
ERROR: [Instance: i-00000000] Command failed on
instance. Return code: 1 Output:
beanstalk/hooks/appdeploy/enact/03start-task.sh failed.
For more detail, check /var/log/eb-activity.log using
console or EB CLI](https://image.slidesharecdn.com/shippinglogstosplunkfromacontainerinawshowto-191021135948/85/Shipping-logs-to-splunk-from-a-container-in-aws-howto-11-320.jpg)
![files:
"/home/ec2-user/setup-available-log-dirvers.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
set -e
if ! grep splunk /etc/ecs/ecs.config &> /dev/null
then
echo 'ECS_AVAILABLE_LOGGING_DRIVERS=["json-
file","syslog","fluentd", "splunk"]' >> /etc/ecs/ecs.config
fi
container_commands:
01-configure-splunk:
command: /home/ec2-user/setup-available-log-dirvers.sh
02-stop-ecs:
command: stop ecs
03-stop-ecs:
command: start ecs
001_splunk.config](https://image.slidesharecdn.com/shippinglogstosplunkfromacontainerinawshowto-191021135948/85/Shipping-logs-to-splunk-from-a-container-in-aws-howto-12-320.jpg)
Shipping logs to splunk from a container in aws howto
- 1. SHIPPING LOGS TO SPLUNK FROM A CONTAINER IN AWS HOWTO. ADVANTAGES OF RUNNING CONTAINERS IN AWS FARGATE.
- 2. Необходимо предоставить решение по доставке логов приложения в Спланк клауд из контейнера в AWS, с использованием HEC Миграция java сервисов с AWS EB на AWS Fargate
- 3. LOGGING Неоходимо отказаться от использования on-premises kafka логгирования и перейти на облачный сервис Спланк используя HTTP Event Collector; Отсылаем все что приложение пишет в stdout stderr; Сделать это нужно не используя CloudWatch; Мы запускаем контейнеры в AWS EB – 90% и AWS FG – 10% The HTTP Event Collector (HEC) is a fast and efficient way to send data to Splunk Enterprise and Splunk Cloud. Notably, HEC enables you to send data over HTTP (or HTTPS) directly to Splunk Enterprise or Splunk Cloud from your application.
- 4. Fluentbit – Cloud native log forwarder Fluentd – Unified logging Layer Filebeat+Logstash Filebeat и Fluentd не подошли по причине своей тяжеловесности.
- 5. Ради экперимента сделали контейнер c приложением + fluentbit RUN mkdir -p /home/fluent-bit && cd /home/fluent-bit && wget https://fluentbit.io/releases/1.0/fluent-bit-1.0.6.tar.gz && tar xzf fluent-bit-1.0.6.tar.gz && rm fluent-bit-1.0.6.tar.gz && cd fluent-bit-1.0.6 && cmake . && make && make install Возникла необходимость скриптом запускать оба процесса и контролировать чтобы один из процессов не завершил работу.
- 6. [SERVICE] Flush 5 Daemon on Log_Level Info Coro_Stack_Size 8092 [INPUT] Name tail Path /var/log/someservice.log Refresh_Interval 7 Tag app [FILTER] Name nest Match * Operation nest Wildcard * Nest_under event [FILTER] Name modify Match * Add index main_dev Add source http: Add sourcetype someservice [OUTPUT] Name splunk Host *.splunkcloud.com Port 443 tls On tls.Verify Off Splunk_Token SPLUNK_TOKEN_PLACEHOLDER Splunk_Send_Raw On Match *
- 7. Что рекомендуют на сайте Спланка? AWS Cloudwatch + AWS lambda
- 8. Docker-compose?!! Application container + Fluentbit container ! Что предлагает AWS EB для мультиконтейнерных приложений? “AWSEBDockerrunVersion”: 2 https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html#create_deploy_docker_v2config_dockerrun_format
- 9. { "AWSEBDockerrunVersion": 2, "containerDefinitions": [ { "name": "app", "image": “some-service:@build_number@", "essential": true, "portMappings": [ { "hostPort": 8080, "containerPort": 8080 } ], "links": [ "fluentd" ], "logConfiguration": { "logDriver": "fluentd", "options": { "fluent-address": "fluentd:24224" } } }, { "name": "fluentd", "image": “fluentd-logger:403323", "essential": true } ] } { "AWSEBDockerrunVersion": "1", "Authentication": { "Bucket": "my-bucket", "Key": "mydockercfg" }, "Image": { "Name": "quay.io/johndoe/private-image", "Update": "true" }, "Ports": [ { "ContainerPort": "1234" } ], "Volumes": [ { "HostDirectory": "/var/app/mydb", "ContainerDirectory": "/etc/mysql" } ], "Logging": "/var/log/nginx" }
- 11. ERROR: Encountered error starting new ECS task: {cancel the command. "failures": [ { "reason": "ATTRIBUTE", "arn": "arn:aws:ecs:ap-northeast- 1:000000000000:container-instance/00000000-0000- 0000-0000-000000000000" } ], "tasks": [] } ERROR: Failed to start ECS task after retrying 2 times. ERROR: [Instance: i-00000000] Command failed on instance. Return code: 1 Output: beanstalk/hooks/appdeploy/enact/03start-task.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI
- 12. files: "/home/ec2-user/setup-available-log-dirvers.sh": mode: "000755" owner: root group: root content: | #!/bin/sh set -e if ! grep splunk /etc/ecs/ecs.config &> /dev/null then echo 'ECS_AVAILABLE_LOGGING_DRIVERS=["json- file","syslog","fluentd", "splunk"]' >> /etc/ecs/ecs.config fi container_commands: 01-configure-splunk: command: /home/ec2-user/setup-available-log-dirvers.sh 02-stop-ecs: command: stop ecs 03-stop-ecs: command: start ecs 001_splunk.config
- 13. AWS ELASTICBEANSTALK Почему получилось так что его использовали для запуска контейнеров
- 14. AWS ELASTICBEANSTALK 01/2011 04/23/2014 We are excited to announce that you can now create and manage Docker containers in AWS Elastic Beanstalk. Elastic Beanstalk is built on top of the proven AWS infrastructure. It takes full advantage of Amazon EC2, Elastic Load Balancing, Amazon CloudWatch, Auto Scaling, and other AWS services. You get all of the economy and scalability of AWS in a form that’s easier and quicker to deploy than ever before. With Elastic Beanstalk you can choose to gradually assert control over a number of aspects of your application. You can start by tuning a number of parameters (see my post on the Elastic Beanstalk Console for more information about this). You can choose the EC2 instance type that provides the optimal amount of RAM and CPU power for your application. You can log in to the EC2 instances to troubleshoot application issues, and you can even take the default Elastic Beanstalk AMI (Amazon Machine Image), customize it, and then configure Amazon Beanstalk to use it for your application. This gradual assertion of control extends all the way to “eleven” — you can choose to move your application off of Elastic Beanstalk and manage the raw components yourself if you so choose. Elastic Beanstalk was designed to support multiple languages and application environments. We are already working with solution providers to make this happen. Each of your Elastic Beanstalk applications will be run on one or more EC2 instances that are provisioned just for your application. Applications running on Elastic Beanstalk have the same degree of security as those running on an EC2 instance that you launch yourself.
- 15. AWS FARGATE Announced 11/29/2017 AWS Fargate is a compute engine for deploying and managing containers without having to manage any of the underlying infrastructure. Fargate makes it easy to scale your applications. You no longer have to worry about provisioning enough compute resources for your container applications. You can launch tens or tens of thousands of containers in seconds. Previously, you needed to manage a cluster of Amazon EC2 instances, pick the instance types, manage the scheduling of the containers, and optimize cluster utilization. With Fargate, all of this goes away. Fargate seamlessly integrates with Amazon ECS. You just define your application as you do today for Amazon ECS. You package your application into task definitions, specify the CPU and memory needed, define the networking and IAM policies each container needs. Once everything is setup, Fargate launches and manages your containers for you. With Fargate, billing is at a per second granularity and you only pay for what you use. You pay for the amount of vCPU and memory resources your containerized application requests. vCPU and memory resources are calculated from the time your container images are pulled until the Amazon ECS Task terminates, rounded up to the nearest second.
- 16. AWS EB pricing There is no additional charge for AWS Elastic Beanstalk. You pay for AWS resources (e.g. EC2 instances or S3 buckets) you create to store and run your application. Fargate pricing per vCPU per hour $0.04048 per GB per hour $0.004445 Daily charge using Docker in AWS EB: t2.small linux 24 * $0.0208 = 0.4992 + 0.0266(EBS price) = 0.5258 Daily charge using Fargate: 1vCPU + 2Gb 24*(0.25*0.04048+2*0.00445) = 0.45648 Effective Jan 07, 2019, we are reducing the price for AWS Fargate by 20% for vCPU and 65% for memory across all regions where Fargate is currently available.
- 17. A few Fargate advantages: • No cluster to manage • Seamless scaling • Much easier to deploy • Supports shipping logs to SPLUNK out of the box