{"id":33850,"date":"2016-04-30T22:04:03","date_gmt":"2016-04-30T16:34:03","guid":{"rendered":"http:\/\/www.tothenew.com\/blog\/?p=33850"},"modified":"2016-12-19T15:02:53","modified_gmt":"2016-12-19T09:32:53","slug":"monitor-aws-ecs-agent-automatically-restart-agent-on-failure","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/monitor-aws-ecs-agent-automatically-restart-agent-on-failure\/","title":{"rendered":"Monitor AWS ECS Agent & Automatically Restart Agent on Failure"},"content":{"rendered":"

Amazon EC2 Container Service<\/strong> is a container management service that makes it easy to manage docker<\/a> containers on EC2 instances. AWS ECS you can create task definition to define container configuration like memory, cpu, environment variables, mount point and services to scale docker containers.<\/p>\n

\"AWS<\/p>\n

Use Case:<\/strong> In one of our project we setup complete QA environment on AWS ECS and after few days we observed ECS agent gets frequently disconnected with the AWS ECS service. As a result AWS ECS service is unable to communicate with ECS agent resulting in no more schedulding and unable to get any status of the existing containers.<\/p>\n

Note: We are using AWS ECS Optimized AMI i.e. Amazon Linux AMI, if you are using other OS AMI few steps may change i.e. install aws and getting metadata.<\/span><\/p>\n

Steps to setup monitoring script on ECS nodes:<\/strong><\/span><\/p>\n

1. Setup SNS topic for recieving notifications<\/strong><\/p>\n

On the AWS console create sns topic and in the subscriber add notification email id, confirm the subscription you recieved from the SNS service.<\/p>\n

2. Install AWS CLI<\/strong><\/p>\n

Our script will use AWS CLI to query AWS to find container instance arn and agent status using awscli ecs command option.<\/p>\n

[js]yum install -y aws-cli[\/js]<\/p>\n

3. Setup IAM policies for SNS and ECS<\/strong><\/p>\n

a. AWS SNS IAM Policy:<\/strong> The below mentioned policy will allow IAM instance role to publish message to the SNS topic we created earlier. This will help us in getting notifications for agent failure.<\/p>\n

[js]
\n{
\n\t "Version": "2012-10-17",
\n\t "Statement": [
\n\t {
\n\t "Sid": "Stmt1460976768000",
\n\t "Effect": "Allow",
\n\t "Action": [
\n\t "sns:GetEndpointAttributes",
\n\t "sns:GetPlatformApplicationAttributes",
\n\t "sns:GetSubscriptionAttributes",
\n\t "sns:GetTopicAttributes",
\n\t "sns:ListEndpointsByPlatformApplication",
\n\t "sns:ListPlatformApplications",
\n\t "sns:ListSubscriptions",
\n\t "sns:ListSubscriptionsByTopic",
\n\t "sns:ListTopics",
\n\t "sns:Publish"
\n\t ],
\n\t "Resource": [
\n\t "arn:aws:sns:ap-southeast-1:<aws-account-id>:<topic-name>"
\n\t ]
\n\t }
\n\t ]
\n\t}
\n[\/js]<\/p>\n

b. AWS ECS IAM Policy:<\/strong> The below mentioned IAM policy will allow IAM instance role to query AWS ECS api to list container instances and check agent connectivity status.<\/p>\n

[js]
\n{
\n "Version": "2012-10-17",
\n "Statement": [
\n {
\n "Sid": "Stmt1460960788000",
\n "Effect": "Allow",
\n "Action": [
\n "ecs:DescribeClusters",
\n "ecs:DescribeContainerInstances",
\n "ecs:DescribeServices",
\n "ecs:DescribeTaskDefinition",
\n "ecs:DescribeTasks",
\n "ecs:DiscoverPollEndpoint",
\n "ecs:ListClusters",
\n "ecs:ListContainerInstances",
\n "ecs:ListServices",
\n "ecs:ListTaskDefinitionFamilies",
\n "ecs:ListTaskDefinitions",
\n "ecs:ListTasks",
\n "ecs:Poll"
\n ],
\n "Resource": [
\n "arn:aws:ecs:ap-southeast-1:<aws-account-id>:cluster\/<cluster-name>"
\n ]
\n }
\n ]
\n}
\n[\/js]<\/p>\n

4. Monitoring Script<\/strong><\/p>\n

The below mentioned script will check for ECS agent connectivity with the ECS service, it first extract all the container instances arns, instance id (using metadata). It will then check for each container instance arn for its current status check weather its on the same instance. If current instance ECS agent is donnected it will trigger a notification and restart ecs service on the instance.<\/p>\n

[js]
\n#!\/bin\/bash
\n# Sourcing the ecs.config file for using the cluster name
\nsource \/etc\/ecs\/ecs.config
\nCONTAINERS_ID=$(aws ecs list-container-instances –cluster $ECS_CLUSTER –output text –query ‘containerInstanceArns’)
\nINSTANCE_ID=$(curl
\nDATE=$(date +%Y-%m-%d-%H:%M)
\nTOPIC="arn:aws:sns:ap-southeast-1:<aws-account-id>:<topic-name>"
\nfor container in $CONTAINERS_ID
\ndo
\nSTATUS=$(aws ecs describe-container-instances –container-instances $container –cluster $ECS_CLUSTER –output json –query ‘containerInstances[0].agentConnected’)
\nCHECK_INSTANCE_ID=$(aws ecs describe-container-instances –container-instances $container –cluster $ECS_CLUSTER –output text –query ‘containerInstances[0].ec2InstanceId’)
\nif [ $INSTANCE_ID == $CHECK_INSTANCE_ID ]
\nthen
\nif [ $STATUS == "false" ]
\nthen
\necho "Agent Disconnected" $DATE &gt;&gt; \/var\/log\/script.log
\naws sns publish –message "AWS ECS Agent Failed $INSTANCE_ID $DATE" –topic $TOPIC
\nsudo stop ecs
\nsudo start ecs
\nelse
\necho "Agent Connected" $DATE &gt;&gt; \/var\/log\/script.log
\nfi
\nfi
\ndone[\/js]<\/p>\n

5. Setup cron to run every 5 minutes<\/strong><\/p>\n

After monitoring QA environment for more than 1 week, we found ECS agent gets disconnected almost twice daily, so I choose to setup cronjob to run every 5 minutes and writing error logs to \/var\/log\/monitor-agent-logs.txt<\/p>\n

[js]*\/5 * * * * bash \/home\/ec2-user\/monitor_agent.sh 2&>1 \/var\/log\/monitor-agent-logs.txt[\/js]<\/p>\n

6. Create AMI and Update ECS Auto Scaling Groups Launch Configuration<\/strong><\/p>\n

Once you create a AMI of running instance, copy the existing launch configuration (i.e. created by AWS ECS Cloudformation Stack), update the AMI and create new launch configuration. Update the auto scaling group to use newly created launch configuration.<\/p>\n","protected":false},"excerpt":{"rendered":"

Amazon EC2 Container Service is a container management service that makes it easy to manage docker containers on EC2 instances. AWS ECS you can create task definition to define container configuration like memory, cpu, environment variables, mount point and services to scale docker containers. Use Case: In one of our project we setup complete QA […]<\/p>\n","protected":false},"author":216,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":52},"categories":[1174,2348],"tags":[3273,2594,3272],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/33850"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/216"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=33850"}],"version-history":[{"count":0,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/33850\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=33850"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=33850"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=33850"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}