{"id":57715,"date":"2023-07-30T10:06:52","date_gmt":"2023-07-30T04:36:52","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=57715"},"modified":"2024-01-02T18:23:29","modified_gmt":"2024-01-02T12:53:29","slug":"boosting-ecs-task-monitoring-with-cloudwatch-input-transformer","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/boosting-ecs-task-monitoring-with-cloudwatch-input-transformer\/","title":{"rendered":"Boosting ECS Task Monitoring with CloudWatch Input Transformer"},"content":{"rendered":"

Introduction<\/span><\/span><\/h2>\n

In the fast-paced world of application delivery, ensuring the health and reliability of our ECS tasks is crucial. Without a reliable alerting mechanism, there\u2019s a risk of overlooking critical task failures that can have a bad impact on our production environment. Just imagine a situation where application tasks fail silently, resource constraints go unnoticed, or container failures go unattended. This can result in costly downtime and leave end users frustrated.<\/span><\/p>\n

But fear not! In this brief article, we will delve into the journey to revolutionize our ECS Cluster monitoring capabilities. By actively detecting and addressing task failures, we can minimize downtime, optimize resource utilization, and ensure a seamless experience for our end users.<\/span><\/p>\n

Join us as we dive deep into the world of\u00a0<\/span>Amazon EventBridge<\/span>,\u00a0<\/span>CloudWatch Input Transformers<\/span>, and\u00a0<\/span>SNS (Simple Notification Service)<\/span>. We will unravel the step-by-step implementation of task failure alerts, illuminating the path to actionable and targeted notifications. Get ready to witness the magic unfold as we unlock the true potential of ECS task monitoring.<\/span><\/p>\n

Problem Statement<\/span><\/span><\/h2>\n

Lack of Task Failure Alerts in ECS Cluster Impacts Production Applications.<\/span><\/strong><\/span><\/p>\n

Description<\/span>:<\/span>\u00a0Our production ECS Cluster experienced a critical issue when tasks within our main application started failing unexpectedly. Unfortunately, we discovered that we were not receiving any alerts specifically for task failures, relying only on the 5xx alert generated by the Application Load Balancer. This absence of task failure alerts hindered our ability to promptly identify and address the root cause of the failures, prolonging the impact on the application’s capabilities.<\/span><\/p>\n

Architecture<\/span><\/span><\/h2>\n

\"\"<\/p>\n

Understanding the Components:-<\/span><\/p>\n

1. Amazon Simple Notification Service (SNS):<\/strong> SNS is a fully managed pub\/sub messaging service that allows you to publish, subscribe, and send messages to various endpoints. In our case, we will use SNS to send notifications whenever an ECS task failure happens.<\/p>\n

2. Amazon EventBridge:<\/strong> AWS EventBridge is a serverless event bus that makes it easy to connect different AWS services together and trigger actions based on events. We will utilize EventBridge to capture and process ECS task failure events.<\/p>\n

3. CloudWatch Input Transformer:<\/strong> CloudWatch Input Transformer is a feature of Amazon CloudWatch Events that allows you to extract, modify, and combine fields from incoming events before sending them to targets like SNS topics or AWS Lambda functions. Using the Input Transformer’s power, we will parse ECS task failure events and extract useful details for our alerting purpose.<\/p>\n

Prerequisites<\/span><\/span><\/h2>\n

1. In this demo, we will be utilizing the ECS Fargate Cluster. Please ensure that you have an active ECS Fargate Cluster running.<\/span><\/p>\n

2. Additionally, you will require a task definition and an ECS service. Make sure your task is up.<\/span><\/p>\n

D<\/span>eployment Of Alerting Mechanism:<\/span><\/span><\/h2>\n

1. In the AWS Management Console, search for “<\/span>SNS<\/span>” in the services search bar and click on “Simple Notification Service” when it appears.<\/span><\/p>\n

2. Click the “Topics” section in the left navigation pane in the SNS console<\/span>.<\/span><\/p>\n

3. Provide the name of your topic in the “Name” field. You can choose a display name that helps you identify the purpose of the topic.<\/span><\/p>\n

\"\"<\/p>\n

4. Click on the “<\/span>Create topic<\/span>” button to create the SNS topic.<\/span><\/p>\n

\"\"<\/p>\n

5. Select the topic by clicking on its name. In the topic, details view, click on the “<\/span>Create subscription<\/span>” button. Choose the protocol as “<\/span>Email<\/span>” from the dropdown menu. Enter the email address where you want to receive the alerts in the “<\/span>Endpoint<\/span>” field.<\/span><\/p>\n

\"\"<\/p>\n

6. Check the inbox of the subscribed email address for a confirmation message from AWS SNS. Click & subscribe to the notifications.<\/span><\/p>\n

7. Once confirmed, the subscription status will change to “<\/span>Confirmed<\/span>” in the SNS console.<\/span><\/p>\n

\"\"<\/p>\n

8. Now, \u00a0 event rule. Open the CloudWatch console by visiting <\/span>\u00a0<\/a><\/span>https:\/\/console.aws.amazon.com\/cloudwatch\/<\/a><\/span>.<\/span><\/p>\n

9. In the navigation pane, click on “Events” and then click on “Create rule”. Provide a name and an optional description for your rule. Click on “Next” to proceed with the configuration of your rule.<\/span><\/p>\n

\"\"<\/p>\n

10. Under the creation method, select “Custom event pattern” (JSON EDITOR).<\/span><\/p>\n

11. Since, <\/span>By default, all three states(Running, stopped & Pending) will be tracked by the Cloudwatch rule but <\/span>we only want stopped tasks alerts only. Enter the following pattern in the custom event pattern tab. Click on next.<\/span><\/p>\n\n\n\n
\n
{\r\n\u00a0<\/span>\"detail\"<\/span>: {\r\n\u00a0 \u00a0<\/span>\"lastStatus\"<\/span>: [<\/span>\"STOPPED\"<\/span>],\r\n\u00a0 \u00a0<\/span>\"stoppedReason\"<\/span>: [{\r\n\u00a0 \u00a0 \u00a0<\/span>\"anything-but\"<\/span>: {\r\n\u00a0 \u00a0 \u00a0 \u00a0<\/span>\"prefix\"<\/span>:\u00a0<\/span>\"Scaling activity initiated by\"<\/span>\r\n\u00a0 \u00a0 \u00a0}\r\n\u00a0 \u00a0}]\r\n\u00a0},\r\n\u00a0<\/span>\"detail-type\"<\/span>: [<\/span>\"ECS Task State Change\"<\/span>],\r\n\u00a0<\/span>\"source\"<\/span>: [<\/span>\"aws.ecs\"<\/span>]\r\n}<\/span><\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\"\"<\/p>\n
\n
Note:<\/span>–<\/span>One thing you might notice is that we have used prefix matching with\u00a0<\/span>anything-but<\/span>\u00a0to ignore alerts when tasks are stopped during deployment or autoscaling. This approach ensures that we don’t receive alerts for task stops initiated during deployment, as it doesn’t make sense to be alerted for those specific cases.<\/span><\/em><\/p>\n
In the provided custom event pattern, `anything-but` and `prefix` are used as matching rules for the `stoppedReason` field. Here’s what both of them mean:<\/span><\/em><\/p>\n