{"id":59999,"date":"2024-01-08T17:18:12","date_gmt":"2024-01-08T11:48:12","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=59999"},"modified":"2024-01-16T17:48:54","modified_gmt":"2024-01-16T12:18:54","slug":"aws-x-ray-configuration-with-eks-cluster","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/aws-x-ray-configuration-with-eks-cluster\/","title":{"rendered":"AWS X-Ray Configuration with EKS Cluster"},"content":{"rendered":"<h2><b>Getting Started with AWS X-Ray<\/b><\/h2>\n<p><b>AWS X-Ray<\/b><span style=\"font-weight: 400;\"> is a service that collects data about requests that your application serves, and provides tools that you can use to view, filter, and gain insights into that data to identify issues and opportunities for optimization.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For any traced request to your application, you can see detailed information not only about the request and response but also about calls that your application makes to downstream AWS resources, microservices, databases, and web APIs.<\/span><\/p>\n<h2><b>Benefits of AWS RUM<\/b><\/h2>\n<p><b>Simple Setup<\/b><b>:\u00a0 <\/b><span style=\"font-weight: 400;\">AWS X-Ray can be used with applications running on <\/span><a href=\"https:\/\/aws.amazon.com\/ec2\/\"><span style=\"font-weight: 400;\">Amazon Elastic Compute Cloud<\/span><\/a><span style=\"font-weight: 400;\"> (EC2), <\/span><a href=\"https:\/\/aws.amazon.com\/ecs\/\"><span style=\"font-weight: 400;\">Amazon EC2 Container Service<\/span><\/a><span style=\"font-weight: 400;\"> (Amazon ECS), <\/span><a href=\"https:\/\/aws.amazon.com\/lambda\/\"><span style=\"font-weight: 400;\">AWS Lambda<\/span><\/a><span style=\"font-weight: 400;\">, <\/span><a href=\"https:\/\/aws.amazon.com\/elasticbeanstalk\/\"><span style=\"font-weight: 400;\">AWS Elastic Beanstalk<\/span><\/a><span style=\"font-weight: 400;\">. It\u2019s easy to get started with X-Ray. You just integrate the X-Ray SDK with your application and install the X-Ray agent.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">With AWS Elastic Beanstalk, you only have to integrate the X-Ray SDK with your application since the X-Ray agent is pre-installed on Elastic Beanstalk.<\/span><\/p>\n<p><b>End-to-end tracing<\/b><b>: <\/b><span style=\"font-weight: 400;\">AWS X-Ray provides an end-to-end, cross-service view of requests made to your application. It gives you an application-centric view of requests flowing through your application by aggregating the data gathered from individual services into a single unit called a trace.<\/span><\/p>\n<p><b>Service map: <\/b><span style=\"font-weight: 400;\">AWS X-Ray creates a map of services used by your application with trace data that you can use to drill into specific services or issues<\/span><\/p>\n<p><b>Data annotation and filtering: <\/b><span style=\"font-weight: 400;\">AWS X-Ray lets you add annotations to data emitted from specific components or services in your application. You can use this to append business-specific metadata that help you better diagnose issues.\u00a0<\/span><\/p>\n<p><b>Console and programmatic access: <\/b><span style=\"font-weight: 400;\">You can use AWS X-Ray with the AWS Management Console, AWS CLI, and AWS SDKs. The X-Ray API lets you programmatically access the service so you can easily export trace data or ingest the data into your own tools and custom analytics dashboards.<\/span><\/p>\n<p><b>Security: <\/b><span style=\"font-weight: 400;\">AWS X-Ray is integrated with <\/span><a href=\"https:\/\/aws.amazon.com\/iam\/\"><span style=\"font-weight: 400;\">AWS Identity and Access Management<\/span><\/a><span style=\"font-weight: 400;\"> (IAM) so that you can control which users and resources have permission to access your traces and how.<\/span><\/p>\n<h2><b>Problem Statement<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">In the case of Latency monitoring in the EKS cluster, we were not able to do it. We have set up the ELK for logging and Grefana and Prometheus for monitoring our environment. But still, we were not able to monitor every request&#8217;s latency or traces of all requests.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We were also facing the issue of tracking the slow part of the application, like in case we are using Kafka, EKS, and RDS in the cluster and there is some slowness in the application so It was very difficult to find which service is performing slow.<\/span><\/p>\n<h2><b>Solution Approach<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">As we have mentioned, there are some problem statements regarding managing and monitoring the EKS cluster. So, the approach to fix this problem is that you can implement the AWS X-ray in your application. Below are the prerequisites and implementation plan for the AWS X-ray.<\/span><\/p>\n<p><b>Prerequisite:<\/b><span style=\"font-weight: 400;\"> \u00a0 The setup process for integrating AWS X-Ray with AWS Distro for OpenTelemetry (ADOT) necessitates adherence to specific prerequisites outlined as follows:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Kubectl installation should align with the respective cluster version.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">eksctl should be installed.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">The installation of eksctl is required.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Meet the TLS Certificate Requirement to ensure end-to-end encryption.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Should have an IAM Role with the (AmazonPrometheusRemoteWriteAccess, AWSXrayWriteOnlyAccess, CloudWatchAgentServerPolicy) permissions.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If installing an add-on version that is v0.62.1 or earlier, grant permissions to Amazon EKS add-ons to install ADOT.<\/span><\/li>\n<\/ul>\n<p><b>Implementation:<\/b> <span style=\"font-weight: 400;\">After satisfying the prerequisites, we can proceed with the step-by-step implementation of AWS X-Ray, adhering to the following points:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If installing an add-on version that is v0.62.1 or earlier, grant permissions to Amazon EKS add-ons to install ADOT.<\/span><\/li>\n<\/ul>\n<pre><b><i>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 kubectl apply -f <\/i><\/b><i><span style=\"font-weight: 400;\">https:\/\/amazon-eks.s3.amazonaws.com\/docs\/addons-otel-permissions.yaml<\/span><\/i><\/pre>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Create an IAM OIDC provider to connect the service account to AWS IAM.<\/span><\/li>\n<\/ul>\n<p><b><i>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 eksctl utils associate-iam-oidc-provider &#8211;region=&lt;<\/i><\/b><b><i>AWS-Region<\/i><\/b><b><i>&gt; &#8211;cluster=&lt;<\/i><\/b><b><i>ClusterName<\/i><\/b><b><i>&gt;<\/i><\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Create your service account and IAM role by using the below command. We need to change the highlighted part per the environment in the command.<\/span><\/li>\n<\/ul>\n<pre><i>eksctl create iamserviceaccount \\<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0--name<\/i><i> adot-collector<\/i><i> \\<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0--namespace <\/i><i>default<\/i><i> \\<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0--cluster <\/i><i>my-cluster<\/i><i> \\<\/i>\r\n<i> \u00a0\u00a0\u00a0--attach-policy-arn arn:aws:iam::aws:policy\/AmazonPrometheusRemoteWriteAccess \\<\/i>\r\n<i> \u00a0\u00a0\u00a0--attach-policy-arn arn:aws:iam::aws:policy\/AWSXrayWriteOnlyAccess \\<\/i>\r\n<i> \u00a0\u00a0\u00a0--attach-policy-arn arn:aws:iam::aws:policy\/CloudWatchAgentServerPolicy \\<\/i>\r\n<i> \u00a0\u00a0\u00a0--approve \\<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0--override-existing-serviceaccounts<\/i><\/pre>\n<ul>\n<li>Install the ADOT Amazon EKS add-on to your AWS EKS Cluster by following the below steps:<\/li>\n<\/ul>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li><span style=\"font-weight: 400;\">Open the Amazon EKS console at<\/span><a href=\"https:\/\/console.aws.amazon.com\/eks\/home#\/clusters\"> <span style=\"font-weight: 400;\">https:\/\/console.aws.amazon.com\/eks\/home#\/clusters<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/li>\n<li><span style=\"font-weight: 400;\">In the left pane, select Clusters, and then select the name of your cluster on the Clusters page.<\/span><\/li>\n<li>Choose the Add-ons tab.<\/li>\n<li>Choose Get more add-ons.<\/li>\n<li>On the Select add-ons page, do the following:\n<ul>\n<li><span style=\"font-weight: 400;\">In the Amazon EKS-addons section, select the AWS Distro for OpenTelemetry checkbox.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Choose Next.<br \/>\n<\/span><\/li>\n<\/ul>\n<\/li>\n<li><span style=\"font-weight: 400;\">On the Configure selected add-ons settings page, do the following:<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">The default version will be selected in the Version drop-down. Select the Version you&#8217;d like to use.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">(Optional) If deploying an ADOT Collector, expand the Optional configuration settings and provide the Configuration values that match your use case for Collector deployment. The Add-on configuration schema provides the available options for your configuration values.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Expand the Optional configuration settings and select Override for the Conflict resolution method if a service account is already created in the cluster without an IAM role.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Choose Next.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">On the Review and Add page choose Create. After the add-on installation is complete, you see your installed add-on. <\/span><span style=\"font-weight: 400;\">Change the ADOT values by using the command below.<\/span><\/p>\n<pre><i>aws eks create-addon \\<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0--cluster-name <\/i><i>&lt;Clustername&gt;<\/i><i> \\<\/i>\r\n<i> \u00a0\u00a0\u00a0--addon-name adot \\<\/i>\r\n<i> \u00a0\u00a0\u00a0--configuration-values \"{\\\"manager\\\":{\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"200m\\\"}}}}\" \\<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0--resolve-conflicts=OVERWRITE<\/i><\/pre>\n<ul>\n<li><span style=\"font-weight: 400;\">Now, we need to install a cert-manager to validate the authentication. We need to run the below command for the installation.<\/span><\/li>\n<\/ul>\n<pre><b><i>kubectl apply -f <\/i><\/b><a href=\"https:\/\/github.com\/cert-manager\/cert-manager\/releases\/download\/v1.8.2\/cert-manager.yaml\"><i>https:\/\/github.com\/cert-manager\/cert-manager\/releases\/download\/v1.8.2\/cert-manager.yaml<\/i><\/a><\/pre>\n<p><span style=\"font-weight: 400;\">Now, we need to deploy the ADOT Collector by following the below steps:<\/span><\/p>\n<p><span style=\"font-weight: 400;\">1. Create the YAML file below with the \u201ccollector-config-xray.yaml\u201d name. In this YAML you need to change the Highlighted part as per your environment.<\/span><\/p>\n<pre><i>apiVersion: opentelemetry.io\/v1alpha1<\/i>\r\n<i>kind: OpenTelemetryCollector<\/i>\r\n<i>metadata:<\/i>\r\n<i>\u00a0\u00a0name: <\/i><i>my-collector-xray<\/i>\r\n<i>spec:<\/i>\r\n<i> \u00a0mode: deployment\u00a0<\/i>\r\n<i> \u00a0serviceAccount: adot-collector\u00a0<\/i>\r\n<i> \u00a0config: |<\/i>\r\n<i> \u00a0\u00a0\u00a0receivers:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0otlp:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0protocols:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0grpc:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0endpoint: 0.0.0.0:4317<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0http:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0endpoint: 0.0.0.0:4318<\/i>\r\n<i> \u00a0\u00a0\u00a0processors:<\/i>\r\n<i> \u00a0\u00a0\u00a0exporters:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0awsxray:<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0region: <\/i><i>&lt;YOUR_AWS_REGION&gt;<\/i>\r\n<i> \u00a0\u00a0\u00a0service:<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0pipelines:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0traces:<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0receivers: [otlp]<\/i>\r\n<i> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0processors: []<\/i>\r\n<i>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0exporters: [awsxray]<\/i><\/pre>\n<p>2. Now, we need to run the above YAML file by using the below command.<\/p>\n<pre><b><i> kubectl apply -f collector-config-xray.yaml<\/i><\/b><\/pre>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Utilize a Sample Application to generate Trace Data by following the below steps:<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">1. Download the traffic-generator.yaml file to your computer by using the below command.<\/span><\/p>\n<pre><b><i>curl -O <\/i><\/b><i><span style=\"font-weight: 400;\">https:\/\/raw.githubusercontent.com\/aws-observability\/aws-otel-community\/master\/sample-configs\/traffic-generator.yaml<\/span><\/i><\/pre>\n<p><span style=\"font-weight: 400;\">\u00a0 2. In traffic-generator.yaml, ensure the second kind value reflects your mode. Which is <\/span><b>\u201ckind: Deployment.\u201d<\/b><\/p>\n<p><span style=\"font-weight: 400;\">\u00a0 3. Now, you need to apply traffic-generator.yaml to your cluster.<\/span><\/p>\n<pre><b><i>kubectl apply -f traffic-generator.yaml<\/i><\/b><\/pre>\n<p><span style=\"font-weight: 400;\">4. Download the sample-app.yaml file to your computer by using the below command.<\/span><\/p>\n<pre style=\"text-align: left;\"><a href=\"https:\/\/raw.githubusercontent.com\/aws-observability\/aws-otel-community\/master\/sample-configs\/sample-app.yaml\"><b><i>curl -O <\/i><\/b><i><span style=\"font-weight: 400;\">https:\/\/raw.githubusercontent.com\/aws-observability\/aws-otel-community\/master\/sample-configs\/sample-app.yaml<\/span><\/i><\/a><\/pre>\n<p><span style=\"font-weight: 400;\">5.\u00a0 In sample-app.yaml, replace the following with your own AWS Region with &#8220;<\/span><b>&lt;YOUR_AWS_REGION&gt;<\/b><span style=\"font-weight: 400;\">&#8220;<\/span><\/p>\n<p>6. In sample-app.yaml, update the value for <b>OTEL_EXPORTER_OTLP_ENDPOINT<\/b> if it doesn&#8217;t match your collector service name. For example, X-Ray requires replacing <a href=\"http:\/\/my-collector-collector:4317\">http:\/\/my-collector-collector:4317<\/a> replace with <a href=\"http:\/\/my-collector-xray-collector:4317\">http:\/\/my-collector-xray-collector:4317<\/a>.<\/p>\n<p>7. Now apply sample-app.yaml to your cluster by following the below command.<\/p>\n<pre><b><i>kubectl apply -f sample-app.yaml<\/i><\/b><\/pre>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Now, the Implementation has been completed. We can validate it by going to the AWS Console.<\/span><\/li>\n<\/ul>\n<h2><b>Debugging<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">There may be an error during the Setup of the AWS X-ray so here are some aspects of debugging in AWS X-Ray setup:<\/span><\/p>\n<p><b><i>Configuration Errors: <\/i><\/b><span style=\"font-weight: 400;\">Verifying that the setup and configuration of X-Ray components, such as SDK integration, permissions, and sampling rules, are correctly implemented.<\/span><\/p>\n<p><b><i>Instrumentation Problems: <\/i><\/b><span style=\"font-weight: 400;\">Identifying any issues with instrumenting services or applications to send trace data to X-Ray. Debugging involves examining SDK integration within each service to ensure it captures relevant information.<\/span><\/p>\n<p><b><i>Permission and Access Issues:<\/i><\/b><span style=\"font-weight: 400;\"> Debugging authorization and IAM roles to ensure X-Ray has the necessary permissions to collect and access tracing information across AWS services and applications.<\/span><\/p>\n<h2><b>Conclusion<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">In conclusion, the implementation of AWS X-Ray has provided us with invaluable insights into our system&#8217;s performance and behavior. This tracing tool has given us a comprehensive understanding of our distributed architecture, allowing us to trace requests, identify bottlenecks, and optimize resource usage. As a result, we are well-positioned to enhance our system&#8217;s efficiency and reliability.<\/span><\/p>\n<div class=\"ap-custom-wrapper\"><\/div><!--ap-custom-wrapper-->","protected":false},"excerpt":{"rendered":"<p>Getting Started with AWS X-Ray AWS X-Ray is a service that collects data about requests that your application serves, and provides tools that you can use to view, filter, and gain insights into that data to identify issues and opportunities for optimization.\u00a0 For any traced request to your application, you can see detailed information not [&hellip;]<\/p>\n","protected":false},"author":1702,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":206},"categories":[4308],"tags":[5599,5600,5601],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/59999"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1702"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=59999"}],"version-history":[{"count":2,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/59999\/revisions"}],"predecessor-version":[{"id":60077,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/59999\/revisions\/60077"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=59999"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=59999"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=59999"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}