{"id":686,"date":"2024-09-09T23:40:00","date_gmt":"2024-09-10T06:40:00","guid":{"rendered":"http:\/\/184.72.63.26\/?p=686"},"modified":"2024-11-08T11:23:54","modified_gmt":"2024-11-08T18:23:54","slug":"how-to-ingest-aws-cloudwatch-metrics-to-dynatrace-using-amazon-data-firehose-and-cloudwatch-metric-stream","status":"publish","type":"post","link":"https:\/\/www.wallacel.com\/index.php\/2024\/09\/09\/how-to-ingest-aws-cloudwatch-metrics-to-dynatrace-using-amazon-data-firehose-and-cloudwatch-metric-stream\/","title":{"rendered":"How to Ingest CloudWatch Metrics to Dynatrace Using Amazon Data Firehose and CloudWatch Metric Stream"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Monitoring the performance of your <strong>AWS ECS<\/strong> (Elastic Container Service) cluster is crucial to ensure your application operates efficiently, especially under varying workloads. While <strong>AWS CloudWatch<\/strong> provides robust tools for tracking metrics, you can enhance your observability by integrating it with other APM tools like <strong>Dynatrace<\/strong> for more advanced dashboards, alerting, and root-cause analysis.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This blog will guide you through streaming <strong>AWS CloudWatch<\/strong> <strong>metrics<\/strong> to <strong>Dynatrace<\/strong> using <strong>Amazon Data Firehose<\/strong> and <strong>CloudWatch Metric Streams<\/strong>. We\u2019ll also cover how to create Dashboard and Service-level Objective (SLO) in Dynatrace to monitor ECS cluster performance.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. <strong>Overview of Streaming CloudWatch Data to Dynatrace<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">AWS CloudWatch offers basic monitoring, but when combined with <strong>Dynatrace<\/strong>, you gain advanced observability features like real-time dashboards, historical data trends, and proactive alerting. This integration allows you to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Monitor <strong>CPU and memory utilization<\/strong> of ECS tasks in real time.<\/li>\n\n\n\n<li>Ingest detailed <strong>logs<\/strong> from ECS tasks for deeper analysis.<\/li>\n\n\n\n<li>Set up custom dashboard and <strong>SLO<\/strong> in Dynatrace to track ECS cluster health.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">We&#8217;ll use:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Amazon Data Firehose<\/strong> to deliver streaming data directly to Dynatrace.<\/li>\n\n\n\n<li><strong>CloudWatch Metric Streams<\/strong> to send metrics to the Data Firehose.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"721\" height=\"471\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/architecture.drawio-1.png\" alt=\"\" class=\"wp-image-729\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/architecture.drawio-1.png 721w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/architecture.drawio-1-300x196.png 300w\" sizes=\"auto, (max-width: 721px) 100vw, 721px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">2. <strong>Prerequisites<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before setting up the streaming pipeline, ensure the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Dynatrace Account<\/strong>: Either you have an active Dynatrace environment or sign up for a free trial account <a href=\"https:\/\/www.dynatrace.com\/signup\/\" data-type=\"link\" data-id=\"https:\/\/www.dynatrace.com\/signup\/\">here<\/a>.<\/li>\n\n\n\n<li><strong>CloudWatch Logs and Metrics<\/strong>: AWS service (e.g. ECS cluster) that is sending logs and metrics to CloudWatch.<\/li>\n\n\n\n<li><strong>IAM Permissions<\/strong>: Your AWS user or role should have appropriate permissions for CloudWatch and Kinesis.<\/li>\n\n\n\n<li><strong>Dynatrace Access Token<\/strong>: Required to authenticate and send data from AWS to Dynatrace.<\/li>\n\n\n\n<li><strong>Dynatrace environment URL<\/strong>: E.g. <em><strong>https:\/\/&lt;your_environment_ID&gt;.live.dynatrace.com<\/strong><\/em><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">3. <strong>Generating a Dynatrace Access Token<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The <strong>Dynatrace Access Token<\/strong> is essential for authenticating your integration with the Dynatrace API, allowing logs and metrics to be sent from AWS to Dynatrace.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step-by-Step: Generating a Dynatrace Access Token<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Log into Dynatrace<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Open your Dynatrace environment and log in with your credentials.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Navigate to API Token Settings<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In the Dynatrace UI, click on the <strong>Settings<\/strong> gear icon located at the bottom left of the navigation pane.<\/li>\n\n\n\n<li>From the settings menu, navigate to <strong>Integration &gt; Dynatrace API<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Create a New Access Token<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Click on the <strong>Generate Token<\/strong> button to create a new Access Token.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Assign Permissions to the Token<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Assign the required permissions to the token based on your use case. To stream CloudWatch Logs and Metrics, ensure you select the following permissions:\n<ul class=\"wp-block-list\">\n<li><strong>Read Metrics<\/strong><\/li>\n\n\n\n<li><strong>Ingest Metrics<\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Save and Copy the Token<\/strong>:\n<ul class=\"wp-block-list\">\n<li>After generating the token, save it in a secure place. You\u2019ll need it for CloudWatch Metric Stream configuration.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"461\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/token-1024x461.png\" alt=\"\" class=\"wp-image-689\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/token-1024x461.png 1024w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/token-300x135.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/token-768x346.png 768w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/token.png 1245w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"672\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/image-1-1024x672.png\" alt=\"\" class=\"wp-image-690\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-1-1024x672.png 1024w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-1-300x197.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-1-768x504.png 768w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-1.png 1044w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">4. <strong>Create Amazon Data Firehose Stream<\/strong><\/h2>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 1: Set Up Amazon Data Firehose<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Create a Data Firehose Stream<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In the <strong>AWS Console<\/strong>, navigate to <strong>Amazon Data Firehose<\/strong> and click on <strong>Create Firehose Stream<\/strong>.<\/li>\n\n\n\n<li>Set the source as <strong>Direct PUT<\/strong> to allow CloudWatch Logs to push data into the stream.<\/li>\n\n\n\n<li>Choose <strong>Dynatrace<\/strong> as the <strong>Destination<\/strong> since you\u2019ll be streaming logs to Dynatrace.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Name the stream:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Enter a name for the Firehose stream<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Transform records:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Make sure data transformation is disabled<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Configure the Destination settings<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In Ingestion type, select <strong>Metrics<\/strong> and choose <strong>Dynatrace &#8211; US<\/strong> as the HTTP endpoint URL<\/li>\n\n\n\n<li>In Authentication, select Use API token and enter your Dynatrace Access Token<\/li>\n\n\n\n<li>In API URL, enter your Dynatrace environment URL (e.g. <code><strong>https:\/\/&lt;your-dynatrace-environment-id&gt;.live.dynatrace.com<\/strong><\/code>)<\/li>\n\n\n\n<li>In Content encoding, select <strong>GZIP<\/strong> to reduce data transfer size<\/li>\n\n\n\n<li>In Retry duration, enter <strong>900<\/strong><\/li>\n\n\n\n<li>In Buffer hints, set an appropriate <strong>Buffer size<\/strong> and <strong>Buffer interval<\/strong> depending on your expected log volume. A buffer size of <strong>3MiB<\/strong> and an interval of <strong>60 seconds<\/strong> is a good starting point<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Set Backup Settings<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In Backup mode, select Failed data only<\/li>\n\n\n\n<li>In S3 backup bucket, either create a new S3 bucket or use an existing one as the backup<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Create the Delivery Stream<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Review your configuration and click on <strong>Create Firehose Stream<\/strong>.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"793\" height=\"833\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/image-2.png\" alt=\"\" class=\"wp-image-693\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-2.png 793w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-2-286x300.png 286w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-2-768x807.png 768w\" sizes=\"auto, (max-width: 793px) 100vw, 793px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"796\" height=\"843\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/firehose.png\" alt=\"\" class=\"wp-image-695\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/firehose.png 796w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/firehose-283x300.png 283w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/firehose-768x813.png 768w\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"798\" height=\"682\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/image-4.png\" alt=\"\" class=\"wp-image-696\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-4.png 798w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-4-300x256.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-4-768x656.png 768w\" sizes=\"auto, (max-width: 798px) 100vw, 798px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"795\" height=\"673\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/firehose-backup.png\" alt=\"\" class=\"wp-image-698\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/firehose-backup.png 795w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/firehose-backup-300x254.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/firehose-backup-768x650.png 768w\" sizes=\"auto, (max-width: 795px) 100vw, 795px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">5. <strong>Create CloudWatch Metric Stream<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>CloudWatch Metric Streams<\/strong> allows you to continually stream CloudWatch metrics to a Data Firehose delivery stream that delivers your metrics to where you want them to go.  You can stream them to a data lake such as Amazon S3, or to any destination or endpoint supported by Firehose including third-party providers. JSON, OpenTelemetry 0.7, and OpenTelemetry 1.0 formats are supported natively, or you can configure transformations in your Firehose delivery stream to convert the data to a different format such as Parquet.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 1: Set Up CloudWatch Metric Stream<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Open the CloudWatch Console<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Navigate to <strong>CloudWatch<\/strong> &gt; <strong>Metrics<\/strong> &gt; <strong>Streams<\/strong>.<\/li>\n\n\n\n<li>Select Create Metric Stream<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Destination:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Choose <strong>Custom setup with Firehose<\/strong> as the destination for the metric stream.<\/li>\n\n\n\n<li>Select the Data Firehose stream you created earlier.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Change Output Format<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Select <strong>OpenTelemetry 1.0<\/strong> as the format for the metrics. Dynatrace supports OpenTelemetry, making this the best format for the data stream.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Metrics to be Streamed<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Choose <strong>All metrics<\/strong> if you want to stream all metrics from namespaces automatically<\/li>\n\n\n\n<li>Choose <strong>Select metrics<\/strong> if you want to include or exclude specific namespaces and metrics<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Custom Metric Stream Name:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Enter a name for your metric stream<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Select <strong>Create metric stream<\/strong> and CloudWatch will start streaming metrics to Dynatrace through the Firehose.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"996\" height=\"818\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/image-6.png\" alt=\"\" class=\"wp-image-700\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-6.png 996w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-6-300x246.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-6-768x631.png 768w\" sizes=\"auto, (max-width: 996px) 100vw, 996px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"991\" height=\"832\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/image-7.png\" alt=\"\" class=\"wp-image-701\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-7.png 991w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-7-300x252.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-7-768x645.png 768w\" sizes=\"auto, (max-width: 991px) 100vw, 991px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 2: Enable AWS Stream in Dynatrace<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">In order to ingest CloudWatch metrics into Dynatrace, you need to install the <strong>AWS Entities for Metric Streaming<\/strong> extension.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Go to Dynatrace Hub<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In Dynatrace, go to <strong>Search <\/strong>and enter <strong>Hub<\/strong>.<\/li>\n\n\n\n<li>Select <strong>Add to environment<\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"712\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/image-8-1024x712.png\" alt=\"\" class=\"wp-image-702\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-8-1024x712.png 1024w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-8-300x209.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-8-768x534.png 768w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/image-8.png 1248w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 3: Verify Metrics in Dynatrace<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Log into Dynatrace<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In Dynatrace, go to <strong>AWS Classic &gt; Go to AWS Metric Streams<\/strong> and select your AWS Account<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Verify Metric Ingestion<\/strong>:\n<ul class=\"wp-block-list\">\n<li>After a few minutes, you should see metrics for your AWS services (like EC2, Containers, RDS, etc.) available.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"752\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/metrics-1024x752-1.png\" alt=\"\" class=\"wp-image-1015\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/metrics-1024x752-1.png 1024w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/metrics-1024x752-1-300x220.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/metrics-1024x752-1-768x564.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">6. <strong>Create SLOs for ECS Performance in Dynatrace<\/strong><\/h2>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 1: Define Your SLOs<\/strong><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Once you have CloudWatch metrics streaming into Dynatrace, you can set up dashboards and create SLO (Service-level Objective) to monitor your AWS service. SLOs are performance targets you aim to achieve over a specific period. For ECS performance, a typical SLO could be:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>CPU Utilization SLO<\/strong>: ECS tasks&#8217; CPU utilization should stay below 80% for 99.9% of the time.<\/li>\n\n\n\n<li><strong>Memory Utilization SLO<\/strong>: ECS tasks&#8217; memory utilization should stay below 70% for 99.9% of the time.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 2: Create SLOs in Dynatrace<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Navigate to SLO Settings<\/strong>:\n<ul class=\"wp-block-list\">\n<li>In Dynatrace, go to <strong>Service-level objectives (SLOs)<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Create a New SLO<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Click on <strong>Add new SLO<\/strong> and provide a descriptive name (e.g., &#8220;ECS CPU Utilization SLO&#8221;).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Define the metric expression that measure the success rate of this SLO<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Enter the following expression to measure the available CPU Utilization of ECS cluster<\/li>\n\n\n\n<li>Replace <strong>CUSTOM_DEVICE-E360A9250EEF25BC<\/strong> with actual id obtained from AWS Metrics Stream for your ECS cluster or other AWS service to be monitored<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"bash\" class=\"language-bash\">100-cloud.aws.ecs.cpuUtilizationByAccountIdClusterNameRegion:filter(\neq(\"dt.entity.cloud:aws:ecs:metric_streams:cluster\",\nCUSTOM_DEVICE-E360A9250EEF25BC)):splitBy(\n\"dt.entity.cloud:aws:ecs:metric_streams:cluster\"):avg<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Evaluation type<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Choose <strong>Aggregate<\/strong><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Entity selector<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Input <strong>entityid(&#8220;<em>CUSTOM_DEVICE-E360A9250EEF25BC<\/em><\/strong><strong>&#8220;)<\/strong> and replace <strong>&#8220;ECSCUSTOM_DEVICE-E360A9250EEF25BC<\/strong>&#8221; with actual id obtained from AWS Metrics Streams<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Timeframe during which the SLO is to be evaluated<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Input <strong>-1d<\/strong> to evaluate the SLO for the last 24 hours<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Target percentage<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Set the SLO target to <strong>20%<\/strong> compliance (i.e., available CPU capacity should stay above 20% for 99.9% of the time).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Warning percentage<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Set the warning to <strong>25%<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Review and Save<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Review your SLO configurations and save them. These SLOs will now be tracked over time, allowing you to evaluate how well your ECS cluster meets the defined objectives.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"903\" height=\"730\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/SLO1.png\" alt=\"\" class=\"wp-image-716\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO1.png 903w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO1-300x243.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO1-768x621.png 768w\" sizes=\"auto, (max-width: 903px) 100vw, 903px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"900\" height=\"818\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/SLO2-2.png\" alt=\"\" class=\"wp-image-717\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO2-2.png 900w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO2-2-300x273.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO2-2-768x698.png 768w\" sizes=\"auto, (max-width: 900px) 100vw, 900px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"898\" height=\"492\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/SLO3.png\" alt=\"\" class=\"wp-image-718\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO3.png 898w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO3-300x164.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/SLO3-768x421.png 768w\" sizes=\"auto, (max-width: 898px) 100vw, 898px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Step 3: Monitor and Analyze SLOs<\/strong><\/h4>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Visualize SLOs in Dashboards<\/strong>:\n<ul class=\"wp-block-list\">\n<li>Go to the <strong>Dashboards<\/strong> section in Dynatrace and create widgets that display the compliance rate of your CPU and memory SLOs over time.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Analyze Breaches<\/strong>:\n<ul class=\"wp-block-list\">\n<li>If your SLOs are not being met, investigate which ECS services are breaching the thresholds and what actions can be taken to improve performance.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"442\" src=\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/09\/dashboard-1-1024x442.png\" alt=\"\" class=\"wp-image-714\" srcset=\"https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/dashboard-1-1024x442.png 1024w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/dashboard-1-300x130.png 300w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/dashboard-1-768x332.png 768w, https:\/\/www.wallacel.com\/wp-content\/uploads\/2024\/09\/dashboard-1.png 1105w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">7. <strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">By following the steps outlined in this blog, you can set up a robust monitoring system that streams AWS CloudWatch Logs and Metrics into <strong>Dynatrace<\/strong> using <strong>Amazon Data Firehose<\/strong> and <strong>CloudWatch Metric Streams<\/strong>. More importantly, instead of relying solely on alerts, you can define and track <strong>Service Level Objectives (SLOs)<\/strong>, providing a clear, long-term view of how well your ECS cluster is performing in relation to the defined performance and reliability targets.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Thank you for reading my blog and I hope you like it!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Monitoring the performance of your AWS ECS (Elastic Container Service) cluster is crucial to ensure your application operates efficiently, especially under varying workloads. While AWS CloudWatch provides robust tools for tracking metrics, you can enhance your observability by integrating it with other APM tools like Dynatrace for more advanced dashboards, alerting, and root-cause analysis. This [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":731,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,70],"tags":[44,46,42,43],"class_list":["post-686","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","category-dynatrace","tag-amazon-data-firehose","tag-cloudwatch-metric-stream","tag-dynatrace","tag-ecs"],"_links":{"self":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/686","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/comments?post=686"}],"version-history":[{"count":22,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/686\/revisions"}],"predecessor-version":[{"id":1016,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/686\/revisions\/1016"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/media\/731"}],"wp:attachment":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/media?parent=686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/categories?post=686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/tags?post=686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}