{"id":962,"date":"2024-10-14T17:06:04","date_gmt":"2024-10-15T00:06:04","guid":{"rendered":"http:\/\/184.72.63.26\/?p=962"},"modified":"2024-12-09T22:04:54","modified_gmt":"2024-12-10T05:04:54","slug":"dynatrace-grail-a-revolutionary-data-platform-for-observability","status":"publish","type":"post","link":"https:\/\/www.wallacel.com\/index.php\/2024\/10\/14\/dynatrace-grail-a-revolutionary-data-platform-for-observability\/","title":{"rendered":"Dynatrace Grail: A Revolutionary Data Platform for Observability"},"content":{"rendered":"\n

In my previous blogs, I explored how to enhance observability in AWS environment by ingesting logs and metrics to Dynatrace. Now I will dive deeper into one of Dynatrace\u2019s most revolutionary features: Dynatrace Grail<\/strong>. This cutting-edge observability data lakehouse not only unifies logs, metrics, and traces into a single platform but also leverages AI-driven analytics<\/strong> to deliver real-time insights and automated root cause analysis. In this blog, I\u2019ll go beyond basic integrations and focus on the unique value Grail brings to observability, especially when compared to AWS CloudWatch and OpenSearch. I\u2019ll explore how Grail\u2019s Dynatrace Query Language (DQL)<\/strong>, real-time ingestion<\/strong>, and AI-powered automation<\/strong> can dramatically improve operational efficiency and reduce the time to resolve incidents.<\/p>\n\n\n\n

What is Dynatrace Grail?<\/strong><\/h3>\n\n\n\n

Dynatrace Grail is the backbone of Dynatrace\u2019s observability platform, designed to unify all your observability data\u2014logs, metrics, traces, events, and security information\u2014into a single, queryable data lakehouse. Unlike traditional log management tools that handle only logs, Grail integrates all observability data types<\/strong> into one system. This allows for real-time ingestion<\/strong>, instant querying<\/strong>, and AI-powered root cause analysis<\/strong>, all from a single platform.<\/p>\n\n\n\n

\"\"
Image source: https:\/\/dynatrace.com\/news\/blog\/new-approach-to-software-intelligence<\/em><\/figcaption><\/figure>\n\n\n\n

Key Features of Dynatrace Grail:<\/strong><\/p>\n\n\n\n

    \n
  1. Real-Time Ingestion:<\/strong> Seamlessly ingests logs, metrics, and traces from multiple sources in real time.<\/li>\n\n\n\n
  2. AI-Powered Insights:<\/strong> Uses Davis AI<\/strong> for automated anomaly detection and root cause analysis.<\/li>\n\n\n\n
  3. Querying and Analysis:<\/strong> Leverages the Dynatrace Query Language<\/strong> (DQL) to query across logs, metrics, and traces.<\/li>\n\n\n\n
  4. Scalable Log Analytics:<\/strong> Designed for dynamic, cloud-native environments, offering real-time scalability without manual reconfiguration.<\/li>\n<\/ol>\n\n\n\n

    Let\u2019s break down these features to have a deeper understanding of Dynatrace Grail with a comparison of similar services from AWS.<\/p>\n\n\n\n

    1. Real-Time Ingestion and Unified Observability<\/strong><\/h3>\n\n\n\n

    AWS CloudWatch:<\/strong><\/p>\n\n\n\n

      \n
    • CloudWatch ingests logs and metrics from AWS services, applications, and on-prem systems, but it splits metrics and logs into separate services (CloudWatch Logs and CloudWatch Metrics).<\/li>\n\n\n\n
    • While AWS CloudWatch is powerful for monitoring individual AWS resources, correlating logs and metrics often requires multiple tools and a fair amount of manual effort. For instance, you need to link CloudWatch with X-Ray<\/strong> for tracing, or with AWS Lambda<\/strong> for serverless monitoring.<\/li>\n<\/ul>\n\n\n\n

      Dynatrace Grail:<\/strong><\/p>\n\n\n\n

        \n
      • Grail ingests logs, metrics, and traces<\/strong> in real time and stores them in a unified data lake.<\/li>\n\n\n\n
      • With automatic correlation<\/strong> between these data types, Grail eliminates the need for manual integration between different monitoring services. For example, a spike in CPU usage is automatically linked to related logs and traces, giving you an immediate holistic view of the issue.<\/li>\n<\/ul>\n\n\n\n

        To see this in action, let’s deploy a Dynatrace OneAgent to an AWS EC2 instance to collect the host metrics using AWS CDK:<\/p>\n\n\n\n

        import * as cdk from 'aws-cdk-lib';\nimport { Stack, StackProps } from 'aws-cdk-lib';\nimport { Vpc, Instance, InstanceType, AmazonLinuxImage, AmazonLinuxGeneration, SecurityGroup, Subnet } from 'aws-cdk-lib\/aws-ec2';\nimport { Role } from 'aws-cdk-lib\/aws-iam';\nimport { Construct } from 'constructs';\nimport * as secretsmanager from 'aws-cdk-lib\/aws-secretsmanager';\nimport * as custom_resources from 'aws-cdk-lib\/custom-resources';\nimport * as iam from 'aws-cdk-lib\/aws-iam';\n\nexport class DynatraceOneAgentAndDashboardStack extends cdk.Stack {\n  constructor(scope: Construct, id: string, props?: StackProps) {\n    super(scope, id, props);\n\n    const region = \"us-west-2\"; \/\/ Replace with your region\n\n    \/\/ Import the existing VPC\n    const vpc = Vpc.fromLookup(this, 'vpc', {\n      vpcId: 'vpc-xxxxxxxxxxxxxxxxx', \/\/ Replace with your VPC ID\n    });\n\n    \/\/ Lookup the existing security group by ID\n    const securityGroup = SecurityGroup.fromSecurityGroupId(this, 'MySG', \n    'sg-xxxxxxxxxxxxxxxxx'); \/\/ Replace with your security group ID\n\n    \/\/ Use an existing IAM role for the EC2 instance\n    const role = Role.fromRoleArn(this, 'InstanceRole', 'arn:aws:iam::xxxxxxxxxxxx:instance-profile\/YourRoleName'); \/\/ Replace with your IAM role ARN\n\n    \/\/ Define the EC2 instance with Amazon Linux\n    const instance = new Instance(this, 'OneAgentInstance', {\n      vpc,\n      instanceType: new InstanceType('t2.micro'),\n      machineImage: new AmazonLinuxImage({\n        generation: AmazonLinuxGeneration.AMAZON_LINUX_2,\n      }),\n      securityGroup: securityGroup,\n      role: role,\n      vpcSubnets: {\n        subnets: [\n          Subnet.fromSubnetAttributes(this, 'MySubnet', {\n            subnetId: 'subnet-xxxxxxxxxxxxxxxxx', \/\/ Replace with your subnet ID\n            availabilityZone: 'us-west-2a',\n          }),\n        ],\n      },\n    });\n\n    \/\/ User Data script for installing Dynatrace OneAgent\n    const userDataScript = `\n    #!\/bin\/bash\n    yum update -y\n    yum install -y wget aws-cli\n    mkdir -p \/opt\/dynatrace\n    cd \/opt\/dynatrace\n  \n    # Retrieve Dynatrace download token from Secrets Manager and extract it using sed\n    DT_TOKEN=\\$(aws secretsmanager get-secret-value --secret-id dynatrace-secret --query SecretString --output text --region us-west-1 | sed 's\/.*\"token\":\"\\\\([^\"]*\\\\)\".*\/\\\\1\/')\n  \n    # Proceed with the download and installation\n    wget -O Dynatrace-OneAgent-Linux-1.299.45.20240924-123410.sh \"https:\/\/your-dynatrace-url.com\/api\/v1\/deployment\/installer\/agent\/unix\/default\/latest?arch=x86\" --header=\"Authorization: Api-Token \\$DT_TOKEN\"\n    sudo \/bin\/bash Dynatrace-OneAgent-Linux-1.299.45.20240924-123410.sh --set-monitoring-mode=fullstack --set-app-log-content-access=true --set-host-group=my-host-group --set-host-tag=environment:prod\n    `;\n\n    instance.addUserData(userDataScript);\n\n    new cdk.CfnOutput(this, 'InstanceId', {\n      value: instance.instanceId,\n      description: 'The ID of the EC2 instance running Dynatrace OneAgent',\n    });\n  }\n}\n<\/code><\/pre>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        Once the OneAgent is deployed, it automatically discovers and collects all relevant monitoring data like cpu usage, network health, processes, and services on your host.<\/p>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        Navigate to Smartscape<\/strong> Topology<\/strong> in the Dynatrace UI to view your AWS host, processes and its dependencies in real time, without manual configuration.<\/p>\n\n\n\n
        \"\"<\/figure>\n\n\n\n
        2. AI-Powered Root Cause Analysis with Davis AI<\/strong><\/h3>\n\n\n\n
        AWS CloudWatch:<\/strong> <\/p>\n\n\n\n
          \n
        • In AWS, you can use CloudWatch alarms<\/strong> and basic anomaly detection to alert you when metrics deviate from thresholds. However, identifying the root cause still involves manually piecing together data from CloudWatch Logs<\/strong>, X-Ray<\/strong>, and other AWS services.<\/li>\n<\/ul>\n\n\n\n
          With Dynatrace Grail<\/strong>:<\/p>\n\n\n\n
            \n
          • Dynatrace\u2019s Davis AI<\/strong> goes beyond basic alerting by automatically correlating data across logs, metrics, and traces to identify the root cause<\/strong> of an issue. This is a massive time-saver, especially in complex environments where problems often span multiple services.<\/li>\n<\/ul>\n\n\n\n
            To simulate a performance issue in my application, I will run a simple python script (cpu.py<\/strong>) to stress the CPU usage to 90% on a EC2 instance:<\/p>\n\n\n\n
            import multiprocessing\nimport time\n\ndef cpu_load():\n    while True:\n        # Busy-wait for 0.9 seconds (90% CPU usage)\n        end = time.time() + 0.9\n        while time.time() < end:\n            pass\n        # Sleep for 0.1 seconds (10% idle)\n        time.sleep(0.1)\n\nif __name__ == \"__main__\":\n    # Run the load on multiple CPU cores (adjust as necessary)\n    for i in range(multiprocessing.cpu_count()):\n        process = multiprocessing.Process(target=cpu_load)\n        process.start()\n<\/code><\/pre>\n\n\n\n
            Open Dynatrace Hosts Classic and after awhile the Davis AI will automatically detect the anomaly. It provides a root cause analysis<\/strong>, showing the metric and host that contributed to the issue.<\/p>\n\n\n\n
            \"\"<\/figure>\n\n\n\n
            From the same page, we can easily identify the python script ‘cpu.py’ is causing the high CPU usage. Davis AI automates what would otherwise require hours of manual log parsing and metric correlation and drastically reducing your MTTR (Mean Time to Resolution)<\/strong>.<\/p>\n\n\n\n
            \"\"<\/figure>\n\n\n\n
            3. Dynatrace Query Language (DQL): Unlocking the Power of Unified Data<\/strong><\/h3>\n\n\n\n
            AWS CloudWatch:<\/strong><\/p>\n\n\n\n
            When using AWS CloudWatch Logs, you have CloudWatch Insights<\/strong> for querying logs, CloudWatch Metrics<\/strong> for querying metrics, and X-Ray<\/strong> for traces. Each of these requires its own interface and query language, making cross-data queries difficult.<\/p>\n\n\n\n
            Dynatrace Grail:<\/strong><\/p>\n\n\n\n
            Dynatrace introduces DQL (Dynatrace Query Language)<\/strong>, a single language to explore, query and process all data persisted in Dynatrace Grail. DQL is a pipeline based data processing language where you can define a set of commands that follow each other where the data is processed step by step. Each command returns an output containing a set of records and that output becomes the input of the next command. You continue this process until you are satisfied with the analysis results.<\/p>\n\n\n\n
            \"\"
            Image source: https:\/\/docs.dynatrace.com\/docs\/platform\/grail\/dynatrace-query-language\/dql-guide<\/em><\/figcaption><\/figure>\n\n\n\n
            Let’s do a simple DQL to fetch the number of problem events that happened over time for the aforementioned EC2 host. First, I go to Smartscape Topology to find out the host id from the URL:<\/p>\n\n\n\n
            \"\"<\/figure>\n\n\n\n
             Then go to Notebook and create a DQL. This query fetches the number of problematic events that happened on my EC2 host with id HOST-480E4F2A782B08B6<\/strong> and their status. With the visualization option, I can choose to visualize the results in a bar chart.<\/p>\n\n\n\n
            fetch events\n| filter event.kind == \"DAVIS_PROBLEM\" and in(affected_entity_ids, \"HOST-480E4F2A782B08B6\")\n| summarize count = count(), by: {bin(timestamp, 15min), event.status}<\/code><\/pre>\n\n\n\n
            \"\"<\/figure>\n\n\n\n
            4. Scalable Log Analytics Without Indexes or Schemas<\/strong><\/h3>\n\n\n\n
            AWS OpenSearch:<\/strong> <\/p>\n\n\n\n
            In a log management system like AWS OpenSearch<\/strong>, data ingestion requires effort in setup to achieve efficient querying and analysis. OpenSearch is based on Elasticsearch and uses indexing<\/strong> and schema definitions<\/strong> to store and organize data. While OpenSearch is a powerful, fully managed service, users must often manually configure index mappings<\/strong> and schemas<\/strong> to optimize search performance and ensure logs, metrics, and traces are stored in the correct format.<\/p>\n\n\n\n
              \n
            • Indexing:<\/strong> OpenSearch requires building indexes to structure and store data. These indexes are created based on the schema you define, which must be updated as new data sources or formats are introduced.<\/li>\n\n\n\n
            • Schema Definition:<\/strong> In OpenSearch, defining the correct schema upfront is important to ensure that fields are correctly indexed for efficient querying. If the schema isn\u2019t set properly, queries can become slow or return incomplete results.<\/li>\n\n\n\n
            • Limitations:<\/strong> While AWS OpenSearch is fully managed, managing indexes and schemas still requires significant planning and ongoing maintenance. This can be cumbersome in dynamic, cloud-native environments where services scale frequently, and new data types constantly emerge.<\/li>\n<\/ul>\n\n\n\n
              Dynatrace Grail:<\/strong> <\/p>\n\n\n\n
              Dynatrace Grail eliminates the need to build indexes<\/strong> or define schemas<\/strong> manually. It offers schema-on-read<\/strong> functionality, allowing you to ingest logs, metrics, and traces and immediately query the data without the overhead of defining how that data should be stored. This makes Grail a more scalable<\/strong> and flexible<\/strong> solution, especially in environments where data structures frequently change.<\/p>\n\n\n\n
              \"\"
              Image source: https:\/\/www.dynatrace.com\/platform\/log-management-analytics<\/em><\/figcaption><\/figure>\n\n\n\n
                \n
              1. No Indexing or Schema Definition Required:<\/strong>\n
                  \n
                • Unlike OpenSearch, where users must predefine indexes and schemas, Grail dynamically understands and organizes your observability data (logs, metrics, and traces) as it ingests it. This makes it easier to handle new data types and instantly start querying without worrying about the underlying data structure.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                • Real-Time Data Ingestion:<\/strong>\n
                    \n
                  • With Grail, data is immediately available<\/strong> for querying upon ingestion. You don\u2019t need to wait for indexing jobs to finish, which can be a bottleneck in OpenSearch when dealing with high data volumes or complex queries.<\/li>\n\n\n\n
                  • Whether you’re ingesting logs from microservices or metrics from infrastructure, Grail ensures that you can access and query this data in real-time<\/strong>, enabling faster troubleshooting and performance analysis.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                  • Unified Data Ingestion:<\/strong>\n
                      \n
                    • Grail unifies logs, metrics, and traces into one Data lakehouse<\/strong>, allowing for cross-data querying without needing separate indexes for each data type. OpenSearch, on the other hand, requires separate index setups for each data type, with separate mapping and storage requirements for logs, metrics, and traces.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                    • Seamless Scalability:<\/strong>\n
                        \n
                      • Grail is built to scale automatically with your environment. As your data grows, the platform seamlessly adapts, providing fast query performance<\/strong> without manual reconfiguration. <\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n
                        Let\u2019s use DQL<\/strong> to demonstrate how to query logs in JSON format<\/strong> and extract specific attributes, such as the user\u2019s question and the chatbot’s answer, without the need to build indexes or define schemas. In this example, I’ve already ingested the logs from my chatbot Lambda function<\/strong> into Dynatrace. These logs capture the interaction between users and the chatbot, specifically showing what users ask and how the chatbot responds.<\/p>\n\n\n\n
                        fetch logs\n| filter matchesValue(aws.log_group, \"\/aws\/lambda\/chatbot\")\n| sort timestamp desc<\/code><\/pre>\n\n\n\n
                        \"\"<\/figure>\n\n\n\n
                        \"\"<\/figure>\n\n\n\n
                        Since the content is in JSON format, I can parse the content using \"json:chat\"<\/code> and extract only the attributes question<\/strong> and answer<\/strong> I am interested by creating objects representing those attributes like chat[question] <\/strong>and chat[answer]<\/strong>. The parse json<\/code> command dynamically understands the structure of the log data so you don’t have to define a schema upfront.<\/p>\n\n\n\n
                        fetch logs\n| filter matchesValue(aws.log_group, \"\/aws\/lambda\/chatbot\")\n| parse content, \"json:chat\"\n| fields timestamp, question=chat[question], answer=chat[answer]\n| filter isNotNull(question)\n| sort timestamp desc<\/code><\/pre>\n\n\n\n
                        \"\"<\/figure>\n\n\n\n
                        \"\"<\/figure>\n\n\n\n
                        Final Thoughts<\/h2>\n\n\n\n
                        Dynatrace Grail offers a powerful solution for modern cloud observability, enabling organizations to unlock deep insights from large volumes of unstructured log data. Its ability to index, store, and query logs in real time, combined with AI-powered analytics, streamlines troubleshooting and enhances decision-making. By integrating seamlessly with existing cloud environments and providing actionable intelligence across applications, infrastructure, and user experiences, Grail helps teams reduce downtime, optimize performance, and gain full visibility into their systems. Its flexibility and performance make it a key tool for driving efficiency and innovation in dynamic, cloud-native environments.<\/p>\n\n\n\n
                        Thank you for reading my blog and I hope you like it!<\/p>\n","protected":false},"excerpt":{"rendered":"
                        In my previous blogs, I explored how to enhance observability in AWS environment by ingesting logs and metrics to Dynatrace. Now I will dive deeper into one of Dynatrace\u2019s most revolutionary features: Dynatrace Grail. This cutting-edge observability data lakehouse not only unifies logs, metrics, and traces into a single platform but also leverages AI-driven analytics […]<\/p>\n","protected":false},"author":1,"featured_media":1000,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,70],"tags":[71,64,65,63,42,62],"class_list":["post-962","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","category-dynatrace","tag-aws-cdk","tag-data-lakehouse","tag-davis-ai","tag-dql","tag-dynatrace","tag-grail"],"_links":{"self":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/962","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/comments?post=962"}],"version-history":[{"count":32,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/962\/revisions"}],"predecessor-version":[{"id":1033,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/962\/revisions\/1033"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/media\/1000"}],"wp:attachment":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/media?parent=962"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/categories?post=962"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/tags?post=962"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}