{"id":783,"date":"2024-09-17T14:39:36","date_gmt":"2024-09-17T21:39:36","guid":{"rendered":"http:\/\/184.72.63.26\/?p=783"},"modified":"2024-10-01T13:02:01","modified_gmt":"2024-10-01T20:02:01","slug":"enhancing-observability-with-aws-opensearch","status":"publish","type":"post","link":"https:\/\/www.wallacel.com\/index.php\/2024\/09\/17\/enhancing-observability-with-aws-opensearch\/","title":{"rendered":"Enhancing Observability with AWS OpenSearch"},"content":{"rendered":"\n

Introduction<\/h3>\n\n\n\n

Observability is a key pillar in Site Reliability Engineering (SRE) that involves monitoring systems, identifying anomalies, troubleshooting issues, and ensuring seamless operations of services. SREs strive to maintain a balance between deploying new features rapidly and maintaining system reliability. To achieve this, observability tools help gather data, metrics, and logs from different layers of the system.<\/p>\n\n\n\n

Observability includes metrics, logging, and tracing to provide deep insights into application health and performance. One of the robust tools for achieving this in the AWS cloud environment is OpenSearch (formerly Elasticsearch Service). AWS OpenSearch is an analytics suite that includes a search engine, a data visualization tool (Kibana, rebranded as OpenSearch Dashboards), and a RESTful API for ingesting, searching, and analyzing data. SREs can use AWS OpenSearch to centralize their logs, metrics, and traces for an integrated observability solution.<\/p>\n\n\n\n

Why Use AWS OpenSearch for Observability?<\/h3>\n\n\n\n
    \n
  1. Centralized Log Management<\/strong>: OpenSearch allows the collection and centralization of logs from multiple sources, making it easy to search and analyze data for troubleshooting.<\/li>\n\n\n\n
  2. Real-time Analytics<\/strong>: Ingest data in near-real-time to monitor system performance and identify issues quickly.<\/li>\n\n\n\n
  3. Scalability<\/strong>: OpenSearch is fully managed on AWS, scaling to handle large volumes of data from diverse sources.<\/li>\n\n\n\n
  4. Visualization<\/strong>: OpenSearch Dashboards provide powerful visualization capabilities for creating graphs, charts, and dashboards, offering an intuitive way to interpret data.<\/li>\n<\/ol>\n\n\n\n

    Setting Up AWS OpenSearch to Ingest VPC Flow Logs<\/strong><\/p>\n\n\n\n

    In this blog, I will explore how AWS OpenSearch can enhance observability by ingesting VPC Flow Logs. VPC Flow Logs provide network traffic insights within your AWS environment, making them a crucial data source to monitor security and network performance.<\/p>\n\n\n\n

    Step 1: Set Up an AWS OpenSearch Domain<\/h3>\n\n\n\n

    For the purpose of evaluation, I will create a single node domain using t3.small.search<\/strong> instance with 10GB<\/strong> EBS storage that is within the AWS free tier.<\/p>\n\n\n\n

      \n
    1. Create an OpenSearch Domain:<\/strong>\n