2. Why Use Hipster Shop as a Demo?<\/strong><\/h3>\n\n\n\nHipster Shop<\/strong> is a microservices-based application designed by Google, simulating an e-commerce platform. It consists of several services (e.g., emailservice<\/code>, checkoutservice<\/code>, cartservice<\/code>, etc.), making it an ideal reference for:<\/p>\n\n\n\n\nMicroservices Monitoring<\/strong>: Hipster Shop\u2019s architecture mirrors real-world applications with interdependent services.<\/li>\n\n\n\nLoad Testing<\/strong>: The demo includes a loadgenerator<\/code> to simulate realistic traffic.<\/li>\n\n\n\nDynatrace Integration<\/strong>: The demo supports full observability with tools like Dynatrace to monitor, analyze, and troubleshoot services.<\/li>\n<\/ul>\n\n\n\n\n\t\t\t\n\t\t\t\tHipster shop demo application’s microservice architecture<\/em><\/figcaption><\/figure>\n\n\n\n3. Setting Up Google Kubernetes Engine (GKE)<\/strong><\/h3>\n\n\n\nPrerequisite:<\/h4>\n\n\n\n Ensure the following tools are installed on your machine:<\/p>\n\n\n\n
\nkubectl<\/li>\n\n\n\n gcloud<\/li>\n\n\n\n git<\/li>\n\n\n\n helm<\/li>\n\n\n\n jq<\/li>\n<\/ul>\n\n\n\nStep 1: Create a GKE Cluster<\/h4>\n\n\n\n Start by creating a GKE cluster via the Google Cloud Console or CLI:<\/p>\n\n\n\n
gcloud container clusters create hipster-shop-cluster \\\n --zone us-west1-a \\\n --num-nodes 2 \\\n --machine-type e2-standard-8\n<\/code><\/code><\/pre>\n\n\n\nThis sets up a Kubernetes cluster with two nodes, each configured for microservices deployment.<\/p>\n\n\n\n4. Deploying Hipster Shop on GKE<\/strong><\/h3>\n\n\n\nClone the Hipster Shop Repository<\/strong>:<\/p>\n\n\n\nhttps:\/\/github.com\/GoogleCloudPlatform\/microservices-demo.git<\/code><\/code><\/pre>\n\n\n\nDeploy Hipster Shop to GKE<\/strong>:<\/p>\n\n\n\nkubectl apply -f .\/release\/kubernetes-manifests.yaml -n hipster-shop<\/span><\/code><\/code><\/pre>\n\n\n\nThis command deploys each microservice along with their configurations, such as services and deployments, into the GKE cluster.<\/p>\n\n\n\n
Verify the pods are ready<\/strong>:<\/p>\n\n\n\nkubectl get pods -n hipster-shop<\/span><\/code><\/code><\/pre>\n\n\n\nAfter a few minutes, you should see the pods are in a running state.<\/p>\n\n\n\n
NAME READY STATUS RESTARTS AGE\nadservice-86d5fd9c56-tprp6 1\/1 Running 0 75s\ncartservice-6bb7677d47-lv25q 1\/1 Running 0 78s\ncheckoutservice-64fd47cdfb-ldm8w 1\/1 Running 0 81s\ncurrencyservice-8587877797-q24bg 1\/1 Running 0 77s\nemailservice-78d967474f-bdl2k 1\/1 Running 0 82s\nfrontend-84fb499859-f66dw 1\/1 Running 0 79s\npaymentservice-c5fb8bfb6-6cvs8 1\/1 Running 0 79s\nproductcatalogservice-64cffd9d98-7vdhl 1\/1 Running 0 78s\nrecommendationservice-549cc495d-285w5 1\/1 Running 0 80s\nredis-cart-867cd85fd4-l6zkm 2\/2 Running 0 76s\nshippingservice-685bc78f7b-jjg9z 1\/1 Running 0 77s<\/code><\/code><\/pre>\n\n\n\nExpose the Frontend<\/strong>:<\/p>\n\n\n\nhelm upgrade --install ingress-nginx ingress-nginx --repo https:\/\/kubernetes.github.io\/ingress-nginx --namespace ingress-nginx --create-namespace\n<\/code><\/code><\/pre>\n\n\n\nThis command installs the NGINX Ingress Controller as a load balancer to exposes the frontend of the Hipster Shop application.<\/p>\n\n\n\n
Get the URL to your deployed Hipster Shop<\/strong>:<\/p>\n\n\n\nkubectl get svc ingress-nginx-controller -n ingress-nginx -ojson | jq -j '.status.loadBalancer.ingress[].ip'\n<\/code><\/code><\/pre>\n\n\n\n5. Implementing Log Analytics in GKE<\/strong><\/h3>\n\n\n\nThere are two ways you can enable Log Analytics to get insights into your GKE cluster. One way is to upgrade an existing bucket. The other is to create a new log bucket with Log Analytics enabled.<\/p>\n\n\n\n
Step 1: Create a new log bucket<\/h4>\n\n\n\n You can configure Cloud Logging to create a new log bucket with Log Analytics enabled. <\/p>\n\n\n\n
\nNavigate to Logging, <\/strong>then click Logs Storage<\/strong> in the Cloud Console<\/li>\n\n\n\nClick CREATE LOG BUCKET<\/strong> at top, name the log bucket as hipstershop_log<\/strong><\/li>\n\n\n\nCheck both Upgrade to use Log Analytics<\/strong> and Create a new BigQuery dataset that links to this bucket<\/strong><\/li>\n\n\n\nType in a dataset name like hipstershop_dataset<\/strong><\/li>\n\n\n\nClick Create bucket<\/strong> to create the log bucket<\/li>\n<\/ol>\n\n\n\nStep 2: Write to the new log bucket<\/h4>\n\n\n\n To write logs from our GKE cluster to the new log bucket, you need to create a log sink<\/strong> and specify the log destination and setup a log filter for logging the gke cluster.<\/p>\n\n\n\nNavigate to Logging, <\/strong>then click Logs Explorer<\/strong> in the Cloud Console, and enable Show query <\/strong>on the top right and add the following filter.<\/p>\n\n\n\nresource.type=\"k8s_container\"<\/code><\/pre>\n\n\n\nClick on Actions<\/strong> and then Create sink<\/strong>. <\/p>\n\n\n\n\nName the sink as hipstershop_sink<\/strong><\/li>\n\n\n\nSelect Logging bucket<\/strong> in the sink service, then choose hipstershop_log<\/strong> bucket<\/li>\n\n\n\nClick CREATE SINK<\/strong><\/li>\n<\/ul>\n\n\n\nStep 3: Perform analytics on logs<\/h4>\n\n\n\n Now we can perform analytics on logs stored in the hipster shop log bucket by running our own query in Log Analytics<\/strong>. In the Log views<\/strong> list, select the hipstershop_log._AllLogs<\/strong>, and then select Query<\/strong>. The Query<\/strong> pane is populated with a default query, which includes the log view that is queried.<\/p>\n\n\n\nTo find the min, max and average latency (in ms) for the frontend service:<\/strong><\/p>\n\n\n\nSELECT hour,\nMIN(took_ms) AS min,\nMAX(took_ms) AS max,\nAVG(took_ms) AS avg\nFROM ( SELECT\n FORMAT_TIMESTAMP(\"%H\", timestamp) AS hour,\n CAST( JSON_VALUE(json_payload,\n '$.\"http.resp.took_ms\"') AS INT64 ) AS took_ms\nFROM `hipster-shop-440423.global.hipstershop_log._AllLogs`\nWHERE\n json_payload IS NOT NULL\n AND SEARCH(labels,\"frontend\")\n AND JSON_VALUE(json_payload.message) = \"request complete\"\nORDER BY took_ms DESC, timestamp ASC )\nGROUP BY 1<\/code><\/pre>\n\n\n\nTo find all sessions with shopping cart checkout:<\/strong><\/p>\n\n\n\nSELECT\n JSON_VALUE(json_payload.session) as session_id ,COUNT(*) as count\nFROM `hipster-shop-440423.global.hipstershop_log._AllLogs`\nWHERE\n JSON_VALUE(json_payload['http.req.method']) = \"POST\"\n AND JSON_VALUE(json_payload['http.req.path']) = \"\/cart\/checkout\"\nGROUP BY\n JSON_VALUE(json_payload.session)<\/code><\/pre>\n\n\n\n6. Monitoring your GKE cluster health with Dashboards and Alert<\/strong><\/h3>\n\n\n\nTo effectively monitor your GKE cluster and observe how it handles the load generated by the Hipster Shop demo application, you can set up a custom Google Cloud Monitoring dashboard to display key metrics such as CPU Request Utilization<\/strong> or Memory Limit Utilization<\/strong>, giving you insights into resource usage. For instance, we want to monitor the CPU utilization because if it consistently nears or exceeds 100%, our cluster may be at risk of CPU throttling, which can degrade application performance.<\/p>\n\n\n\nStep 1: Create a Custom Dashboard to monitor your pods<\/h4>\n\n\n\n\nGo to the Google Cloud Console and open Monitoring<\/strong><\/li>\n\n\n\nSelect Metrics Explorer<\/strong> from the left menu and click on Select a metric<\/strong><\/li>\n\n\n\nSelect Kubernetes Container > Popular Metrics > CPU request utilization<\/strong><\/li>\n\n\n\nClick Apply<\/strong><\/li>\n\n\n\nNow click the Save Chart<\/strong> button and name the dashboard as K8s Container Dashboard<\/strong><\/li>\n<\/ol>\n\n\n\nStep 2: Create an Alert to Identify Incident<\/h4>\n\n\n\n We will create an alert policy to detect high CPU utilization among the containers and send notifications when the CPU usage exceeds a specific threshold, indicating potential resource constraints. Then we can use the dashboard to identify and respond to the incident.<\/p>\n\n\n\n
\nIn the Cloud Console, navigate to Monitoring > Alerting<\/strong><\/li>\n\n\n\nClick + Create Policy<\/strong><\/li>\n\n\n\nClick on Select a metric<\/strong> dropdown<\/li>\n\n\n\nUncheck the Active<\/strong> option<\/li>\n\n\n\nClick on Kubernetes Container > Container<\/strong>->CPU request utilization<\/strong>, then click Apply<\/strong><\/li>\n\n\n\nSet Rolling windows<\/strong> to 1 min, click Next<\/strong><\/li>\n\n\n\nSet Threshold position to Above Threshold<\/strong><\/li>\n\n\n\nChoose Threshold<\/strong> and define a Threshold value<\/strong> (e.g., 0.8 for 80% utilization)<\/li>\n\n\n\nUnder Advanced Options<\/strong>, set Retest window<\/strong> to 5 min<\/strong>. By using a retest window, you can reduce the number of unnecessary alerts that might occur due to temporary anomalies in metrics. Click Next<\/strong><\/li>\n\n\n\nConfigure Notification Channels<\/strong> – Choose how you’d like to be notified (e.g., email, Slack, SMS), name the alert policy and then click Create Policy<\/strong><\/li>\n<\/ol>\n\n\n\nStep 3: Monitor and Respond<\/h4>\n\n\n\n With the dashboard and alert policy in place, you can now:<\/p>\n\n\n\n
\nView real-time insights<\/strong> on CPU usage and active users.<\/li>\n\n\n\nReceive alerts<\/strong> when CPU usage exceeds the set threshold, allowing you to investigate and address potential issues before they impact the application\u2019s performance.<\/li>\n<\/ul>\n\n\n\n7. Load Testing and Stress Analysis<\/strong><\/h3>\n\n\n\nWe will use the loadgenerator<\/code> included in the Hipster Shop demo application to simulate some user traffic.<\/p>\n\n\n\nUpdate Kubernetes Configuration for the Loadgenerator<\/strong><\/p>\n\n\n\nIn k8s-manifest.yaml<\/code>, you can adjust how the loadgenerator runs in your cluster:<\/p>\n\n\n\n\nConcurrency and Duration<\/strong>: Define how many users and how long they should interact with the app.<\/li>\n\n\n\nResource Limits<\/strong>: Adjust resource requests\/limits to control how much CPU and memory the loadgenerator uses.<\/li>\n<\/ul>\n\n\n\napiVersion: batch\/v1\nkind: CronJob\nmetadata:\n name: loadgeneratorservice\n labels:\n app.kubernetes.io\/component: loadgenerator\n app.kubernetes.io\/part-of: hipster-shop\n app.kubernetes.io\/name: loadgenerator\nspec:\n schedule: \"*\/15 * * * *\" # Runs every 15 minutes\n jobTemplate:\n spec:\n template:\n spec:\n restartPolicy: OnFailure\n containers:\n - name: loadgenerator\n image: loadgenerator-image\n args: [\"-u\", \"50\", \"-d\", \"10m\"]<\/code><\/pre>\n\n\n\n\nSchedule<\/strong>: \"*\/15 * * * *\"<\/code> triggers the CronJob every 15 minutes.<\/li>\n\n\n\nNumber of virtual users<\/strong> (-u <\/code>50) is set to 50. For heavier load, increase this value.<\/li>\n\n\n\nDuration<\/strong>: Set the duration (-d 10m<\/code>) in the args<\/code> to 10 minutes to allow a short break between runs. This prevents overlap between consecutive jobs and allows each job to finish before the next one starts<\/li>\n<\/ul>\n\n\n\nAfter configuring, you can deploy the modified loadgenerator to your GKE cluster:<\/p>\n\n\n\n
kubectl apply -f k8s-manifest.yaml -n hipster-shop<\/code><\/pre>\n\n\n\nThis will apply a load to the web store and after a while we will start receiving alert notifications. We can open the incident summary and respond to it by clicking on Acknowledge Incident<\/strong>. The incident status now shows Acknowledged, but that doesn’t solve the problem. In reality, you need to fix the root cause of the problem.<\/p>\n\n\n\n10. Conclusion<\/strong><\/h3>\n\n\n\nMonitoring and log analytics are essential for managing complex microservices applications in Kubernetes. Using Google Log Analytics and Monitoring<\/strong>, you can ensure comprehensive observability for your GKE cluster and container application. But monitoring doesn\u2019t stop at metrics; actionable insights into logs and traces help you detect and resolve issues faster, ensuring your applications are always running smoothly. <\/p>\n\n\n\nThank you for reading my blog and I hope you like it!<\/p>\n","protected":false},"excerpt":{"rendered":"
This blog will guide you through setting up log analytics, monitoring and alert for a Google Kubernetes Engine (GKE) cluster using the Hipster Shop Observability Demo Application as reference. 1. Introduction to Kubernetes Monitoring and Log Analytics Monitoring and log analytics in Kubernetes are essential for ensuring: Effective monitoring and logging provide insights into application […]<\/p>\n","protected":false},"author":1,"featured_media":1076,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[69],"tags":[66,68,23,67],"class_list":["post-1034","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-gcp","tag-gke","tag-hipster-shop","tag-kubernetes","tag-log-analytics"],"_links":{"self":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/1034","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/comments?post=1034"}],"version-history":[{"count":32,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/1034\/revisions"}],"predecessor-version":[{"id":1126,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/posts\/1034\/revisions\/1126"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/media\/1076"}],"wp:attachment":[{"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/media?parent=1034"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/categories?post=1034"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wallacel.com\/index.php\/wp-json\/wp\/v2\/tags?post=1034"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"http:\/\/184.72.63.26\/wp-content\/uploads\/2024\/11\/hipster-shop-1024x454-1.png\")