Kubernetes is the favored option for developing cloud-native / containerized applications due to its scalability, portability, and availability. Additionally, it offers other benefits, including:

• Automated Deployment and Scaling: Kubernetes provides the capability to automatically deploy and scale applications in response to demand.
• Self-healing: Kubernetes has the ability to self-heal, ensuring your applications / services remain continuously available.
• Load Balancing: Kubernetes distributes traffic among multiple containers, preventing any single container from becoming overloaded.
• Storage Orchestration: Kubernetes manages persistent data by automatically mounting storage volumes to containers, simplifying data management.
• Security: Kubernetes offers a range of security features, including Role-Based Access Control and network policies.
• Multi-Cloud Support: Kubernetes is an ideal choice for organizations aiming for cloud-agnostic solutions, providing compatibility across different cloud environments.

Understand the Tooling

Prometheus is an open-source monitoring system (originally developed by SoundCloud), which is widely adopted for monitoring cloud-native applications due to its scalability, reliability, and user-friendliness.

Prometheus gathers metrics from multiple sources, including applications, services, and infrastructure components, and stores them in a time series database. This allows Prometheus to maintain a record of system performance.

Prometheus utilizes software programs known as exporters. These exporters collect metrics from various sources and provide them to Prometheus. When new exporter are detected, Prometheus initiates the collection of metrics from that source, storing them in its database. Further configuration options enable Prometheus to scrape metrics from specific endpoints.

• Scalability: Prometheus is crafted for scalability, making it suitable for monitoring multiple systems.
• Reliability: Prometheus is recognized for its reliability.
• User-Friendliness: Prometheus is known for its ease of use and straightforward configuration.
• Open Source: Being an open-source project, Prometheus is freely available for use and can be modified to meet specific requirements.

Prerequisites

You will need an Azure account, the Azure CLI, Kubectl and Helm.

• Azure account
• Azure CLI
• Kubectl
• Helm

You can install the latest versions of Kubectl and Helm using the Azure CLI or install them manually.

Install the CLI tools on your local machine since you will need to forward a local port to access both the Prometheus and Grafana web interfaces.

Create an Azure Kubernetes Service (AKS) Cluster

Sign into the Azure CLI by running the login command.

az login

Install or update kubectl.

az aks install-cli

Create two bash variables which we will use in subsequent commands. (You may need to change the syntax below if you are using another shell.)

RESOURCE_GROUP=aks-prometheus
AKS_NAME=aks1

Create a resource group. I have chosen to create this in the eastus2 Azure region.

az group create --name $RESOURCE_GROUP --location eastus2

Create a new AKS cluster using the az aks create command. Here we create a 3-node cluster using the B-series Burstable VM type which is cost-effective and suitable for this test.

az aks create --resource-group $RESOURCE_GROUP \
  --name $AKS_NAME \
  --node-count 3 \
  --node-vm-size Standard_B2s \
  --generate-ssh-keys

Authenticate to the cluster we have just created.

az aks get-credentials \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_NAME

We can now access our Kubernetes cluster with kubectl. Use kubectl to see the nodes we have just created.

kubectl get nodes

Install Grafana and Prometheus

Prometheus can be installed through the official operator or by leveraging Helm. I’ll use the Helm chart.

Add its repository to our repository list and update it.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install the Helm chart into a namespace called monitoring, which will be created automatically.

helm install prometheus \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

The helm command will ask you to check on the status of the deployed pods.

kubectl --namespace monitoring get pods -l "release=prometheus"

Ensure all pods are in the “Running” state before continuing.

Explore the Prometheus and Grafana Web UI

By default, all the monitoring options for Prometheus will be enabled.

Create a port forward to access the Prometheus interface.

kubectl port-forward --namespace monitoring svc/prometheus-kube-prometheus-prometheus 9090

Access http://localhost:9090 via your web browser and access the user interface to examine the raw metrics within Prometheus.

The default username for Grafana is admin and the default password is prom-operator (You can change it in the Grafana UI later).

Note: For security reasons, you should avoid exposing your Prometheus or Grafana endpoints to the public internet using a Service or Ingress.

Since AKS operates as a managed Kubernetes service, it restricts access to internal components like the etcd store, the controller manager, the scheduler, etc. Attempting to pull metrics from these components within the cluster is not necessary. Therefore, I am disabling this option by upgrading our Prometheus release.

helm upgrade prometheus \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set kubeEtcd.enabled=false \
  --set kubeControllerManager.enabled=false \
  --set kubeScheduler.enabled=false

This ensures that resources are not wasted in attempting to retrieve its metrics.

Note: In the case of running an outdated Kubernetes version, you may need to deactivate the HTTPS metrics served from the kubelet as they are exposed over HTTP. To do this, you will need to set the kubelet.serviceMonitor.https parameter in the helm chart to false.

helm upgrade prometheus \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set kubeEtcd.enabled=false \
  --set kubeControllerManager.enabled=false \
  --set kubeScheduler.enabled=false \
  --set kubelet.serviceMonitor.https=false

To clean up Azure resources and prevent further charges, execute the following command, which will delete all items within your resource group.

az group delete --name $RESOURCE_GROUP

Keep learning!