Context

GitOps part of the asynchronous encryption-decryption application. The application code is hosted in a separate repository which has a CI pipeline set-up (based on GitLab hosted runner) that performs two tasks upon a commit to master

• Run the unit and integration tests
• Upon successful completion of the test, pushes images to the relevant Dockherhub registries

Goal

Monitor the following items

• this Git repository (for push to master)
• two relevant dockerhub registries (for push to :latest tag) Upon any of these two events, redeploy the application.

Non Goals

• Any Git write back, or automatic modification of the manifest itself
• Provisioning of K8s cluster

Commands

We are using ArgoCD and Keel for the continuous deployment pipeline. The following commands must be run in sequence to get it set up. It was run in an Ubuntu 22.04 environment with the relevant application binaries (installed via apt-get or snap set up)

# Optional. Not necessary if kubectl is preconfigured to use 
# an existing cluster
minikube delete
minikube start --nodes 4 # Cluster of a master node and three workers
kubectl create namespace argocd
# Following line gets the argocd set up
kubectl apply --namespace argocd --filename https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# The Helm set up is necessary to set up the Dockerhub monitoring with Keel
helm repo add keel https://charts.keel.sh 
helm repo update
helm upgrade --install keel --namespace=kube-system keel/keel # The namespace is important to retrieve the logs later
sleep 2m # Allow enough time for argocd services to fire up
# This to access the argocd service via localhost:8080
kubectl port-forward svc/argocd-server --namespace argocd 8000:443 # 8000:local port, 443: Service port of argocd

The default ArgoCD username is admin. The password is auto-generated randomly each time, and can be found by running

argocd admin initial-password --namespace argocd # To find password

Once we have the password, we can log-in to the web interface by directing our browser to localhost:8000 and to the command line using the following command.

argocd login localhost:8000 # This to login on the terminal, different from UI login

Monitoring a private GitLab repository also requires a personal access token (PAT) generated for the user with appropriate read-access. The following command assumes

• the user has access to this repository
• the token is stored in the environment variable PAT
• the GitLab user name is stored in the environment variable GIT_USER.

# GIT_USER: GitLab username, PAT: Access token
argocd repo add https://gitlab.com/barmanroys/gitops-dbs.git --username $GIT_USER --password $PAT # This to access the private repository
kubectl apply --filename application.yaml # This will fire up the application and relevant monitoring services 

These steps, when executed correctly, set up the following components

• The external Load-balancer service for encryption-decryption
• The monitoring system controlled by ArgoCD that updates the deployment upon Git push to this manifest repository
• The continuous deployment system by Keel that updates the images deployed once a new push is detected.

To see the pod running the keel service, use

kubectl --namespace=kube-system get pods -l "app=keel"

To check the image monitoring log, use

kubectl logs -n kube-system $KEEL_POD_NAME -f

Diagram

Here's an example image to show what's happening

Graphana and Prometheus Setup

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack
helm install grafana grafana/grafana

Get access to Graphana WebUI by running this in Bash. Graphana should run on localhost:3000

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000

To get Graphana password, use

kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

To use Prometheus is scraping Kubernetes metrics

kubectl port-forward -n default svc/prometheus-kube-prometheus-prometheus 9090

Go to connection and add data source on graphana dashboard. Get the correct prometheus ClusterIP by running

kubectl get service | grep prometheus-kube-prometheus-prometheus 

and going to port 9090 of the IP. This is the socket that Graphana needs to connect to Prometheus data source.