News

Developing An Internal Developer Platform

stp2y0 Comments
Developing An Internal Developer Platform


In this blog we will be discussing about Internal Developer Platform (IDP), what it is , how to set one up as a lead DevOps Engineer.Present tech landscape implementing an IDP helps to streamline workflows and boost productivity.



What is an Internal Developer Platforming

It is a platform or an application which is built by platform engineecloring team for building golden paths and to facilitate developer self service. It consists of various tools and technologies integrated together so that it lowers cognitive load on developers without abstraction of context and underlying technologies.



Why is there a need to build IDP

It Centralizes and simplifies application deployment and management.



Simplify Deployment:

Helps engineers to deploy applications with minimal manual intervention.



Increase Reliability:

This ensures that deployments are consistent and error free.



Enhance Productivity:’

Reduces time and efforts invested for managing infrastructure



Tools and Technologies Used



Kubernetes

This is a container orchestration tool which is used for automating deployments , scaling and management of applications which are containerized.

Documentation: https://kubernetes.io/docs/home/
Youtube: https://youtu.be/X48VuDVv0do?si=9aPNky0DDPyATiFM
Interview Questions: https://kuberty.io/blog/top-kubernetes-interview-questions/



Cyclops

It is a user friendly UI and streaming tool for managing kuberenetes clusters.

Documentation: https://cyclops-ui.com/docs/about/
Youtube: https://www.youtube.com/live/QQAqHGEYVRM?si=GEgZ3bDyhsCh_cHV
Github Repository: https://github.com/cyclops-ui/cyclops



Docker

It is a tool which is used for easy developing , shipping and running applications inside a container

Documentation: https://docs.docker.com/
Youtube: https://youtu.be/3c-iBn73dDE?si=FI7ZKDi3Kij1MYMo
Interview Questions: https://hackr.io/blog/docker-interview-questions



Helm

It is a package manager for kubernetes which helps in simplification of deployments and services

Documentation: https://helm.sh/docs/
Youtube: https://youtu.be/5_J7RWLLVeQ?si=8DPo9rD5cYf-moJd
Interview Questions: https://www.devopsschool.com/blog/top-50-helm-interview-questions-and-answers/



Prometheus

It is used for monitoring and alert management

Documentation: https://prometheus.io/docs/introduction/overview/
Youtube: https://youtu.be/h4Sl21AKiDg?si=2Heu9OAkG_FkZSNJ
Interview Questions: https://www.devopsschool.com/blog/top-50-prometheus-interview-questions-and-answers/



Graphana

It is a visualization tools for visualizing data

Documentation: https://grafana.com/docs/
Youtube: https://youtu.be/CjABEnRg9NI?si=QrZ3aBVUy_B9LOKH
Interview Questions: https://www.devopsschool.com/blog/top-50-grafana-interview-questions-and-answers/



PostgreSQL and MongoDB

These are popular relative and NoSQL databases respectively and is used for storing application data.

Documentation-PostgreSQL: https://www.postgresql.org/docs/
Youtube-PostgreSQL: https://youtu.be/qw–VYLpxG4?si=rory5ndP3nqGzKjZ
Interview Questions-PostgreSQL: https://www.datacamp.com/blog/top-postgresql-interview-questions-for-all-levels
Documentation-MongoDB: https://www.mongodb.com/docs/
Youtube-MongoDB: https://youtube.com/playlist?list=PL4cUxeGkcC9h77dJ-QJlwGlZlTd4ecZOA&si=N49ED6H2sztBQrFu
Interview Questions: https://hackr.io/blog/mongodb-interview-questions



Istio

It is a service mesh for advanced traffic management , security , observability for microservice applications

Documentation: https://istio.io/latest/docs/
Youtube: https://youtube.com/playlist?list=PLI4xy7phW54nbfjf7ZMnlEx1O5cHneigh&si=bs3RdBT6Tm6SF9Fc
Interview Questions: https://www.devopsschool.com/blog/top-50-istio-interview-questions-and-answers/



Jenkins

It is an open source automation server tool which is used for Continuous Integration (CI) and Continuous Delivery (CD), which automates building, testing and deploying applications via pipelines.

Documentation: https://www.jenkins.io/doc/
Youtube: https://youtube.com/playlist?list=PLy7NrYWoggjw_LIiDK1LXdNN82uYuuuiC&si=yjaMJdICFYJlCFZY
Interview Questions: https://hackr.io/blog/jenkins-interview-questions



Argo CD

it is a gitops CD tool for kubernetes , it is declarative and ensures the desires state of deployed application is matched as defined in git repository

Documentation: https://argo-cd.readthedocs.io/en/stable/
Youtube: https://youtu.be/MeU5_k9ssrs?si=PUzJxpQobUiVcqvm
Interview Questions: https://www.lwplab.com/forum/questions-answers/interview-questions-answers-on-argo-cd



Implementation



Step 1: Setup Docker

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce
Enter fullscreen mode

Exit fullscreen mode



Step 2: Start Docker

sudo systemctl start docker
sudo systemctl enable docker
Enter fullscreen mode

Exit fullscreen mode



Step 3: Set up kubernetes

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
Enter fullscreen mode

Exit fullscreen mode



Step 4: Set Up Minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64
Enter fullscreen mode

Exit fullscreen mode



Step 5: Start Minikube

minikube start



Step 6: Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Enter fullscreen mode

Exit fullscreen mode



Step 7: Install Cyclops-UI

kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.8.2/install/cyclops-install.yaml && kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.8.2/install/demo-templates.yaml
Enter fullscreen mode

Exit fullscreen mode



Step 8: Expose cyclops and access it

kubectl port forward svc/cyclops-ui 3000:3000 -n cyclops
Enter fullscreen mode

Exit fullscreen mode

cyclops is now accesssible in browser by using this url: http://localhost:300



Step 9: Install and Configure Databases



PostgreSQL

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgresql bitnami/postgresql
Enter fullscreen mode

Exit fullscreen mode



MongoDB

helm install mongodb bitnami/mongodb



Step 10: Install Monitoring and Logging Tools



Prometheus

helm install prometheus bitnami/prometheus
Enter fullscreen mode

Exit fullscreen mode



Grafana

helm install grafana bitnami/graphana
Enter fullscreen mode

Exit fullscreen mode

configure grafana to add prometheus as data source, below is example yaml file named as grafana-datasource.yaml



Step 11: Install and implement Service Mesh with Istio

curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH
istioctl install --set-profile=demo -y
kubectl label namespace default istio-injection=enabled
Enter fullscreen mode

Exit fullscreen mode

deploy a sample app using istio , below is example yaml file named as sample-app.yaml used for deployment

sample-app.yaml

deploy using kubectl apply -f sample-app.yaml



Step 12: Install ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argocd/stable/manifests/install.yaml
Enter fullscreen mode

Exit fullscreen mode

  • Expose the argocd to get access 
kubectl port-forward svc/argocd-server -n argocd 8080:443
Enter fullscreen mode

Exit fullscreen mode 

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
Enter fullscreen mode

Exit fullscreen mode



Step 13: Install Jenkins

kubectl create namespace jenkins
helm repo add jenkins https://charts.jenkins.io
helm repo update
helm install jenkins jenkins/jenkins --namespace jenkins
Enter fullscreen mode

Exit fullscreen mode

  • Expose jenkins to get access 
kubectl port-forward svc/jenkins -n jenkins 8080:8080
Enter fullscreen mode

Exit fullscreen mode 

kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo
Enter fullscreen mode

Exit fullscreen mode



Step 14: Set up Jenkins Pipeline;

Access jenkins using the url http://localhost:8080, login using the admin password and username as admin, below is the sample image of jenkins console

Jenkins Console

now create a sample app in your local directory, it can be in any programming language , below is the sample app structure in node js format

App.js

next create a Dockerfile to run docker container , sample Dockerfile is shown below

Dockerfile

now create a Jenkinsfile which contains the steps for jenkins pipeline , a sample pipeline is shown below

Jenkinsfile

at last create a deployment.yaml file for kubernetes and other services to run , below is sample file for deployment.yaml

deployment.yaml



Usecase

Let’s consider an E-commerce company which requires frequent updates and deployments and engineers are struggling with their deployments which is manual , inconsistent , lack of visibility. The answer to tackle these issues is building and IDP. Lets see how each tool helps in solving these problems.



Cyclops-Ui

  1. Pre Deployment: Cyclops Ui can be accessed to set parameters
    like environment settings and resource allocation.

  2. Deployment: the pipeline will be triggered by Cyclops-UI
    which leverages tools like kubernetes , docker and helm and
    ensures consistent and reliable manner of deployment.



Jenkins

  1. Automated Builds: The source code repository in github is
    monitored by jenkins and an immediate trigger for new build
    happens when any change is detected in the repository.

  2. Automated Tests: Once the changes are applied jenkins also
    tests the changes to ensure no bugs or regressions.

  3. Deployment: After tests are successful the application is
    deployed on staging environment by jenkins and after validations
    are done it is deployed to production environment.



Argocd

  1. Continuous Delivery: Detects changes in repository and
    synchronizes the kubernetes cluster to the desires state
    mentioned in manifest file.



Prometheus

This can be used to collect metrics from application and kubernetes cluster which provides insights into performance and cluster health.



Graphana

This is used for visualizng the metric data taken from prometheus
and create dashboards and monitor the application in real time



PostgreSQL and MongoDB

These are the primary databases of the application which uses transactional and non relational data respectively, it is managed by helm charts for consistency and scalability.



Istio

  1. Traffic Management: Istio provides features for managing
    traffic between microservices , for loadbalancing and for
    routing as well.

  2. Security: Security policies are enforced by istio to ensure
    secure communication between services.



Benefits

  1. Efficient Deployment: Since deployment process is automated
    manual errors are reduced and consistency is ensured.

  2. Enhanced Monitoring: With realtime dashboards ,visualization
    performance and issues are being monitored effortlessly.

  3. Scalable Infrastructure: With tools like kubernetes , docker
    , helm applications can be scaled and managed easily also
    effective resource allocation is done without any difficulty.

  4. Streamlined CI/CD: With jenkins building , testing and
    deployment is automated and consistent as and when changes in
    source repository are detected.

  5. GitOps Workflow: With tool like ArgoCD deployed applications
    will always seam lessly match desired states mentioned in
    manifest file in git repository.



Final Thoughts

Building an IDP using cyclops and kubernetes includes setting up a variety of tools whcih ensures smooth and efficient deployment and management process. By using tools like Docker , Helm , ArgoCD, prometheus , graphana, PostgreSQL, MongoDB, Istio , Jenkins we have built a comprehensible and reliable platform which makes deployments and management of projects very easy.

By using a range of DevOps and CI/CD tools we have built a powerful environment which accelerates development and deployments. We have also seen an example usecase of E-Commerce website which shows how efficiently deployments can be managed and how it enhances both productivity and reliability



Source link
lol

By stp2y

Leave a Reply

Your email address will not be published. Required fields are marked *

No widgets found. Go to Widget page and add the widget in Offcanvas Sidebar Widget Area.