Once a friend of mine wanted to learn Android development, like every starter he took a course, watched a few Youtube tutorials, and began his application development journey. Little did he know that it was not enough for his app to reach the audience. Disheartened he ended up attending a technical talk where he was introduced to the world of DevOps, pipelines, and finally twin processes CI/CD.
Let us see what he explored:
Continuous Integration
Continuous Integration or CI is a step where you integrate a set of processes before delivering your application to the customer.
Continuous Delivery
Continuous Delivery or CD is a process where you deploy or deliver your application to a specific customer.
Let us understand using a real-life example:
Suppose you have developed an application on your local machine and your customer is from India and you are in a different part of the world.
Question: How your app will be delivered from your machine to a customer who is in a different part of the world?
The CI/CD process goes through every organization, startup, MNC, and MAANG company. Your application has to be delivered in a prompt, efficient reliable way, it has to be tested and scanned for security vulnerabilities, and there should be a report that states that your app is perfect to be deployed. Finally, you deploy your application somewhere from where your customer will be able to access it.
These are certain steps that we have to follow, this may vary from product to product. Android apps may consist of more steps. In general, there have to be certain standards that companies follow to deliver apps to customers.
CI/CD is where you automate all of these processes:
Unit Testing
Static Code Analysis
Code Quality/Vulnerability Test
Automation
Reporting(All checks are verified)
Deployment
Unit Testing
Testing the code concerning the specific functionality.
You cannot run this process manually. Whenever there is a small change in the whole application, you cannot run it hundred times manually before delivering it to the customer. So, instead of running the whole app, we run a specific part of an app that changes.
Static Code Analysis
Suppose a developer declared twenty variables but used only four so, a lot of memory is being wasted.
To avoid this we do static code analysis which verifies if you are syntactically right, well-formatted, right indented, and did not declare any unnecessary variables.
Code Quality/Vulnerability Testing
Sometimes the code reviewer might not catch these so for that we have Code quality/vulnerability testing.
Suppose a new version of Android is launched and we updated to it but later came to realize that there is a security vulnerability. Hence, leading to a bad User Experience so before shipping the application to the customer, code quality is checked.
Automation
Automate the functional testing.
In functional testing, we check if the changes done to one functionality do not impact the other functionality. We verify the application in an end-to-end manner not just the function that changes.
Reports
To document the number of unit tests passed(unit test coverage), how many end-to-end passed, and how is the code quality.
Reports are very crucial for every organization.
Deployment
Deploying an application to a platform from where the customer can access it.
Doing all these steps manually will take a lot of time. Therefore, you divide this process into chunks and submit them one by one to the git repository every time you are confident about the change. All the versions of an application will be stored on Git(VCS or Version Control System). CI/CD is executed when you push your code to VCS.
Hence, once you are confident about your local changes as a developer you can submit the changes to Github and then CI/CD pipeline will take place.
How CI/CD pipeline takes place?
We deploy Jenkins in our Github repo and it will always look after the repo, whenever there is a new commit on a specific branch then Jenkins will be notified and as part of the automation, it will run a set of actions like code quality, vulnerability testing, static code analysis, unit testing.
Here, Jenkins acts as an orchestrator or a pipe. It will automate or orchestrate tools.
If you want to do unit testing, let's say you have a Java application. Integrate Maven for building the app and running unit tests and run them using Junit for code coverage. Sonar for code quality and testing, ALM for reports.
DevOps engineer will configure all these tools inside Jenkins so that they will run whenever your code is committed to the GitHub repository.
Further, you can integrate K8S, docker platform, EC2 instance, or cloud-based VMs.
Jenkins Pipelines takes care of automating and executing all the tools.
Jenkins can promote your app to different stages:
Dev Environment: Simple EC2 instance/simple environment where automation occurs. Once it is successful, Jenkins promotes the app to the staging environment.
Staging Environment: Here, you might have a Kubernetes cluster. (Like three master and five worker nodes).
Production Environment: It is like an exact environment that the customer uses. Here let's say you have three master and thirty worker nodes.
So before deploying your app on production, you might want to test your app on the Dev environment(single node k8s cluster), then the staging environment which is a little more complex cluster, and then eventually move forward to the production environment.
Question: Why you can't have a production-like environment in staging?
Because the production environment is a very costly setup.
History of DevOps tools
DevOps engineers started with Hudson and Jenkins, they first dealt with on-premises then they moved to the Cloud platform. Twitter, Amazon, or Flipkart is made up of thousands of microservices.
To deploy all these services Jenkins is a platform. It is binary. You install Jenkins on one host (Laptop or EC2 instance) and you keep adding machines(Multiple EC2 instances) to it. Suppose we have 10 teams, each Jenkins machine is dedicated to one team. Now if we want to set up additional VMs for running the nodes, every time you scale up the instances, you will have to create a lot of compute which is very costly.
Compute is RAM, CPU, and Hardware. Keep adding compute to the Jenkin instances, the setup becomes costly as well as the maintenance becomes huge. So instead look for setups that can scale up and scale down automatically. Integrating Jenkins with autoscaling groups isn't a real-time solution.
For example, if we want to scale the Jenkins to zero, we don't even want the Jenkins master node to exist. Like during weekends or odd times when there are no changes to be made, or zero pipelines to be executed. In the case of thousands of microservices, we have around 20-30 Jenkins instances. 20-30 master nodes and 3-4 worker nodes so if they are not configured well then every time there will be hundreds of VMs that are created but not used.
We want zero servers when we are not making any code changes so in such cases we avoid Jenkins.
Question- Then how do the modern-day applications work?
Modern Day CI/CD
Example of a highly scalable and open-source application ie., Kubernetes.
How does k8s ensure that it isn't wasting any computing power?
They are utilizing Github actions. GitHub actions are another way of doing CI/CD like Jenkins. Whenever a code change will be made, it will spin up a Kubernetes pod or a docker container for you, and everything gets executed on that docker container. If you are not using it then this server or a docker container will be used by some other project (Concept of shared resources).
So, instead of wasting resources(Jenkins instances) for every project, you can create one common GitHub Actions which can be used by multiple projects in your organization. This saves your compute and if nobody is using it, there is no wastage because the server size is shared across everyone so literally you are using zero compute instances when there is no code change.
One more advantage is scalability. In the case of Jenkins, if you want to add one more server, you will have to add one more worker node whereas, in terms of k8s, you can easily scale up to hundreds or thousands of nodes.
Conclusion
Attending technical talks always brings up new perspectives and skills that add up to our tech stack. Let us summarize our learnings:
Summary
Without CI/CD, an application would be delivered to customers after months, hence it ensures that the application is delivered to customers rapidly whenever there is a change introduced by the application developer. That change will go through the CI/CD Pipeline orchestrated by tools like Jenkins or GitHub Actions which will take care of unit testing, reporting, code quality, vulnerability analysis, etc.
Hence, learning tools like Jenkins, and GitHub Actions as a DevOps engineer is quite important and useful.