Solution Spotlight #4 – KubeRocketCI (a.k.a. EPAM Delivery Platform)

Project Idea

When a new project starts, everyone has ambitious plans to make the world a better place, to use proven practices from previous projects, to build the CI/CD of their dreams. Sure there are projects where we can make these dreams come true, but there are also projects where it is not possible for various reasons. For example, the team disagrees, due to different moments, the customer has a specific infrastructure, etc. In the end, the chances are about 50/50: either it works or it doesn't. Sooner or later, an initiative tends to tip the scales in the direction of more successful project launches.

At EPAM, one such initiative was our project called KubeRocketCI (hereafter abbreviated as EDP). The idea itself originated at the end of 2017. The main task of KubeRocketCI is to provide minimum time for the so-called Sprint 0 when the formation of basic practices on the project, the configuration of CI/CD, and other processes take place. After that, start in Sprint 1 to develop business features that can be demonstrated to the customer, automatically tested, and run through configurable Quality Gates at various stages of the CI/CD process.

KubeRocketCI goal: to make Sprint 0 a maximum of 2 weeks, which includes deploying the infrastructure, configuring CI/CD for a specific project, and training the team on the tools and techniques used in the solution.

Team: assembled by architects and engineers (software engineers, DevOps, testing engineers) who have sufficient expertise and experience in developing projects in Java, JavaScript, .NET using modern CI/CD practices.

Tagline: "Business value from Sprint 1".

The Main Technologies Chosen For the Project

Naturally, to ensure such an ambitious goal and repeatable result, we made a number of important decisions. On the one hand, they narrow the application of KubeRocketCI to specific projects, while on the other hand, they provide a repeatable solution and the ability to create a critical set of training materials.

Key KubeRocketCI technologies:

• Docker-based containerization

• Container orchestration via Kubernetes

• Platform as a Service over Kubernetes - OpenShift

Schematically, the solution can be represented as the following set of layers. Installation on any infrastructure where it is possible to use the 3 technologies described above.

Experienced OpenShift or Kubernetes users will notice that there are quite a few ready-made solutions or reference articles, such as:

• CI/CD with OpenShift

• Continuous Integration and Continuous Delivery with Fabric8

• OpenShift.io

• A solution based on Kubernetes and Istio

A closer look at adapting these solutions raises a number of complexities. For example, OpenShift's CI/CD only describes the basic principles of how CI/CD can be organized around the platform and does not detail the implementation and specifics of use.

The Fabric8 team, on the other hand, has done a huge, I would even say titanic amount of work on adapting Java applications based on Spring Cloud + Netflix, promising the ability to deploy microservices solutions to both OpenShift and Kubernetes. By the way, it was this product that had the biggest impact on the architecture and capabilities of EDP.

At the same time, from an end-user point of view, Fabric8 does not look easy to use. In reviewing Fabric8, the KubeRocketCI team even joked that it was the most bleeding cutting edge they'd ever seen. The reason for that was the trivial inability to install a stable version of the platform, whether on minikube/minishift or full OpenShift and Kubernetes clusters. This may be due to the fact that judging from Red Hat summits (like this video), the Fabric8 development team is helping to build OpenShift.io, which has caused the pace of development of Fabric8 itself to slow down.

OpenShift.io looks like a very promising platform, which, however, has a couple of important "buts."

The integrated project management system is far inferior to the familiar Jira, Rally, Trello, etc. How many will want to use a new tool without the already familiar functionality?

Developing in a browser-based on Eclipse Che looks very attractive in terms of manager and Ops, but I find it hard to imagine there joyful working senior developers who are used to adult full-featured IDEs.

OpenShift.io deserves a more detailed study and a separate article. As well as a new rising star - KNative. It's too early to say much about this technology, let's see how far they can go. The beginning looks promising.

What KubeRocketCI Consists Of and How It Works

To further talk about the architecture and the usefulness of KubeRocketCI for startup projects at EPAM, it's worth describing its main features right away:

• An integrated set of tools for organizing CI/CD, with the ability to replace tools with peers

• Centralized authentication and authorization system. SSO for all CI/CD tools and integration with enterprise SSO. RBAC-control of access to the platform resources

• Transparent integration of source code applications on CI/CD tools

• Configurable continuous delivery pipeline with the possibility to set up the sequence of its stages

• Providing automated testing (functional, security, performance, etc.) for each stage of the CD pipeline, as well as configuration and control of quality gates for each of them

• Cluster monitoring for development and centralized log collection

The KubeRocketCI architecture is a set of interrelated abstract components and APIs for them. They are used to implement the core functions of the platform. The components can be broken down into three major logical blocks:

• Tools for organizing CI/CD

• Components for running automated tests

• Continuous delivery pipeline, consisting of an arbitrary set of stages. They launch automated tests and control passing by means of quality gates analysis

The tools for CI/CD organization, their purpose, and supported implementations are given in the list below:

• VCS - Version control and management system. It provides secure storage in separate repositories for the code of applications that are added to the platform. Implementation: GitLab, Bitbucket

• IdentityService - A service that stores KubeRocketCI user data and the roles and privileges they have in each tool. It is a user authentication service, allowing you to delegate authentication to other IdentityServices. Implementation: KeyCloak

• AutorizationServer - A server that manages user rights, providing a central and primary source of IdentityService information. The IdentityService converts received rights or roles into those used in KubeRocketCI components. Implementation: LDAP

• CorporateSSO - With this server, users can enter a one-time password and gain access to all registered resources according to their privileges. Implementation: ADFS

• CodeReview - The main component for validating source code. All changes by team members in their application repositories go through the Code Review process. In addition to the manual step (performed by the team leader, for example), it also includes automated steps (checking compilability, running unit tests, etc.) KubeRocketCI defines a set of privileges for CodeReview by which to delimit the ability of team members to manage changes to the source code. For example, approve code changes, submit code changes, block code changes, etc. Implementation: Gerrit

• CICDServer - A central component for performing various kinds of tasks related to the development lifecycle. Implementation: Jenkins

• СodeAnalyzer - Static code analyzer. Allows controlling its quality in terms of different quality attributes (maintainability, security, etc.). This is done based on the configured rules and thresholds (quality gates). Such a tool also allows you to analyze historical data and understand the trend of code quality changes. Implementation: SonarQube

• ArtifactsStorage - Storage for all "binary" artifacts (jar, zip, etc.) that are created by CI server during pipelines startup based on events from CodeReview or VCS. Implementation: Nexus

• DockerStorage - A repository for docker images that are created by the CI server. Implementation: Docker Registry

• BinaryPackager - Tools for creating "binary" artifacts from source code. Implementation: Maven, NPM

• DockerPacager - Tools for creating "binary" artifacts from source code. Implementation: OpenShift Source to Image

• Cockpit - An KubeRocketCI tool that allows you to add applications and automated tests for them from existing repositories and integrate them into the tools listed above. Implementation: EDP custom tool

• CentralizedLogging - A repository for logs from all running applications in OpenShift, including the tools of KubeRocketCI itself. Implementation: EFK

• Monitoring - A tool for monitoring cluster nodes and dockers running in a cluster. Implementation: Prometheus

The diagram below is a logical KubeRocketCI design based on the toolkit described above:

As you can see from the list and diagram above, KubeRocketCI relies on well-known and widely used opensource tools, through which CI/CD is usually built on most projects. The main value of EDP is the standardization of API integration and interoperability of these tools, both during the installation of the platform and when using it. The main goal is to provide extension points for EDP users and the ability to replace components that are part of the platform. Let's look at more specific examples. As mentioned above, EDP distinguishes between two main types of APIs:

Integration

Each tool needs to be installed in the platform, configured, and integrated into the SSO. Therefore, the basic functions are as follows:

• install()

• configure()

• integrateWithSSO()

More specific is the integration of tools with each other to provide the declared KubeRocketCI functions. For example, for the CodeAnalyzer we can distinguish such:

• uploadEdpQualityProfiles()

• registerCIServerFunctionalUser()

Runtime

After installing the platform, users can add their own applications and configure the CD pipeline for them. These features make up the second part of the API that KubeRocketCI provides for the tools described. Some of the most comprehensible examples are as follows:

CodeReview:

• registerNewApplication(), which allows you to register a new application and perform a code review process.

CICDServer:

• registerFeatureBranchPipeline(), which provides for adding a CI pipeline to run per commit in the feature branch.

• registerPullRequestPipeline() registers a pipeline for each added application, which is responsible for building it after confirming the pull request.

• addStageToPipeline() allows you to add a new stage to the CD pipeline, providing a mechanism for automatic promotion between stages.

Schematically, the KubeRocketCI components and their APIs can be represented in the following diagram:

The topic of the CI/CD pipeline, recommended by the KubeRocketCI, was touched on above, which can be represented as follows:

Where

• SIT - System Integration testing - the stage at which it is recommended to run automated tests.
• QA - Quality Assurance - a stage where the verification of the new functionality of applications under development is performed.
• UAT - User Acceptance Testing - a stage for demonstrating the application to the end users and giving them an opportunity to work with the new functionality by themselves.

For each stage there is deployment of all applications to the dedicated environment. After a successful deployment, the automated tests configured for this stage are launched and the quality gates are controlled. Schematically, the pipeline can be represented as the following high-level steps:

The EDP Cockpit deserves special attention, which is a web interface for the Runtime API EDP. The Cockpit allows you to add new applications, configure the sequence of steps for the CD pipeline, determine which automated tests to run at which step, etc.

Let's summarize what an KubeRocketCI user gets after installing in an OpenShift cluster:

An integrated set of CI/CD tools with configured SSO, centralized authentication, and authorization. This facilitates end-user access to platform resources and allows flexible control over the level of that access.

• KubeRocketCI source code with the ability to customize it to the needs of projects.

• Quickly add application source code and projects with automated tests via Cockpit.

• Customize the CD pipeline, manage the set of automated tests run at each stage, and assign quality gates to ensure quality control.

• Tools for centralized logging and monitoring.

Furthermore, multiple KubeRocketCI instances can be installed in a single OpenShift cluster, allowing you to develop multiple projects on the same infrastructure.

What's Now and What's Next

KubeRocketCI is no longer just an internal EPAM initiative. It is now a platform on which several projects have been implemented and new ones are starting. One of them involves a team size of up to 100 people with different expertise, which increases the level of complexity and responsibility of setting up the platform. AWS and Azure are the Cloud Provider for these projects, and therefore EDP teams gain important experience in managing OpenShift clusters in the public cloud according to the latest Red Hat guidelines.

Naturally, the KubeRocketCI team has no plans to stop there. Our next steps:

• Adding new tools that implement the KubeRocketCI component APIs.

• More fine-tuning of the tools at the stage of adding them.

• Expanding the list of supported technologies for development and frameworks.

• Tighter integration with managed services of public clouds.

• Development of infrastructure and OpenShift cluster configuration to comply with the strictest security standards.

• Integration of servers for storing and analyzing the results of performance and security testing and much more.

An important point of KubeRocketCI development is the introduction of this platform for projects, which are being developed as part of EPAM RD (Resource Development). This allows young professionals to immediately get used to the highest standards of code development on modern projects.