Exploring Git Repo Architecture in GitOps

Introduction

With the increasing popularity of GitOps, solutions like ArgoCD, Flux, etc., have helped numerous Kubernetes environments address the challenges and processes of Continuous Deployment (CD).

However, GitOps, which combines Git and Operations (Ops), involves varying ways of utilizing Git depending on the environment and team.

This article attempts to organize several potential usage methods. In practice, there’s no perfect solution, so it’s important to compare and clarify team requirements and capabilities to establish suitable approaches.

Challenges of GitOps When deploying applications to Kubernetes using GitOps, the following diagram represents a possible workflow:

The entire process includes:

1. Application source code
2. Containerized application
3. Storing containers in a remote container registry
4. Simultaneously updating K8s YAML to prepare for the latest changes
5. GitOps using Webhooks or Pulling mechanisms to detect YAML modifications
6. Applying changes to the target Kubernetes cluster
7. Kubernetes updates the application by fetching the latest version from the container registry

While this workflow seems straightforward and simple, its implementation can have various nuances. Several common issues are frequently discussed:

1. Which tool to use for maintaining K8S YAML? Helm? Kustomize? Jsonnet?
2. How to manage the ownership of Source Code and K8s YAML? Should they be in the same Git repo or separated?
3. After generating a new version of the container, who is responsible for updating K8s YAML? The application developer or DevOps personnel?

Next, I will share solutions for these three issues. As mentioned earlier, there’s no perfect solution, and each implementation approach has reasons behind its selection, influenced by factors like team size, project timelines, and more.

K8s Manifest

Kubernetes resource descriptions can be in either YAML or JSON formats. The prevailing approach mostly involves using YAML for descriptions.

Here’s a quick recap of some common tools:

1. Vanilla YAML
2. Helm
3. Kustomize
4. Jsonnet

Vanilla YAML

Vanilla YAML directly describes all Kubernetes objects using YAML format.

Pros:

• Intuitive and suitable for beginners
• No need for additional tools;
• YAML files can be used with kubectl

Cons:

• Lacks flexibility;
• Customization requires maintaining multiple files
• No versioning concept
• Becomes complex and cumbersome with more deployment environments

Due to these reasons, few teams directly use this method to manage complex and multi-environment applications.

However, some third-party solutions define their usage through this approach for reasons like:

• Involves a small number of K8s objects;
• Not too complex a project focuses on application development, leaving deployment to user.
• Maintains different versions’ differences using Git branches

This approach is also suitable for beginners to learn K8s concepts and practice. After becoming familiar with the concepts, they can use the following tools to manage without feeling unfamiliar.

In summary, each environment requires maintaining a nearly identical set of files, making maintenance challenging.

Helm

Helm is one of the most well-known tools for defining Kubernetes applications. Using Helm Chart and Go Template mechanisms, you can easily customize resource deployment based on the environment.

Pros:

• Uses Go Template to dynamically generate final K8s resources
• Can be packaged for easy distribution and installation, with version control support
• Supports Dependency concept, enabling Umbrella Chart composition
• Rich ecosystem with third-party projects supporting Helm Chart installations, reducing obstacles for users

Cons:

• Steep learning curve;
• Template syntax might be complex
• Requires additional Helm command installation
• Might need an additional Helm Chart Server for maintenance

Helm allows teams to maintain a single file for the main application and corresponding values.yaml files for different deployment environments, dynamically generating K8s objects tailored to each environment’s needs. The workflow is as follows:

Apart from direct local usage, Helm Charts can be packaged and published to a Helm Chart server, which essentially functions as a web server. Many open-source projects use this method to distribute their Helm Charts, making it easy for users to select versions and customize using values.yaml. The process is similar to the diagram shown.

Additionally, some teams deliver not just a single application but a service composed of multiple applications. In such cases, an Umbrella Chart is employed. The concept is simple:

1. Delivery and deployment are for a complete service, not just a single application
2. The Umbrella Helm Chart relies on many other Helm Charts
3. Deploying the Helm Chart also deploys all dependent Helm Charts
4. Umbrella Helm Chart’s values.yaml can be extensive and complex, dynamically determining which dependencies to install and passing parameter settings

As shown in the example diagram, in an environment with five Helm Charts, one primary Helm Chart links to Helm Chart {A/B/C/D} through helm dependency.

Users only need to prepare a values.yaml and deploy the Umbrella Chart to deploy all four Helm Charts to the target cluster seamlessly.

Umbrella Helm

Kustomize

Unlike Helm’s templating approach, Kustomize offers a simpler and cleaner way to customize Kubernetes resource objects. All files maintain the native K8s format, and dynamic modifications to resources are achieved through the concept of patches.

Pros:

• Based on the native YAML format
• No need to learn additional templating syntax
• Integrated into kubectl
• No need to install additional commands
• Modifications through overlay and patching mechanisms

Cons:

• Ecosystem not as rich as Helm’s
• Lacks versioning concept, making distribution to different users and clients inconvenient
• Although lacking templates, learning the overall structure and usage still requires some effort.

In the example below, we prepare base objects for our application within the base folder. Then, each environment folder contains customized content. Everything is connected based on the contents of kustomization.yaml.

Kustomize Example

While previously requiring a separate kustomize command, it’s now integrated into kubectl, so you can use kubectl apply -k to operate based on the kustomize deployment format.

Jsonnet

Jsonnet is a programming language based on JSON format. All operations are based on JSON, and the output is also in JSON format. Since Kubernetes also accepts JSON-based descriptions of resources, some people use this approach to maintain K8s applications.

In essence, Jsonnet utilizes programming language characteristics to generate object description files for Kubernetes.

Pros:

• Supports programming language logic like if/else, loops, functions, etc.
• Supports the concept of libraries to reduce code duplication

Cons:

• Requires learning a new tool
• Jsonnet Documentation and ecosystem might not be as extensive

Jsonnet operates on Json objects, and jsonnet k8s and k8s-libsonnet provide various interfaces to quickly generate object resources suitable for different Kubernetes versions.

For instance, the following syntax can generate a K8s deployment + service object. These files can be reused and dynamically overridden. The process is quite flexible:

local k = import "vendor/1.27/main.libsonnet";

local s= {
  name: "demo",
};

[
   k.apps.v1.deployment.new(name="demo", containers=[
     k.core.v1.container.new(name="demo", image="hwchiu/netutils")
   ]),
   k.core.v1.service.new("demo", s, 5000)
]

azureuser@course:~/jsonnet$ jsonnet --yaml-stream main.jsonnet
---
{
   "apiVersion": "apps/v1",
   "kind": "Deployment",
   "metadata": {
      "name": "demo"
   },
   "spec": {
      "replicas": 1,
      "selector": {
         "matchLabels": {
            "name": "demo"
         }
      },
      "template": {
         "metadata": {
            "labels": {
               "name": "demo"
            }
         },
         "spec": {
            "containers": [
               {
                  "image": "hwchiu/netutils",
                  "name": "demo"
               }
            ]
         }
      }
   }
}
---
{
   "apiVersion": "v1",
   "kind": "Service",
   "metadata": {
      "name": "demo"
   },
   "spec": {
      "ports": [
         5000
      ],
      "selector": {
         "name": "demo"
      }
   }
}

The overall flow involves writing various jsonnet files, generating files suitable for K8s using jsonnet commands, and then using kubectl commands to apply them to the target cluster.

Tools

The four mentioned tools have no absolute good or bad, and they are all supported by ArgoCD. Therefore, as long as their features and pros and cons align with the team’s needs, they can all be tried.

In practice, a more common concept involves mixing these tools. After all, many open-source projects are built upon Helm for distribution, while teams may use Kustomize/Jsonnet to manage their applications. Thus, these different approaches might coexist, as ArgoCD supports them all for deployment. The only challenge is that users need the relevant background knowledge for utilization, debugging, and configuration.

Git Repo

From the perspectives of Source Code and K8s Manifest, there are two common choices:

1. Each application has a dedicated Git Repo containing Source Code, Dockerfile, and K8s manifest.
2. Each application has a dedicated Git Repo containing Source Code and Dockerfile. Additionally, there’s a separate Git Repo containing all K8s manifest files needed for the applications.

Code/YAML in the Same Repo

The first concept is as follows:

Features of this mode include:

1. From application development to K8s deployment, all resources are kept together for easy cross-referencing.
2. If the application requires any modifications, the deployment YAML can be modified together, such as adding the reading of a new environment variable.
3. The CI/CD pipeline can handle everything, even testing K8s deployments using frameworks like KIND.
4. If using Helm Charts, they can be packaged and distributed within the CI/CD pipeline.
5. Using jsonnet/kustomize for this approach makes handling files shared across applications challenging.
6. Version control can be managed using Git branches/tags.
7. Operational personnel find it difficult to understand the comprehensive deployment methods and content used on the system and must search each Git repo step by step.

Additionally, this model often faces challenges related to handling image tags.

For example:

1. A developer opens a PR to add a new feature.
2. Since users don’t know the final container image tag, they cannot update the K8s YAML files simultaneously.

Common solutions include using the latest tag for deploying applications, opening a second PR for updates, or creating a GitHub App to dynamically update PR content.

Therefore, implementing this process still involves many details to consider and discuss, and it’s not as perfectly simple as imagined.

Code/YAML in Different Repos

A more common approach is the second one, involving separation of responsibilities. Developers can focus on their application development and containerization, while a dedicated Git Repo handles all GitOps management and deployment.

The basic concept is illustrated in the diagram below:

Features of this architecture often include:

1. All deployment objects are kept together, making it easy to quickly understand the differences in all deployments.
2. Application development is separate, making it convenient for developers and operational personnel to handle their respective responsibilities.
3. If a new version of the application requires any YAML changes, they need to be handled separately in this repo; it cannot be done all at once.
4. Since environments are grouped together, concepts like kustomize/jsonnet are easier to share.
5. Different deployment resources for various environments can be distinguished using Branch/Folder methods.
6. Dedicated CI/CD pipelines can validate overall service deployments, rather than just individual applications.

Structures like the one below are common for Kustomize or Jsonnet-based approaches.

├── base
│   ├── app
│   │   ├── foo
│   │   │   └── app.jsonnet
│   │   └── foo2
│   │       └── app.jsonnet
│   └── component
│       ├── deployment.jsonnet
│       └── service.jsonnet
└── env
    ├── production
    │   ├── foo
    │   │   ├── env.jsonnet
    │   │   └── main.jsonnet
    │   └── foo2
    │       ├── env.jsonnet
    │       └── main.jsonnet
    └── staging
        ├── foo
        │   ├── env.jsonnet
        │   └── main.jsonnet
        └── foo2
            ├── env.jsonnet
            └── main.jsonnet

├── base
│   ├── foo
│   │   ├── deployment.yaml
│   │   ├── kustomization.yaml
│   │   └── service.yaml
│   └── foo2
│       ├── deployment.yaml
│       ├── kustomization.yaml
│       └── service.yaml
└── overlays
    ├── dev
    │   ├── foo
    │   │   ├── kustomization.yaml
    │   │   └── resource.yaml
    │   └── foo2
    │       ├── kustomization.yaml
    │       └── resource.yaml
    └── prod
        ├── foo
        │   ├── kustomization.yaml
        │   └── resource.yaml
        └── foo2
            ├── kustomization.yaml
            └── resource.yaml

Different environments are separated using folders, and if needed, Git tags can be used for versioning to track deployment content for specific versions in the future.

Additionally, since Kustomize currently supports using Helm Charts for deployment, you might encounter variations like the one below, where Kustomize and Helm are combined.

├── base
│   ├── foo
│   │   ├── deployment.yaml
│   │   ├── kustomization.yaml
│   │   └── service.yaml
│   └── foo2
│       ├── deployment.yaml
│       ├── kustomization.yaml
│       └── service.yaml
├── helm
│   └── app
│       ├── charts
│       ├── Chart.yaml
│       ├── templates
│       │   ├── deployment.yaml
│       │   ├── _helpers.tpl
│       │   ├── hpa.yaml
│       │   ├── ingress.yaml
│       │   ├── NOTES.txt
│       │   ├── serviceaccount.yaml
│       │   ├── service.yaml
│       │   └── tests
│       │       └── test-connection.yaml
│       └── values.yaml
└── overlays
    ├── dev
    │   ├── foo
    │   │   ├── kustomization.yaml
    │   │   └── resource.yaml
    │   └── foo2
    │       ├── kustomization.yaml
    │       └── resource.yaml
    └── prod
        ├── foo
        │   ├── kustomization.yaml
        │   └── resource.yaml
        └── foo2
            ├── kustomization.yaml
            └── resource.yaml

Mix

Since Helm Charts can be packaged and pushed to a Helm Chart server, using Helm Charts may lead to both tools being combined, as shown in the diagram below.

In this scenario, features include:

1. The application repo focuses solely on maintaining the release and configuration of the Helm Chart.
2. The application repo packages the Helm Chart using CI/CD and pushes it to the Helm Chart server.
3. The K8s YAML repo then fetches the built files from the Helm Chart server using the Helm Chart and deploys them to different environments using different values.
4. Separate repo characteristics are retained, roles are separated, and each has its own Git workflow for management.

As this architecture usually requires an additional Helm Chart server, Helm Chart v3 supports OCI formats, allowing the use of OCI-supported Container Registries like ECR, Harbor, or traditional Chart Museum for deployment.

Image Updater

In the second and third architecture approaches, many teams explore enhancing workflows, especially when a Container Image has a new tag version. They consider how to automatically update descriptor files in that repo.

Some teams use tools like Argo Image Updater or create their own CI/CD processes for updates.

Summary

1. This article summarized common Git structure issues in the context of GitOps architecture and discussed common tools and usage approaches.
2. K8s applications can be managed using native YAML, Helm, Jsonnet, Kustomize, etc., each with their own characteristics and suitable scenarios.
3. There’s no one-size-fits-all solution; all planning discussions should consider team size, processes, capabilities, and workloads to determine suitable tools.