Docs

Air-gapped GitOps with OCM & Flux

Introduction

In this guide, we will show you how the tools provided by OCM make it possible to automate your air-gapped deployments.

Air-gapped can mean different things depending on the context. For this guide, we’ll assume it means your deployment artifacts are stored in a private registry protected by the security controls at your organization. Your applications only have access to this private registry and little to no public internet access.

We’ll take the same podinfo component that we deployed in the Deploy Applications with OCM & GitOps guide but this time we will use the OCM CLI to transfer the component to our own registry. The application will then be deployed from this “private” registry. This, of course, mimics a real-world air-gap scenario. In practice, there could be many layers of security between the two registries; however, the mechanics are ultimately the same.

Table of Contents

Requirements

Component Content

The podinfo component contains three resources:

  • a container image for podinfo
  • a kubernetes deployment manifest for podinfo
  • a configuration file read by the ocm-controller

We can list these resources using the ocm CLI:

ocm get resources ghcr.io/phoban01//phoban.io/podinfo -c v6.3.5

NAME       VERSION IDENTITY TYPE      RELATION
config     6.3.5            PlainText local
deployment 6.3.5            Directory local
image      6.3.5            ociImage  external

If we examine the config file, we will see a section named localization:

ocm download resource ghcr.io/phoban01//phoban.io/podinfo -c v6.3.5 config -O -

apiVersion: config.ocm.software/v1alpha1
kind: ConfigData
metadata:
  name: ocm-config
...
localization:
- name: image # rule name
  file: deployment.yaml # target file for substitution
  image: spec.template.spec.containers[0].image # path in file to insert image name
  resource: # ocm resource from which to resolve the image location
    name: image

The localization section contains a list of rules that describe the substitutions the ocm-controller needs to perform to ensure that the Local copy of our image is deployed. OCM provides an identifier for each resource which can always be resolved to a specific storage location at which the resource can be accessed. This secret sauce makes it possible to automate air-gapped deployments using OCM.

We can examine the image resource to see precisely where the image can be accessed:

ocm get resources ghcr.io/phoban01//phoban.io/podinfo -c 6.3.5 image -owide

NAME  VERSION IDENTITY TYPE     RELATION ACCESSTYPE  ACCESSSPEC
image 6.3.5            ociImage external ociArtifact {"imageReference":"ghcr.io/stefanprodan/podinfo:6.3.5"}

Component Transfer

We can use the ocm CLI to transfer this public component into our “private” registry. Because we are simulating an air-gapped install, we instruct the ocm CLI to copy the resources along with the component metadata:

AIR_GAPPED_REGISTRY=ghcr.io/phoban01/air-gapped

ocm transfer component --copy-resources ghcr.io/phoban01//phoban.io/podinfo $AIR_GAPPED_REGISTRY

It will take few moments to complete the transfer. Once it is complete we can view the component in the air-gapped registry:

ocm get component ghcr.io/phoban01/air-gapped//phoban.io/podinfo

COMPONENT         VERSION PROVIDER
phoban.io/podinfo 6.2.3   phoban.io
phoban.io/podinfo 6.3.5   phoban.io

Let’s examine the image resource on the component in our private registry:

ocm get resources $AIR_GAPPED_REGISTRY//phoban.io/podinfo -c 6.3.5 image -owide

NAME  VERSION IDENTITY TYPE     RELATION ACCESSTYPE  ACCESSSPEC
image 6.3.5            ociImage external ociArtifact {"imageReference":"ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5"}

We can see that the image reference now points to an image stored in our air-gapped registry.

GitOps & Localization

Now that our component has been successfully transferred, let’s deploy it using GitOps.

We assume you have completed the Deploy Applications with OCM & GitOps guide and will use that repository as the starting point for our air-gapped deployment.

Because our air-gapped OCM repository is private, we need to provide credentials. This will enable the ocm-controller to retrieve components from the repository.

We can do this using a ServiceAccount. First, create an Kubernetes Secret to hold the credentials:

kubectl create secret docker-registry -n ocm-system ghcr-cred \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USER \
  --docker-password=$GITHUB_TOKEN

Then, create the ServiceAccount:

cat > ./components/service_account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: air-gapped-ops
  namespace: ocm-system
imagePullSecrets:
- name: ghcr-cred
EOF

Next, let’s modify the ComponentVersion manifest so that it points to our air-gapped OCM repository and references the ServiceAccount:

apiVersion: delivery.ocm.software/v1alpha1
kind: ComponentVersion
metadata:
  name: podinfo
  namespace: ocm-system
spec:
  interval: 1m0s
  component: phoban.io/podinfo
  version:
    semver: ">=v6.3.5"
  repository:
    url: ghcr.io/phoban01/air-gapped
  serviceAccountName: air-gapped-ops

Now we need to tell the ocm-controller to use the Localization rules we discussed earlier. To do this, we create a Localization Custom Resource:

cat > ./components/localization.yaml >>EOF
apiVersion: delivery.ocm.software/v1alpha1
kind: Localization
metadata:
  name: podinfo-deployment
  namespace: ocm-system
spec:
  interval: 5m
  sourceRef:
    kind: Resource
    name: podinfo-deployment # this is the podinfo deployment manifest resource we created previously
  configRef:
    kind: ComponentVersion
    name: podinfo
    resourceRef:
      name: config # here we reference the resource containing localization rules
EOF

You can see that we have used the existing Resource as the source for the Localization and have provided the localization rules using the spec.configRef field. The ocm-controller enables us to freely chain resources together in order to perform a sequence of transformations upon an OCM resource.

Because the output we want to deploy is now generated by the Localization CR rather than the Resource CR, we need to update our FluxDeployer:

apiVersion: delivery.ocm.software/v1alpha1
kind: FluxDeployer
metadata:
  name: podinfo
  namespace: ocm-system
spec:
  sourceRef:
    kind: Localization
    name: podinfo-deployment
  kustomizationTemplate:
    interval: 1m0s
    path: ./
    prune: true
    targetNamespace: default

Let’s commit, push, and reconcile these changes:

git add ./components

git commit -m "move to air-gapped repository"

git push

flux reconcile source git flux-system

Verification

Flux should now be reconciling the Localized manifest with image references pointing to our private OCM repository.

We can easily verify this using kubectl:

kubectl get deployment -n default podinfo -oyaml | grep image | xargs

image: ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5

To Be Continued

If we look closer, however, we will see that our application has not successfully rolled out:

kubectl get po -n default

NAME                       READY   STATUS             RESTARTS   AGE
podinfo-7b7d874bf8-xv75x   0/1     ImagePullBackOff   0          1m4s

If we filter the events we can see that Kubernetes cannot pull the image owing to missing credentials:

kubectl get events --field-selector involvedObject.kind=Pod

LAST SEEN   TYPE      REASON      OBJECT                         MESSAGE
7m31s       Normal    Scheduled   pod/podinfo-7b7d874bf8-xv75x   Successfully assigned default/podinfo-7b7d874bf8-xv75x to kind-control-plane
6m7s        Normal    Pulling     pod/podinfo-7b7d874bf8-xv75x   Pulling image "ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5"
6m6s        Warning   Failed      pod/podinfo-7b7d874bf8-xv75x   Failed to pull image "ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5": failed to resolve reference "ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
6m6s        Warning   Failed      pod/podinfo-7b7d874bf8-xv75x   Error: ErrImagePull
2m31s       Normal    BackOff     pod/podinfo-7b7d874bf8-xv75x   Back-off pulling image "ghcr.io/phoban01/air-gapped/stefanprodan/podinfo:6.3.5"
5m44s       Warning   Failed      pod/podinfo-7b7d874bf8-xv75x   Error: ImagePullBackOff

Check out our GitOps Driven Configuration of OCM Applications guide to see how we can use the ocm-controller to configure our application at runtime and solve exactly this kind of problem!

Conclusion

In this tutorial we have shown how we can automate the process of delivering software to air-gapped environments using the Open Component Model and Flux.

We have shown how the process of Localization is enabled via OCM and combined with GitOps delivers a seamless application deployment model suitable for any environment.

Prev
Component Subscription
Next
Best Practices