Skip to main content

kcp-aware Service Provider Runtime

Moritz Marby
Moritz Marby
OpenControlPlane Contributor

Context and Problem Statement​

The OpenControlPlane platform currently supports one way for a service provider to onboard tenants: a tenant creates a Service API object on the onboarding cluster, the service provider reconciler picks it up, orders cluster access (MCP + optional workload cluster), and installs the service.

There is no defined way for a service provider to expose its service into a kcp workspace. kcp (Kubernetes-like Control Plane) lets tenants work in isolated logical workspaces rather than on a shared cluster. A tenant in a kcp workspace should be able to request a service the same way a tenant on the onboarding cluster does - by creating a Service API object - and receive the same result.

At the same time, every service provider today duplicates the same generic reconcile plumbing: finalizer handling, ProviderConfig loading, cluster-access ordering, kubeconfig delivery, status management, and deletion sequencing. This duplication makes providers hard to maintain and makes adding kcp support per-provider prohibitively expensive.

This ADR proposes:

  1. A design for how service providers expose their service into kcp workspaces.
  2. A unified runtime abstraction that handles both the existing onboarding flow and the new kcp flow, so service providers only write install/delete logic.

Considered Options​

  • Unified runtime with a single provider seam for both onboarding and kcp modes
  • Per-provider kcp implementation (status quo - each provider copies and adapts kcp plumbing)

Decision Outcome​

Chosen option: "Unified runtime with a single provider seam", because it eliminates per-provider duplication, establishes kcp onboarding as a first-class platform capability, and reduces new provider development to only service-specific logic.

Core principle​

A service provider implements exactly one interface: install and delete. The runtime owns everything else - for both the onboarding cluster path and the kcp workspace path. From the provider's perspective, a service request looks identical regardless of where it came from.

Architecture​

1. Provider declares what it offers​

The service provider defines:

  • Its Service API type - the CRD a tenant creates to request the service (e.g. Kro, Crossplane). One type is used for both onboarding and kcp modes.
  • Its ProviderConfig type - platform-operator configuration (available versions, kcp settings, workload cluster preferences).
  • Its install/delete logic - a single Reconciler with two methods: CreateOrUpdate (install or update) and Delete.

Everything else is the runtime's responsibility.

2. How kcp onboarding works​

kcp onboarding is a new capability this ADR introduces. There is no existing implementation to migrate from. The flow is:

Provider side (one-time setup):

  1. The provider defines an APIExport in its kcp provider workspace, backed by an APIResourceSchema derived from the same Service API type used on the onboarding cluster. One API type, two delivery surfaces.
  2. The runtime provisions this APIExport at startup from the provider's ProviderConfig.

Tenant side:

  1. A tenant's workspace creates an APIBinding to the provider's APIExport.
  2. The Service API type becomes available in the tenant workspace.
  3. The tenant creates a Service API object in their workspace, just as they would on the onboarding cluster.

Runtime reconciliation:

  1. The runtime watches for Service API objects across all bound consumer workspaces via the APIExport virtual workspace (multicluster-runtime).
  2. When a Service API object appears in a workspace, the runtime:
    • Mints a scoped ServiceAccount token in the consumer workspace (via TokenRequest API) so the workload can reach back at the workspace.
    • Resolves (or orders) a workload cluster if the provider requested one.
    • Calls the provider's Reconciler.CreateOrUpdate with all credentials pre-resolved.
  3. The provider installs the service exactly as it would in standard mode - it receives an end user cluster client and kubeconfig, a workload cluster client and kubeconfig, and a stable tenant identity. No kcp-specific code in the provider.

3. Unified provider seam​

Both discovery paths (onboarding cluster and kcp workspace) funnel into the same two methods:

CreateOrUpdate(ctx, serviceAPIObject) -> (result, error)
Delete(ctx, serviceAPIObject) -> (result, error)

The runtime carries pre-resolved credentials and config on ctx, and the framework offers functions to retrieve them, e.g. cluster.WorkloadFromCtx(ctx).

The context carries:

  • MCPCluster - ready client for the end user cluster. In standard mode: the MCP control plane. In kcp mode: the tenant's kcp workspace. Same field, same type, either mode.
  • MCPKubeconfig - raw end user kubeconfig. Providers embed this into child resources (e.g. a HelmRelease whose worker needs KUBECONFIG pointing at the end user cluster).
  • WorkloadCluster - ready client for the workload cluster (nil if not requested or not yet granted). Workload clusters are shared: the platform may grant access to an existing cluster rather than provisioning a new one.
  • WorkloadKubeconfig - raw workload kubeconfig.
  • Identity - stable per-tenant identifier (MCP object key in standard mode, kcp logical cluster name in kcp mode). Used for deriving stable per-tenant resource names.
  • Mode - standard or kcp. Most providers ignore this; it exists for the rare case where install logic must genuinely differ.

4. ProviderConfig drives mode selection​

The ProviderConfig CRD is defined by the provider. Adding a kcp section activates kcp mode. No change to main.go code is required:

spec:
  versions:
    - name: v1.2.3
      chartURL: oci://...
  kcp:
    providerWorkspace: root:my-org:services
    kubeconfigSecret:
      name: kcp-kubeconfig
      namespace: my-provider-ns

When spec.kcp is absent the provider runs standard mode only. When present, both modes run in the same process (each with its own manager and leader election ID).

kcp compatibility depends on how the service deploys its worker.

There are two deployment patterns:

  • Workerless: The service worker runs on a separate workload cluster, with its kubeconfig pointed at the end user control plane or kcp workspace. The end user cluster is only used as an API server - no pods run on it. This pattern is kcp-compatible because kcp workspaces are API servers and can serve the service's API surface without running compute.

  • Classic (not kcp-compatible): The service worker runs directly on the end user control plane, on the same cluster the service is operating on. This pattern cannot support kcp because kcp workspaces have no compute - pods cannot be scheduled in a kcp workspace.

A provider using the classic pattern must explicitly opt out of kcp:

  • Omit the kcp field from its ProviderConfig CRD schema entirely, so platform operators cannot accidentally configure it.
  • Not supply a Provisionable or kcp wiring in its main.go.

The runtime will not start kcp mode unless the provider explicitly wires it up. kcp support is opt-in, not default.

5. Workload cluster ordering​

The provider declares at startup whether it needs a workload cluster. This maps directly to the two deployment patterns described above:

  • WorkloadCluster(true) (workerless - kcp-compatible): the runtime places a ClusterRequest + AccessRequest and waits until a cluster is granted before calling CreateOrUpdate. The service worker runs on this separate workload cluster with its kubeconfig pointing at the end user control plane or kcp workspace. The granted cluster may be shared with other tenants.
  • WorkloadCluster(false) (classic - not kcp-compatible): no workload cluster is ordered. The service worker runs directly on the end user control plane. This pattern cannot be used with kcp.

In kcp mode the runtime additionally mints a workspace ServiceAccount token so the workload controller can authenticate back to the end user workspace.

6. Provider declares its kcp API surface (kcp mode only)​

For kcp mode the provider supplies a Provisionable implementation that defines:

  • The APIResourceSchema (the kcp equivalent of a CRD) for the Service API, derived from the same Go type as the onboarding CRD.
  • The APIExport (name, resource list, permission claims).
  • The watched GVK for the multicluster controller.

The runtime calls Provision once at startup. After that the provider never touches kcp infrastructure again - reconciliation is driven by workspace events, resolved by the runtime, and delivered as a standard ClusterContext.

7. Deletion​

On deletion the runtime:

  1. Calls provider.Delete with the same resolved ClusterContext.
  2. Removes the workload cluster access (ClusterRequest + AccessRequest, or kcp workspace SA/RBAC).
  3. Removes the end user cluster access (MCP AccessRequest or workspace token cleanup).
  4. Removes the finalizer.

The provider is not responsible for any access cleanup.

What the provider does NOT need to write​

  • Finalizer add/remove logic.
  • ProviderConfig loading or watching.
  • kcp multicluster manager setup.
  • APIExport provisioning lifecycle (schema immutability, upsert).
  • Workspace token minting or refresh.
  • Workload cluster ordering and wait loop.
  • Kubernetes status patching. The provider reports status content; the runtime writes it to the object.
  • Mode detection or branching.
  • Deletion sequencing (access teardown before finalizer removal).

Non-goals​

  • The provider's install mechanism (Flux, Helm SDK, raw manifests) is not specified.
  • Multi-tenancy isolation guarantees on shared workload clusters.
  • Cross-provider APIExport composition.
  • The specific fields of any provider's ProviderConfig.

Open Questions​

  1. CRD install ownership: Should the runtime own the init step (install the Service API CRD on the onboarding cluster + register the GVK at the ServiceProvider), or do providers retain a thin init command for custom pre-flight steps?

  2. APIBinding activation: Should the runtime watch APIBinding objects and trigger a reconcile when a workspace binds the APIExport (proactive first-install), or is watching the Service API objects across virtual workspaces sufficient (reactive)?

  3. ProviderConfig hot-reload in kcp mode: When ProviderConfig changes (e.g. new chart version), should the runtime re-reconcile all active workspaces? Proposal: yes, same fan-out behavior as standard mode.

  4. Singleton enforcement: In kcp mode, the runtime enforces that only one Service API object per workspace is active. Default policy: oldest wins - if multiple objects exist, the oldest is reconciled and newer ones are rejected with a status message. In standard mode this is not needed as the object is named after the corresponding cluster, ensuring uniqueness by convention.

Consequences​

  • Good, because service provider development is reduced to only service-specific logic.
  • Good, because kcp onboarding becomes a first-class platform capability, not a per-provider burden.
  • Good, because improvements to cluster-access, token handling, and status management benefit all providers immediately.
  • Good, because existing providers can migrate incrementally: adopt the unified runtime first (standard mode, no behavior change), add kcp support later.
  • Bad, because the runtime becomes a more complex shared dependency - changes to the runtime interface affect all providers.

Discoverable Components

Moritz Marby
Moritz Marby
OpenControlPlane Contributor
Valentin Gerlach
Valentin Gerlach
OpenControlPlane Contributor
Maximilian Techritz
OpenControlPlane Contributor

This ADR defines how consumers discover the location of OCI artifacts for different services.

For example, a service provider such as service-provider-crossplane, running inside the OpenControlPlane platform, needs to know where to fetch the Crossplane Helm chart and container image from. This ADR defines the solution.

Terminology​

For full background on the Open Component Model, see ocm.software. The entries below cover only what is needed to read this ADR.

  • OCM - Open Component Model. Secure delivery for sovereign clouds - deliver and deploy your software securely, anywhere, at any scale. A specification and toolset for describing software components and their resources (images, charts, blobs) in a transport-agnostic way.
  • OCM Component - a single, versioned unit described by OCM. It carries a set of resources (e.g. an OCI image, a Helm chart) and may reference other components via componentReferences.
  • Service OCM Component - an OCM component that represents a single service the platform can install (e.g. flux, crossplane). It lists all OCM resources that this service needs at a given version (chart, controller images, etc.).
  • Umbrella OCM Component - an OCM component that contains no resources of its own and only carries componentReferences to other OCM components. Used to group a set of service OCM components into one entry point. In this ADR, the Releasechannel is the umbrella OCM component.
  • Platform cluster - the cluster on which the OpenControlPlane platform runs its ServiceProvider, PlatformService, and ClusterProvider controllers.
  • Consumer - in this ADR, a technical consumer of Discoverable Components. The ServiceProvider, PlatformService, and ClusterProvider controllers running on the platform cluster are the consumers: they resolve where to fetch their OCI artifacts from.

Current state​

The platform cluster runs three types of controllers, each operating at a different scope - see Plug & Play for a full overview:

  • Service Providers add functionality to individual ControlPlanes (e.g. GitOps tooling, cloud provider APIs).
  • Platform Services add functionality to the OpenControlPlane environment as a whole (e.g. network services, audit logs).
  • Cluster Providers handle the dynamic creation and deletion of Kubernetes clusters behind a homogeneous interface.

Service Providers​

Each Service Provider is deployed by creating a ServiceProvider resource in which the OCI image URL of the provider is specified directly:

# From service-provider-flux
apiVersion: openmcp.cloud/v1alpha1
kind: ServiceProvider
metadata:
  name: flux
  namespace: openmcp-system
spec:
  image: ghcr.io/openmcp-project/images/service-provider-flux:v0.1.0

The ProviderConfig resource of a Service Provider defines which versions of the managed service are available. For each version, all OCI artifact URLs (Helm chart and container image) as well as pull secrets for private registries must be specified manually:

# From service-provider-crossplane
apiVersion: crossplane.services.open-control-plane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: default
spec:
  versions:
    - version: v2.0.2
      chart:
        url: "ghcr.io/openmcp-project/openmcp/charts/crossplane:2.0.2"
        secretRef:
          name: ghcr
      image:
        url: "ghcr.io/openmcp-project/openmcp/images/crossplane:2.0.2"
        secretRef:
          name: xyz
    - version: v1.20.0
      chart:
        url: "ghcr.io/openmcp-project/openmcp/charts/crossplane:1.20.0"
        secretRef:
          name: ghcr
      image:
        url: "ghcr.io/openmcp-project/openmcp/images/crossplane:1.20.0"
        secretRef:
          name: xyz
  providers:
    availableProviders:
      - name: provider-kubernetes
        package: xpkg.upbound.io/upbound/provider-kubernetes
        versions:
          - v0.16.0
          - v0.15.0
    imagePullSecretRefs:
      - name: secretforprivateproviders
# From service-provider-flux
apiVersion: flux.services.open-control-plane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: flux
spec:
  versions:
    - version: "2.8.3"
      chartVersion: "2.18.2"
      chartUrl: "oci://ghcr.io/fluxcd-community/charts/flux2"
      chartPullSecret: "chart-registry-credentials"
      values:
        imagePullSecrets:
          - name: "image-registry-credentials"
        helmController:
          image: my-registry.example.com/fluxcd/helm-controller
          tag: v1.5.3
        sourceController:
          image: my-registry.example.com/fluxcd/source-controller
          tag: v1.8.1

Platform Services​

When installing a Platform Service, the OCI image URL of the controller is specified directly in the PlatformService resource:

# From platform-service-gateway
apiVersion: openmcp.cloud/v1alpha1
kind: PlatformService
metadata:
  name: gateway
spec:
  image: ghcr.io/openmcp-project/images/platform-service-gateway:v0.1.0

A Platform Service may additionally expose a service-specific configuration resource in which image URLs and Helm chart locations for its dependencies must be configured manually:

# From platform-service-gateway - GatewayServiceConfig
apiVersion: gateway.openmcp.cloud/v1alpha1
kind: GatewayServiceConfig
metadata:
  name: gateway
spec:
  envoyGateway:
    images:
      proxy: "ghcr.io/openmcp-project/components/github.com/openmcp-project/openmcp/images/envoy-proxy:distroless-v1.36.2"
      gateway: "ghcr.io/openmcp-project/components/github.com/openmcp-project/openmcp/images/envoy-gateway:v1.5.4"
      rateLimit: "ghcr.io/openmcp-project/components/github.com/openmcp-project/openmcp/images/envoy-ratelimit:99d85510"
    chart:
      url: "oci://ghcr.io/openmcp-project/components/github.com/openmcp-project/openmcp/charts/envoy-gateway"
      tag: "1.5.4"
  clusters:
    - selector:
        matchPurpose: platform
    - selector:
        matchPurpose: workload
  dns:
    baseDomain: dev.openmcp.example.com

Cluster Providers​

When installing a Cluster Provider, the OCI image URL is specified directly in the ClusterProvider resource:

# From cluster-provider-gardener
apiVersion: openmcp.cloud/v1alpha1
kind: ClusterProvider
metadata:
  name: gardener
spec:
  image: "ghcr.io/openmcp-project/images/cluster-provider-gardener:v0.2.0"

Requirements​

A centrally managed, platform-provided mechanism for resolving the location of OCI images must satisfy the following.

  • Multiple versions of an artifact must be storable.
  • The solution must support local development.
  • Pull secrets for an artifact must be discoverable alongside the artifact itself, not configured separately by every consumer. In OCM, the controllers propagate a parent component's secret reference down to every child resource automatically, so each resolved resource carries its own pull-secret reference. For the Artifacts API the same concept applies, but expressed in-cluster: the Artifact resource carries its pull-secret references directly in spec.pullsecrets, populated by the translation layer when it materialises the OCM content.

Flows That Must Be Supported​

  1. Retrieve all versions of an artifact, e.g. crossplane-chart.
  2. Retrieve the pull secrets for artifact crossplane-chart at version v1.10.
  3. Different versions of an artifact may originate from different registries.

Scope​

This ADR covers platform-internal artifact discovery only. It is sufficient on its own for ServiceProvider, PlatformService, and ClusterProvider to resolve where to fetch their OCI artifacts from. The user-facing API for selecting a service version is decided in a follow-up ADR and is not required to implement what is proposed here.

In Scope​

  • OCI images and artifacts only.
  • How ServiceProvider, PlatformService, and ClusterProvider discover what artifacts are available and where to fetch them from.
  • Defining the structure of the Releasechannel OCM component.

Out of Scope​

  • The user-facing API for selecting a service version (e.g. how an end user requests "Flux v2.2.0"). To be discussed in a follow-up ADR.
  • How a Releasechannel is built, signed, or published. Only its structure is defined here.
  • Lifecycle and rollout of Releasechannel versions.

Releasechannel OCM Component​

A single umbrella OCM component, Releasechannel, acts as the entry point for all artifacts the platform knows about. The cluster is configured with one piece of information - the Releasechannel version to consume - and discovers everything else from its content. In practice this is expressed as an OCM Component resource managed by the OCM controllers, pointing at the published Releasechannel and pinning its version.

Structure​

Releasechannel is a pure pointer component. It contains no resources of its own; it only carries componentReferences to per-service OCM components.

Each componentReference is a direct (name, version) reference to another OCM component. There is no grouping layer in between, so multiple versions of the same service appear as separate entries:

Releasechannel v1.8.9
├── componentReference: flux       v2.2.0
├── componentReference: flux       v2.1.0
├── componentReference: crossplane v2.0.0
└── …

Each per-service OCM component lists its OCM resources - for example, the flux component at v2.2.0 carries:

  • helm chart
  • helm-controller image
  • kustomize-controller image
  • source-controller image
  • notification-controller image
  • …

A consumer that picks "flux at version v2.2.0" resolves through the matching componentReference into the concrete set of resources (chart + controller images) that this version of flux ships.

Why this structure​

The customer installs a service component version, not individual resource versions. Saying "install flux v2.2.0" must be enough for the platform to know which chart and which controller images to pull. OCM performs that resolution natively once the components are structured this way; the platform does not need its own version-mapping layer.

Listing each version as its own componentReference also makes "which versions of flux are available?" answerable by enumerating the references of the current Releasechannel.

Why OCM​

The choice of OCM as the underlying mechanism is deliberate. OCM gives the platform two properties that are hard to get otherwise:

  • Localization / transport. A whole Releasechannel, including every referenced service OCM component and every resource (chart, image) it transitively pulls in, can be transported to a different OCI registry as a single unit using OCM tooling. Consumers then point at the new registry and continue to operate without any other change.
  • Air-gapped deployments. The same property makes air-gapped environments approachable: mirror the Releasechannel once, and all artifacts the platform needs are reachable inside the isolated network.

Building a comparable mechanism on top of plain OCI references would mean re-implementing transport, signing, and integrity checks that OCM already provides.

These same properties are the reason the whole OpenControlPlane platform is itself bundled and released as one umbrella OCM component - see openmcp-project/openmcp. Using OCM for the Releasechannel keeps the discovery layer aligned with how the platform itself is delivered.

Scope of consumers​

Although the ADR is motivated by ServiceProvider resolving service artifacts, the concept is intentionally open: any part of the platform that needs to know where an OCI artifact lives can infer it from the same Releasechannel. In particular, the install image of a PlatformService or a ClusterProvider can be discovered the same way as a service's chart or controller image. The Releasechannel is the single source of truth for "what is installable, in which versions, and where to fetch it from".

Hosting​

In production, a Releasechannel is published to an OCI registry. The cluster only needs to know the registry URL and Releasechannel version; everything else is discovered in-cluster from the resolved component graph.

For local development, KIND (Kubernetes IN Docker) based e2e, or quick try-out scenarios, requiring an OCI registry to host a Releasechannel is a hard external dependency. The two approaches below address this differently.

Proposal​

The Releasechannel defined above is the single source of truth for what is installable, in which versions, and where to fetch it from. This is common to both approaches below; they differ only in how consumers read from the Releasechannel:

  • Approach A - Direct OCM consumption. Consumers query the Releasechannel through OCM controllers. Requires upstream OCM controller features that do not exist yet.
  • Approach B - Artifacts API translation layer. The platform translates the resolved OCM content into Artifact Kubernetes resources. Consumers read those instead of OCM directly. Optional addition, independent of the upstream OCM roadmap.

Approach A - Direct OCM consumption​

Consumers (ServiceProvider, PlatformService, ClusterProvider) read the Releasechannel through OCM controllers directly. No intermediate Kubernetes API on the platform side.

Required OCM controller capabilities​

This approach depends on two capabilities of the upstream OCM controllers that do not exist in their current form:

  1. Discovery from an umbrella component. Today, OCM controllers require both the component name and the resource name to be specified explicitly (1:1 lookup). Approach A needs the controllers to enumerate componentReferences of a Releasechannel and, per service component, enumerate the available resources in-cluster. Tracked upstream in open-component-model/ocm-project#1153 ("Make components and resources discoverable through ocm-k8s-toolkit").
  2. Local-YAML fallback for offline scenarios. A way to back the lookup with a plain YAML file pointing at upstream registries, so that local development and e2e tests do not require an OCI registry hosting a full Releasechannel. Discussed with OCM maintainers; no public issue yet.

Approach B - Artifacts API translation layer​

An optional addition on top of Approach A. The platform introduces a new Kubernetes API that materialises the resolved content of the Releasechannel as in-cluster resources. Consumers read those resources instead of querying OCM directly.

The Releasechannel remains the source of truth in production: a translation layer reads it via OCM and creates the corresponding Artifact resources. In local or e2e setups, Artifact resources can be applied directly without an OCI registry, removing the dependency on the upstream OCM features listed in Approach A.

Artifacts​

A new resource on the platform cluster called Artifact. Each Artifact points to exactly one OCI image. The CRD is cluster-scoped. Because Artifacts live on the platform cluster, they are not reachable by end customers.

Artifact resources are created in one of two ways, depending on environment:

  • Local / e2e. Artifacts are applied manually to the (KIND) cluster. No OCM registry or translation layer is involved.
  • Production. A translation layer materialises Artifacts from the resolved OCM components on a configurable interval. On each reconcile it overwrites any local edits, keeping the Releasechannel the single source of truth.

Two version concepts appear in the examples below and must not be confused:

  • componentVersion is the version of the OCM service component the user selects, e.g. flux v2.8.8. This is what an Artifact's spec.version carries. Picking a componentVersion resolves into a fixed set of Artifact resources.
  • artifactVersion is the tag of the underlying OCI image, or in OCM terms the resource version. It is what ends up in the artifact's url (e.g. :v1.5.5 for the flux helm-controller image while the chart is at :v2.18.4).

A single componentVersion therefore typically pins multiple, independent artifactVersions. Consumers select by componentVersion; the corresponding artifactVersions come along automatically.

kind: Artifact
metadata:
  name: flux-chart-v2.8.8
spec:
  name: flux-chart
  version: v2.8.8
  url: example-reg.io/charts/flux:v2.18.4@sha256:23426aabcd...
  pullsecrets:                      # if from authenticated registry
    - name: registry-pull-secret    # required. `Artifact` is cluster-scoped, so namespace is mandatory.
      namespace: openmcp-system

The metadata.name of an Artifact is arbitrary. As a best practice, use <artifact-name>-<componentVersion> (e.g. flux-chart-v2.8.8) for readability. Artifacts are intended to be retrieved by their spec fields using a fieldSelector. For this to work, the Artifact CRD must declare spec.name and spec.version in spec.versions[*].selectableFields (see CRD selectable fields).

The spec.version field on an Artifact is the componentVersion, not the OCI tag of the underlying image. The OCI tag (the artifactVersion) is encoded in spec.url.

For example, flux at componentVersion v2.8.8 is a component that consists of one chart and multiple controller images at independent artifactVersions. The resulting Artifact resources are:

kind: Artifact
metadata:
  name: flux-chart-v2.8.8
spec:
  name: flux-chart
  version: v2.8.8
  url: example-reg.io/charts/flux:v2.18.4@sha256:23426aabcd...
  pullsecrets:                      # if from authenticated registry
    - name: registry-pull-secret
      namespace: openmcp-system
---
kind: Artifact
metadata:
  name: flux-helm-controller-v2.8.8
spec:
  name: flux-helm-controller
  version: v2.8.8
  url: ghcr.io/fluxcd/helm-controller:v1.5.5@sha256:23426aabcd...
  pullsecrets:                      # if from authenticated registry
    - name: registry-pull-secret
      namespace: openmcp-system
---
kind: Artifact
metadata:
  name: flux-kustomize-controller-v2.8.8
spec:
  name: flux-kustomize-controller
  version: v2.8.8
  url: ghcr.io/fluxcd/kustomize-controller:v1.8.5@sha256:23426aabcd...
  pullsecrets:                      # if from authenticated registry
    - name: registry-pull-secret
      namespace: openmcp-system
---
kind: Artifact
metadata:
  name: flux-source-controller-v2.8.8
spec:
  name: flux-source-controller
  version: v2.8.8
  url: ghcr.io/fluxcd/source-controller:v1.8.5@sha256:23426aabcd...
  pullsecrets:                      # if from authenticated registry
    - name: registry-pull-secret
      namespace: openmcp-system
# ... additional controller Artifacts (image-automation, image-reflector, notification, ...) follow the same shape.

All Artifact resources share the same componentVersion v2.8.8 while carrying independent artifactVersions in their url. A consuming entity can therefore resolve: "for component version v2.8.8, pull chart tag v2.18.4, helm-controller tag v1.5.5, kustomize-controller tag v1.8.5, source-controller tag v1.8.5, ..."

Discoverable Artifacts

ProviderConfig Example​

Every ServiceProvider has a ProviderConfig resource. To inform the ServiceProvider how to locate the relevant Artifact resources, selectors are defined in the ProviderConfig.

Multiple componentVersions of the same service coexist as independent Artifact sets on the platform cluster. A single ProviderConfig selects across all of them via fieldSelector on spec.name; the end user then picks the concrete componentVersion per ControlPlane (out of scope for this ADR).

Example: crossplane at componentVersion v1.20.1 and v2.0.2. Both versions ship a matching chart and image artifactVersion, so eight Artifact resources exist on the platform cluster:

kind: Artifact
metadata:
  name: crossplane-chart-v1.20.1
spec:
  name: crossplane-chart
  version: v1.20.1
  url: example-reg.io/charts/crossplane:v1.20.1@sha256:23426aabcd...
---
kind: Artifact
metadata:
  name: crossplane-image-v1.20.1
spec:
  name: crossplane-image
  version: v1.20.1
  url: xpkg.crossplane.io/crossplane/crossplane:v1.20.1@sha256:23426aabcd...
---
kind: Artifact
metadata:
  name: crossplane-chart-v2.0.2
spec:
  name: crossplane-chart
  version: v2.0.2
  url: example-reg.io/charts/crossplane:v2.0.2@sha256:23426aabcd...
---
kind: Artifact
metadata:
  name: crossplane-image-v2.0.2
spec:
  name: crossplane-image
  version: v2.0.2
  url: xpkg.crossplane.io/crossplane/crossplane:v2.0.2@sha256:23426aabcd...

A single ProviderConfig selects both versions by matching on spec.name only:

apiVersion: crossplane.services.open-control-plane.io/v1alpha1
kind: ProviderConfig
spec:
  chart:
    selectors:
      matchFields:
        - artifactName: crossplane-chart
  image:
    selectors:
      matchFields:
        - artifactName: crossplane-image
  providers:
    selectors:
      matchLabels:
        crossplane.services.open-control-plane.io/type: "provider"
  functions:
    selectors:
      matchLabels:
        crossplane.services.open-control-plane.io/type: "functions"

Each selector resolves to multiple Artifact resources (one per componentVersion). The consumer picks the concrete spec.version per ControlPlane at request time.

Pros and Cons​

Both approaches share the Releasechannel OCM component as the source of truth. The trade-off is purely in the consumption path.

Approach A - Direct OCM consumption​

Pros

  • Single source of truth, single read path: consumers resolve the Releasechannel directly via OCM.
  • No additional CRD or controller logic to maintain on the platform side.
  • Lowest overall complexity once the upstream OCM features are in place.

Cons

  • Blocked on upstream OCM controller features that do not yet exist (umbrella-component discovery, see open-component-model/ocm-project#1153 "Make components and resources discoverable through ocm-k8s-toolkit"; local-YAML fallback for offline use).
  • Local development, KIND-based e2e, and try-out scenarios are only viable once the local-YAML fallback exists upstream.
  • Platform's delivery timeline is coupled to the OCM upstream roadmap.

Approach B - Artifacts API translation layer​

Pros

  • Decouples the platform from the upstream OCM roadmap: ships independently of open-component-model/ocm-project#1153 ("Make components and resources discoverable through ocm-k8s-toolkit") and the local-YAML fallback.
  • Local development and e2e tests work by applying Artifact resources directly into a KIND cluster - no OCI registry, no OCM controller features required.
  • Consumers see a plain Kubernetes API with selectors, familiar to anyone working with CRDs.

Cons

  • Adds a CRD that the platform must own and maintain on top of the OCM-based source of truth.
  • Requires a translation layer from resolved OCM components to Artifact resources, which is additional code to write, test, and operate.
  • Increases overall complexity: the platform maintains both the API and the OCM-to-Artifact bridge instead of relying on OCM end-to-end.
  • Two representations of the same data in production raise the surface for drift and bugs.

Decision​

The team decided on Approach A - Direct OCM consumption.

Consumers (ServiceProvider, PlatformService, ClusterProvider) resolve the Releasechannel directly via OCM controllers, with no intermediate Kubernetes API on the platform side. This keeps a single source of truth and a single read path, avoids maintaining an additional CRD and translation layer, and keeps the discovery layer aligned with how the platform itself is delivered as an umbrella OCM component.

This choice depends on the upstream OCM controller capabilities described above - umbrella-component discovery (tracked in open-component-model/ocm-project#1153) and the local-YAML fallback for offline scenarios.

Autonomous Platform Clusters

Valentin Gerlach
Valentin Gerlach
OpenControlPlane Contributor
Moritz Marby
Moritz Marby
OpenControlPlane Contributor

The goal of this proposal is to streamline and simplify the bootstrapping and lifecycle management of the OpenControlPlane platform for platform operators and contributors.

Current state​

The current setup uses the openmcp-bootstrapper CLI to bootstrap a production installation. Before running it, a platform owner must manually provision a target Kubernetes cluster, set up a non-empty Git repository, and ensure access to the OCI registry. The bootstrapper then pulls component versions and manifest templates from the OCI registry, generates and writes rendered manifests into the Git repository, and deploys FluxCD to the cluster pointing at that repository. From that point on, FluxCD continuously reconciles the cluster state with what is in Git.

This means the Git repository is a required external dependency that must be maintained alongside the cluster. The manifests written into it are generated and templated (similar to Helm output), making them difficult to read, understand, or modify by hand. Any configuration change requires re-running the bootstrapper or editing large generated YAML files directly.

Verifying the setup is also a manual process: a platform owner must check that FluxCD's GitRepository and Kustomization resources are in a Ready state, confirm that core components are running in the openmcp-system namespace, and create a test ControlPlane resource to validate end-to-end functionality.

Learnings​

Operating the platform across multiple landscapes has shown that the Git repository introduces meaningful overhead. Because manifests are generated from a single large configuration file, they are not easy to inspect or adjust by hand, and any change requires going back through the bootstrapper. This adds friction during initial setup, local development, and for smaller teams who want to operate a landscape without the need for complex configurations.

Lifecycle management is currently spread across three layers: the bootstrapper generates and writes manifests before FluxCD takes over, FluxCD manages the base operators and infrastructure, and the different OpenControlPlane operators handle service providers, platform services, and cluster providers. This split has served its purpose but means there is no single place where the full operational state of a platform is described.

A related pattern is that deployment configuration lives separately from the release artifacts themselves. Platform Owners need to follow external guides to understand how to install or upgrade a given version, rather than the components carrying that information with them.

These observations point toward a model where the cluster itself holds and reconciles its own configuration, components ship their own deployment instructions, and the overall setup is simpler to get started with regardless of scale.

Desired state​

The core idea is that a platform cluster should be autonomous: configuration and landscape description are stored in-cluster rather than in an external Git repository. This allows the cluster to operate independently, even in air-gapped environments or during network disruptions, and to detect and repair configuration drift on its own.

In-cluster configuration should not depend on a specific delivery mechanism. FluxCD remains a valid option for bringing configuration into the cluster, but it should not be a hard requirement. The platform should be agnostic toward how configuration arrives.

Each component should ship its own deployment configuration as part of its release artifact. This means an operator installing or upgrading a version gets everything they need from the artifact itself, without having to consult a separate guide.

The setup experience should scale in both directions: straightforward enough for a single developer running a local environment, and manageable enough for a team operating multiple landscapes without needing dedicated tooling expertise. Local and production setups should follow the same model.

Finally, the platform should provide clear, machine-readable status reporting - both for the overall platform health and for individual components - so operators have a reliable way to understand what is running and whether it is healthy.

Terms​

Component​

A component in OpenControlPlane is a discrete part of the platform. Examples include the openmcp-operator, platform services, cluster providers, and service providers.

Solution Proposal​

Distribution Configuration in Releases​

Every component in OpenControlPlane ships its own ResourceGraphDefinition as part of its OCM component artifact. Each RGD describes how to install that component into the cluster it is deployed on.

Example for the openmcp-operator:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: openmcp-operator
  annotations:
    kro.run/allow-breaking-changes: "true"
spec:
  # kro uses this simple schema to create your CRD schema and apply it
  # The schema defines what users can provide when they instantiate the RGD (create an instance).
  schema:
    apiVersion: v1alpha1
    group: bootstrap.open-control-plane.io
    kind: OpenMCPOperator
    spec:
      ocmRepoName: string
      environment: string | default="main"
      configData: string
    # status:

  # Define the resources this API will manage.
  resources:
    - id: ocmrepo
      externalRef:
        apiVersion: delivery.ocm.software/v1alpha1
        kind: Repository
        metadata:
          name: ${schema.spec.ocmRepoName}

    - id: component
      readyWhen:
        - ${component.status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')}
      template:
        apiVersion: delivery.ocm.software/v1alpha1
        kind: Component
        metadata:
          name: openmcp-operator
        spec:
          repositoryRef:
            name: ${ocmrepo.metadata.name}
          component: github.com/openmcp-project/openmcp-operator
          semver: v0.19.1
          interval: 1m

    - id: resourceImage
      readyWhen:
        - ${resourceImage.status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')}
      template:
        apiVersion: delivery.ocm.software/v1alpha1
        kind: Resource
        metadata:
          name: openmcp-operator-image
        spec:
          componentRef:
            name: ${component.metadata.name}
          resource:
            byReference:
              resource:
                name: openmcp-operator-image
          additionalStatusFields:
            oci: resource.access.toOCI()
    - id: serviceaccount
      template:
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: openmcp-operator

    - id: clusterrolebinding
      template:
        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRoleBinding
        metadata:
          name: ${serviceaccount.metadata.namespace}:openmcp-operator
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: cluster-admin
        subjects:
          - kind: ServiceAccount
            name: ${serviceaccount.metadata.name}
            namespace: ${serviceaccount.metadata.namespace}

    - id: configmap
      template:
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: openmcp-operator-scheduler-config
        data:
          config: ${schema.spec.configData}

    - id: deployment
      readyWhen:
        - ${deployment.status.readyReplicas == deployment.spec.replicas && deployment.status.availableReplicas == deployment.spec.replicas}
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          labels:
            app: openmcp-operator
          name: openmcp-operator
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: openmcp-operator
          template:
            metadata:
              labels:
                app: openmcp-operator
              annotations:
                prometheus.io/scrape: "true"
                prometheus.io/port: "8080"
                prometheus.io/path: "/metrics"
            spec:
              serviceAccountName: ${serviceaccount.metadata.name}
              imagePullSecrets: "${resourceImage.status.effectiveOCMConfig.map(s, {'name': s.name})}"
              initContainers:
                - name: openmcp-operator-init
                  image: ${resourceImage.status.resource.access.imageReference}
                  args:
                    - init
                    - --environment=${schema.spec.environment}
                    - --config=/etc/config/openmcp-operator/config
                    - --provider-name=managedcontrolplane
                  env:
                    - name: POD_NAME
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.name
                    - name: POD_NAMESPACE
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                    - name: POD_IP
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: status.podIP
                    - name: POD_SERVICE_ACCOUNT_NAME
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: spec.serviceAccountName
                  resources:
                    requests:
                      cpu: 100m
                      memory: 128Mi
                    limits:
                      cpu: 1000m
                      memory: 1024Mi
                  terminationMessagePath: /dev/termination-log
                  terminationMessagePolicy: File
                  volumeMounts:
                    - mountPath: /etc/config/openmcp-operator
                      name: openmcp-operator-scheduler-config
                      readOnly: true
              containers:
                - name: openmcp-operator
                  image: ${resourceImage.status.resource.access.imageReference}
                  args:
                    - run
                    - --environment=${schema.spec.environment}
                    - --config=/etc/config/openmcp-operator/config
                    - --provider-name=managedcontrolplane
                    - --metrics-bind-address=:8080
                    - --metrics-secure=false
                  env:
                    - name: POD_NAME
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.name
                    - name: POD_NAMESPACE
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                    - name: POD_IP
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: status.podIP
                    - name: POD_SERVICE_ACCOUNT_NAME
                      valueFrom:
                        fieldRef:
                          apiVersion: v1
                          fieldPath: spec.serviceAccountName
                  resources:
                    requests:
                      cpu: 100m
                      memory: 128Mi
                    limits:
                      cpu: 1000m
                      memory: 1024Mi
                  terminationMessagePath: /dev/termination-log
                  terminationMessagePolicy: File
                  ports:
                    - name: metrics-http
                      containerPort: 8080
                      protocol: TCP
                  volumeMounts:
                    - mountPath: /etc/config/openmcp-operator
                      name: openmcp-operator-scheduler-config
                      readOnly: true
              volumes:
                - name: openmcp-operator-scheduler-config
                  configMap:
                    name: ${configmap.metadata.name}

The RGD can then be installed using the OCM controllers. For the example above, this would look as follows:

apiVersion: delivery.ocm.software/v1alpha1
kind: Repository
metadata:
  name: openmcp-repository
spec:
  repositorySpec:
    baseUrl: ghcr.io/openmcp-project/components
    type: OCIRegistry
  interval: 1m
---
apiVersion: delivery.ocm.software/v1alpha1
kind: Component
metadata:
  name: openmcp-operator-rgd
spec:
  repositoryRef:
    name: openmcp-repository
  component: github.com/openmcp-project/openmcp-operator-rgd
  semver: v0.0.2
  interval: 1m
---
apiVersion: delivery.ocm.software/v1alpha1
kind: Resource
metadata:
  name: openmcp-operator-rgd
spec:
  componentRef:
    name: openmcp-operator-rgd
  resource:
    byReference:
      resource:
        name: rgd
---
# Once the RGD is installed in the cluster, create an instance to install the OpenMCP Operator.
apiVersion: delivery.ocm.software/v1alpha1
kind: Deployer
metadata:
  name: openmcp-operator-rgd
spec:
  resourceRef:
    name: openmcp-operator-rgd
apiVersion: bootstrap.open-control-plane.io/v1alpha1
kind: OpenMCPOperator
metadata:
  name: default
spec:
  ocmRepoName: openmcp-repository
  environment: local
  configData: |
    managedControlPlane:
      mcpClusterPurpose: mcp-worker
      reconcileMCPEveryXDays: 7
    scheduler:
      scope: Cluster
      purposeMappings:
        mcp:
          template:
            spec:
              profile: kind
              tenancy: Exclusive
        mcp-worker:
          template:
            spec:
              profile: kind
              tenancy: Exclusive
        platform:
          template:
            metadata:
              labels:
                clusters.openmcp.cloud/delete-without-requests: "false"
            spec:
              profile: kind
              tenancy: Shared
        onboarding:
          template:
            metadata:
              labels:
                clusters.openmcp.cloud/delete-without-requests: "false"
            spec:
              profile: kind
              tenancy: Shared
        workload:
          tenancyCount: 20
          template:
            spec:
              profile: kind
              tenancy: Shared

Once this is applied, the OpenMCP Operator is installed. When the OCM component is upgraded, the operator is automatically updated without any configuration change.

A semver filter can be defined on the OCM component resource to automatically update to new minor versions.

The core principle is that all components ship their deployment configuration alongside their release artifacts using RGDs. This enables loosely coupled deployments and makes it possible to bundle components into distributions (see the next section).

Distributions​

A distribution bundles a set of components into a single top-level RGD, so the whole platform is described as one resource graph and bootstrapped from a handful of Kubernetes resources.

The distribution RGD does not repeat each component's install logic. Instead it instantiates the per-component RGDs (OpenMCPOperator, ClusterProviderKind, PlatformService, ...), passing each the OCM repository to pull from. Each instance triggers its own component's RGD, which handles the actual install as shown in the previous section.

The following example shows a Kind distribution of OpenControlPlane.

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: open-control-plane-platform
  annotations:
    kro.run/allow-breaking-changes: "true"
spec:
  # kro uses this simple schema to create your CRD schema and apply it
  # The schema defines what users can provide when they instantiate the RGD (create an instance).
  schema:
    apiVersion: v1alpha1
    group: bootstrap.open-control-plane.io
    kind: OpenControlPlanePlatform
    spec:
      ocmRepoName: string
    # status:

  # Define the resources this API will manage.
  resources:
    - id: ocmrepo
      externalRef:
        apiVersion: delivery.ocm.software/v1alpha1
        kind: Repository
        metadata:
          name: ${schema.spec.ocmRepoName}

    - id: openmcpoperator
      template:
        apiVersion: bootstrap.open-control-plane.io/v1alpha1
        kind: OpenMCPOperator
        metadata:
          name: default
        spec:
          ocmRepoName: ${ocmrepo.metadata.name}
          environment: local
          configData: |
            managedControlPlane:
              mcpClusterPurpose: mcp-worker
              reconcileMCPEveryXDays: 7
            scheduler:
              scope: Cluster
              purposeMappings:
                mcp:
                  template:
                    spec:
                      profile: kind
                      tenancy: Exclusive
                mcp-worker:
                  template:
                    spec:
                      profile: kind
                      tenancy: Exclusive
                platform:
                  template:
                    metadata:
                      labels:
                        clusters.openmcp.cloud/delete-without-requests: "false"
                    spec:
                      profile: kind
                      tenancy: Shared
                onboarding:
                  template:
                    metadata:
                      labels:
                        clusters.openmcp.cloud/delete-without-requests: "false"
                    spec:
                      profile: kind
                      tenancy: Shared
                workload:
                  tenancyCount: 20
                  template:
                    spec:
                      profile: kind
                      tenancy: Shared

    - id: clusterprovider
      template:
        apiVersion: bootstrap.open-control-plane.io/v1alpha1
        kind: ClusterProviderKind
        metadata:
          name: default
        spec:
          ocmRepoName: ${schema.spec.ocmRepoName}

    - id: platformservicegateway
      template:
        apiVersion: bootstrap.open-control-plane.io/v1alpha1
        kind: PlatformService
        metadata:
          name: platform-service-gateway
        spec:
          ocmRepoName: ${schema.spec.ocmRepoName}
          ocmComponent:
            name: github.com/openmcp-project/platform-service-gateway
            semver: v0.0.10

    - id: platformservicedns
      template:
        apiVersion: bootstrap.open-control-plane.io/v1alpha1
        kind: PlatformService
        metadata:
          name: platform-service-dns
        spec:
          ocmRepoName: ${schema.spec.ocmRepoName}
          ocmComponent:
            name: github.com/openmcp-project/platform-service-dns
            semver: v0.0.5

The distribution RGD is itself shipped as an OCM component (open-control-plane-platform) whose component references pull in the per-component RGD components transitively. Bootstrapping the platform then reduces to: register the OCM Repository, pull the distribution component, deploy its RGD, and instantiate OpenControlPlanePlatform. From that point on the whole graph is upgradeable through OCM.

apiVersion: delivery.ocm.software/v1alpha1
kind: Repository
metadata:
  name: openmcp-repository
spec:
  repositorySpec:
    baseUrl: ghcr.io/openmcp-project/components
    type: OCIRegistry
  interval: 1m
---
# Pull the distribution component which carries the OpenControlPlanePlatform RGD.
apiVersion: delivery.ocm.software/v1alpha1
kind: Component
metadata:
  name: open-control-plane-platform
spec:
  repositoryRef:
    name: openmcp-repository
  component: github.com/openmcp-project/open-control-plane-platform
  semver: v0.0.4
  interval: 1m
---
apiVersion: delivery.ocm.software/v1alpha1
kind: Resource
metadata:
  name: open-control-plane-platform-rgd
spec:
  componentRef:
    name: open-control-plane-platform
  resource:
    byReference:
      resource:
        name: rgd
---
# Installs the RGD shown above into the cluster.
apiVersion: delivery.ocm.software/v1alpha1
kind: Deployer
metadata:
  name: open-control-plane-platform-rgd
spec:
  resourceRef:
    name: open-control-plane-platform-rgd
---
# Once the RGD is registered, instantiating it bootstraps the whole platform.
apiVersion: bootstrap.open-control-plane.io/v1alpha1
kind: OpenControlPlanePlatform
metadata:
  name: default
spec:
  ocmRepoName: openmcp-repository

Current limitation: staged bootstrap​

The single-apply flow above is the target shape, but it is not yet achievable with kro. kro resolves the schema of all resources in an RGD up front. The ClusterProviderKind and PlatformService instances expand to RGDs that reference openmcp.cloud/* CRDs, and those CRDs only exist after the OpenMCPOperator has booted and registered them. kro therefore hard-fails the distribution RGD before its own CRD is ever created, so no instance can be applied.

This is tracked upstream in kro#1293 (deferred schema resolution). Until it lands, the platform is bootstrapped in stages, ordering the applies by dependency and waiting between them:

  1. Register the OCM Repository, the RBAC binding for the OCM controller, and the distribution Component.
  2. Deploy the per-component RGDs (OpenMCPOperator, ClusterProviderKind, PlatformService). Their kro CRDs become Established; the ClusterProviderKind and PlatformService RGDs stay inactive because the openmcp.cloud/* CRDs they reference do not exist yet.
  3. Apply the OpenMCPOperator instance on its own. Once it is Ready the operator registers the openmcp.cloud/* CRDs, kro re-resolves the pending schemas, and the two stuck RGDs flip to active.
  4. Apply the ClusterProviderKind and PlatformService instances.

Steps 3 and 4 are separate because the OCM Deployer applies its manifest atomically: bundling the operator instance together with the platform instances fails on the first apply, since the platform kinds are still unknown at that point.

Once kro#1293 is resolved, this collapses back into the single-apply flow above: one Deployer for the distribution RGD plus one OpenControlPlanePlatform instance, with the ordering handled inside kro.

An additional caveat is RBAC: the OCM controller applies the RGDs, kro instances, and downstream openmcp.cloud/* resources, so it needs broad permissions on the platform cluster. The reference implementation grants it cluster-admin via a ClusterRoleBinding, which is acceptable for a demo but should be scoped down for production.

ControlPlane Namespace Strategy for Platform Cluster

Maximilian Techritz
OpenControlPlane Contributor

Context and Problem Statement​

In the OpenControlPlane platform, we need to determine how to organize resources in the Platform Cluster that belong to ControlPlanes. Each ControlPlane represents a separate tenant or customer environment that needs to be isolated and managed independently. The key question is: Should every ControlPlane on the Platform Cluster have its own Namespace to ensure proper isolation, resource management, and security boundaries?

Without proper namespace isolation, ControlPlanes could interfere with each other, leading to security vulnerabilities, resource conflicts, and operational complexity.

Considered Options​

  1. mcp-{hash-mcp-name-and-namespace} - Create namespaces using a hash of the ControlPlane name and original namespace (hash < 63 chars)
  2. mcp-{uid} - Create namespaces using the UID of the ControlPlane resource

Decision Outcome​

Option 1: mcp-{hash-mcp-name-and-namespace}, because it provides unique namespace isolation for each ControlPlane while avoiding conflicts and maintaining deterministic naming that survives backup/restore operations. Option 2 would have been simpler but does not work well with backup/restore scenarios, as the UID changes after a restore operation, breaking the link between the ControlPlane and its namespace.

The hash algorithm we will use with Option 1 is SHAKE128. It is an 128-bit algorithm that is FIPS compliant. The 128-bit output allows us to print the hash in the UUID v8 format. So the namespace could look something like mcp-3f4b2c1d-8e9a-7b6c-5d4e-3f2a1b0c9d8e.

Consequences​

  • The old hash function K8sNameHash should not be used anymore.

Local DNS

Rene Schünemann
OpenControlPlane Contributor

Context and Problem Statement​

When creating services on a Kubernetes cluster, they shall be accessible from other clusters within an OpenControlPlane landscape. To achieve this a Gateway and HTTPRoute resource is created. The Gateway controller will assign a routable IP address to the Gateway resource. The HTTPRoute resource will then be used to route traffic to the service.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: example-gateway
  namespace: default
spec:
  gatewayClassName: openmcp
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same
status:
    addresses:
      - type: IPAddress
        value: "172.18.201.1"
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: example-route
  namespace: default
spec:
  parentRefs:
    - name: example-gateway
  hostnames:
    - svc-abc123.workload01.openmcp.cluster
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: example-svc
          port: 80

The problem is that the service is only reachable via the IP address and not via the hostname. This is because the DNS server in the OpenControlPlane landscape does not know about the service and therefore cannot resolve the hostname to the IP address. The Kubernetes dns service only knows how to route to service within the same cluster. On an OpenControlPlane landscape however, services must be reachable from other clusters by stable host names.

Therefore there is a need for an OpenControlPlane DNS solution that makes these host names resolvable on all clusters that are part of the OpenControlPlane landscape.

OpenControlPlane DNS System Service​

To solve the stated problem, an OpenControlPlane DNS System Service is needed. This system service will be responsible for the following tasks:

  • Deploy a central OpenControlPlane DNS server in the OpenControlPlane landscape. This DNS server will be used to resolve all host names in the OpenControlPlane base domain openmcp.cluster.
  • For each cluster in the OpenControlPlane landscape, the system service will configure the Kubernetes local DNS service to forward DNS queries for the OpenControlPlane base domain to the central OpenControlPlane DNS server. This will ensure that all clusters can resolve host names in the OpenControlPlane base domain.
  • For each Gateway or Ingress resource, the system service will create a DNS entry in the central OpenControlPlane DNS server. The DNS entry will map the hostname to the IP address of the Gateway or Ingress resource.
  • For each cluster in the OpenControlPlane landscape, the system service will annotate the Cluster resource with the OpenControlPlane base domain. This will help service providers to configure their services to use the OpenControlPlane base domain for their host names.

This shall be completely transparent to a service provider. The service provider only needs to create a Gateway or Ingress resource and the DNS entry will be created automatically.

Example Implementation​

For the example implementation, following components are used:

The DNS Provider is running on the platform cluster. The DNS Provider is deploying an ETCD and the cental CoreDNS instance on the platform cluster. The ETCD instance is used to store the DNS entries. The CoreDNS is reading the DNS entries from the ETCD instance and is used to resolve the host names.

# CoreDNS configuration to read DNS entries from ETCD for the OpenControlPlane base domain `openmcp.cluster`
- name: etcd
    parameters: openmcp.cluster
    configBlock: |-
      stubzones
      path /skydns
      endpoint http://10.96.55.87:2379

The DNS Provier is reconciling Cluster resources. For each Cluster resource, the DNS Provider is deploying an External-DNS instance on the cluster. The External-DNS instance is used to detect the Gateway and Ingress resources and to automatically create the DNS entries in the ETCD instance running on the platform cluster.

# external-dns configuration to detect Gateway and Ingress resources and write DNS entries to ETCD
containers:
    - name: external-dns
    image: registry.k8s.io/external-dns/external-dns:v0.16.1
    args:
    - --source=ingress
    - --source=gateway-httproute
    - --provider=coredns
    - --domain-filter=platform.openmcp.cluster # only detect hostnames in the OpenControlPlane base domain belonging to the cluster
    env:
    - name: ETCD_URLS
        value: http://172.18.200.2:2379 # external routable IP of the ETCD instance running on the platform cluster

The DNS Provider is updating the CoreDNS configuration on the platform cluster and on all other clusters. The CoreDNS configuration is updated to forward DNS queries for the OpenControlPlane base domain to the central CoreDNS instance running on the platform cluster. This will ensure that all clusters can resolve host names in the OpenControlPlane base domain.

openmcp.cluster {
    forward . 172.18.200.3
    log
    errors
}

Then on any pod in any cluster of the OpenControlPlane landscape, the hostname can be resolved to the IP address of the Gateway or Ingress resource.

Example

Valentin Gerlach
Valentin Gerlach
OpenControlPlane Contributor

Context and Problem Statement​

(Describe the context and problem statement, e.g., in free form using two to three sentences or in the form of an illustrative story. You may want to articulate the problem in form of a question and add links to collaboration boards or issue management systems.)

Considered Options​

  • (title of option 1)
  • (title of option 2)
  • (title of option 3)
  • …

Decision Outcome​

Chosen option: "(title of option 1)", because (justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force (force) | … | comes out best (see below)).

Consequences​

  • Good, because (positive consequence, e.g., improvement of one or more desired qualities, …)
  • Bad, because (negative consequence, e.g., compromising one or more desired qualities, …)
  • …