Develop

This guide shows you how to create Service Provider for the OpenControlPlane ecosystem from scratch. Service Providers are the heart of the OpenControlPlane platform, as they provide the capabilities to offer Infrastructure as Data services to end users.

In this guide, we will walk you through the steps of creating a Service Provider using the service-provider-template, explain the context a service provider operates in, and demonstrate how to run end-to-end tests for it.

By the end of this guide, you should have a solid understanding of how a service works and be ready to build a real world provider such as service-provider-velero.

Let's get started!

Overview

A service provider consists of the following two major parts, similar to a regular kubernetes controller:

A user-facing ServiceProviderAPI: This allows end users to request a DomainService for a ControlPlane, e.g. FooService or Velero.
A controller that reconciles the ServiceProviderAPI: This controller manages the lifecycle of the provided DomainService and its API (such as Foo or the CRDs of Velero).

For a visual overview of how these components fit into an OpenControlPlane installation, refer to the service provider deployment model.

Cluster and Namespace Context

A service provider operates across multiple clusters. The following table shows which cluster each code-level accessor points to and what you typically find there:

Code Access	Cluster	What lives there
`r.PlatformCluster`	Platform Cluster	ProviderConfig (cluster-scoped); image pull secrets (in pod namespace); AccessRequests and kubeconfig secrets (in the per-tenant `mcp--<uuid>` namespace)
`r.OnboardingCluster`	Onboarding Cluster	ServiceProviderAPI objects in `project-<p>--ws-<w>` workspace namespaces
`clusters.MCPCluster`	ControlPlane cluster (per tenant)	DomainServiceAPI CRDs and deployed resources in a namespace you choose (e.g., `crossplane-system`)
`clusters.WorkloadCluster`	Workload Cluster (shared, optional)	DomainService controller workloads in a namespace you choose

info

In the previous table, r refers to the receiver of the CreateOrUpdate and Delete methods, while clusters refers to the parameter of type ClusterContext.

For a complete overview of namespaces across all cluster types, including real-world examples, see Clusters and Namespaces.

Prerequisites

Start by creating a new repository for your service provider using the service-provider-template. Click "Use this template" button on the GitHub page and give your new repository a name that reflects the domain service it provides, e.g. service-provider-velero for a service provider that deploys Velero.

Clone the newly created repository to your local machine and open it with your favorite IDE. Finally, ensure that you have Go installed. You can download it from go.dev.

Service Provider Template Usage

The template allows you to create a service provider without requiring deep knowledge of the underlying OpenControlPlane platform.

Run the following command to generate a new provider. Replace velero with the kind of your service:

go run ./cmd/template -v -module github.com/openmcp-project/service-provider-velero -kind Velero -group velero

Optional flags:

-v generates sample code to get started quickly
-w enables workload cluster support (see below)
-s generates a secret watcher implementation

The template generates a fully functional service provider that can be executed and deployed on your local machine using cluster-provider-kind and openmcp-testing.

To run the the generated end-to-end test using task, init the build submodule and execute the e2e test:

git submodule update --init --recursive
task test-e2e

This test bootstraps a complete local OpenControlPlane installation with all required components, including:

The platform cluster where your service provider is managed by the openmcp-operator
The onboarding cluster where end users request the domain service you provider offers.
A ControlPlane cluster where your service provider installs its DomainServiceAPI and optionally its workload.
An optional workload cluster, provisioned when using the template flag -w. Your service provider then requests a workload cluster from the openmcp-operator to deploy its workload outside the ControlPlane. This will result in another kind cluster.

info

This means running the above command from the Service Provider Template Usage section with -w like this:

go run ./cmd/template -v -w -module github.com/openmcp-project/service-provider-velero -kind Velero -group velero

For a detailed guide to the openmcp-testing testing framework, test structure, and examples, see the Testing guide. For running your controller locally outside the cluster and troubleshooting common issues, see the Debug guide.

The template generator removes its own code after execution. If you want to revert your changes and start fresh, simply use git and delete any generated untracked files. For this reason, remove template-generation step from the e2e test .github/workflows/go.yaml before committing your changes (otherwise your workflow will fail).

      - name: Generate template
        run: |
          go run ./cmd/template -v -w

Project Structure

The service provider template is built with kubebuilder, so the project follows the conventions of typical Kubernetes controllers:

api/ includes generated types and their CRDs which the crd manager will install during the initCommand.
internal/controller/ contains the Reconciler where you implement your domain specific reconcile logic.

If you are new to implementing Kubernetes controllers, consider completing building a CronJob tutorial before returning to this guide. The rest of this guide highlights the most important steps to create a service provider and the differences compared to a regular Kubernetes controller.

Create your ServiceProviderAPI

The ServiceProviderAPI type defines the options available to end users when consuming your managed service offering. This API is watched by your ServiceProviderReconciler. The template starts with a simple example field:

// FooServiceSpec defines the desired state of FooService
type FooServiceSpec struct {
 // foo is an example field of FooService. Edit fooservice_types.go to remove/update
 // +optional
 Foo *string `json:"foo,omitempty"`
}

Modify the spec to expose the configuration your provider supports.

For example, Velero allows the user to choose a version and which plugins to install:

apiVersion: velero.services.open-control-plane.io/v1alpha1
kind: Velero
metadata:
  name: test-controlplane
spec:
  version: "v1.17.1"
  plugins:
    - name: "aws"
      version: "v1.13.0"

Each onboarding API type must include common status fields. These are included in the template via the commonapi.Status inline struct and you can add additional optional fields.

import (
 commonapi "github.com/openmcp-project/openmcp-operator/api/common"
)

// FooServiceStatus defines the observed state of FooService.
type FooServiceStatus struct {
 commonapi.Status `json:",inline"`
}

For example, Velero includes a list of managed resources.

// Constants representing the phases of a velero instance lifecycle.
const (
 Pending     InstancePhase = "Pending"
 Progressing InstancePhase = "Progressing"
 Ready       InstancePhase = "Ready"
 Failed      InstancePhase = "Failed"
 Terminating InstancePhase = "Terminating"
 Unknown     InstancePhase = "Unknown"
)

type VeleroStatus struct {
 commonapi.Status `json:",inline"`
 // Resources managed by this velero instance
 // +optional
 Resources []ManagedResource `json:"resources,omitempty"`
}

// ManagedResource defines a kubernetes object with its lifecycle phase
type ManagedResource struct {
 corev1.TypedObjectReference `json:",inline"`

 Phase   InstancePhase `json:"phase"`
 Message string        `json:"message,omitempty"`
}

Edit the ProviderConfig API

Service providers must expose a ProviderConfig which platform operators use to configure provider behavior. Because provider deployment does not support passing arguments to the binary directly, configuration must be expressed via this API.

ProviderConfig Guidelines

At the time of writing, the design of a ProviderConfig must address the following concerns:

Deployment prerequisites for restricted environments: A ProviderConfig must expose the configuration required to deploy a service provider in private or aig-gapped environments, for example by specifying image and Helm chart pull secrets.
Constraining deployable domain service versions: A ProviderConfig must expose configuration that constraints which versions of a domain service the service provider is allowed to deploy. This enables platform operators to explicitly define an approved set of versions. It is required to add this for new service providers until dedicated Registry and TenantPolicy systems will replace this in the future (see Discoverability of OpenControlPlane components).
Runtime behavior: A ProviderConfig may expose configuration settings that influence the runtime behavior of a service provider, such as the poll interval used in the service provider template or support for custom CA bundles.

ProviderConfig Example

Velero's ProviderConfig currently includes both operational settings and deployment artifacts:

apiVersion: velero.services.open-control-plane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: default
spec:
  # OPERATIONAL SETTINGS (correct — permanent)
  pollInterval: 1m

  # DEPLOYMENT ARTIFACTS (temporary — will move to separate API)
  availableImages:
    - name: velero
      versions: ["v1.17.1"]
      image: "velero/velero"
    - name: aws
      versions: ["v1.13.0"]
      image: "velero/velero-plugin-for-aws"

  # DEPLOYMENT ARTIFACTS (temporary — will move to separate API)
  imagePullSecrets:
    - name: privateregcred

Note that these image pull secrets reference secrets stored on the platform cluster. It is the responsibility of the service provider to ensure that the referenced secrets are copied to the cluster in which the deployments run.

To synchronize the image pull secrets between the platform cluster and the cluster where your workloads run, you can use the SecretMutator from controller-utils inside your CreateOrUpdate logic.

Velero implements this synchronization as follows:

// internal/controller/velero_controller.go
func (r *VeleroReconciler) CreateOrUpdate(ctx context.Context, obj *apiv1alpha1.Velero, pc *apiv1alpha1.ProviderConfig, clusters clusteraccess.ClusterContext) (ctrl.Result, error) {
 ...
 workloadCluster := resources.NewManagedCluster(clusters.WorkloadCluster.Client(), clusters.WorkloadCluster.RESTConfig(), instance.Namespace(obj), resources.WorkloadCluter)
 secret.Configure(workloadCluster, r.PlatformCluster, pc.Spec.ImagePullSecrets, r.PodNamespace)
 ...
}

// pkg/secret/secret.go
func Configure(cluster resources.ManagedCluster, platformCluster *clusters.Cluster, imagePullSecrets []corev1.LocalObjectReference, sourceNamespace string) {
 for _, pullSecret := range imagePullSecrets {
  secret := resources.NewManagedObject(&corev1.Secret{
   ObjectMeta: metav1.ObjectMeta{
    Name:      pullSecret.Name,
    Namespace: cluster.GetDefaultNamespace(),
   },
  }, resources.ManagedObjectContext{
   ReconcileFunc: func(ctx context.Context, o client.Object) error {
    oSecret := o.(*corev1.Secret)
    sourceSecret := &corev1.Secret{
     ObjectMeta: metav1.ObjectMeta{
      Name:      pullSecret.Name,
      Namespace: sourceNamespace,
     },
    }
    // retrieve source secret from platform cluster
    if err := platformCluster.Client().Get(ctx, client.ObjectKeyFromObject(sourceSecret), sourceSecret); err != nil {
     return err
    }
    mutator := openmcpresources.NewSecretMutator(pullSecret.Name, cluster.GetDefaultNamespace(), sourceSecret.Data, corev1.SecretTypeDockerConfigJson)
    return mutator.Mutate(oSecret)
   },
   StatusFunc: resources.SimpleStatus,
  })
  cluster.AddObject(secret)
 }
}

Finally, ensure that the synchronized image pull secrets are referenced in any workload you reconcile.

Watch Secrets for Changes

If your service provider references secrets on the platform cluster (e.g. image pull secrets configured in the ProviderConfig), you may want to automatically trigger reconciliation when those secrets change. The opencontrolplane-runtime provides an optional serviceprovider.SecretWatcher interface for this purpose.

When enabled, the runtime watches Kubernetes Secret objects in a configured namespace on the platform cluster. Whenever a secret changes, it calls your IsReferencedSecret method to determine whether the change is relevant. If it is, all ServiceProviderAPI objects are re-enqueued for reconciliation, ensuring that your provider picks up the updated secret data.

Generate with the template

Pass the -s flag to the template generator to scaffold the secret watcher:

go run ./cmd/template -v -s -module github.com/openmcp-project/service-provider-velero -kind Velero -group velero

This generates:

An IsReferencedSecret method stub in your controller
A WithSecretNamespace(podNamespace) call in the setup code

Implement `IsReferencedSecret`

Your reconciler must implement the serviceprovider.SecretWatcher interface by providing an IsReferencedSecret method. This method receives the changed secret and the current ProviderConfig, and returns true if the secret should trigger reconciliation.

// IsReferencedSecret returns true if the given secret should trigger
// reconciliation. See serviceprovider.SecretWatcher for details.
func (r *VeleroReconciler) IsReferencedSecret(ctx context.Context, secret *corev1.Secret, pc *apiv1alpha1.ProviderConfig) bool {
    if pc == nil {
        return false
    }
    for _, ref := range pc.Spec.ImagePullSecrets {
        if ref.Name == secret.Name {
            return true
        }
    }
    return false
}

info

The pc parameter may be nil if the ProviderConfig has not been loaded yet. Always guard against this before accessing its fields.

Enable secret watching in setup

In your main.go, chain WithSecretNamespace when building the SPReconciler. The namespace should be the pod's own namespace on the platform cluster, which is where secrets like image pull secrets are typically stored:

spr := serviceprovider.NewAPIReconcilerBuilder[*velerosv1alpha1.Velero, *velerosv1alpha1.ProviderConfig]().
    EmpyObjectProvider(func() *velerosv1alpha1.Velero { return &velerosv1alpha1.Velero{} }).
    PlatformCluster(platformCluster).
    OnboardingCluster(onboardingCluster).
    SecretNamespace(podNamespace).
    Reconciler(&controller.VeleroReconciler{...}).
    // ...

The watch is only active when both conditions are met:

Your reconciler implements the serviceprovider.SecretWatcher interface
SecretNamespace is configured with a non-empty namespace

When a referenced secret changes, the runtime lists all ServiceProviderAPI objects from the onboarding cluster and enqueues them for reconciliation. This ensures every service instance picks up the latest secret data during its next reconcile cycle.

Edit the ServiceProviderReconciler

Your Reconciler must implement two methods: CreateOrUpdate and Delete. The template sets up your reconciler together with the generic reconciler from pkg/runtime, which handles OpenControlPlane-specifics (cluster access, config updates, etc.).

CreateOrUpdate Operation

The template example contains a basic implementation that installs a CRD into the tenant ControlPlane.

// CreateOrUpdate is called on every add or update event
func (r *FooServiceReconciler) CreateOrUpdate(ctx context.Context, svcobj *apiv1alpha1.FooService, _ *apiv1alpha1.ProviderConfig, clusters clusteraccess.ClusterContext) (ctrl.Result, error) {
 serviceprovider.StatusProgressing(svcobj, "Reconciling", "Reconcile in progress")
 managedObj := &apiextensionsv1.CustomResourceDefinition{
  ObjectMeta: metav1.ObjectMeta{
   Name: "foos.example.domain",
  },
 }
 _, err := ctrl.CreateOrUpdate(ctx, clusters.MCPCluster.Client(), managedObj, func() error {
  managedObj.Spec = fooCRD().Spec
  return nil
 })
 if err == nil {
  serviceprovider.StatusReady(svcobj)
 }
 return ctrl.Result{}, err
}

In a real world example like Velero, this step contains installing and reconciling every required resource, including CRDs, namespace(s), service account(s), deployment(s), etc. into the ControlPlane and workload cluster.

The ClusterContext provides access to all request specific clusters. This always includes the ControlPlane and, when requested, the workload cluster associated with the current request.

The workload cluster is optional and will not be available for providers that deploy their workload directly to the ControlPlane.

info

Cluster and Namespace Context provides an overview of available clusters, their purpose and how to access them.

In addition, ClusterContext exposes two ObjectKeys that can be used to retrieve the kubeconfig for either the ControlPlane or the workload cluster. These kubeconfigs can be passed to a HelmReleases spec when a service provider uses Flux to deploy its service.

type ClusterContext struct {
 MCPCluster *clusters.Cluster
 MCPAccessSecretKey client.ObjectKey
 WorkloadCluster *clusters.Cluster
 WorkloadAccessSecretKey client.ObjectKey
}

In contrast, the platform and onboarding clusters are static clusters that are assigned to the reconciler at initialization.

type FooServiceReconciler struct {
 OnboardingCluster *clusters.Cluster
 PlatformCluster   *clusters.Cluster
}

Any update to either the ServiceProviderAPI or ProviderConfig triggers reconciliation.

Delete Operation

When a user removes a service, the provider must delete all managed resources. A managed resource is any resource the service provider explicitly creates in its CreateOrUpdate Operation. For example, the HelmRelease in Flux based service providers is considered a managed resource. The Kubernetes objects created by Flux from that HelmRelease are managed by Flux and not by the service provider.

The basic template example deletes the Foo CRD:

func (r *FooServiceReconciler) Delete(ctx context.Context, obj *apiv1alpha1.FooService, _ *apiv1alpha1.ProviderConfig, clusters clusteraccess.ClusterContext) (ctrl.Result, error) {
 l := logf.FromContext(ctx)
 serviceprovider.StatusTerminating(obj)
 managedObj := fooCRD()
 if err := clusters.MCPCluster.Client().Delete(ctx, managedObj); client.IgnoreNotFound(err) != nil {
  l.Error(err, "delete object failed")
  return ctrl.Result{}, err
 }
 if err := clusters.MCPCluster.Client().Get(ctx, client.ObjectKeyFromObject(managedObj), managedObj); err != nil {
  return reconcile.Result{}, client.IgnoreNotFound(err)
 }
 // object still exists
 return ctrl.Result{
  RequeueAfter: time.Second * 10,
 }, nil
}

Block deletion while user resources exist

End users interact with two clusters in this context:

The Onboarding cluster, where they create the ServiceProviderAPI object that requests the service. This object is controlled by the service provider.
The tenant ControlPlane cluster, where they create domain custom resources (e.g. GitRepository, Bucket, ExternalSecret) of the CRDs your provider installs. These CRs are controlled by the domain controllers your provider manages.

A naive Delete implementation removes those domain CRDs from the ControlPlane cluster as soon as the user deletes the ServiceProviderAPI object on the Onboarding cluster. If the domain controller does not protect its CRs with finalizers removing the CRDs while user instances still exist may orphan those resources and leave external systems (object storage, git repositories, secrets in vaults) outside platform control.

A service provider can improve the user experience and prevent silent data loss by blocking its own deletion until the user has cleaned up the domain CRs first. Quality criterion Deletion behaviour asks providers to do this whenever the underlying domain controller lacks its own deletion safeguards. The expected flow inside Delete:

Look up the user's domain custom resources on clusters.MCPCluster for the CRDs your provider installs (e.g. GitRepository for a Flux provider, ClusterPolicy for a Kyverno provider).
If any are still there, set Status.Phase = Terminating on the ServiceProviderAPI object, add a condition that names the kinds blocking deletion, and requeue. Skip the rest of Delete.
If none are left, run your normal teardown.

warning

There is no generic way to enumerate every CRD a domain service installs, so each provider must list the kinds it knows it owns. A force-delete escape hatch is intentionally not offered today; if you need one, raise it as a discussion before implementing it ad-hoc.

info

The service-provider-template does not yet provide a reference implementation for this check. Progress is tracked in openmcp-project/service-provider-template#88.

Cluster Access

This section covers all things related on how to control what your service provider is allowed to do on a target control plane.

Restrict Cluster Access

The template code runs the provider with full admin permissions. Before releasing your provider, restrict its required permissions using the advanced ClusterAccessReconciler.

Before

controlPlaneTokenAccess := &clustersv1alpha1.TokenConfig{
    Permissions: []clustersv1alpha1.PermissionsRequest{
        {
            Rules: []rbacv1.PolicyRule{
                {
                    APIGroups: []string{"*"},
                    Resources: []string{"*"},
                    Verbs:     []string{"*"},
                },
            },
        },
    },
    RoleRefs: []common.RoleRef{
        {
            Name: "cluster-admin",
            Kind: "ClusterRole",
        },
    },
}

After

controlPlaneTokenAccess := &clustersv1alpha1.TokenConfig{
    Permissions: []clustersv1alpha1.PermissionsRequest{
        {
            Rules: []rbacv1.PolicyRule{
                {
                    APIGroups: []string{""},
                    Resources: []string{"configmaps"},
                    Verbs:     []string{"get", "list", "watch"},
                },
                {
                    APIGroups: []string{"coordination.k8s.io"},
                    Resources: []string{"leases"},
                    Verbs:     []string{"get", "create", "patch", "update"},
                },
                {
                    APIGroups: []string{""},
                    Resources: []string{"events"},
                    Verbs:     []string{"create", "patch"},
                },
                ...
            },
        },
    },
}

Dynamic Cluster Access

It is also possible to dynamically generate the TokenConfig of your provider which is enabled through registering a TokenAccessGenerator and AdditionalDataGenerators. These generators will be called during reconciliation. The AdditionalDataGenerators can generate arbitrary data which is looped through the various ClusterAccessReconciler methods down to the TokenAccessGenerator. There it can be used to tailor the TokenConfig to the permission requirements of your service provider.

This is for example useful if your service provider offers a namespace override meaning the namespace is only known during reconciliation and using a TokenAccessGenerator allows you to take that information into account.

info

Using dynamic cluster access is only possible when using the advanced ClusterAccessReconciler. If your service provider is still using the legacy simple ClusterAccessReconciler please refer to the migration guide on how to migrate.

To see a real-world example on how to implement dynamic cluster access please check out the implementation of service-provider-external-secrets.

As an example we will demonstrate how you can dynamically insert an overridden namespace into your TokenConfig in the following code snippets.

1. Define a Namespace Resolver

We first need to define a function which resolves a potential overridden namespace from the ProviderConfig.

type fooServiceNamespace string

func resolveFooServiceNamespace(_ context.Context, obj *fooservicesv1alpha1.FooService, providerConfig *fooservicesv1alpha1.ProviderConfig) (any, error) {
    namespace := "fooservice-system"

    if obj != nil && providerConfig != nil {
        for _, version := range providerConfig.Spec.Versions {
            if version.Version != obj.Spec.Version {
                continue
            }

            if version.NamespaceOverride != "" {
                namespace = version.NamespaceOverride
            }
            break
        }
    }

    return fooServiceNamespace(namespace), nil
}

2. Define a `TokenAccessGenerator`

Now we define a TokenAccessGenerator which will replace our static TokenAccess definition. It reads the namespace from the provided additionalData and then sets the Namespace of our Permissions.

tokenAccessGenerator := func(_ reconcile.Request, additionalData ...any) (*clustersv1alpha1.TokenConfig, error) {
    namespace := "fooservice-system"
    for _, data := range additionalData {
        ns, ok := data.(fooServiceNamespace)
        if ok && ns != "" {
            namespace = string(ns)
            break
        }
    }
    return &clustersv1alpha1.TokenConfig{
        Permissions: []clustersv1alpha1.PermissionsRequest{
            {
                Namespace: namespace,
                Rules: []rbacv1.PolicyRule{
                    {
                        APIGroups: []string{"*"},
                        Resources: []string{"*"},
                        Verbs:     []string{"*"},
                    },
                },
            },
        },
    }, nil
}

3. Build the `ClusterRequest`

When we build our ClusterRequest we need to register our TokenAccessGenerator.

controlPlaneClusterRequest := advanced.ExistingClusterRequest("mcp", "mcp", func(req reconcile.Request, _ ...any) (*common.ObjectReference, error) {
    namespace, err := utils.StableMCPNamespace(req.Name, req.Namespace)
    if err != nil {
        return nil, err
    }
    return &common.ObjectReference{
        Name:      req.Name,
        Namespace: namespace,
    }, nil
}).
    WithNamespaceGenerator(advanced.DefaultNamespaceGeneratorForMCP).
    WithTokenAccessGenerator(tokenAccessGenerator).
    WithScheme(controlPlaneScheme).
    Build()

4. Build the `ClusterAccessReconciler`

When building our ClusterAccessReconciler we need to register our ClusterRequest.

clusterAccessReconciler := advanced.NewClusterAccessReconciler(platformCluster.Client(), "FooServiceOperator").
    WithManagedLabels(func(controllerName string, req reconcile.Request, reg advanced.ClusterRegistration) (string, string, map[string]string) {
        _, managedPurpose, _ := advanced.DefaultManagedLabelGenerator(controllerName, req, reg)
        return controllerName, managedPurpose, map[string]string{
            openmcpconst.OnboardingNameLabel:      req.Name,
            openmcpconst.OnboardingNamespaceLabel: req.Namespace,
        }
    }).
    WithRetryInterval(10 * time.Second).
    Register(controlPlaneClusterRequest)

5. Register with the `APIReconciler`

To tie it all together we have to register resolveFooServiceNamespace as AdditionalDataGenerator with our APIReconciler along with our just created ClusterAccessReconciler.

spr := serviceprovider.NewAPIReconcilerBuilder[*fooservicesv1alpha1.FooService, *fooservicesv1alpha1.ProviderConfig]().
    EmptyObjectProvider(func() *fooservicesv1alpha1.FooService {
        return &fooservicesv1alpha1.FooService{}
    }).
    PlatformCluster(platformCluster).
    OnboardingCluster(onboardingCluster).
    Reconciler(&controller.FooServiceReconciler{
        OnboardingCluster: onboardingCluster,
        PlatformCluster:   platformCluster,
        PodNamespace:      podNamespace,
    }).
    AdvancedClusterAccessReconciler(clusterAccessReconciler).
    AdditionalDataGenerators(resolveFooServiceNamespace).
    MustBuild()

Migrate from simple to advanced ClusterAccessReconciler

This section walks through migrating a service provider from the legacy simple ClusterAccessReconciler to the advanced ClusterAccessReconciler. The simple ClusterAccessReconciler is a facade over the advanced one and the logic we now build manually was previously hidden behind its functional options.

info

To see a real-world implementation have for example a look at service-provider-flux or service-provider-external-secrets

1. Update dependencies

First we need to bump opencontrolplane-runtime in our go.mod to a version >= 0.4.0:

github.com/openmcp-project/opencontrolplane-runtime >= v0.4.0

Then run go mod tidy to update our dependencies.

2. Add imports

Next we need to add the advanced and reconcile package main.go imports:

import (
    // existing imports …
    "github.com/openmcp-project/openmcp-operator/lib/clusteraccess/advanced"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
)

3. Define a `TokenConfig`

Instead of calling WithMCPPermissions / WithMCPRoleRefs on the NewClusterAccessReconciler we need to create a standalone *clustersv1alpha1.TokenConfig:

tokenAccessConfig := &clustersv1alpha1.TokenConfig{
    Permissions: []clustersv1alpha1.PermissionsRequest{
        {
            Rules: []rbacv1.PolicyRule{
                {
                    APIGroups: []string{"*"},
                    Resources: []string{"*"},
                    Verbs:     []string{"*"},
                },
            },
        },
    },
    RoleRefs: []common.RoleRef{
        {
            Name: "cluster-admin",
            Kind: "ClusterRole",
        },
    },
}

info

Replace the wildcard rules with the minimum set of permissions your provider actually needs. If you need the permissions to vary per reconcile please refer to Dynamic Cluster Access.

4. Build the `ClusterRequest`

Next we need to create a ClusterRequest for our Control Plane cluster where we register our TokenConfig

controlPlaneClusterRequest := advanced.ExistingClusterRequest("mcp", "mcp", func(req reconcile.Request, _ ...any) (*common.ObjectReference, error) {
    namespace, err := utils.StableMCPNamespace(req.Name, req.Namespace)
    if err != nil {
        return nil, err
    }
    return &common.ObjectReference{
        Name:      req.Name,
        Namespace: namespace,
    }, nil
}).
    WithNamespaceGenerator(advanced.DefaultNamespaceGeneratorForMCP).
    WithTokenAccess(tokenAccessConfig).
    WithScheme(controlPlaneScheme).
    Build()

5. Build the `ClusterAccessReconciler`

Now we are ready to build our new advanced ClusterAccessReconciler where we register our ClusterRequest

clusterAccessReconciler := advanced.NewClusterAccessReconciler(platformCluster.Client(), "YourOperatorName").
    WithManagedLabels(func(controllerName string, req reconcile.Request, reg advanced.ClusterRegistration) (string, string, map[string]string) {
        _, managedPurpose, _ := advanced.DefaultManagedLabelGenerator(controllerName, req, reg)
        return controllerName, managedPurpose, map[string]string{
            openmcpconst.OnboardingNameLabel:      req.Name,
            openmcpconst.OnboardingNamespaceLabel: req.Namespace,
        }
    }).
    WithRetryInterval(10 * time.Second).
    Register(controlPlaneClusterRequest)

6. Register with the `APIReconciler`

Lastly we have to register our advanced ClusterAccessReconciler with our APIReconciler. To do that we replace the NewAPIReconcilerBuilder.ClusterAccessReconciler with NewAPIReconcilerBuilder.AdvancedClusterAccessReconciler in the builder chain.

// Before
spr := serviceprovider.NewAPIReconcilerBuilder[*FooService, *ProviderConfig]().
    // …
    ClusterAccessReconciler(clusteraccess.NewClusterAccessReconciler(platformCluster.Client(), "Foo").
        WithMCPScheme(controlPlaneScheme).
        WithRetryInterval(10 * time.Second).
        WithMCPPermissions(adminPermissions).
        WithMCPRoleRefs([]common.RoleRef{{Name: "cluster-admin", Kind: "ClusterRole"}}).
        SkipWorkloadCluster(),
    ).MustBuild()

// After
spr := serviceprovider.NewAPIReconcilerBuilder[*FooService, *ProviderConfig]().
    // …
    AdvancedClusterAccessReconciler(clusterAccessReconciler).
    MustBuild()

Taskfile Usage

OpenControlPlane controllers use task instead of make to generate code, build your controller image, etc. The most important developer commands are:

task generate to regenerate code after API changes
task build:img:build-test to build an image for local testing
task test-e2e for a full pipeline run including code generation, validation and e2e test execution.

Run task -l to see all available tasks.

Release and Publish Workflows

The template includes two reusable GitHub workflows to create releases and publish build artifacts. These workflows are defined in the build submodule.

To reuse these workflows outside of the github.com/openmcp-project organization, you must provide your own GitHub App values as input parameters, e.g. in /.github/workflows/release.yaml:

name: Versioned Release

jobs:
  release_tag:
  uses: openmcp-project/build/.github/workflows/release.lib.yaml@2857d51e2d98261233de19e6df4e49f20db8bbda # main
  with:
    app_id: "YOUR_APP_ID"
  secrets:
    app_private_key: ${{ secrets.YOUR_APP_PRIVATE_KEY }}

Best Practices

For simple deployments

For simple deployments that don't require too much hassle, it is wise to just implement the deployment generation and creation in the Go code using plain Deployment objects and ServiceAccount objects for access setups. For example:

// in authz package
// Configure adds a managed ClusterRoleBinding object to the given cluster.
// The passed in service account is granted the cluster-admin role.
func Configure(cluster resources.ManagedCluster, msa *authn.ManagedServiceAccount) {
	crb := resources.NewManagedObject(&rbacv1.ClusterRoleBinding{
		ObjectMeta: metav1.ObjectMeta{
			Name: clusterRoleBindingName,
		},
	}, resources.ManagedObjectContext{
		ReconcileFunc: func(_ context.Context, o client.Object) error {
			oCRB := o.(*rbacv1.ClusterRoleBinding)
			oCRB.Subjects = []rbacv1.Subject{
				{
					Kind:      rbacv1.ServiceAccountKind,
					Name:      msa.Name,
					Namespace: msa.Namespace,
				},
			}
			oCRB.RoleRef = rbacv1.RoleRef{
				APIGroup: rbacv1.GroupName,
				Kind:     "ClusterRole",
				Name:     "cluster-admin",
			}
			return nil
		},
		StatusFunc: resources.SimpleStatus,
	})
	cluster.AddObject(crb)
}

And then, during CreateOrUpdate or the Delete flows, you could call them as such:

// CreateOrUpdate is called on every add or update event
func (r *VeleroReconciler) CreateOrUpdate(ctx context.Context, obj *apiv1alpha1.Velero, pc *apiv1alpha1.ProviderConfig, clusters clusteraccess.ClusterContext) (ctrl.Result, error) {
	l := log.FromContext(ctx)
	serviceprovider.StatusProgressing(obj, "Reconciling", "Reconcile in progress")
	err := r.ensureInstanceID(ctx, obj)
	if err != nil {
		return ctrl.Result{}, err
	}
	mgr, err := r.configResources(obj, pc, clusters)
	if err != nil {
		return ctrl.Result{}, err
	}
	results := mgr.Apply(ctx)
	errRes := false
	for _, r := range results {
		if r.Error != nil {
			l.Error(r.Error, objectutils.ObjectID(r.Object.GetObject()))
			errRes = true
		}
	}
	managedResources := resultsToResources(results)
	obj.Status.Resources = managedResources
	if allResourcesReady(managedResources) {
		serviceprovider.StatusReady(obj)
	}
	if errRes {
		return ctrl.Result{}, errors.New("reconciliation result contains errors")
	}
	return ctrl.Result{}, nil
}

Where configResources calls each individual package's Configure function like this:

namespace.Configure(mcpCluster, resources.Orphan)
...
authz.Configure(mcpCluster, mcpServiceAccount)
...
deployment.ConfigureMcp(mcpCluster, images["velero"], instance.GetID(obj))
...
...

This will always reconcile all the objects handled by this service provider.

For complex deployments with lot of moving parts

Now, consider a more complex deployment that requires and has a lot of buttons and switches to set during the deployment. This would be rather cumbersome and difficult to manage from Go code.

We recommend using Flux ( or any other deployment handling operator that understand Helm ) to do such deployments.

info

At the time of this writing, a flux provider implementation is in the works.

Simply create a HelmRelease object with all the bells whistles set using the ProviderConfig optionally.

Here is a plain example of an OCIRepository pointing to a configurable URL with a configurable version coming from either the ServiceProviderAPI:

// createOciRepository creates a repository pointing to the helm repo containing the to be deployed chart.
func createOciRepository(url, version string) *sourcev1.OCIRepository {
	return &sourcev1.OCIRepository{
		ObjectMeta: metav1.ObjectMeta{
			Name:      OCIRepositoryName,
			Namespace: metav1.NamespaceDefault,
		},
		Spec: sourcev1.OCIRepositorySpec{
			Interval: metav1.Duration{Duration: time.Minute},
			URL:      url,
			Reference: &sourcev1.OCIRepositoryRef{
				Tag: version,
			},
		},
	}
}

// createHelmRelease creates a release for a chart with some optional helm values.
func createHelmRelease() (*helmv2.HelmRelease, error) {
	values := make(map[string]interface{})
	values["manager"] = map[string]interface{}{
		"concurrency": map[string]interface{}{
			"resource": 21,
		},
		"logging": map[string]interface{}{
			"level": "debug",
		},
	}
	content, err := json.Marshal(values)
	if err != nil {
		return nil, fmt.Errorf("failed to marshal helm value overrides: %w", err)
	}

	return &helmv2.HelmRelease{
		ObjectMeta: metav1.ObjectMeta{
			Name:      HelmReleaseName,
			Namespace: metav1.NamespaceDefault,
		},
		Spec: helmv2.HelmReleaseSpec{
			Interval:        metav1.Duration{Duration: time.Minute},
			TargetNamespace: metav1.NamespaceDefault,
			ChartRef: &helmv2.CrossNamespaceSourceReference{
				Kind:      "OCIRepository",
				Name:      OCIRepositoryName,
				Namespace: metav1.NamespaceDefault,
			},
			Values: &apiextensionsv1.JSON{Raw: content},
      // ServiceAccountName: --> could be created first and then passed in here.
			KubeConfig: &meta.KubeConfigReference{
				ConfigMapRef: &meta.LocalObjectReference{},
				SecretRef:    &meta.SecretKeyReference{},
			},
		},
	}, nil
}

In here, the values, for example, could come from the ProviderConfig. Or further image accesses could be configured by other service accounts and secrets being created as a first step.

warning

Any resource created in a tenant namespace on the platform cluster (e.g. a pull secret to access an OCIRepository with a Helm chart) must have a carefully chosen name to avoid conflicts with other service providers. Use a service‑provider–specific prefix for the resource, e.g. rename privateregcred to sp-crossplane-privateregcred for pull secret copies. Make sure the resulting name does not exceed the allowed max name length.

Also notice the KubeConfig section. Your service provider controller is deployed in the Platform cluster. The target controller that your provider is deploying will run in either the ControlPlane or the Workload cluster. This depends on your choice, however, it is recommended to deploy into the workload cluster as documented here.

This means that Flux will need a KubeConfig in order to deploy to the ControlPlane cluster. And this is the KubeConfig that is given to Flux in this section.

info

Note, that as of this writing, it's a bit difficult to obtain the KubeConfig for the Workload Cluster. MCPCluster KubeConfig is obtained via a AccessRequest. For an example, please look at how the service-provider-crossplane is doing this.

Overview​

Cluster and Namespace Context​

Prerequisites​

Service Provider Template Usage​

Project Structure​

Create your ServiceProviderAPI​

Edit the ProviderConfig API​

ProviderConfig Guidelines​

ProviderConfig Example​

Watch Secrets for Changes​

Generate with the template​

Implement IsReferencedSecret​

Enable secret watching in setup​

Edit the ServiceProviderReconciler​

CreateOrUpdate Operation​

Delete Operation​

Block deletion while user resources exist​

Cluster Access​

Restrict Cluster Access​

Before​

After​

Dynamic Cluster Access​

1. Define a Namespace Resolver​

2. Define a TokenAccessGenerator​

3. Build the ClusterRequest​

4. Build the ClusterAccessReconciler​

5. Register with the APIReconciler​

Migrate from simple to advanced ClusterAccessReconciler​

1. Update dependencies​

2. Add imports​

3. Define a TokenConfig​

4. Build the ClusterRequest​

5. Build the ClusterAccessReconciler​

6. Register with the APIReconciler​

Taskfile Usage​

Release and Publish Workflows​

Best Practices​

For simple deployments​

For complex deployments with lot of moving parts​

Overview

Cluster and Namespace Context

Prerequisites

Service Provider Template Usage

Project Structure

Create your ServiceProviderAPI

Edit the ProviderConfig API

ProviderConfig Guidelines

ProviderConfig Example

Watch Secrets for Changes

Generate with the template

Implement `IsReferencedSecret`

Enable secret watching in setup

Edit the ServiceProviderReconciler

CreateOrUpdate Operation

Delete Operation

Block deletion while user resources exist

Cluster Access

Restrict Cluster Access

Before

After

Dynamic Cluster Access

1. Define a Namespace Resolver

2. Define a `TokenAccessGenerator`

3. Build the `ClusterRequest`

4. Build the `ClusterAccessReconciler`

5. Register with the `APIReconciler`

Migrate from simple to advanced ClusterAccessReconciler

1. Update dependencies

2. Add imports

3. Define a `TokenConfig`

4. Build the `ClusterRequest`

5. Build the `ClusterAccessReconciler`

6. Register with the `APIReconciler`

Taskfile Usage

Release and Publish Workflows

Best Practices

For simple deployments

For complex deployments with lot of moving parts