Skip to main content

Develop

Cluster Providers manage Kubernetes clusters and provide access to them within the OpenControlPlane ecosystem. They follow the same general patterns as Service Providers.

In this guide, we will walk you through the steps of creating a Cluster Provider using the cluster-provider-template. The template provides a no-op Cluster Provider that is ready to be deployed by the openmcp-operator in a real OpenControlPlane environment. It also provides a test skeleton that uses cluster-provider-kind to demonstrate how a Cluster Provider can be validated with openmcp-testing.

By the end of this guide, you should have a solid understanding of how the template and the resulting cluster provider works and be ready to build a real world service such as cluster-provider-gardener.

Prerequisites​

Start by creating a new repository for your Cluster Provider using the cluster-provider-template. Click "Use this template" button on the GitHub page and give your new repository a name that reflects the cluster kind it provides, e.g. cluster-provider-kind creates Kubernetes clusters using kind.

Clone the newly created repository to your local machine and open it with your favorite IDE.

Finally, ensure that you have the following binaries available in your path to execute the code generation of the template:

Cluster Provider Template Usage​

Run the following command to verify that everything works without applying any changes to disk.

task template:generate-provider dryrun=true

The generate-provider task supports the following arguments to customize the resulting Cluster Provider:

  • dryrun: Print in-memory result to stdout without altering any files (default false)
  • name: Name of the Cluster Provider (default "example"). Note that it is expected to be the suffix of cluster-provider-x, e.g. cluster-provider-foo -> name=foo.
  • module The go module name of your Cluster Provider (default "github.com/openmcp-project/cluster-provider-example")

Run generate-provider without dryrun to apply the result to disk, e.g.:

task template:generate-provider name=foo module=github.com/yourorg/cluster-provider-foo

The template generates a fully functional provider with 3 controllers that can be executed and deployed on your local machine using cluster-provider-kind and openmcp-testing.

To run the generated end-to-end test, init the build submodule and execute task test-e2e:

git submodule update --init --recursive
task test-e2e

This test bootstraps a complete local OpenControlPlane installation with all required components, including:

  • The platform cluster where your Cluster Provider is managed by the openmcp-operator.
  • The onboarding cluster that end users interact with.

For a visual overview of how a cluster provider fits into an OpenControlPlane installation, refer to the cluster provider deployment model.

Project Structure​

The cluster provider template is built with kubebuilder, so the project structure is similar to most Kubernetes controllers:

  • api/ contains the types and their CRDs which the crd manager will install during the initCommand.
  • cmd/ contains the entrypoint of the provider with init + run commands expected by the openmcp-operator to deploy the Cluster Provider.
  • internal/controller/ contains Reconcile functions for each controller where you implement your provider specific reconcile logic.

If you are new to implementing Kubernetes controllers, consider completing building a CronJob tutorial before returning to this guide. The rest of this guide highlights the most important steps to create a cluster provider and the differences compared to a regular Kubernetes controller.

Implementing a ClusterProvider​

Provider Configuration​

Most ClusterProviders will require some form of configuration (see Cluster Provider Design). Since the provider deployment does not allow passing in configuration via an argument to the binary directly, they need to read the configuration from a k8s resource. Depending on the provider, it might even allow multiple configuration resources and/or reconcile them instead of just reading them statically.

The template contains a basic config controller and an empty ProviderConfig API that is installed by the init job:

api/v1alpha1/config_types.go
// ProviderConfigSpec defines the desired state of ProviderConfig
type ProviderConfigSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "task generate" to regenerate code after modifying this file
}
cmd/cluster-provider-foo/app/init.go
func (o *InitOptions) Run(ctx context.Context) error {
  ...
	// apply CRDs
	log.Info("Creating/updating CRDs")
	crdManager := crdutil.NewCRDManager(openmcpconst.ClusterLabel, crds.CRDs)
	crdManager.AddCRDLabelToClusterMapping(clustersv1alpha1.PURPOSE_PLATFORM, o.PlatformCluster)
	if err := crdManager.CreateOrUpdateCRDs(ctx, &log); err != nil {
		return fmt.Errorf("error creating/updating CRDs: %w", err)
	}

	log.Info("Finished init command")
	return nil
}

The template controller contains a basic reconcile implementation where you need to add your provider specific logic (see the following Cluster Profiles section):

internal/controller/config/controller.go
func (r *ProviderConfigReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
  ...
}

Cluster Profiles​

Out of the configuration(s), the ClusterProvider has to generate ClusterProfile resources. They serve as some kind of service discovery and look like this:

apiVersion: clusters.openmcp.cloud/v1alpha1
kind: ClusterProfile
metadata:
  name: default.gardener.mcpd-gcp-large
spec:
  providerConfigRef:
    name: mcpd-gcp-large
  providerRef:
    name: gardener
  supportedVersions:
  - version: 1.33.3
  - deprecated: true
    version: 1.33.2
  - version: 1.32.7
  - deprecated: true
    version: 1.32.6
  - deprecated: true
    version: 1.32.5
  - deprecated: true
    version: 1.32.4
  - deprecated: true
    version: 1.32.3
  - deprecated: true
    version: 1.32.2

spec.providerRef is the name of the ClusterProvider that created this ClusterProfile. It should be filled with the value that the provider received via its --provider-name argument.

spec.providerConfigRef is the name of the provider configuration that is responsible for this profile. Whether this refers to an actual k8s resource, an internal value or just a static string depends on the provider implementation. It is used as a label value though and therefore has to match the corresponding regex.

spec.supportedVersions is a list of kubernetes versions that are supported by this provider for this profile.

The name of the ClusterProfile can be freely chosen. In this example, it follows the format X.Y.Z, where X is the environment name, Y is the name of the ClusterProvider, and Z is the name of the provider configuration that created this profile. A naming scheme like this avoids potential conflicts between multiple ClusterProviders (or multiple instances of the same ClusterProvider).

ClusterProfile resources are cluster-scoped and do not have a status.

Note that each ClusterProvider must at least generate one ClusterProfile in order to be usable.

Cluster Management​

The main purpose of ClusterProviders is the management of k8s clusters. Each ClusterProvider therefore needs a controller that reconciles the Cluster resource, which looks like this:

apiVersion: clusters.openmcp.cloud/v1alpha1
kind: Cluster
metadata:
  annotations:
    clusters.openmcp.cloud/providerinfo: foobar
  labels:
    clusters.openmcp.cloud/k8sversion: 1.31.11
    clusters.openmcp.cloud/provider: gardener
  name: my-cluster
  namespace: my-namespace
spec:
  kubernetes:
    version: 1.32.8
  profile: default.myprovider.myprofile
  purposes:
  - my-purpose
  tenancy: Shared

Some information about the different fields:

  • The clusters.openmcp.cloud/k8sversion and clusters.openmcp.cloud/provider labels are not set by default. The cluster provider can populate them to allow for easier filtering or better column information in kubectl get.
    • Note that spec.kubernetes.version contains a desired k8s version, which does not have to match the actual k8s version that is displayed in the label.
  • The clusters.openmcp.cloud/providerinfo annotation can be used to hold additional provider-specific information. It is displayed as a column on kubectl get -o wide.
  • spec.kubernetes.version can contain a desired k8s version. If not set, the provider has to derive it from its configuration. The provider can decide to either throw an error or choose a version if an invalid/unsupported version is specified.
  • spec.profile is the most important field for a ClusterProvider. It references the ClusterProfile that should be used for this cluster.
    • The referenced profile contains a reference to the ClusterProvider it belongs to. Since multiple ClusterProviders can run in parallel, this allows a ClusterProvider to determine whether it is responsible for this cluster resource or not.
      • ClusterProviders must only ever act on Cluster resources that reference profiles belonging to themselves!
      • The profile is immutable.
    • This can also contain further configuration, e.g. for the Gardener ClusterProvider, each provider configuration (which is referenced in the profile) can specify a different Gardener landscape and/or project to use.
  • spec.purposes and spec.tenancy are mostly relevant for the scheduler and usually don't need to be evaluated by the ClusterProvider.

The template generates a basic cluster controller with an additional cluster profile watch to create reconcile requests on profile changes:

internal/controller/cluster/controller.go
func (r *ClusterReconciler) SetupWithManager(mgr manager.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&clustersv1alpha1.Cluster{}).
		Watches(&clustersv1alpha1.ClusterProfile{}, handler.EnqueueRequestsFromMapFunc(func(ctx context.Context, obj client.Object) []reconcile.Request {
			if obj == nil {
				return nil
			}
			// reconcile all clusters that reference this profile
			clusters := &clustersv1alpha1.ClusterList{}
			if err := r.platformCluster.Client().List(ctx, clusters, client.MatchingFields{
				"spec.profile": obj.GetName(),
			}); err != nil {
				logf.FromContext(ctx).Error(err, "failed to list cluster profiles")
				return nil
			}
			reqs := make([]reconcile.Request, len(clusters.Items))
			for i, cluster := range clusters.Items {
				reqs[i] = reconcile.Request{
					NamespacedName: client.ObjectKeyFromObject(&cluster),
				}
			}
			return reqs
		})).
		Complete(r)
}

Reconciliation Logic​

The reconciliation logic has to be placed into the Reconcile function of the cluster controller:

internal/controller/cluster/controller.go
func (r *ClusterReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
  ...
}

Before doing anything in a reconciliation, the ClusterProvider needs to check the operation annotation and whether it is responsible for the Cluster resource or not. For this, it has to check if it created the ClusterProfile that is referenced in spec.profile itself or if it was created by a different ClusterProvider. It can either keep track of created ClusterProfile resources internally or compare spec.providerRef.name in the profile to its own name (passed in via the --provider-name argument). If the name differs, another ClusterProvider is responsible for this resource and the ClusterProvider must not touch it.

The rest of the reconciliation logic is pretty much provider specific: If the Cluster resource has a deletion timestamp, delete the k8s cluster and everything that belongs to it and then remove the finalizer. Otherwise, ensure that there is a finalizer on the Cluster resource and create/update the actual k8s cluster.

Status Reporting​

Since creating, updating, or deleting k8s clusters can easily take several minutes, reporting the current status is very important here. It is recommended to make good use of the conditions that are part of the status. ClusterProviders must adhere to general status reporting rules.

In addition to the common status, the Cluster status contains a few more fields that can be set by the ClusterProvider:

  • apiServer should be filled with the k8s cluster's apiserver endpoint, as soon as it is known.
  • providerStatus can hold arbitrary data and is meant for provider-specific information. Using it is optional and no other controller will evaluate the contents of this field.

Note that any kind of kubeconfig should not be part of the cluster's status - access to the cluster is managed via AccessRequest resources.

Access Management​

ClusterProviders are not only responsible for creating and deleting k8s clusters, but also for managing access to their clusters. Controllers and human users can request access to a cluster by creating an AccessRequest resource which looks like this:

apiVersion: clusters.openmcp.cloud/v1alpha1
kind: AccessRequest
metadata:
  name: my-access
  namespace: my-namespace
  labels:
    # ClusterProviders must only act on AccessRequests where these two labels are set
    # and the value of the first one matches their own provider name.
    clusters.openmcp.cloud/provider: myprovider
    clusters.openmcp.cloud/profile: default.myprovider.myprofile
spec:
  clusterRef: # optional, takes precedence over requestRef if set
    name: my-cluster
    namespace: foo

  requestRef: # optional, at least one of clusterRef and requestRef must be set
    name: my-request
    namespace: bar

  token: # either token or oidc
    permissions:
    - name: foo # optional, not required usually
      namespace: test # optional, results in Role if set and in ClusterRole otherwise
      rules:
      - apiGroups:
        - "*"
        resources:
        - "*"
        verbs:
        - "*"
    roleRefs:
    - kind: ClusterRole
      name: my-clusterrole

  oidc: # either token or oidc
    name: my-oidc-provider
    issuer: https://oidc.example.com
    clientID: my-client-id
    usernameClaim: sub # optional
    usernamePrefix: "my-user:"
    groupsClaim: group # optional
    groupsPrefix: "my-group:"
    extraScopes:
    - foo
    roleBindings:
    - subjects:
      - kind: User
        name: foo
      - kind: Group
        name: bar
      roleRefs:
      - kind: ClusterRole
        name: my-cluster-role
      - kind: Role
        name: my-role
        namespace: default
    roles:
    - name: my-admin
      rules:
      - apiGroups:
        - "*"
        resources:
        - "*"
        verbs:
        - "*"

Note that, while the example shows both, an AccessRequest must have exactly one of spec.token and spec.oidc set, not both.

The reconciliation logic has to be placed into the Reconcile function of the access request controller:

internal/controller/accessrequest/controller.go
func (r *AccessRequest) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
  ...
}

Token-based Access​

If spec.token is set, a token-based access is requested. The ClusterProvider is expected to create a ServiceAccount, create Role (if namespace is not empty) and ClusterRole (if namespace is empty) resources for each entry in spec.token.permissions, and create RoleBinding and ClusterRoleBinding resources for each entry in spec.token.permissions and each entry in spec.token.roleRefs.

Since token-based access is based on standard RBAC and TokenRequest APIs, it should work on any k8s cluster and is expected to be supported by every ClusterProvider.

OIDC-based Access​

If spec.oidc is set, OIDC-based access is requested. Most fields within spec.oidc are required for setting up the trust relationship. extraScopes is meant to be used for the oidc-login kubectl plugin that handles OIDC authentication. roleBindings specifies (Cluster)RoleBindings that should be created, while roles can be used to construct additonal (Cluster)Roles.

Note that not every ClusterProvider might support OIDC-based access and requesting it could result in an error or a denied request.

The spec.oidc field contains a nested struct named OIDCProviderConfig that has a Default() method. Whenever reading data from this field, it is strongly recommended to have run the Default() method first, because it will take care of setting some defaults, such as appending a : suffix to the username and groups prefixes, if it doesn't exist.

The Preparation of AccessRequests​

From a 'raw' AccessRequest, it is not immediately obvious which ClusterProvider is responsible: If spec.clusterRef is not set, first the ClusterRequest that is referenced in spec.requestRef needs to be fetched. From there, the Cluster needs to be fetched, which again leads to the ClusterProfile and only then the provider knows whether it is responsible or not.

To avoid having to implement this flow in every ClusterProvider and have all ClusterProviders executing it whenever any AccessRequest changes, there exists a 'generic' AccessRequest controller that takes over this task. This generic controller reacts only on AccessRequest resources that do not have both the clusters.openmcp.cloud/provider and the clusters.openmcp.cloud/profile labels. It modifies the AccessRequest in the following way:

  • It adds the clusters.openmcp.cloud/provider label with the provider name (extracted from the ClusterProfile) as value.
  • It adds the clusters.openmcp.cloud/profile label with the ClusterProfile name as value.
  • If spec.clusterRef is empty, it resolves the ClusterRequest reference and fills spec.clusterRef with the information from the ClusterRequest's status.

This means that the AccessRequest controller in a ClusterProvider must only act on AccessRequests that have both of the aforementioned labels set. They can then expect spec.clusterRef to be set and don't need to check for spec.requestRef.

It is recommended to use event filtering to avoid reconciling AccessRequests that belong to another provider or have not yet been prepared by the generic controller. The template configures the access request controller accordingly with the IsClusterProviderResponsibleForAccessRequest filter:

internal/controller/accessrequest/controller.go
func (r *AccessRequestReconciler) SetupWithManager(mgr manager.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&clustersv1alpha1.AccessRequest{}, builder.WithPredicates(
			predicate.NewPredicateFuncs(func(obj client.Object) bool {
				ar, ok := obj.(*clustersv1alpha1.AccessRequest)
				if !ok {
					return false
				}
				return libutils.IsClusterProviderResponsibleForAccessRequest(ar, r.providerName)
			}),
		)).
		Owns(&corev1.Secret{}). // watch the managed kubeconfig
		Complete(r)
}