Skip to main content
Version: main 🚧

Custom resources

Supported Configurations
Running the control plane as a container with:

vCluster allows you to sync custom resources from the tenant cluster to the control plane cluster. This is useful for syncing resources that are not included in the default sync behavior.

This feature only works for resources that have a corresponding CustomResourceDefinition (CRD) installed in the control plane cluster.

If a synced custom resource creates additional resources in the control plane cluster, vCluster attempts to detect and sync those resources back to the tenant cluster. For example, a custom resource that creates a Secret has that Secret automatically synced into the tenant cluster.

vCluster also adds the necessary cluster and namespace-level RBAC permissions to retrieve the CRD and sync the corresponding resources.


Only Namespace-Scoped Resource

This feature currently only works for namespace-scoped resources only.

Namespace syncing

If you want to sync many custom resources, consider using namespace syncing.

Enable custom resource syncing​

To enable custom resource syncing from the tenant cluster to the control plane cluster, first identify which CustomResourceDefinitions (CRDs) you want to sync by running kubectl get crds. Then, add the name of each CRD to the customResources section under sync in your vcluster.yaml configuration file.

Although the custom resources themselves are synced from the tenant cluster to the control plane cluster, vCluster also copies the corresponding CRDs from the control plane cluster into the tenant cluster. This ensures the tenant cluster can understand and manage those custom resources correctly.

By default, you do not need to specify an API version when syncing a custom resource. If the CRD defines multiple versions and no version is explicitly set, vCluster uses the storage version. To sync a specific version, such as when the storage version is not suitable, you can define that version directly in the vcluster.yaml.

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true

Versioned example​

To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a /, using the format <resource>/<version> in your configuration.

The specified version must exist in the control plane cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the control plane cluster's storage version, vCluster automatically converts custom resources between the two versions.

For example, to use version v1 of the certificaterequests.cert-manager.io CRD, specify:

Configure CR sync to host
sync:
  toHost:
    customResources:
      certificaterequests.cert-manager.io/v1:
        enabled: true
Unique CRD versions

vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the tenant cluster fails to start.

Upgrade custom resources with explicit CRD API versions​

vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the tenant cluster fails to start.

vCluster modifies the CRD inside the tenant cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the tenant cluster or change the sync configuration.

Upgrade from no version to specified version​

When upgrading from an unversioned CRD to a versioned one (certificaterequests.cert-manager.io → certificaterequests.cert-manager.io/v1), vCluster checks the CRD's storage version. If the storage version is not v1, vCluster updates the CRD to include v1 as a new version in addition to the current storage version.

Upgrade from one specified version to another specified version​

When upgrading between versioned CRDs (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io/v2), the CRD is updated to include the new version v2, and the old version v1 is kept as well.

Upgrade from a specified version to no version specified​

When upgrading from a versioned CRD to an unversioned one (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io), the behavior depends on the host's storage version:

  • If the CRD's storage version in the control plane cluster is v1, nothing happens and the CRD in the tenant cluster remains as certificaterequests.cert-manager.io/v1.
  • If the storage version is not v1, vCluster updates the CRD to include the storage version as a new version in addition to v1.

Patches​

Use sync.toHost.customResources.<resource>.patches to transform custom resource fields while syncing to the control plane cluster. Custom resources support expression patches, reference patches, and labels patches. See Patching synced resources for the full syntax and directionality rules.

Reference patches are useful when a custom resource field points to another synced resource, such as a Secret:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.secretName
          reference:
            apiVersion: v1
            kind: Secret

Expression patches are useful when the tenant and control plane cluster need different field values:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.hosts[*]
          expression: "value.startsWith('prod-') ? value : `prod-${value}`"
          reverseExpression: "value.startsWith('prod-') ? value.slice(5) : value"

Configure Istio waypoint Gateway via custom resources​

Use native Gateway API sync for core Gateway API resources

For syncing core Kubernetes Gateway API resources (Gateway, HTTPRoute, TLSRoute, ReferenceGrant, BackendTLSPolicy), use native Gateway API sync instead of custom-resource syncing. The example below covers Istio waypoint Gateways using the HBONE listener, which is outside the standard Gateway API channel and still requires the custom-resource path.

To use Istio waypoint Gateways with custom resources, install the Gateway API CRDs in the control plane cluster and then create the waypoint Gateway.

Install Gateway CRD in the host​

To create Gateway resources, install the Gateway API CustomResourceDefinition in the control plane cluster if it's not already present:

Install Gateway CRD
kubectl --context="${HOST_CTX}" get crd gateways.gateway.networking.k8s.io &> /dev/null || \
kubectl --context="${HOST_CTX}" apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Create waypoint gateway​

Create a waypoint gateway configuration:

waypoint-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Apply it to your control plane cluster:

Create Waypoint Gateway
kubectl --context="${HOST_CTX}" create -f waypoint-gateway.yaml --namespace="${VCLUSTER_HOST_NAMESPACE}"

After it is configured, you can configure your custom resources to sync the waypoint Gateway between the tenant and control plane clusters.

Config reference​

customResources {key: object} ​

CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work.

enabled required boolean ​

Enabled defines if this option should be enabled.

scope string ​

Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported.

patches object[] ​

Patches patch the resource according to the provided specification.

path required string ​

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression string ​

Expression transforms the value according to the given JavaScript expression.

reverseExpression string ​

ReverseExpression transforms the value according to the given JavaScript expression.

reference object ​

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string ​

APIVersion is the apiVersion of the referenced object.

apiVersionPath string ​

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string ​

Kind is the kind of the referenced object.

kindPath string ​

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath string ​

NamePath is the optional relative path to the reference name within the object.

namespacePath string ​

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels object ​

Labels treats the path value as a labels selector.