CustomResourceDefinition must have at most 262144 bytes

Argo CD is a popular GitOps tool that offers a way for developers to run continuous deployments in Kubernetes through declarative configuration. It runs as a Kubernetes controller that monitors the state of resources in your cluster and compares this against the desired state for the cluster. Resources that are “out-of-sync” — have a different live state than the desired state — can be synced to the desired state either manually or automatically. Thus Argo CD is commonly used to manage the life cycle for Kubernetes workloads.

Enforcing the desired state is meant to help with situations like changing the container image tag (.e.g., upgrading or downgrading) in which case the resolution would be to spin-up new pods with the new image and then remove the out pods. However sometimes an Argo CD Application can be out-of-sync for superficial reasons. One such example is this message:

$ kubectl get application seldon-core-operator -o=jsonpath="{.status.conditions[0].message}"

Failed sync attempt to 1.13.1: one or more objects failed to apply, reason:
CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid:
metadata.annotations: Too long: must have at most 262144 bytes (retried 5 times).

In this case, Argo CD shows as out-of-sync and displays a SyncError for the seldon-core-operator Application. This can happen for Applications with large resources, like Custom Resource Definitions (CRDs), because they are too large to diff.

The best solution I’ve found is to use the ‘replace’ sync options (set Replace=true and ApplyOutOfSyncOnly=true). This tells Argo CD to replace (kubectl replace) the existing resources when the Seldon Core Operator resources are out-of-sync, rather than applying (kubectl apply) changes. See the manifest below for the details:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: seldon-core-operator
spec:
  project: default
  source:
    repoURL: 'https://storage.googleapis.com/seldon-charts'
    targetRevision: 1.13.1
    chart: seldon-core-operator
  destination:
    server: 'https://kubernetes.default.svc'
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - Replace=true
    - ApplyOutOfSyncOnly=true

So far I’ve only seen this problem for Custom Resource Definitions. It’s something I first encountered with Seldon Core but I’ve since noticed other cases (like Prometheous). It would be nice if developers could try to reduce the size of their CRDs but that’s unlikely and may be too constraining. Alternatively, enabling developers to configure this behavior for all CRDs (or another resource type) at a global or project-level would be another great option.