Best practices

Best practices in OKD/Kubernetes

A good resource to get you started with best practices is learnk8s suggestions.

• Try to avoid using persistent volumes for "state"(for example DB-clusters in OKD) that is difficult to maintain/manage, rather use a dedicated database server as backend instead in such cases.
• Have everything in git and use our ArgoCD-service to publish applications. If not, we cannot assure a proper restore of data if something gets removed by accident.
• Secrets should not be stored plain-text in git, nor in container images. Use Gitlab CI/CD variables, Kubernetes secrets or use our SOPS-plugin in ArgoCD to have encrypted secrets stored in git.
• Try to keep your image sizes small to speed up your deployments. This will also increase your security since there are fewer potential vectors of attack.
• When using large images, try to avoid imagePullPolicy: Always. Rather use imagePullPolicy: IfNotPresent or ImageStreams
• Always try to define resources and limits in your deployments directly. This way one can avoid Out-Of-Memory events caused by specific pods taking up too much resources!

meaning:

Always configure memory request == memory limit

Always configure cpu request, but without cpu limits

• Avoid overcommitting cpu/memory requests and limits in your available nodes, since this can lead to a node evicting pods to other nodes when being overcommitted over a long time. This will also trigger a billable alert to oncall.

For example, when 4vCPU is available on a specific node, this roughly translates to 4000 millicores where 3500 are actually usable. If 4 containers then are running on this node with 1000 millicores request each, you have overcommitted the node.

In most cases a simple adjustment of the requests and limits in all manifests is the way to go though. If more resources are really needed when specifying requests/limits, additional nodes or higher specs can be a solution.

• Try to consider adding Liveness and Readiness probes to your deployments.

• Due to increasing security requirements of upstream Kubernetes/Openshift, run your pods/containers as least-privilege as possible(without root). We can contribute here whenever this desired.

• Deployments of applications should be redundant(=multiple replicas) and spread over multiple worker nodes. Every deployment in the cluster should be able to withstand random restarts/evictions. There are multiple ways to achieve this.

Advanced methods for spreading pods over multiple worker nodes: Pod Anti-affinity is one of the best ways to achieve this.

Example:

  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                - grafana
            topologyKey: kubernetes.io/hostname
          weight: 100

- When running deployments, we advise you to consider quotas. In fact, we apply quotas to non-production namespaces by default. These quotas varies based on the underlying hardware. We do this to protect production namespaces incase of a bad configuration in other namespaces. Afterall, namespaces share the same nodes and therefore the same computational power.

To see if you have a quota enabled:

oc get quota -o yaml

To see if this affects your deployments:

oc get events

Consider applying from the following example:

spec:
  template:
    spec:
      containers:
        name: rlcp-app
        ports:
          - containerPort: 8080
            protocol: TCP
        resources: {}

to your pods or deployment manifests:

spec:
  template:
    spec:
      containers:
        name: rlcp-app
        ports:
          - containerPort: 8080
            protocol: TCP
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi

Consult the upstream kubernetes documentation for more information about resource management.