Production guidelines on Kubernetes
Cluster and capacity requirements
Dapr support for Kubernetes is aligned with Kubernetes Version Skew Policy.
For a production-ready Kubernetes cluster deployment, we recommended you run a cluster of at least 3 worker nodes to support a highly-available control plane installation.
Use the following resource settings as a starting point. Requirements will vary depending on cluster size, number of pods, and other factors, so you should perform individual testing to find the right values for your environment:
|Operator||Limit: 1, Request: 100m||Limit: 200Mi, Request: 100Mi|
|Sidecar Injector||Limit: 1, Request: 100m||Limit: 200Mi, Request: 30Mi|
|Sentry||Limit: 1, Request: 100m||Limit: 200Mi, Request: 30Mi|
|Placement||Limit: 1, Request: 250m||Limit: 150Mi, Request: 75Mi|
|Dashboard||Limit: 200m, Request: 50m||Limit: 200Mi, Request: 20Mi|
NoteFor more info, read the concept article on CPU and Memory resource units and their meaning.
When installing Dapr using Helm, no default limit/request values are set. Each component has a
resources option (for example,
dapr_dashboard.resources), which you can use to tune the Dapr control plane to fit your environment.
The Helm chart readme has detailed information and examples.
For local/dev installations, you might simply want to skip configuring the
The following Dapr control plane deployments are optional:
- Placement: needed to use Dapr Actors
- Sentry: needed for mTLS for service to service invocation
- Dashboard: needed to get an operational view of the cluster
Sidecar resource settings
To set the resource assignments for the Dapr sidecar, see the annotations here. The specific annotations related to resource constraints are:
If not set, the Dapr sidecar will run without resource settings, which may lead to issues. For a production-ready setup it is strongly recommended to configure these settings.
For more details on configuring resource in Kubernetes see Assign Memory Resources to Containers and Pods and Assign CPU Resources to Containers and Pods.
Example settings for the Dapr sidecar in a production-ready setup:
|Limit: 300m, Request: 100m||Limit: 1000Mi, Request: 250Mi|
NoteSince Dapr is intended to do much of the I/O heavy lifting for your app, it’s expected that the resources given to Dapr enable you to drastically reduce the resource allocations for the application.
The CPU and memory limits above account for the fact that Dapr is intended to support a high number of I/O bound operations. It is strongly recommended that you use a monitoring tool to get a baseline for the sidecar (and app) containers and tune these settings based on those baselines.
When deploying Dapr in a production-ready configuration, it is recommend to deploy with a highly available (HA) configuration of the control plane, which creates 3 replicas of each control plane pod in the dapr-system namespace. This configuration allows the Dapr control plane to retain 3 running instances and survive individual node failures and other outages.
For a new Dapr deployment, the HA mode can be set with both the Dapr CLI and with Helm charts.
For an existing Dapr deployment, enabling the HA mode requires additional steps. Please refer to this paragraph for more details.
Deploying Dapr with Helm
Visit the full guide on deploying Dapr with Helm.
Instead of specifying parameters on the command line, it’s recommended to create a values file. This file should be checked into source control so that you can track its changes.
For a full list of all available options you can set in the values file (or by using the
--set command-line option), see https://github.com/dapr/dapr/blob/master/charts/dapr/README.md.
Instead of using either
helm install or
helm upgrade as shown below, you can also run
helm upgrade --install - this will dynamically determine whether to install or upgrade.
# Add/update a official Dapr Helm repo. helm repo add dapr https://dapr.github.io/helm-charts/ # or add/update a private Dapr Helm repo. helm repo add dapr http://helm.custom-domain.com/dapr/dapr/ \ --username=xxx --password=xxx helm repo update # See which chart versions are available helm search repo dapr --devel --versions # create a values file to store variables touch values.yml cat << EOF >> values.yml global: ha: enabled: true EOF # run install/upgrade helm install dapr dapr/dapr \ --version=<Dapr chart version> \ --namespace dapr-system \ --create-namespace \ --values values.yml \ --wait # verify the installation kubectl get pods --namespace dapr-system
This command will run 3 replicas of each control plane service in the dapr-system namespace.
NoteThe Dapr Helm chart automatically deploys with affinity for nodes with the label
kubernetes.io/os=linux. You can deploy the Dapr control plane to Windows nodes, but most users should not need to. For more information see Deploying to a Hybrid Linux/Windows K8s Cluster.
Upgrading Dapr with Helm
Dapr supports zero-downtime upgrades. The upgrade path includes the following steps:
- Upgrading a CLI version (optional but recommended)
- Updating the Dapr control plane
- Updating the data plane (Dapr sidecars)
Upgrading the CLI
To upgrade the Dapr CLI, download the latest version of the CLI and ensure it’s in your path.
Upgrading the control plane
See steps to upgrade Dapr on a Kubernetes cluster.
Updating the data plane (sidecars)
The last step is to update pods that are running Dapr to pick up the new version of the Dapr runtime.
To do that, simply issue a rollout restart command for any deployment that has the
kubectl rollout restart deploy/<Application deployment name>
To see a list of all your Dapr enabled deployments, you can either use the Dapr Dashboard or run the following command using the Dapr CLI:
dapr list -k APP ID APP PORT AGE CREATED nodeapp 3000 16h 2020-07-29 17:16.22
Enabling high-availability in an existing Dapr deployment
Enabling HA mode for an existing Dapr deployment requires two steps:
Delete the existing placement stateful set:
kubectl delete statefulset.apps/dapr-placement-server -n dapr-system
Issue the upgrade command:
helm upgrade dapr ./charts/dapr -n dapr-system --set global.ha.enabled=true
You delete the placement stateful set because, in the HA mode, the placement service adds Raft for leader election. However, Kubernetes only allows for limited fields in stateful sets to be patched, subsequently failing upgrade of the placement service.
Deletion of the existing placement stateful set is safe. The agents will reconnect and re-register with the newly created placement service, which will persist its table in Raft.
Recommended security configuration
When properly configured, Dapr ensures secure communication. It can also make your application more secure with a number of built-in features.
It is recommended that a production-ready deployment includes the following settings:
Mutual Authentication (mTLS) should be enabled. Note that Dapr has mTLS on by default. For details on how to bring your own certificates, see here
App to Dapr API authentication is enabled. This is the communication between your application and the Dapr sidecar. To secure the Dapr API from unauthorized application access, it is recommended to enable Dapr’s token based auth. See enable API token authentication in Dapr for details
Dapr to App API authentication is enabled. This is the communication between Dapr and your application. This ensures that Dapr knows that it is communicating with an authorized application. See Authenticate requests from Dapr using token authentication for details
All component YAMLs should have secret data configured in a secret store and not hard-coded in the YAML file. See here on how to use secrets with Dapr components
The Dapr control plane is installed on a dedicated namespace such as
Dapr also supports scoping components for certain applications. This is not a required practice, and can be enabled according to your security needs. See here for more info.
Service account tokens
By default, Kubernetes mounts a volume containing a Service Account token in each container. Applications can use this token, whose permissions vary depending on the configuration of the cluster and namespace, among other things, to perform API calls against the Kubernetes control plane.
When creating a new Pod (or a Deployment, StatefulSet, Job, etc), you can disable auto-mounting the Service Account token by setting
automountServiceAccountToken: false in your pod’s spec.
It is recommended that you consider deploying your apps with
automountServiceAccountToken: false to improve the security posture of your pods, unless your apps depend on having a Service Account token. For example, you may need a Service Account token if:
- You are using Dapr components that interact with the Kubernetes APIs, for example the Kubernetes secret store or the [Kubernetes Events binding]https://docs.dapr.io/reference/components-reference/supported-bindings/kubernetes-binding/).
Note that initializing Dapr components using component secrets stored as Kubernetes secrets does not require a Service Account token, so you can still set
automountServiceAccountToken: falsein this case. Only calling the Kubernetes secret store at runtime, using the Secrets management building block, is impacted.
- Your own application needs to interact with the Kubernetes APIs.
Because of the reasons above, Dapr does not set
automountServiceAccountToken: false automatically for you. However, in all situations where the Service Account is not required by your solution, it is recommended that you set this option in the pods spec.
Tracing and metrics configuration
Dapr has tracing and metrics enabled by default. It is recommended that you set up distributed tracing and metrics for your applications and the Dapr control plane in production.
If you already have your own observability set-up, you can disable tracing and metrics for Dapr.
To configure a tracing backend for Dapr visit this link.
For metrics, Dapr exposes a Prometheus endpoint listening on port 9090 which can be scraped by Prometheus.
To setup Prometheus, Grafana and other monitoring tools with Dapr, visit this link.
The Dapr Operator service includes an injector watchdog which can be used to detect and remediate situations where your application’s pods may be deployed without the Dapr sidecar (the
daprd container) when they should have been. For example, it can assist with recovering the applications after a total cluster failure.
The injector watchdog is disabled by default when running Dapr in Kubernetes mode and it is recommended that you consider enabling it with values that are appropriate for your specific situation.
Refer to the documentation for the Dapr operator service for more details on the injector watchdog and how to enable it.
Watch this video for a deep dive into the best practices for running Dapr in production with Kubernetes