Distributed tracing overview
Dapr uses the Open Telemetry (OTEL) and Zipkin protocols for distributed traces. OTEL is the industry standard and is the recommended trace protocol to use.
Most observability tools support OTEL, including:
The following diagram demonstrates how Dapr (using OTEL and Zipkin protocols) integrates with multiple observability tools.
Tracing is used with service invocaton and pub/sub APIs. You can flow trace context between services that uses these APIs. There are two scenarios for how tracing is used:
- Dapr generates the trace context and you propagate the trace context to another service.
- You generate the trace context and Dapr propagates the trace context to a service.
Scenario 1: Dapr generates trace context headers
Propagating sequential service calls
Dapr takes care of creating the trace headers. However, when there are more than two services, you’re responsible for propagating the trace headers between them. Let’s go through the scenarios with examples:
Single service invocation call
service A -> service B.
Dapr generates the trace headers in
service A, which are then propagated from
service A to
service B. No further propagation is needed.
Multiple sequential service invocation calls
service A -> service B -> propagate trace headers to -> service C and so on to further Dapr-enabled services.
Dapr generates the trace headers at the beginning of the request in
service A, which are then propagated to
service B. You are now responsible for taking the headers and propagating them to
service C, since this is specific to your application.
In other words, if the app is calling to Dapr and wants to trace with an existing trace header (span), it must always propagate to Dapr (from
service B to
service C, in this example). Dapr always propagates trace spans to an application.
NoteNo helper methods are exposed in Dapr SDKs to propagate and retrieve trace context. You need to use HTTP/gRPC clients to propagate and retrieve trace headers through HTTP headers and gRPC metadata.
Request is from external endpoint
from a gateway service to a Dapr-enabled service A.
An external gateway ingress calls Dapr, which generates the trace headers and calls
Service A then calls
service B and further Dapr-enabled services.
You must propagate the headers from
service A to
service B. For example:
Ingress -> service A -> propagate trace headers -> service B. This is similar to case 2.
Dapr generates the trace headers in the published message topic. These trace headers are propagated to any services listening on that topic.
Propagating multiple different service calls
In the following scenarios, Dapr does some of the work for you, with you then creating or propagating trace headers.
Multiple service calls to different services from single service
When you are calling multiple services from a single service, you need to propagate the trace headers. For example:
service A -> service B [ .. some code logic ..] service A -> service C [ .. some code logic ..] service A -> service D [ .. some code logic ..]
In this case:
service Afirst calls
service B, Dapr generates the trace headers in
- The trace headers in
service Aare propagated to
- These trace headers are returned in the response from
service Bas part of response headers.
- You then need to propagate the returned trace context to the next services, like
service D, as Dapr does not know you want to reuse the same header.
Scenario 2: You generate your own trace context headers from non-Daprized applications
Generating your own trace context headers is more unusual and typically not required when calling Dapr.
However, there are scenarios where you could specifically choose to add W3C trace headers into a service call. For example, you have an existing application that does not use Dapr. In this case, Dapr still propagates the trace context headers for you.
If you decide to generate trace headers yourself, there are three ways this can be done:
Standard OpenTelemetry SDK
You can use the industry standard OpenTelemetry SDKs to generate trace headers and pass these trace headers to a Dapr-enabled service. This is the preferred method.
You can use a vendor SDK that provides a way to generate W3C trace headers and pass them to a Dapr-enabled service.
W3C trace context
You can handcraft a trace context following W3C trace context specifications and pass them to a Dapr-enabled service.
Read the trace context overview for more background and examples on W3C trace context and headers.
- Observability concepts
- W3C Trace Context for distributed tracing
- W3C Trace Context specification
- Observability quickstart
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.