Telemetry

Since Camel 4.11

This module is a common interface and API for distributed tracing/telemetry. It is not intended to be used directly by end users. This component is implemented concretely in following dependencies:

  • camel-opentelemetry2

  • camel-micrometer-observability

  • camel-telemetry-dev

Tracing an application in a distributed deployment is an important feature supported by Camel. As there are plenty of potential telemetry implementation out there we have created an abstract component whose goal is to define the structure and lifecycle of a trace from Camel point of view. This component acts as a trace manager, making sure that regardless of the concrete technology used, the traces are consistently treated by Camel applications.

Using any of the concrete implementations above will instrument your Camel application with the tooling required to generate and expose traces to be collected from an external system (typically a trace collector via an agent to be run when executing the application). However, this is hidden to the abstract implementation and must be delegated to the concrete implementation instead.

Configuration

The configuration properties for the Telemetry component are:

Option Default Description

traceProcessors

false

Trace inner processors.

excludePatterns

Sets exclude pattern that will disable tracing for those spans that matches the pattern. The variable is a comma separated values of filters to execute (eg, log*,direct*,setBody*, …​)

traceHeadersInclusion

false

Add the generated telemetry CAMEL_TRACE_ID and CAMEL_SPAN_ID Exchange headers.

Tracing structure

There are two important concept we are using in the abstraction. Trace and Span. A Trace is the resource we use to track the execution of a Route. It starts when the Route is fired and is closed when the Route execution ends. A Span is the unit of execution of any event fired by the Route execution. A Trace typically contains one or more Span, depending on the complexity ot the Route.

In order to clarify these concepts, let’s use the following Route as a reference:

from("timer:mytimer")
    .routeId("timer")
    .log("in timer route")
    .to("direct:anotherRoute")
    .to("http://my-service");

For each event triggered, we expect the generation of traces with a Span for the timer component which contains 2 Spans, one for direct and the other for http endpoints. Mind that the log is not considered as a span as indeed is an execution event of the timer endpoint. It’s interesting to notice that the direct endpoint would contain as many Spans as its Route defines. All those spans are wrapped in a Trace and would look like as it follows:

timer
├── direct
│   ├── ...
└── http

The above model is likely to be mapped one to one to the same related standard model (which is, Traces and Spans) in any of the concrete technology adopted. If the technology has a different or richer mapping, then, it will be the implementation that has to take care to do the mapping accordingly.

Tracing lifecycle

The camel-telemetry component takes care to hook into the Camel application lifecycle and capture those meaningful events we want to turn into traces. The component capture the following events:

  • RoutePolicySupport.onExchangeBegin()

  • RoutePolicySupport.onExchangeDone()

  • CamelEvent.ExchangeSendingEvent

  • CamelEvent.ExchangeSentEvent

The first two are in charge to generate a new Trace (or a Span) when an Exchange begins and close it accordingly. The last two are in charge to create and close a Span for each of the events executed by the Route.

Additionally we are also capturing:

  • LogListener.onLog()

The LogListener.onLog() is a special case which we use in order to capture any logging trace and store it as an event into the component endpoint Span.

Main features

The abstract component provides some important features. Here the most important ones.

Span storage

Camel uses a proprietary storage mechanism. The trace is serialized into each generated Exchange and the component is taking care to maintain the hierarchy and the consistency of the Spans, regardless of the threading model which is executing the application logic. Each concrete implementation may provide its additional storing mechanism which should be ignored for the scope of Camel application tracing.

Component exclusions

The component provide the possibility to exclude the trace of any component when using the excludePatterns parameter. This feature is not implementation specific.

Exchange headers inclusion

The component provide the possibility to include the generated trace and span into the Exchange header. These headers can be then used for any observability purposes, eg, included in MDC via camel-mdc component.

Component Span decoration

The component automatically includes certain useful parameter out of the box for the different components you may use within Camel. As an example, if you’re using the Kafka component (camel-kafka), then, it will include in the Kafka endpoint span a few useful information such as partition or offset which you will be able to verify later in the trace collector.

Implement a new component decorator

You can customize each component by adding or updating the specific decorator class contained in this package. Each class should extend either AbstractSpanDecorator, AbstractInternalSpanDecorator, AbstractHttpSpanDecorator or AbstractMessagingSpanDecorator, depending on the component kind (for example, the components dealing with messaging should extend the abstract messaging class).

There are two "identification" methods to override:

  • String getComponent()

  • String getComponentClassName()

They are used to provide a component name and above all to identify the component to which this decorator is bound (via its fully qualified class name).

Additionally you can alter the trace extending the following methods:

  • void beforeTracingEvent(…​);

  • void afterTracingEvent(…​);

These methods are the ones in charge to alter the trace before and after it reaches the telemetry endpoint (hence, altering it accordingly).

Beside that you will need to include the fully qualified name in the resources/META-INF/services/org.apache.camel.telemetry.SpanDecorator service provider file. This is the mechanism used by the telemetry component to load dynamically the available decorators which will be matched by the component fully qualified class name.

Distributed Tracing

Distributed tracing are required to be correlated between each other. This is quite important above all when you’re running a microservice oriented architecture. When a Camel application calls another Camel applications, then, there must be in place a mechanism to correlate traces. This is done via context propagation.

The upstream application must inject the context into the event sent (typically a traceparent header in the Exchange). The downstream application must extract the context from the event received (same traceparent header). The result will be a unique distributed tracing with the same Trace ID.

This feature is implementation specific, the abstraction just provide the interface that must be implemented concretely in each of the implementation.

Processor tracing

When this feature is enabled, you will be able to collect a finer grain number of Spans into a Trace. Each of the different endpoint processors will be collected. You can enable the feature using the traceProcessors parameter (default false).

enabling this feature will provide many more Spans for each Trace.

Implementation specific abstraction

the following chapter is dedicate exclusively to developers willing to create a concrete implementation for this component.

In order to simplify the implementation of any tracing technology the abstraction provides the following method to implement:

    /*
     * It has to be provided by the specific implementation
     */
    private SpanLifecycleManager spanLifecycleManager;

    protected abstract void initTracer();

The initTracer() is in charge to inject a concrete implementation of SpanLifecycleManager whose abstraction is:

public interface SpanLifecycleManager {

    Span create(String spanName, Span parent, SpanContextPropagationExtractor extractor);

    void activate(Span span);

    void deactivate(Span span);

    void close(Span span);

    void inject(Span span, SpanContextPropagationInjector injector);

}