Dead Letter Channel

Camel supports the Dead Letter Channel from the EIP patterns using the DeadLetterChannel processor which is an Error Handler.

The Dead Letter Channel is an Error Handler that implements the principles from the Dead Letter Channel EIP. From the illustration above we can see the pattern is that if a message cannot be processed or fails during sending, it should be moved to a dead letter queue. The dead letter queue, is based on a Camel Endpoint, allowing you to use any of the many Camel components, such as a file system, database, or a log.

The Dead Letter Channel is similar to the default error handler, but with the following differences:

The dead letter channel is the only error handler that supports moving failed messages to a dedicated error queue, known as the dead letter queue (is a Camel endpoint).
Unlike the default error handler, the dead letter channel will, by default, handle exceptions and move the failed messages to the dead letter queue.
The dead letter channel is by default configured to not log any activity when it handles exceptions.
The dead letter channel supports using the original input message when a message is moved to the dead letter queue.

Using Dead Letter Error Handler

When using the dead letter channel error handler, you must configure the dead letter queue as an endpoint, so the handler knows where to move the failed messages. This is done a bit differently in the Java DSL and XML DSL.

For example, here’s how to log the message at ERROR level in Java DSL:

errorHandler(deadLetterChannel("log:dead?level=ERROR"));

And in Spring XML

<errorHandler id="myErrorHandler" type="DeadLetterChannel"
              deadLetterUri="log:dead?level=ERROR"/>

Pay attention to Spring XML DSL, the type attribute is used to declare which error handler to use, here its DeadLetterChannel. And in XML the <errorHandler> must be configured with an id. The id must then be enabled on either the <camelContext> (global scope), or per route that should use this error handler.

The following listing is an equivalent of the prior Java DSL listing:

<camelContext errorHandlerRef="myErrorHandler">
  <errorHandler id="myErrorHandler" type="DeadLetterChannel"
                deadLetterUri="log:dead?level=ERROR"/>
  <route>
    <from uri="direct:newOrder"/>
    <bean ref="orderService" method="validate"/>
    <bean ref="orderService" method="store"/>
  </route>
</camelContext>

Notice how we have enabled the error handler on the <camelContext> via errorHandlerRef which means this error handler is used globally. You can override this per route, and have individual routes used another error handler.

The DSLs is planned to be improved in the near future to have a unified way of configuring error handling across all DSLs, with Route Configuration. When fully implemented then configuring error handler in Java and XML DSL would be much more similar than currently.

Redelivery

It is common for a temporary outage or database deadlock to cause a message to fail to process; but the chances are if its tried a few more times with some time delay then it will complete fine. So we typically wish to use some kind of redelivery policy to decide how many times to try to redeliver a message and how long to wait before redelivery attempts.

The RedeliveryPolicy defines how the message is to be redelivered. You can customize things like:

how many times a message is attempted to be redelivered before it is considered a failure and sent to the dead letter channel
the initial redelivery timeout
using exponential backoff (i.e. the time between retries increases using a backoff multiplier)
whether to use collision avoidance to add some randomness to the timings
delay pattern
whether to allow redelivery during stopping/shutdown

Only when all redelivery attempts have failed then the message is moved to the dead letter queue.

Redelivery default values

Redelivery is disabled by default.

The default redeliver policy will use the following values:

maximumRedeliveries = 0
redeliveryDelay = 1000L (1 second)
maximumRedeliveryDelay = 60 * 1000L (60 seconds)
And the exponential backoff and collision avoidance is turned off.
The retriesExhaustedLogLevel are set to LoggingLevel.ERROR
The retryAttemptedLogLevel are set to LoggingLevel.DEBUG
Stack traces is logged for exhausted messages
Handled exceptions is not logged
allowRedeliveryWhileStopping is true
logExhaustedMessageHistory is false
logExhaustedMessageBody is disabled by default to avoid logging sensitive message body/header details. If this option is true, then logExhaustedMessageHistory must also be true.

The maximum redeliver delay ensures that a delay is never longer than the value, default 1 minute. This can happen if you turn on the exponential backoff.

The maximum redeliveries are the number of redelivery attempts. Setting the maximumRedeliveries to a negative value such as -1 will then always redelivery (unlimited). Setting the maximumRedeliveries to 0 will disable any redelivery attempt.

Camel will log delivery failures at the DEBUG logging level by default. You can change this by specifying retriesExhaustedLogLevel and/or retryAttemptedLogLevel.

You can turn logging of stack traces on/off. If turned off Camel will still log the redelivery attempt, it’s just much less verbose.

There are more options for fine-tuned logging configurations which you can find at RedeliveryPolicy.

What happens when an exception is thrown during sending to the dead letter queue

When the Dead Letter Channel moves a message to the dead letter endpoint, and a new Exception is thrown, then the new caused exception is logged at WARN level (can be turned off by setting logNewException to false) and Camel stops processing the message. This ensures that the Dead LetterChannel will always succeed.

If you do not want this behaviour you can configure deadLetterHandleNewException to false, which then causes the dead letter channel to fail and propagate back the new Exception, which causes Camel to fail processing the message.

What happens when an Exchange is moved to the dead letter queue

When all attempts of redelivery have failed the Exchange is moved to the dead letter queue. The exchange is then complete, and from the client point of view, the message is done being processed.

For instance configuring the dead letter channel as:

errorHandler(deadLetterChannel("jms:queue:dead")
    .maximumRedeliveries(3).redeliveryDelay(5000));

And in XML:

<errorHandler id="myErrorHandler" type="DeadLetterChannel"
              deadLetterUri="jms:queue:dead">
  <redeliveryPolicy maximumRedeliveries="3" redeliveryDelay="5000"/>
</errorHandler>

The Dead Letter error handler will clear the caused exception (setException(null)), by moving the caused exception to a property on the Exchange, with the key Exchange.EXCEPTION_CAUGHT. Then the Exchange is moved to the jms:queue:dead destination, and the client will not notice the failure.

Moving the original message to the dead letter queue

The option useOriginalMessage is used for routing the original input message instead of the current message that potentially is modified during routing.

For instance if you have this route:

from("jms:queue:order:input")
   .to("bean:validateOrder")
   .to("bean:transformOrder")
   .to("bean:handleOrder");

The route listen for JMS messages and validates, transforms and handle it. During this the Exchange payload is transformed/modified in the various bean stages.

Now suppose that if an exception is thrown we want to move the message to the dead letter queue. However the message that is moved to the dead letter queue (by default) is the current message. Suppose at one time there is an exception in the validateOrder, and another time an exception thrown by transformOrder, and yet also in handleOrder. In all these different situations the message may be changed.

By enabling useOriginalMessage on the dead letter channel, then the message that is moved to the dead letter queue, would be the original incoming message.

There is also a useOriginalBody option, which only keeps the original message body, and does not change the message headers.

// will use original message (body and headers)
errorHandler(deadLetterChannel("jms:queue:dead")
   .useOriginalMessage().maximumRedeliveries(5).redeliveryDelay(5000);

And in XML, you set useOriginalMessage=true on the <errorHandler> as shown:

<errorHandler id="myErrorHandler" type="DeadLetterChannel" useOriginalMessage="true"
              deadLetterUri="jms:queue:dead">
  <redeliveryPolicy maximumRedeliveries="5" redeliveryDelay="5000"/>
</errorHandler>

Then the messages routed to the jms:queue:dead is the original input. If we want to manually retry we can move the JMS message from the failed to the input queue, with no problem as the message is the same as the original we received.

Boundary of original message

The original input means the input message that are bounded by the current unit of work. A unit of work typically spans one route, or multiple routes if they are connected using internal endpoints such as direct or seda. When messages are passed via external endpoints such as JMS or HTT then the consumer will create a new unit of work, with the message it received as input as the original input. Also, some EIP patterns such as splitter, multicast, will create a new unit of work boundary for the messages in their sub-route (i.e. the split message); however these EIPs have an option named shareUnitOfWork which allows combining with the parent unit of work and ends up usinguse the parent original message.

Calling a processor before redelivery using OnRedelivery

When the Dead Letter Channel is doing redeliver its possible to configure a Processor that is executed just before every redelivery attempt. This can be used for the situations where you need to alter the message before its redelivered.

For example in Java DSL you can do:

errorHandler(deadLetterChannel("jms:queue:dead")
  .maximumRedeliveries(3)
  .onRedeliver(new MyOnRedeliveryProcessor());

And in XML DSL, you specify a bean id via onRedeliveryRef on the <errorHandler> as shown:

<bean id="myRedeliveryProcessor" class="com.foo.MyRedeliveryProcessor"/>

<errorHandler id="myErrorHandler" type="DeadLetterChannel" onRedeliveryRef="myRedeliveryProcessor"
              deadLetterUri="jms:queue:dead">
  <redeliveryPolicy maximumRedeliveries="3"/>
</errorHandler>

Camel also supports onException to use onRedeliver. This means you can do special on redelivery for different exceptions, as opposed to onRedelivery set on Dead Letter Channel (or Default Error Handler) can be viewed as global scoped.

Calling a processor before sending message to the dead letter queue using OnPrepareFailure

Before the exchange is sent to the dead letter queue, you can use onPrepare to allow a custom Processor to prepare the exchange, such as adding information why the Exchange failed.

For example the following processor adds a header with the exception message:

public static class MyPrepareProcessor implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        Exception cause = exchange.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class);
        exchange.getIn().setHeader("FailedBecause", cause.getMessage());
    }
}

Then configure the error handler to use the processor as follows:

errorHandler(deadLetterChannel("jms:dead").onPrepareFailure(new MyPrepareProcessor()));

Configuring this from Spring XML is done with the onPrepareFailureRef to refer to the processor as a <bean> as shown:

<bean id="myPrepare"
      class="org.apache.camel.processor.DeadLetterChannelOnPrepareTest.MyPrepareProcessor"/>

<errorHandler id="dlc" type="DeadLetterChannel" deadLetterUri="jms:dead" onPrepareFailureRef="myPrepare"/>

Calling a processor when an exception occurred

With the onExceptionOccurred you can call a custom processor right after an exception was thrown, and the Dead Letter Channel is about to decide what to do (either to schedule a redelivery, or move the message into the dead letter queue).

In other words this happens right after the exception was thrown, where you may want to do some custom logging, or something else.

For example, you may have a Processor that does some special logging:

public static class OnErrorLogger implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        Exception cause = exchange.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class);
        String msg = "Something went wrong due to " + cause.getMessage();
        // do some custom logging here
    }
}

You can then configure the Dead Letter Channel to use this as shown:

errorHandler(deadLetterChannel("jms:dead").onExceptionOccurred(new OnErrorLogger()));

Configuring this from Spring XML is done with the onExceptionOccurredRef to refer to the processor as a <bean> as shown:

<bean id="myErrorLogger" class="com.foo.OnErrorLogger"/>

<errorHandler id="dlc" type="DeadLetterChannel" deadLetterUri="jms:dead" onExceptionOccurredRef="myErrorLogger"/>

Redeliver Delay Pattern

Delay pattern is used as a single option to set a range pattern for delays. If used then the following options does not apply: (delay, backOffMultiplier, useExponentialBackOff, useCollisionAvoidance, maximumRedeliveryDelay).

The idea is to set groups of ranges using the following syntax: limit:delay;limit 2:delay 2;limit 3:delay 3;…;limit N:delay N

Each group has two values separated with colon

limit = upper limit
delay = delay in milliseconds. The groups are separated with semicolon. The rule of thumb is that the next groups should have a higher limit than the previous group.

Let’s clarify this with an example:

delayPattern=5:1000;10:5000;20:20000

That gives us 3 groups:

5:1000
10:5000
20:20000

Resulting in these delays for redelivery attempt:

Redelivery attempt number 1..4 = 0 millis (as the first group start with 5)
Redelivery attempt number 5..9 = 1000 millis (the first group)
Redelivery attempt number 10..19 = 5000 millis (the second group)
Redelivery attempt number 20.. = 20000 millis (the last group)

The first redelivery attempt is 1, so the first group should start with 1 or higher.

You can start a group with limit 1 to have a starting delay:

delayPattern=1:1000;5:5000

Redelivery attempt number 1..4 = 1000 millis (the first group)
Redelivery attempt number 5.. = 5000 millis (the last group)

There is no requirement that the next delay should be higher than the previous. You can use any delay value you like. For example with:

delayPattern=1:5000;3:1000

We start with 5 sec delay and then later reduce that to 1 second.

State of redelivery as message headers

When a message is redelivered the Dead Letter Channel will append the following headers to the message with the state of the redelivery:

Header Type Description

Header	Type	Description
`Exchange.REDELIVERED`	`Boolean`	Whether the message is redelivered
`Exchange.REDELIVERY_COUNTER`	`Integer`	The current redelivery attempt
`Exchange.REDELIVERY_MAX_COUNTER`	`Integer`	The maximum number of redeliveries configured (if any). This header is absent if you use `retryWhile` or have unlimited maximum redelivery configured.

Exchange.REDELIVERED

Boolean

Whether the message is redelivered

Exchange.REDELIVERY_COUNTER

Integer

The current redelivery attempt

Exchange.REDELIVERY_MAX_COUNTER

Integer

The maximum number of redeliveries configured (if any). This header is absent if you use retryWhile or have unlimited maximum redelivery configured.

Which endpoint failed

During routing messages Camel will store an exchange property with the most recent endpoint in use (send to):

String lastEndpointUri = exchange.getProperty(Exchange.TO_ENDPOINT, String.class);

The Exchange.TO_ENDPOINT have the constant value CamelToEndpoint.

This information is updated when Camel sends a message to any endpoint.

When for example processing the Exchange at a given Endpoint and the message is to be moved into the dead letter queue, then Camel also decorates the Exchange with another property that contains that last endpoint:

String failedEndpointUri = exchange.getProperty(Exchange.FAILURE_ENDPOINT, String.class);

The Exchange.FAILURE_ENDPOINT have the constant value CamelFailureEndpoint.

This allows for example you to fetch this information in your dead letter queue and use that for error reporting. This is particular usable if the Camel route uses dynamic EIPs such as Recipient List or ToD, where the target endpoint uri would be stored as information.

This information is kept on the Exchange even if the message was successfully processed by a given endpoint, and then later fails for example in a local Bean EIP processing instead. So beware that this is a hint that helps pinpoint errors to Endpoints, and not EIPs.

from("activemq:queue:foo")
    .to("http://someserver/somepath")
    .bean("foo");

Now suppose the route above, and a failure happens in the foo bean. Then the Exchange.TO_ENDPOINT and Exchange.FAILURE_ENDPOINT will still contain the value of http://someserver/somepath.

Which route failed

When a message is moved into the dead letter queue, then Camel will store the id of the route, where the message failed. This information can be obtained in Java via:

String failedRouteId = exchange.getProperty(Exchange.FAILURE_ROUTE_ID, String.class);

The Exchange.FAILURE_ROUTE_ID have the constant value CamelFailureRouteId.

This allows for example you to fetch this information in your dead letter queue and use that for error reporting.

Control if redelivery is allowed during stopping/shutdown

The option allowRedeliveryWhileStopping (default is true) controls whether redelivery is allowed or not, during stopping Camel or the route. This only applies for any potential new redelivery attempts; any currently ongoing redeliveries is still being executed.

This option can only disallow any redelivery to be executed after the stopping of a route/shutdown of Camel has been triggered.

If a redelivery is disallowed then a RejectedExecutionException exception is set on the Exchange, and it stops being routed. The outcome of the Exchange is then a failure due the RejectedExecutionException.