Aggregator Pattern Pitfalls

One of the features you would expect BizTalk to accomplish is in built BATCHING, where you need to send messages to an external system in Batch Mode, as usual a batching system depends on two variables:

• Maximum Batch Size

OR

• Batching Interval (Time Frame)

Microsoft was kind enough to provide a sample solution for batching. The solution is found here, after BizTalk is installed:

C:\Program Files (x86)\Microsoft BizTalk Server 2006\SDK\Samples\Pipelines\Aggregator

Or

C:\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\Pipelines\Aggregator

Just run the setup.bat file to get it installed on your BizTalk instance. They are even kind enough to provide 4 instance documents to drop in the receive location and then what will happen is the final send location will have them batched into one message, nice hey!

How it works is an envelope is used to batch messages within a pipeline. I highly recommend you play with this SDK, even though it is bugged, you gain good understanding of envelopes in BizTalk.

BUT NOT SO FAST!

What Microsoft fails to tell you is that if you need to batch more than 4 messages you will get ZOMBIES in BizTalk, yes that is right, install the SDK, and create 8 or more instance1.txt files and drop them all at the same time in the receive location, let’s say we drop 80, we should get one message out, instead, you get failed instances in BizTalk, as Zombies are created, how it happens is due to the Orchestration design problem. The orchestration is configured to listen for messages, the problem is when it is finalizing the final batch message it is possible that a new message from the BizTalk message box can sneak into the Orchestration receive handler, but the final message is already created, so this new message that crept into the process becomes a zombie and you get stuck messages in the BizTalk message box!

Many people on the net use a listen shape, and use a timer to manage batching, Microsoft’s SDK approach is a loop and use MAX BATCH SIZE, and either method will cause zombie records. Look at the orchestration below:

 

When the loop ends, and processing continues, it is possible that more messages from the message box arrive! This is what causes stuck messages in the BizTalk Message box. Errors will be like this:

0xC0C01B4C The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended.

So as far as I am concerned Microsoft have some serious work to do on BizTalk to address this problem, I feel that Batching is core requirement for any workflow system, and they should address this, if you read their articles they mention the problem, BUT DO NOT PROVIDE A SOLUTION.

There are solutions, we use a MSMQ or SQL table to manage batching, but I feel this is not a nice and elegant approach.

Summary

Microsoft need to address this problem in BizTalk or provide the community with a Batching Adapter, it is about time they get this done, this problem has been around now for far too long!

http://msdn.microsoft.com/en-us/library/bb203853.aspx

http://msdn.microsoft.com/en-us/library/aa561361.aspx

http://msdn.microsoft.com/en-us/library/ms942189.aspx

Thanks Microsoft for providing us with a flawed SDK sample that only works when you batch small message numbers (Less than 8)! Please take this opportunity to please the BizTalk community and provide us with a professional batching adapter.

What interested me the most or amuses me is the various authors on BizTalk books that have discussed this pattern, and not once mentioned the bug, and reviewing their solution has shown to also have the bug, but then again, most of the examples for BizTalk on educational resources are always with small batch sizes like a flight agency/hotel booking company, where you have a convoy of max 2 messages, not one of them have discussed enterprise batching solutions.

Hopefully we see some improvements in this regard soon.

• Uncategorized