Designing BizTalk Solutions Tips

Hi Folks,

I have decided to start a series of posts in regards to implementing BizTalk solutions. In these solutions I will assume that the following goals are desirable:

Services

Loosely coupled architecture
Abstraction by implementing web service or WCF solutions
- These services must provide discrete components, that when orchestrated as a whole can provide logical business processes, this means that they should be re-useable.
Just because you implement web services or WCF solutions, does not mean you implementing SOA.
Entry points into the Service Bus must PROTECT data integrity. This means all WCF/WS* services must implement strict message contracts
- How many of you have been given a web service to consume by a third part and when you download their WSDL, you notice that you can submit any data, just a generic XmlDocument? Not good.
- Secondly by protecting the Service Bus from badly formed data, you will save yourself heaps of problems.
A â€œInternalâ€ XSD schema is used to orchestrate, route and update common artefacts. This schema is usually referred to as a canonical schema.
- This means a C# version of the XSD is generated either with xsd.exe, svcutil.exe or xsd2class (my favourite). Therefore always ensure these classes that are used to expose your service contracts are PARTIAL. You can then create a seperate partial class for CUSTOM implementations such as Schema/Serialization and manipulation routines.
- You may ask, why, we can have a BizTalk pipeline do this for us. Yes indeed you can. However many EXTERNAL facing services must be decoupled from the BizTalk system. This is where you can leverage strict service contracts to the outside world. Just because a WCF or Web Service accepts a message, does not mean it conforms to the XSD. Class generated code from XSD does not implement all the rules! e.g. Maximum Occurence and Minimum Occurrence.
- There can be some disadvantages here. If you implement a canonical schema and not much thought went into the design, then you go to have some serious problems later on down the line. INVEST allot of TIME with business analysts and users, so that your canonical schema matches all business processes.
- Any CHANGES to the canonical schema MUST be backward compatible.
- Use namespaces to provide logical sections in the canonical schema to limit the IMPACT of new changes to the schema.
- Let me put it this way. If the Canonical schema changes, your BizTalk Maps that map to this schema should still work.
Use Document Literal if you need the messaging payload in your services, this message is not bound to any distinct operation.
Use RPC style if you need a small subset of the data and need to clearly define what operation the message payload supports.
if you unsure of the above, then go for Document Literal, most flexible.
If using web services, implement SOAP security headers in external facing web services.
Ensure core services have decent logging capabilities and perhaps even a web page to view the Inbound data, this can save users allot of hassles when chatting to clients about submitted documents that are not processed.
You can use a generic XSD schema (ANY type) for internal processing, when the schema has already been validated. Why always validate the schema during multiple processes, if the entry point is valid? Sometimes this is necessary, however, when working deep within the service bus, you may find it easy to use a generic schema e.g. A Dispatching system, that already received validated documents and just needs to dispatch them to other services.
Ensure you web.configs etc for services are tuned to support multiple connections (Default is 2). if they are not you will see many send port instances in BizTalk ready to send, because the max connections on the web service is defaulted to 2!
http://grounding.co.za/blogs/romiko/archive/2009/04/11/biztalk-2006-optimising-soap-send-ports-and-sql-query-optimizer.aspx

   <system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "50"/>
    </connectionManagement>
    </system.net>

Ensure repeated calls to the same service will not changed the state of the message, unless you really intend to do so.
If there is a chance that the same message will be delivered may times during the day (Customer updates), then perhaps a subscription service model can be used, where the system will PULL the latest record. This will solve problems where you cannot guarantee the latest message for the same customer will be processed last. Also, by doing so, you can avoid messaging queuing (which can be slow).
There is nothing wrong incorporating solutions that PULL from web services, as the example above explains.
Use Windows Communication Foundation to communicate between services within the organisation if you can, especially if the communication will be on the same server (netbinding etc)

Canonical Schema

As I mentioned, INVEST allot of time defining, designing and implement this schema. get intermit with it.
Appending Service Bus metadata to the canonical schema is extremely Powerful. You can use this meta data for Property Schemas and Distinguished fields. You can put this data in a separate namespace. This gives you allot of flexibility.
When extracting data from SQL for BizTalk to consume, this is a good opportunity to see what SQL fields can be used in the canonical schema to provide metadata.
- Example: You may have a SharePoint/FTP/WCF address associated with the data from SQL. Why not extract this on the SQL receive location and append it to the metadata of the message payload? You can then leverage Dynamic Send Ports 🙂
In a nutshell, think through this, it can save you allot of time, and allow you to implement some call dynamic routing patterns, however the Schema must be really well designed and i did mentioned backward compatible.

Routing

Never use a pattern where an Orchestration is used to invoke a Business Rule Engine policy and uses the result of the rule execution to define a message routing. I have seen this happen allot, NEVER do it. It slows down the messaging engine. Use filters on Send Ports, this is loosely coupled.
Use Filters on Send Ports. PLAN your Property Schema.
If your content filters for any BizTalk application is more that 10 types of property schema members, then you have a design issue. Keep your Property Schema Small, ideally fewer that 10 elements.
Try to use dynamic send ports where it is possible to dynamically gather meta data about the destination end point.

Orchestration

Try not to use BizTalk Maps in an orchestration. Why? You are going to tightly couple your solution here. Rather implement maps at the entry and exit points. Send and Receive Ports.
Keep the persistent points on your orchestrations down to a minimum, use SCOPES to control persistent points.
Do not use the expression editor for complex logical, and do not TRY hack the expression editor by using a multitude of static serializable classes to get the job done, this is a sign something is wrong, usually exposing the correct distinguished properties can solve this problem!
I have seen to many implementations where Orchestration are used, just because well â€œThey easy to implementâ€. Most of the times you do not need to use them, where the BizTalk messaging engine will suffice. Reasons why people use them so much, is that not allot of thought goes into the design of the canonical schema.
DO NOT couple your orchestration directly to a WCF/Web Service port. It is just UGLY. Secondly you expose your internal orchestration logic to the outside world! You have no control how the contract is generated.
- Try and get orchestrations to receive data from DIRECT ports bound to the message box.
So direct ports is the way to go. Keep you service logic separate from orchestrations as mentioned. Use direct ports to the message box and just use filters on the message receive shaped. This gives you the best loosely coupled pattern.
If you are call external classes, ensure they are THREAD SAFE!

Failures

Use BizTalk’s routing of failed message subscription features to reroute failed messages. Do not underestimate how useful this can be e.g. Route failed messages to different document libraries in SharePoint for manual intervention, or to a custom human workflow database for web based fixing/editing.

BizTalk Mapper

Keep your maps SIMPLE. Do not implement complex logic here, if you do, then something is wrong with the design.
As I mentioned implement maps on the latest or earliest stage, not in the orchestration engine, unless you cannot help it.

In fact, I worked on a project where the Business required full control of mapping flat files to the canonical schema. They did not want to use the BizTalk Mapper. We built a custom ASP.NET mapper for them, and they can generate maps during runtime without the need for deployments of new maps. We leveraged Altova products, where they can use XSLT bases mapping and then load them into the cache (Compiled version of the xslt) where our service can call them at runtime. Altova has a really cool XSLT engine. We also extended this for extremely complex files so that a software factory and implement code, however these sort of files require an assembly update. However such files are only 5% of the file feeds, and dealing with over 300 file feeds, we can see that a web based mapper, that biztalk can consume via a service is an awesome way to loosely couple mappings from business processes.

I have seen some bad ass maps, where all I see is BLACK, not even the lines, with all sort of functoids, function calls and weird things going on, avoid complex mappings.

Remember large xml files BLOW up when processed by an XSLT engine. There are other ways to transform flat files if they are extremely large, do not always have to use XSLTâ€™s. However, design a solution that is flexible in this regard if you dealing with flat files that are really complex, where xslts or other transformation solutions can be implemented and are loosely coupled.

SharePoint Integration

The SharePoint Adapter for BizTalk is really powerful and rock solid, leverage SharePoint if you require message collaboration! We use dynamic SharePoint send ports so we have FULL control over custom columns in the document library. This allows us to publish the message AND metadata to SharePoint and leverage SharePoint document library views. E.g. Sending a file to SharePoint and using a custom column in the document library to mentioned the Country where the file came from, or other properties.

Well, I think thatâ€™s enough tips I can think of for today! I hope this helps. Now, back on the bright side, here are some pics i took on a wildlife expeditions 🙂 Yes, life is not all about geeking it to the max!

This was in the Kruger National Park, I was in a concession area, no fences, just me and the wild animals 🙂