Hi Folks,

I wanted to see if I could use the WCF-Custom Adapter to match my scenario below:

Scenario:

I want to call a stored procedure with multiple arguments without using orchestrations and one of the parameters of the stored procedure is the actual BizTalk Message as XML SQL data type.

Consider a scenario where you do not want the overhead of generating metadata, creating an orchestration, deploying the orchestration, and executing the operation. For such a basic scenario, where the stored procedure takes a single parameter, you do not need to create an orchestration. Rather, you can configure a WCF-Custom or WCF-SQL send port to directly invoke the stored procedure. This topic demonstrates how to perform these tasks using the BizTalk Server Administration console.

Prerequisites:

WCF LOB Adapter SDK SP2

and the following

Microsoft BizTalk Adapter for SQL Server:

Ok, so either way, lets continue!

I managed to do it with this template in the WCF-Custom properties, however I found some limitations.

Here is a template that works, notice I HARD coded the Stage, Status and Id, call then constants for now.

<SetWorkflowRecord xmlns="http://schemas.microsoft.com/Sql/2008/05/Procedures/dbo">
<Message><bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007" encoding="string"/></Message>
<Stage>TWO</Stage>
<Status>SUC</Status>
<Id>968819</Id>
</SetWorkflowRecord>

How the above template will probably not be cool, who hard codes values into send ports….

However this template is pretty cool, we can actually use WCF to call a proc directly with multiple parameters, as long as the other parameters besides the Message is constants. Note, you can generate a default template using the Consume Adapter Service BizTalk Project Add-In, and then generate an instance.

I while back I developed a custom SQL adapter that can call a SQL procedure directly and match the scenario above, you know what I love about the custom adapter, is that it uses a STREAM and not a String to send the XML data to the stored procedure. So for large files, I can imagine a Out Of Memory Exception. Also the custom SQL adapter can parse Constants and promoted properties as parameters to a stored procedure, and that is very powerful stuff. We need this power in WCF-Custom or WCF-SQL adapters! I mean it. This allows us to avoid orchestrations and updategrams etc.

However, here is my wish list, and I hope a Microsoft BizTalk Developer finds this wish list:

I wish for a WCF-Custom adapter, where you can reference promoted properties in the WCF template properties. This would match what I can do with the custom SQL adapter we built. Where I can send XML, and promoted property values from BizTalk directly to a WCF adapter, without the need for developing complex WCF Chanel Model or WCF Service Model tools. I also wish that the Message is streams to the SQL server procedure.

So, perhaps in the near future, this type of template will work in WCF-Custom. Where the template has access to the BizTalk Property Schema. Notice below we no longer use encoding type of string to the message, will be faster, of course this sort of template will never work now, but nice to have one day? It could be an add in for the Consume Adapter Service BizTalk Project Add-In where you can drag promoted property fields into the template. This should be possible, since the following assembly has direct access to the message and all properties:

System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

<SetWorkflowRecord xmlns="http://schemas.microsoft.com/Sql/2008/05/Procedures/dbo">
<Message><bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007"/></Message>
<Stage><StageParam xmlns="http://MMIT.Workflow.Common.Schemas.WorkflowRecordPropertySchema" encoding="string"/></Stage>
<Status><StatusParam xmlns="http://MMIT.Workflow.Common.Schemas.WorkflowRecordPropertySchema" encoding="string"/></Status>
<Id><IdParam xmlns="http://MMIT.Workflow.Common.Schemas.WorkflowRecordPropertySchema" encoding="string"/></Id>
</SetWorkflowRecord>

StageParam, StatusParam and IdParam are promoted properties of the message going into the adapter, and this should be possible, the WCF architecture can support this, since they already have the fixed property schema for:

bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007

I am excited about WCF, but it can be much more powerful with BizTalk if it can hook into Promoted Properties and not encode the message as a string, but default it to a stream.

This will allow for some awesome low latency BizTalk Patterns…. I understand that we need to keep promoted properties under 10 or so, which is the case, but sometime you want to access promoted properties for database level operations, thus avoiding XPath on them again in the proc to save time if you already have it promoted.

I have migrated all send ports to WCF, so my solution is still simple, all I do is change the stored procedure to accept one parameter and then in the SQL proc, we XPATH the stage, status and ID values, which should be fine, however it will be slightly slower as in the stored procedure I run a SQL Message.Query to get the stage, status and ID :), I will do some performance test against both ways, WCF and the Custom Adapter. However if the WCF is slightly slower, I will probabyl stick with so that in the long run we are WCF compliant!

One thing I like very much is the Transaction Isolation Level and the fact we can use Read Committed 🙂 Also, I prefer Custom as it is more flexible than the WCF-SQL 🙂

Below are screenshots for setting up the adapter:

Click Configure

Then add your SQL URI and stored procedure name:

Then change the binding to SQLBinding, to do this, you must install the following:

WCF LOB Adapter SDK SP2
Microsoft BizTalk Adapter for SQL Server:

, eitheway, you should now be able to choose the SQLBinding.

Then you can create your template on the messages tab, NOTE I change the transaction isolation level here, as I want READPAST hints.

That is all there is to it. If you having trouble making the template, just use Visual Studio to do it for you, then copy past the instance:

Just add a generated item to a existring BizTalk project and choose Consume WCF Service:

Since you NOT using orchestrationsm, you do not need to save the generated items 🙂 🙂

Once you got an instance, that is the template to copy paste into the Template section for the messages.

Hope this helps thos of you who want high performance writes to SQL Server.

 

This is the best option to use, a long while back I had a custom SQL adapter developed for this as posted in the blogs.

Cheers

Hi Folks,

I have decided to start a series of posts in regards to implementing BizTalk solutions. In these solutions I will assume that the following goals are desirable:

Services

• Loosely coupled architecture
• Abstraction by implementing web service or WCF solutions
  • These services must provide discrete components, that when orchestrated as a whole can provide logical business processes, this means that they should be re-useable.
• Just because you implement web services or WCF solutions, does not mean you implementing SOA.
• Entry points into the Service Bus must PROTECT data integrity. This means all WCF/WS* services must implement strict message contracts
  • How many of you have been given a web service to consume by a third part and when you download their WSDL, you notice that you can submit any data, just a generic XmlDocument? Not good.
  • Secondly by protecting the Service Bus from badly formed data, you will save yourself heaps of problems.
• A “Internal” XSD schema is used to orchestrate, route and update common artefacts. This schema is usually referred to as a canonical schema.
  • This means a C# version of the XSD is generated either with xsd.exe, svcutil.exe or xsd2class (my favourite). Therefore always ensure these classes that are used to expose your service contracts are PARTIAL. You can then create a seperate partial class for CUSTOM implementations such as Schema/Serialization and manipulation routines.
  • You may ask, why, we can have a BizTalk pipeline do this for us. Yes indeed you can. However many EXTERNAL facing services must be decoupled from the BizTalk system. This is where you can leverage strict service contracts to the outside world. Just because a WCF or Web Service accepts a message, does not mean it conforms to the XSD. Class generated code from XSD does not implement all the rules! e.g. Maximum Occurence and Minimum Occurrence.
  • There can be some disadvantages here. If you implement a canonical schema and not much thought went into the design, then you go to have some serious problems later on down the line. INVEST allot of TIME with business analysts and users, so that your canonical schema matches all business processes.
  • Any CHANGES to the canonical schema MUST be backward compatible.
  • Use namespaces to provide logical sections in the canonical schema to limit the IMPACT of new changes to the schema.
  • Let me put it this way. If the Canonical schema changes, your BizTalk Maps that map to this schema should still work.
• Use Document Literal if you need the messaging payload in your services, this message is not bound to any distinct operation.
• Use RPC style if you need a small subset of the data and need to clearly define what operation the message payload supports.
• if you unsure of the above, then go for Document Literal, most flexible.
• If using web services, implement SOAP security headers in external facing web services.
• Ensure core services have decent logging capabilities and perhaps even a web page to view the Inbound data, this can save users allot of hassles when chatting to clients about submitted documents that are not processed.
• You can use a generic XSD schema (ANY type) for internal processing, when the schema has already been validated. Why always validate the schema during multiple processes, if the entry point is valid? Sometimes this is necessary, however, when working deep within the service bus, you may find it easy to use a generic schema e.g. A Dispatching system, that already received validated documents and just needs to dispatch them to other services.
• Ensure you web.configs etc for services are tuned to support multiple connections (Default is 2). if they are not you will see many send port instances in BizTalk ready to send, because the max connections on the web service is defaulted to 2!
  http://grounding.co.za/blogs/romiko/archive/2009/04/11/biztalk-2006-optimising-soap-send-ports-and-sql-query-optimizer.aspx

   <system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "50"/>
    </connectionManagement>   
    </system.net>

• Ensure repeated calls to the same service will not changed the state of the message, unless you really intend to do so.
• If there is a chance that the same message will be delivered may times during the day (Customer updates), then perhaps a subscription service model can be used, where the system will PULL the latest record. This will solve problems where you cannot guarantee the latest message for the same customer will be processed last. Also, by doing so, you can avoid messaging queuing (which can be slow).
• There is nothing wrong incorporating solutions that PULL from web services, as the example above explains.
• Use Windows Communication Foundation to communicate between services within the organisation if you can, especially if the communication will be on the same server (netbinding etc)

Canonical Schema

• As I mentioned, INVEST allot of time defining, designing and implement this schema. get intermit with it.
• Appending Service Bus metadata to the canonical schema is extremely Powerful. You can use this meta data for Property Schemas and Distinguished fields. You can put this data in a separate namespace. This gives you allot of flexibility.
• When extracting data from SQL for BizTalk to consume, this is a good opportunity to see what SQL fields can be used in the canonical schema to provide metadata.
  • Example: You may have a SharePoint/FTP/WCF address associated with the data from SQL. Why not extract this on the SQL receive location and append it to the metadata of the message payload? You can then leverage Dynamic Send Ports 🙂
• In a nutshell, think through this, it can save you allot of time, and allow you to implement some call dynamic routing patterns, however the Schema must be really well designed and i did mentioned backward compatible.

Routing

• Never use a pattern where an Orchestration is used to invoke a Business Rule Engine policy and uses the result of the rule execution to define a message routing. I have seen this happen allot, NEVER do it. It slows down the messaging engine. Use filters on Send Ports, this is loosely coupled.
• Use Filters on Send Ports. PLAN your Property Schema.
• If your content filters for any BizTalk application is more that 10 types of property schema members, then you have a design issue. Keep your Property Schema Small, ideally fewer that 10 elements.
• Try to use dynamic send ports where it is possible to dynamically gather meta data about the destination end point.

Orchestration

• Try not to use BizTalk Maps in an orchestration. Why? You are going to tightly couple your solution here. Rather implement maps at the entry and exit points. Send and Receive Ports.
• Keep the persistent points on your orchestrations down to a minimum, use SCOPES to control persistent points.
• Do not use the expression editor for complex logical, and do not TRY hack the expression editor by using a multitude of static serializable classes to get the job done, this is a sign something is wrong, usually exposing the correct distinguished properties can solve this problem!
• I have seen to many implementations where Orchestration are used, just because well “They easy to implement”. Most of the times you do not need to use them, where the BizTalk messaging engine will suffice. Reasons why people use them so much, is that not allot of thought goes into the design of the canonical schema.
• DO NOT couple your orchestration directly to a WCF/Web Service port. It is just UGLY. Secondly you expose your internal orchestration logic to the outside world! You have no control how the contract is generated.
  • Try and get orchestrations to receive data from DIRECT ports bound to the message box.
• So direct ports is the way to go. Keep you service logic separate from orchestrations as mentioned. Use direct ports to the message box and just use filters on the message receive shaped. This gives you the best loosely coupled pattern.
• If you are call external classes, ensure they are THREAD SAFE!

Failures

• Use BizTalk’s routing of failed message subscription features to reroute failed messages. Do not underestimate how useful this can be e.g. Route failed messages to different document libraries in SharePoint for manual intervention, or to a custom human workflow database for web based fixing/editing.

BizTalk Mapper

• Keep your maps SIMPLE. Do not implement complex logic here, if you do, then something is wrong with the design.
• As I mentioned implement maps on the latest or earliest stage, not in the orchestration engine, unless you cannot help it.

In fact, I worked on a project where the Business required full control of mapping flat files to the canonical schema. They did not want to use the BizTalk Mapper. We built a custom ASP.NET mapper for them, and they can generate maps during runtime without the need for deployments of new maps. We leveraged Altova products, where they can use XSLT bases mapping and then load them into the cache (Compiled version of the xslt) where our service can call them at runtime. Altova has a really cool XSLT engine. We also extended this for extremely complex files so that a software factory and implement code, however these sort of files require an assembly update. However such files are only 5% of the file feeds, and dealing with over 300 file feeds, we can see that a web based mapper, that biztalk can consume via a service is an awesome way to loosely couple mappings from business processes.

I have seen some bad ass maps, where all I see is BLACK, not even the lines, with all sort of functoids, function calls and weird things going on, avoid complex mappings.

Remember large xml files BLOW up when processed by an XSLT engine. There are other ways to transform flat files if they are extremely large, do not always have to use XSLT’s. However, design a solution that is flexible in this regard if you dealing with flat files that are really complex, where xslts or other transformation solutions can be implemented and are loosely coupled.

SharePoint Integration

The SharePoint Adapter for BizTalk is really powerful and rock solid, leverage SharePoint if you require message collaboration! We use dynamic SharePoint send ports so we have FULL control over custom columns in the document library. This allows us to publish the message AND metadata to SharePoint and leverage SharePoint document library views. E.g. Sending a file to SharePoint and using a custom column in the document library to mentioned the Country where the file came from, or other properties.

Well, I think that’s enough tips I can think of for today! I hope this helps. Now, back on the bright side, here are some pics i took on a wildlife expeditions 🙂 Yes, life is not all about geeking it to the max!

This was in the Kruger National Park, I was in a concession area, no fences, just me and the wild animals 🙂

This elephant was not happy with me at all! I immediately turned into Casper the ghost!

 

This is me 🙂 Yes a Black Mamba in the tent.

This dude is hardcore bushmen I ever met!

I am not sure, but I think this is a cameroptra bashing a worm, poor thing…

 

Found these lions, eating a DONKEY!!!!

 

 

The Saddle billed stork above is really a rare site to see!

Handsome chap isn’t he?

I hope you enjoy the photos. I picked up a Canon 450d and the book for dummies, so hopefully i will improve!

Hi!

A quick note that I must say in regards to hosting BizTalk in our virtual environment that Sun Virtual Box has really out did them. They have a really stable virtual machine product. BizTalk, SQL and MOSS have been running really smooth on my development box which runs Sun Virtual Box.

Another nice thing is that they continually update the product.

This is my strategy for templates.

I will build a Windows 2003/2008 Virtual Box base template.

1. Install Visual Studio 2008
2. Install SQL 2005/2008
3. Install BizTalk (DO NOT CONFIGURE)
4. Adapter Packs e.g. WCF LOB, SQL
5. DebugView
6. Resharper (if You like it)
7. Notepad++
8. Orchestration Profiler
9. BizTalk Environment Configuration: http://biztalkconfigloader.codeplex.com/

Rudolf Henning on CodePlex updated my original BizTalk Environment Configuration Tooland made some cool enhancements.

Then you can just close the template above for new developers or for new projects, of course you need a new computer name and IP address. The only trick is for SQL. When you close it, you just run the following command:

sp_dropserver <old_name>

GO

sp_addserver <new_name>, local

GO

A really stable product!

Sun Virtual Box is much better than VMware, hands down. Another aspect I like about Virtual Box is that you can use the console to work on the machine, and the mouse pointers etc feel as if the machine was real, no VMware tools or mouse pointer acceleration to deal with, or even remote desktop as Virtual Box has it’s own built in addin which is a must to install.

HTH

Hi Folks,

Once you have gone live in production with your BizTalk system, you will certainly need to tune the system in order to get a high throughput.

There were two things causing the throughput of our system to be slow.

1. Limitation of connections by the windows server to SOAP/HTTP servers. Limit is 2
2. A bug in the SQL query optimizer when doing implicit data type conversions

We were processing around 10 000 messages per hour when the two settings above were incorrect. After making some adjustments we bumped up the processing to 100 000 messages per hour. Each messages does the following:

1. Imported into the database
2. Sent to a GEO Coder using SOAP
3. Sent to XSLT engine using SOAP
4. Sent to a SQL Algorithm system using SOAP
5. Send to a CRM system using SQL

In all the steps, a transaction log and custom database is used to keep track of the information and also for a custom human workflow application to edit records that are not well formed.

The following is what was needed to get the throughput higher.

SOAP

Edit the BTSNTSvc.exe.config

Add the following section:

    <system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "50"/>
    </connectionManagement>   
    </system.net>

What this does is ensures the server can have more than 2 connections (by default) simultaneously when calling a web service or http service. The setting above should be 12 x (Number of processors). We have a quad core processor, so 48-50 should be a good number for us.

You can read more here:  http://msdn.microsoft.com/en-us/library/aa545389.aspx

One way of knowing if this is a solution, is checking the event log in BizTalk. If you see allot of web service timeouts and you have set high timeout settings in your configuration files, proxy class and message context properties in orchestrations, then it must be the machine limiting the service calls or the web service needs some tuning. In our case, the server was limiting the number of calls.

What you will see in the Group Hub Page, is many send ports with this status:

READY TO RUN

Once we made the change in the BTSNTSVc.exe.config and restart the host instances, there SOAP send ports no longer had a ready to run status.

After the change, the timeouts in the event log disappeared and the query page looks much better, with ACTIVE status for all SOAP send port.

SQL

For the SQL side, we had many receive locations calling the same stored procedure, but using different values for the parameters. The parameters are the BATCH SIZE and STAGE.

CREATE  PROCEDURE [dbo].[GetWorkflowRecord]
@BatchSize int,
@Stage nvarchar(3) = null
AS
BEGIN

In the actual SQL table the Stage column has a data type of varchar(3). If you look above, the the data type is not the same, it is nvarchar(3), this was causing the SQL optimizer to not successfully calculate how the query should run and caused table scans. TO solve the problem we just changed it to varchar(3), and this would stop the stored procedure from doing an implicit conversion when querying the table, since the @stage is in a where clause:

wfr_wfs_code = (‘SUC’)
            AND        wfr_Batch is null
            AND        wfr_stg_code = @Stage

Here is a nice blog about it:

http://statisticsio.com/tabid/36/articleType/CategoryView/categoryId/10/query-optimizer.aspx

CREATE  PROCEDURE [dbo].[GetWorkflowRecord]
@BatchSize int,
@Stage varchar(3) = null
AS
BEGIN

Once this change was made, I was even able to change the 8 receive locations to call SQL every 1 second and increase the batch size from 150 to 500-1000.

Conclusion

Sometimes small trivial changes can make a huge different in performance, here, we were able to increase the throughout of the system from 240 000 records per day to 1.4 million records per day, with an average rate of 900-1000 messages per minute where each message was executing 10-12 transactions and calling 5-6 web services. Two factors limited the system, SOAP calls and a stored procedure causing table scans since implicit conversion of data in the procedure could not be optimised and detect the correct non-clustered index to use. Two simple change with major positive results! What more can I ask for before the Easter break.

 

Chronicles of South Africa

 

Here is a picture of two giraffes in a defensive pose. All the animals on this day were very frisky!

Nothing beats a good boerwors!

I love spiders!

Hi Folks,

I was away on holiday in South Africa and on my return, a colleague of mine mentioned that the BizTalk receive locations were unexpectedly shutting down. It would always be a random receive location.

We had just reached 50 receive locations and growing. In the Event Log BizTalk would complain:

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: (1)
Event ID: 5649
Date: Date
Time: Time
User: N/A
Computer: ComputerName
Description:
The receive location "FileReceive" with URL "\\ServerName\FileDrop\FileName.FileNameExtension" is shutting down. Details:"The FILE receive location \\ServerName\FileDrop\FileName.FileNameExtension exhausted the network retry attempts. ".

 

The problem here is due to the application level protocol management system within windows. This all has to do with the LanMan server and LanMan workstation service within windows. Since these services are used to manage UNC connections to folder shares and the tried and trusted Service Message Block application protocol is used here.

To resolve this problem, increase the MaxCmds and MaxMpxCt registry values on the BizTalk Server computer and on the remote UNC Share computer. To do this, follow the steps that are described in Microsoft Knowledge Base article 810886.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:

This issue may occur if the following conditions are true:

• This issue may occur if the client computer submits simultaneous, long-term requests against a file server that uses the Server Message Block (SMB) protocol. An example of a long-term request is when a client computer uses the FindFirstChangeNotification function to monitor a server share for changes.
• This issue may occur if the MaxCmds registry value setting on the client is less than 50, or the MaxMpxCtregistry value setting on the server is less than 50.
  Note The MaxMpxCt registry value setting may have a different name on other SMB/CIFS implementations. The Common Internet File System (CIFS) specification refers to it as MaxMpxCount.

In our case it was the former that caused it, since we had 50 receive locations. So we need to increase the value from 50 to let says 200.

To resolve this issue, verify that the MaxCmds and MaxMpxCt registry values are set to 50 or more. To do this, follow these steps:

1. Click Start, click Run, type regedit, and then click OK.
2. Locate and then click the following key in the registry:

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters

3. In the right pane, double-click the MaxCmds value.
4. In the Value data box, verify that the value is 50 or more.
   Note In a Microsoft Windows Terminal Services environment, you may have to set the MaxCmds registry value to 500 or more. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

   232476 Terminal Server client connections and logon limited by MaxWorkItem and MaxMpxCt values

5. Locate and then click the following key in the registry:

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters

   Note Make sure that you make this change to the lanmanserver\parameters registry key and not to thelanmanworkstation\parameters registry key mentioned in step 2.

6. In the right-pane, double-click the MaxMpxCt value.
   Note On SMB servers that are running a version of Windows earlier than Windows 2000, the MaxMpxCt registry entry is named MaxMpxCount, but has the same function.
7. In the Value data box, verify that the value is 50 or more.
   Note The MaxMpxCt value determines the maximum number of simultaneous, active requests that the server permits from an individual client.
8. Quit Registry Editor.

Note The MaxCmds and MaxMpxCt registry entries are REG_DWORD decimal entries. If they do not exist on your computer, you can create them as new REG_DWORD values. The range of values for these registry entries is between 0 and 65535.

It is important that you make this registry change on:

1. All BizTalk Servers
2. All File Share Servers being monitored

Happy BizTalking!

Here is a recent picture of South Africa where I am visiting a herd of elephants at the Thula Thula private game reserve:

Yes the elephants were not happy with us. On this day, we actually got mocked charged by a female elephant which is very rare, since the matriarch is usually responsible for this, maybe he was on sick leave! So with everything in life, and as Nature teaches us, we all have our limits, and with these elephants, it was overstaying our invitation.  The female in the background was within 4 metres from us by the time we put the peddle to the metal!

Hi Folks,

Sometimes you are running a rather large SQL stored procedure, and would like to know which part of the code is running and executing.

Below is a query you can use to see what code is executing at the current moment. I find this very useful, not only for detecting code blocks that are running, but also for identifying long running transaction and so forth.

select r.blocking_session_id,             r.session_id             ,status             ,substring(qt.text,r.statement_start_offset/2,                   (case when r.statement_end_offset = -1                   then len(convert(nvarchar(max), qt.text)) * 2                   else r.statement_end_offset end – r.statement_start_offset)/2)             as query_text   — this is the statement executing right now             ,qt.dbid             ,qt.objectid             ,r.cpu_time             ,r.total_elapsed_time             ,r.reads             ,r.writes             ,r.logical_reads             ,r.scheduler_id from sys.dm_exec_requests r cross apply sys.dm_exec_sql_text(sql_handle) as qt where r.session_id > 50 order by r.blocking_session_id desc, r.scheduler_id, r.status, r.session_id

Hope this helps with your queries and code re-factoring.

 

Hi Folks,

This blog will discuss the high level overview for developing a proxy on BizTalk to interface with composite web services. What we want to do is create a multi-part message on the fly when sending the data to the web service. BizTalk deals with data, but the workflow system deals with another message type, specify the stage and status of a message. This is where a SOAP proxy can be of use, by leveraging serializing etc.

The hardest part about making this work is supplying the correct message to the SOAP adapter at run time. If the specified proxy class method takes more than one parameter, BizTalk expects you to supply a multi-part message where each part corresponds to a parameter in the signature. BizTalk will deserialize each message part into the corresponding .NET Framework type before invoking the Web service proxy method, which then serializes everything back into a SOAP message.

Imagine we have a web service that manipulates data. It calls an XSLT engine and then does some formatting to the data. In our case, we use Altova XSLT engine, since users can develop maps and not need to be concerned with the BizTalk Mapper.

On the BizTalk side we would call the web service and parse in as parameters the XML from the BizTalk Message Store that subscribed to this particular send port.

You basically create a standard Send Port and then link the custom dll you have to represent your web proxy in the Web Service tab:

So how do you create a proxy dll. First I always use a dedicated proxy Visual Studio project. So in this case I call it MMIT.Workflow.Common.Proxies.<SubProxyName>

Sp the SubProxyName would be DataManipulation for a web service we need to call via BizTalk.

Proxies are cool, since you can do some formatting and extra logic to the data before sending it to the web service.

I assume in the blog, you all familiar with developing web services, if not, you can always read my other blogs about developing web services.

So here is the template code for calling a DataManipulation proxy service.

Here is how you make a proxy class, then just LINK the method name in BizTalk (see picture above) to the method in the proxy dll.

 

using System; using System.Net; using System.Web.Services.Protocols; using System.Xml; using MMIT.Common.Data.Internal; using MMIT.Workflow.Common.BOL; namespace MMIT.Workflow.Common.Proxies.DataManipulation {     public partial class MMITDataManipulation     {         [SoapDocumentMethodAttribute("http://MMIT.DataManipulation/BizTalkExecute", RequestNamespace = "http://MMIT.DataManipulation/", ResponseNamespace = "http://MMIT.DataManipulation/", Use = System.Web.Services.Description.SoapBindingUse.Literal, ParameterStyle = SoapParameterStyle.Wrapped)]         public WorkflowRecord BizTalkExecute(WorkflowRecord msg)         {             msg.WorkflowData.Stage = WorkflowStage.DataManipulation;             try             {                 XmlDocument doc = msg.MMITData.MMIT_Data_Record.Serialize();                 msg.MMITData.MMIT_Data_Record = MMIT_Data_Record.BuildDataRecord(Execute(doc));                 msg.WorkflowData.Status = WorkflowStatus.Succeeded;                 return msg;             }             catch (WebException)             {                 // To use the retry functionality of the send port                 // This exception type includes time outs and service unavailable from the web service.                 throw;             }             catch (Exception e)             {                 msg.WorkflowData.Status = WorkflowStatus.Failed;                 msg.WorkflowData.Error = e.Message;                 return msg;             }         }     } }

In the above code the Execute is a method call to from a web service reference, which accepts an Xml document.

That is all there is to it, whenever BizTalk calls the send port, it will execute the above code and send the data to the web service. In the above, we converting a XML message to a workflow message and then sending it to another system for processing. The main reason we have this feature is to create multi-part messages.

You can read more about this stuff here:

http://msdn.microsoft.com/en-us/magazine/cc163464.aspx

So I hope this blog gets you started.

Hi Folks,

Scenario

You have a web form where you need to do the following:

• Retrieve XML data from a SQL Data Store
• Provide an interface for Viewing and editing the data depending on the status of the data being viewed.
• The form view for the XML data should be easy to generate when the XSD changes
• The template used to publish the elements and attributes of the XML will be designed using InfoPath
• InfoPath will use Web Service data connections, which are published as Data Connection Libraries on a
  SharePoint 2007 Data Connection Library
• InfoPath will use a Edit and View mode in one template, so data can be controlled and secured for viewers and authors of data.
• InfoPath templates will be published on SharePoint 2007 InfoPath Forms library
• A custom ASPX page will leverage the XmlFormView control to embed the web based Info Path form which is hosted on SharePoint.

Design the Web Services

The first stage is to have the web services ready to Submit and Receive Data. All Info Path forms are designed with data connections to
Submit and Receive Data.

So the first step is to have the Data Layer built.

Design the InfoPath Form

This is divided into two stages

Stage 1

This is where you will create a Data Connection Library and publish it to the MOSS 2007 Server. The Data Connection library will have
all the properties to Submit and Receive data from the web service. When InfoPath forms are loaded it will retrieve the data connections|
from the SharePoint 2007 Data connection library. Below is a sample Library:

Notice above I have two libraries:

DCL for the data connections and WorkflowForms for the InfoPath Template.

Also notice I have no files saved in the WorkflowForms, the template is there, just no need to have a XML based file. The XmlFormView
can still use the template J

Notice the two types:

Main Query (Receive) and Submit.

If we click on of these published DCL’s we have this information:

MainQuery:

  <?xml version="1.0" encoding="UTF-8" ?>

  <?MicrosoftWindowsSharePointServices ContentTypeID="0x010100B4CBD48E029A4ad8B62CB0E41868F2B0"?>

– <udc:DataSource MajorVersion="2" MinorVersion="0" xmlns:udc="http://schemas.microsoft.com/office/infopath/2006/udc">

  <udc:Name>Main query</udc:Name>

  <udc:Description>Format: UDC V2; Connection Type: WebService; Purpose: ReadOnly; Generated by Microsoft Office InfoPath 2007 on 2009-05-08 at 13:25:16 by romiko.</udc:Description>

– <udc:Type MajorVersion="2" MinorVersion="0" Type="WebService">

  <udc:SubType MajorVersion="0" MinorVersion="0" Type="" />

  </udc:Type>

– <udc:ConnectionInfo Purpose="ReadOnly" AltDataSource="">

  <udc:WsdlUrl>http://Romiko.com/Workflow/WorkflowService.asmx?WSDL</udc:WsdlUrl>

– <udc:SelectCommand>

  <udc:ListId />

  <udc:WebUrl />

  <udc:ConnectionString />

  <udc:ServiceUrl UseFormsServiceProxy="false">http://Romiko.com/Workflow/WorkflowService.asmx</udc:ServiceUrl>

  <udc:SoapAction>http://Romiko.WebServices/GetWorkflowRecordByTransactionId</udc:SoapAction>

  <udc:Query />

  </udc:SelectCommand>

– <udc:UpdateCommand>

  <udc:ServiceUrl UseFormsServiceProxy="false" />

  <udc:SoapAction />

  <udc:Submit />

  <udc:FileName>Specify a filename or formula</udc:FileName>

  <udc:FolderName AllowOverwrite="" />

  </udc:UpdateCommand>

– <!–

udc:Authentication><udc:SSO AppId=” CredentialType=” /></udc:Authentication

  –>

  </udc:ConnectionInfo>

  </udc:DataSource>

 

Submit

  <?xml version="1.0" encoding="UTF-8" ?>

  <?MicrosoftWindowsSharePointServices ContentTypeID="0x010100B4CBD48E029A4ad8B62CB0E41868F2B0"?>

– <udc:DataSource MajorVersion="2" MinorVersion="0" xmlns:udc="http://schemas.microsoft.com/office/infopath/2006/udc">

  <udc:Name>Main submit</udc:Name>

  <udc:Description>Format: UDC V2; Connection Type: WebService; Purpose: WriteOnly; Generated by Microsoft Office InfoPath 2007 on 2009-05-08 at 13:24:00 by romiko.</udc:Description>

– <udc:Type MajorVersion="2" MinorVersion="0" Type="WebService">

  <udc:SubType MajorVersion="0" MinorVersion="0" Type="" />

  </udc:Type>

– <udc:ConnectionInfo Purpose="WriteOnly" AltDataSource="">

  <udc:WsdlUrl>http://Romiko.com/Workflow/WorkflowService.asmx?WSDL</udc:WsdlUrl>

– <udc:SelectCommand>

  <udc:ListId />

  <udc:WebUrl />

  <udc:ConnectionString />

  <udc:ServiceUrl UseFormsServiceProxy="false" />

  <udc:SoapAction />

  <udc:Query />

  </udc:SelectCommand>

– <udc:UpdateCommand>

  <udc:ServiceUrl UseFormsServiceProxy="false">http://Romiko.com/Workflow/WorkflowService.asmx</udc:ServiceUrl>

  <udc:SoapAction>http://Romiko.WebServices/SetWorkflowRecord</udc:SoapAction>

  <udc:Submit />

  <udc:FileName>Specify a filename or formula</udc:FileName>

  <udc:FolderName AllowOverwrite="" />

  </udc:UpdateCommand>

– <!–

udc:Authentication><udc:SSO AppId=” CredentialType=” /></udc:Authentication

  –>

  </udc:ConnectionInfo>

  </udc:DataSource>

 

This is all done in the InfoPath form designer and published via the designer as well.
These are basically udcx files published on SharePoint.

Stage 2

Now that you have defined the Data Connection Libraries, you can now design the look and feel.

A nice way to control access to the form, is to use views in the SAME template. Why?

SharePoint ALLOWS only ONE type of template per InfoPath Form Library. So if you need different views on the same data,
use ONE template with multiple views:

We can then access these views programmatically, based on business rules in the ASPX page we build J

In my case we have three views:

Default View

Read Only

Write

Programmatically Access Fields in View

You can also programtically access these fileds in your view, for example. When you retrieve the data, you
may need the primary key (TransactionID), so when you call the web service to update, you can then
send the XML data back in the ViewState and also have the TransactionID sent to the webservice, so it
knows what to update.

Another nice thing is you can make the textboxes readonly and programmatically insert the key in the box,
where the data is retrieved, so users cannot hack in and try edit records they do not have permission to!

Web Browser Compatible

Ensure that the form is Web Browser Compatible, in the Form Options set the options:

Notice to that the submit options are for a web service:

WSDL

InfoPath will automatically show all fields in the DataSource pane for the WSDL data, so it is easy to drag
the fields onto the form and then design!

Rules for switching views

You can switch views by linking buttons in the form to a custom rule, so need for custom code for the views.

If you are not familiar with InfoPath forms, there is more than enough material on the internet to get affiliated
with this.

Here are some links I liked:

http://msdn.microsoft.com/en-us/library/aa701078.aspx

 

ASPX Development

Before we get cracking you can read about the XmlFormView control here:

http://msdn.microsoft.com/en-us/library/microsoft.office.infopath.server.controls.xmlformview.aspx

Now, all you need to do is create a basic aspx page that will host the XmlFormView. Notice the xsnlocation is the
InfoPath forms document library on SharePoint!

 

Link Custom Page to XmlFormView Page

All you need to do now, is link your page where users can select data from a view (gridview, repeater etc) and then
the corresponding data is shown in a user friendly manner where they can edit/view the xml J

You can then add the XmlFormview you to your page in a panel or so:

 

You will need a way to reset the XmlFormView, this can be called from GridView_RowCommand event, when
you want to select data based on a key (transactionid in my case) from a row in a gridview.

 

Here is an overview of the form integrated into a aspx page.

 

So basically you can select a record, which triggers the RowCommand and then do the magic!

 

Here is a sample gridview row command that’s writes the transactionid to the form, so the user can then
click the button to get the data:

 

When users click the button to view the form, you can do a response.redirect:

 

You can also choose which view to write to, based on the status of the message, users can get a different
view:

WriteToInfoPathForm("false", "/dfs:myFields/my:FormReadOnly");

So we can now click the view button, and voila, the InfoPath on SharePoint gets the data:

SharePoint Server

The SharePoint server must have Info Path forms services enabled, and the feature must be activated.

Also you MUST enable the FormView is the SharePoint Central Administration pages of the MOSS server:

You need to enable Enterprise Features or have InfoPath Form Services Installed.

Also, you must ENABLE form view, this is needed to store the data from the form back to the server,
on post backs.

Summary

So basically your InfoPath form which is embedded in a Panel, will be seamlessly integrated into your
custom web application.

The infrastructure is:

• SharePoint Server with Info Path Forms Server
• IIS Server with custom web application
• XmlFormView.aspx page
• A Panel is a custom web form where you link the XmlFormView
• Some events to loads the Form and run rules
• Views are used to control access to the different forms options
• Very little logic in the code

So, this article should give you some ideas on how to integrate InfoPath forms with existing web applications.

So if XSD/WSDL changes, you can just upload/publish a new template to SharePoint and your users will get the
latest form with new fields etc J

Thanks to a colleague, Leon Droog who did allot of the prototyping and development of this solution. It
can sometimes be a real pain to get things going, but once they going it is allot easier than anticipiated!

Hi Folks,

Just finished the BizTalk 2006 Exam with a score of 985, I got one question wrong in the Enable Business Activity Monitoring section, I would like to take the opportunity to explain what you need to do to pass this exam from a practical perspective and get the best out of the experience!

1. Develop a Custom Send Pipeline
   1. Write a custom Disassemble Component
   2. Configure XSD Validation
2. Develop a Custom Receive Pipeline
   1. Write a custom Assemble Component

While we are on the subjects of pipelines, I think one of the biggest grey areas is debugging these buggers, I will provide an article at a later stage on how to effectively debug a Custom Pipeline Component.

 

1. Buy the book BizTalk 2006 Recipes and do all the exercises pertaining to BAM
2. Understand how to configure a Schema for Tag Identifiers, so that you can exclude Header and Footer Info or repeating fields
   1. Read about Tag Identifiers in the properties of Schema’s
3. Buy the book BizTalk 2006 Recipes and do all the exercises pertaining to BRE
4. Understand Assert and Retract built in functions in BRE and how custom fact retrievers work
5. Write your own Custom fact Retriever in C# and assert it into the Rule Engine at runtime
6. Understand how to priorities rules
7. Be familiar with the basics of the BizTalk Mapper (Lovely tool for small schemas, a death trap for large and complicated schema’s)
8. Orchestrations
   1. Learn about exception handling
   2. Learn about Persistence points and what shapes cause them!
   3. Understand Correlation inside out
9. Do not underestimate the usefulness of Dynamic Ports and Role Links!
10. The HAT tool was cool for school in BizTalk 2004, but in 2006 the HAT tool is there for those dev’s that were used to using it in 2004, so please focus on BizTalk Administration Tool and HUB Page.
11. Write at least one batch file to deploy an MSI package automatically with BTSTASK (BTSDEPLOY sucks, never use it!)
12. Know how to export and import MSI packages
13. Know how to export and import Rules
14. Have a good understanding of how you can use host instances to organize where receive and send locations run on
15. Remember the golden rule; always put common artifacts in one application. This makes deployment much easier
16. Read the BizTalk 2006 book by Wrox Press
17. Read the BizTalk 2006 book by Wrox Press again

I cannot delve to deep about the exam, but I think the above list is a good place to start, I never used any of the MOC material for the exam, the MOC material is great to get an idea, but to really understand the details, much better to get the BizTalk 2006 Recipes book and develop hands-on!

Good Luck

Hi Folks,

Sometimes in Orchestrations you would like to throw a custom exception. Before we begin, remember this rule:

• Never throw a general exception outside the catch block in an Orchestration

The rule above is a bit funny, because when you drag a default exception shape in a try block within an orchestration, it will default to General Exception and you cannot choose any other exception type.

Imagine we need to call a web service and then handle the response from it within an orchestration. However, sometimes I may get a timeout from the web service or I would only like to wait a 1 minute or so to get the response. What I usually do in these cases is create a listen shape after I have called the web service and then I use a delay shape to wait a specified amount of time for the web service response, if it does not respond within the time span , I simple throw a Timeout Exception.

 

However, as mentioned when you drag a default throw shape, the only option is to throw a general exception, to solve the problem, create a variable of type .NET Class

Then, navigate to the mscrolib library:

And choose the exception type as illustrated below.

Give the variable a friendly name in the Orchestration View

Add a throw exception under the Delay shape and assign the type of exception to the variable name you created (Weird I know, but this is how it is done)

Below are the properties of the Throw Exception Shape

As you can see above, I have a listen shape after sending a message to a web service and am now waiting for a response.

And you nearly done, what you then do is add a Catch Exception by right clicking the scope and clicking add exception handler:

• Remember all transaction should be in a scope, so your catch blocks will automatically be appended to it. Now it is time to customize the catch block.

Right click the new Catch Block and set the type of exception to catch, in my case, a timeout exception.

There are two properties to set here:

• Exception Object Type
• Exception Object Name

In my case this will be

Exception Object Type = System.TimeoutException (I had to choose .NET Class and navigate through the mscrolib again)

Exception Object Name = e ( I chose e, as this is what we use in coding most of the time, you can choose anything you like)

So the Exception Object Name is similar to doing this:

catch (Exception ex)

{

}

This is how looks is now.

 

Don’t forget to add custom code to handle the exception in the Catch Block, maybe to update a log table or send a message somewhere, like I have done above, so now when a web service times out, the orchestration will throw a timeout exception, this is much more intuitive than seeing a general exception!

SOAP exceptions from a web service

In order to catch soap exception from a web service, what you do is add a reference to the BizTalk project for:

System.Web.Services

 

Then add an exception handle block:

 

 

You can then access the information of the exception message like this:

e.Message

e.g.

 

Or you can create a Soap Exception message and send it to the message box for failure routing, here is an example from:

http://www.digitaldeposit.net/blog/2007/05/orchestration-handle-soap-exception-and.html, accessed 26 May, 2008

Source: Saravana Kumar, http://www.digitaldeposit.net/blog/2007/05/orchestration-handle-soap-exception-and.html

 

A word of warning, if a web service throws a soap exception, the port will retry by default 3 times at intervals of 5 minutes, I changed the retry here to 0 for my example, so that the soap exception is caught and a log in made in SQL, however if you using a delay shape make sure the amount in the delay is greater than the total interval in the send point, else you will never catch the soap exception and will just catch timeout exceptions!

So below, works fine with a delays of 2.5 minutes, however if I made retry count = 1, then the delay should be greater than 5 minutes! You can really get confused when a soap exception occurs and the orchestration does not catch it, it will only catch it after the retry count is done in the send port of the web service!

 

Here is the exception in SQL

To recap:

1. Create a VARIABLE in the orchestration view of the exception type you want to catch, use mscorlib to assign the variable type to an exception
2. Drag a throw shape and set the type of exception to throw to the variable name (yes it is automatically instantiated)
3. Add a custom Catch block to the SCOPE where the throw shape is located and once again, set the properties of Exception Object Name and Exception Object Type, where type will come from mscorlib and name is anything you like!

I would appreciate any comments you have or if you need any assistance with a BizTalk project. In later blogs I will be discussing Windows Communication Foundation and BizTalk R2.

Tip of the Day: Orchestrations should be used scarcely! The BizTalk books out their do not do any justice about this, since their main content is orchestrations and less on pure Content Based Routing using the core messaging engine i.e. Send Ports, Receive Ports, Adapters and Pipelines.

Secondly remember to always utilize serializable components if calling external components from within an orchestration, if you cannot encapsulate them by calling a web service, this ensures the calling code is dehydration friendly!

Cheers