Render XML as HTML

Hi Folks,

I had to make a report on a website, where users could see data being sent into the Enterprise Service Bus. So basically all requests sent to web services is stored as XML in the SQL database.

So, I created a GridView with a link to display the XML data if needed. However IE will display it like text unless you use a Response.Write mechanism and modify the content-type to text/xml etc or load IE with xml file direcrlty. I find this a bit dirty, since you need to finalise with a Response.End(), and this can cause the browser to not allow navigatiing back on your pages. It is much better to have a button on page to view data, then the user clicks it, a popup box shows with the nicely html view of the xml, just like IE does it when opening an XML file!!

Also I can imagine a user getting annoyed, where evertime he views the XML data he needs to click the back button, so popups are cool here. You can also of course just put it into another page if you want etc.

So the solution is the following:

1. AJAX – modalpopupextender

2. System.Web.UI.WebControls.Xml

3. XSLT to tranform the XML to HTML – http://grounding.co.za/files/folders/documents/entry2193.aspx

Ok, what you do is put in a event handler on the gridview for the tenmplate column:

<asp:TemplateField>
<ItemTemplate>
<asp:LinkButton ID="linkButtonShowData" Text="Show" Font-Underline="true" Font-Bold="true"
ForeColor="Pink" CommandArgument="<%# Bind(‘aib_RequestData’) %>" runat="server"
OnClick="linkButtonShowData_Click" />
</ItemTemplate>
</asp:TemplateField>

Then the event handler will parse the xml data to a helper method which will call the modalpopupextender.

protected void linkButtonShowData_Click(object sender, EventArgs e)
{
LinkButton showDataButton = (LinkButton)sender;
bool isXml = false;
XmlDocument doc = new XmlDocument();
try
{
doc.LoadXml(showDataButton.CommandArgument);
isXml = true;
}
catch
{

}

if (isXml)
showXml(doc);
else
showText(showDataButton.CommandArgument);
}

protected void showText(string text)
{
if (text == string.Empty)
{
text = "No request data found.";
lblRequestData.Text = @"<H5>" + text + @"</H5>";
lblRequestData.ForeColor = System.Drawing.Color.Red;
}
else
{
lblRequestData.Text = text;
lblRequestData.ForeColor = System.Drawing.Color.Black;
}
CreateModalPopupExtender.Show();
}

protected void showXml(XmlDocument xml)
{
lblRequestData.Text = string.Empty;
XmlRequestData.DocumentContent = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + Environment.NewLine;
XmlRequestData.DocumentContent += xml.OuterXml;
XmlRequestData.TransformSource=@"~\XSLT\XMLToHtml.xslt";
CreateModalPopupExtender.Show();
}

From the above, we have to methods, if the XML is not valid, then we just show the text data as some wbe requests are not XML based.

Ok, so as you can see we load the xml into the XmlRequestData.DocumentContent. This is just a simple System.Web.UI.WebControls.Xml.

<asp:Xml ID="XmlRequestData" runat="server" />

Which is located in the popup control of ajax, see below:

<asp:Button ID="HiddenPopupButton" Style="display: none" runat="Server" />
<ajaxtoolkit:modalpopupextender id="CreateModalPopupExtender" targetcontrolid="HiddenPopupButton"
popupcontrolid="CreateModelPanel" cancelcontrolid="Close" backgroundcssclass="ModalBackground"
dropshadow="true" runat="server" />
<asp:Panel ID="CreateModelPanel" Style="display: none;" CssClass="ModalPopup" ScrollBars="Both"
runat="server">
<div class="PopUpContent">
<div>
<asp:Label ID="lblRequestData" runat="server" Width="100px" />
<asp:Xml ID="XmlRequestData" runat="server" />
</div>
<div>
<asp:Button ID="Close" Text="Close" runat="server" /><br />
<br />
<br />
</div>
</div>
</asp:Panel>

Now here is the magic!

XmlRequestData.DocumentContent += xml.OuterXml;
XmlRequestData.TransformSource=@"~\XSLT\XMLToHtml.xslt";
CreateModalPopupExtender.Show();

This will transform the xml control data into HTML using the xslt.

Below is the xslt:

——————————————————————-

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="no" method="html"/>

<xsl:template match="/">
<HTML>
<HEAD>
<SCRIPT>
<xsl:comment>
<![CDATA[
function f(e){
if (e.className=="ci") {
if (e.children(0).innerText.indexOf("\n")>0) fix(e,"cb");
}
if (e.className=="di") {
if (e.children(0).innerText.indexOf("\n")>0) fix(e,"db");
} e.id="";
}
function fix(e,cl){
e.className=cl;
e.style.display="block";
j=e.parentElement.children(0);
j.className="c";
k=j.children(0);
k.style.visibility="visible";
k.href="#";
}
function ch(e) {
mark=e.children(0).children(0);
if (mark.innerText=="+") {
mark.innerText="-";
for (var i=1;i<e.children.length;i++) {
e.children(i).style.display="block";
}
}
else if (mark.innerText=="-") {
mark.innerText="+";
for (var i=1;i<e.children.length;i++) {
e.children(i).style.display="none";
}
}
}
function ch2(e) {
mark=e.children(0).children(0);
contents=e.children(1);
if (mark.innerText=="+") {
mark.innerText="-";
if (contents.className=="db"||contents.className=="cb") {
contents.style.display="block";
}
else {
contents.style.display="inline";
}
}
else if (mark.innerText=="-") {
mark.innerText="+";
contents.style.display="none";
}
}
function cl() {
e=window.event.srcElement;
if (e.className!="c") {
e=e.parentElement;
if (e.className!="c") {
return;
}
}
e=e.parentElement;
if (e.className=="e") {
ch(e);
}
if (e.className=="k") {
ch2(e);
}
}
function ex(){}
function h(){window.status=" ";}
document.onclick=cl;
]]>
</xsl:comment>
</SCRIPT>
<STYLE>
BODY {font:x-small ‘Verdana’; margin-right:1.5em}
.c {cursor:hand}
.b {color:red; font-family:’Courier New’; font-weight:bold;
text-decoration:none}
.e {margin-left:1em; text-indent:-1em; margin-right:1em}
.k {margin-left:1em; text-indent:-1em; margin-right:1em}
.t {color:#990000}
.xt {color:#990099}
.ns {color:red}
.dt {color:green}
.m {color:blue}
.tx {font-weight:bold}
.db {text-indent:0px; margin-left:1em; margin-top:0px;
margin-bottom:0px;padding-left:.3em;
border-left:1px solid #CCCCCC; font:small Courier}
.di {font:small Courier}
.d {color:blue}
.pi {color:blue}
.cb {text-indent:0px; margin-left:1em; margin-top:0px;
margin-bottom:0px;padding-left:.3em; font:small Courier;
color:#888888}
.ci {font:small Courier; color:#888888}
PRE {margin:0px; display:inline}
</STYLE>
</HEAD>
<BODY class="st">
<xsl:apply-templates/>
</BODY>
</HTML>
</xsl:template>

<xsl:template match="processing-instruction()">
<DIV class="e">
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">
<xsl:text>&lt;?</xsl:text>
</SPAN>
<SPAN class="pi">
<xsl:value-of select="name(.)"/>
<xsl:value-of select="."/>
</SPAN>
<SPAN class="m">
<xsl:text>?></xsl:text>
</SPAN>
</DIV>
</xsl:template>

<xsl:template match="processing-instruction(‘xml’)">
<DIV class="e">
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">
<xsl:text>&lt;?</xsl:text>
</SPAN>
<SPAN class="pi">
<xsl:text>xml </xsl:text>
<xsl:for-each select="@*">
<xsl:value-of select="name(.)"/>
<xsl:text>="</xsl:text>
<xsl:value-of select="."/>
<xsl:text>" </xsl:text>
</xsl:for-each>
</SPAN>
<SPAN class="m">
<xsl:text>?></xsl:text>
</SPAN>
</DIV>
</xsl:template>

<xsl:template match="@*">
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*/@*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
</SPAN>
<SPAN class="m">="</SPAN>
<B>
<xsl:value-of select="."/>
</B>
<SPAN class="m">"</SPAN>
</xsl:template>

<xsl:template match="text()">
<DIV class="e">
<SPAN class="b"> </SPAN>
<SPAN class="tx">
<xsl:value-of select="."/>
</SPAN>
</DIV>
</xsl:template>

<xsl:template match="comment()">
<DIV class="k">
<SPAN>
<A STYLE="visibility:hidden" class="b" onclick="return false"
onfocus="h()">-</A>
<SPAN class="m">
<xsl:text>&lt;!–</xsl:text>
</SPAN>
</SPAN>
<SPAN class="ci" id="clean">
<PRE>
<xsl:value-of select="."/>
</PRE>
</SPAN>
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">
<xsl:text>–></xsl:text>
</SPAN>
<SCRIPT>f(clean);</SCRIPT>
</DIV>
</xsl:template>

<xsl:template match="*">
<DIV class="e">
<DIV STYLE="margin-left:1em;text-indent:-2em">
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">&lt;</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
<xsl:if test="@*">
<xsl:text> </xsl:text>
</xsl:if>
</SPAN>
<xsl:apply-templates select="@*"/>
<SPAN class="m">
<xsl:text>/></xsl:text>
</SPAN>
</DIV>
</DIV>
</xsl:template>

<xsl:template match="*[node()]">
<DIV class="e">
<DIV class="c">
<A class="b" href="#" onclick="return false" onfocus="h()">-</A>
<SPAN class="m">&lt;</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
<xsl:if test="@*">
<xsl:text> </xsl:text>
</xsl:if>
</SPAN>
<xsl:apply-templates select="@*"/>
<SPAN class="m">
<xsl:text>></xsl:text>
</SPAN>
</DIV>
<DIV>
<xsl:apply-templates/>
<DIV>
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">
<xsl:text>&lt;/</xsl:text>
</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
</SPAN>
<SPAN class="m">
<xsl:text>></xsl:text>
</SPAN>
</DIV>
</DIV>
</DIV>
</xsl:template>

<xsl:template match="*[text() and not (comment() or processing-instruction())]">
<DIV class="e">
<DIV STYLE="margin-left:1em;text-indent:-2em">
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">
<xsl:text>&lt;</xsl:text>
</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
<xsl:if test="@*">
<xsl:text> </xsl:text>
</xsl:if>
</SPAN>
<xsl:apply-templates select="@*"/>
<SPAN class="m">
<xsl:text>></xsl:text>
</SPAN>
<SPAN class="tx">
<xsl:value-of select="."/>
</SPAN>
<SPAN class="m">&lt;/</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
</SPAN>
<SPAN class="m">
<xsl:text>></xsl:text>
</SPAN>
</DIV>
</DIV>
</xsl:template>

<xsl:template match="*[*]" priority="20">
<DIV class="e">
<DIV STYLE="margin-left:1em;text-indent:-2em" class="c">
<A class="b" href="#" onclick="return false" onfocus="h()">-</A>
<SPAN class="m">&lt;</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
<xsl:if test="@*">
<xsl:text> </xsl:text>
</xsl:if>
</SPAN>
<xsl:apply-templates select="@*"/>
<SPAN class="m">
<xsl:text>></xsl:text>
</SPAN>
</DIV>
<DIV>
<xsl:apply-templates/>
<DIV>
<SPAN class="b">
<xsl:call-template name="entity-ref">
<xsl:with-param name="name">nbsp</xsl:with-param>
</xsl:call-template>
</SPAN>
<SPAN class="m">
<xsl:text>&lt;/</xsl:text>
</SPAN>
<SPAN>
<xsl:attribute name="class">
<xsl:if test="xsl:*">
<xsl:text>x</xsl:text>
</xsl:if>
<xsl:text>t</xsl:text>
</xsl:attribute>
<xsl:value-of select="name(.)"/>
</SPAN>
<SPAN class="m">
<xsl:text>></xsl:text>
</SPAN>
</DIV>
</DIV>
</DIV>
</xsl:template>

<xsl:template name="entity-ref">
<xsl:param name="name"/>
<xsl:text disable-output-escaping="yes">&amp;</xsl:text>
<xsl:value-of select="$name"/>
<xsl:text>;</xsl:text>
</xsl:template>

</xsl:stylesheet>

——————————————————————-

That is all there is to it, now you XML will be displayed as HTML, as rendered when opening a file in Internet Explorer with a nice format to it!

You can download the XSLT here:

http://grounding.co.za/files/folders/documents/entry2193.aspx

XSD Code Generation Xsd2Code vs XSD.exe

Hi Folks,

A quick post for those of you who need to generate code based on complex and rather large XSD schemas.

I found this tool very handy:

http://www.codeplex.com/Xsd2Code

It has allot more customisation than the xsd.exe tool from Visual Studio. Xsd2Code can be customised to use Lists or Arrays and supports large schemas with allot of Import statements, which you do not need to include int he command line, very handy! Wish microsoft would jack up their xsd.exe gen tool.

Here is a sample command line for a schema that references around 40 xsd files.

C:\Code\MMIT.GMDS\MMIT.GMDS.WebServices\Schemas\Star\Rev5.3.\BODs\Standalone>"C
:\Program Files\Xsd2Code\Xsd2Code.exe" AcknowledgeSalesLead.xsd Romiko.GMDS.BOL.S
chemas.Star /c Array

NOTE: Above I do not use /cb Array as it states in the help file, I use /c Array, this is a bug in the help file or in the code, I will report it to CodePlex.

I am currently using this to manage code generation for this automative schema:

http://www.starstandard.org/SIGXMLSTAR5/XMLSchemas

As you can see for the acknowledge sales lead, there is allot of imports, which this tool handles nicely.

<?xml version="1.0" encoding="utf-8"?>
<xsd:schema xmlns:udt="http://www.openapplications.org/oagis/9/unqualifieddatatypes/1.1" xmlns:oacl="http://www.openapplications.org/oagis/9/codelists" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sqdt="http://www.starstandard.org/STAR/5/qualifieddatatypes/1.0" xmlns="http://www.starstandard.org/STAR/5" xmlns:ccts="urn:un:unece:uncefact:documentation:1.1" xmlns:qdt="http://www.openapplications.org/oagis/9/qualifieddatatypes/1.1" xmlns:scl="http://www.starstandard.org/STAR/5/codelists" xmlns:oagis="http://www.openapplications.org/oagis/9" xmlns:star="http://www.starstandard.org/STAR/5" targetNamespace="http://www.starstandard.org/STAR/5" elementFormDefault="qualified" attributeFormDefault="unqualified" blockDefault="#all">
<xsd:import namespace="http://www.starstandard.org/STAR/5/qualifieddatatypes/1.0" schemaLocation="../../Resources/Components/Common/QualifiedDataTypes.xsd" />
<xsd:import namespace="http://www.starstandard.org/STAR/5/codelists" schemaLocation="../../Resources/Components/Common/CodeLists.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/qualifieddatatypes/1.1" schemaLocation="../../Resources/Components/OAGIS/CoreComponents/QualifiedDataTypes.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/unqualifieddatatypes/1.1" schemaLocation="../../Resources/Components/OAGIS/CoreComponents/UnqualifiedDataTypes.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/codelists" schemaLocation="../../Resources/Components/OAGIS/Common/CodeLists.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/currencycode/54217:2001" schemaLocation="../../Resources/Components/OAGIS/CoreComponents/CodeList_CurrencyCode_ISO_7_04.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/languagecode/5639:1988" schemaLocation="../../Resources/Components/OAGIS/CoreComponents/CodeList_LanguageCode_ISO_7_04.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/IANAMIMEMediaTypes:2003" schemaLocation="../../Resources/Components/OAGIS/CoreComponents/CodeList_MIMEMediaTypeCode_IANA_7_04.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9/unitcode/66411:2001" schemaLocation="../../Resources/Components/OAGIS/CoreComponents/CodeList_UnitCode_UNECE_7_04.xsd" />
<xsd:import namespace="http://www.openapplications.org/oagis/9" schemaLocation="../../Resources/Components/OAGIS/Common/Components.xsd" />

Beast XSD.exe from Microsoft hands down from a useability perspective.

SQL 2005 – Rebuilding Indexes

Hi,

Below is a script to rebuild indexes, it will try online operations, which can fail, in the event of a failure it will switch to offline mode.

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

SET NOCOUNT ON

GO

CREATE procedure [dbo].[RebuildIndexes]

(

@DatabaseName VARCHAR(255)

)

as

DECLARE @Database VARCHAR(255)

DECLARE @Table VARCHAR(255)

DECLARE @cmd NVARCHAR(500)

DECLARE @fillfactor INT

SET @fillfactor = 75

DECLARE DatabaseCursor CURSOR FOR

SELECT name FROM master.dbo.sysdatabases

WHERE name IN (@DatabaseName)

ORDER BY 1

OPEN DatabaseCursor

FETCH NEXT FROM DatabaseCursor INTO @Database

WHILE @@FETCH_STATUS = 0

BEGIN

SET @cmd = ‘DECLARE TableCursor CURSOR FOR SELECT table_catalog + ”.” + table_schema + ”.” + table_name as tableName

FROM ‘ + @Database + ‘.INFORMATION_SCHEMA.TABLES WHERE table_type = ”BASE TABLE”’

— create table cursor

EXEC (@cmd)

OPEN TableCursor

FETCH NEXT FROM TableCursor INTO @Table

WHILE @@FETCH_STATUS = 0

BEGIN

— SQL 2000 command

–DBCC DBREINDEX(@Table,’ ‘,@fillfactor)

— SQL 2005 command

print ‘Rebuilding ‘ + @Table

Select @Table = Substring(@table, 0, LEN(@table) PATINDEX(‘%.%’, REVERSE(@table)) + 2) + ‘[‘ +

SUBSTRING(@table, LEN(@table) PATINDEX(‘%.%’, REVERSE(@table)) + 2, LEN(@table)) + ‘]’

SET @cmd = ‘ALTER INDEX ALL ON ‘ + @Table + ‘ REBUILD WITH (FILLFACTOR = ‘ + CONVERT(VARCHAR(3),@fillfactor) + ‘, ONLINE = ON)’

BEGIN TRY

EXEC (@cmd)

END TRY

BEGIN CATCH

print ‘Table: ‘ + @table + ‘ has been skipped as ONLINE, switching to OFFLINE’

SELECT

ERROR_NUMBER() AS ErrorNumber

,ERROR_SEVERITY() AS ErrorSeverity

,ERROR_STATE() AS ErrorState

,ERROR_PROCEDURE() AS ErrorProcedure

,ERROR_LINE() AS ErrorLine

,ERROR_MESSAGE() AS ErrorMessage;

SET @cmd = ‘ALTER INDEX ALL ON ‘ + @Table + ‘ REBUILD WITH (FILLFACTOR = ‘ + CONVERT(VARCHAR(3),@fillfactor) + ‘, ONLINE = OFF)’

EXEC (@cmd)

END CATCH

FETCH NEXT FROM TableCursor INTO @Table

END

CLOSE TableCursor

DEALLOCATE TableCursor

FETCH NEXT FROM DatabaseCursor INTO @Database

END

CLOSE DatabaseCursor

DEALLOCATE DatabaseCursor

Upgrading BizTalk 2006 to BizTalk 2009

Hi,

When you want to upgraded to BizTalk 2009 from BizTalk 2006 R2, it is a relatively easy process to do on Windows 2003.

MSMQT will no longer work, so if you implement message queuing then I advise to build another solution than use MSMQT.

Here is the upgrade warning:

Upgrade Required

Setup has detected that BizTalk Server 2006 R2 is installed on this computer. Setup will upgrade BizTalk Server 2006 R2 to BizTalk Server 2009.

Prior to the upgrade, you must stop all BizTalk Services in the BizTalk Group and backup the BizTalk databases.

Warning: The following features are no longer available in BizTalk Server 2009.

BizTalk Deployment Command Line Tool
Human Workflow Services Runtime Components
Human Workflow Services Base Assemblies
Human Workflow Service Administration Tools
BizTalk Message Queuing

If you proceed with the upgrade current applications may not function correctly. If you do not wish to proceed with the upgrade, press Cancel.

Database Upgrade

Setup will upgrade the following databases. Please backup these databases before you click upgrade on the final summary screen.

  • -Database: BizTalkMgmtDb on SQL Server: PP\WorkRomiko.
  • -Database: BizTalkMsgBoxDb on SQL Server: PP\EUWorkRomiko.
  • -Database: BizTalkDTADb on SQL Server: PP\EUWorkRomiko.
  • -Database: BizTalkRuleEngineDb on SQL Server: PP\EUWorkRomiko.

Also you will need to install the redistribution cab files:

BTSRedistW2K3EN32.cab (32 bit servers)

or

BTSRedistW2K3EN64.cab (64 bit servers)

The best is:

1. Run the setup for the first time and choose the option: Download the redistributable prerequisites CAB file.

2. Rerun setup and choose the last option. Automatically install the redistributable prerequisites from a CAB file.

Doing it this way, gives you the opportunity to have the file and then you can reuse it for development, preproduction and production installs.

Also stpo the rule engine and www services before the upgrade and any Biztalk host instances.

Once the upgrade is complete I do notice that the SQL BizTalk jobs owner is changed to the Windows Account that did the upgrade, so if you like them to run under SA or not use windows accounts, then you must go to the SQL jobs and change the owner of the jobs, else you might find your SQL jobs failing!

I was pretty happy with the upgrade and everything went smooth.

However I have one gripe, and I hate moaning, but this is really not cool. The BizTalk administrator console is also upgraded, however it can no longer administer BizTalk 2006 servers, and it can only administer 2009 servers. This can pose a problem on an administration machine when you in a upgrade project cycle, if you want to administer 2006 servers, then you need to logon locally or use another admin console.

I also notice that the BizTalk install folder in program files will still be BizTalk 2006, which is nice, if you use the BTSNTSvc.exe.config file, so those settings are still kept e.g. Enterprise Library profiles etc.

I heard of an issue in Visual Studio 2008 Maps and Schemas can cause a problem, I have yet to see this, so if I do, I will write something up about it! I am just happy to get ion Visual Studio 2008 and get rid of Visual Source Safe and work on Team Server!

I must say assemblies compiled on Visual Studio 2005 still work on 2009 🙂 However if you upgrade your development machine, the BizTalk projects of 2006 will not be recognised, you will need to upgrade to Visual Studio 2008 and upgrade the BizTalk 2006 solution files etc.

Another thing to note, is that when you upgrade a BizTalk project with an orchestration using a web reference, then in visual studio 2008, the upgraded web reference will be updated, however the orchestration odx.cs code behind is not updated. The solution is to DELETE the web references and recreate them from scratch and then reconfigure the web port types 🙂

Another issue is Visual Studio 2008 COPY LOCAL setting for reference dll’s is very buggy, since it is not actually boolean, so migrated settings will not work even if copy local is on, careful of this, else you will get manifest problems thinking it copied the recompiled referenced assemblies. So solution to fix this, is set COPY LOCAL to false in the interface and then set it back to true, thanks to this dudes blog:

http://underground.infovark.com/2009/09/09/visual-studio-2008-and-the-copylocal-setting/

Hope this helps for anyone considering an upgrade.

SQL Server 2005 Enterprise Reporting Solutions leveraging Service Broker Synchronization

Hi!

This blog will outline some high level aspects of implementing service broker, this is targeted at people who are already familiar with the basics of service broker, if you need the basics, check this tutorial:

http://msdn.microsoft.com/en-us/library/bb839483.aspx

Scenario:

You have a production system that is processing millions of records a month.
Reporting requires frequent complex reports to run on the metrics and also on the actual data stored.
The actual data payload is stored as XML data type. This means any reporting metrics on data payload must be done on a fact table of some sort.
Data must be archived for 5 years. (This will be around 500 million records)

Objectives:

Decouple the reporting from the production system
Ensure up to date synchronization from production to the reporting server
Running reports will not affect the production system

Solution:

  • We will use service broker to define ONE message contract for all tables that need to be synced. Synchronization will be activated by a job which activates the SQL service broker process
  • A message type per table is created
  • Synchronization will work in batch mode, supporting running of 40 to 50 simultaneous SQL processes (Optimal for a quad core process = 48)
  • After synchronization of the main transaction tables a job on the reporting server will update the FACTS table.
  • The initial queue on the sender side will have a PROC activation configured on the queue. When the SQL job runs it will activate this job.
  • We will have a service to Trigger the archiving process
  • We will create a Route to the destination system
  • Up to you how to use the Service Broker security JIn this sample I keep it simple, no encryption. You can use Asymmetric Encryption/security etc, not in scope of this article! In this example I USE A SQL USER MYUser and User, notice the service and route user names are swapped on either server J

The fact table is used to extract information from the XML data. This allows the data to be indexed and reports to run smooth.

Do you have a transaction database and need a custom reporting solution, without the complexity of log shipping or replication? Then look no further, SQL Service broker is the ideal tool to synchronize two databases, based on simple tables.

Source SQL Server

On this server you create a Service Broker Database to activate sending the data contracts payload. So for example you can create a database called SSBSender, this will manage the starting and spawning of processes, queues etc

Contract:

USE

[SSBSender]

GO

CREATE CONTRACT [ArchivingTables] AUTHORIZATION [dbo] ([CONF_AuditTrail] SENT BY ANY,

[Table_Error] SENT BY ANY,

[Table_Record] SENT BY ANY,

[Table_RecordData] SENT BY ANY,

[Table_Transaction] SENT BY ANY)

MessageTypes:

USE [SSBSender]

GO

CREATE MESSAGE TYPE [Table_Error] AUTHORIZATION [dbo] VALIDATION = WELL_FORMED_XML

CREATE MESSAGE TYPE [Table_Record] AUTHORIZATION [dbo] VALIDATION = WELL_FORMED_XML

CREATE MESSAGE TYPE [Table_RecordData] AUTHORIZATION [dbo] VALIDATION = WELL_FORMED_XML

CREATE MESSAGE TYPE [Table_Transaction] AUTHORIZATION [dbo] VALIDATION = WELL_FORMED_XML

Sender Queue:

CREATE QUEUE [dbo].[InitialQueue] WITH STATUS = ON , RETENTION = OFF , ACTIVATION ( STATUS = ON , PROCEDURE_NAME = [dbo].[SSB_ReportTransmission] , MAX_QUEUE_READERS = 20 , EXECUTE AS OWNER ) ON [PRIMARY]

Service:

CREATE SERVICE [TriggerArchiving] AUTHORIZATION [MyUser] ON QUEUE [dbo].[InitialQueue] ([ArchivingTables])

Route

CREATE ROUTE [SSBReceiver] AUTHORIZATION [dbo] WITH SERVICE_NAME = N’ReportingReceivingService’ , ADDRESS = N’TCP://DestinationServer:4022′

Remote Service Binding

CREATE REMOTE SERVICE BINDING [ReceivingServiceBinding] AUTHORIZATION [dbo] TO SERVICE N’ReportingReceivingService’ WITH USER = [MyUserName] , ANONYMOUS = OFF

Activation Job

Schedule a job to activate the process.

declare @datetime datetime

select @datetime = getdate()

EXEC [SSBSender].[dbo].[Start_Report_Transmission] @batch_size = 2500, @archive_date_limit = @datetime

Sender Procedures

USE [SSBSender]

GO

CREATE PROCEDURE [dbo].[Start_Report_Transmission] @batch_size INT = NULL, @archive_date_limit DATETIME = NULL

AS

DECLARE @xmldata XML

DECLARE @InitDlgHandle UNIQUEIDENTIFIER

SET @batch_size = ISNULL(@batch_size, 100)

SET @archive_date_limit = ISNULL(@archive_date_limit, GETDATE())

–@xmldata contains the batch size for every table

SET @xmldata = N'<batch>’ + CAST(@batch_size AS VARCHAR(30)) + N'</batch><archive_date_limit>’ + CONVERT(NVARCHAR(20), @archive_date_limit, 120)+ N'</archive_date_limit>’

–create a dialog from TriggerArchiving to itself

;BEGIN DIALOG @InitDlgHandle

FROM SERVICE TriggerArchiving

TO SERVICE N’TriggerArchiving’

ON CONTRACT ArchivingTables

WITH ENCRYPTION = OFF;

–send as many messages as the tables that need to be sent to reporting

;SEND ON CONVERSATION @InitDlgHandle MESSAGE TYPE [Table_Error](@xmldata);

;SEND ON CONVERSATION @InitDlgHandle MESSAGE TYPE [Table_Transaction](@xmldata);

;SEND ON CONVERSATION @InitDlgHandle MESSAGE TYPE [Table_Record](@xmldata);

;SEND ON CONVERSATION @InitDlgHandle MESSAGE TYPE [Table_RecordData](@xmldata);

END CONVERSATION @InitDlgHandle;

The procesure above will TRIGEER the Queue activation proc below!!!

USE [SSBSender]

GO

/****** Object: StoredProcedure [dbo].[SSB_ReportTransmission] Script Date: 03/10/2010 11:08:23 ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATE PROCEDURE [dbo].[SSB_ReportTransmission]

AS

DECLARE @batch INT;

DECLARE @archive_date_limit DATETIME;

DECLARE @RecvReqDlgHandle UNIQUEIDENTIFIER;

DECLARE @RecvReqMsg XML;

DECLARE @RecvReqMsgName sysname;

DECLARE @ReportTransmissionHandle UNIQUEIDENTIFIER;

DECLARE @list XML;

DECLARE @far_service NVARCHAR(256)

WHILE 1=1

BEGIN

SELECT @RecvReqDlgHandle = NULL, @RecvReqMsg = NULL, @RecvReqMsgName = NULL, @far_service = NULL

WAITFOR

(

RECEIVE TOP(1)

@RecvReqDlgHandle = conversation_handle,

@RecvReqMsg = message_body,

@RecvReqMsgName = message_type_name

FROM InitialQueue

), TIMEOUT 10000

IF @RecvReqMsgName = N’http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog&#8217;

OR @RecvReqMsgName = N’http://schemas.microsoft.com/SQL/ServiceBroker/Error&#8217;

BEGIN

IF @RecvReqDlgHandle IS NOT NULL

BEGIN

END CONVERSATION @RecvReqDlgHandle;

END

RETURN 0

END

–at this point there is a message.

–check where the message is coming from (the far service)

SELECT @far_service = far_service

FROM sys.conversation_endpoints

WHERE conversation_handle = @RecvReqDlgHandle

–based on the source of the message perform different actions

IF @far_service = N’ReportingReceivingService’ –that means this is an answer from the reporting server

BEGIN

–set the status of records

EXEC dbo.SetReportingFlag @tabletype = @RecvReqMsgName, @list = @RecvReqMsg

CONTINUE –to receiving loop

END

–at this point the message came from a local call (triggering repeatable transmission)

–get the batch size

SET @batch = @RecvReqMsg.value(‘(batch/text())[1]’, ‘INT’)

SET @archive_date_limit = @RecvReqMsg.value(‘(archive_date_limit/text())[1]’, ‘DATETIME’)

IF ISNULL(@batch, 0) = 0 OR @archive_date_limit IS NULL

BEGIN

RETURN 0

END

;BEGIN DIALOG @ReportTransmissionHandle

FROM SERVICE TriggerArchiving

TO SERVICE N’ReportingReceivingService’

ON CONTRACT ArchivingTables

WITH ENCRYPTION = OFF;

WHILE 1=1

BEGIN

SET @list = NULL

–call procedure that updates the appropriate table

–and returns a list of ids that need to be transferred

EXEC dbo.GetArchiveIdsforReporting @tabletype = @RecvReqMsgName, @batch = @batch, @archive_date_limit = @archive_date_limit, @list = @list OUTPUT

–break the loop when list is null (no records to be transferred)

IF @list IS NULL

BEGIN

BREAK

END

–transmit the list, with the initially received message type

;SEND ON CONVERSATION @ReportTransmissionHandle MESSAGE TYPE @RecvReqMsgName(@list);

END

–end the conversation when no more records are to be transmitted

END CONVERSATION @ReportTransmissionHandle;

–break receiving loop

BREAK

END –of receiving looping

RETURN 0

USE [SSBSender]

GO

/****** Object: StoredProcedure [dbo].[GetArchiveIdsforReporting] Script Date: 03/10/2010 11:09:22 ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATE PROCEDURE [dbo].[GetArchiveIdsforReporting] @tabletype SYSNAME = NULL, @batch INT = NULL, @archive_date_limit DATETIME = NULL, @list XML = NULL OUTPUT

AS

–first NULLify the output variable

–this is important for always returning the right value (when output param is NULL)

–to the caller

SET @list = NULL

IF ISNULL(@tabletype, ”) = ” OR ISNULL(@batch, 0) = 0 OR @archive_date_limit IS NULL

BEGIN

RETURN

END

DECLARE @tmp TABLE (archive_id BIGINT, tmpid INT IDENTITY(1,1) PRIMARY KEY CLUSTERED (archive_id, tmpid))

–trigger update of table and sending of records

–based on the message type

IF @tabletype = ‘Table_Error’

BEGIN

UPDATE TOP (@batch) MyTransactionDatabase.dbo.tfe_TableError

SET sent_to_report = 1

OUTPUT inserted.tfe_id INTO @tmp(archive_id)

WHERE sent_to_report = 0

AND archive_date <= @archive_date_limit

END

ELSE IF @tabletype = ‘Table_Transaction’

BEGIN

UPDATE TOP (@batch) MyTransactionDatabase.dbo.tft_TableTransaction

SET sent_to_report = 1

OUTPUT inserted.tft_id INTO @tmp(archive_id)

WHERE sent_to_report = 0

AND archive_date <= @archive_date_limit

END

ELSE IF @tabletype = ‘Table_Record’

BEGIN

UPDATE TOP (@batch) MyTransactionDatabase.dbo.tfr_TableRecord

SET sent_to_report = 1

OUTPUT inserted.tfr_id INTO @tmp(archive_id)

WHERE sent_to_report = 0

AND archive_date <= @archive_date_limit

END

ELSE IF @tabletype = ‘Table_RecordData’

BEGIN

UPDATE TOP (@batch) MyTransactionDatabase.dbo.trd_TableRecordData

SET sent_to_report = 1

OUTPUT inserted.trd_id INTO @tmp(archive_id)

WHERE sent_to_report = 0

AND archive_date <= @archive_date_limit

END

IF NOT EXISTS(SELECT * FROM @tmp)

BEGIN

RETURN

END

SET @list = NULL

–create the list of tbl_id’s

SET @list = (SELECT TOP 100 PERCENT archive_id AS id

FROM @tmp

ORDER BY archive_id

FOR XML PATH(”)

)

USE [SSBSender]

GO

/****** Object: StoredProcedure [dbo].[SetReportingFlag] Script Date: 03/10/2010 11:11:55 ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATE PROCEDURE [dbo].[SetReportingFlag] @tabletype sysname = NULL, @list XML = NULL

AS

DECLARE @TransferStatus TINYINT

IF @tabletype IS NULL OR @list IS NULL

BEGIN

RETURN 0

END

–get the transmission flag value

SET @TransferStatus = @list.value(‘(TransferStatus/text())[1]’, ‘TINYINT’)

IF @TransferStatus IS NULL

BEGIN

RETURN 0

END

–shred the @list and get id values

DECLARE @tmp TABLE (archive_id BIGINT, tmpid INT IDENTITY(1,1) PRIMARY KEY CLUSTERED (archive_id, tmpid))

–shred the @list of values

INSERT INTO @tmp(archive_id)

SELECT a.col.value(‘text()[1]’, ‘BIGINT’)

FROM @list.nodes(‘id’) a(col)

IF NOT EXISTS(SELECT TOP 1 * FROM @tmp)

BEGIN

RETURN 0

END

–update proper table with @TransferStatus

IF @tabletype = ‘Table_Error’

BEGIN

UPDATE archive_tbl

SET sent_to_report = @TransferStatus

FROM MyTransactionDatabase.dbo.tfe_TableError archive_tbl

JOIN @tmp tmp ON archive_tbl.tfe_id = tmp.archive_id

END

ELSE IF @tabletype = ‘Table_Transaction’

BEGIN

UPDATE archive_tbl

SET sent_to_report = @TransferStatus

FROM MyTransactionDatabase.dbo.tft_TaleTransaction archive_tbl

JOIN @tmp tmp ON archive_tbl.tft_id = tmp.archive_id

END

ELSE IF @tabletype = ‘Table_Record’

BEGIN

UPDATE archive_tbl

SET sent_to_report = @TransferStatus

FROM MyTransactionDatabase.dbo.tfr_tableRecord archive_tbl

JOIN @tmp tmp ON archive_tbl.tfr_id = tmp.archive_id

END

ELSE IF @tabletype = ‘Table_RecordData’

blah blah blah

RETURN 0

Destination SQL Server

This is the server that will receive the data and process the messages in the queue. The database here to manage synchronization will be SSBReceiver, this will manage the receiving queuing and parsing the message payload to the receiver procedure.

On the receiving database, you will have the SAME contract and message types.

Queue

CREATE QUEUE [dbo].[ReceivingQueue] WITH STATUS = ON , RETENTION = OFF , ACTIVATION ( STATUS = ON , PROCEDURE_NAME = [dbo].[SSBReceiveMessage] , MAX_QUEUE_READERS = 40 , EXECUTE AS OWNER ) ON [PRIMARY]

Service

CREATE SERVICE [ReportingReceivingService] AUTHORIZATION [User] ON QUEUE [dbo].[ReceivingQueue] ([ArchivingTables])

Route

CREATE ROUTE [TriggerArchiving] AUTHORIZATION [dbo] WITH SERVICE_NAME = N’TriggerArchiving’ , ADDRESS = N’TCP://SourceServer:4022′

Remote Service Binding

CREATE REMOTE SERVICE BINDING [TriggerArchivingServiceBinding] AUTHORIZATION [dbo] TO SERVICE N’TriggerArchiving’ WITH USER = [MyUserName] , ANONYMOUS = OFF

Now all you need to do is get the stored procedure that is activated on the receiving queue to manage the message payload JEasy as that!

USE [SSBReceiver]

GO

/****** Object: StoredProcedure [dbo].[SSBReceiveMessage] Script Date: 03/10/2010 11:20:17 ******/

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATE PROCEDURE [dbo].[SSBReceiveMessage]

AS

print ‘SSBReceiveMessage started’

DECLARE @RecvReqDlgHandle UNIQUEIDENTIFIER;

DECLARE @RecvReqMsg XML;

DECLARE @RecvReqMsgName SYSNAME;

DECLARE @ReportAnswerHandle UNIQUEIDENTIFIER;

–begin an answering dialogue with the TriggerArchiving service

;BEGIN DIALOG @ReportAnswerHandle

FROM SERVICE ReportingReceivingService

TO SERVICE N’TriggerArchiving’

ON CONTRACT ArchivingTables

WITH ENCRYPTION = OFF;

WHILE 1=1

BEGIN

SELECT @RecvReqDlgHandle = NULL, @RecvReqMsg = NULL, @RecvReqMsgName = NULL;

WAITFOR

(

RECEIVE TOP(1)

@RecvReqDlgHandle = conversation_handle,

@RecvReqMsg = message_body,

@RecvReqMsgName = message_type_name

FROM [ReceivingQueue]

), TIMEOUT 10000

IF @RecvReqMsgName = N’http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog&#8217;

OR @RecvReqMsgName = N’http://schemas.microsoft.com/SQL/ServiceBroker/Error&#8217;

BEGIN

IF @RecvReqDlgHandle IS NOT NULL

BEGIN

END CONVERSATION @RecvReqDlgHandle;

END

BREAK

END

–in case the message is null, break looping

IF @RecvReqMsg IS NULL

BEGIN

BREAK

END

BEGIN TRY

–execute procedure that pulls archived records

EXEC dbo.PullArchiveRecords @tabletype = @RecvReqMsgName, @list = @RecvReqMsg

–apply Success Status element in @RecvReqMsg

SET @RecvReqMsg.modify(‘insert element TransferStatus {2} as first into .’)

END TRY

BEGIN CATCH

–at this point an error occured

print ‘Error occured during PullArchiveRecords: ‘ + ERROR_MESSAGE()

–apply Error Status element in @RecvReqMsg

SET @RecvReqMsg.modify(‘insert element TransferStatus {3} as first into .’)

END CATCH

–send the action answer to TriggerArchiving

;SEND ON CONVERSATION @ReportAnswerHandle MESSAGE TYPE @RecvReqMsgName(@RecvReqMsg);

END

–end conversation with TriggerArchiving service

END CONVERSATION @ReportAnswerHandle

RETURN 0

From here, you CREATE your own logic on the calling proc from the above proc to transfer the batches to the reporting/Archive tables.

USE [SSBReceiver]

GO

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATE PROCEDURE [dbo].[PullArchiveRecords] @tabletype SYSNAME = NULL, @list XML = NULL

AS

IF @tabletype IS NULL OR @list IS NULL

BEGIN

RETURN 0

END

DECLARE @tmp TABLE (archive_id BIGINT, tmpid INT IDENTITY(1,1) PRIMARY KEY CLUSTERED (archive_id, tmpid))

–shred the @list of values

INSERT INTO @tmp(archive_id)

SELECT a.col.value(‘text()[1]’, ‘BIGINT’)

FROM @list.nodes(‘id’) a(col)

IF NOT EXISTS(SELECT TOP 1 * FROM @tmp)

BEGIN

RETURN 0

END

–pull records from Archiving server

–based on @tabletype

BEGIN TRY

IF @tabletype = ‘Table_Error’

BEGIN

–PUT LOGIC HERE to insert from payload joining on @tmp

END

ELSE IF @tabletype = ‘Table_Transaction’

BEGIN

–PUT LOGIC HERE to insert from payload joining on @tmp

END

ELSE IF @tabletype = ‘Table_Record’

BEGIN

–PUT LOGIC HERE to insert from payload joining on @tmp

END

ELSE IF @tabletype = ‘Table_RecordData’

BEGIN

–PUT LOGIC HERE to insert from payload joining on @tmp

END

END TRY

BEGIN CATCH

INSERT INTO dbo.myerrors(errormessage)

values(ERROR_MESSAGE())

END CATCH

Reporting Server Facts Job

You can then create a FACT Table job to update the facts table with data that was just imported. You can use a logical flag to ensure data is flagged in the procedure above, where you implement your logic when inserting the new data. The fact job can then use a flag value to update facts, similar to a CUBE on an OLAP server.

Conclusion

You can develop some super fast and stream lime synchronization mechanisms with service broker, transferring millions of records is a breeze, since batch sizes are kept small and we spawn 48-50 parrallel batching processes at once into the queue, you ensure your Transaction log checkpoints are efficient, especially on high processing load days. Much more efficient that Replication and Log Shipping, and MUCH easier to support. This means, we can always keep the queue size at a max of +- 50 🙂 Easy to manage on large transaction days.

I have transferred around 300 million records and not had a hiccup since.

Windows Communication Foundation – Biztalk Adapter – WCF-Custom

Hi Folks,

I wanted to see if I could use the WCF-Custom Adapter to match my scenario below:

Scenario:

I want to call a stored procedure with multiple arguments without using orchestrations and one of the parameters of the stored procedure is the actual BizTalk Message as XML SQL data type.

Consider a scenario where you do not want the overhead of generating metadata, creating an orchestration, deploying the orchestration, and executing the operation. For such a basic scenario, where the stored procedure takes a single parameter, you do not need to create an orchestration. Rather, you can configure a WCF-Custom or WCF-SQL send port to directly invoke the stored procedure. This topic demonstrates how to perform these tasks using the BizTalk Server Administration console.

Prerequisites:

WCF LOB Adapter SDK SP2

and the following

Microsoft BizTalk Adapter for SQL Server:

Ok, so either way, lets continue!

I managed to do it with this template in the WCF-Custom properties, however I found some limitations.

Here is a template that works, notice I HARD coded the Stage, Status and Id, call then constants for now.

<SetWorkflowRecord xmlns="http://schemas.microsoft.com/Sql/2008/05/Procedures/dbo">
<Message><bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007" encoding="string"/></Message>
<Stage>TWO</Stage>
<Status>SUC</Status>
<Id>968819</Id>
</SetWorkflowRecord>

How the above template will probably not be cool, who hard codes values into send ports….

However this template is pretty cool, we can actually use WCF to call a proc directly with multiple parameters, as long as the other parameters besides the Message is constants. Note, you can generate a default template using the Consume Adapter Service BizTalk Project Add-In, and then generate an instance.

I while back I developed a custom SQL adapter that can call a SQL procedure directly and match the scenario above, you know what I love about the custom adapter, is that it uses a STREAM and not a String to send the XML data to the stored procedure. So for large files, I can imagine a Out Of Memory Exception. Also the custom SQL adapter can parse Constants and promoted properties as parameters to a stored procedure, and that is very powerful stuff. We need this power in WCF-Custom or WCF-SQL adapters! I mean it. This allows us to avoid orchestrations and updategrams etc.

However, here is my wish list, and I hope a Microsoft BizTalk Developer finds this wish list:

I wish for a WCF-Custom adapter, where you can reference promoted properties in the WCF template properties. This would match what I can do with the custom SQL adapter we built. Where I can send XML, and promoted property values from BizTalk directly to a WCF adapter, without the need for developing complex WCF Chanel Model or WCF Service Model tools. I also wish that the Message is streams to the SQL server procedure.

So, perhaps in the near future, this type of template will work in WCF-Custom. Where the template has access to the BizTalk Property Schema. Notice below we no longer use encoding type of string to the message, will be faster, of course this sort of template will never work now, but nice to have one day? It could be an add in for the Consume Adapter Service BizTalk Project Add-In where you can drag promoted property fields into the template. This should be possible, since the following assembly has direct access to the message and all properties:

System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

<SetWorkflowRecord xmlns="http://schemas.microsoft.com/Sql/2008/05/Procedures/dbo">
<Message><bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007"/></Message>
<Stage><StageParam xmlns="http://MMIT.Workflow.Common.Schemas.WorkflowRecordPropertySchema" encoding="string"/></Stage>
<Status><StatusParam xmlns="http://MMIT.Workflow.Common.Schemas.WorkflowRecordPropertySchema" encoding="string"/></Status>
<Id><IdParam xmlns="http://MMIT.Workflow.Common.Schemas.WorkflowRecordPropertySchema" encoding="string"/></Id>
</SetWorkflowRecord>

StageParam, StatusParam and IdParam are promoted properties of the message going into the adapter, and this should be possible, the WCF architecture can support this, since they already have the fixed property schema for:

bts-msg-body xmlns="http://www.microsoft.com/schemas/bts2007

I am excited about WCF, but it can be much more powerful with BizTalk if it can hook into Promoted Properties and not encode the message as a string, but default it to a stream.

This will allow for some awesome low latency BizTalk Patterns…. I understand that we need to keep promoted properties under 10 or so, which is the case, but sometime you want to access promoted properties for database level operations, thus avoiding XPath on them again in the proc to save time if you already have it promoted.

I have migrated all send ports to WCF, so my solution is still simple, all I do is change the stored procedure to accept one parameter and then in the SQL proc, we XPATH the stage, status and ID values, which should be fine, however it will be slightly slower as in the stored procedure I run a SQL Message.Query to get the stage, status and ID :), I will do some performance test against both ways, WCF and the Custom Adapter. However if the WCF is slightly slower, I will probabyl stick with so that in the long run we are WCF compliant!

One thing I like very much is the Transaction Isolation Level and the fact we can use Read Committed 🙂 Also, I prefer Custom as it is more flexible than the WCF-SQL 🙂

Below are screenshots for setting up the adapter:

Click Configure

Then add your SQL URI and stored procedure name:

Then change the binding to SQLBinding, to do this, you must install the following:

WCF LOB Adapter SDK SP2
Microsoft BizTalk Adapter for SQL Server:

, eitheway, you should now be able to choose the SQLBinding.

Then you can create your template on the messages tab, NOTE I change the transaction isolation level here, as I want READPAST hints.

That is all there is to it. If you having trouble making the template, just use Visual Studio to do it for you, then copy past the instance:

Just add a generated item to a existring BizTalk project and choose Consume WCF Service:

Since you NOT using orchestrationsm, you do not need to save the generated items 🙂 🙂

Once you got an instance, that is the template to copy paste into the Template section for the messages.

Hope this helps thos of you who want high performance writes to SQL Server.

 

This is the best option to use, a long while back I had a custom SQL adapter developed for this as posted in the blogs.

Cheers

Designing BizTalk Solutions Tips

Hi Folks,

I have decided to start a series of posts in regards to implementing BizTalk solutions. In these solutions I will assume that the following goals are desirable:

Services

  • Loosely coupled architecture
  • Abstraction by implementing web service or WCF solutions
    • These services must provide discrete components, that when orchestrated as a whole can provide logical business processes, this means that they should be re-useable.
  • Just because you implement web services or WCF solutions, does not mean you implementing SOA.
  • Entry points into the Service Bus must PROTECT data integrity. This means all WCF/WS* services must implement strict message contracts
    • How many of you have been given a web service to consume by a third part and when you download their WSDL, you notice that you can submit any data, just a generic XmlDocument? Not good.
    • Secondly by protecting the Service Bus from badly formed data, you will save yourself heaps of problems.
  • A “Internal” XSD schema is used to orchestrate, route and update common artefacts. This schema is usually referred to as a canonical schema.
    • This means a C# version of the XSD is generated either with xsd.exe, svcutil.exe or xsd2class (my favourite). Therefore always ensure these classes that are used to expose your service contracts are PARTIAL. You can then create a seperate partial class for CUSTOM implementations such as Schema/Serialization and manipulation routines.
    • You may ask, why, we can have a BizTalk pipeline do this for us. Yes indeed you can. However many EXTERNAL facing services must be decoupled from the BizTalk system. This is where you can leverage strict service contracts to the outside world. Just because a WCF or Web Service accepts a message, does not mean it conforms to the XSD. Class generated code from XSD does not implement all the rules! e.g. Maximum Occurence and Minimum Occurrence.
    • There can be some disadvantages here. If you implement a canonical schema and not much thought went into the design, then you go to have some serious problems later on down the line. INVEST allot of TIME with business analysts and users, so that your canonical schema matches all business processes.
    • Any CHANGES to the canonical schema MUST be backward compatible.
    • Use namespaces to provide logical sections in the canonical schema to limit the IMPACT of new changes to the schema.
    • Let me put it this way. If the Canonical schema changes, your BizTalk Maps that map to this schema should still work.
  • Use Document Literal if you need the messaging payload in your services, this message is not bound to any distinct operation.
  • Use RPC style if you need a small subset of the data and need to clearly define what operation the message payload supports.
  • if you unsure of the above, then go for Document Literal, most flexible.
  • If using web services, implement SOAP security headers in external facing web services.
  • Ensure core services have decent logging capabilities and perhaps even a web page to view the Inbound data, this can save users allot of hassles when chatting to clients about submitted documents that are not processed.
  • You can use a generic XSD schema (ANY type) for internal processing, when the schema has already been validated. Why always validate the schema during multiple processes, if the entry point is valid? Sometimes this is necessary, however, when working deep within the service bus, you may find it easy to use a generic schema e.g. A Dispatching system, that already received validated documents and just needs to dispatch them to other services.
  • Ensure you web.configs etc for services are tuned to support multiple connections (Default is 2). if they are not you will see many send port instances in BizTalk ready to send, because the max connections on the web service is defaulted to 2!
    http://grounding.co.za/blogs/romiko/archive/2009/04/11/biztalk-2006-optimising-soap-send-ports-and-sql-query-optimizer.aspx

   <system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "50"/>
    </connectionManagement>   
    </system.net>

  • Ensure repeated calls to the same service will not changed the state of the message, unless you really intend to do so.
  • If there is a chance that the same message will be delivered may times during the day (Customer updates), then perhaps a subscription service model can be used, where the system will PULL the latest record. This will solve problems where you cannot guarantee the latest message for the same customer will be processed last. Also, by doing so, you can avoid messaging queuing (which can be slow).
  • There is nothing wrong incorporating solutions that PULL from web services, as the example above explains.
  • Use Windows Communication Foundation to communicate between services within the organisation if you can, especially if the communication will be on the same server (netbinding etc)

Canonical Schema

  • As I mentioned, INVEST allot of time defining, designing and implement this schema. get intermit with it.
  • Appending Service Bus metadata to the canonical schema is extremely Powerful. You can use this meta data for Property Schemas and Distinguished fields. You can put this data in a separate namespace. This gives you allot of flexibility.
  • When extracting data from SQL for BizTalk to consume, this is a good opportunity to see what SQL fields can be used in the canonical schema to provide metadata.
    • Example: You may have a SharePoint/FTP/WCF address associated with the data from SQL. Why not extract this on the SQL receive location and append it to the metadata of the message payload? You can then leverage Dynamic Send Ports 🙂
  • In a nutshell, think through this, it can save you allot of time, and allow you to implement some call dynamic routing patterns, however the Schema must be really well designed and i did mentioned backward compatible.

Routing

  • Never use a pattern where an Orchestration is used to invoke a Business Rule Engine policy and uses the result of the rule execution to define a message routing. I have seen this happen allot, NEVER do it. It slows down the messaging engine. Use filters on Send Ports, this is loosely coupled.
  • Use Filters on Send Ports. PLAN your Property Schema.
  • If your content filters for any BizTalk application is more that 10 types of property schema members, then you have a design issue. Keep your Property Schema Small, ideally fewer that 10 elements.
  • Try to use dynamic send ports where it is possible to dynamically gather meta data about the destination end point.

Orchestration

  • Try not to use BizTalk Maps in an orchestration. Why? You are going to tightly couple your solution here. Rather implement maps at the entry and exit points. Send and Receive Ports.
  • Keep the persistent points on your orchestrations down to a minimum, use SCOPES to control persistent points.
  • Do not use the expression editor for complex logical, and do not TRY hack the expression editor by using a multitude of static serializable classes to get the job done, this is a sign something is wrong, usually exposing the correct distinguished properties can solve this problem!
  • I have seen to many implementations where Orchestration are used, just because well “They easy to implement”. Most of the times you do not need to use them, where the BizTalk messaging engine will suffice. Reasons why people use them so much, is that not allot of thought goes into the design of the canonical schema.
  • DO NOT couple your orchestration directly to a WCF/Web Service port. It is just UGLY. Secondly you expose your internal orchestration logic to the outside world! You have no control how the contract is generated.
    • Try and get orchestrations to receive data from DIRECT ports bound to the message box.
  • So direct ports is the way to go. Keep you service logic separate from orchestrations as mentioned. Use direct ports to the message box and just use filters on the message receive shaped. This gives you the best loosely coupled pattern.
  • If you are call external classes, ensure they are THREAD SAFE!

Failures

  • Use BizTalk’s routing of failed message subscription features to reroute failed messages. Do not underestimate how useful this can be e.g. Route failed messages to different document libraries in SharePoint for manual intervention, or to a custom human workflow database for web based fixing/editing.

BizTalk Mapper

  • Keep your maps SIMPLE. Do not implement complex logic here, if you do, then something is wrong with the design.
  • As I mentioned implement maps on the latest or earliest stage, not in the orchestration engine, unless you cannot help it.

In fact, I worked on a project where the Business required full control of mapping flat files to the canonical schema. They did not want to use the BizTalk Mapper. We built a custom ASP.NET mapper for them, and they can generate maps during runtime without the need for deployments of new maps. We leveraged Altova products, where they can use XSLT bases mapping and then load them into the cache (Compiled version of the xslt) where our service can call them at runtime. Altova has a really cool XSLT engine. We also extended this for extremely complex files so that a software factory and implement code, however these sort of files require an assembly update. However such files are only 5% of the file feeds, and dealing with over 300 file feeds, we can see that a web based mapper, that biztalk can consume via a service is an awesome way to loosely couple mappings from business processes.

I have seen some bad ass maps, where all I see is BLACK, not even the lines, with all sort of functoids, function calls and weird things going on, avoid complex mappings.

Remember large xml files BLOW up when processed by an XSLT engine. There are other ways to transform flat files if they are extremely large, do not always have to use XSLT’s. However, design a solution that is flexible in this regard if you dealing with flat files that are really complex, where xslts or other transformation solutions can be implemented and are loosely coupled.

SharePoint Integration

The SharePoint Adapter for BizTalk is really powerful and rock solid, leverage SharePoint if you require message collaboration! We use dynamic SharePoint send ports so we have FULL control over custom columns in the document library. This allows us to publish the message AND metadata to SharePoint and leverage SharePoint document library views. E.g. Sending a file to SharePoint and using a custom column in the document library to mentioned the Country where the file came from, or other properties.

Well, I think that’s enough tips I can think of for today! I hope this helps. Now, back on the bright side, here are some pics i took on a wildlife expeditions 🙂 Yes, life is not all about geeking it to the max!

This was in the Kruger National Park, I was in a concession area, no fences, just me and the wild animals 🙂

_MG_1456 _MG_1459

DSC02096 DSC02169

This elephant was not happy with me at all! I immediately turned into Casper the ghost!

 IMG_1107 IMG_1365

This is me 🙂 Yes a Black Mamba in the tent.

IMG_1428 IMG_1042

_MG_1087 _MG_1094

This dude is hardcore bushmen I ever met!

_MG_1166 _MG_1336

I am not sure, but I think this is a cameroptra bashing a worm, poor thing…

 

_MG_1334 _MG_1347

Found these lions, eating a DONKEY!!!!

 

  _MG_1271 _MG_1381

The Saddle billed stork above is really a rare site to see!

_MG_1442 Handsome chap isn’t he?

I hope you enjoy the photos. I picked up a Canon 450d and the book for dummies, so hopefully i will improve!

BizTalk 2009 and Sun Virtual Box

Hi!

A quick note that I must say in regards to hosting BizTalk in our virtual environment that Sun Virtual Box has really out did them. They have a really stable virtual machine product. BizTalk, SQL and MOSS have been running really smooth on my development box which runs Sun Virtual Box.

Another nice thing is that they continually update the product.

This is my strategy for templates.

I will build a Windows 2003/2008 Virtual Box base template.

  1. Install Visual Studio 2008
  2. Install SQL 2005/2008
  3. Install BizTalk (DO NOT CONFIGURE)
  4. Adapter Packs e.g. WCF LOB, SQL
  5. DebugView
  6. Resharper (if You like it)
  7. Notepad++
  8. Orchestration Profiler
  9. BizTalk Environment Configuration: http://biztalkconfigloader.codeplex.com/

Rudolf Henning on CodePlex updated my original BizTalk Environment Configuration Tooland made some cool enhancements.

Then you can just close the template above for new developers or for new projects, of course you need a new computer name and IP address. The only trick is for SQL. When you close it, you just run the following command:

sp_dropserver <old_name>

GO

sp_addserver <new_name>, local

GO

A really stable product!

Sun Virtual Box is much better than VMware, hands down. Another aspect I like about Virtual Box is that you can use the console to work on the machine, and the mouse pointers etc feel as if the machine was real, no VMware tools or mouse pointer acceleration to deal with, or even remote desktop as Virtual Box has it’s own built in addin which is a must to install.

HTH

BizTalk 2006: Optimising Soap Send Ports and SQL query optimizer

Hi Folks,

Once you have gone live in production with your BizTalk system, you will certainly need to tune the system in order to get a high throughput.

There were two things causing the throughput of our system to be slow.

  1. Limitation of connections by the windows server to SOAP/HTTP servers. Limit is 2
  2. A bug in the SQL query optimizer when doing implicit data type conversions

We were processing around 10 000 messages per hour when the two settings above were incorrect. After making some adjustments we bumped up the processing to 100 000 messages per hour. Each messages does the following:

  1. Imported into the database
  2. Sent to a GEO Coder using SOAP
  3. Sent to XSLT engine using SOAP
  4. Sent to a SQL Algorithm system using SOAP
  5. Send to a CRM system using SQL

In all the steps, a transaction log and custom database is used to keep track of the information and also for a custom human workflow application to edit records that are not well formed.

The following is what was needed to get the throughput higher.

SOAP

Edit the BTSNTSvc.exe.config

Add the following section:

    <system.net>
    <connectionManagement>
        <add address = "*" maxconnection = "50"/>
    </connectionManagement>   
    </system.net>

What this does is ensures the server can have more than 2 connections (by default) simultaneously when calling a web service or http service. The setting above should be 12 x (Number of processors). We have a quad core processor, so 48-50 should be a good number for us.

You can read more here:  http://msdn.microsoft.com/en-us/library/aa545389.aspx

One way of knowing if this is a solution, is checking the event log in BizTalk. If you see allot of web service timeouts and you have set high timeout settings in your configuration files, proxy class and message context properties in orchestrations, then it must be the machine limiting the service calls or the web service needs some tuning. In our case, the server was limiting the number of calls.

What you will see in the Group Hub Page, is many send ports with this status:

READY TO RUN

image

Once we made the change in the BTSNTSVc.exe.config and restart the host instances, there SOAP send ports no longer had a ready to run status.

After the change, the timeouts in the event log disappeared and the query page looks much better, with ACTIVE status for all SOAP send port.

image

SQL

For the SQL side, we had many receive locations calling the same stored procedure, but using different values for the parameters. The parameters are the BATCH SIZE and STAGE.

CREATE  PROCEDURE [dbo].[GetWorkflowRecord]
@BatchSize int,
@Stage nvarchar(3) = null
AS
BEGIN

In the actual SQL table the Stage column has a data type of varchar(3). If you look above, the the data type is not the same, it is nvarchar(3), this was causing the SQL optimizer to not successfully calculate how the query should run and caused table scans. TO solve the problem we just changed it to varchar(3), and this would stop the stored procedure from doing an implicit conversion when querying the table, since the @stage is in a where clause:

wfr_wfs_code = (‘SUC’)
            AND        wfr_Batch is null
            AND        wfr_stg_code = @Stage

Here is a nice blog about it:

http://statisticsio.com/tabid/36/articleType/CategoryView/categoryId/10/query-optimizer.aspx

CREATE  PROCEDURE [dbo].[GetWorkflowRecord]
@BatchSize int,
@Stage varchar(3) = null
AS
BEGIN

Once this change was made, I was even able to change the 8 receive locations to call SQL every 1 second and increase the batch size from 150 to 500-1000.

Conclusion

Sometimes small trivial changes can make a huge different in performance, here, we were able to increase the throughout of the system from 240 000 records per day to 1.4 million records per day, with an average rate of 900-1000 messages per minute where each message was executing 10-12 transactions and calling 5-6 web services. Two factors limited the system, SOAP calls and a stored procedure causing table scans since implicit conversion of data in the procedure could not be optimised and detect the correct non-clustered index to use. Two simple change with major positive results! What more can I ask for before the Easter break.

 

Chronicles of South Africa

 DSC00466

Here is a picture of two giraffes in a defensive pose. All the animals on this day were very frisky!

DSC00458

DSC00447

Nothing beats a good boerwors!

DSC00481

DSC00547

I love spiders!

BizTalk File Receive locations: Exhausted Network – SMB limits

Hi Folks,

I was away on holiday in South Africa and on my return, a colleague of mine mentioned that the BizTalk receive locations were unexpectedly shutting down. It would always be a random receive location.

We had just reached 50 receive locations and growing. In the Event Log BizTalk would complain:

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: (1)
Event ID: 5649
Date: Date
Time: Time
User: N/A
Computer: ComputerName
Description:
The receive location "FileReceive" with URL "\\ServerName\FileDrop\FileName.FileNameExtension" is shutting down. Details:"The FILE receive location \\ServerName\FileDrop\FileName.FileNameExtension exhausted the network retry attempts. ".

 

The problem here is due to the application level protocol management system within windows. This all has to do with the LanMan server and LanMan workstation service within windows. Since these services are used to manage UNC connections to folder shares and the tried and trusted Service Message Block application protocol is used here.

To resolve this problem, increase the MaxCmds and MaxMpxCt registry values on the BizTalk Server computer and on the remote UNC Share computer. To do this, follow the steps that are described in Microsoft Knowledge Base article 810886.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:

This issue may occur if the following conditions are true:

  • This issue may occur if the client computer submits simultaneous, long-term requests against a file server that uses the Server Message Block (SMB) protocol. An example of a long-term request is when a client computer uses the FindFirstChangeNotification function to monitor a server share for changes.
  • This issue may occur if the MaxCmds registry value setting on the client is less than 50, or the MaxMpxCtregistry value setting on the server is less than 50.
    Note The MaxMpxCt registry value setting may have a different name on other SMB/CIFS implementations. The Common Internet File System (CIFS) specification refers to it as MaxMpxCount.

In our case it was the former that caused it, since we had 50 receive locations. So we need to increase the value from 50 to let says 200.

To resolve this issue, verify that the MaxCmds and MaxMpxCt registry values are set to 50 or more. To do this, follow these steps:

  1. Click Start, click Run, type regedit, and then click OK.
  2. Locate and then click the following key in the registry:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters

  3. In the right pane, double-click the MaxCmds value.
  4. In the Value data box, verify that the value is 50 or more.
    Note In a Microsoft Windows Terminal Services environment, you may have to set the MaxCmds registry value to 500 or more. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

    232476 Terminal Server client connections and logon limited by MaxWorkItem and MaxMpxCt values

  5. Locate and then click the following key in the registry:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters

    Note Make sure that you make this change to the lanmanserver\parameters registry key and not to thelanmanworkstation\parameters registry key mentioned in step 2.

  6. In the right-pane, double-click the MaxMpxCt value.
    Note On SMB servers that are running a version of Windows earlier than Windows 2000, the MaxMpxCt registry entry is named MaxMpxCount, but has the same function.
  7. In the Value data box, verify that the value is 50 or more.
    Note The MaxMpxCt value determines the maximum number of simultaneous, active requests that the server permits from an individual client.
  8. Quit Registry Editor.

Note The MaxCmds and MaxMpxCt registry entries are REG_DWORD decimal entries. If they do not exist on your computer, you can create them as new REG_DWORD values. The range of values for these registry entries is between 0 and 65535.

It is important that you make this registry change on:

  1. All BizTalk Servers
  2. All File Share Servers being monitored

Happy BizTalking!

Here is a recent picture of South Africa where I am visiting a herd of elephants at the Thula Thula private game reserve:

DSC00800

DSC00813

Yes the elephants were not happy with us. On this day, we actually got mocked charged by a female elephant which is very rare, since the matriarch is usually responsible for this, maybe he was on sick leave! So with everything in life, and as Nature teaches us, we all have our limits, and with these elephants, it was overstaying our invitation.  The female in the background was within 4 metres from us by the time we put the peddle to the metal!