Sunday, December 30, 2007

Finding the Original Filename Processed by BizTalk Server

A common BizTalk implementation starts with business partners (customers, vendors, etc.) transmitting files to you via FTP in their own file format. Then BizTalk transforms that file into your company's standard XML format. Next, your existing systems process that XML file as normal. This simplifies application architecture, because for each new business partner you simply need to add an FTP site, a BizTalk Document Specification, a BizTalk Map, a BizTalk Port, and a BizTalk File Receive Function. While it seems like a large number of items that need to be created for each new business partner, it is much, much easier than creating a new application for each new business partner. The combined time to create all of these new items should be 5 - 30 minutes, depending on how complex the documents are.
If you have a business partner that sends information in the filename (e.g., Check12345.txt) that you need during your processing, you will need to perform a not-so-obvious step, that is not well documented, in order to find out the name of the file sent in by the business partner.

Where Does BizTalk Store Original Filenames
BizTalk tracks the files that it processes in its database (InterchangeDTA). If you have BizTalk write the transformed file out as file%tracking_id%.xml then you will end up with a filename like this: file{544191B6-6755-4AF7-8B82-C53C24B2A2D4}.xml. You can easily parse the Tracking GUID out of this.

Once you have the Tracking GUID, which is the unique identifier that BizTalk uses internally, you can query the InterchangeDTA database for the original name of the file. The query joins four tables:
      Select data.nvcOriginalFileName as Filename
      From  dta_indoc_details inDoc,
            dta_outdoc_details outDoc,
            dta_interchange_details details,
            dta_interchange_data data
      Where outDoc.nInDocKey = inDoc.nInDocKey and
            inDoc.nInterchangeKey = details.nInterchangeKey and
            details.nInterchangeDataKey = data.nInterchangeDataKey and
            outDoc.uidTrackingGUID ='{544191B6-6755-4AF7-8B82-C53C24B2A2D4}'

And there you have it. A (somewhat) simple query will return the original filename from the Tracking GUID. 

Thursday, April 19, 2007

Peeping Inside BizTalk

Developers can easily develop Interfaces using BizTalk. On the Stage the BizTalk Show looks cool but have you thought about who writes the Scripts in background. Its only Drag and Drop for developers and once the solution is deployed and it's up to be tested. At times one must have thought what happens when I drop a message, where it goes and how the whole mechanism works. Just go through the artifacts below ….its worth having a look and I bet you would say….”Hey I dint knew this at all”.
1.      Information about the promoted properties is extracted and stored in the bts_documentSpec table in the Management database.
2.      Items that are marked with PropertyField annotations in your message schema will lead to the pipeline disassembler putting a Promoted property in the context. Items that are marked with DistinguishedField annotations in your message schema will lead to the pipeline disassembler putting a Written property into the context.
3.      One of the benefits of promoted properties is that the value of the element that is promoted is available in the context of the message. This means that retrieving that value is inexpensive, as it does not require loading the message into memory to execute an XPath statement on the message. Instead, a simple property bag can be used along with a key to get the value.
4.      Writing a value into the context with the same name and namespace that were used previously to promote the property causes that property to no longer be promoted. The write essentially overwrites the promotion.

Suscriptions:-Instance, Activation
1.      An activation subscription is one specifying that a message that fulfills the subscription should activate, or create, a new instance of the subscriber when it is received. Examples of things that create activation subscriptions include send ports with filters or send ports that are bound to orchestrations, and orchestration receive shapes that have their Activate property set to true. An instance subscription indicates that messages that fulfill the subscription should be routed to an already-running instance of the subscriber. Examples of things that create instance subscriptions are orchestrations with correlated receives and request/response-style receive ports waiting  for a response from BizTalk Server.

2.      The difference between the two types of subscription at the information level is that an instance subscription includes the unique instance ID, stored in the subscription table in the master MessageBox database. When an  orchestration instance or receive port completes processing, instance subscriptions are removed from the MessageBox while activation subscriptions remain active as long as the orchestration or send port is enlisted.

Picture above shows what Biztalk actually does when a message comes in or is going out.We dont see it in Biztalk architecture diagram.

What Adapter does with the Message
1.      The adapter creates a message (an implementation of the Microsoft.BizTalk.Message.Interop.IBaseMessage interface), adds a part to it (an implementation of the Microsoft.BizTalk.Message.Interop.IBasePart interface), and provides the  stream of data as the part content, the adapter writes and promotes into the message context properties related to the location, adapter type, and others related to the adapter. After the message and its context have been created, the adapter passes the message to the Endpoint Manager. The message is then processed through the receive pipeline, which has been configured for the receive location. After the message has been processed by the pipeline, a map may  be used to transform the message into the format desired before the Endpoint Manager publishes the message with the  Message Agent.

Out of Memory Problems: Why?
1.      Routing only If BizTalk Server is only used only for routing messages based upon promoted message properties, then  the message is streamed into the Messagebox database using the .NET XmlReader interface, and message parts are not  individually loaded into memory. In this scenario, out of memory errors are not an issue and the primary  consideration is the amount of time that is required to write very large messages (over 100 MB) into the Messagebox  database.The BizTalk Server development team has successfully tested the processing of messages up to 1 GB in size   when performing routing only.

2.       Mapping Transforming a document with a map is a memory-intensive operation. When a document is transformed by a map, BizTalk Server passes the message stream to the .Net XslTransform class, which then loads the document into a .NET  XPathDocument object for processing. Loading the document into the .NET XPathDocument can potentially expand the original file size in memory by a factor of 10 or more. This expansion may be more pronounced when mapping flat files because flat files must be parsed into XML before they can be transformed.

1.       Adjust the message size threshold above which documents are buffered to the file system during mapping. To modify the size threshold, create a DWORD value named TransformThreshold at the following location in the BizTalk Server  registry:
        HKLM\Software\Microsoft\BizTalk Server\3.0\Administration\TransformThreshold
        After you have created this value, enter a decimal value with the number of bytes to set the new threshold to. For  example, enter a decimal value of 2097152 to increase the message size threshold to 2 MB (from the default of 1 MB). Increase this value on systems with a large amount of available memory to improve throughput. Buffering documents to disk conserves memory at a slight cost to overall throughput.
All Inputs above are Courtesy to Microsoft.