This was one of those very frustrating errors that had a very simple solution.  I had an orchestration that was dynamically loading a map using a Fully Qualified Name (FQN) that is stored in BRE.  The exception looked like this:

{ORCHESTRATION_NAME} encountered an error: Object reference not set to an instance of an object.
at Microsoft.XLANGs.Core.Service.ApplyTransform(Type mapRef, Object[] outParams, Object[] inParams)
at {ORCHESTRATION_NAME}.segment3(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

 Many times, this is the result of not deploying your DLL to the GAC, or not having a schema available in the Management Database.  I double checked both of those, and they were there.  I removed them and manually reinstalled them using gacutil, still got the same error.  It finally occurred to me to use reflection on the assembly to see if the FQN was wrong, and that was indeed my culprit.  The class name had .Outbound appended to it, and my orchestration was trying to load a class that ended in .Maps.  I could have changed the FQL in the BRE, but that would have been inconsistent with naming conventions elsewhere in the project.

The source of the error was adding the map to a Solution subfolder in Visual Studio:

This made the Namespace of the map default to Maps.Outbound:

Changing the highlighted property to .Maps and rebuilding/deploying fixed the issue.

I’m currently working on a solution that exposes a BizTalk Orchestration as a RESTful webservice using WCF-WebHttp.  Upon successful processing of the message, the service returns binary data for the client to download.  However, if the processing fails, I wanted the service to return application/json data with an error message that would be suitable for consumers – I did not want Stack Traces or other internal details to be provided that could potentially be used by malicious consumers.  Further, I wanted to ensure this information was sent in the same session.

Initially, I created a schema with my sanitized error information, set the Fault Operation Message Message Type to that schema, and enabled “Include exception details in faults.”  While this did end up including my custom error message, it also included the extra information I was trying to avoid.  It also meant always sending back the error reply using HTTP status code 500, which is undesirable if the error is actually in the message submitted.  Finally, the message was being returned in XML, which would be more challenging for my JavaScript-based clients to parse.

To resolve these issues, I created a custom WCF Behavior with three classes.

The class which does most of the work, SanitizedJsonFaultHandler, implements IErrorHandler, and contains a subclass JsonErrorBodyWriter to help with JSON serialization.  In the ProvideFault override, I parse the exception message into an XDocument.  One of the fields in that document is the HTTP Status Code I wish to use for that message (400 if the validation of the file failed, 500 if an internal exception occurred); this field gets removed, as it would be redundant to include it in the message body.  I then set the fault message using the JsonErrorBodyWriter class to serialize the XML message to a JSON message.  The message has only a root node and string value child nodes.

Two other classes help make this behavior available to the BizTalk adapter.  One implements IEndpointBehavior, adding my custom error handler to the endpointDispatcher:

And another overrides BehaviorExtensionElement so that this behavior will be visible to the system:

Finally, I added the following line to my machine.config files (32 and 64 bit; replace FULLNAMESPACE with your namespace and FULLY_QUALIFIED_NAME with the FQN of the DLL you create and GAC.  This information can be found by running the command ‘gacutil /l | find "WCFBehaviors"‘).


<system.serviceModel>
<extensions>
<behaviorExtensions>
........
<add name="SanitizedJsonFault" type="FULLNAMESPACE.WCFBehaviors.SanitizedJsonFaultBehaviorElement, FULLY_QUALIFIED_NAME"/>


With this done, I restarted IIS and BizTalk.  Then I was able to add the endpoint behavior by right clicking on EndpointBehavior:

Now, my orchestration can log the sensitive information for later review (or send it to an administrator directly), and the fault message is sent back to the client like so:

500 error:

400 error:

Microsoft Azure BizTalk Services (MABS) has a lot to offer for companies looking for a PaaS Middleware solution.  EAI bridges provide B2B communication as well as LoB access functionality for EDI, XML, and flat file interchanges.  The new mapping system offers some exciting and powerful new functionality as well, vastly simplifying certain tasks that previously required several functiods, and opening up new possibilities and enhanced performance with lists.

However, it is a new technology, and certain tasks that have been very straightforward in BizTalk Server 2013 require a different way of thinking for MABS.  For example, it is a fairly trivial task to create an orchestration that accesses a LoB adapter (using, for example, WCF slqBinding) to do data validation or enhancement, and publishing this orchestration as a web service for client consumption. If your SQL database is SQL Azure, there is some built in functionality to do a “Lookup” in the Azure database, but this may not be an option for an infrastructure that is makes use of features not currently available in SQL Azure, such as the ability to encrypt at rest.  It may also just be possible that an existing LoB SQL database cannot easily be moved for various other reasons.  In this series of posts, I will outline the process for implementing this functionality using the powerful custom code functionality available in MABS.

The tasks include the following (this post will cover steps 1-6):

1. Creating the BizTalk services
2. Setting up BizTalk Adapter Services in a local (or IaaS) environment to run a stored procedure in SQL Server
3. Creating a sample table and stored procedure
4. Creating a ServiceBus namespace with ACS
5. Create the Relay to the LOB system
6. Creating an EAI bridge to access the LoB system
7. Testing and debugging the bridge with a Visual Studio add on
8. Writing a custom component to call the LoB adapter in a EAI Bridge Stage and parse the response
9. Having the component send an email notification using an Office 365 server

Step 1: Create BizTalk Service This is fairly straightforward.  Log into your Azure portal, click BizTalk Service, click New, and then Custom Create.  Choose a name, edition, and region, and your service will begin provisioning:

You’ll need the ACS Information for your newly created service for several of the following steps.  You can get this information by selecting the service you created and clicking “Connection Information” at the bottom.  That will display the following information.  You’ll need similar information for the ServiceBus relay (created in step 4) as well; I found it convenient to copy this information into an Excel sheet for quick reference during development (however, be sure this is stored in a secure manner and is consistent with your IT security policies – this is sensitive information that should not be disclosed to non-administrators of the service):

Step 2: Setting up BizTalk Adapter Services For this, you’ll need to get the BizTalk Services SDK: http://www.microsoft.com/en-us/download/details.aspx?id=39087.  The instructions for this can be found at: http://msdn.microsoft.com/en-us/library/azure/hh689760.aspx#BKMK_Installation.  Take note of the requirement regarding machine.config if you have previously installed Azure BizTalk SDK components, and install the optional runtime (this had me going in circles for a while!).  Also take note of the requirements for downloading and installing the certificate from the Azure:

To add it to your security store (see http://msdn.microsoft.com/en-us/library/azure/hh949825.aspx):

1. On your test/development machine, double-click the .cer file. Select Install Certificate.
2. Select Place all certificates in the following store, select Browse, and select Trusted Root Certification Authorities.
3. Complete the installation. You’ll get a Security Warning; which is normal. Click Yes.

When prompted for the service identity, be sure to use an account that has internet access and permissions SQL (or whatever LoB item you’re working with).   A successful install should result in the following page when you navigate to https://localhost:8080/BAService/ManagementService.svc/

Step 3: Creating a sample stored procedure For this sample, I’m working with a SQL Server database running in a VM behind Tallan’s firewall.  I did not have to do any special firewall configuration for the VM, my host machine, or IT to get this up and running, nor did I have to make use of any kind of tricky proxying methods.  In my database (named BTSTrainDb, a database I have for testing and samples), I created a table: dbo.sn.  This table has an ID column, a varchar SerialNumber column, and a bit IsValid column.  I also created a simple stored procedure that takes a varchar as a parameter and uses it to do a look up on the dbo.sn table:

USE [BTSTrainDb]
GO

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[sn](
 [ID] [int] IDENTITY(1,1) NOT NULL,
 [SerialNumber] [varchar](50) NOT NULL,
 [IsValid] [bit] NOT NULL
) ON [PRIMARY]

GO

SET IDENTITY_INSERT dbo.sn ON
GO

INSERT INTO [dbo].[sn] (SerialNumber, IsValid) VALUES ('asdf', 1), ('fdsa', 0), ('test', 1);'
GO

SET IDENTITY_INSERT dbo.sn OFF
GO

CREATE PROCEDURE [dbo].[usp_ValidateSN]
(
 @sn varchar(50)
)
AS
BEGIN
SET NOCOUNT ON;

SELECT IsValid from dbo.sn WHERE SerialNumber = @sn;
END

Obviously, this procedure could contain any amount of business logic and/or perform other CRUD operations if desired.  I also have a SQL user that has permissions on the database, table, and stored procedure. Step 4: Create a ServiceBus relay with an ACS namespace Microsoft has recently removed the ability to do this from the management portal, and so it has to be done through a PowerShell cmdlet.  On top of that, the current documentation for this command is out of date!  Cesar Cordero (a colleague here at Tallan) just recently wrote a blog about this with some more details here: http://blog.tallan.com/2014/11/06/service-bus-authentication-and-authorization/ Here’s an overview of what you need to do:

1. Get the Azure PowerShell module(s) if you don’t already have them: http://azure.microsoft.com/en-us/documentation/articles/install-configure-powershell/
2. Add-AzureAccount; follow the steps in the dialog that pops up
3. Select-AzureSubscription (if your account is associated with more than one subscription; this information can be found by clicking on the “Subscriptons” button near your username at the top of the Azure Portal)
4. New-AzureSBNamespace -Name <name> -Location “Eastern US”; at the prompt for NamespaceType, type Messaging.  Note: The name of this namespace must be different from the name of the BizTalk service.  If it is not, there will be a clash between the ACS namespace used for the BizTalk Services ACS and the ServiceBus ACS. 
5. You can now retrieve the connection information for this ServiceBus namespace from the management portal (or copy it from the output of the PowerShell cmdlet).  I made a note of it as it’s required for configuration in step 5.

Step 5: Create the LoB relay Open Visual Studio.  In the Server Explorer, right click on BizTalk Adapter Services, and select Add a BizTalk Adapter Service.  Enter the URL set up in step 2 (https://localhost:8080/BAService/ManagementService.svc/). Expand the newly created adapter service.  You will be prompted to enter ACS information; enter the information for the BizTalk Adapater Service ACS from Step 1. Right click on SQL, and click “Add a Target”.  For connection parameters, fill in the server information

Operations should look familiar to anyone who’s used the WCF SQL binding.  Go to Strongly-Typed Procedures and add the dbo.usp_ValidateSN:

Choose your runtime security.  Here, I entered the username that has access to the procedures and its password as a Fixed username:

Specify the LOB Relay URL using your newly created service bus namespace; for LOB Relay path, choose something that will be memorable this particular SQL relay path, and for the sub-path choose a unique identifier (like the InboundID parameter for WCF SQL binding in on prem BizTalk):

Review your information on the next page and click create.   Step 6: Create an EAI Bridge for this relay In Visual Studio, create a new BizTalk Service project (Under Visual C#).  Right click on your new LOB SQL type and click “Add schemas to <ProjectName>”; enter the project folder and filename prefix you want, and enter the credentials you set up in the previous step under runtime security.  This will generate two schemas that should look pretty familiar if you’ve used WCF SQL before. Drag the LOB SQL type over onto your design surface (named MessageFlowItnerary.bcs by default).  Drag an XML Request-Reply bridge onto the surface from the toolbox.  Drag a connector between the two. On the connector, set the following properties: Filter Condition: Match All (will set to 1=1): this sets the connector to always send messages through; we’ll cover content routing in the next article in this series

Route Action: configure as follows (again, this should look familiar!): this sets the SOAP action header

Your surface should look like this now (after renaming the XML bridge to SQLRequest):

Double click the SQL LOB icon (circled). In the config file that pops up, edit the sharedSecret node with  the information from your ServiceBus relay ACS (not BizTalk Services ACS):

Build and deploy the solution (you’ll be prompted for your BizTalk Services ACS information on deployment).  This operation should not require a refresh of the service, but go ahead and select it anyway.  Grab the MessageSender app: https://code.msdn.microsoft.com/windowsazure/Windows-Azure-BizTalk-EAI-af9bc99f.  Build it and run the following command with a generated instance of the request schema:

messagesender.exe <BizTalk Service Namespace Name> owner <BizTalk Service ACS Key> https://<name>.biztalk.windows.net/default/SQLRequest <instance of the request schema.xml> application/xml

A sample request message looks like this:

<ns0:usp_ValidateSN xmlns:ns0="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo">
 <ns0:sn>asdf</ns0:sn>
</ns0:usp_ValidateSN>

And here’s the successful output:

Stay tuned for part 2!

Last week month (sorry about that!), I wrote a post about using MABS to access a LoB system (in the example, SQL Server) behind several layers of firewalls (here).

We looked at the following tasks

1. Creating the BizTalk services
2. Setting up BizTalk Adapter Services in a local (or IaaS) environment to run a stored procedure in SQL Server
3. Creating a sample table and stored procedure
4. Creating a ServiceBus namespace with ACS
5. Create the Relay to the LOB system
6. Creating an EAI bridge to access the LoB system

This week, we’ll look at these tasks:

1. Testing and debugging the bridge with a Visual Studio add on
2. Writing a custom component to call the LoB adapter in a EAI Bridge Stage and parse the response
3. Having the component send an email notification using an Office 365 server

Using the MessageSender application from Microsoft is handy for a quick test (and for some sample code), but to do more serious work there’s a Visual Studio add on for debugging and sending test messages to EAI bridges called the BizTalk Service Explorer.  It’s available here: https://visualstudiogallery.msdn.microsoft.com/1f75a6a6-a54e-44eb-8b11-1b5ea8928754 (this can also be found through the Visual Studio Extensions manager).  This will add a new item to the Server Explorer window in Visual studio:

Right click on the highlighted item, and click “Add New BizTalk Service”.  Here you’ll have to provide the service URL and the ACS information about the service, as well as a “Friendly Name” used to display the service in the Server Explorer:

(if you can’t remember your ACS information, see my last post linked at the top of this entry; hopefully you saved it somewhere, otherwise you can get it from the Azure portal).

Once you enter that information, you will see your assemblies, bridges, certificates, schemas, and transforms deployed to MABS.  There are two particularly helpful functions for bridges: sending a test message and debugging the bridge:

Right click on SQLREQUEST (created last time), and click “Send Test Message…”  In this dialogue, you can load and send a test message and get the response (here, I got back a response of valid because asdf is a valid serial number in my local database):

Debugging this bridge is a similar process, but it’s not really doing a whole lot yet.  The next phase of this project is to call the LoB bridge from a one way bridge to make a routing decision.

There is no built in way to do that in MABS currently; a two way bridge’s output cannot be routed to a one way destination.  However, various stages of a bridge can all be extended with custom .NET code, and it’s possible to send a message to a bridge and get the response using a custom .NET module.  This is something like a BizTalk Pipeline Component.  I wrote a C# library to place in a new EAI bridge.  The library has two classes:

• MessageSender class (based off of Microsoft MessageSender application, but refactored to take string inputs instead of files; source included at end of this post)
• SetPropertiesInspector, which implements IMessageInspector.  This class reads some promoted properties, sends a message to the LoB system, parses the response, and promotes some new properties for routing.

Note that the http scheme is used, not https; this is to ensure that the ACS lookup will happen correctly.  Obviously, a “real” integration would have more sophisticated message parsing, perhaps using an XDocument to load and parse the data.

The bridge itself needs to be created and configured as well.  I dropped a new XmlOneWayBridge on to the MessageFlowItinerary design surface, and two Queues:

On the properties for the connector to the first queue, I added the filter condition “Valid=’True’”, and on the other, “Valid=’False’” (see the last post for more about filter conditions).  Valid is a property that gets promoted by the custom component.

For the Bridge configuration, I have the following: the message type is a new Request Schema that has a serial number and other information as well.  In the enrich stage, I added two XPath property definitions to promote the serial number node and another node:

1. Click Enrich
2. Open the Property Definitions
3. Add/Edit your properties
4. Enter the information; Xpath is to promote properties by XPath expression.

Finally, I set up the “On Exit Inspector”to refer to the Fully Qualified Name of my signed assembly:

1. Select the outter Enrich stage box
2. Open the properties for the On Exit Inspector
3. Provide the fully qualified name of the class in the assembly.  You can also pass parameters to the assembly in here if desired.

Deploy your solution (ensure that the custom assembly is referenced and set to Copy Local by the MABS project) as before.

Deployment make take a minute or two to complete, but now we can debug the bridge.  Right click the bridge in Server Explorer and instead of sending a test message, this time we’ll debug; each stage will have new information to show, including showing the message body and any properties getting promoted.  This example is really only working with the requestMessageExtractor stage, so I’ll show those.  First, the “Before requestMessageExtractor”: (two properties have been promoted by earlier stages):

The XPath promotions come next:

Then the Exit Inspector portion of this stage; our new properties are promoted, including “Valid” to true because asdf is a valid serial number in this case; I would not ordinarily promote the outgoing and return messages, but for debugging it’s helpful:

And that’s that!  The custom component can do other things with the message or message data if desired.  This solution has the data going to a queue, but the data could also be sent via Exchange 365, like so:

There’s a pretty wide range of possibilities here!

Finally, here’s the MessageSender class:

using System;
using System.Collections.Specialized;
using System.Configuration;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Mime;
using System.ServiceModel.Channels;
using System.Text;
using System.Web;
using System.Xml;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Collections.Generic;

namespace LoBLookup
{

 /// <summary>
 /// MessageSender is used to send a message to a deployed bridge end point
 /// </summary>
 public class MessageSender
 {
 /// <summary>
 /// Send message bytes to the runtime adresss of the deployed bridge end point
 /// </summary>
 /// <param name="acsNamespace">ACS namespace</param>
 /// <param name="issuerName">Issuer name for the specified namespace</param>
 /// <param name="issuerKey">Issuer key for the specified namespace</param>
 /// <param name="runtimeAddress">Runtime address of the deployed bridge</param>
 /// <param name="msg">String of the message to be sent</param>
 /// <param name="contentType">Content type of the message</param>
 /// <returns></returns>
 public static string SendMessage(string acsNamespace, string issuerName, string issuerKey, string runtimeAddress, string msg, string contentType)
 {
 string runtimeToken;
 UriBuilder endpointToGetAcsTokenForBuilder = new UriBuilder(runtimeAddress);
 endpointToGetAcsTokenForBuilder.Scheme = Uri.UriSchemeHttp;
 endpointToGetAcsTokenForBuilder.Port = -1;
 runtimeToken = GetAccessControlToken(endpointToGetAcsTokenForBuilder.ToString(), issuerName, issuerKey, acsNamespace);
 return SendMessageToBridge(runtimeAddress, msg, runtimeToken, contentType);
 }

 /// <summary>
 /// Get the Access Control token for the Service Bus URI
 /// </summary>
 /// <param name="endpointUri">Represents the End Point URI</param>
 /// <param name="issuerName">Issuer name for the Service Bus URI</param>
 /// <param name="issuerKey">Issuer key for the Service Bus URI</param>
 /// <returns>Access Control token</returns>
 private static string GetAccessControlToken(string endpointUri, string issuerName, string issuerKey, string acsNamespace)
 {
 string acsAddress = GetAcsAddress(acsNamespace);
 return GetAcsToken(acsAddress, issuerName, issuerKey, endpointUri);
 }

 /// <summary>
 /// Get the ACS address from the ACS namespace
 /// </summary>
 /// <param name="acsNamespace">Represents ACS Namespace</param>
 /// <returns>ACS Address</returns>

 private static string GetAcsAddress(string acsNamespace)
 {
 //UriBuilder acsUri = new UriBuilder(Uri.UriSchemeHttps + "://" + acsNamespace + "." + "accesscontrol.windows.net");
 //return acsUri.ToString();
 return "https://" + acsNamespace + ".accesscontrol.windows.net:443/";
 }

 /// <summary>
 /// Gets the ACS token for the specified Service Bus URI
 /// </summary>
 /// <param name="acsAddress">Represents ACS address</param>
 /// <param name="issuerName">Issuer name for the specified Service Bus namespace</param>
 /// <param name="issuerKey">Issuer key for the specified Service Bus namespace</param>
 /// <param name="appliesToAddress">Represents Service Bus URI</param>
 /// <returns>ACS Token</returns>
 private static string GetAcsToken(string acsAddress, string issuerName, string issuerKey, string appliesToAddress)
 {
 HttpClient client = new HttpClient();

 HttpContent content = new FormUrlEncodedContent(new Dictionary<string, string>
 {
 {"wrap_name", issuerName},
 {"wrap_password", issuerKey},
 {"wrap_scope", appliesToAddress}
 });

 var message2 = client.PostAsync(acsAddress + "WRAPv0.9/", content).Result;
 string response = message2.Content.ReadAsStringAsync().Result;
 //string response = Encoding.UTF8.GetString(responseBytes);

 // Extract the SWT token and return it.
 return response
 .Split('&')
 .Single(value => value.StartsWith("wrap_access_token=", StringComparison.OrdinalIgnoreCase))
 .Split('=')[1];

 }
 

 /// <summary>
 /// Sends message
 /// </summary>
 /// <param name="address">Represents the runtime address of the bridge end point</param>
 /// <param name="msg">Represents the string of the message to be sent</param>
 /// <param name="token">Represents ACS token</param>
 /// <param name="contentType">Content type of the message</param>
 /// <returns>Success/Failure message of the Send operation</returns>
 private static string SendMessageToBridge(string address, string msg, string token, string contentType)
 {
 string response;
 //WebClient webClient = new WebClient();
 //webClient.Headers[HttpRequestHeader.Authorization] = "WRAP access_token=\"" + HttpUtility.UrlDecode(token) + "\"";
 //webClient.Headers["Content-Type"] = contentType;
 //byte[] uploadData = webClient.UploadData(address, "POST", messageBytes);
 //response = Encoding.UTF8.GetString(uploadData);
 HttpClient c = new HttpClient();

 c.DefaultRequestHeaders.Add("Authorization", "WRAP access_token=\"" + HttpUtility.UrlDecode(token) + "\""); //Authorization = new AuthenticationHeaderValue("WRAP access_token", HttpUtility.UrlDecode(token));
 //c.DefaultRequestHeaders.Add("Content-type", "application/xml");

 //string msg = Encoding.UTF8.GetString(messageBytes);
 HttpContent content = new StringContent(msg);
 content.Headers.ContentType = new MediaTypeHeaderValue("application/xml");

 var message2 = c.PostAsync(address, content).Result;
 response = message2.Content.ReadAsStringAsync().Result;
 return response;
 }

 }
}

Muenchian Grouping is a powerful technique for to allow grouping by common values among looping/repeating nodes in an XML document.  BizTalk does not have out of the box support for this, but it can be achieved by adding custom XSLT to a map.  Chris Romp wrote a post about this years ago that serves as an excellent example of the idea in BizTalk: http://blogs.msdn.com/b/chrisromp/archive/2008/07/31/muenchian-grouping-and-sorting-in-biztalk-maps.aspx.  The drawback of his method is that you lose all other Mapper functionality by using completely custom XSLT, and custom XSLT is more difficult to maintain than a BizTalk map.

Enter Sandro Periera’s phenomenal tutorial on Muenchian Grouping in BizTalk maps (https://code.msdn.microsoft.com/windowsdesktop/Muenchian-Grouping-and-790347d2).  His solution is particularly powerful because it allows you to maintain the functionality and simplicity of the BizTalk Mapping engine while extending it to allow for the Muenchian Grouping technique as well.  However, there is still a limitation to this approach; the XSLT functoids will still be responsible for any transformation of child nodes that are grouped.  That poses a problem if your grouping logic requires that a parent (or perhaps even a root) node gets grouped on the criteria and many child nodes must be appended to the proper parent.

I recently faced just this situation while working for a client.  The XML data coming in needed to be extensively transformed, and in particular, duplicate child nodes had to be converted to unique parent nodes, with the original parents being appended to the correct new unique node.  Custom XSLT is clearly required here, but a hybrid approach can be used to still allow a regular BizTalk map to transform the resultant data.

My keys look like the following: each duplicate node has a pair of elements that, when joined, make it unique (ContactID and ContactType):

 <xsl:key name="Contacts" match="Contact" use="concat(ContactID, '|', ContactType)"/>

I could then use a xsl:for-each loop to join these nodes as the new parent node; the rest of the map was pretty straightforward:

<Contacts>
  <xsl:for-each select="//Root/Node1/Node2/Contact[generate-id(.) = generate-id(key('contacts', concat(ContactID, '|', ContactType)))]">
    <Contact>
    ... parent nodes appended here ...
    </Contact>
  </xsl:for-each>
</Contacts>

To avoid having very complicated logic and needing to replicate custom functoids in that “…” part, I started considering ways to have two maps: one using custom XSLT and the second being a regular BizTalk map.

The most basic way to achieve this strategy would be to have two maps which are called in sequence from an orchestration.  The first map has the custom XSLT to do the Muenchian grouping, the second map is a regular BizTalk map that works from the output of the first map.  This would work, but not if you want to do mapping on a receive port (which the architecture called for), where only the first map would get triggered.  It also is a bad idea of the message sizes are large (which occasionally happens with this trading partner).  Another method would be the one discussed here (again by Chris Romp): http://blogs.msdn.com/b/chrisromp/archive/2008/08/06/stacking-maps-in-biztalk-server.aspx, but this would involve creating several dummy receive locations to achieve something that can be done fairly simply in a pipeline component.

My eventual solution was pretty simple: create a pipeline component with the custom XSLT as an embedded resource.  I decided to put it in the decode stage, but it could have gone in the XML Disassembler stage, or in a C# helper library that gets called by an orchestration.  Here’s the relevant code:

A quick explanation of the code: A VirtualStream is used to manipulate the incoming message; the embedded XSLT is loaded as a resource, and XslCompiledTransform is used to run transform the schema.  The new message is added to the Context’s ResourceTracker to ensure that the garbage collector will properly dispose of streams when the pipeline is done with them, and the message is passed on to the next stage of the pipeline component.

This approach does have a drawback.  If the trading partner changes the schema, there will now be several updates to make: the pipeline component’s XSLT, the map, and the inverted schema.  However, this change is not likely to occur often, and using this method means that future developers can continue to use the BizTalk mapper functionality (including some custom functoids) when mapping the partially transformed trading partner data to the internal canonical format.

And the upside is worth it.  The complexity of the XSLT grouping is handled in one place, and it will be easier to maintain changes to that separately from changes to the rest of the mapping logic.  Plus, new developers on the solution will be able to quickly see how the data is getting sent into the canonical, a task that is more difficult to understand when looking at raw XSLT.

So you design your strongly typed stored procedure to take table types from BizTalk and it’s running great with your test cases.  It works well through the unit testing, but then you start running larger jobs and suddenly SQL is choking on it.

Ever wish you could just run that SQL call directly in SSMS with those exact several thousand rows for the table type parameters, and step through it using the debugger?  Well, you can using SQL Server Profiler (and/or Server Traces).  I used this technique recently to help a client resolve a particularly thorny issue that came up when they tried to process some larger messages.

To walk through the process of doing this, I’ll use a database named BTSTrainDb with a stored procedure (dbo.usp_DemoTableTypeSP) that takes a user-defined Table Type (dbo.DemoTableType) as a parameter and then just selects * from it (echoing it back to the caller).

First, fire up SQL Server Profiler and create a new trace.  Uncheck all of the event boxes.  On the Events Selection tab, check off Show all events and Show all columns, and then select the RPC:Completed and the RPC:Starting event; it’s not a bad idea to uncheck the Text column for the RPC:Completed event (as this will repeat the data from the RPC:Starting event).

Next, you’ll want to add filters that will capture only calls to usp_DemoTableTypeSP on BTSTrainDb; here, I’m filtering on the database and object names – you would want a username filter as well if there were other users calling the procedure that you’d like to ignore:

Click OK and then click Run to start the trace.

This will work fine for RPCs using small amounts of data (like test cases with only a few records).  That can be helpful if you just need to get a quick template for running the stored procedure directly during testing, but once you start feeding in larger amounts of data you’ll see the message “Trace Skipped Records.”  The sample below shows both types of occurrences:

So, I have my small sample (super handy for quickly debugging the procedure when I make changes), but when I tried to send in a large sample the Profiler tried to avoid crashing its GUI by loading several megabytes of text data.  To resolve this, you’ll need to start a server side trace.  This is easily accomplished by exporting the trace you just made; click “File->Export->Script Trace Definition->For SQL Server 2005-SQL11…”.  Load the resulting file up in SSMS.  You’ll want to increase the @maxfilesize parameter  and change the filename to something SQL Server will be able to write cool; take a note of the TraceID that it creates as well, as you’ll probably want to stop or dispose of it later:

When I run the big file through again now, I get the whole procedure call in the c:\temp\DemoTrace.trc file.  I can stop the trace by running the following stored procedure (the first parameter is the TraceID from earlier, the second is the status value):

exec sp_trace_setstatus 2, 0

If I wanted to remove the trace entirely it’d be exec sp_trace_setstatus 2, 2; or to start it up again exec sp_trace_setstatus 2, 1.  See http://msdn.microsoft.com/en-us/library/ms176034.aspx for more information.  When I open up the DemoTrace.trc file in profiler, I can get the whole call:

(note the ID column here, which indicates how many rows there are – the file generated almost 165000 rows!)

You can export it by going to File->Export->Extract SQL Server Events->Extract Transact-SQL Events… and saving it as a .sql file.  I now have full debugging access to the SQL stored procedure with the exact data that’s been causing problems!

EDI files offer many advantages in business and health care messaging, and BizTalk offers a premier suite for handling EDI interchanges and agreements.  At the same time, EDI files are notoriously difficult to read, even for EDI experts and highly skilled Business Analysts. The T-Connect EDI Viewer is designed to address this gap.  It supports converting the HIPAA EDI 837 transaction sets (for Professional, Institutional, and Dental) as well as the HIPAA EDI 834 Enrollment transaction set to a human-readable and printable PDF format.

The T-Connect EDI Viewer offers a library and a pipeline component that can be used to convert EDI messages sent to BizTalk into a PDF version of their Paper Form equivalents.  This conversion could be done as part of an existing integration, or using archived/historical data on a case by case basis.  The PipelineComponent offers a very simple way of adding this functionality to an existing BizTalk application with little to no development effort.  The library offers a simple but powerful API for more fine grained control over the process for BizTalk application developers.

A fully functional 30-day trial is available here.

Keep reading to see a sample conversion!

To install the trial, simple start the MSI.  The User Installation Guide offers step by step support for each option during the installation process.

Once installed, you can test the application with any 837P, I, or D file.  It will consume the file and write PDF versions of the transactions to a specified location.  An 837P file that starts out as this:

ISA*00* *00* *ZZ*EDIConvTPsamp1 *ZZ*Tallansamp1 *120926*0642*U*00501*000000252*0*T*:~
GS*HC*XXXXX-CPS*Tallan*20120926*0642*252*X*005010X222A1~
ST*837*000000252*005010X222A1~
BHT*0019*00*000000252*20120926*0642*CH~
NM1*41*2*Mayo Clinic*****46*S32433~
PER*IC*Mayo Clinic*TE*9547487111*FX*9547487222~
NM1*40*2*Zirmed*****46*Zirmed~
HL*1**20*1~
NM1*85*1*John*Smith*J***XX*6565656565~
N3*5537 Some Street*APT ZZ~
N4*Big City*NY*000008888~
REF*EI*222222222~

....

is converted to this:

Does your organization have custom needs for this tool?  Tallan can provide custom solutions around this product, including:

• Use of your own in-house custom form
• Alternate or nonstandard coding/value meanings
• Providing additional information from the EDI transactions that isn’t normally captured on the paper form

Simply contact BizTalk@tallan.com!

Receiving an untyped message (a System.Xml.XmlDocument) in a BizTalk orchestration offers a lot of flexibility in design.  The same code can be used for many different message types, and schema changes don’t require reworking your ports and receive shapes.  However, you lose some of the advantages of working with strongly typed messages – your orchestration could very well get a malformed message that is invalid by structure or content, and you won’t know until further down the line when another component that does validate the message (such as a send port) throws an exception.  Many times this isn’t a concern, but I was recently working on a solution where the orchestration received untyped messages and needed to know whether the message would validate against its schema or not.  Other components in the integration that processed the message later were not open for modification, and since the message was being constructed by the orchestration I couldn’t just add the XML Validator component to the receive pipeline for the orchestration.

There are a few different ways to handle this scenario. XmlDocument provides a Validate() method that takes a callback function it sends warnings and errors to.  This method requires that you add the correct schema for the message type.  This could be accomplished with a C# helper class:

This class could be invoked from an orchestration as follows (where tSchema is a System.Type varaible, strFQN is the fully qualified name of the library, and varXmlDoc is the System.Xml.XmlDocument):

varSchema = System.Type.GetType(strFQN);
blnIsValid = Utilities.XmlValidator.Validate(varXmlDoc, varSchema);

However, this method becomes fairly cumbersome.  You will have to keep track of the fully qualified names of the schemas for your particular message type, and you start facing painful design decisions about where and how to store that information.

A better approach involves calling a custom pipeline that has the default XML Validator component built in.  Adding in the XML assembler provides an additional benefit of properly promoting the standard context properties for a constructed message.  I designed the following pipeline:

The default properties were fine here.  By leaving the Document schemas property empty, it will perform the default schema resolution (using the BTS.MessageType property) to automatically find the deployed schema for the message.  This pipeline is in an assembly named Tallan.BizTalk.Pipelines, with the class name XmlValidatorPipeline.  Then, the following code can be used in a Construct Message or Expression shape in an Atomic scope:

// the default XMLRecieve pipeline will promote the properties, but won't do XSD validation on the document
// varXMLReceivePipelineType = System.Type.GetType("Microsoft.BizTalk.DefaultPipelines.XMLReceive, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35");
// varXMLReceivePipelineType is a System.Type; varRecievePipelineOutputMessages is a Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages
varXMLReceivePipelineType = System.Type.GetType("Tallan.BizTalk.Pipelines.XmlValidatorPipeline, Tallan.BizTalk.Pipelines, Version=1.0.0.0, Culture=neutral, PublicKeyToken=<PKT>");
varReceivePipelineOutputMessages = Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(varXMLReceivePipelineType, msgXml);
varReceivePipelineOutputMessages.MoveNext();
varReceivePipelineOutputMessages.GetCurrent(msgXml);

Now, the pipeline will take care of throwing an exception if the document is invalid, and the schema will be automatically resolved based on the message type!  If you’re not familiar with calling pipelines from orchestrations, check out the MSDN article here: https://msdn.microsoft.com/en-us/library/aa562035.aspx

Consider the following scenario: you have a tested BizTalk Orchestration that effectively takes a parameter (say, a message identifier) and returns the message from a database for consumption.  Now you want to expose a WCF Service for consumers of that orchestration to use.  While you could design a SOAP based interface, a simple RESTful interface that accepts a GET verb would do the trick more elegantly (and in a way that would be much easier for clients to work with). It also may meet a requirement when you already have clients that expect a RESTful interface for data retrieval.

There’s plenty out there on handling REST GET requests using BizTalk Orchestrations as a consumer:

• https://msdn.microsoft.com/en-us/library/jj248685.aspx
• http://www.quicklearn.com/blog/2013/08/16/biztalk-server-2013-support-for-restful-services-part-15/
• https://seroter.wordpress.com/2012/11/19/exploring-rest-capabilities-of-biztalk-server-2013-part-2-consuming-rest-endpoints/

Unfortunately, the same can’t quite be said about handling REST GET requests as a provider; there are resources:

• https://msdn.microsoft.com/en-us/library/jj572859.aspx
• https://seroter.wordpress.com/2012/11/12/exploring-rest-capabilities-of-biztalk-server-2013-part-1-exposing-rest-endpoints/

…but this side of the equation tends to focus on POST and PUT verbs where there will be a message body.  The GET verb is not going to provide a message body (only parameters).

Using the above two articles should be enough to get a WCF webHttp service deployed and ready, and I’m not going to rehash every step of that process (MSDN and Richard Seroter provide excellent instructions including pictures and diagrams).

However, when you get to the configuration of the receive location, there is some additional work needed to properly handle GET requests:

In the Transport Type Configuration of the receive location, you’ll want the following HTTP Method and URL Mapping:

<BtsHttpUrlMapping>
  <Operation Name='OpName' Method='GET' Url='/OpName?param1={param1}&amp;id={id}' />
</BtsHttpUrlMapping>

Then, you can modify your variable mapping like so:

You’ll notice there’s a property namespace, which means you’ll need a property schema.  Create your property schema with those properties, set the Property Schema Base to MessageContextPropertyBase.

I then set up my orchestration to receive a System.Xml.XmlDocument, and had the receive shape filter on the BTS.Operation == “OpName” && BTS.ReceivePortName == “ReceivePortName”.  However, when I tried to read these properties from the context of the message it failed.  This is because the context properties are not automatically promoted by the adapter.  Looking at the pipeline, it made sense – by default the pipeline was a PassThruTransmit, which does no promotion.

I started working on a custom pipeline component to solve this problem, but then I realized I could still use the XML Disassembler if I set the “Allow Unrecognized Document” property to true:

Now, the assembler will let a message with an empty body (like from a GET request) pass through, but still do property promotion.

The last piece of this process is to add logic in your orchestration to test for these properties before trying to read them, like so:

strId = "";
if (SchemaProjectNameSpace.Id exists msg_Request)
{
  strId = msg_Request(SchemaProjectNameSpace.Id);
}

Viola!  I can now have this receive location process RESTful GET requests (that will always have empty bodies) and my orchestration can use the variables passed in like parameters.  I can even use the same pipeline and receive location to deal with other HTTP verbs by modifying the BtsHttpUrlMapping!

Just a quick one today.  I’ve been working on several BizTalk pipeline components lately, and had been using some code that’s been floating around here for a while – I’ve found them posted on other blogs/message boards as well, but the earliest posting I’ve found of it is from another Tallan blog post from 2006: http://blog.tallan.com/2006/11/22/custom-pipeline-components-part-2-archive-same-continued/.

The relevant code from that post is this:

private object ReadPropertyBag(Microsoft.BizTalk.Component.Interop.IPropertyBag pb,string propName)
{
  object val = null;
  try
  {
    pb.Read(propName, out val, 0);
  }
  catch (System.ArgumentException)
  {
    return val;
  }
  catch (System.Exception e)
  {
    throw new System.ApplicationException(e.Message);
  }
  return val;
}

This code works well enough, and isn’t terribly difficult to use (especially if you’re only dealing with a few properties).  Using it (from the same post), looks like this:

object val = null;
val = this.ReadPropertyBag(pb, “path”);
if ((val != null))
{
  this._path = ((string)(val));
}

This becomes a little tedious when dealing with multiple properties, and is difficult to edit if you have to refactor property names or values.  Instead, using generic types provides more flexibility.  We can also add a some logic to remove spaces so that a display name of the property can have spaces and still be used in the property bag (trying to save a property name with spaces causes a very unhelpful error; see http://geekswithblogs.net/michaelstephenson/archive/2013/07/17/153431.aspx for more details) :

private T ReadPropertyBag<T>(Microsoft.BizTalk.Component.Interop.IPropertyBag pb,string propName)
{
 object val = null;
 try
 {
   pb.Read(propName.Replace(" ", ""), out val, 0);
   return (T)val;
 }
 catch (System.ArgumentException)
 {
   return default(T);
 }
 catch (System.InvalidCastException)
 {
   // Optional: This would be the place to provide some event log warning that the property propName couldn't be parsed
   return default(T);
 }
 catch (System.Exception e)
 {
   throw new System.ApplicationException(e.Message);
 }
}

private void WritePropertyBag(Microsoft.BizTalk.Component.Interop.IPropertyBag pb, string propName, object value)
{
  try
  {
    pb.Write(propName.Replace(" ", ""), ref val)
  }
  catch (Exception e)
  {
    // provide logging functionality here
    throw;
  }
}

and now, loading and saving the property bag becomes quite a bit clearer and easier to maintain:

const string PATH_PROP_NAME = "File Path";
const string ENABLED_PROP_NAME = "Enabled";

[DisplayName(PATH_PROP_NAME)] // the properties GUI will display 'File Name'
public string PathName { get; set; }
[DisplayName(ENABLED_PROP_NAME)]
public bool IsEnabled { get; set; }
...

public void IPersistPropertyBag.Load(IPropertyBag pb, int errorLog)
{
  string PathName = this.ReadPropertyBag<string>(pb, PATH_PROP_NAME); // the property will be read as 'FileName'
  bool bVal = this.ReadPropertyBag<bool>(pb, ENABLED_PROP_NAME);
}

public void IPersistPropertyBag.Save(IPropertyBag pb, bool clearDirty, bool saveAllProperties)
{
  WritePropertyBag(pb, PATH_PROP_NAME, PathName);
  WritePropertyBag(pb, ENABLED_PROP_NAME, IsEnabled);
}

The BizTalk mapper is a phenomenal tool and part of what sets BizTalk apart as an enterprise middleware platform.  Developers with little knowledge of XSLT can rapidly and accurately develop complicated transformations, and like all other aspects of the BizTalk engine it can be extended and heavily customized when necessary.  This more than makes up for any limitations in the mapping engine.  At the same time, sometimes the designer interface goes wrong and it’s difficult to figure out why: a functoid that has placeholders on a multipage map, or the desire to delete all functoids of a specific type from a specific page.  On smaller maps, this usually isn’t a problem, but when dealing with larger documents (like HIPAA 837 claim or 834 enrollment files), understanding the file structure of .BTMs can be a huge time saver.  It’s also possible to manually correct changes in your schema that would otherwise break the map (and perhaps cause a significant loss of development time and effort).  For demo purposes, I’ll use a fairly simple map with just a few links, using the 837I schema.

This map has two pages, makes use of a couple functoids, and only has a few links.  Ordinarily it wouldn’t be very difficult to refactor if needed, but in real life cases maps for this file type can have dozens of pages and thousands of links.  If we right click the map in the solution explorer, we get the option to open it with the XML editor:

With the exception of scripting functoid values, the map is saved as one long line of text.

You can easily fix that by clicking “Format document” or Ctrl+K, Ctrl+D.  (If you wish to edit the .btm file in another editor such as Notepad++, you’ll have to change the encoding attribute to “utf-8″ before XML Tools will pretty print it.)

Once formatted properly, the document is actually fairly easy to work with.

The mapsource root node has a few attributes that can be customized and are already well documented here: https://msdn.microsoft.com/en-us/library/aa561485.aspx.  Some of these attributes can only be changed by directly editing the map file.

The SrcTree and TrgTree nodes specify the source and target schemas.  For this demo, my map and schema are in the same project – but ordinarily they’d be compiled into separate projects/assemblies.  If a fully qualified name or file name of a map changes, it can be easily fixed by editing this node.  Note that the mapper will allow you to replace a schema (and will prompt you to do so if a FQN or file name changes), but it sometimes has mixed results on preserving links and nodes.

ScriptTypePrecedence allows you to enable or disable Scripting functoid options.  These parameters can be controlled in the properties window in VisualStudio as well, and don’t usually have to be changed.

TreeValues allows you to see/edit default values for testing (set on the source schema) or constants (set on the destination schema).  This is immensely useful, as there is no obvious visual key about default values set in the mapper.  It’s easy to forget where you set a default value if you use this feature, but using this method can quickly identify, clear, or change those values.  The format is:

<Value value="123" Query="/*[local-name()='&lt;Schema&gt;']/*[local-name()='X12_00501_837_I']/*[local-name()='ST']/*[local-name()='ST01_TransactionSetIdentifierCode']" />

The value attribute sets the value, and the Query attribute specifies an XPath to the node to set. Note, however, that the XPath has an additional pseudo-element the mapper uses: <Schema> (encoded as &lt;Schema%gt;). This must be present in any XPaths used in the map.

Finally, we come to the Pages node.  Under the Pages node, there are Page nodes, which have name attributes.  These correspond to the visible pages in BizTalk.  Each Page Element has two children: Links and Functoids.

Links contain Link elements.  Each Link element has three attributes: LinkID, LinkFrom, and LinkTo.

• The LinkID is unique among other LinkIDs throughout the entire file, so if you manually add or change any you must update this LinkID.  The LinkID is also used by functoid elements to reference their inputs and outputs.
• The LinkFrom is either a number (in which case it corresponds to a FunctoidID) or an XPath (with the <Schema> prefix).  This is the source of the link.
• The LinkTo is either a number (in which case it corresponds to a FunctoidID) or an XPath.

Functoids contain Functoid Elements, which have four attributes: FunctoidID, Functoid-FID, X-Cell, and Y-Cell:

• FunctoidID is similar to LinkID, but for functoids.  They are kept unique among other FunctoidIDs throughout the file.  This is what a LinkFrom or LinkTo attribute refers to when it’s numeric in a Link element.
• Functoid-FID is an integer that specifies what kind of functoid it is.  In the screen shot, 424 is looping.  I have not fully documented what each one does, but it’s possible to figure out what functoid you’re looking at from the Link values coming into and out of it and their XPaths.  Fromt his you can derive the meaning of the Functoid-FID value
• X-Cell: the X position of the functoid (horizontal) on the grid
• Y-Cell: The Y Position; these values should not be altered in a way that would cause two functoids to occupy the same cell for obvious reasons.

Functoid elements have up to two children: Input-Parameters and ScripterCode (for Scripting Functoids)

Input-Parameters has Parameter elements one or more children.  Parameter nodes have four attributes: Type, value, linkIndex, Guid.

• Type: either “link” or “constant”.
• Value: if Type is “link”, then the value refers to a LinkID in the map; if the Type is “constant”, then Value is a constant value parameter for the functoid.
• linkIndex: numerical index for the parameter/value.  0 based.
• Guid: a GUID used by the mapper.  These can be changed without breaking the map, but they must be valid GUIDs.  At the end of this post, I’ll include a PowerShell script that can regenerate GUIDs in a file, which is handy if you copy and paste functoids.

ScripterCode nodes contain a child called Script with a Language attribute, which specifies which language is used for this scripting functoid:

In the CDATA node, there is the code for my scripting functoid.  Again, this is immensely useful when using inline Script functoids: by directly editing the file, you could search for code in these functoids and even edit that code without having to find the exact scripting functoid.  You can also manually edit external assembly script functoids.    For external assemblies, the language is “ExternalAssembly”  and three additional attributes are used:

• Assembly: the FQN of the Assembly
• Class: The namespace qualified name of the class in the assembly
• Function: the name of the function

So when else is this useful?

• Lets say you (or another developer) refactors the schema for a map you’ve been working on.  While this shouldn’t happen, it certainly does (sometimes as a direct result of finding out how challenging it is to map that particular structure!).  While this could potentially break your entire map, it doesn’t have to this way.  You can manually add the part of the XPath that would be missing to relevant LinkTo or LinkFrom nodes.
• You get a horribly unhelpful error message that Functoid X has placeholders and that functoid occurs on many pages in your map.  Look for functoids with that FID that don’t have the right number of Parameter nodes (too many or too few), or that have a GUID of 00000000-0000-0000-0000-000000000000 (sometimes this happens when the map gets corrupted).
• You need to quickly find or change a constant value, or a value in a scripting functoid
• You need to map a nearly identical structure in another part of the schema (such as the Loop1 loops in the 837 schemas): I was able to save a lot of time on this by copying and pasting entire pages in the text and using search/replace (sometimes with regexes) to update the XPaths on the new pages.

Finally, here’s a PowerShell script to regenerate GUIDs; I wish I could take credit for this, but I only slightly modified a version that was found on StackOverflow (http://stackoverflow.com/questions/2201740/replacing-all-guids-in-a-file-with-new-guids-from-the-command-line):

## GuidSwap.ps1
##
## Reads a file, finds any GUIDs in the file, and swaps them for a NewGUID
##
param (
[string]$inputFilename = $( Read-Host "Enter input filename" ),
[string]$outputFilename = $( Read-Host "Enter output filename")
)

$text = [string]::join([environment]::newline, (get-content -path $inputFilename))

$sbNew = new-object system.text.stringBuilder

$pattern = "[a-fA-F0-9]{8}-([a-fA-F0-9]{4}-){3}[a-fA-F0-9]{12}"

$lastStart = 0
$null = ([regex]::matches($text, $pattern) | %{
$sbNew.Append($text.Substring($lastStart, $_.Index - $lastStart))
$guid = [system.guid]::newguid()
$sbNew.Append($guid)
$lastStart = $_.Index + $_.Length
})
$null = $sbNew.Append($text.Substring($lastStart))

$sbNew.ToString() | out-file -encoding utf8 $outputFilename

Write-Output "Done"

Note that it uses utf8 encoding even though the file says it’s utf-16 – because the file actually has a BOM indicating it’s utf8.

Updated: part 2

Application integration is challenging in that it requires a wide breadth of knowledge: server and network architecture, object oriented programming and design patterns, messaging and communication patterns, workflow design, message translation, database design and programming, business rule processing, data validation and quality handling… the list goes on and on.  On top of that, it’s important to be familiar with the data you’re working with from various sources and feeds to be able to effectively transform it.

EDI presents a whole host of challenges, and the 837 Professional, Institutional, and Dental transactions are among the most challenging to grapple with.  This will be the first in a series covering a high level introduction and overview of the 837 Claim format in the BizTalk HIPAA EDI schemas.  It is not quite going to the level of a competent EDI Analyst, but should give an integration developer enough to make a good go at understanding the data.  I’ll focus on Professional claims, but much of this post would also apply to Institutional and Dental claims; and in this post, we’ll be starting closer to the raw EDI data.

The 837P is a Professional Claim file.  It is what a doctors office or similar entity would send your insurance company to describe what doctors saw you, what diagnoses and procedures were made, and how much was charged.  Like other EDI transactions, it’s organized into “loops” and “segments,” and has certain identifiers to indicate what kind of transaction it is.

To identify an 837P, you’d look to the GS08, ST01, and ST03 values.  Here is a sample of the first three lines of an 837P file with those fields in bold:

ISA*00* *00* *ZZ*SenderParty *ZZ*ReceiverParty *131212*0703*^*00501*201334690*0*P*:~
GS*HC*999999*999999*20151112*0703*2013690*X*005010X222A1~
ST*837*20110*005010X222A1~

ST01 tells us this is an 837 claim file; GS08 and ST03 specify what kind it is.  As you can see, GS08 and ST03 should match each other.  The valid values for an 837P (5010 version) are 00501X222, 00501X222A1, and 005010X222A2; the “A” number specifies which errata version of the transaction is being used. In BizTalk, only the ST segment will be left in the schema – the GS and ISA values are promoted into the message context.

A BHT segment indicates the beginning of the transaction, mainly with some fixed values.

Then we have what BizTalk will call the 1000A and B loops.  This contains the submitter and receiver information, including names, addresses, contact information, and identifiers.  The submitter is the group or organization sending the claim, the receiver is the entity receiving it.

Next is the 2000A loop.  This contains information about the billing provider: name, IDs, contact information, and so on.  It is possible to have multiple billing providers in the same claim file, and the file represents the beginning of the 2000A loop with an HL segment like so:

HL*1**20*1~

HL01 says which HL segment this is in the file (the first will always be a 2000A).  HL02 is like a foreign key to HL01  - since the 2000A has no parent, it’s empty here.  HL03 says what kind of loop (20 is for 2000A), and HL04 says whether this loop has any child HL segments (2000As always do).  When generating claim files, it’s important to get these values right; each successive HL segment in the file should increment from its parent, and the key relationships between segments must be respected.

The 2000B loop, which contains Subscriber (insurance holder) information, will have a similar HL segment:

HL*2*1*22*0~

This HL segment is saying: I am the second HL segment in the file, my parent is #1, I’m a 2000B loop, and I have no HL children.  A 2000B loop will have a 2000C child if the subscriber is not the patient; for example: my child is a patient and I am the insurance holder; my information is found in the 2000B segment, and my child’s information would be found in the 2000C segment.  The 2000B loop has an important segment that really determines whether or not the patient and subscriber at the same person: the SBR segment

SBR*P*18*******CI~

SBR01 is saying whether this is the primary insurance (or S for secondary, T for tertiary, etc).  SBR02 says the subscriber is the patient when it’s 18; it’s left blank otherwise, and a different code is used to indicate patient/subscriber relationships.  SBR09 is a claim filing indicator code, here indicating commercial insurance usage.  Checking whether SBR02 = 18 is a safe way to determine where you’ll find the claim and service information.

Under the 2000B  (or 2000C for non-subscriber patients), there will be 2300 (Claim) and 2400 (Service Line) loops.  The claim loops will describe diagnoses, what doctor(s) saw the patient in what capacity, what facility(ies) the patient was at, how much the claim is for, a claim ID, and so on.  They always start with a CLM segment, which has an ID (CLM01), total amount (CLM02), information about what kind of service/loctaion (CLM05), and other information about patient and physician consent:

CLM*54321*250***12:B:1*Y*A*Y*Y~

Service lines are used to break down individual services, and give more details if other doctors/providers were involved.  They start with an LX segment that increments sequentially for each service line:

LX*1~

So if I go to my doctor’s office and have an exam, blood work, and some other procedure that toal $250, that $250 would be in the claim line.  In the service lines, there’d be a break down ($75 for blood work, $75 for another procedure, $100 for the exam).  Those service lines would indicate whether my primary doctor or some other doctor did various procedures, and also provided code information indicating what procedures were run.  If it was actually a visit for my child, all of that information would come under a 2000C loop instead.

Some common segements you’ll see in these loops are:

NM1: Contains name and ID information.

• NM101 will tell you what kind of NM loop (more on that next post).
• NM102 will specify whether this is a person (1) or business (2).
• NM103 will be a last or organization name
• NM104 will be a first name
• NM105 will be a middle initial/name
• NM106 will be a prefix
• NM107 will be a suffix
• NM108 will be an ID qualifier (for example, ‘EI’ for federal tax ID; more next post)
• NM109 will be the ID

N3 and N4: Address information

• N301: Street 1
• N302: Street 2
• N401: City
• N402: State
• N403: Zip

DMG segments contain demographic information

• DMG01: Date qualifier
• DMG02: Date of birth
• DMG03: Gender code

PER Segments contain contact information (telephone, fax, email, etc.)

REF segments contain a qualifier that specifies what kind of REF (REF01) and usually a numerical ID based on the qualifier.

HI segments contain diagnosis information specific to a qualifier.

SV1/2 segments contain service line specific information.

DTP segments contian an identifier, a date qualifier, and a date or timespan describing some part of an event.

Next post: Using the BizTalk EDI schemas to simplify and explain these mysterious qualifiers!

In the last post, I wrote about the overall structure of 837P claim messages.  837I and 837D messages have similar structures, though with some differences – for example, the 837I has many more diagnosis code options, and the 837D has more specific nodes for tooth identification, etc.  One important thing to note is that 837P, 837I, and 837D files are very similar, but not the same.  It’s not enough to simply change the version numbers in the GS and ST segments and then try to process one the same as the other!  If it were that easy, we wouldn’t need separate schemas for them in BizTalk!

BizTalk provides out of the box functionality to support these message types, with well developed functionality to translate the X12 EDI to XML and vice versa.  The schema nodes also have friendly names, translating something like this:

NM1*IL*1*Field*Dan****MI*123456789~

To this:

...
<MM1_SubscriberName>
  <NM101_EntityIdentifierCode>IL</NM1_EntityIdentifierCode>
  <NM102_EntityTypeQualifier>1</NM1_EntityTypeQualifier>
  <NM103_SubscriberLastName>Field</NM1_SubscirberLastName>
  ...

etc.

There are a few things to take note of here:

1. BizTalk will not allow completely invalid EDI to be transformed to XML.
2. The schemas avoid reuse of the same node name, even if it’s used in a different context.
3. BizTalk will not automatically populate qualifiers, even if there’s only one possible value and it’s required.

Invalid EDI and BizTalk

To point 1, this is both a strength and a weakness.  It’s a weakness if your trading partner(s) occasionally (or always!) send X12 data that is formally invalid (missing required segments, non-standard qualifier use, etc.).  However, it’s also a strength because you don’t want that bad data in your system anyway!  There are three methods to deal with bad X12 coming in:

1. Refuse to accept it.  Let your trading partner know in a 999 or 277CA that the claim has been rejected because it’s invalid.
2. Correct it before trying to convert it to XML.  At Tallan, we’ve written custom components for clients to do field replacements, check for and/or remove duplicate fields, etc. before the data hits the EDI to XML engine in BizTalk.
3. Create a custom version of the EDI schema that will validate against the document.

Number 1 is like a teacher who refuses to grade a paper until gross errors are corrected by the student: this teacher won’t be popular, but the students will learn a lot (if they do the work).  The drawback is that your trading partners not be able to do that work, for whatever reason, and your business may not be in a position to compel them to do so.

Number 2 and 3 both have the advantage of requiring less from a trading partner, but the disadvantage of changing the message coming from your partner.  This is like the teacher who accepts the major errors and just gives the student a so-so grade.  The danger is that trading partners may become more and more lax, or that the claim data you’re correcting is corrected using bad assumptions.  However, this option may be necessary if you and a trading partner have a non-standard usage of the X12 file that you both adhere to.

Of options 2 and 3, 2 is preferable as it is more flexible ad requires less delving into the internals of the BizTalk EDI engine; some schema modifications could work, but some might break the pipeline’s ability to serialize the raw X12 to XML.

Node naming in EDI Schemas

The BizTalk schemas give unique names to record nodes.  When a claim loop shows up under the 2000B loop (a claim for a subscriber who is a patient), it will be in the TS837_2300_Loop node.  If it shows up under the 2000C loop, it will be in the TS837_2300_Loop1 node.  Similarly, subloops are postfixed (such as NM1_SubLoop, NM1_SubLoop_2, NM1_SubLoop_3, etc.)  This offers the advantage of easily and quickly identifying whether a fragment of XML came from one loop or another; it also can give you an idea of how many times that loop occurred elsewhere in the document.  However, the numbers can be confusing, and mean that designing Maps or writing XSLT for the schemas will have to take those XPaths into account.  There’s no easy way to select the claim number regardless of whether this file is for a patient or a subscriber – you will have to specify the XPaths (or source links in the Mapper) for both if you need both.

EDI Qualifiers

BizTalk does not automatically populate qualifiers when constructing or mapping a message.  It’s important that you map the qualifiers.  This makes sense – in many cases different qualifiers are possible, for example D8 would specify a CCYYMMDD date in a DTP segment, whereas RD8 would specify that the DTP segment is a CCYYMMDD-CCYYMMDD format.  EI would specify that an identifier is a Federal Tax ID, where as SN would specify that it is a social security number.

BizTalk does offer some help on this – qualifier fields that only accept certain values will have an enumeration on the node specifying that.

This is very helpful when trying to figure out what value to put in a NM101 field, or a DTP01 field.   In the XML fragment above, the NM101 qualifier ‘IL’ specifies that this is the NM1 segment for a Subscriber Name – and that would be the qualifier you’d find in the BizTalk schema designer.  To get the full specification of all allowed qualifiers and their meanings, you’d have to look at the WPC consolidated guide for the 837 claim you’re working with; these are not freely available, but certainly worth the purchase if you’ll be working with EDI exensively.  BizTalk will, however, give you at least a hint about what values are valid – and frequently their meanings can be inferred.

In the end, a BizTalk developer will have to work with an EDI specialist when attempting to construct a file.  However, BizTalk drastically reduces the time it takes to do so by ensuring structural integrity and providing many hints and helps when a file is invalid.  This means that the EDI analyst and the BizTalk developer can be more focused on meeting business needs than worrying about structural validity.

Finally, I highly recommend this post from Senthil Muthiah – it’s not BizTalk specific, but it does provide some more details on some of the more commonly used segments of the 837P:

http://emrpms.blogspot.in/2012/09/edi-5010-documentation-837-health-care.html

The short version of this story is: Before uploading a VHD to Azure, make sure you’ve enabled remote RDP (installing Azure PowerShell is a good idea too).  But if you forget (and your VM is running Windows Server 2012), it can be fixed without having to redo a massive upload.

We recently did a demo for a prospective client, and wanted to give that client access to the VM the demo was on to further explore and tinker with the solution.  Azure IaaS makes perfect sense here – upload the VM right to Azure and let the client have access.  The VM can be spun up on demand, and can be removed when it’s no longer needed – and it won’t require any special permissions or network/firewall rules on either end.

There are several helpful guides to prepping and uploading a VHD to Azure:

• http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-create-upload-vhd-windows-server/
• http://michaelwasham.com/windows-azure-powershell-reference-guide/uploading-and-downloading-vhds-to-windows-azure/
• http://blogs.technet.com/b/canitpro/archive/2014/05/08/step-by-step-upload-your-own-vhd-to-add-to-the-gallery-in-azure.aspx

The upload process went fairly smoothly – we let it run over night and in the morning our VHD was there and ready to be attached and booted.

For better or worse (maybe just because it should be so obvious), none of the guides mention the fact that remote RDP should be enabled before uploading your VHD (and I forgot to do so before starting the upload).    The process to do that is fairly trivial: run SystemPropertiesRemote and then use the GUI to configure your settings.

There are ways to enable RDP on an Azure VM from the portal or using PowerShell – if you created the VM using the portal, or if you have Azure PowerShell installed before uploading the VHD.  Neither of these was true in my case.  However, Windows 2012 enabled RemotePowershell by default, and allows public access to remote PowerShell from the LocalSubnet by default.  This was our way in.

We created another VM on Azure on the same local subnet as the uploaded VM.  The Uploaded VM had an IP of 10.0.0.4, and the new VM was 10.0.0.5.  After logging into the new (RDP accessible) VM, and ran the following commands in PowerShell:

# Add IP to the trusted hosts
Set-Item WSMAN:\LocalHost\Client\TrustedHosts -value 10.0.0.4 -Force
# restart to update WinRM
Restart-Service WinRM

# Login information
$user = "Administrator"
$pwd = "MyPasswordForAdministrator"
$secPwd = ConvertTo-SecureString $pwd -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($user, $secPwd)

# enter the remote PowerShell Session
Enter-PSSession 10.0.0.4 -Credential $credential

# Prompt should change to [10.0.0.4] C:\users\Administrator\Documents >

# Allow remote RDP connections
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server'-name "fDenyTSConnections" -Value 0
# Allow through firewall
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
# Set up authentication for RDP
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp' -name "UserAuthentication" -Value 1

And then, we could successfully RDP into the remote machine. In all, this process took about an hour to work out – instead of another overnight upload!