Windows Azure BizTalk Services (WABS) provides capabilities for EAI and B2B in the cloud. This relative new service was made available for customers in November 2013. WABS is a cloud integration platform or integration platform as a service or IPaaS. Characteristic of IPAAS is that you build your integration on premise, deploy in the cloud, where it is hosted in a service (set of dedicated hardware). The service is offered through a subscription model for operating the service in the cloud (for BizTalk Services this means Windows Azure hence the name Windows Azure BizTalk Services).
Currently there are quite some players offering that service like Dell, Attunity, MuleSoft, TIBCO, and so on. Microsoft is currently pushing the service hard to catch up with some of these players and try to quicken/hasten the maturity of the product. Therefore, Microsoft has committed to have a release cadence for new features every three months. The recent release was in February 2014, which added new features for WABS. These new features are:
- EDIFACT Protocol Support and X12 Schema Updates
- Pulling Messages from Service Bus Queues and Topics
- Service Bus Shared Access Signatures (SAS) support with Service Bus Queues and Topics
- BizTalk Adapter Services No Longer Needs SQL On Premises
- Backup and Restore Support
- Operation Log Support
This post will dive into the new feature of pulling messages from a Service Bus Queue. When a developer designs and build a bridge he/she can now choose two new sources i.e. Service Bus Queues and Topics. A Bridge with WABS is a workflow that processes a message pulled from a source and delivered to a destination (see picture below).
Tracing messages and Azure Service Bus monitoring across multiple Queues and Topic Subscriptions provide powerful toolsets and actionable insights to troubleshoot messaging issues.
Benefits of pulling messages from Service Bus queues and topics are:
- Durable messaging
- Leveraging the support for multiple protocols
Durable messaging
The ability to pull messages from Service Bus queues and topics increases the durability of a bridge solution. You can leverage the pub-sub mechanism of the topics and subscriptions. It is even possible to have multiple bridges pulling messages from different subscriptions and have them send to multiple destinations. A single bridge will not enable you to do that.
Leveraging multiple protocols
Another benefit is that you as a developer will have more input channels for the bridge besides FTP, and HTTP (REST). Indirectly through the Service Bus Queues you could have support for AMQP.
Scenario
To explore the new capability of pulling message from the Service Bus we will explore the following scenario. A payroll company wants to offer a service in Windows Azure BizTalk Services for organizations to send their payroll data for pay rolling. Organizations that like to outsource their pay rolling can connect to this service in Azure. Basically the organizations can send their payroll data to the service in any kind of format and structure they want. The payroll company will provide a bridge to provide connectivity to the source the organization pushes its data to. This can be FTP(S), HTTP or the Service Bus Queue or Topic (subscription). The data will be picked up from the source and processed to one unified format to be routed to a LOB system or database for a later pay roll run that the payroll company will do. Below you will find a diagram detailing the scenario.
I have used this scenario also in my talk during the BizTalk Summit 2014 in London discussing the manageability of WABS.
Building the solution
The solution will be a bridge for an organization that requires to push out its payroll data to the payroll company. In this case the data is pushed to a Service Bus Queue and the Bridge will pull the data from the queue, transform it to a format that enables a table operation in SQL Server on premise. To build this solution the following parts need to build, configured and/or created:
- a LOB Target that needs to be configured for access to SQL Server through a LOB Relay endpoint,
- generation of a schema for the table operation (insert operation),
- a mapping,
- a bridge design,
- connection of the source to the bridge and to the destination,
- configuration of the source, the bridge and the destination.
When solution is build it needs to be deployed to the BizTalk Service in the cloud.
Create a BizTalk Service Project
To build a WABS solution you need to create a BizTalk Service project: Open Visual Studio and on the File menu, point to New, and then click Project. Use the details in the following table to create the project and then click Ok:
Installed Templates: Click BizTalk Services, then click BizTalk Service.
Name: Specify a name.
Location: Set this to a location on your computer.
Create directory for solution: Select this if you want this solution to have a separate folder in Windows Explorer.
LOB Target/Relay
A part of the Windows Azure BizTalk Services is on-premise infrastructure. By downloading the WABS SDK you can set up that infrastructure. By infrastructure in this context is addition of WABS templates in Visual Studio 2012 and installation/configuration of the BizTalk Adapter Service. This service is basically a IIS wrapper around the adapter pack that provides access to on-premise LOB systems. The BizTalk Adapter Services can be viewed as a gateway to premise LOB systems that can be securely accessed indirectly through WABS. The concept of BizTalk Adapter Service is to ability to create kind of endpoints to operation on your LOB systems. You create a LOB Target, which is your on-premise Line-of-Business (LOB) system and the operations (like SELECT or INSERT) exposed to your client applications (a Bridge). During the configuration of a LOB Target you create a LOB Relay, which is a URL that provides a connection to the cloud using Service Bus Relays.
For installation of the BizTalk Adapter Service see TechNet Wiki article BizTalk Adapter Service Installation.
Creating a LOB Target/Relay
A LOB Target is created through Visual Studio. You have to perform the following steps to create the LOB Target:
- In the BizTalk Service project, open Server Explorer, right-click BizTalk Adapter Service and expand.
- You will need to authenticate to further expand to LOB Types.
- Provide credentials to access the BizTalk Adapter Service.
- Click Ok. You can now expand LOB Types, right-click SQL, and select Add SQL Target.
- You will see a Wizard pop-up and the first screen will be a welcome screen.
- Click Next and configuration process of LOB target will start.
- In the Connection parameters tab you specify the connection details of SQL Server and the credentials.
- Click Next.
- In the Operations tab you can select object, operations and add them to the selected operations.
- Click Next.
- In the Runtime Security tab you specify the way you like the authentication to LOB target i.e. SQL Server.
- Click Next.
- In the Deployment tab you specify LOB Relay. You can select an existing relay or specify a new one.
- To create a new LOB Relay, enter the following details: Service Bus Namespace
Specify the Service Bus namespace on which the LOB relay endpoint is created.
Service Bus
Issuer Name
Specify the issuer name for the Service Bus namespace
Service Bus
Issuer Secret
Specify the issuer secret for the Service Bus namespace
LOB Relay Path
Enter a name for the relay.
LOB Target Sub-path
Enter a sub-path to make this target unique.
Target runtime URL
This read-only property displays the URL where the relay is deployed on Service Bus. This is the path where you could send a message to be inserted into the on-premises SQL Server. In the payroll scenario, this is where the bridge routes the message. - Click Next.
- The summary tab will appear and you review your LOB Target/Relay setup.
- By click Create the LOB target/relay will be created. What basically will happen is that LOB target will be created in the BizTalk Adapter Service and is created as an application in IIS.
Generate schema for the insert operation
When a LOB Target is created you can generate the schema(s) for specified operation(s). The following steps describe how to generate the schema for the insert operation:
- In the BizTalk Service project, in the Server Explorer, right click the LOB target you created, and then click Add Schemas To Payrollservice….
- A dialog will appear, where you specify the credentials, folder name, …
- Click Ok.
- The schema(s) will be added to the BizTalk Service project.
Generate schema for request
In BizTalk Service project you can create the schema for the incoming request message containing the payroll data. The data is delivered per employee. The schema can be created as visualized by screenshot below.
Create Mapping from source schema to destination schema (XML Transform)
The following steps show how to transform the incoming request message (source) to a request message for LOB Target (destination):
- In Visual Studio, right click the your project, point to Add, and then click New Item.
- In the Add New Item dialog box, select Map, specify the map name as “.trfm”, and then click OK.
- In the Solution Explorer, double click the “.trfm” file to open the Transform. On the Transform surface, select the source schema to Request.xsd and the destination schema to xxxxx_PAYROLLSERVICE_payrolldata_TableOperation.dbo.Employee.xsd.
- Drag lines between the various elements in the source and destination schema like below:
Configure a one-way Bridge
The following steps describe how to configure the one-way bridge with a source and destination. You select the MessageFlowItinerary.bcs and:
- Drag and drop the LOB Target Entity from the BizTalk Adapter Service onto the design surface (destination).
- Drag and drop XML one-way bridge onto the design surface.
- Finally drag and drop Service Bus Queue source to the design surface (source).
- Double click the XML One-Way Bridge on the itinerary designer.
- On the XML One-Way Bridge design surface, within the Message Types box, click the add icon [+] to open the Message Type Picker dialog box.
- In the Message Type Picker dialog box, from the Available message types box, select the Request (PayRolldata) message type, click the right arrow icon to associate the request schema with the XML One-Way Bridge, and then click OK.
- Configure the bridge to use the Transform created earlier. Within the Transform stage, select the Xml Transform activity, and then from the Properties window click the ellipsis button (…) against the Maps property to open the Map Selection dialog box.
- Save the bridge configuration and go back to the Bridge Configuration designer surface
Configure the Service Bus Queue source
The source of the bridge needs to be configured (connectivity) to enable the bridge to receive or pull a message from. For the service bus queue you need to specify the connection string in the following format:
Endpoint=sb://<your service bus name>.servicebus.windows.net/;SharedSecretIssuer=<issuer name>;SharedSecretValue=<issuer secret>
Beside the connection string you specify the name (entity) of the source and name of the queue.
Configure the LOB Target Destination
Once you have dragged a LOB Target on the design surface you will see a <your lob target>.config appear. You can double click this file in the solution explorer. This file resembles a kind of binding file where you need to specify the credentials of your LOB Target Relay. These are service bus credentials. After specifying those in the file you can save it.
Connect the components on the bridge design surface
After configuration of the source, bridge and destination it is time to connect the source to the bridge and the bridge to the destination. The following steps describe how to connect the components and connection configuration from the XML One-way bridge to the LOB Target:
- From the Toolbox, select Connection, and drag-drop the mouse pointer from the Payroll Queue component (source) to the bridge component to connect the two.
- Select a Connection again from the Toolbox, and drag-drop the mouse pointer from the bridge component to the LOB Target/Relay component to connect the two.
- Set the filter condition on the connection between the bridge and the LOB Relay entity.
- Click the connection between XML One-Way Bridge and the LOB Relay entity.
- In the Properties window, click the ellipsis (…) button for Filter Condition.
- In the Route Filter Configuration dialog box, set the filter condition to Match All. This ensures that all the messages that are processed through the bridge are routed to the LOB entity.
- Click Ok.
- Set the Route action so that the outgoing message to the LOB application has the appropriate SOAP action header.
- Open Server Explorer and navigate to the SQL LOB Relay created earlier. Right click the relay, click Properties, and for the Operations property, copy the value of the first operation (insert). This value denotes the value of the SOAP action header that must be set on the message that is routed to the LOB Relay.
- On the Bridge Configuration surface, click the connection between XML One-Way Bridge and the LOB Relay entity.
- In the Properties window, click the ellipsis (…) button for Route Action. In the Route Actions dialog box, click Add to open the Add Route Action dialog box. In the Add Route Action dialog box, do the following:
- Under Property (Read From) section, select Expression, and then paste the value that you copied earlier.
- Under Destination (Write To) section, set the Type to SOAP and the Identifier to Action. The dialog box resembles the following:
- Click OK in the Add Route Action dialog box to add the route action. Click OK in the Route Actions dialog box and then click Save to save changes to an BizTalk Service project. The end-to-end message flow resembles the following:
- Save change to the project.
Build, deploy and Test the solution
You have finished creating the application. In this steps, you will build and deploy the application under your BizTalk Services subscription. Subsequently you will use the Service Bus Explorer to test your solution.
To build and deploy the solution
To build and deploy your solution you can perform the following steps:
- In Visual Studio, right click your bridge solution, and then click Build Solution.
- Once the build succeeds, right click your solution, and then click Deploy Solution.
- In the deployment window, the Deployment Endpoint is a read-only property and the value is derived from the BizTalk Service URL/Namespace set in the message flow surface. However, you must provide the ACS Namespace for BizTalk Services, Issuer Name, and Shared Secret.
- Click Deploy. The Visual Studio Output pane displays the deployment progress and result. The URL where the bridge is deployed is also displayed in the Output pane. For this scenario, the bridge is deployed at http://sltn2014.biztalk.windows.net.
Test the solution
To test the solution you can use the Windows Azure Service Bus Explorer. This tool can be used to sent a message to a service bus queue. The following steps will describe the way to test the solution:
- Open the Service Bus Explorer.
- Connect to the appropriate namespace within the service bus where the queue resides that the bridge will pull the messages from.
- Right click the queue and select send messages.
- Paste/Load a message payload in the Message Text pane.
- Click Start.
- Message will be sent to the queue.
- You can see the log displaying the result.
- In SQL Server you can view the Employee table to verify if the data is present in the table.
The addition of queues and topics as sources to WABS has extended the flexibility and ability to create durable cloud solutions. However, WABS will need more adapters (sources/destinations) to be able to compete with some of the other IPaaS vendors like MuleSoft, who offers a great deal of adapters. Hence more versatile integration capabilities.
This post demonstrated the use of pulling messages from a Service Bus Queue. This ability can add durability to your cloud integration solution using WABS. The walk through of the payroll scenario demonstrated the ease of creating a bridge solution. Most of the configuration is done through Visual Studio a main component currently to build, deploy and manage a bridge solution.
The dependency of Visual Studio for building WABS can be considered undesirable. Changes to the configuration of sources and destinations for a bridge has to be done in Visual Studio accordingly and then redeployed to the BizTalk Service hosting the bridge. Therefore, managing some parts of a bridge solution like the configuration of sources and destinations is preferable better off in a different tool like a management console/web application or within Windows Azure itself. Since this is still the first release after the go-live of WABS last year in November you can expect improvement of the service in many areas including the manageability.
Cheers,
Steef-Jan
