Currently I confronted with a dilemma concerning synchronous or asynchronous processing of batches. A customer I work for currently process batch in the back-end system. This customer is demanding a solution in which processing can be controlled and back-end systems are efficiently used. At the moment this not the case, systems are sometimes running idle and batch processing is done during non-working hours (17:00 – 8:00). Everything in these batches are processed at once and sometimes the batch windows is not sufficient to have everything processed, so this processed the next day. Systems are used during working hours by staff and processing during these hours interferes greatly with their work (long duration for data to pop-up their screen or transactions are not preformed).
Therefore they suggested a drop wise batch processing, where processing can be controlled and preformed during day as well on moments that systems are not too busy. I suggested a solution with BizTalk, that is heavily used inside this company. BizTalk acts as broker that sends messages to one or more message queue to divers back-end applications that can process batches. Not all at one but small parts of each batch during day and night. Size of parts (as messages is controlled dynamically) , see picture below where I used some notation form Enterprise Integration Pattern book by Greogor Hophe to visualize the solution.
This asynchronous pattern, where messages on the queue are written by BizTalk through MSMQ Adapter and these queue are being read by a windows service that delivers messages directly to the application (a legacy app written in mainframe language wrapped by .NET code). Results are returned in form of a message by the windows service that writes messages to the queue (general), where BizTalk reads them and by interpreting the message translating it to a result message for verifying processing or error message to be handled by a different BizTalk process (there for routing after the translation). I think this is a very robust and solid solution, though there is also a desire or demand for a more direct approach by more synchronous way of doing processing using web services on top off the applications (this pattern is also described in the book of Enterprise Integration Patterns: Synchronous Implementation using Web Services). In this case BizTalk would play a lesser role or should not be part of it all. Batches or parts of them can be delivered directly through web services to back-end applications. I do have some worries through this approach concerning time-outs and how to pin-point bottlenecks. It will be challenge to choose the right solution or convince the customer to go for asynchronous solution. What do you think?
Technorati:
BizTalk
BizTalk Patterns
BizTalk Architecture
