Asynchronous Logging using MSMQ in EL

The Logging and Instrumentation App Block in EL has a cool feature that harnesses MSMQ for asynchronous logging.

The advantages of asynchronous messaging based on MSMQ are multiple; Most importantly, I feel it is the decoupling of the message (event) source and the message handler that results in a higher system stability and reliability. In most solutions, the message source and the message handling logic usually resides in the same process, and the message handling logic will include some form of persistence like writing the messages to a database. Now, the problem comes when the database is down. The system can still receive the message, but cannot persist the message, and the messages are lost... With MSMQ, we add a layer of indirection by having MSMQ as an interim storage when the message is received. Then the permanent persistance will retrieve messages from MSMQ and write them to the database. So when the database is down, the messages are not lost as they can still be persisted to MSMQ when the database is back in operation.

Hence in my current project, I was intenting to use MSMQ to log SNMP traps and have another custom listener service to check (and subsequently process) the MSMQ for incoming traps. Stumbling upon this cool feature was like a great bonus, and I wasted no time in getting my hands dirty with this finding.

Digging into the details, I realised some finer caveats of this implementation that needs attention:

1. MSMQ Distributor Service is NOT installed by default. The MSMQ Distributor Service is a windows service that comes built into the app block. Its purpose is to check the MSMQ periodically for new messages. Then based on its configuration, the service will call the appropriate LogSink classes to persist the messages. However, note that the MSMQ Distributor Service is NOT installed by default when installing the EL. "Installutil.exe" that comes with .NET Framework SDK must be ran to install this service.

2. The MSMQ Distributor Service requires additional application-specific configuration. Even after the installing of the service, it must be configured to enable it to run. Example includes the queue in MSMQ that this service will check.

3. MSMQ's Custom Queue must be created manually. The custom queue in MSMQ must be created manually by the developer and deployment team. The EL installation process does not create the required queue in MSMQ.

4. Authentication for Custom Queue must be matching with Service's credentials. Permissions must be given to the account under which the service is running to access the custom queue. E.g. if the MSMQ Distributor Service is running under account 'NT Authority\Local Service', under the custom queue's security setting, the same account must be given 'Peek Message' and 'Receive Message' permissions. Otherwise the service will failed and stop.

5. MSMQ Distributor Service only supports ONE queue at a time. This limitation may mean that the service can only supports one application. My thoughts on working around this is to have this service use a generic queue, so that more than one application can use this same queue. The app block allows application-specific LogSinks to handle the messages in the applications context.

Well, most of these are mainly about design and deployment issues. If we are mindful about these and factored them into the deployment plan and installation setup projects, the benefits of a readily available asynchronous logging mechanism should outweight the above finer issues.