Tag: NServiceBus

NServiceBus-ServiceMatrix Saga To Saga Request/Response Pattern

Summary

This document explains how to setup a SAGA to SAGA request Response pattern. Bus.Reply is used, as ReplyToOriginator is not supported. We will simulate a service receiving an order, then sending it to an Order Service, which then has a request/response pattern to process payment.

We will create 3 endpoints, OrderSender , OrderService, PaymentService
We will configure an order to send an initial Order from OrderSender to OrderSaga
We will then configure OrderSaga to send a Request/Response to PaymentSaga
Note, I have message correlation as well. This is needed for ReplyToOriginator to work between Saga’s from a timeout.

Saga to Saga Request/Response supports Bus.Reply, however do not use it in TimeOut handlers, as it will try reply to the timeout queue.

ReplyToOriginator also works, when you need to call the originating Saga, however there were issues on the bug report https://github.com/Particular/NServiceBus/issues/2018. However you can get it working by doing two things:

Ensure the Calling Saga outlives the Called Saga (Create a Long TimeOut that marks Saga completed in Calling, Create a shorter timeout in the Called Saga that MarksItComplete)
Add this code to the ProcessOrderHandlerConfigureHowToFindSaga.cs

You can download the source code at:

git@bitbucket.org:romiko/sagatosagarequestresponse.git

https://romiko@bitbucket.org/romiko/sagatosagarequestresponse.git

Just enabled NUGET package restore J

TimeOuts and responding to the original calling (originator) saga

Never use Bus.Reply within a TimeOut handle in the Saga, as it will try to reply to the timeout queue, as Bus.Reply will always respond to the source of the LAST incoming message.

To get ReplyToOriginator working between Saga’s, you need to:

Ensure the called Saga (ProcessOrder) lives LONGER than the called (Payment), by using timeouts in both sagas
You need to add a correlation

This is the message pattern with timeouts and a polling pattern, which you can run indefinitely if ever needed.

Create three endpoints

Click New endpoint
Create an OrderReceiver, as an NServiceBus Host. Do the same for OrderSaga and PaymentSaga
Your canvas will look like this

Send a message from OrderReceiver to OrderSaga

So now we will simulate a service (OrderReceiver) that receives orders on a backend system and then sends them to our OrderSaga for long running transaction processing

Click the OrderReceiver and click “Send Command”
Set the Service name to Orders (Domain) and the command ProcessOrder

Your canvas should look like this
Click the undeployed component and select Deploy
Select OrderSaga as the destination and click Done

Your canvas should look like this, with a bit of interior design J
Edit the ProcessOrder message and add the following properties
OrderId
Amount
Open the ProcessOrderSender.cs file under the Orders folder, we will configure it to send 3 orders. We will implement IWantToRunWhenBusStartsAndStops

Note that I am not in the infrastructure folder, as this is generated code.

Build the solution

Configure the OrderSaga as a Saga and Message Correlation

Great, so now we have the minimum need to set the OrderSaga endpoint to a real Saga, as a SAGA MUST have a message to handle. In this case ProcessOrder.

Click the ProcessOrderHandler and click “Convert To Saga”
This will open the ProcessOrderHandlerConfigureHowToFindSaga.cs file. Build the solution, so that partial classes are generated.
We want to correlate order messages based on the orderId to the correct Saga instance. So here we will set the properties on how to find it. Add the following code:
Open the file ProcessOrderHandlerSagaData.cs and add the OrderId, set the property to Unique, as this is how the Saga will correlate messages to the correct instance.

Excellent, so now we have correlation established between the OrderReceiver and the OrderSaga. So if ever the Saga receives order updates for the same order, the infrastructure will no which instance to send the processorder command to.

Configure Saga To Saga Command

Here we will configure the OrderSaga to send a message to the PaymentSaga, then we will update the PaymentSaga to become a Saga.

Click the ProcessOrderHandler and click SendCommand
Name the command ProcessOrderPayment
Click Copy to Clipboard. This will then open the ProcessOrderHandler.cs file. Paste the code.
Open the Canvas, it should look like this
Click the ProcessOrderPaymentHandler, and Click Deploy.
Select the Payment Saga, as this will handle the ProcessOrderPayment Request.
Your canvas will look like this. BUILD SOLUTION
Let’s CONVERT PaymentSaga endpoint to a Saga, as we have the minimum needed to do this!
WARNING: NEVER convert an endpoint to a saga unless it has at least one message handler, else it cannot implement IAmStartedByMessages interface. You would have to wire it up manually, since the Infrastructure code generator will not know how.
Click ProcessOrderPaymentHandler and click Convert to Saga…
This will open the ProcessOrderPaymentHandlerConfigureHowToFindSaga
Build the solution, to auto generate the Saga partial classes and infrastructure
We want the payment instance to correlate to the correct order Id, so add this:

Build the solution! We added properties so ConfigureHowToFindSaga will compile J

Configure Saga To Saga Response and Bus.Reply

Open the ServiceMAtrix Canvas, confirm your canvas looks like this

notice the icon for saga’s has a circle in it with a square.
Click the ProcessOrderPaymentHandler in the payment Saga and click Reply with Message…
Click Ok
Copy the code to Clipboard
Click the Copy To Clipboard, note the mouse pointer will show as busy, however you can still click copy to clipboard.
This will open the ProcessOrderPaymentHandler.cs, paste the code here. Put in a Thread.Sleep to simulate a credit card payment.
Your canvas will look like this now
Add the following code to ProcessOrderHandler.cs file
Build the solution

Testing the solution

Following the following in Order, so the msmq’s are created in the correct order, to avoid race conditions on the first time is starts.

Start the SagaToSagaRequestResponse.PaymentSaga
Start the SagaToSagaRequestResponse.OrderSaga
Start the SagaToSagaRequestResponse.OrderReceiver
You should see

And

In ServiceInsight we see:

Source Code

You can download the source code at:

git@bitbucket.org:romiko/sagatosagarequestresponse.git

https://romiko@bitbucket.org/romiko/sagatosagarequestresponse.git

Just enabled NUGET package restore J

NServiceBus – Some Benefits

Hi,

I am not sure why, but in many organisation, there is allot of unnecessary complexity when looking at source code. From Programmers using try/catch blocks all over the show to unnecessary synchronous service calls with multiple threads, causing deadlocks on SQL Server.

I would like to give a simple example, consider this code

static ProcessOrder()
{
    const int workerCount = 4;
    WorkQueue = new ConcurrentQueue();
    Workers = new List();

    for (var index = 0; index; workerCount; index++)
    {
        Worker = new BackgroundWorker();
        Workers.Add(Worker);
        Worker.DoWork += Worker_DoWork;
        Worker.RunWorkerAsync();
    }
}

The above code is extremely dangerous. You have no control over the load it will cause to backend calls, especially when it is calling Stored Procs etc. Secondly, it is not durable.

The above code, can easily be replaced by a NServiceBus Saga or handler, in the above, a Saga is appropriate, as this message is a Root Aggregate, we have an order to process. Sagas will provide an environment to preserve state and all the threading and correlation is all handled for you.

partial void HandleImplementation(ProcessOrder message)
{
    Logger.Info("ProcessOrder received, Saga Instance: {0} for OrderId: {1}", Data.Id, Data.OrderId);
    RunOnlyOncePerInstance(message);         
    AlwaysRunForEveryMessage(message); //counts and extracts new order items.

    if (Data.OrderCount == 1)
    {
        SendSaveOrder(message);
        return;
    }

    if (Data.InitialDelayPassed) //Expired workflow
        SendUpdateOrderItem();
}

From the above, you can see state is preserved for each unique OrderId. This Saga processes multiple order items for an OrderId. Which can be submitted at any time during the day. We do not have to worry about multiple threads, correlation is dealt with automatically, well we set the correlation id to the OrderId, so order items coming in can be matched to the correct Saga instance.

We can now get rid of the unreliable issues with in memory worker threads, and uncontrolled pounding of the database.

By using a MSMQ infrastructure and leveraging NServiceBus, these sort of issues, you find in code, can be easily mitigated.

Install NServiceBus as a service with Powershell

Hi,

I have just completed a script that can be used to install a service bus host as a windows service. It is nice to use, when delegating the installs to the support team, this is always done via NServiceBus.Host.exe.

It can always be tricky parsing arguments in PowerShell. The trick is to use the Start-Process command, and isolate the arguments into it’s own variable.

Here is the two scripts.

Download the script

The one will install the license and the other installs your endpoints as a windows service. All you now need to do, is just type in the password.

It relies on a directory structure, where.

Root\powershellscripts.ps1
Root\ServiceNameA\Debug or Root\ServiceNameA\Release
Root\ServiceNameB\Debug or Root\ServiceNameB\Release

I hope this makes installing all your endpoints easy.

Download the script

Now, remember, if you install Subscriber endpoints before publishers, you will get a race condition, so with this script, add the endpoints in the array argument in the correct dependent order. First install all the publisher endpoints first, then the subscriber endpoints.

Romiko

PayPal Payment Standard IPN/PDT–Asynchronous Processing

Hi Guys,

I am currently working on a personnel project to interface with PayPal Payment Standard on an MVC3 and Windows Azure based application. I needed to find a nice way to convert request/response objects from PayPal to their corresponding IPN/PDT objects.

I want a payment solution that can be easily scaled on high load. So we will leverage NServiceBus as the front end –> back end service bus infrastructure. This allows decoupling of payments from the functionality of the site, so under high load the payments will not restrict the usability of the site, This is done with MSMQ and NServicebus. I will show a basic example of how.

Below is a DTO you can use for IPN or PDT data. Of course you will need logic to convert http request/response data to this DTO and you have many ways of doing so.

More information about PDT/IPN variables can be found here:

https://www.paypal.com/IntegrationCenter/ic_ipn-pdt-variable-reference.html
https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/e_howto_html_IPNandPDTVariables#id091EB0901HT

You can then take and IPN and convert the request form/query string values to a DTO e.g.

  _myIpnNotification.Invoice = _myRequest["invoice"];

The same applies for the PDT which you will receive out of band on a separate listener.

Here are the different type of transaction types and status

 [Serializable]
    public enum PdtStatus
    {
        Unknown = 0,

        [Description("SUCCESS")] Success,
        [Description("FAIL")] Fail
    }

  [Serializable]
    public enum IpnStatus
    {
        Unknown,
        [Description("verified")] Verified,
        [Description("invalid")] Invalid
    }

    [Serializable]
    public enum TransactionType
    {
        //[Description(null)]
        Unknown = 0,

        [Description("cart")] Cart,

        [Description("express_checkout")] ExpressCheckout,

        [Description("merch_pmt")] MerchantPayment,

        [Description("send_money")] SendMoney,

        [Description("virtual_terminal")] VirtualTerminal,

        [Description("web_accept")] WebAccept,

        [Description("masspay")] MassPayment,

        [Description("subscr_signup")] SubscriptionSignUp,

        [Description("subscr_cancel")] SubscriptionCancellation,

        [Description("subscr_failed")] SubscriptionPaymentFailed,

        [Description("subscr_payment")] SubscriptionPayment,

        [Description("subscr_eot")] SubscriptionEndOfTime,

        [Description("subscr_modify")] SubscriptionModification
    }

Security, if you want the best security, never trust anything from Paypal, always double check the data, so when an IPN comes in, take that data and do a check with Payal and then verify the results match, and the same with PDT data.

This means checking the IPN data – currency, amount, date, subscription type etc is the same as what is in your transaction log, this way, any spoofing is protected, here is an example of a rule.

 public interface IRulesCommon
    {
        List<Error> ApplyRulesStandardPayments( decimal resultAmount, Currency resultCurrency, string receiver, TransactionLog transaction);
    }

 public interface IRulesPaypal
    {
        List<Error> ComparePaymentFrequencyTransactionStatus(TransactionLog transaction, TransactionType transactionType);

        //Used by IPN Only for SignUp
        List<Error> CheckPaymentFrequencyOnSignupIpn(TransactionLog transaction, TimePeriod subscriptionPeriod);

        List<Error> CheckSubscriptionPayment(decimal mcAmount3, Currency currency, string receiver,
                                             TransactionLog transaction);



        List<Error> CheckSubscriptionSignUpCancel(decimal mcAmount3, string recurring, Currency currency, string receiver,
                                     TransactionLog transaction);
    }

Unfortunately, I cannot show you implementation logic for this, as it may breach the security of my site if something is found which can be compromised, however this interface is a good start.

As you can see above, I do allot of checks, I ensure the amount, currency, receiver all match the original transaction object.

So, now we have an IPN or PDT listener, so say I have an MVC3 listener controller like so:

 public class IpnController : Controller
    {
        public readonly ILog Log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
        readonly IBus _bus; //NServiceBus Object
        readonly IIpnDataProvider _ipnDataProvider;
        readonly ISettingsDataProvider _settings;

        
        //Constructor Inject with Autofac
        public IpnController(IBus bus, IIpnDataProvider ipnDataProvider, ISettingsDataProvider settings)
        {
            _bus = bus;
            _settings = settings;
            _ipnDataProvider = ipnDataProvider;
        }

        /// <summary>
        /// Expects a post with variables like: ?mc_gross=19.95&protection_eligibility=Eligible&address_status=confirmed&payer_id=LPLWNMTBWMFAY&tax=0.00&address_street=1+Main+St&payment_date=20%3A12%3A59+Jan+13%2C+2009+PST&payment_status=Completed&charset=windows-1252&address_zip=95131&first_name=Test&mc_fee=0.88&address_country_code=US&address_name=Test+User&notify_version=2.6&custom=1||1||myredirecturl&payer_status=verified&address_country=United+States&address_city=San+Jose&quantity=1&verify_sign=AtkOfCXbDm2hu0ZELryHFjY-Vb7PAUvS6nMXgysbElEn9v-1XcmSoGtf&payer_email=gpmac_1231902590_per%40paypal.com&txn_id=61E67681CH3238416&payment_type=instant&last_name=User&address_state=CA&receiver_email=gpmac_1231902686_biz%40paypal.com&payment_fee=0.88&receiver_id=S8XGHLYDW9T3S&txn_type=express_checkout&item_name=&mc_currency=USD&item_number=&residence_country=US&test_ipn=1&handling_amount=0.00&transaction_subject=&payment_gross=19.95&shipping=0.0
        /// </summary>
        /// <returns></returns>
        public ActionResult Ipn()
        {
            string rawData = Extract.GetHttpRawData(Request);

            if (_settings.AuditHttpEnabled)
                Audit.AuditHttp(rawData, _bus, Request.RawUrl); //Asynchronous auditing of the IPN message for audit tracking

            if (_ipnDataProvider != null && _ipnDataProvider.MandatoryDataSpecified)
            {
                try
                {
                    var message = new PaypalIpnMessage
                        {
                            OriginalHttpRequest =
                                rawData,
                            MessageId = Guid.NewGuid(),
                            InvoiceId = _ipnDataProvider.TransactionId,
                            Notification =
                                new ResponseToIpnNotification(Request.Form).GetIpnNotification()
                        };
                    _bus.Send(message); //Asynchronous processing of the payment message to the backend systems
                }
                catch (Exception e)
                {
                    Log.Info(e);
                    Log.Info(rawData);
                    return new HttpStatusCodeResult(400);
                }
                return View();
            }

            Log.Info("Request data does not contain mandatory fields e.g. MerchantId and MerchantTransactionId");
            Log.Info(rawData);
            return new HttpStatusCodeResult(“_codeResult”);
        }
    }

As you can see above, any payment processing is now totally decoupled, the Front End and Back End work independent of each other with no synchronous calls. Yes this is all wrapped within a MSDTC transaction. I prefer this than using WCF, due to no issue in dealing with latency and I have guaranteed delivery of my crucial message.,You will write a similar controller for the PDT out of band response as well. We also leveraging an IoC container to automatically inject the service bus object into the controller as well as other dependencies. My favourite is Autofac.

 public ActionResult Process()
        {
            string rawData = Extract.GetHttpRawData(Request);
            if (_settings.AuditHttpEnabled)
                Audit.AuditHttp(rawData, _bus, Request.RawUrl);

            if (_successDataProvider != null && _successDataProvider.MandatoryDataSpecified)
            {
                try
                {
                    var message = new PaypalPdtMessage
                        {
                            OriginalHttpRequest =rawData,
                            MessageId = Guid.NewGuid(),
                            TransactionIdForPdtToken = _successDataProvider.TransactionIdForPdtToken,
                            AmountPaidRecordedByPdt = _successDataProvider.AmountPaidRecordedByPdt,
                            InvoiceId = _successDataProvider.TransactionId,
                            
                        };

                    _bus.Send(message);
                   
                }
                catch (Exception e)
                {
                    Log.Info(e);
                    Log.Info(rawData);
                }

                return Redirect(_successDataProvider.MerchantRedirectURL);
            }


            Log.Info("Request data does not contain mandatory fields e.g. MerchantId");
            Log.Info(rawData);
            return new HttpStatusCodeResult(“_codeResult”);
        }
    }

I hope this gives you some ideas for developing robust payment options for your site and provide a nice user experience for users Smile

Remember to also deal with your dates in UTC format, and ensure the date kind property is set to UTC as well for extra safety when storing the transaction date.

So in a nutshell, you can have top notch validation that protects from spoofing attacks if you ALWAYS take a IPN and PDT and compare it with the original transaction object! If you keep your transaction id’s unpredictable, users can never guess someone else transactionid and hijack it for payment. So you can have a nice invoice number e.g. 90124, but the transactionid is not easy to predict, it could be a Guid etc.

Imagine if the above was not the case a smart user could then create two transactions, one for a cheap item and one for an expensive one, here can then hijack the ipn/pdt, or send a FAKE pdt/ipn to your system, and then swap the item information around, and then later cancel the expensive item transaction, however, he hijacked the cheap item and changed the item list and transactionid, hence why I say, check AMOUNT, CURRENCY, Items, TranactionID etc. There are allot of sites out there that are easy to hack due to them not doing these sort of double checks on Ipn and Pdt messages from Paypal. If I know you IPN or PDT listener, I can send fake messages in and try guess weaknesses by using methods such as generating two transactions on the site and trying to swap items around, currencies, etc In fact, because they cannot guess my transaction number, I do not need to check items in the list, as this is not processed in my listener, so if they changed a flower pot to a BMW, who cares, I ignore this sort of data in an IPN/PDT, as long as the fundamentals are the same we in good shape.

Matthew Will and I spent allot of time think these sort of scenarios through and nutted out a nice secure solution. The above samples should get you started in the right direction for Paypal Standard integration. Allot of implementation logic is left out, and this is done on purpose.

Cheers