In this latest article, Udi Dahan describes an experience implementing a new order system in which large size message passing was affecting the scalability and even bringing down servers in the system. The article describes how they diagnosed the problem and their solution, by "changing our service contracts and introducing stateful interactions we were able to manage the performance critical state of the system."
Read Spectacular Scalability with Smart Service Contracts.
The problem:
Read Spectacular Scalability with Smart Service Contracts.
The problem:
Message size seemed to be have a large influence on all the other dimensions. As messages got larger, it took more CPU to deserialize them, more memory to hold the resulting data, more network and IO to get that data in and out of the database, altogether affecting the total processing time. However, even small requests like those to discount all pending orders for a partner were impacted by the amount of data they had to process.The solution:
Instead of one "create order message", partners could send us many "order messages" over time keyed by their partner id and purchase order number. When they were done with all the items for that purchase order number, they would send us an "order message" with the flag "complete" set to true. It was a statefull interaction....Once resource utilization per request dropped, obviously latency dropped as well but throughput increased even higher than expected.And Udi's key takeaway:
The key takeaway for me was this: scalability isn't a Boolean value. Beyond the number of concurrent users or servers online, scalability is a multi-dimensional cost function. Given certain response time requirements, peak and average request rates, message sizes, formats, memory working set sizes per request, CPU/IO utilization per request, how much does a given solution cost? Technology choices which made sense for strategic partners were not cost effective for regular partners. Always run the results by the business - they just might change the performance requirements once they see that costs (up-front and ongoing) overshadow projected revenue.Have you had similar experiences?