StackExchange is built on the ASP.NET and SQL Server stack. Recently, Sam Saffron and Marc Gravell blogged about their experience identifying and solving a performance problem that was finally traced to the .NET GC GEN-2 objects. There is a lot to be taken away from their experience for everyone tuning performance for applications in production.
The issue was caused by a custom tag engine, that kept large object lists in memory. The blog posts are both worth reading, Sam’s being the more detailed one including details on how the problem was diagnosed. Key takeaways -
- Use profilers. You can use something like MVC-MiniProfiler for profiling method calls and a memory Profiler like the .NET Memory Profiler or the Redgate ANTS Memory profiler for tracking memory usage.
- If possible, have Queryable logs – the StackExchange team has a background process that parses raw logs and inserts it into a large, dedicated, SQL Server instance. Without the ability to query logs, you can’t graph on metrics and see trends or patterns.
- Understand how the GC works – Garbage collection in .NET works with 3 generations of objects (GEN 0 to GEN 2) with the longest living data making it to GEN-2. GEN-2 scans are least frequent and most expensive to perform and even block other threads. Sam recommends reading Rico’s Garbage Collector Basics and Performance Hints for more details
In the end, the team ended up switching from object arrays to index arrays and from classes to structs, amongst several other optimizations, for that particular part of the application. The first one is a fairly useful pattern wherever the array is just an index to a main immutable list, but the second one is not commonly done – Sam recommends it only if there are more than half-a-million objects or so, that are released shortly after they reach generation 2.
The StackExchange team takes performance very seriously – for instance you can see how StackOverflow uses caching at three different levels to improve response times and reduce server loads.
There are several resources you can refer to on InfoQ about performance and scalability as well as on MSDN, including this article by Rob Howard. After all, quoting Sam, “Doing a ton less work is always going to be less expensive than doing more work!”
Do you have any similar experiences to share about finding and resolving bottlenecks in scaling your ASP.NET application?