My current client, Charlie of PearBudget fame, is actually expressing interest in learning more about Rails backend code. How cool is that? I started writing an introductory primer for him about how to approach backend optimization, then realized others may enjoy reading it as well.
The techniques and tools mentioned here have been discussed in detail all over the web, so rather than provide detailed pointers this article paints a broad picture about how all the approaches fit together.
Guided vs Intuitive Optimization
Early in the development process, efficient optimization is driven by intuition. The developer isn’t studying benchmarks, he’s just trying to clean up code whenever it’s worthwhile and doesn’t interfere with development. The moment that optimization itself becomes the goal, it should be driven by reliable and reproducible numbers, otherwise you risk spending a day optimizing something that has no discernable effect on the application’s performance.
Finding the Problem
Reliable and reproducible numbers come in three flavors described below. The numbers should always be collected in production mode, though. In development Rails reloads certain files like your models and controllers, which is awesome, but the time spent reloading will throw off your benchmarks. It’s easy to run in production mode on your own machine by first opening your config/database.yml and copying the development configuration settings to the production configuration settings (don’t check this into Subversion!), then running `script/server -e production`.
Average response times
There’s not much to learn from any single response time, due to the wide variation. To determine a reliable time for any given page you need to use average the response times over a number of requests. Generic tools for this include Apache AB, Httperf, and RailsBench. Note that these tools can themselves slam your processor, so they’re best when not run from the machine processing the requests.
Traffic patterns
A page may on average respond slowly, but unless it’s being run with any significant frequency that doesn’t matter. A page that’s slow once per month per user just isn’t worth optimizing until the common pages are ticking along smoothly. Traffic patterns can be determined either through log analysis or through web analytics (Google, Overture, etc.). A common log analysis tool for Rails logs is Rails Analyzer, although it can be difficult to use because it requires your log to be in a certain format (Hodel3000CompliantLogger helps here).
Per-request times
Once you’ve identified a slow page, it’s time to determine why it’s slow. You can time the duration of any given request and even break that request down into component times. This is what the benchmarks in the Rails log attempt to do, though I believe I’ve improved on them with the BenchmarkForRails plugin.
Benchmarks on this level are mostly good for determining ratios. Two identical requests will never have the same exact times, but they will have generally consistent ratios between the components. For example, two requests for the home page may take 0.2 and 0.3 seconds (a significant difference), but the important take-home lesson from these times might be that rendering is 50% of each request.
Once you’ve identified a component that appears slow, you can start digging into the real cause. This is where in-depth tools like RubyProf are useful. Profilers are too slow to leave on for normal operation, and they have a learning curve for deciphering the output, but they can really open your eyes to unknown speed hits in the codebase. Note that these profilers will not help you profile database query performance.
Solving the Problem
Once you’ve determined what exactly is slow, it’s time to determine what can be done about it. Sometimes simply restructuring a code block can be all that’s needed. But in web applications it usually just comes down to database usage.
Adding Database Indexes
If you find a slow query, the first thing to check is that it’s well-written. Assuming that’s the case, you should immediately ask yourself whether you can index a database column (or set of columns) to aid the database in searching through the massive amounts of data your application has picked up. A simple database index can easily cut a query’s time by 90% or more.
Caching
Are you making the same query or running a calculation multiple times per page? Cache it. But don’t break encapsulation – when possible, cache it without the application really even knowing. For example, make sure the methods on your fat models embrace the “@var ||= calculation” pattern.
Query Merging
Efficient queries can become very inefficient in bulk. This is when it’s time to consider merging the queries, whether through eager-loading or by querying a set of records and sorting them into groups afterwards (instead of querying them in groups). The reason is that there’s some overhead in every database query, ranging from the costs of constructing the query to the costs of communicating with the database to the database’s own overhead costs.
Note that it’s entirely possible to take this too far. When in doubt, check the benchmarks.
Denormalization
In the end, your queries may just be performing as well as they can given your pristine database schema. But if you find some massive queries that join 75% of the tables in your database, you might consider denormalizing. Rails makes denormalization manageable with well-placed callbacks on your ActiveRecord models, and careful denormalization can create shortcuts between key data objects and really cut down the query complexity.
Better Hardware
When all else fails, upgrade your hosting. Add more memory. Upgrade to a cluster. These are perfectly valid ways to make performance problem go away until you have more time to focus on them.
Optimal Optimization
The final thing to realize is that your application will never be performing as fast as it could. Accept it! There will always be something you can optimize, but it won’t always be worth it. Careful application of the above techniques (or cascade of techniques) can help you keep your optimization passes efficient and focused.