Advanced Rails - Building Industrial-Strength Web Apps in Record Time

Measurement Tools | 151

We see that the server that produced the first set of samples will usually (95% of the
time) respond in between 8 and 12 ms. On the other hand, the data from the second
server varies so wildly as to be nearly meaningless.

When comparing two data sets, it is important to compare not just their means but
their standard deviations and confidence intervals as well. It may look as if you have
made an improvement, but if the confidence intervals overlap by a significant
amount, there is no statistical significance to the result.

Black-box analysis with httperf

Now that we know how to analyze results, we can benchmark a site. The best refer-
ence is Zed Shaw’s instructions about tuning Mongrel using httperf (http://mongrel.
rubyforge.org/docs/how_many_mongrels.html). We will not repeat the procedure
here, but we will give some caveats:

• Ensure that you have the front end server (usually Apache, lighttpd, or nginx)
  configured to serve static files. Then do a baseline measurement that requests a
  static file from the front end web server. You will never get Rails faster than this.


• Run your tests from a machine as close as possible (in network terms) to the
  server. You want to eliminate latency andjitter(variance in latency) from the
  results.


• Do not run performance tests from your web server. There are too many interac-
  tions between the server and the analyzer that will be confounded with your
  results. Even if CPU utilization is not a problem (such as on a multiprocessor
  machine), you will not know if I/O contention has skewed the results.

Code Timing

The Ruby standard library includes Benchmark, which can be used to answer simple
questions about code performance. The key word here issimple: it is all too easy to
ignore confounding factors and take the numbers that Benchmark gives you as
gospel.

Suppose we want to compare conventional method dispatch to the idiom of using
method_missingand then examining the method name to decide what action to take.
Here is a simple code example that benchmarks the two options:

require 'benchmark'

class Test

def test_one

1 + 1

end

def method_missing(method_id)

case method_id

when :test_unknown: 1 + 1