Split testing with Rails Part 2 – The Results Are In

In my last article I ask you to visit Blabs and participate in a simple split test. A few hundred of you were nice enough to humor me and we were able to collect some statistically significant data. It turns out you like pictures of us, and love blink and marquee tags.

To elaborate, I started with a goal of driving traffic to our “Who” page, with the idea that this would be a good proxy for how well we’re getting readers of our blog interested in Pivotal (people are our secret sauce). I then created two new versions of the Blabs page (below), one that added Pivots’ pictures to their posts, and a second that added pictures and a big blinking “click me” button.

Control Page: The Current Blabs Landing Page

Variation 1: The Blabs Landing Page With User Images

Variation 1: The Blabs Landing Page With ‘Click Me’ Link

Web Optimizer requires that each of these pages be at unique URLs, so I added a query parameter to my controller ‘p’ that could be set to 1 to create the first variation and 2 for the second, e.g /blabs?p=2. You may have noticed this if you visited the page and weren’t in the control group. Here’s what the Web Optimizer (WO) summary screen looks like:

I wont get into configuring WO as it’s pretty dang easy. So, without further ado onto the code…

I started with a simple base class to take care of the JS and conditional logic around it (web_optimizer_test.rb). WO requires a specific bit of JS on the control page that does not appear on the variation pages, which web_optimizer_test takes care of for me.

I then created a subclass that contains functionality specific to the two permutations I was testing:

class BlabsSplitTest < WebOptimizerTest
  def initialize(id, control, plea=false)
    super(id, control)
    @plea = plea
  end

  def show_images?
    !control?
  end

  def shameless_plea?
    @plea
  end
end

With this done the controller code became pretty simple. I just needed created this object with the test id (created when making the test in WO), and the parameter onfo (e.g ?p=1) from the request:

class BlabsController < ApplicationController
  before_filter :setup_ab_test

  protected

  def setup_ab_test
      @profile_image_test = BlabsSplitTest.new('2726898050',
                                        params[:p].nil?, params[:p] == '2')
       @profile_image_ab_testing_control_js  = @profile_image_test.control_script
      @profile_image_ab_testing_tracking_js = @profile_image_test.tracking_script
  end
end

The second and third arguments to BlabsSplitTest.new are a bit confusing. The second (control) is true if we request the page with no p param. If p is defined we’re on a variation page. The second argument is necessary because I have two variations, and is true when we’re on the “shameless plea” version of the page (p=2).

Since all the logic for what to do is encapsulated in this object, the blabs template is pretty easy to understand:

<!-- images -->
<% if @profile_image_test.show_images? %>
  <%= image_tag(user_img)%>
<% end %>

<!-- sidebar -->
<% if @profile_image_test.shameless_plea? %>
    <div class="sidebar-node alarming">
      <marquee>Split Test in progress!</marquee>
      <blink>
        <%= link_to "please click here", '/who' %>
      </blink>
  </div>
<% end %>

That’s all there is to setting up the test. All that’s left is to track conversions, which we do on the “Who” page, which happens to be served up by the User controller. Here I could’ve used a BlabsSplitTest, but since I apways print out the same JS (and I refactored to BlabsSplitTest) I just use the WebOptimizerTest directly with the same test id:

class UsersController < ApplicationController
  before_filter :setup_ab_testing

  protected

  def setup_ab_testing
    @profile_image_test = WebOptimizerTest.new('2726898050')
    @profile_image_ab_testing_conversion_js = @profile_image_test.conversion_script
  end
end

and include the JS in the corrisponding template:

<%= @profile_image_ab_testing_conversion_js %>

That’s all there is to it. In terms of results, WO provides a running summary of the test and can be turned on/off without a redeploy whixh is very nice. Here’s my final summary:

which indicates we should add blinking marquee tags to all our pages, so stay tuned….

In terms of my opinion having implemented a simple test, I like WO overall and recommend it for simple tests and/or new testers. The reports kept me from having to bust out my stats book, and I didn’t have to keep the data myself, which was a plus and would be great at scale.

That said, WO seems limited in some key ways and I expect to outgrow it rapidly. For example, the Blabs controller and our “Official Blog” controller have a common ancestor, so it should be easy to have a parallel test there with one line of code. I would have loved to do that, however the conversion script uses a global variable to track which test it’s for so — as far as I can tell — you can’t have multiple tests end on the same page (e.g eight tests that all convert when a user registers).

This was fun for me, I hope y’all found it useful as well.

About the Author

Biography