A/B Testing - nathanbarry http://highscalability.com/blog/2011/9/10/ab-testing-and-the-gaming-industry.html ====== sjbach Very interesting, especially since we use A/B testing quite a bit here at Squarespace. A nice feature of AB testing is that the best answer typically isn't obvious, in a way that is hard to predict. With a good amount of experimentation and user data, though, you can get a pretty high confidence in the best answer. ------ NathanBarry The link to the full PDF version of the post is missing a "n" in the URL. This should be fixed by the time you click the link, but it may have been deleted when I posted. ~~~ chc Sorry about that, there were a lot of edits happening at the same time, but it should be fixed now. ~~~ nathanbarry Thanks! It looks like it works now. The version we are distributing will be a bit different from the first draft since there were some typos in it. ------ NathanBarry This is a really good read, I hadn't seen many of these concepts used before, and I've always thought it was interesting to test and optimize new designs. ------ davewicket Excellent write-up! One key thought that I took from this article is the importance of A/B testing at scale. As a company grows, it's important to make sure that everything you do at a large scale works equally well at a smaller scale. I also agree with some of the comments that if you use these concepts in a high-traffic application, it needs to be monitored rigorously. By setting up some alerting and tracking, you can ensure that there are no failures or interference with the application. ~~~ nathanbarry I have a related post coming up about A/B testing at scale. If you're curious to see how we do it at Squarespace, check out and ------ buro9 This is interesting as our company does things pretty much the same way. We have 5 minute cycles and try to split things into as small of tasks as possible. We have a feature being developed to allow a user to give a URL and it will redirect to that on page load, we call that the 'redirect hack'. We've just added support for non-www and www versions. ~~~ aaronblohowiak Do you handle the 301 Redirects at the load balancer? ~~~ buro9 My answer is: no. But then I have a second question: What is the benefit? As a very small website, I'm struggling to grasp the benefit, which is that the HTTP request and response get delayed. But we have a load balancer which will take a look at all HTTP requests. My assumption is that we're talking about something that is extremely rare, even if it's 2% of traffic. The load balancer has multiple servers which share the load. I can't see how we can reduce that even a little bit, even as a load test. Even if we were to find a way to get rid of a few servers, and only load one, there's the challenge of load balancing again. ~~~ aaronblohowiak For your question about if the load balancer handles the 301s: you probably shouldn't. A browser doesn't generally send a 301 for every possible redirect -- it does it when it knows that the redirect is permanent. At some point the browser will cache this fact, and 301s will start to get dropped in favor of direct hits. The reason I am pushing you in this direction is that you will want to be able to reproduce (possibly even share) the load you expect for your load test when you actually have traffic. This way, if you accidentally load balance across a proxy, for example, you can make sure that the request stays on the HTTP server. That way you can reproduce the number of requests to the site which made you consider using a redirector in the first place. ------ nathanbarry This article is just one part of a series about scaling applications, as needed. The next article is coming soon: Scaling A/B Testing at Scale: Squarespace's Story