Optimize Your Site
I was surprise to find people actually reading my blog. Especially a poorly written post on Offermatica’s dominance.
So here is another post on some basics tenets of running tests. The goal: provide users with a consistent view of the site while exposing different variants of the site to different people.
Users Must Have A Consistent Experience
Within a session, a user must have the same experience. If you expose a users to two or more variants within a single session you will ruin the test. There will be no way of separating activity by variant. You may also create an unworkable navigation.
For example, lets say you decide to make a brighter background on some google ad words. If the users are split into two groups, you may simple compare the conversion rates of each group. In addition users will consistently see the same background and get the same “message”. If you randomly serve up different backgrounds, users will see ad words in two different presentations.
If a user sees both “messages” it can be very difficult to figure out which “message” should be credited with the conversion.
Good SEO Means a Single Variant
Remember all that hard work you put into optimizing your site for search engines? With some poorly planned multivariate testing it can all go down the drain.
Ya see, if search engines find more than one URL with almost the same content, that URL is viewed as a link farm and the page rank plummets. If your testing creates different URLs for different variants, try using robots.txt to block search engines from seeing your “experimental” pages.
The other problem is rapid changes. To the best of my knowledge, if the agents from a search engine see multiple version of the same page, they will only take one version. With each new version no one is sure what happens to the pages rank, it may go up it may go down. Sometimes these changes take 2-3 weeks to digest, and problems show up after testing is complete.
In this case filtering by user agent is a good idea. Send agents with “bot” in their UA string to a nice stable site. In addition, at this point search agents don’t look at java script. With DOM manipulation via java script your site is safe too.
June 27th, 2006 at 9:32 am
If you want a lot of traffic there is nothing more you have to do than go after a industry “leader”. : )
You do make a excellent point about consistent experience and it is interesting that solutions out there won’t even do simple things to ensure that tests are not “cross polinated”. Something even as dumb as setting a persistent cookie and then checking that cookie value when the person comes back again to ensure the same exeprience is provided (so as to avoid people in single session or multi session getting into different tests). This is not even really intelligent but will catch most “cross polinators” (if there is such a word).
Actually there is a really nice way to avoid the SEO problem you mention. Our site will detect the user_agent_id of the requestor and if it is one of our pre identified big search engines they will simply never get placed in a test, they’ll see the “control” version of our website which almost always is the normal version.
Ok I should stop now but in light of your other post and this one I wanted to share a quick resource that outlines all the possibilites for testing, and it highlights pro’s and con’s of each method (you touched on some of these for multivariate in your other post):
http://www.kaushik.net/avinash/2006/05/experimentation-and-testing-a-primer.html
Thanks for this great posts.
-Avinash.
Web Insights Blog: www.kaushik.net/avinash