Calder Group Inc

I was surprise to find people actually reading my blog. Especially a poorly written post on Offermatica’s dominance.

So here is another post on some basics tenets of running tests. The goal: provide users with a consistent view of the site while exposing different variants of the site to different people.

Users Must Have A Consistent Experience

Within a session, a user must have the same experience. If you expose a users to two or more variants within a single session you will ruin the test. There will be no way of separating activity by variant. You may also create an unworkable navigation.

For example, lets say you decide to make a brighter background on some google ad words. If the users are split into two groups, you may simple compare the conversion rates of each group. In addition users will consistently see the same background and get the same “message”. If you randomly serve up different backgrounds, users will see ad words in two different presentations.

If a user sees both “messages” it can be very difficult to figure out which “message” should be credited with the conversion.

Good SEO Means a Single Variant

Remember all that hard work you put into optimizing your site for search engines? With some poorly planned multivariate testing it can all go down the drain.

Ya see, if search engines find more than one URL with almost the same content, that URL is viewed as a link farm and the page rank plummets. If your testing creates different URLs for different variants, try using robots.txt to block search engines from seeing your “experimental” pages.

The other problem is rapid changes. To the best of my knowledge, if the agents from a search engine see multiple version of the same page, they will only take one version. With each new version no one is sure what happens to the pages rank, it may go up it may go down. Sometimes these changes take 2-3 weeks to digest, and problems show up after testing is complete.

In this case filtering by user agent is a good idea. Send agents with “bot” in their UA string to a nice stable site. In addition, at this point search agents don’t look at java script. With DOM manipulation via java script your site is safe too.

Jeff has a nice collect of Yahoo shirts on display. Someday I will show off my collection of shirts.

I only tracked back 4 posts. TrackBack doesn’t seem to work too well as an API/Protocol. If anyone wants to re-write it I’ll pay them $100.

How do I know Ubuntu is taking over?

Observation Number 1
Well just last night I upgraded my Debian etch (aka unstable) to X11R7. Now X wouldn’t restart, so I started browsing the web using another computer. I found the answers pretty quickly, but the Ubuntu forums had so many more posting, and I have to say more useful answers.

Observation Number 2
Yesterday I was sitting down with someone from our Operations group. I noticed a familar interface, so I asked, “Are you running Debian?”. The answer which suprised me but shouldn’t have, “No, its Ubuntu”.

Observation Number 3
The Wikipedia entry for Ubuntu is about the same size as the entries Fedora and Debian.

• Ubuntu
• Fedora
• Debian

Oh, and here were my steps for fixing problems like could not open default font fixed or /ect/X11/X not executable
* Remove symlinks for /etc/X11/X and /usr/bin/X
sudo /bin/rm /etc/X11/X; sudo /bin/rm /usr/bin/X
* Install xfs
sudo aptitude install xfs
* Re-install xserver-xorg
sudo aptitude reinstall xserver-xorg
* Install some fonts
sudo aptitude install xfonts-100dpi xfonts-75dpi xfounts-base
* Re-create links to X
sudo ln -s /usr/bin/Xorg /etc/X11/X
sudo ln -s /usr/bin/Xorg /usr/bin/X
* Remove old font references, deleting anything referenceing /usr/X11R6/lib/X11/fonts in xorg.conf
sudo vi /etc/X11/xorg.conf
* Re-start gdm
sudo /etc/init.d/gdm restart

After the last one, x should start working again. It may flicker a few times and the screen will go black. Be patient.

Offermatica http://www.offermatica.com/ is a great piece of software. Their product does multivariate testing as a web service. For example you could pick 5 different background colors for you text ads and find out which performed the best.

Little changes like this made a huge difference. I’ve seen click through rates go up 8 times with a different background color. I’ve seen SMALLER buttons with new spicy graphics improve click through by 20%.

Problem number 1
Offermatica does have its problems. Lets start with the technology. I have no problem with the overall architecture. In fact, I would pretty much build the same thing if I had the technology reigns, no the problem is scalability. Offermatica doesn’t scale for large websites. Don’t send Offermatica more than a few million hits a day. Keep it around 2-4 million. Their platform is new technology it shouldn’t be that hard to scale, but Offermatica is more interested in sales than performance.

Problem number 2
There are many types of conversion, but just about every website will be interested in following a multi-step, mulit-page process. Offermatica doesn’t do a good job of tracking changes through multiple user interactions. Offermatica does work well with multivariate testing on a single page. So if you want to figure out if the “blue” background makes shoppers buy more widgets, you have quite a big of work cut out for you. This too could be improved, but I’m not seeing any real changes in the technology as of late.

How Does it Work
Well Offermatica askes you to insert a special div tag on your page, with default HTML inside of that div tag. You then insert a URL call which loads javascript functions from Offermatica.

At page load time, the user is issues an Offermatica cookie, and Offermatica changes the HTML inside of the div tag based on a has of the user cookie.

Double AB-Testing
Since Offermatica can’t handle huge volumes of traffic, large volume sites end up splitting there traffic and sending Offermatica a small, yet significant fraction. This amounts to site running its own A-B test sending, on top of Offermatica’s own tests.

Why I love Symbioware & you should too http://symbioware.com/overview.html

Testing web applications is tough. It’s so hard, that I encourage most developers to keep as much code out of the webapp as possible. For java this means turning off scriplets and using JSTL as much as possible. Ruby has a pretty good setup already, but making your code testable via modules is a must. With PHP templeting with Smarty and other addons is a good idea.

Why is it so tough? First you need to spin up a webcontainer and add test harnesses to talk to the webapp. Many times the only option is to use HTTP to exercise the test cases, and using HTTP mean scraping web pages, not fun.

Second using HTTP, means the testing client is getting active scripting. There are almost no clients our there which can handle active scripts. This is a shame because a good web enable application may have lots good features in active script.

Finally, there is every changing presentation. A form which says email may say username tomorrow. The testing code, which is often written in XML or some scripting language can’t keep up.

The solution comes in the form of products like Symbioware. It uses IE browser to parse web pages and it fully supports everything that IE does. That solves problems 1 & 2.

Symbioware, has a great WSYSIWYG interface. Creating new test cases is easy, you see can interact with your web page. It is also easy to keep up with changes, you see it. No more digging through code.

Its fantastic. Higher Quality means less time fixing bugs, which means more features faster.