Offermatica Needs a Competitor
Offermatica http://www.offermatica.com/ is a great piece of software. Their product does multivariate testing as a web service. For example you could pick 5 different background colors for you text ads and find out which performed the best.
Little changes like this made a huge difference. I’ve seen click through rates go up 8 times with a different background color. I’ve seen SMALLER buttons with new spicy graphics improve click through by 20%.
Problem number 1
Offermatica does have its problems. Lets start with the technology. I have no problem with the overall architecture. In fact, I would pretty much build the same thing if I had the technology reigns, no the problem is scalability. Offermatica doesn’t scale for large websites. Don’t send Offermatica more than a few million hits a day. Keep it around 2-4 million. Their platform is new technology it shouldn’t be that hard to scale, but Offermatica is more interested in sales than performance.
Problem number 2
There are many types of conversion, but just about every website will be interested in following a multi-step, mulit-page process. Offermatica doesn’t do a good job of tracking changes through multiple user interactions. Offermatica does work well with multivariate testing on a single page. So if you want to figure out if the “blue” background makes shoppers buy more widgets, you have quite a big of work cut out for you. This too could be improved, but I’m not seeing any real changes in the technology as of late.
How Does it Work
Well Offermatica askes you to insert a special div tag on your page, with default HTML inside of that div tag. You then insert a URL call which loads javascript functions from Offermatica.
At page load time, the user is issues an Offermatica cookie, and Offermatica changes the HTML inside of the div tag based on a has of the user cookie.
Double AB-Testing
Since Offermatica can’t handle huge volumes of traffic, large volume sites end up splitting there traffic and sending Offermatica a small, yet significant fraction. This amounts to site running its own A-B test sending, on top of Offermatica’s own tests.
June 23rd, 2006 at 8:19 am
You should investigate SiteSpect, an A/B and multivariate testing platform that is built to support large scale, high performance, multi-objective optimization. http://www.sitespect.com
June 23rd, 2006 at 8:53 pm
To be fair there are 4 companies in this space. Sitespect is one Offermatica and Kefta are two others.
http://www.kefta.com/
http://optimost.com/
http://www.sitespect.com/
http://www.offermatica.com/
June 24th, 2006 at 5:09 am
Hi Eric. Thank you for asking for my opinion.
There is at least one more competitor in that space, Vertster, and perhaps many more. But that really speaks to the title of your post, doesn’t it — there are many Offermatica competitors and we work hard to figure out who they are. I found SiteSpect, for example, when Dave Morgan commented on my conversion blog. I found Optimost when I saw them at Shop.org in Atlanta. But we see Offermatica everywhere - they have done a stunning job of promotion themselves. Isn’t that the business we’re all in — marketing?
I can’t speak to the issue that you pointed out, i.e. large site with issues. I know of at least one very large household name that uses them and is happy. Probably more, one need only look at their case studies. I also think there are technical issues with regard to the way some testing companies redirect which can mess up testers who are well optimized for search, and I believe Offermatica evades that problem. In any case, they do have competitors, they have many customers who love them and they probably aren’t perfect. But last time I checked, I wasn’t either.
Am glad I found your blog. - Robbin
June 26th, 2006 at 4:16 pm
A very interesting blog and this thread is particularly good. I just added it to my RSS
This industry is just like the CRM space of 2000… each month the buzz around the market is growing nearly as fast as the list of competitors. However, just like the example, the list will eventually shorten once a clear industry direction is determined and the market matures. The future promises to be very interesting!
The key in shopping for this form of technology is exactly the same as it is for any other strategic technology purchases… you needs to be clear on what it is that you’re trying to solve before you pull the trigger. If the issue is conversion and revenue increases, a plain multivariate or A|B test may or may not be the right solution.
What does that mean? By their nature, these tests randomly split traffic coming to the page. Inherent with that technology is the presumption that all site traffic is the same, that there is no differences in needs or expectations. This may be the case for your site but consider this example: a person using the search query “cheap widgets” is clearly different from one using the query “quality widgets”, however, using a multivariate test, this distinction is lost and the best “recipe” would simply highlight the needs of the larger of the groups, not the best for either individually.
As well, as The Calder Group pointed out in the initial blog entry, scalability is critical for large commercial sites. However, in addition to the potential latency that an overtaxed system can bring, there are other issues to consider… what if their servers are not available? Imagine a large website with blank wholes all over the page, that’s what would typically happen. At Kefta, we’ve filed for patent rights around this issue as a response to many of our large clients.
Finally, although we have had the technology to do testing for several years, our clients find greater results in targeting first, then testing. We call our approach Dynamic Targeting. Basically, it’s website personalization that helps you deliver content specific to each visitor, then test to see what the best content is for visitors who share interests and needs.
As a side note, Kefta has been around for over 6 years, has been profitable for the last three, and services large and small clients around the globe.
Thanks!
Mark
June 28th, 2006 at 9:15 am
Eric, Two comments on two days on your blog (!!).
Competition is good in all industries and if the testing and experimentation mind-set takes off, as I hope it does, then even more players will jump in. We live in the great US of A after all.
Two quick thoughts to hopefully provide more context (I actually am lucky enough to actually work in a company where we run atleast a couple different experimentation platforms, so this comes from a practitioner’s perspective):
Offermatica does scale well. This season we were using it on one of our premier properties which gets multi ten million hits a month (they’ll kill me if I tell you the actual numbers : )) and Offermatica worked just fine as we sent several million participants into various tests. From your post it seems your experience might have been different, but I would recommend you push Offermatica to give you more details and data because of your experience.
The second problem you have highlighted is far more complex and I am not sure we can lay it all on Offermatica’s door (by the way your critique would apply to Optimost and others just the same). The definition of a multi-page test is critical first. If you are changing common “modules” / areas in the website then Offermatica can handle that just fine, and with first party cookies. We have done multi-stage reporting with them as well (the tool allows that natively).
Where this gets really really complex is if you are doing complex experience tests where you change too many things and try to measure multi-goal success (think more than conversion, think for example measuring customer satisfaction, qualitative, along with quant which MVT vendors will do just fine). It gets complex but can be done. I have a post coming on my blog in a week or so that will cover this more.
I think all current MVT vendors will evolve to allow us to do more kinds of test (they all suffer from the same limitations today) and allow us to measure more.
I hope this helps give some real world context, I am adding a couple of links from my blog below that might be of interest.
Thanks for stirring up the conversation.
-Avinash.
http://www.kaushik.net/avinash/2006/05/overview-importance-of-qualitative-metrics.html
http://www.kaushik.net/avinash/2006/05/experimentation-and-testing-a-primer.html
July 2nd, 2006 at 12:45 pm
Eric
Thank you for appreciating the elegance of the Offermatica integration.
I’d like to clarify a few of your points that may be based on old information:
1. Offermatica is supporting clients that send over 100 Million requests a day.
2. Offermatica supports multiple conversion events across multiple pages and multiple user visits.
3. By default Offermatica uses first-party cookies
Mark
(Offermatica Engineering Team)
August 31st, 2006 at 7:37 pm
I’ve read a lot of concerns in this thread, ones which we’ve all addressed with Offermatica, and it seems like people just aren’t using the software to it’s full potential. I’ve only been using the software for a few months, and I’m just starting to get into some of it’s stronger features, but already I could come up with some possible solution for the problems listed above. I’m by no means a representitive of the company though, so definitely ask your reps if these will work before you put them into play.
When talking about targetting specific groups of people, you simply target the mbox to segment out specific groups of people. On your paid search, which I’m assuming is the main place that people get specific groups of visitors from, simply throw on some sort of url parameter onto your links so you can identify your groups of keywords that are targeted towards specific groups of shoppers.
For example, for all your cheap widgit shoppers, give them a parameter like “widigit1″ and for expensive widgits, have a parameter like “widigit2″ on those keyword groups. You can then target those specific groups in two separate campaigns running the same tests. You should be able to see how different users behave to different content.
As for there being “blank spots” if offermatica goes down, just put something in the tag in the html as your default content. If the offermatica script doesn’t load for some reason, it just displays the default stuff.
The “10% of traffic” problem took me a while to think of a solution, but once I thought of it, it’s actually pretty easy. Lets say you have an A/B/C test on your home page, and then an 1/2/3 test in the checkout process. If you want to have that same 10% of traffic included into both tests, simply set up a 9 recipe test. A1, A2, A3, B1, B2, B3, C1, C2, C3.
Now,if you wanted to run something where 10% of homepage traffic saw the test, but you want 100% of visitors to see the checkout test, simply change the priority of each test. Put the A/B/C/1/2/3 test at high priority and 10% of traffic, and put the checkout test at low priority with 100% of traffic.
The results will split themselves correctly and will actually provide you with some cool results.
The only problem I have is that the program is web-based, so setting up a test takes 3 minutes instead of 1, but, if that’s my only problem, and I’m setting up 10-12 tests a month, I’m not losing too much sleep over it.
My number 1 advice would be to consult the offermatica person assigned to your account and brainstorm some ideas for solutions.