Is there a way you can incorporate the A/B testing into an actual product?
You can release two pieces that are functionally equivalent but have a slightly different UI on it. If you send stats back to the server you can probably get a good amount of data.
You mean like downloadable software? I did it once.
There are many ways to distribute Java applications to customers. That probably bores you. To make a long story short, the tradeoff is usability versus download size.
I made two functionally identical (or so I thought -- long story) versions of BCC, and randomly redirected half of the people downloading from my site to one (1 MB, required JRE) and half to the other (10 MB, did not).
I was able to check which versions were causing sales by incorporating a URL parameter naming the version of the executable when people clicked from the nag prompts to the website. Set the version they're using in a cookie, associate it with a later purchase, blah blah bob's your uncle.
Early testing suggested that the larger one was statistically significantly more effective at producing sales, but this trend did not continue.
There are numerous downsides with this. It took a LOT of work to set up. I got to maintain two versions of the same executable, which introduced bugs (I'll spare you the details, aside from one: Vista). It cost me in support requests. I will be supporting those executables for years to come (trust me -- I still get people buying from software not on my website in 2+ years). Changing the A/B tests used takes a full release cycle.
On the web, these are all so easily surmountable it isn't even funny.
You can release two pieces that are functionally equivalent but have a slightly different UI on it. If you send stats back to the server you can probably get a good amount of data.