Intro
A/b testing is great! Wikipedia uses it every year to pick the best fundraising banner and a/b testing makes them millions of dollars more.The contextual bandits algorithm can do even better.
2 improvements over traditional a/b testing
- The algorithm doesn't have to have a single winner. It will try to show the most effective version for each user.
- The algorithm automatically optimizes. If it has a good guess at which version a user will prefer, it uses that. If not, it picks randomly to collect more data.
How to Use It
There's a paid service at http://www.synference.com/ with a free beta.My Questions After Reading
- How do we generate the versions to use for contextual bandits?
- Does the algorithm support adding new versions later?
- Does the algorithm support removing poorly performing versions?
Hey, I'm one of the co-founders of Synference, and it's nice to see what questions you're left with. I'm happy to answer your questions.
ReplyDelete1. You generate the variants (which we call choices) yourself, and associate a name with each choice. Then, when you set up your optimization task on the Synference website, you fill out a form in which you list the name of each choice.
2. In theory, it's possible to add new choices on-the-fly to an already existing optimization task. In practice, Synference currently supports this functionality only if you have an enterprise account.
3. You do not need to remove poorly-performing choices. If they perform poorly, then we will hardly ever suggest to use them anyway. The idea behind our API is that you do not need to babysit the optimization tasks. There may however be a situation in which you need to guarantee that a choice is removed completely. This is currently only possible on our enterprise accounts, but we may add this functionality to the cheaper accounts soon.
If you want a more in-depth example of how this all works, you can check out the "banner tutorial" on the Synference website.