Nanos gigantium humeris insidentes!

Using A/B test for other purpose

  • April 3, 2015 9:52 pm

A/B test is broadly used in many customer facing systems to measure the significance of a hypothesis. But most applications are focusing on revenue or other short-term impact metrics. For applications like personalized search engine, personalized recommendation engine, the results of this, they are getting more and more monotonic.

We should try to apply A/B test to see whether the personalized content truly interests the user. This should include two aspects:

  1. replace the recommendations with other randomness (meaning not recommend that item), will the customer eventually find it interesting in other way.
  2. once this item being discovered (via personalized recommendations or spontaneous discovery), whether user show significant interest to this item than other items along the way (e.g. those random items before discovering this item).

The philosophy behind this is same as Organic Training Data that we are making the rich richer, and that’s what we should avoid.

Print Friendly