A/B testing with Optimizely
The performance of Nosto Recommendations can easily be validated using A/B testing. One tool commonly used is Optimizely.
If you are new to Optimizely, it might make sense to start with A/A testing to get a feel for the software, its conversion tracking and in general to make sure the grouping is correct. Often even with the A/A testing the results favor one variant (see this article).
Please be aware of the following points when setting up testing of Nosto Recommendations
1. Always keep the nosto tagging included in all variants
This ensure that the behavioral data and product updates are tracked correctly regardless of your testing setup.
2. How to control which recommendations are shown to which variation
For the variants that do not show Nosto recommendations, remove the Nosto element divs from the page before Nosto runs. To disable all Nosto recommendations from a page with jQuery for example, you could add a script block just before </body> tag and call jQuery(“.nosto_element”).remove();
3. Make sure your test conversion tracking numbers match your backend information
You should always compare the number of orders and revenue logged by your testing setup to the authoritative numbers from your backend. If the numbers do not match, then you should make sure you understand why. Unfortunately Optimizely doesn’t allow you to register order numbers with revenue tracking, but by linking the experiment to your google analytics account you can actually achieve that as well (https://help.optimizely.com/hc/en-us/articles/200039995-Integrating-Optimizely-with-your-Google-Analytics-Account ).
Common technical problems
- Removing the full Nosto script from some variants which causes problems with recommendation learning, Nosto statistics & Triggered emails
- Just hiding the Nosto recommendation elements instead of removing them. With this setup Nosto always creates the recommendations and of course assumes that they were shown. This causes discrepancies in Nosto’s behavioral data and messes up Nosto’s analytics section, for example recommendation CTR.
- Large unexplained discrepancies in conversion and revenue tracking when compared to the backend numbers. If there are discrepancies you should make sure what is the root cause of those and if possible, fix them before making conclusions based on the results.
If you are considering A/B testing, please consult Nosto Support. We will assist you with any questions you might have.