In data-driven product development, A/B, or split testing, is key. While A/B tests are routine across many industries and product types, with numerous tools and learning resources available, running reliable A/B tests remains difficult in some contexts, such as B2B recommender systems. These systems are essential for optimizing customer experiences and driving business outcomes, yet they present unique challenges that diverge from traditional testing setups.

In this talk, I will provide practical insights into the complexities and unexpected hurdles we've encountered during our A/B testing journey for Product Recommenders at METRO, and how we've addressed them. We'll delve into the hard lessons learned about integrating recommenders across multiple touchpoints in a complex ecosystem, which complicates data collection and measurement quality, as well as the implications of a smaller user base. Additionally, I'll outline key do's and don'ts for conducting reliable tests in these challenging environments.