Ever wonder if we do AB Testing, aka Experiments?
Yes, we do experiments in our app! Most experiments are run by the Growth team on the SaaS product, and the majority are done on our registration, trial and purchase flows. We also collaborate with stage groups and run experiments around increasing the visibility, usability and learnability of a feature. These experiments are ultimately aimed at improving KPIs that tie back to revenue. Things like increasing engagement with the product, increasing the speed at which a customer uses a feature or upgrades their account, or even increasing the number of stages used by a namespace.
Experimentation is a way to validate and measure improvements to a digital product against particular goals. Basically, we have an idea that if we change something in our product, our users will respond to it, and the metric that we want to impact will be affected by that. Experimentation works well when there's a large number of users and a good baseline product (see the When is an experiment not a good idea? section).
Our vocabulary is a bit different when we're talking about experimentation and UX, so here are a few important terms and how we use them here.
The Growth section has groups related to Conversion, Activation, Expansion and Adoption. The things these groups work on relate back to the definitions above, but they aren't exactly the same.
Use an experiment when you need to quantify the impact of product changes (eg. click rate, conversion rates, other metrics) or when you need to validate a solution.
There are some situations where we'd prefer to make a change to the product and roll it straight out to product. These are:
Much of the product design process for Growth will remain familiar, but there are some differences. There are also considerations and concepts we need to understand and follow.
No, they don't. This is because experiments tell us what is happening while user research can tell us why something is happening. We do both problem and solution validation research in Growth. Here are examples of things we might ask:
Experimenation is an awesome way to improve your product. There are a lot of considerations though, so think through your goals and constraints to ensure it's the right approach.