From Failing to Learning
The Birth of a Culture of Product Team Experiments
By: Elisa Zanoni and Matheus de Lima Gonçalves Leite, from the Sensorama Design team
Disclaimer
Before we begin, we would like to point out first that all the processes and approaches that we will highlight throughout this article are NOT an absolute truth that makes sense in all scenarios. So, be very mature to understand and recognize the context in which your team, product or company is inserted before implementing something that you can discover here.
And of course, on the other hand, this article IS ABOUT our learnings while trying to build a culture of experimentation to ensure that at the end of the day we can deliver a product that generates value for users and businesses. It’s always about that.
Knowing this, we can get started. :)
Context
Together with the product squad, we are responsible for designing a new solution for the trade promotion management (TPM) system for a large beverage company, in an agile way, following the business and engineering needs.
The expected result at the end of the project is the delivery of a more suitable tool for everyone who manages product promotions that have taken place or will take place in supermarket chains throughout Brazil through this system, making possible a more efficient and effective routine.
It is precisely this expected result, which led us to the following question: how can we mitigate risks as much as possible and validate the ideal solution to achieve business OKRs and facilitate the user’s day-to-day?
To answer this and countless other questions that arose throughout the project, we used the Opportunity Solution Tree, a method created by Teresa Torres that we will detail below.
Opportunity Tree
Presented to the world at the 2016 Productized Conference by Teresa, it is a tool to help product teams design solutions that deliver higher value results for the business, organize discovery and facilitate the understanding of all those involved in the project about the objectives and what path will be taken to reach the desired result.
The tree is composed of the following topics: objective, key result, opportunity, solution hypotheses, solution, feature. Below we describe in steps how this tool works.
Step 1: To start building our tree, we help the Business team to define what the objective would be, what they really want to achieve as a company.
Step 2: We set the KR to compose the top of our tree (highlighted in strong blue). It is essential to have metrics that can measure and understand if the objective was actually achieved.
Step 3: After we had a clear direction, we defined our opportunities based on pains and needs raised in interviews, dynamics carried out in workshops, and data from the support team. That is, something that brings us benefit and helps us achieve the goal. At this time, on a more macro level, the opportunity does not necessarily need to be related to the team, product or company.
Step 4: Based on the understanding of the opportunities, we held a co-creation workshop with the business, engineering, design and users team with the objective of generating as many hypotheses as possible for solutions and continuing to feed our tree. At this time, we were looking for quantity and encouraging different ideas to deconstruct as much as possible the biases on how to achieve this goal, for that, we resorted to “What If…”, a method that stimulates creativity and allows us to visualize several different ways to possibly solve a problem.
Some data about our workshop: there were more than 25 participants over 2 meetings, we held 1 per KR, and more than 80 solution hypotheses.
Right after this moment of ideation, we prioritize the hypotheses through the DVF method (desirable, viable, feasible), to later be validated or invalidated through experiments, as we will see below.
Experiments
They are concept tests that must be performed simply, quickly and at the lowest possible cost to validate or invalidate something. In this case we are referring to a hypothetical solution, and they allow us to learn and investigate desirability and feasibility before investing any considerable effort, whether money or time, for example.
There are several methods to carry out this investigation, and each of them could easily be a single article, we will not cover them all here. However, regardless of which method, even before starting to make the experiment viable, it is important to define and align with the entire team what the parameters will be defined to validate or invalidate the hypothesis. In other words, what would be successful for this experiment and makes it stop being a hypothesis and can become a solution to achieve the established objective.
Now that we have given a macro context about the experiments, let’s go to the ones we performed and adapt different methods to validate or invalidate our hypothesis according to our context*:
*Our entire engineering team was committed to another major delivery that we had worked on previously, not involving engineering was one of the requirements.
The hypothesis: “What if…through reading the sell-out we allowed users to approve a promotion carried out by the supermarket chain? If approved, this promotion would be automatically registered.”
Our first validation attempt: Fake door
It is basically the creation of a banner or button, displaying a call to a new functionality, without the functionality being actually ready. By clicking, the user proves the interest in that feature.
We sent an email to the user base indicating a weekly summary of all promotions carried out by the networks that were not registered in the system and that if they recognized that promotion, it would be automatically registered in the system. To learn more, he should click on “see all promotions” which would redirect him to a form informing that we were still working on that functionality and a field to leave the email if he was interested in the functionality.
Our metric: 70% of our users will leave email to receive notification when the feature is ready.
In the course of the experiment, we identified that we were not learning as well as we should, and we quickly thought of another way to experiment.
Second Validation Attempt: The Wizard of OZ
Tools are used to give the impression to the user that he is performing an automated task, but behind there is a person operating without his knowledge.
We sent a message via WhatsApp to the user base as if it were a company chatbot, indicating that we had identified some promotions in supermarket chains that were not registered in the system, and if the user recognized that promotion, it would be automatically registered in the system. To learn more, he should click on the link we sent at the end of the message that would redirect him to a form informing that we were still working on that functionality and a field to leave the email if he was interested in the functionality.
Our metric: 70% of our users will recognize the actions and choose to automatically register the promotions in the system.
This is an experiment that gives us more autonomy and agility to create and put to work. But because it is an experiment that requires manual effort, it ends up taking up more time to execute. It was an experiment with greater engagement than the previous one, we believe that perhaps because it was on a platform that is part of the daily lives of users, in addition to being a notification via cell phone. But still, we were not able to meet the minimum number of participants to validate or invalidate our hypothesis.
Third validation attempt: Survey
It is a type of quantitative investigation that can be defined as a way of collecting data through the opinions of groups of individuals.
As our isolated experiments (so far we talked about two here but we were running others in parallel) were not fulfilling their main objective — to learn and support us in decision making if in fact that hypothesis of solution contributes to the achievement of our objective, we chose to consolidate all the prioritized hypotheses, and we sent a form through Typeform, with open and closed questions, for validation or not in a unique way.
Our metric: 70% of our users will answer B in question number 3 (note: we had different metrics for each group of questions on the form).
It was the experiment with the highest number of engagement, but we also did not reach the minimum number of responses that would support us in making an assertive decision.
Conclusion
The main lesson that remains is that, in fact, if we are going to operate with a culture of experimentation, we have to have a safe environment for errors. And that goes a long way in having the entire team aligned and committed to the main objective, the one we started working on at the top of our tree.
As Peter Drucker would say: “Culture eats strategy for breakfast”.
The truth is that this was our real experiment, containing other small experiments to validate or invalidate solution hypotheses along the way, giving birth to a culture of experimentation in the product in which we operate. After all, when it comes to product design, we need to understand that our purpose is not only to make the user’s life easier, but also just as importantly, it is to generate value for the business.
The tree is alive and was built with several people and teams, so we couldn’t help but thank everyone here who made themselves available and contributed in some way.
In particular (in alphabetical order): Daniel, Fernanda, Heitor, Katherine, Larissa, Nana, Otavio and Sergio for co-creating with us a more efficient and effective product to manage promotions in supermarket chains throughout Brazil!
