5 Tips for Fostering a Culture of Product Experimentation

It’s time to take the guesswork out of product optimization—and empower your teams to maximize the business value of every software feature and product change.

Through product experimentation, your teams can test small, incremental changes at scale. Experiment results can then drive smarter decisions about which features to ship, which audiences to ship them to, and the overall customer experience. 

Over time, product experimentation can unlock continuous product improvements, wider margins, and greater competitive advantage. That’s why successful brands like Amazon, Booking.com, Facebook, Google, and Microsoft perform tens of thousands of product experiments each year.

These companies are the exception, however. According to Stefan Thomke of Harvard Business School, most companies run less than “a few dozen” experiments per year that have little impact.

If product experimentation is so valuable, then why is it so rare? The reasons are varied and complex—but, if you trace them back far enough, many of them are rooted in the same cause: culture.

In this post, we’ll provide five tips for building a strong culture of experimentation at your organization.

Building an experimentation culture

Companies that succeed with product experimentation all have one thing in common: they foster a culture of experimentation. They encourage their employees to stay curious, take risks, and learn from failures. They stress that good ideas can come from any person or department, and they understand that fear is anathema to innovation.

Building a culture of experimentation is not always easy. It means leaving old habits behind and rethinking processes that may have been in place from day one. It takes time, intent, and steadfast commitment to create the atmosphere and buy-in for teams to learn and adopt a new way of building products.

If you start with the right plan and follow best practices, however, you can minimize the difficulty and rise to the challenge.

1. Set clear goals and objectives

Successful experiments start with a clear, agreed-upon goal that aligns with your business objectives. Setting specific, measurable targets—like "increase checkout completion by 15%”—can help your business, technology, and product teams collaborate toward success throughout the product workflow.

You’ll also want to set goals for your experimentation program as a whole. For example, the more experiments you run, the more opportunities you have to test and identify a product feature that performs exceedingly well. Similarly, more experiments can also help prevent shipping changes that underperform. Thus, we suggest tracking the number of experiments your team runs per month. First, determine your baseline, then set a goal for the number of experiments you’d like to run in subsequent months and quarters. See, for example, how Ritual increased its experimentation rate.

You could also set goals around the time it takes from running an experiment to shipping the winning variation of an experiment to all intended users. You’ll want to shorten the interval between experimenting and releasing over time. Over time, your team will learn how to select better target measures and design their experiments to return clearer outcomes.

2. Foster a data-first mindset

Data is the language of product experimentation. You can generally express the problem an experiment is meant to solve without “speaking” data—i.e., “customers don’t stay on the app long enough.” But to design, run, and/or interpret the results of an experiment, you’ll need some degree of data literacy.

Through data literacy initiatives, you can help everyone at your organization—including business teams—grow more confident in their ability to run experiments and explore new ideas. Over time, you can work to instill a “data-first” mindset throughout the organization, encouraging (or, when appropriate, mandating) teams to inform every decision with data.

Culture flows from the top-down, so it’s critical to obtain buy-in from management. Senior executives can provide the authority needed to drive adoption of the data-first mindset and other cultural changes. They can also expedite experiment approval and ensure the company acts on the insights derived from experiment results.

Use caution, however, as the inclusion of senior managers in experimentation can be a double-edged sword. According to a Harvard Business School study, the involvement of senior management may lead to bolder experiments and bigger wins—but can also undermine the cause-and-effect learning that is essential to product optimization.

You can run all the experiments you want. But if the highest-paid person's opinion (HiPPO) reigns supreme in your organization, and senior managers reject data that conflicts with their intuition, all those experiments will have been done to no avail.

Thus, it's critical for executives to endorse a culture of experimentation—even if those experiments disprove their opinions in some cases.

3. Implement rigorous experimentation processes

Encourage those who are just starting with experimentation to begin with simple A/B tests that measure a single variable. Just start measuring each new feature rollout without overthinking it—build the habit with low stakes. From there, gradually progress toward more advanced tests.

As your experimentation practice grows, develop rigorous documentation and detailed policies. Be sure to explain and create guidelines around recommended audience size and test run-times, statistical validity, and ensuring reproducibility. These steps will give your team a support structure to guide their thinking—while also improving the accuracy, reliability, and usefulness of your experiment results.

4. Promote cross-functional collaboration

Cross-functional collaboration is critical to developing a healthy workplace culture in any context, and building a culture of experimentation is no different. As we mentioned previously, setting clear goals helps keep your teams aligned. And promoting data literacy across all teams allows them to “speak the same language,” so they can more effectively leverage experiments throughout the product workflow. 

Companies with successful experimentation practices take collaboration a step further by establishing cross-functional teams, where people from different departments come together to work toward a common goal. 

In the context of software product experimentation, collaboration is necessary between software developers, product teams, and data scientists. Engineering and product leaders may also be involved, to a certain extent. Be sure to consider these roles and support them to working together by setting shared goals and granting access to the necessary tools as you build cross-functional teams and foster larger cultural changes.

To be successful, cross-functional teams need the autonomy to function as mini business units, as well as the authority to design, conduct, and make decisions based on experiment results. These factors allow cross-functional teams to move through the product workflow with a speed and efficiency many times greater than the larger organization.

Other steps to improving cross-functional collaboration include breaking down silos between departments during feature planning, establishing regular communication channels, and involving key stakeholders early and often to prevent missteps and work smoothly as a team.

5. Invest in experimentation infrastructure

Your efforts to foster a culture of experimentation will ultimately be limited by the capabilities of the tools you provide to your teams. Unfortunately, many experimentation tools are siloed from the development process. This creates friction and slows the process of acting on experiment results. 

LaunchDarkly aligns your experimentation tooling with your software delivery pipeline, integrating both into a single implementation. It layers on top of feature management—where new features are already being controlled for launch—making it easy to add controlled experiments and ship winning features based on the results.  

Further, you’ll need a solution that is scalable and performant enough to meet your needs today—and continue meeting them as the number and types of experiments you run and people running them expands. Look no further than LaunchDarkly, which enables safe innovation at scale while reducing complexity, delivering full transparency, and keeping everything tied to your development efforts.

LaunchDarkly empowers more teams to run more experiments—and get more out of them—by providing easy access control, self-service, full-stack integration, fast remediation, and powerful, intuitive visualizations. It also enables you to drill deep into every feature and product change to surface insights beyond basic events and clicks.

Finally, LaunchDarkly provides clear lines of sight into both user experience and back-end performance, so you can continuously measure and improve the business value of your products. It’s the ultimate catalyst and the perfect foundation for building a culture of experimentation.

By combining these best practices with LaunchDarkly’s powerful experimentation capabilities, you can cultivate a spirit of curiosity and innovation across your teams. As this new culture of experimentation begins to flourish, it will begin yielding extraordinary results—including the continuous improvement of every feature across every product, plus the ability to validate and quantify the business value of every product change.