产品分析师实验设计指南-第1/3部分

By 伊丽莎白Reitmayr

At AG真人旗舰厅在美国,AG真人国际厅做了很多实验,为用户改进产品. Our experiment design guidelines for product analysts give guidance on how to set up those experiments from the analytical and statistics perspective to ensure we can evaluate the experiment as intended. 本指南给出了一些提示,但没有完全涵盖产品管理, 用户研究, 和设计的角度, i.e. 做什么实验. 在这篇文章中,AG真人国际厅将关注在开始实验之前需要做的工作.

This post is the first part of a series in which we publish some of our internal guidelines and frameworks to make the way we work more transparent. AG真人国际厅对您对这些指导方针的反馈很感兴趣——请将其发送给伊丽莎白.reitmar@researchgate.净.

实验目的


实验是最好的方法吗?


Experiments are a very powerful tool in the methodological repertoire of a product analyst because they allow us to causally infer from a treatment (product change) to an effect. 这是比相关分析更有力的证据, 哪一个不允许AG真人国际厅得出因果结论. 所以AG真人国际厅为什么不做些实验呢? 实验是昂贵的, 它们需要大量的准备工作和产品管理方面的监控, 用户研究, 产品分析, 和设计团队, 最终规划实施和解决方案的时间. They also come with opportunity cost: we only have a limited amount of traffic and time to experiment, AG真人国际厅应该确保将其用于最具影响力的变革和创新. 因此, we should choose the assumptions and hypotheses we experiment on carefully based on previous insights.



书中建议的那样 这篇博客, we should only test assumptions that have the potential to provide high user value and which have high risk associated. As we want to minimize the uncertainty of the most impactful assumptions that our experimental hypotheses are based on, AG真人国际厅依赖于“最风险假设测试”的概念(RAT -阅读更多关于这个概念的内容) 在这里). The idea behind the RAT is to test the assumptions that can potentially have a strong effect on the product (high risk). “风险”可以定义为对用户行为的潜在影响, 或者AG真人国际厅不确定这个假设是否成立. If we rely on an assumption that we did not gather any previous insights, the uncertainty is high.

Whether an experiment is the best method to test the assumption depends on various factors such as:

  • 这个实验的费用是多少?

  • AG真人国际厅是否有足够的流量来快速评估实验?

  • AG真人国际厅最终实现测试过的解决方案的可能性有多大?


We add a limitation to our interpretation of “riskiest” in the RAT concept: in case the solution we are testing is associated with very high risk, AG真人国际厅最终不实施它的可能性也更大. 因此, a usability test with mockups might be a better (cheaper) first step to test the underlying assumptions before running an experiment:


AG真人国际厅通过实验来了解AG真人国际厅的用户


AG真人国际厅通过实验来改进AG真人国际厅的产品,更好地满足用户的需求. 因此, we have to make sure that we have a solid understanding of our users’ needs in the specific domain we are experimenting on. 例如, 如果AG真人国际厅想支持AG真人国际厅的用户在AG真人国际厅的产品中发现相关的内容, we should have a good understanding about the different tasks that users are trying to accomplish with our product before we run experiments.

每个实验的设置都应该让AG真人国际厅能够了解用户. AG真人国际厅经常可以把学到的东西从一个语境转移到另一个语境. That's why we want to make sure we test the assumptions about our users in the most direct way possible so that we can update our theories about our users with the new insights we generate via the experiment. 例如, in most cases we should not test two changes at the same time (unless you use a full-factorial设计 — read more in the next part of 这篇博客) because we will not be able to attribute the result of the experiment to the different changes we introduced. AG真人国际厅还应该致力于测试关于用户需求的假设.g., "People don’t want to click like on a story if they dislike the title”) rather than testing specific solutions ("Users will click more on stories if we introduce a dislike button") (read more 在这里).

在实施实验之前需要做的工作


正确地设计一个实验需要大量的前期工作——在编写任何代码之前. The first step for designing an experiment is defining the follow-up action you take in case you gather the evidence you are interested in:
统计学是在不确定性下改变你想法的AG真人国际厅, so the first order of business is to figure out what you’re going to do unless the data talk you out of it ... That’s why everything begins with a physical action/decision that you commit to doing if you don’t gather any (more) evidence." (永远不要从假设开始)

Defining such a follow-up action often requires 用户研究 to make sure we actually address our users’ needs and not only experiment towards moving a certain metric. AG真人国际厅应该清楚地了解AG真人国际厅正在进行的用户旅程, 并在假设的基础上定义一个明确的假设. 在这种背景下, a hypothesis does not refer to the Null hypothesis we define for the experiment (we call this "statistical hypothesis"), 这里AG真人国际厅讨论的是关于用户的假设. 假设通常有以下格式:
"We believe that , and if we provide for them, they will "

应根据量化的预期确定后续行动. This means that we do not only say “we expect a lift in conversion rate” but rather “we expect at least a 5% lift in conversion rate”. This helps to prevent the implementation of marginal improvements and is also important for determining the required sample size (“minimum detectable effect — read more in the third part of 这篇博客).

The following summarizes these requirements based on an example experiment to improve the usability of the bookmarking option on the AG真人旗舰厅 feed:





在书签功能的例子中, we can learn whether it is merely the visibility of the bookmarking feature that prevents users from using it. 如果不是这样的话, we will have to do more research or run more experiments to identify the reasons why users are not adopting it. We might for example have cluttered the feed with too many interaction options such that users feel overwhelmed. 在本例中,AG真人国际厅可以从不同的方向进行测试,以使feed更干净. Bookmarking might also feel like a burden to the user because the feed provides endless scrolling and potentially serves users too much content. 在这种情况下, AG真人国际厅可能要尝试一个更大胆的方向, like making less but more relevant recommendations to users before working on the visibility details of the bookmarking feature.

实验设置模板


作为产品分析师, you will save a lot of time if you ensure clarity about all prerequisites for experiment analysis upfront. We strongly recommend writing down the background/context section of the experiment documentation and to gather feedback from the design, PM和用户研究团队在实验前由工程师实施. AG真人国际厅建议使用 这个模板.

这篇博文的下一部分将集中于实验的设置.
分享