Frequently Asked Questions
What is a field experiment?
A field experiment, like a medical trial, applies the scientific method to experimentally examine the effect of an intervention in the real world. Economic experiments randomize subjects into treatment and control groups and compare outcomes between these groups. Randomization is what guarantees that if we observe a difference between a treatment and a control group, that difference must be caused by the treatment. Randomization also allows you to control for several potential explanatory factors such as age, gender, and income level. This enables you not only to understand what works, but also why. Field experiments allow you to discover the most efficient way to allocate your resources.
What can our organization expect from using field experiments? Four Key Benefits:
Business insights from science: field experiments - like medical trials - are the "gold standard" for learning about the world. Harness the power of the scientific method to improve and revolutionize your business.
Bespoke support, for free: the Institute's economists are not consultants: they won't charge you for their time. Working with you allows them to generate original research and publish it in top academic journals.
Minimal distraction: field experiments are set up to take place organically within your business activity. Our scholars will take care of the field experiment from A to Z, with minimal/no interference with your regular business activities.
Mastery with experimentation: Every large successful company, from Google to Walmart to IEA, routinely uses randomized experiments and behavioral economics to improve their business. At the Institute you will learn how to easily incorporate these methods into your core management practice.
How much does it cost to participate in SIFE?
The Summer Institute on Field Experiments is free of charge for both scholars and firms. Scholars will receive a travel allowance.
We already use surveys and focus groups to inform our decisions. Why would we want to run an experiment?
Surveys and focus groups are an excellent way to learn about people’s intentions. However, what people say they would do is often very different from what they actually do. This is because people are affected by the context in which they make a decision, by the amount of attention they put in that decision, and by their emotions. Surveys may not capture these factors because people often do not know ex-ante what they will pay attention to or they do not consciously realize that context clues or emotions affect their decisions. Experiments allow us to observe how different interventions affect behavior in natural environments.
Can't we use econometric and statistical analysis to analyze our existing data to understand what works and what doesn't?
Analyzing existing data is an important component to research, but falls short on establishing a causal relationship. Consider the following example from a development office: an organization chooses to add a small gift to their solicitation and after one year observes that average donations have increased. This is good news, but why is this happening? The improvement may surely be due to the gift itself, but without proper randomized tests one cannot establish causation. It is possible that a third, unobserved factor is driving this effect (e.g. better general economic conditions). If this were the case, the increase would have happened also without the gift, thus spending resources on gifts was inefficient. It is also possible that donors’ responses to gifts vary, but the lack of proper randomization hides this effect. For instance new donors may be attracted by the gift, but existing donors may consider it a waste and thus reduce their giving. The average effect is positive, but the organization is losing donors, which may backfire in the long run.
We already use A/B tests, what is the difference with field experiments?
Many companies, particularly young startups, routinely use A/B testing: from choosing the right user interface, to choosing the best day to send out newsletters, to finding the right wording for communications with clients. However, A/B testing alone can hardly be used for anything besides optimizing business practices at the margin. Without a motivating theory, A/B testing alone can merely reveal what works, but are silent about why things work the way they do. This limits the generalizability of A/B tests, as well as their predictive power. To truly unleash the power of randomization and use experiments to address core business issues, A/B tests need to be informed by theory. The combination of tested (and testable) theories about human behavior and randomized controlled experiments designed to falsify these theories is what allows the most successful firms in the world to use experiments to gain a deep and transformative understanding of their business. It is no coincidence that major firms and institutions all have dedicated teams of economists that closely work with their data scientists to understand the economics behind their businesses. SIFE provides you with an opportunity to gain the same insights, for free.
Will the name of our organization be on the academic paper?
It is your choice whether the name of your organization will or will not appear in an academic paper.
Can I be a coauthor on the academic paper?
We are happy to include you as a coauthor, if you wish to.
Will you acquire personal identifiable information?
Like medical experiments, economic experiments need to be approved by an Institutional Review Board (or Ethical Review Board). The IRB process is aimed at guaranteeing the safety of participants. In the case of economic experiments, safety typically means privacy. To receive IRB approval, two general conditions need to hold: 1) the experiment does not involve deception of any kind; 2) no personally identifiable information is retained. We abide by what academics call data de-identification: while we retain individual-level data that is useful for research purposes (e.g. age, gender, income, level of education etc.), we try to design our experiments so that no personal identifiable information is collected (e.g. names, addresses, phone numbers, social security numbers). If the nature of the experiment requires acquiring some sensitive information, we will ensure that the data is de-identified before any dataset is built.
If we run an experiment, will the scholars own the data?
Absolutely not. We routinely prepare with our partners a Data Usage Agreement— a contract that specifies that the organization remains the sole owner of all data generated. The partner University, however, will own the results and analysis performed on the data generated by the experiment. This is to guarantee the academic freedom of publishing novel results in scientific journals.
What is the timeline of an experiment?
Economic experiments are set up to take place organically within your business activity. How long the experiment is run and when it is implemented will be decided in its planning, with all partners involved.
Will experiments "poison" my pool of customers?
Firms and organizations are rightly concerned that some "bad" treatments may drive some customers away. While possible, experiments are set up precisely to avoid applying a "bad" intervention to all customers. Only a subset of your customers will be exposed to new treatments and once we find the most effective way to solve your problem, you will be able to use the solution with your entire pool. As the proverb goes, it is better to be safe than sorry.
Further questions? Please email firstname.lastname@example.org.