Overview

ExperimentPilot is a free online platform for adaptive experimentation. ExperimentPilot allows users to design, run, and monitor experiments with complex adaptive treatment assignment policies in a simple online interface.

Why ExperimentPilot?

Researchers or businesses conducting experiments frequently want to condition treatment assignment on information that only becomes available while the experiment is running. Two obstacles make such “adaptive” experimentation time-intensive and costly to implement.

1. Existing solutions for adaptive experimentation usually require users to specify their design using code. This can be a steep hurdle for users unfamiliar with the specific experimentation library, the programming language it is implemented in, or computer programming, more generally.
2. To deploy adaptive experiments, the platform where the experiment is conducted must communicate in real-time with the code implementing the treatment assignment. If the experiment is conducted online, this usually means that researchers or businesses must set up their own web server running the adaptive experimentation algorithm and link it to their experiment platform. Setting up a web server can be a difficult and error-prone task unfamiliar to many users. Furthermore, running a web server can create significant maintenance costs.

ExperimentPilot solves both problems. It provides users with a simple online interface where they can specify their adaptive experiment with a few clicks and without a single line of code. Once specified, ExperimentPilot provisions the adaptive treatment assignment algorithm on its own servers, eliminating the need for users to set up a web server. Users can then connect data collection and treatment assignment by simply embedding a link supplied by ExperimentPilot in their experiment platform of choice.

Supported Methods

Sequential Blocking: To improve the precision of treatment effect estimates, researchers may want to block treatment assignment on covariates known to strongly affect the outcome of interest. However, in many experiments, units arrive sequentially, and covariate distributions are, at best, imprecisely known prior to the start of the experiment. This makes it difficult to specify blocking policies before the start of the experiment. Sequential blocking methods can be used to implement valid blocking schemes for units arriving sequentially, but such methods must be executed in real-time while the experiment is running. ExperimentPilot allows users to choose from a variety of common sequential blocking methods, making it easy for researchers to use blocking in any experiment.

Multi-Armed Bandits: Coming soon...