Starting your first A/B experiment

This section of the guide covers designing and implementing an A/B experiment. If you're creating a multivariate experiment, review the multivariate section of the guide.

•Step 1: Identify experiment pages
•Step 2: Add tags to experiment pages
•Step 3: Preview and start experiment
•View your reports
•Stop the experiment
Before you can do anything, you'll need to access Website Optimizer. You can do so by following the steps below.

1.If you're using Website Optimizer within AdWords, sign in to your AdWords account and visit the Campaign Management tab, then choose the Website Optimizer subtab. If you're using Website Optimizer through the standalone site, just sign in there.
2.If this is your first time accessing the tool, you'll see an opening page with links to some useful resources. Click the Get started button to continue.
3.You'll reach the Experiment List page, which displays a summary of all your experiments. If this is your first experiment, your list will be empty. Click Create new experiment to proceed with setting up your experiment. Since we'll be creating an A/B experiment in this section of the guide, click the Create link next to A/B experiment.
The Website Optimizer tool presents a four-step work flow for setting up a new experiment. As you finish each step, this page will track your progress and display basic information about the new experiment. To begin, click on the Step 1 "Continue" button.

Step 1: Identify experiment pages


--------------------------------------------------------------------------------

You need to supply three pieces of information here:

Experiment name: For your first experiment, names aren't quite so important, so just enter something simple like "My first experiment". For future experiments, you'll want to choose something more meaningful that will help you easily distinguish between all your experiments on the Experiment List page.

Test pages: Enter the URL of the original test page you chose during Step 0. As shown in the example on the Step 1 page, the URL you enter should not contain any information after the page's file name (such as index.htm or productpage.html). If you include query parameters, they will be ignored. Please also enter the URLs for each alternate page you have created. Remember, each alternate page you create, needs to be saved at a unique URL in order to be used in an A/B test.

Conversion page: Enter the URL of the conversion page you chose during Step 0. Again, the URL should not have any extra information after the page's file name.

When you're finished entering this information, you're ready to move on. When you click "Continue", Website Optimizer will try to validate the URLs you entered. If you leave either the test page or conversion page field blank, or if Website Optimizer got an error trying to access either page, it will generate an error message. In most cases, public web pages will not cause any validation problems; if it is, however, you will be given the option of continuing the experiment setup without URL validation.

Once Website Optimizer successfully validates your page URLs, it'll send you back to the experiment work flow page. As you will see, Step 1 has been successfully checked off and your test and conversion pages are listed. This work flow page is an important tool to help you track your progress through the setup and keep track of scope of your experiment. Before you launch the experiment, you will be able to revise any of the setup information from the work flow page.

Step 2: Add tags to experiment pages


--------------------------------------------------------------------------------

Who will install the tags?


The first decision you need to make in this step is whether you'll install the HTML script tags or whether you'll hand that task off to someone else. This task needs to be done by someone familiar with HTML and with the structure of your webpages. If you choose to hand the task off, you'll need to communicate some specific information to that person.

Regardless of who installs the tags, Website Optimizer provides specific script to be installed, as well as detailed instructions available in our Installation Guide. Be aware that this script sets a cookie, so you may wish to ensure that your site's privacy policy covers the setting of cookies.

Install the tags

If someone else will install the tags, Website Optimizer will give you a link to that you'll need to send to the person who'll be installing the tags. Stay in touch with the colleague who's installing the tags. They'll need you to provide the following information:

•Exactly which elements on the test page will be varied. They need to mark these elements with "page section" tags.
•A page section name for each element. Later, these names will help you identify the element in question. For this experiment, the names "Headline" and "Image" should be sufficient.
Make sure your colleague notifies you once they've installed the tags.

If you will install the tags, follow the Installation Guide, which provides complete instructions. That guide goes into more detail, but we've given you a brief overview of the task below.

There are two types of script that needs to installed on your pages. Although it looks complicated, each set of script serves a purpose:

•First, there's the control script. Among other things, the control script makes sure that the experiment variations are switched randomly and that all variations are displayed an equal number of times. For this experiment (and in most cases), place the control script immediately after the tag. You'll need to install the control script just on your original test page. For this experiment (and in most cases), place the tracking script immediately before the closing tag in each page.
•The second set of script is the tracking script. It ensures that visits to both the test page and the conversion page are tracked by Website Optimizer for the experiment. For this experiment (and in most cases), place the tracking script immediately before the closing tag in each page. You'll need to add the tracking script to your original test page, each of the alternate variation pages you've created, and your conversion page.
Validate tag installation.

Once you've installed all the tags, validate the tags within Website Optimizer. Our validation tool examines your pages and verifies that the tags have been installed properly. If the validation tool detects any problems with the installation, you'll need to fix these before continuing. Website Optimizer won't let you go on to the next step without validation.

The validation tool will check for the following potential errors, which you will need to fix before continuing:

•Was any javascript found on the test page?
•Were any page sections defined on the test page? The tool looks for beginning and ending tags.
•Was tracking code found on the test page?
•Was tracking code found on the conversion page?
There are two methods of validating your pages. The first is to provide the URLs for your test and conversion pages. If your pages are externally visible, Website Optimizer will access them and provide any errors as described above.

Alternatively, if Website Optimizer can't access the pages on your live site, you can also upload HTML source files for your test and conversion pages. This is helpful if your pages are part of a purchase process, behind a login, or inaccessible for some other reason. Simply save the HTML source of your pages, upload the files, and Website Optimizer will validate them.

If you're unable to validate the tags using either method -- for example, if you're using a dynamic webpage that Website Optimizer cannot view -- you can opt out of the validation step. However, please be sure that the tags have been installed correctly on your own if this is the case.

Once your pages pass validation, select the Tags have been added to the pages option, and click Continue to finish this step.

Step 3: Preview and start experiment


--------------------------------------------------------------------------------

At this point, the hard work is done. You've tagged your pages and created new variations. Now it's time to set the experiment off and running. But before you do, make one final check of your experiment settings. Once you start the experiment, you won't be able to change the parameters of the experiment, so be certain they are as you intend.

If you do find you need to update your settings, though, click the "Back" button to return to the experiment work flow page, and click the "view or edit" link on Step 1. Keep in mind that if you change the page URLs now, you will need to go back through Step 2, reinstall the code on the new pages, and re-validate.

Percentage traffic. Remember earlier in the experiment preparation you were supposed to decide how much of your traffic to include in the experiment? Here's where you enter that value. The more traffic is included, the faster the experiment will run.

Variations. Here you get one last chance to preview the alternate page variations that will be displayed to visitors during the experiment. If you need to change any variation, return to the experiment work flow page (click the "Back" button) and click the "view or edit" link.

Ready, set, start your experiment!

Once you click "Start", you'll return to the experiment work flow page. Step 3 is checked off, and now there's an additional section describing the progress of this experiment, including estimated duration and the number of impressions and conversions tracked during the experiment. Though your test page will start showing different variations immediately, there's a delay of about an hour before your reports will begin displaying data. The progress of the experiment, and the estimated duration, depends entirely on the amount of traffic seen on your test and conversion pages.

Once you start seeing some impressions and conversions recorded, the reports will start to have some preliminary results. Click "View report" to see the experiment results.

View your reports


--------------------------------------------------------------------------------

Once the experiment is launched, impressions and conversions will be recorded almost immediately. You can view this data on the reports page as soon as it's recorded. Plan to check that impressions and conversions are being recorded soon after starting; there are some tagging errors that are not revealed until an experiment is actually running. If you're not getting any impressions or conversions, check the troubleshooting guide for some suggestions on what might be causing this error.

Until a minimum amount of data has been collected, you'll likely get a message like: "We have not gathered enough data yet to produce a complete report with analysis. Please check back in a day or so."

This message may be accompanied by a table/graph combination showing the performance for the individual performance of each variation, showing which variations are most effective. For more information on how to read this report, see What do my reports mean? If you love numbers and want to learn specifics about how we derive results, see the Technical Overview.



As the experiment progresses, the Website Optimizer will identify a "winning" variation. How long this takes depends mainly on how much traffic the experiment pages receive, as well as how similar the variations are. For example, if your experiment has run beyond the estimated duration, it's getting lots of impressions and conversions, but no variation is emerging as a clear winner, it's likely that visitors are not experiencing strong reactions to any of the alternate variations you created. It's probably time to stop the experiment and try some bolder variations.