As a quick disclaimer, we will focus on Optimizely Web and not on Optimizely Full Stack. You can find more information about the difference between them here.


In essence, Optimizely Web requires less technical knowledge and is more focused on optimizing conversion rates, whereas Full Stack is more focused on product performance.


We will guide you through each step necessary to set up your experiments through Optimizely, a user-friendly optimization tool that will help you achieve your business goals. To give you a quick outline, here are the 5 main steps required to get your experiment up and running.

  1. Target Your Pages
  2. Create the Variations
  3. Target Specific Audiences
  4. Define Your Events & Set Up Your Metrics
  5. Set Traffic Allocations 

Before diving in and getting started with your experiment, I’ll go over some prep work in order to ensure a successful experiment.



Foundation of a Successful Experiment

The foundation of a successful experiment is a data-driven hypothesis and an excellent measurement plan. Doing experiments “by ear” without having data points to back the experiment will result in wasted time and resources.


Here are questions that you need to consider before getting started:

  • What is the hypothesis behind the experiment?
  • What is the key metric which you want to improve?
  • How much do you expect it to improve? 
  • What are the secondary metrics?
  • At what point will you ramp up the population of the experiment?
  • How long will it take to reach statistical significance?
  • When do you consider it unsuccessful?


All of these questions need to be answered in the measurement plan. After you have that in place, you are ready to start creating your experiment!



Prep Work for Your Experiment

When getting started, don’t forget to be specific with naming your experiments.


This is critical prep work for keeping tabs on everything you will test. What I recommend is including the name of the page or URL you are experimenting on in the name and also what functionality is changed.


Also, I highly advise having the hypothesis which led to the creation of this experiment in the description. That way, new project members can go through past experiments and understand the logic behind them.


You are also required to enter a URL or choose a page, but I will get to that in the steps below.



How-To Set Up Your Experiment With Optimizely


Step 1: Target Your Pages

Targeting pages is a key step for a successful experiment. You have two options here, you can either:


A) Target specific URLs on the website






B) Create pages that are categorized URLs that you can reuse in other experiments.



We recommend that you create a library of Pages before starting any experiment


Creating a page is a straightforward process:

  1. Give it a name.
  2. Input an Editor URL. This URL will be displayed when creating a variation in the Editor.
  3. Find a pattern in the URLs that will help you include all of them.

This is necessary when targeting URLs which might have slight variations such as product pages.


A simple and effective way of making sure that you are targeting the right URLs is using the same conditions in a report in Google Analytics or Adobe Analytics.


Step 2: Create the Variations

On the Variations tab, you can access the Original and the Variation. Here you can create multiple variations if you want to run a multivariate test.



You will be able to edit the Variation in the editor only after you have set-up an Editor URL when creating the Page. The Editor gives you the freedom of modifying the design you want the users to see. 


There are options such as modifying elements, rearranging them, adding pictures, writing in-line CSS, and more. You can also create more advanced experiment designs through the editor but that may require the assistance of a developer or knowledge of code.


Step 3: Target Specific Audiences

Depending on the scope of your experiment, you want to think about the segments of users you want to test. 


For example, if you want to change an element on the mobile design, excluding desktop users from your audience will give clearer results and will make interpreting results that much easier.


Another example would be Google One Tap Login, which is not supported on all browsers. You can create a specific audience that targets only supported browsers. That way, the results you see in Optimizely will be more relevant to what you are testing.


On the audience creation page, you can choose dimensions such as browser, device type, language, and even location as part of the standard dimension list.



Sometimes you can’t just rely on the standard dimensions given by Optimizely. What you can do is define your audience by custom events and pages you have already defined. This can be especially useful if you want your audience to be defined by a complex criterion.


Step 4: Define Your Events & Set Up Your Metrics

Next, you need to define the events which make up your metrics. These can either be pageview events, custom events, or click events.


Pageview events are set up automatically as soon as you define a page. Click events can be defined from the editor or the implementation tab while the custom events require custom javascript to be set up (these can be form completions, impressions, or anything that doesn’t qualify as a click).



Depending on what you are trying to achieve with your experiment, you need to have metrics that are relevant to your goal created. Optimizely gives you a wide array of options when creating a metric, ranging from page views to custom events.


Setting up the metrics you want to measure is a straightforward process:


From the metrics tab, pick one of the events or page views you have created and press the plus sign to add it as a metric.


After that, you need to decide on the winning direction, numerator,  and denominator for your event. Make sure that the winning direction matches what you are trying to achieve. It is set to increase by default, but depending on the metric, you might want it to decrease.



The numerator can have multiple values, from unique conversions to the bounce rate or exit rate of a page.



The denominator is the way your metric measures your numerator. It can have one of three values: per visit, per visitor, or per conversion. 


Some numerators give you the option to change the denominator and some are locked in place.  For example, unique conversions can only be at a visitor level.


After creating your metrics, make sure to assign a primary metric that is the most relevant to what you are trying to achieve. This main metric will make or break your hypothesis. Optimizely will try to achieve statistical significance for your main metric first and then the secondary metrics.



The main takeaway from this step is that you need an accurate measurement plan (which I will go over how to build one in another article) in order to choose the right metrics to prove your hypothesis. Just adding metrics on the go without having a clear plan will negate your efforts up until this point.



Step 5: Set Traffic Allocations

At this step, you can decide how much of your website’s traffic will be allocated to the test. You can also set exclusion groups so that there is no overlap between this test and others you might be running.


Depending on the scale of the change, it might be a good idea to start with a smaller percentage of total traffic before ramping up to a bigger chunk of your user base. The decision of ramping up traffic can only be made based on the performance of the metrics you have set in place for the experiment.


In terms of distributing traffic among variations, it is good practice to distribute equally. Optimizely also gives you the option to use “Stats accelerator” which uses AI to reach statistical significance faster for winning variations by dynamically distributing traffic.




After going through these five essential steps, you are now ready to start your experiment. Depending on how significant the change in the variation is and the amount of traffic you have on the page you are testing, the test might need to run anywhere from a couple of days to a couple of weeks in order to achieve statistical significance.


Even though your experiment might achieve statistical significance in two days, it is good practice to let it run at least one week to account for weekday seasonality.


If you have any questions on testing or experimentation or want to learn more about our A/B Testing & Personalization service, contact us today.

About the author

Robert Bobosa

Robert Bobosa
Senior Product & Optimization Analyst

eBook: Understand Your Customers

Cognetik eBook: Guide to User Journey Analysis


Related Articles