Set up an A/B test

A/B testing is a great way to compare versions of your site to see if one is more popular with users than another. With mPulse's A/B test integration, you can compare your tests for performance. All you need to do is tag the code with the name of the currently running test and let mPulse report on each test for you.

Measurements that fall into an A/B test bucket do not affect your main site measurement. So, you can use your main site as the control or baseline measurement for separate tests on new features. This means that the total beacon count on your main site will be lower than first expected. There is only one A/B variable available per app, but you can assign it many values. It's a good idea to keep the different values assigned as small a number as possible for easier analysis.

How to

  1. Define a test name in your page.
  2. Define a JavaScript variable in your page that contains the name of a page group. For example, HOMEPAGE.ab_test = “test home”. You can namespace the variable if you want to.
    mPulse uses these regular expressions to confirm an A/B test name, which you can use as well to verify your code:
    /^[a-zA-Z0-9_ -]{1,25}$/
    Note: Test names cannot exceed 25 characters. Use alphanumeric characters, dashes, underscores and spaces only.
  3. In mPulse, click Central.
  4. In the left pane, click Apps, then double-click the app that you want to use for your test.
  5. Click General.
  6. In the A/B (bucket) tests variable name field, enter the test variable name, then click OK to save the configuration.

Next steps

To see your test results, use the A/B Test filter in these system dashboards:
  • App Error List
  • App Error Summary
  • Dimensions
  • Metrics
  • Overview
  • Summary
  • Waterfall
  • What-If