What is Realized Performance in Experimentation?

Evolv AI offers a number of ways to understand how your optimization is performing overall. One of these indicators is realized performance, which reflects the impact of all of your testing over time compared to having not run any test at all. This is an important metric in determining the return on your optimization efforts. 

When optimizing a conversion rate, like add-to-carts or checkouts, the realized performance is the relative change in the number of conversions as a result of running your tests.

Additionally, when a conversion has an average value associated with it, like completed orders, you can multiply that value by the conversions gained or lost to calculate 
an estimated value impact.

How Realized Performance is Calculated

To understand this metric, we need to start by looking at the two distinct groups in your optimization: those allocated to test experiences (non-control combinations), while the remainder goes to the control.

Let’s say your website receives 100,000 sessions in a month and your project is configured to evenly allocate your traffic across 10 different combinations at any given time.

This would result in 90% of your traffic receiving a non-control combination and 10% of your traffic receiving the control.

Each of these two groups has a conversion rate associated with them. In the example above, only 100 out of 10,000 sessions that received control ended up converting, resulting in a 1% conversion rate.

The non-control group saw 90,000 sessions and only 945 converted, a 1.05% conversion rate.

If those non-control sessions were sent to control, that 1% conversion rate would result in only 900 conversions!

To calculate realized performance, take the difference in conversions for your non-control sessions, given the non-control conversion rate, and the control conversion rate, and then divide by the number of conversions you would have had if all traffic converted at control’s conversion rate:

• Difference in conversions: 945 (non-control) - 900 (if sent to control) = 45
• Expected conversions if all traffic went to control: (100,000 * 1%) = 1,000
• Realized Performance: 45 / 1,000 = 4.5%

Using Realized Performance in Business Decisions

Realized performance is an essential metric for any business running an optimization.

• It ensures that you have a clear read on if your optimization efforts are moving in the right direction.
• It provides a straightforward way to measure the return on investment for your optimization effort as a whole. 
• It allows for continuous monitoring of experimentation efforts, ensuring strategies are continuously refined and enhanced.

For more on Evolv AI reporting, check out our latest webinar: How Evolv AI measures impact through Continuous Optimization