# How to calculate signifiance on an A/B test on revenue

I am having trouble calculating the statistical significance of a split test based on revenue. Most online calculators only look at the number of visitors and the number of conversions, outputting a p-value (and thus measuring the significance). I want to take this to the next level and measure if a test is significant based on revenue, not just the number of purchases.

Here is an example calculator using the "traditional" method: http://visualwebsiteoptimizer.com/ab-split-significance-calculator/

Here' an example to try to illustrate this a bit better:

Pretend you have baseball card store, and you are testing a new version of your website. Version A is the control, Version B is the variation.

With normal split testing, you would measure the number of people who visit both websites, and the number of purchases.

But what if you had more data? Instead of just tracking wether or not a purchase was made, what if you knew how much each customer spent, and what version of the website they saw? Over the course of a month you track purchases from customers depending on the variation they saw. At the end of the month, you want to know - with 95%+ confidence - if the new version of the website resulted in more revenue, not just the number of sales.

Is this possible to calculate in excel? Here's some sample data to use:

Version A 2000 visitors, 500 sales: $123 115 103 177 188 120 114 189 134 80 126 142 144 108 141 139 114 119 181 82 154 114 128 166 163 116 112 180 122 159 109 185 126 178 40 161 101 175 184 150 180 174 159 132 166 154 125 151 126 157 111 210 152 151 130 107 118 105 113 162 118 163 142 124 132 157 128 101 106 167 147 148 158 160 156 168 177 149 155 830 113 100 136 152 176 171 129 158 189 188 156 187 178 78 130 130 171 163 120 164 128 168 145 110 186 189 149 140 142 134 145 189 187 161 164 129 104 167 166 186 142 169 137 60 126 157 174 183 176 166 136 190 170 109 141 125 187 105 178 147 113 137 141 106 60 167 153 155 174 141 117 154 140 118 118 177 187 149 159 151 109 111 115 149 182 138 168 115 171 167 123 178 104 120 121 126 176 154 182 164 139 106 162 139 138 138 166 137 152 182 172 106 120 162 107 117 103 114 162 168 160 122 161 160 138 159 124 179 124 112 138 170 146 137 153 122 163 167 156 150 186 127 153 187 141 187 143 172 184 104 121 104 176 145 102 156 143 167 139 137 125 101 109 183 100 110 127 106 119 179 131 116 166 169 108 141 189 143 109 107 124 108 146 147 158 155 183 140 101 100 111 107 151 177 130 147 186 156 117 162 187 139 131 177 147 146 111 112 142 107 141 164 157 159 169 148 190 165 172 181 190 111 175 190 103 127 132 173 101 178 101 161 160 144 123 189 121 182 147 141 166 144 180 126 161 141 116 152 130 164 147 163 177 126 163 167 135 164 116 104 169 126 175 179 185 137 130 139 131 139 147 123 124 172 130 100 189 111 186 105 106 147 136 122 171 165 151 123 126 132 169 106 158 120 165 144 120 182 131 174 107 127 165 105 139 127 161 103 114 143 184 101 105 185 153 148 142 162 187 167 154 149 158 135 180 179 182 139 104 159 164 149 168 187 131 107 175 111 189 162 133 131 180 101 141 153 139 171 175 124 119 102 130 152 121 159 122 124 123 188 104 130 184 109 126 114 188 100 139 179 187 104 165 169 152 169 127 152 165 177 103 187 167 157 141 150 120 180 101 129 104 157 104 161 120 162 139 110 182 150 105 170 121 160 133 151 116 190 153 189 101 171 184 179 153 188 173 141 164 100 Version B 2,200 visitors, 510 sales:$

144 143 159 194 90 158 149 145 197 192 139 132 467 185 164 159 166 190 159 170 168 132 157 134 151 183 183 180 135 320 128 185 141 184 181 126 126 167 134 120 163 138 345 150 199 156 173 194 183 186 360 165 183 149 143 156 167 174 177 430 258 175 143 183 122 145 184 1029 170 183 126 146 165 193 157 154 199 192 145 199 129 156 157 173 125 127 182 164 159 182 147 126 160 184 193 133 154 127 183 125 189 151 192 182 178 169 189 132 157 142 310 147 158 164 196 155 135 167 146 167 123 172 198 129 135 176 131 120 124 166 157 280 149 165 146 158 128 199 147 198 149 153 157 154 133 195 198 385 192 195 123 158 199 146 163 183 164 173 130 187 168 123 196 120 176 179 197 150 173 161 150 187 193 195 192 164 130 194 180 124 142 180 152 125 186 181 166 181 187 159 127 125 130 179 134 143 148 154 139 161 162 161 147 133 160 143 126 127 130 160 133 156 168 194 164 120 130 168 188 127 141 127 171 150 162 189 163 177 160 145 197 176 134 142 124 137 166 197 188 191 193 178 121 162 134 175 199 146 157 147 127 151 159 140 125 163 197 187 165 155 137 163 139 136 181 151 148 176 178 153 162 151 164 199 147 161 148 185 132 141 157 134 158 145 150 127 140 165 138 163 182 176 173 173 184 196 174 121 137 121 159 145 139 145 191 152 178 163 171 198 175 145 192 126 159 165 128 161 129 134 192 136 175 132 176 144 159 147 131 138 188 134 132 193 183 166 176 170 136 146 139 151 197 173 161 121 194 150 179 120 174 196 139 137 140 148 161 195 157 174 197 193 156 156 183 168 192 146 170 120 145 151 134 173 182 172 167 134 145 132 182 178 158 156 168 170 167 195 159 181 177 126 152 142 134 131 176 139 190 162 195 139 193 172 169 153 184 172 167 154 147 124 141 156 169 146 142 168 147 163 143 154 175 179 175 127 157 129 190 189 175 159 146 183 183 164 154 167 162 156 145 151 181 121 140 145 158 183 185 166 122 136 189 121 191 136 166 192 134 195 153 168 146 133 144 123 162 181 125 181 196 182 165 196 146 168 138 146 144 179 152 149 134 195 158 135 129 189 177 142 137 199 144 193 170 122 184 194 130 146 213 394 128 394 231 495 201 120 90 50

-
Huge lists of numbers are not conducive to getting an answer. It looks like you have a small number of columns and a large number of rows, but you don't post the titles, nor have linebreaks so we can see it the way you do. Please come back to your question to see what it looks like to the rest of us. – Ross Millikan Jul 19 '13 at 3:28
I just made up those numbers if they were helpful, it's not actually a part of the problem. – Jeremy Jul 19 '13 at 3:40
I see two long lists of numbers without any indication of what the numbers represent. No columns, no titles on columns. The numbers are then completely useless. They all seem to be in the range $100-200$, but so what? – Ross Millikan Jul 19 '13 at 3:45

If you have large enough samples the CLT lets you just use the mean, $\mu$ and the standard error on the mean, $\sigma_\mu = \sigma / \sqrt{N}$. You can then calculate your 95% CL through the comparison of the two Gaussians formed by $\mu$ and $\sigma_\mu$ for each scenario in your split test.