Here are some quick tips on how to make better decisions on your ab tests with statistics
When you run an experiment or analyze data, you want to know if your findings are “significant”. You essentially want to make sure your results aren’t due to chance instead of the variable your testing on (example: message copy). Running a chi square test is what I recommend and can be done without pulling out the good old pencil, paper or dusty TI-84 from high school. Here’s a reliable handy chi-square calculator that I use.
HERE ARE THE STEPS:
- Determine the metric. In this example I’ll pick CTR
- Calculate your CTR (clicks / sends) for the two test variants you’re looking to compare. (In this screenshot below I did this based on 20% vs 25% CTR)
- Plug those numbers into the two top boxes
- Click “Result”
- In this example I need 1504 sends in each group of my test. Because I picked a power of 0.9, I can be 90% sure that group 2 will result in higher CTR than group 1