JASON DROHNmake a version of the page with that. Call that version C. Now it's time to test A
against C. Since B already lost, there is no need to use it again.
I suggest keeping a log of your test results. Record the dates of your test, details
of what you were testing, and the results. Your software might keep track of
these, but it's always good to have a backup. When it's time to create versions C,
D, or E, check your log and make sure you don't accidentally use headline B
again. The only version you should test again is the winning one.
The nature of split testing is that sometimes A will win, sometimes B will win.
In other words, sometimes the newer version will do worse than the older one.
Of course we want to find things that do better, but don't feel bad when the new
one does worse. Just like the baseball player that knows he won't hit a home run
(or even a single) every time, realize that sometimes the older version will do
better. All you really need to do is make sure you keep track of your all-time
best performing version (which in copywriting circles is called the control), and
always test that against something new.