Simple testing can boost CS education 29% overnight!


On October 14, we launched the Hour of Code, with unprecedented partners and backers, uniting behind a campaign to to convince teachers and schools to teach computer science to their classrooms for one hour, instead of their usual classes, during Computer Science Education Week. As expected, the initial press coverage created a spike of traffic.

After the spike, we started measuring educator signups. Our ridiculous goal is to reach 10,000,000 students, and we had 7 weeks to recruit the support of educators to hit this goal.

Based on my experience as an advisor to Facebook and co-founder of one of the fastest-growing apps on the early Facebook platform, I’d already taught our team the most important tool for any growth-hacker: A/B testing. So we immediately began testing numerous alternative headlines and buttons. What a difference it made!


Original text / button:

Welcome to the 21st Century. Help us introduce 10 million students to computer science. All it takes is one Hour of Code. Learn more

.. other tested versions not included here...

Winning text / button:

An Hour of Code for every student. Computer Science is a foundation for every student. Help us introduce it to 10 million. All it takes is one Hour of Code. Join us

The winning combination of text/button performed 29% better than the original, so we swapped it out after just a few days of testing.

As you can see from the charts below, even though our original spike of visitors had ended, and our web site traffic was flat, our total signups accelerated after implementing our A/B test.


What caused signup growth to accelerate without a growth in visitors?

The answer is easy: A/B testing increased our signups/visit. This last graph is most telling. Mid-day on 10/20 is when we switched 100% of our visitors to see the testing-optimized headline and button, and we saw an immediate growth in signups per visit.


Graphic: leosapiens / shutterstock

Roderick Hunnicutt, CPD

Lead Instructor: Firearms | Emergency Medical | Active Shooter

10y

Thanks for sharing this information. Constantly testing and optimizing your page can increase revenue, donations, leads, registrations, downloads, and user generated content, while providing teams with valuable insight about their visitors. Whatever your goals it may be, you can assure that every changes you made it produce positive results.

Like
Reply

Salam Hadi Partovi, I think that article shows that big PR launches are not always helpful. Similar to A/B testing, there should be several "PR iterations". A startup should leave the PR iteration with the highest reach to the point, where it is likely that people signup, use the service or come back. So in general being confident to get a big chunk of people through the funnel. One question: how many people have visited your website at peak-time? Khodafez

Like
Reply
Ken Collins

Owner at Ken Collins Marketing

10y

A/B testing (or split testing) can continue to improve results when you continue to use it. Test two versions, pick the best of the two, drop the other. Now, make small tweaks to your chosen version to create another version and A/B test them next to each other. Pick the best, drop the other. Continue that process until you can no longer find a version that performs better than your chosen version.

Like
Reply

Not to split hairs, but I'd argue that it was copywriting and not A/B testing that increased your signups/visit. But that A/B testing helped you identify which copy was ultimately more effective.

To view or add a comment, sign in

Insights from the community

Explore topics