About CBS Sports - Fantasy Football
The flagship Fantasy product of CBSSports.com is Fantasy Football Commissioner.
As a league manager product that allows individuals to organize their league using robust customization capabilities, it is the most profitable of all fantasy football product offerings and is the primary focus of marketing efforts. Commissioner is subscription based and must be re-purchased each year.
Annual league retention of 90% year over year.
It Started With: “Ask the UX Guy”
At CBS Interactive, User Experience (UX) Research and Design was new to the organization and not an established part of the product process.
The organization did not clearly understand the scope and value of UX, so I was not surprised when one day I was asked to look at some marketing product pages and pick the “most user-friendly” design.
The Opportunity
While I appreciated being leveraged to make product decisions, the initial ask was not the right question. I needed to understand how much time I had to invest in this evaluation and how I could best help the organization make a design decision.
Thankfully it was several months before Football season so there was time to evaluate the pages via an A/B test and conduct a parallel set of user evaluations.
But it was not going to be easy.
An A/B Test and User Evaluations wasn’t what they wanted to hear!
Marketing Designs
The Visual Design team worked with Product to create (3) variants to the current Commissioner subscription page.
Current Page (Control)
This is the current page and served as the Control test artifact.
Long Text & Visual Variant
Print ad style presentation: Graphically intensive while providing extensive product information without messaging price. Bright blue call to action button “Start a Free Trial Now” at the bottom of the text.
Price Focus Variant
Simple presentation with bold call out of price: Minimal graphics and minimal product information provided with a text link call to action. Price is messaged loudly in the header graphic.
Simple Text Variant
Same page as PRICE – minimal graphics, minimal product information, price is not mentioned.
A/B Testing Strategy
I created a detailed Landing Page Optimization plan to share with the team outlining why the A/B testing strategy was recommended and how we could execute and analyze the results to make an informed data decision. Below are excerpts.
A/B Test Results
Key Criteria Evaluated:
Primary metric: % of viewers creating and paying for a league
Secondary metric: % of viewers creating leagues
Summary of Findings
Two variants stood out from the four total marketing creatives tested (LONG and PRICE).
The LONG variant had higher % of created leagues while PRICE showed higher % of paid leagues.
LONG is very likely (99.99%) to generate higher % of created leagues
PRICE has a high probability (about 90%) to create higher % of paid leagues than SIMPLE and REG.
PRICE has higher % of total paid leagues than LONG (0.07%), but not significantly different.
LONG Variant
Generates creation of more leagues
PRICE Variant
Generates more paid subscriptions after leagues creation
LONG and PRICE variants performed better but for different metrics. Now what?
Usability Test
Although my strategy for evaluating the marketing creatives was successful, there were still questions.
While the A/B tests were still in flight, I planned and conducted a usability test of CBS Sports with (20 participants) to get a baseline of the user experience.
Fantasy participants rated CBS Sports.com higher in comparison to Non-Fantasy participants based on (14) ratings of the site.
Eye Tracking Evaluation
As part of each Usability session, I asked participants to start by taking (1) minute to view and “take in” each of the marketing designs run in the A/B test. Below are the results of the Eye Tracking data.
Summary:
Participants ignore the subscription buttons altogether.
Participants did spend time reading the copy.
In Summary
The research strategy I conducted examined the original question of which marketing creative was “most user friendly” by making it a research problem.
A/B Test: The A/B test reduced the variants to (2) but more questions remained. Why did LONG create more subscriptions but lose out to PRICE in terms of number of paid subscriptions (the Primary metric).
Usability Test: Through careful questioning during the Usability Test, participants remarked that they needed enough information (text) to make an informed decision to spend nearly $200. A combination of elements from both LONG and PRICE could be tested.
Eye Tracking: The data backed what participants said that reading some of the text was important. In addition, there is a distraction to the signup buttons and/or not shown prominently.
7% increase in paid subscriptions
Note: Subsequent to my research and testing, a net new marketing variant was created based on the data collected and tested against the prior subscription page. I was excited to see such a lift. Even a modest uptick resulted in significant increased revenue.
