nfl all day
Perfecting the NFL ALL DAY Playbook experience
Scope of work
Goal: Identify desirability of a new product feature, reveal areas of confusion and uncover opportunities to improve the user experience
Research Approach: User interviews, usability testing
Role: Planning, recruiting, executing, synthesizing and reporting
Timeframe: 2 weeks
Methods: Remote user interviews
Tools: Google Meet, DoveTail, Figma
Project Background
NFL All Day is a digital collectible marketplace that lets football fans buy, own and trade officially licensed video highlights (Moments) of their favourite players or teams in the form of NFTs. Moments can either be purchased through limited-time pack drops, earned by competing in challenges, or through the All Day marketplace, where fans from all over the world buy and sell around the clock.
The NFL ALL DAY team was in the process of designing a new product feature called the “Playbook”, as a way to increase site engagement for both new and existing users.
Through the Playbook, users would be rewarded by completing specific tasks at set milestones called “Yards” during a one-week period. The type of reward a user could collect would be based on a Free “free-to-play” rewards path or a Premium “pay-to-play” rewards path.
The challenge
The Playbook was originally designed without user feedback, leaving the NFL ALL DAY team uncertain about whether the concept would be intuitive or engaging.
Although the team anticipated that the Playbook could boost retention and engagement, there was little evidence to show that new or existing users wanted this feature or would interact with it regularly.
To address these challenges, the NFL ALL DAY team asked me to:
Evaluate the desirability of the Playbook concept
Identify and resolve usability and comprehension issues before launch
my approach
After an initial kick-off meeting to better understand goals and desired outcomes, I proposed and got alignment on the following research plan:
Research Questions
Desirability: Does the overall Playbook concept resonate with users? What do they like and dislike about it?
Comprehension: Do users understand the differences between the “Free” path and the “Premium” path? Is it clear on how to earn rewards within each path?
Usability: What challenges or issues do users come across while interacting with the Playbook?
Methodology
45-minute remote user interviews covering:
Problem discovery: Identify reasons for why users become disengaged and their overall JTBD for joining the site.
Concept validation: Identify whether or not the Playbook concept resonates with users.
Usability testing: Identify areas of confusion and uncover opportunities to improve the user experience.
Recruitment
5 x New users who joined in the last 30 days and have yet to purchase a Moment, or who have yet to participate in a Drop or Challenge.
5 x Existing users who have previously purchased a Moment, or who have previously participated in a Drop or Challenge, but have stopped in the last 30 days.
Users were selected based on their responses to screening criteria.
methodology deep dive
Problem Discovery
I asked a series of problem discovery questions prior to showing users the Playbook concept, as a way to uncover existing problems, behaviours, needs and goals.
The feedback that was collected from this stage was then used to more explicitly identify if the Playbook concept would meet the needs of disengaged users.
Through problem discovery I sought to answer the following:
Why did the user join NFL ALL DAY?
What problem was the user trying to solve by joining?
Why is the user no longer participating in Challenges or Drops?
What would make it easier for the user to continue using NFL ALL DAY?
Concept Validation
Users were then shown the Playbook concept without any contextual information provided upfront. They were given 1-2 minutes to explore and learn more about the concept before I followed up with questions to evaluate sentiment and comprehension.
Through concept validation I sought to answer the following:
What is the users’ initial reactions to this Playbook concept?
What does the user think is the purpose of the Playbook concept?
What does the user find unclear or confusing with the Playbook concept?
Can the user see themselves benefiting from the Playbook concept? In what ways?
How does the user think others will respond to the Playbook concept?
Usability Testing
After providing initial feedback on the Playbook concept, users were given a series of tasks to complete. I observed and documented how long it took for users to complete each task, and what they struggled with as they proceeded to do so.
Through usability testing I sought to evaluate the following:
Do users know how to complete a task within the Playbook?
Do users know how to gain Yards?
Do users know how to opt-in to the Premium Playbook path?
Do users know where to go to learn more about a specific reward in the Playbook?
Key takeaways
Problem Discovery
New users struggled to take action on NFL ALL DAY as they did not know where to start
For new users who have yet to purchase a Moment or participate in a Drop or Challenge, the main reason was not knowing how they should first spend their money on the platform. They were anxious about not wanting to acquire a Moment that they would regret later, and so it resulted in them not taking any action at all.
Existing users started losing interest in the platform once the value of Moments started depreciating
Most existing users joined NFL ALL DAY as they saw it as an opportunity to make money. However, the value of their Moments depreciated as prices were largely impacted by the fall in value of cryptocurrencies. As a result, existing users are critical to spend more money on the platform until they start seeing the value of their Moment collection going up.
Implications
The Playbook will have a positive impact on new users as it will provide them with a set of guided tasks that they are able to complete on the platform, which in turn alleviates anxieties of what to do first or what to buy first.
The Playbook will have a positive impact on existing users as it encourages and rewards the “burning” of lower-value Moments as a way to acquire higher-value Moments. Burning Moments reduces the supply of Moments in the marketplace, which in turn increases the value of Moments left in circulation.
“Honestly, I just don’t know how to get started. What do I do? How do I buy and trade? How do I make money? I don’t have the time to figure it out and I don’t want to regret what I buy.” – New User
“As much as everyone tells you that they do it for the fun and the love, nobody wants to lose money. Once they (NFL ALL DAY) burn Moments and rewards, and get our accounts to where they should be.. engagement will go up.” – Existing User
Concept Validation
The Playbook concept was highly desired
Overall, users were excited by the Playbook feature and could not wait to see it live on the NFL ALL DAY website. They liked that they would be rewarded and incentivized for doing things on the site, and that these rewards can only be acquired by completing the Playbook.
The Playbook concept was easy to understand as it felt familiar to most
A majority of users were able to quickly identify the purpose of the Playbook as they had prior exposure to similar concepts via video games such as Fortnite and Call of Duty.
Implications
The Playbook concept resonated well with users as they saw it transforming the NFL ALL DAY platform from being a buy-and-sell marketplace and more of a rewards-based gaming website.
“It almost reminds me of like an RPG game.. where you kind of collect experience points, and then you just kind of unlock better and better items every time“ – New User
“If I’m buying a Moment, not only am I buying it, but I’m also counting it towards Playbook rewards. I really like that.” – Existing User
Comprehension & Usability
The design of the Reward table left a lot to be desired
75% of users could not clearly identify what needed to be done in order to acquire Premium rewards. They assumed that clicking the “unlock premium” button would unlock one premium reward at a time. This was mainly due to multiple unlock call-to-actions in the design.
50% of users also assumed that they had to select between either “Free” rewards or “Premium” rewards, and that they would be collecting both if they proceeded with unlocking premium rewards. This was mainly due to seeing two distinct rows (paths) in the Playbook design.
Implications
Simplify the Rewards table by combining Free and Premium rewards into a single lane. This will help ensure that users don’t think that there are “missing” or “empty” slots in the reward path.
At first glance, users could not identify the relationship between Tasks and Rewards.
40% of users assumed that each task corresponded to one of the rewards displayed in the Rewards table, e.g. that the top-most task would unlock the bronze reward at the 10 Yards milestone, and that the second task at the 20 Yards milestone, and so on.
65% of users indicated that it would be easier to see the tasks on the main Rewards screen, since they would want an easier way to see what tasks still need to be completed without having to navigate through multiple screens.
Implications
Use the principle of proximity to show the relationship between yards, tasks and rewards. Tasks should not be separated from rewards, as users believe that certain tasks give them certain rewards. Instead, the design should group tasks and yards together so that the causal relationship is more clear.
Outcome
As a result of the research findings, the NFL ALL DAY team felt more confident moving the Playbook concept into production, following a redesign to better meet the needs of their users.
Changes to the design include reducing the Rewards table into a single “lane” and moving tasks to the first screen.
In terms of engagement and desirability, within the first week of launching the Playbook, the NFL ALL DAY team reported the following results:
50,000+ users participating in week one of the Playbook
125,000+ Moments “burned” to fulfill the different task requirements