One of the more frequent questions that we get asked by our clients or by our seminar attendees is: "Is a redesign for usability worth it?" In other words, what is the return-on-investment (ROI) of a redesign?
In recent years, we've seen a decline of the ROI for usability, most likely due to approaching a ceiling of usability improvements: With more than 20 years of web-design experience under their belt, designers have learned a few things and fixed quite a few problems. But a redesign for usability can still save you a significant amount of money.
In this article we tell the story of Mozilla's support site, which was able to get a 233% improvement out of a redesign for usability. (Here's an explanation of how the improvement score was computed.) Thus, we can say that the redesigned site was about 3 times better than the original site.
The cost involved in this redesign was 14 person-weeks or 560 person-hours.
Is it worth spending 14 weeks to become 3 times better? This depends on the hourly rate for your staff and the value of your site and thus cannot be answered in general. However, for Mozilla, which gets huge amounts of traffic, the improvement is certainly worth the trouble, as it would be for almost any big site or company that does substantial business online.
So how did Mozilla do it? What was involved in this redesign?
Who is Mozilla
Mozilla is a large, open-source, worldwide, software organization staffed by both employees and volunteers. It makes one of the most popular web browsers (Firefox), along with other useful products and services.
Pain Points
Millions of people come to Mozilla's support website every year to get help with Firefox and other products and services. When users cannot get an answer from the information existent on the site, the Mozilla staff aims to help by answering each person's question in the user-support forum, and to respond to questions as quickly as possible.
As Mozilla's website had grown organically, users were having difficulty finding information and the support staff couldn't keep up with the number of questions in the forums. Specifically:
- At about 400 pages or so, the help documentation had become a difficult place in which to locate particular information quickly.
- The forum staff (employees and volunteers) were having trouble responding to questions as quickly as they wanted to, because of the increasing number of incoming questions for the rapidly updating Firefox.
- The forum overload was also making it difficult for staff to find time to write new help articles for frequently asked questions. More articles could help, but the growing number of articles also caused more findability problems.
Action Plan
Mozilla decided to invest in discovery and iterative usability testing in order to improve the IA of its support site. The research questions aimed to understand (1) how people (users and staff) used the support system; (2) which types of information were really important.
Top research questions
- How do users and staff interact with the support system?
- Which problems are the most important to address in the website redesign?
- What is the most-wanted information?
- Which words do people use when they search?
- Which desired information seems to be missing?
- How can the information best be organized and presented in order to match what users most want to do on the website?
The UX Team
The UX team consisted of 3 members:
Susan Farrell, Senior User Experience Specialist, Nielsen Norman Group. Susan conducted the research, did data discovery, analyzed data, and made design-change recommendations.
Crystal Beasley, Product Designer, Mozilla. Crystal led the project, coordinated with Mozilla stakeholders, and played the computer during paper-prototype research sessions.
Bram Pitoyo, Web User Experience Designer, Mozilla. Bram designed the task flows and prototypes and supervised the interaction-design changes to the website. He also tested the old IA so we could compare results with the tests of the proposed new IA.
The Steps
We employed a variety of research methods intended to help us understand users' needs and also to redesign the IA and the workflow on the support site:
- Doing data discovery and analysis, to understand how users behave on the support site and why
- In particular, we looked at a variety of data sources to identify users' top tasks, as well as difficulties that they had with the current site. The table below summarizes the methods that we used.
| UX data |
|
| Behavioral data |
|
| Content analysis |
|
| Interviews with staff |
|
- Testing old and new navigation schemes using card sorting and tree testing to improve the information architecture
- Testing variations of the most-important task flows for desktop and mobile web designs, using paper prototypes
We conducted the paper-prototype research with users in Portland, Oregon, USA. Bram designed the evolving prototypes in Jakarta, Indonesia after watching each day's sessions on video.
Bram changed the designs and sent them back to us as PDF files, which we sent to an office supply store to print for us on their large-format printer. Because of the 14-hour time difference, it was possible to work around the clock as a team most days.
By revising designs between test days, we were able to progress key pages in the design through 7 versions in only 2 weeks of testing. Prototype-design time, including testing and final revisions, took place over 9 weeks.
Although not every project can test 7 design versions in 2 weeks, this example is certainly proof that usability can be agile (whether with a lower-case a or an upper-case A) if the team is sufficiently efficient and decides to emphasize throughput and fast research methods.
Redesign Results
A 70% decrease in support questions means forum staff are less overwhelmed and able to respond more quickly.
After prototype testing, but before implementation of the new design, Mozilla staff implemented a temporary quicklinks navigation item on the homepage to test whether direct access to the identified, most-wanted content would decrease the volume of new questions coming into the forum. Quicklinks are often a workaround for poor navigation structures, so we don't recommend them, but in this case it was important to test those key research findings at scale before implementing the new navigation scheme.
As a result of the website redesign and content improvement effort, the Mozilla Support help documentation was expanded to sufficiently address the most general questions we discovered. Because articles on basic and frequent issues are more findable now, visitors ask fewer questions, and their questions are about more-specific topics.
Providing easy access to the most-wanted information caused the volume of new help questions in the forum to drop immediately by about 70% (from about 7000 to about 2000 questions per month), allowing forum staff to exceed their quality goals. A month after the UX activities, Mozilla staff revised the support documentation to make it more findable, and ongoing data analysis allows the hot topics to change as needed.
Lessons Learned
Support websites are made of answers to frequently asked questions, which are a moving target. By periodically analyzing user data of various kinds, customer service can document the standard answers that users need, allowing support staff to focus on new issues and questions that require unique answers.
Data is key to gaining needed resources. Data mining proved the problems existed, and analytics proved the solutions worked. It was easy for support-site stakeholders to gain the resources and momentum to implement recommended changes once some of the fixes measurably addressed their pain points. This early success led to more support for UX activities and new hires: an information architect and a content manager, who now optimize content, navigation and search.
Collaboration over time helped the UX team, support staff, and developers, who were located in many places around the world, to share knowledge and develop a shared vision. This unification of purpose and concerns while working closely together to solve problems resulted in a great website design, one that helps meet everyone's needs.
Clearly, Mozilla's investment in UX design for its support website has paid off beautifully, and it showed measurable improvement almost immediately. We hope this example helps you make the case for user-experience ROI and paper prototyping — for iterative, user-centered design.
We'd like to thank Mozilla for allowing us to share this experience with the UX community.
Learn more about ROI in our ROI for Usability report or in our course on Measuring UX.
