Documentation Usability Testing
Premise
Dell’s APEX Navigator for Multicloud Storage had recently launched a major update but there had not been any generative user research done prior to this update’s design or release. I was brought in post-release by the UX Writers team. Tasked with creating a seamless user experience from in-console activities to the Dell Support website, the UX Writing team ensured that all support documentation was accessible to unauthenticated users, in line with business requirements. Before this iteration of research, the focus had primarily been on the latest product updates, with minimal attention to the overall console experience. This gap left usability concerns unaddressed, which the UX Writing team had been raising with limited impact. This study set out to 1. validate the design and 2. capture qualitative user feedback on the live site to help bolster change initiatives made by the UX Writers team. It was clear we needed to convey as many captivating stories from participants to bridge the empathy gap for stakeholders.
The UX Writers team had a goal to have one document per product that gets updated with each release. Their hypothesis was that users would prefer to have a single linkable page that they could refer back to for everything related to that product’s release updates. Whereas the prior model was to have a documentation page per update that were not connected. We set out to test this hypothesis in the research study.
Study Design
There were three key questions we set out to answer with this study:
When a user enters APEX or when a user is interacting with the console where are they're going to go to find support documentation, do they get lost?
Authentication and federation is the hardest step. It is not intuitive and requires reading documentation to complete. Is the documentation effective?
How easy is it for someone to use a centralized document per product that gets updated with each release?
With time constraints in mind, I chose to run a single usability testing study with 6 participants*. Even though the median age of Dell’s IT Admin study participants is 35y, I expected that we might see generational difference between the way participants pursue documentation and support. I chose to recruit for an even distribution across the following categories:
(2) Millenials (age 28-43)
(2) Generation X (age 44-59)
(2) Baby Boomers (60-69)
*the final participant was a no-show and a part of the Baby Boomer age category. If we had more time I would have pushed to find an adequate replacement.
Findings
“It’s not as though you can use natural language search to say, ‘Hey, this is my situation. This is my constraints. This is what I’m currently using. Can you tell me what the issue is with my setup?’ You have to try to figure that out through the documentation like, how does this compare to my setup.”
Searching for documentation
When given a task, the research revealed that 3/5 users reached for the search function without scrolling below the fold. AND 3/5 users started the task from outside of the console, using the global search instead of the console search.
Users were prompted by the task to find a specific answer. When pursuing that answer in the documentation, many users assumed that the config guide had the answer without pursuing further.
Some participants used the left nav instead of the top nav to get to support. Some did the reverse. This indicated that it was unclear to users that there was a distinction between global support and the in-console support.
Search Experience
Time on task ranged from 5 to 15 minutes when the tasks were expected to take less than 5 clicks. Moreso, one user had to give up on their task 13 pages deep into a search. Throughout longer searches like this users observed a lack of brand cohesion in their search experience and expressed disorientation throughout their search.
The image below shows three different screen examples from the interview of Dell branded content with similar components, but inconsistent look and feel.
Technical Pitfalls
Sometimes filter functions didn’t retain after first attempt, for example, two user found results in another language(s).
The Recommended feature (chip UI) was often on articles not related to the user’s search needs so they never bothered to click.
Users specifically complained that their search results recommended visiting a page that was not hyperlinked and could not be located via the search function.
Research Answers
Q: When a user enters APEX or when a user is interacting with the console where are they're going to go to find support documentation, do they get lost?
Answer: Users reach for global search functions often before inspecting the page. Users got lost or disoriented on nearly every task.
Q: Authentication and federation is the hardest step. It is not intuitive and requires reading documentation to complete. Is the documentation effective?
Answer: Users seek detailed notification/alert information before making decisions to reach for documentation. Task success is contingent on the way we chose to direct users from an alert. Users are asking why Dell can’t link the document with the solution to the alert about the problem.
Q: How easy is it for someone to use a centralized document per product that gets updated with each release?
Answer: Users enjoy having a single link for all documentation relating to one product. However, that page then needs to have a searchable navigation.
“I thought the search function would be more feature rich. As soon as I started typing I was expecting it to give me some suggestions across the categories (manuals, KB articles, etc.) before even clicking search. I want it to bring up suggestions. Other sites are taking advantage of the AI capabilities and I can put in language rather than a Boolean search.”
Key Insights and Recomendations
In addition to answering the research questions. The research was able to reveal the following areas that need improvement
User Interface Improvements
Make a clearer distinction between in-product and global search.
Embed direct links to relevant articles in alerting messages.
Ensure filter functions retain settings.
Community and Support Improvements
Strengthen brand cohesion across support pages.
Expand and integrate community features alongside the knowledge base.
Provide clear descriptions for support options like “expert guidance,” “tech support,” etc.
Search Function Improvements
Enhance search robustness to ensure relevant results
Integrate AI capabilities for natural language processing.
Impact
The findings of the research were well received by the UX Writers team. The information and stories that were surfaced reinforced the issues that the UX Writers team had already been advocating for. With this new anecdotal data, we were able to approach multiple teams with supporting evidence for proposed changes. These findings started new conversations around ways that Dell can develop a competitive edge by leveraging new technologies like AI enabled search functions to improve our product offerings.
