Complete user research no matter when

Sometimes content design work comes with incredibly tight deadlines, deadlines that don’t allow for much, if any, user research. That’s okay, you can complete your research at any point in the process, even if it’s after the publish date.

In 2023-4, we completed work with the Public Sector Commission (PSC) Communications and Engagement team that required this approach. They had new HTML content to publish that previously would have been published as PDFs. It aligned to best practice, but we did not have specific user experience data to support the change or guide us in our design choices.

We committed to going back after the publish date to complete our user research, and ensured the subject matter experts (SMEs) supported this approach.

Key learnings

• It’s never too late to complete user research.
• HTML is proven again to be the best way to present content for most online users.
• Quantitative research is good for understanding content performance.
• Qualitative research helps you understand the human experience and make meaningful improvements.

What we did

The PSC Communications and Engagement team engaged us to help them publish a suite of new directives as HTML rather than PDFs. This aligned to best practice, but there was no time to complete the research, prototyping and testing that we would usually recommend, to ensure the best design.

We published without testing

We designed the new HTML directives based on our understanding of our users and the content. We also implemented a print-to-PDF solution. This meant that our users could still access the information in a way they were used to if the HTML version did not meet their needs.

We committed to user testing after publishing

We ensured the SMEs understood we would complete our user research after publishing to:

• confirm our design choices
• understand if a PDF and HTML version were needed
• learn if we could make further improvements.

We ensured the SMEs understood that while the content itself could not change (because of the nature of the content), the design might need to, if that’s what the user research indicated.

We completed quantitative research

We believed that a round of quantitative research would help us validate our design, determine if a PDF version of the HTML was needed, and uncover areas for improvement.

We wanted to gain an understanding of who our users were, what their needs were and their experience when interacting with the new HTML version of the directives.

We completed:

• Google analytics and Mouseflow analysis to understand how users interact with the HTML directives pages
• A/B online testing to test different layout choices, heading placements, accordion styles and button placements.

This gave us data that showed most users had adapted to using the HTML version of the directive. It also gave us clear endorsement of our design, with some suggested areas of improvement.

It did not tell us with certainty what users wanted or why, and it did not tell us why users were continuing to download the PDF version even after interacting with the HTML.

We progressed to qualitative research

We completed qualitative research to better understand how our users were navigating through the content, and why they were doing what they were doing. This would ensure that any changes we were considering based on the quantitative feedback would improve user experience. We’d also understand why these changes were needed. This would help us better communicate the need for change back to the SMEs.

We made improvements

We used our quantitative and qualitative research to present back evidence to the business that our initial best practice design was a success. We provided insights on how the content was performing with users and what potential improvements we could make. We gained support to continue with the HTML approach and approval to make our suggested improvements.

What we discovered

It’s never too late to complete user research

When faced with tight deadlines, a research and test after publishing approach can work. It helped us get approval from the business area to proceed with a best practice design. It also helped us get approvals for important changes after publishing.

Quantitative and qualitative research combined provides the best insights

A combination of quantitative and qualitative research is best. The quantitative research gives you good data in terms of what most of your users like and expect. It tells you what they are doing and when. However, it does not tell you why they are doing what they are doing, and it often cannot give you clear reasons or endorsement for the changes you’re considering.

Qualitative research gives you the context you need for your data. Talking directly to users helps you understand the human experience better, meaning you can make certain the changes you implement enhance the user experience.

What else we can use

Completing research after we published content helped us support our design decisions. It gave us the data we needed to support our best practice approach, and it helped us make informed enhancements. It told us:

• HTML is preferred over PDF, but an option to save or print to PDF or similar allows us to meet all our user needs:
  • 48% of users used the HTML version
  • 28%& of users used both the HTML and PDF versions
  • 24% of users used the PDF version
• For government website users (82%) prefer accordions with clear headings and sub-headings to help them navigate difficult content
• For government website users (80%) prefer wider page layouts for difficult content.

Conclusion

Research is always valuable, no matter what stage of the process you complete it. Set the expectations with your stakeholders early, and it will help you get the very best from them and their content.

Quantitative and qualitative research should always be completed together. One without the other means you’ll only ever get part of the story. Having both types of research will make your arguments stronger and get you better outcomes for your users.

Finally, it’s never a waste to gather good research data to supports best practice. It can help remove any doubts your business area may have.