What We Learned From User Testing Core Publisher’s Responsive Design

Mar 28, 2016

In the process of creating and rolling out our responsive theme for Core Publisher, we did a lot of research to help us make an outstanding product. However, now that more than five million end-users visit Core Publisher sites every month, we wanted to investigate how it performs — usability-wise — in the real world.

Specifically, can users find stories that interest them whether they’re on a desktop or a smartphone? Can they listen to live-streaming and on-demand audio? Can they easily navigate the site to find content they came for? In short, is Core Publisher’s responsive design effective?

And the answer is — drumroll please … yes! The responsive site works well. Okay, it’s obviously more complicated than that. Not everything worked perfectly, but at a high-level, it’s fair to say that we didn’t discover any major usability problems.

Before digging into some of the details of what we learned, however, I’d like to explain how we carried out this project by working with stations to recruit and interview volunteers remotely — it’s a method that has a lot of promise for us to do user research more frequently going forward. If you’d rather jump down to the findings, skip ahead.

Goal 1: Collaborate With Stations

It was important to us that we collaborate with Member stations on this research for a couple of reasons. First, by giving stations a chance to help us craft interview questions, we ensured that the results would be informed by authentic station concerns — and would therefore be as meaningful as possible.

Second, stations could help us identify and recruit volunteers from among their members inexpensively and relatively easily — and we’d get to observe actual listeners interacting with their local station websites. Such users don’t represent everybody, but they can nevertheless give us insights we might not otherwise get — say, if we used arbitrary testers selected by an online recruiter.

For this particular round of testing, we reached out to a few stations and ended up working with two, WUWM in Milwaukee and KUOW in Seattle. Together, we agreed on an interview script that asked users to perform a number of tasks related to finding, reading, and listening to content that would interest them. In addition, we asked users to find program information; to see if they could donate online; and to reflect on station branding — for example, what kind of branding would they expect to see and how did that compare with what they actually saw?

Goal 2: Investigate Remote Testing Methods

Since Member stations and their listeners are all over the country, we wanted to explore the viability of interviewing volunteers remotely. Could we speak with volunteers in their own homes or places of work? Would people be willing and able to share their screens with Skype — or to point their webcam at a mobile phone — so we could observe them in action?

While there were a number of logistical challenges in carrying out remote testing, they were easier than arranging to interview users in an office. And asking someone to set aside an hour for a phone call — which could be at pretty much any time — is a much smaller request than asking them to come into an office during regular working hours.

The biggest challenge involved observing people use the sites on a mobile phone, because it wasn’t enough to have them talk to me about what they were doing. I wanted to see what they were doing with my own eyes — how were they scrolling? what did they click on? at what points did they get confused? — because people will often say one thing but do another. For example, I asked one person what they would do to find a recent episode of a particular program, and they said they weren’t sure. But as they were saying it, they clicked on the navigation and went straight to the correct program page. I believe they were being honest: they weren’t sure what to do. But they weren’t stuck either — they had a guess — and it turned out to work successfully.

We solved the problem of observing people on their mobile phones with a funny-sounding trick called “laptop-hugging.” Here’s a picture (below) demonstrating how it works — as long as you have a webcam and the willingness to sit in a slightly awkward position, it’s possible to share your phone’s screen!

UX researcher Adam Kiryk demonstrates how to hug your laptop in order to share a phone screen with a remote viewer.
Credit Adam Kiryk

Goal 3: Explore The Effectiveness Of Core Publisher's Responsive Design

User testing is a research method that’s particularly good for evaluating the effectiveness of a given design because, even with a small sample size, it can reliably expose usability problems. It doesn’t necessarily give you hard-and-fast answers, but it can suggest areas that need improvement and ideas that might warrant further inquiry.

For example, if three or four people have trouble completing a task that is supposed to be easy, it’s probably because the design is inadequate in some way. Similarly, if seven or eight can complete a task without trouble, you’re pretty safe assuming that there are other aspects of the site in greater need of design attention. 

For this study, we recruited fourteen volunteers, ranging in age from 30 to 70, and divided them into two groups, one using smartphones and the other desktops. All were KUOW or WUWM members, so — while they didn’t represent a random sample of the general public — they provided an excellent opportunity to glean insights about what station listeners look for on their local Member station’s website.

Our Findings: What We Learned That Stations Can Put To Use

Local news matters.

As we’ve written about before, local news is a big driver of engagement for NPR Member stations, and this certainly applied to the listeners we interviewed. When asked to recall a time when they came to WUWM.com or KUOW.org in the past, almost all described an instance when they were looking for local news. They had heard part of a story online and wanted to follow up, or they were simply browsing for recent stories about their community.

Stations should continue to focus on creating high-quality local news for the web. Publishing audio for a story that was played on the air along with that story’s transcript is a bare minimum — people are looking for well-written stories, pictures, additional audio, relevant links, and anything else that can add to a story. And, even if they’d like to listen on their device, people aren’t always in a location where it’s possible — they need other forms of content to engage with. 

If you create web-only content, promote it.

Several testers were surprised to see stories on the website that didn’t have audio and that didn’t appear to be related to an on-air story. In other words, they didn’t think their stations produced web-first or web-only content, and would not have come to the website looking for it. This isn’t the first time we’ve heard users express this sentiment — it came up in user testing we did a few years ago. Given that, it’s worth considering how to raise the profile of this type of potentially valuable content among listeners. For example, talk about your web-only content on-air or highlight it using homepage promotional blocks.

Use all the tools at your disposal to keep visitors engaged.

At Digital Services, we’re very interested in how we can modify Core Publisher in ways that improve user-engagement — and we’ll be working on this later in the spring and into the summer. In the meantime, stations can use a variety of ways to help users find interesting content.

I asked people to find a story that interested them, and, after looking it over, to tell me what they would do to find other interesting stories. They used a variety of techniques. When it was present, many found the Related Content section at the bottom of a post. Others used tags or the author’s byline, and some went straight to search, typing in keywords that interested them.

KUOW has experimented with putting links to other posts right inside the flow of a story — for example, in this post about air pollution, the text is broken up, here and there, with links to other content that the editors thought would be worth promoting (see image below). Testers who saw posts with these inline links noticed them and said they might click on them.

KUOW places inline links in some posts to highlight popular or related stories. For example, the screenshot above shows a link to “7 Things You Need To Know…”
Credit kuow.org

For people who clicked on tags, the specific wording was important — and it mattered that the tags linked to a range of stories. One user clicked on a tag called “Muslim Employees,” which went to a page with only one story — the same one he’d been reading. Another, considering the tag “WUWM News,” said he felt it was too broad.  

Apply care in writing navigation labels.

Screenshot from an iPhone displaying a “hamburger” icon (three horizontal lines) circled in orange.

Nobody had trouble finding or using the navigation on desktop, and most people could navigate using the “hamburger” icon on mobile (see the screenshot at left). We did, however, discover a usability issue with nested menus that stations can improve right now with better labeling.  

Specifically, don’t assume mobile users will understand that your navigation has nested menus (that is, menus that only display if a user hovers or clicks on the parent label, as in the screenshot below). For example, if your station uses “News” as the label for a nested menu containing subcategories like “Local,” “National,” “Business,” etc — consider changing it to something like “News Categories.” None of the testers using smartphones understood that clicking “News” would do this, believing instead that it would take them to a page of local stories.

Specifically, don’t assume mobile users will understand that your navigation has nested menus (that is, menus that only display if a user hovers or clicks on the parent label, as in the screenshot below). For example, if your station uses “News” as the label for a nested menu containing subcategories like “Local,” “National,” “Business,” etc — consider changing it to something like “News Categories.” None of the testers using smartphones understood that clicking “News” would do this, believing instead that it would take them to a page of local stories.

Mobile phone users didn’t understand that clicking on News (at left, circled in orange) would display the nested menu at right. They thought it would take them to a page of news stories — like the homepage — and therefore never found the news subcategories.

Similarly, don’t assume that desktop users will click on a menu item that contains a nested menu. Unlike mobile users, they were able to discover the nested items without trouble — maybe because all they had to do was hover — but they didn’t realize the parent item was clickable. In the example screenshot below, users hovered over Programs and saw Lake Effect, UWM Today, and It’s Alright, Ma, but they never clicked on Programs — and didn’t expect it would take them to a page with dozens of additional titles. Consider including a link to “All Programs” inside of this dropdown menu. 

Since many users won’t think to click on the “Programs” label, consider including an item called All Programs in the dropdown menu.
Credit wuwm.com

Stay on top of branding.

Stations can update their logo and tagline in Core Publisher as well as in other DS products, so take advantage of this to ensure your logo and tagline is consistent in different places. Although this was not a major issue for most users, some noticed discrepancies between the logo and tagline on different station properties. One user said she was very skeptical of a donation page with a logo that was different from the logo on the Core Publisher site. She said it made her wonder if the site had been hacked and was compromised.

Link to the daily schedule (not weekly) by default.

The weekly schedule works well enough for desktop users, but it has a usability problem when viewed on a phone because content can get clipped. Seeing users encounter this problem was fascinating, and it gives us valuable insights about how users handle trouble-spots (some fixed the problem by turning their phone sideways, others switched to the daily view). While we can’t promise to address this issue right away, we will certainly discuss putting it into our roadmap of work for this year. In the meantime, stations can help their users by linking “Schedule” in the navigation to the daily view by default. The testers generally liked this view, and could still get to the weekly view by clicking on a toggle.

The schedule’s daily view (above left) tended to work better than the weekly view (middle) in user testing on a smartphone, although some users found that the weekly view functioned well in landscape mode (right).

Help users find recent program episodes.

It was fairly easy for users to find the program page for a program that interested them. However, when I asked them to listen to the latest episode, some had a harder time. Observing how users look for audio content is very useful for Digital Services because it can guide our decisions as we consider how to improve various aspects of Core Publisher over the next year, but it can also be useful to stations right now. There are several things stations can modify to help users find audio content on program pages.

  • Consider making audio as findable as possible by putting it above the break in a post. 
  • Write brief program descriptions. Users appreciated the schedule information, but were less interested in reading a general description of the program — they mostly wanted to find recent episodes, and long descriptions pushed episode content further down the page. 
  • Consider publishing both an entire episode and its segments separately — that is, one post would contain an entire 50 minute show along with links to each of the separate segments. KUOW does this with The Record in a way that seemed to work well for testers, who liked that they could either listen to the whole thing or go straight to the part they cared about.

Promote your content

Even though the people I spoke with were regular listeners and members, not all were aware that there was so much local news content on the site. Nor were all aware that they could listen to live streaming online. Don’t assume that your listeners have a sense of all your station does online. 

Our Findings: What We Learned That Digital Services Can Put To Use

Before going into detail, I want to point out that we can’t promise doing all the work necessary to take advantage of these findings. We have a lot of great ideas for enhancing and improving Core Publisher, and we need to make choices about how to focus our efforts. That said, these findings can help us develop and prioritize our roadmap — and they will certainly inform any changes we make to Core Publisher in the coming year.

Explore how to keep users engaged

This is a topic we’ll be working on a lot in coming months, and user testing can help guide our explorations. Some ideas that emerged from these interviews included:

  • Can we provide better tools for stations to highlight interesting content?
  • Can we change the layout of post pages in a way that guides users to more content? Related Content seems to perform okay, but not all users were able to find it. Is it too far down the page? Are there better places to put it?
  • Could the post page display content differently depending on the type of user — that is, might a fan, coming in through the homepage, want something slightly different than a user visiting via Facebook? 

Encourage users to share

A lot of these users said they would share a story, but only by word-of-mouth. Some are afraid to click on share links, carefully avoiding “Twitter” and “Facebook” icons. Some weren’t confident about what would happen — for example, one thought that clicking “Facebook” might publish to their feed without providing a chance to edit. Another wondered if clicking “Email” would open their email client, display a pop-up window, or do something else.

  • Can we reduce some of the hesitation to share electronically?
  • Can we make the process more transparent so users know what will happen?
  • Can we make sharing links easier to find and more friendly to engage with?

Continue thinking about the listening experience.

The volunteers I spoke with had no trouble using Core Publisher to listen to on-demand audio clips or live streams. That said, we may be missing opportunities to increase how much time people spend listening.

  • Could we make it easier for users to keep listening after an on-demand clip ends?
  • Could we make it easier to find recent episodes?
  • If a user can’t listen in the moment, can we help them listen later?

What's Next?

It’s clear that remote user testing can be a useful tool for us, and it’s likely that we’ll do more of it soon. Depending on what questions we have, it might make sense to explore ways to reach other types of users — for example, how does a younger demographic interact with Core Publisher sites? What about music-station listeners? And what about ‘flybys,’ the sort of user who follows a link on Facebook or Twitter, reads or listens to one story on the site, and then disappears? It’s possible that user testing isn’t the best way for us to learn about this type of user — analytics might tell us more — but it’s worth investigating.

For now, we have some reassuring confirmation that — at least among station fans — the responsive Core Publisher theme does a good job of helping people find, read, and listen to local news stories on desktop or mobile. And we've made a lot of interesting observations that can help us focus our work in the coming months. In the meantime, we’ll keep you updated whenever we learn more about our users.