How to Usability Test a Government Service

An overview of my favorite approach to getting government services in front of real users, and making use of the feedback as quickly as possible.

As you may have heard, some folks have had a hard time trying to conduct usability testing in the Federal government (and other governments!). Not only can it be easy and fun, I have a *favorite* easy and fun approach to conducting usability testing — which I’m excited to share with you today.

For the feds: This process does not require PRA review by OMB or even your agency PRA Officer! It’s on less than 9 people, and it’s nonstandard oral communication, and it’s observation! Three exceptions, when even one of them would do!

Step 1: Decide what you’re going to test.

If you’re already working on a project to build/update a service, you should make sure to do usability testing starting *before a single line of code is written.* Use paper (or PDF) prototypes to test. You will get a ton of critical information.

So, in this case, we’re going to pretend that the service is applying for Veterans Widgets (not a real thing [I think]) and that the owner is pumped to make the changes.

Step 2: Draft your field guide.

For a field guide, you’re making a list of what you want to go over in your testing sessions, and writing example questions as a guide. For our test of the Veteran Widget application, that field guide could look like this:

  1. You’re a former servicemember and need a contraption from the VA. What would you do now?
  2. Are you eligible for the widget? How do you know?
  3. Sign up to get the widget.
  4. Check the status on your widget application.
  5. They denied your application! Appeal your widget determination.

“I dunno man, this seems like an exact list of questions, kind of like a survey.” Nay! Check out a video of how this goes in real life. Usability testing involves a nonstandard conversation — you’re responding to a human in real time. Reading an exact list of questions wouldn’t work.

Step 3: Recruit your users.

Approach A: Contact your correspondence office (or mail center or call center or wherever someone would go if they gave feedback to your office), and ask if they have any correspondence they can share from your target users. In our example, let’s say it’s Veterans who love electronics. They send you 40 letters from Veterans who wrote in about widgets and contraptions and other electronics. Choose a few, and ask if they’d like to help out on a design project to make widget applications work better for Veterans. They will be pumped.

Note: When we did this at the White House, people were SO excited to help. A USDS team once needed Veterans who were LGBTQ to help test a new, faster process to remove less-than-honorable discharges (“bad papers”) from Veteran records so they could receive the appropriate benefits. The White House correspondence office shared a stack of letters, and it was a delight to contact folks who had written in, to help fix an unfair system.

Approach B: Work with your community affairs office (or whoever connects with outside groups!) to find a great community group, and ask their leaders for help finding the right people (referring them or forwarding along an ask). I strongly recommend that you explain it’s a “design project” so that they don’t assume it’s the normal feedback that advocacy groups offer government agencies — it’s a real potential user, trying to finish a task.

TIP — Recruit more folks than you need, and work with them on future rounds of testing!

Step 4: Get the right team in place

Step 5: Test!

Instead, if it’s in person (and not a pandemic!), bring snacks and offer coffee and water. If you have swag (like mugs or stickers), that’s often appreciated. If it’s online, a hand-written note, ideally on the government stationary, saying “thank you” and also how their contributions will help other people sent in the mail will be very meaningful. If possible, have your executive (mayor, secretary, commissioner etc.) sign the card as well.

The word “issue” written above a list of problems with a service like “couldn’t find the home button”
The word “issue” written above a list of problems with a service like “couldn’t find the home button”
Example notes

For the observer(s), have them take notes about every time a participant gets stuck/lost/frustrated in the process.

The same “issue” list as above, but with additional issues listed, and an “X” next to them, with each participant represented
The same “issue” list as above, but with additional issues listed, and an “X” next to them, with each participant represented
Example notes for multiple participants

Ideally, have the test be 45 minutes, and then at the conclusion of the test, thank the participant and spend 15 minutes collecting notes from your observers to put in one spreadsheet, and make a “P1” column to represent Participant 1, and put an X next to each issue they had. Do the same for Participant 2 (P2) and Participant 3 (P3), also marking “X” for issues that they shared with other participants. This is a process I learned many years ago at UX Camp NYC, and it’s perfect for government usability testing.

The same chart as above, but with an “action” column with actions like “enlarge button” and “n/a.”
The same chart as above, but with an “action” column with actions like “enlarge button” and “n/a.”
Example notes with draft actions

Then, at the end of your testing sessions for the day, keep the same folks in the room to discuss what should be done about each issue. Sometimes, the answer is “nothing,” and sometimes the answer has nothing to do with the user interface, and that’s ok! It will be very clear from your tests what the most important items are, and making this a discussion with your observers will help build buy in (shout out to general counsels across government!) from folks who may otherwise resist the simplification of a process. Draft the proposed actions together.

The same day as testing, email your feedback and actions spreadsheet to as large a group as will not be mad at you for the spam, and include reflections about what the participants were like. Things like “They were so excited, because they had never been asked for their feedback before,” or “One of the participants is a 3rd generation Veteran, and is from my home state of Ohio!” Include an invitation to attend the next session as an observer, and stress that no experience is required.

This is critical, because in my roughly 10 years of on-and-off Federal service, I have never once seen a beautifully designed, narrative report about the findings from a single usability test 1) result in those changes actually being made, or 2) be read and understood by anyone not on the design team. Never. This is not the time to make a glossy, beautiful report. Make it casual, make it fast, make it simple, make it action-oriented. One spreadsheet, some notes, send email.

Step 6: Make it a system!

  • Book the same conference room for a few hours, recurring every month, for testing.
  • When you get more participants who are willing to help than you need for that month, book them for next month.
  • Government agencies typically have a very formal review process for launching things. Work with your “executive secretary” or general counsel (or whoever is in charge of reviewing and approving things before they go live) to add “How many users was this tested with?” to the clearance cover sheet or process. This both helps make it something folks “need” before a thing can go live, and also helps raise awareness that this is expected.

Troubleshooting

My agency is too scared to show something before it’s final, even if it’s only to 3 people!

As a fed, won’t I need PRA clearance if even though I’m only testing 3–5 people with this specific test, I will eventually test more than 10 people total?

GIF of 19 different prototypes of a tool
GIF of 19 different prototypes of a tool
Prototypes by the brilliant Stephanie T. Nguyen. Public Domain GIF

I’m not a designer or design expert!

The people who own “services” here are super skeptical because this has been so expensive and hard in previous attempts.

We’re not building anything new right now.

I totally agree with all of this in theory, but I feel really alone.

I don’t have a remote way to host a usability test because no videoconference solutions are approved at my agency.

Note: This is just one way to accomplish this work. There are different ways that work well, too!

America’s foremost technologist named after a Great Lake. Now @CodeforAmerica! Co-founder @techladymafia + @usds. Former @harvard @whitehouse @cfpb.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store