How to Usability Test a Government Service
An overview of my favorite approach to getting government services in front of real users, and making use of the feedback as quickly as possible.
As you may have heard, some folks have had a hard time trying to conduct usability testing in the Federal government (and other governments!). Not only can it be easy and fun, I have a *favorite* easy and fun approach to conducting usability testing — which I’m excited to share with you today.
For the feds: This process does not require PRA review by OMB or even your agency PRA Officer! It’s on less than 9 people, and it’s nonstandard oral communication, and it’s observation! Three exceptions, when even one of them would do!
Step 1: Decide what you’re going to test.
This may seem obvious, but this choice is important, especially on your first round of testing. Choose something that has a chance of getting changed based on the feedback because the “owner” of the service is interested in it working well. A response letter from your office? The IVR (phone maze) that folks hear when they call you? If you already have buy-in and can choose anything you want, run, don’t walk, to the most important service that you provide.
If you’re already working on a project to build/update a service, you should make sure to do usability testing starting *before a single line of code is written.* Use paper (or PDF) prototypes to test. You will get a ton of critical information.
So, in this case, we’re going to pretend that the service is applying for Veterans Widgets (not a real thing [I think]) and that the owner is pumped to make the changes.
Step 2: Draft your field guide.
Unlike a survey, where there are a list of exact questions that are presented in the same way to every user, usability testing is a conversation. Surveys, especially government service feedback surveys, in my opinion don’t produce useful feedback about how well a service is performing. For example, a real, live government website currently surveys users on a 1–9 scale and asks “*Please rate the site procedures to accomplish tasks on this site.” (Emphasis theirs.) What does that mean? Who knows! What will they learn? Nothing useful!
For a field guide, you’re making a list of what you want to go over in your testing sessions, and writing example questions as a guide. For our test of the Veteran Widget application, that field guide could look like this:
- You’re a former servicemember and need a contraption from the VA. What would you do now?
- Are you eligible for the widget? How do you know?
- Sign up to get the widget.
- Check the status on your widget application.
- They denied your application! Appeal your widget determination.
“I dunno man, this seems like an exact list of questions, kind of like a survey.” Nay! Check out a video of how this goes in real life. Usability testing involves a nonstandard conversation — you’re responding to a human in real time. Reading an exact list of questions wouldn’t work.
Step 3: Recruit your users.
In non-government contexts, this can be super hard and expensive! In government, it’s actually super lovely and convenient. And remember, for this round of testing you need *at most* five people. I like to have a mix of approach A and B.
Approach A: Contact your correspondence office (or mail center or call center or wherever someone would go if they gave feedback to your office), and ask if they have any correspondence they can share from your target users. In our example, let’s say it’s Veterans who love electronics. They send you 40 letters from Veterans who wrote in about widgets and contraptions and other electronics. Choose a few, and ask if they’d like to help out on a design project to make widget applications work better for Veterans. They will be pumped.
Note: When we did this at the White House, people were SO excited to help. A USDS team once needed Veterans who were LGBTQ to help test a new, faster process to remove less-than-honorable discharges (“bad papers”) from Veteran records so they could receive the appropriate benefits. The White House correspondence office shared a stack of letters, and it was a delight to contact folks who had written in, to help fix an unfair system.
Approach B: Work with your community affairs office (or whoever connects with outside groups!) to find a great community group, and ask their leaders for help finding the right people (referring them or forwarding along an ask). I strongly recommend that you explain it’s a “design project” so that they don’t assume it’s the normal feedback that advocacy groups offer government agencies — it’s a real potential user, trying to finish a task.
TIP — Recruit more folks than you need, and work with them on future rounds of testing!
Step 4: Get the right team in place
At bare minimum, you should have someone conducting the test (using the field guide) and someone observing. I strongly recommend having anyone on your team who will later make suggestions to make it harder to use (your general counsel, perhaps!), as well as the person who will get to approve any changes (This can even be your executive! Sitting in these testing sessions are extraordinary) also sit in to observe and take notes.
Step 5: Test!
It’s possible to get “thank you” gifts for participants in government, but may be more red tape than is worth it for quick rounds of usability testing that you’ll (hopefully!) be conducting frequently. Be upfront so no one is surprised.
Instead, if it’s in person (and not a pandemic!), bring snacks and offer coffee and water. If you have swag (like mugs or stickers), that’s often appreciated. If it’s online, a hand-written note, ideally on the government stationary, saying “thank you” and also how their contributions will help other people sent in the mail will be very meaningful. If possible, have your executive (mayor, secretary, commissioner etc.) sign the card as well.
For the observer(s), have them take notes about every time a participant gets stuck/lost/frustrated in the process.
Ideally, have the test be 45 minutes, and then at the conclusion of the test, thank the participant and spend 15 minutes collecting notes from your observers to put in one spreadsheet, and make a “P1” column to represent Participant 1, and put an X next to each issue they had. Do the same for Participant 2 (P2) and Participant 3 (P3), also marking “X” for issues that they shared with other participants. This is a process I learned many years ago at UX Camp NYC, and it’s perfect for government usability testing.
Then, at the end of your testing sessions for the day, keep the same folks in the room to discuss what should be done about each issue. Sometimes, the answer is “nothing,” and sometimes the answer has nothing to do with the user interface, and that’s ok! It will be very clear from your tests what the most important items are, and making this a discussion with your observers will help build buy in (shout out to general counsels across government!) from folks who may otherwise resist the simplification of a process. Draft the proposed actions together.
The same day as testing, email your feedback and actions spreadsheet to as large a group as will not be mad at you for the spam, and include reflections about what the participants were like. Things like “They were so excited, because they had never been asked for their feedback before,” or “One of the participants is a 3rd generation Veteran, and is from my home state of Ohio!” Include an invitation to attend the next session as an observer, and stress that no experience is required.
This is critical, because in my roughly 10 years of on-and-off Federal service, I have never once seen a beautifully designed, narrative report about the findings from a single usability test 1) result in those changes actually being made, or 2) be read and understood by anyone not on the design team. Never. This is not the time to make a glossy, beautiful report. Make it casual, make it fast, make it simple, make it action-oriented. One spreadsheet, some notes, send email.
Step 6: Make it a system!
Use bureaucracy to bake it into your agency’s work. Here are just a few ideas, but think creatively about what would make usability testing “a thing” in your agency.
- Book the same conference room for a few hours, recurring every month, for testing.
- When you get more participants who are willing to help than you need for that month, book them for next month.
- Government agencies typically have a very formal review process for launching things. Work with your “executive secretary” or general counsel (or whoever is in charge of reviewing and approving things before they go live) to add “How many users was this tested with?” to the clearance cover sheet or process. This both helps make it something folks “need” before a thing can go live, and also helps raise awareness that this is expected.
Troubleshooting
My agency is too scared to show something before it’s final, even if it’s only to 3 people!
Start with a service that’s live on your site today, and have folks who are scared sit in to observe. They’ll learn how important the work is and meet the very un-scary test participants who are excited to serve their country in a small way. Then try again with a new service and work with those previously scared folks to advocate with you.
As a fed, won’t I need PRA clearance if even though I’m only testing 3–5 people with this specific test, I will eventually test more than 10 people total?
Nope! Each round of testing is different. Here’s an example where the United States Digital Service and the Federal Student Aid office tested with dozens and dozens of people and did not need PRA clearance because, for example, there were 19 different prototypes tested and each one was only tested with 1–5 people.
I’m not a designer or design expert!
That’s ok! You don’t need a design degree to do a usability test. Here are two books that can help increase your confidence.
The people who own “services” here are super skeptical because this has been so expensive and hard in previous attempts.
Start with something super simple like a “closing letter,” (the letter a person gets at the end of a process) and ask some of the skeptical folks to sit in and watch. Empathize and don’t excuse past things that went wrong — acknowledge that it sounds very frustrating. The previous folks were likely trying to help, but got stuck in some common design-meets-government pitfalls.
We’re not building anything new right now.
That’s ok! Try something that’s live on your site now.
I totally agree with all of this in theory, but I feel really alone.
Good news! There are sweethearts all over government working on doing this work that would love to cheer you on. If you have a .gov or a .mil email address, join this UX Community listserv hosted by the General Services Administration (GSA)!: https://digital.gov/communities/user-experience/
I don’t have a remote way to host a usability test because no videoconference solutions are approved at my agency.
Reach out to folks at the GSA on that listserv, and ask if they’d observe *and digitally host* your test. They have very simple videoconferencing approved and love collaborating with different government agencies.
Note: This is just one way to accomplish this work. There are different ways that work well, too!
