Usability testing a CLI tool is easier than you might think. It can be done by anyone working on a product that has a user interface. Website? You can test that. CLI? You can test that. Instant pot interface? You can test that (Oh, I wish they would).
We test products to get real, unbiased feedback. You can ship your product and hope for the best, but chances are, you’ve missed something that would improve the experience. You can show people your product and just ask them if they like it, but chances are, it’ll be too awkward for them to say “No.” However, when you ask people to actually use your tool, and watch them fumble with it, you learn infinitely more. That’s usability testing.
The biggest difference between usability testing a GUI and a CLI is that 90% of testing a CLI tool is going to be about whether the documentation is clear and intuitive. If you don’t create documentation for your tool before user testing, you’re going to have to walk the user through how and why to use your product, which will make the test a bit pointless. Make your documentation the best you can before the test (whether that documentation is in a published website, or a text file attached to an email sent to the participant, or a super robust —help flag). This contrasts heavily with GUI testing, where almost any amount of documentation would be considered a obvious failure of the solution.
There are a few different ways of setting up your CLI prototype. Short of literally creating your product, you can protoype a CLI with paper, digital paper (Google docs), chat tools, bash, and javascript.
Many years ago, Cloud Foundry posted a few articles and talks about how they prototyped in a Google doc. In one case study, their team approached users with 3 versions of a prototype in a Google doc, without having to code anything. Their hypothesis was that the users would really want the option where users could apply flags to a generic command. But after testing it, the users all liked the option where they would write a more specific command without flags. According to folks at Cloud Foundry, that option involved fewer changes for their team in the end.
So, testing it actually saved Cloud Foundry engineering hours and effort and, thanks to a scientific process, resulted in a product that they knew users would prefer.
Another super low fidelity option is to use a chat program. CLI tools are much like chat programs in that there is a call and response. You could create a chatbot (or have a very fast person “behind the curtain” on the other end of the chat) that replies to user input with expected responses, and interview users about their experience with interacting with the fake CLI tool right from Slack.
These options might be best suited for when a PM or Designer has no engineering resources but still needs to get input on direction or CLI style guidelines.
But, these options might be too low fidelity for most situations. Although it’s going out of style, designers have often used wireframes to test out products or features. Wireframes allow us to generate the mere idea of an app using greyscale boxes and un-styled text to give users just enough information to make decisions. This prevents designers from spending tons of cycles pushing pixels on details that don’t matter after finding out the entire user flow doesn’t work.
For testing CLIs, there are a few mid fidelity options, including bash or javascript applications with minimal lines of code and no database, that replicate call and response interaction for users in their CLI.
For example, one of the HashiCorp founders, Mitchell, initially tested some of the HashiCorp products in bash on colleagues and friends before building them out for real. Another example is Matt Rothenberg, who tested out a Runway CLI tool by coding in Javascript, which just felt comfortable and easy to prototype in. In particular, he tested out a loading animation, which faked a slight delay to see how users would respond to latency and how to communicate that latency.
Mid fidelity options are most often going to be the best route to take because they allow you to test the user experience of your product in the most realistic interface with some but minimal engineering time.
Finally, keep in mind, you can do usability testing any time, even if you have a fully featured production product. Regardless of fidelity, here are some basic best practices for user testing:
- Recruitment: Aim for 4-6 participants. Try to find people who are most like your users. You’re building a CLI tool. Easy! Find other engineers at your company or in your network who are not already familiar with your project, but who might be prospective users.
- Ask open ended questions. If your participant says something like, “Is this a command or a flag?” don’t direct them to the answer. Don’t even tell them to type —help. Simply ask them, “What would you do to find out?” They might try a few methods you hadn’t considered in your design. They might use —help and give you feedback on the help content. Or, they might just pop open Google, which is a good way to find out if your website has good SEO and good documentation.
- Let users struggle for awkward amounts of time. It’s painful to watch users fumble with something that you designed. But if you let them struggle for long enough, they’ll start talking about how they wish it was designed, giving you valuable feedback on routes you may want to consider.
Photos related to this post are from WOCinTech Chat.
