Into the Maelstrom

LJI Week 1: Quality
June 15 marked the ten year anniversary of working for my current employer. This was both gratifying and slightly amusing to me because of a comment made by one of my former coworkers when I left my previous job.

"An accessibility job for a large mainstream company?" they reportedly scoffed. "That'll probably only last a year, if that long."

Hello, still here!

Oh, don't get me wrong, it's definitely been a bumpy ride at times. Our team has always been that troublesome puzzle piece that company leadership wasn't quite sure where or how to slot into their operations. It was a bit like being part of a jigsaw puzzle where the assembler could change a piece's location, but also its size, and shape, and name, and overall purpose. And, don't let the default past tense of storytelling fool you, they're still at it!

The disadvantages to being that puzzle piece are obvious, but believe it or not, there were advantages too. For example, it meant I did the routine stuff like reviewing content to identify accessibility issues, but also worked with developers to design feature enhancements so that the content was easier to understand and navigate. Myself and other colleagues were encouraged to share our knowledge by responding to calls for papers at national conferences, and if we were accepted, got to travel to the conference and present are findings. Without a doubt though, customer research was one of the most interesting activities I got to participate in.

Of course, ten years ago, I didn't know anything about research or usability studies seeking customer feedback. When I was asked to help out with our first study, I sorta figured, you put together a prototype, showed it to the customer, let them play with it for a bit, and then asked a few questions. Easy-peasy! Yeah, not so much.

• What are the research questions your study is trying to answer?


• What prototypes or study materials will be shown to each participant?


• Is there enough time to prepare the prototypes/study materials before the study is scheduled to begin?


• Will participants need to sign nondisclosure agreements in order to view the prototypes/study materials, and has legal approved the language for those agreements?


• Do you plan to compensate participants? If so, how much will each participant receive? Have you secured funding for the compensations, and if so, for how many participants?


• Where will the study sessions take place, and is there a fee to use that space?


• How much time will be allotted for each session?


• How do you plan to recruit participants for the study?


• Will any study participants be less than 18 years of age? If so, have you written a parent/guardian consent form? Has legal approved the consent form language?

You know that old Star Trek joke about the different management styles of captain Kirk verses Captain Picard? With Kirk, the strategy with a planet was beam down and poke any aliens with a stick, and if it was another ship, "Fire phasers!" Picard's strategy, no matter what was in the offing? "Lets hold a meeting."

Yup, you guessed it, I was on Picard's ship. So many meetings!

If you're reading the list of logistics above and thinking to yourself, "Dude, where's the fun?" I don't blame you. The answer was, in the beginning, I didn't have to worry about most of that stuff. I got to bounce potential research questions around with other team members, and then, once we had nailed down what we were trying to discover, I helped write the research protocol. The protocol was important because when you're conducting a usability study seeking feedback, the golden rule you're supposed to follow is to not bias the participants. The prototypes or study materials should be evaluated the same way by every participant, and one of the ways to ensure that consistency was to write a script that would be read in every session.

Now, what about the prototypes or study materials? Should you show those to participants in the same order every time as well? If you're like me and said, "Yes," I regret to inform you that you're wrong.

Think about it this way. You're participating in a research study lasting an hour and a half and looking at three different prototypes for the researchers. By the time you reach the third prototype, the likelihood is that you're going to be getting tired and imagining how nice it will be when the session's over and they stop asking you questions. If I always presented the prototypes in the same order, prototype number three would probably get a pretty raw deal. So, as researchers you counterbalance or change the order of items being evaluated.

Yes, it was a lot to keep track of, and it did get just a bit tedious reading the same script to six or eight participants every day during the study, but the payoff was being able to meet and speak with people with no filters in-between you and their opinions. The hard part, if you can believe it in this age of social media where people seem compelled to find fault with every little thing, was convincing participants to answer honestly and in depth. "We want to know what you like and what you don't like about each prototype, and you won't hurt our feelings in any way by telling us what you think about them."

There were typically three or four of us in the room with a study participant, one person reading the script and asking questions, and the others observing reactions and taking notes. We encouraged participants to "talk through" their experience while exploring a prototype, and I imagine they sometimes may have felt as though they were under a microscope. If they had a pensive expression or made an unhappy noise, one of us would invariably say, "Could you please share with us what you're thinking right now?"

It was time consuming, and exhausting, and while a study was going on, we'd barely had time to grab a meal in-between sessions. Is it weird if I say I miss the experience?

The last study I participated in was during the summer of 2022 at a conference in New Orleans, just after the July 4th holiday. We all wore masks during each session because COVID was very much still a thing. While venturing out for a late dinner one evening, a coworker accidently led me into a hole in a New Orleans sidewalk. Before the conference was over, we had to push the abort button and head home because everyone on the research team, except me, caught COVID. And finally, when I did arrive home, I decided to self-quarantine in a hotel for a few days, just in case I might still be a carrier of the dreaded virus.

Yeah, even with all of that, I still miss conducting research studies.

Dan