This was my first time hearing the exact term behavioral prototyping, but the method makes sense for if you’re testing something that might be too expensive to fully develop. The term Wizard of Oz prototyping is much more fun, though, and accurately describes the idea of “the man behind the curtain” in this method. Working in a team of 4 I helped develop a chat bot that would help users find out what times they are all available within one day. Each user would send their availability in a group text message, saying “I’m free…” or “I’m available…”. Once all the users entered their time availability, one person would message “That’s everyone” and then the chat bot would respond with the period of time where everyone’s availability overlaps.

Chat history from test. Emily in blue, Andrew in pink, and participant in green.

For our user test, we brought in one person, then formed a group chat between the facilitator (myself), note-taker (Andrew Briggs)and the participant. Myself and the note-taker walked the participant through the process while the wizards (Michael Beach and Greg Yu) were in charge of sending messages to the user pretending to be the chat bot. A video of the user test is shown below.

The primary challenge faced during testing was environmental–our test location introduced significant latency in message response time, due to cell phone support. Our first test has slower responses which made our chat bot less effective. Later tests in different location allowed for faster messages and made the chat bot more believable. Also through watching the way other groups implemented chat bots, it looked like Facebook Messenger might have had faster messaging speeds. Something that surprised me is that users thought that a person was on the other end of the chat bot because they didn’t have a specific syntax they had to follow. The fact that they had some variation in how they typed out their availability sparked their suspicions.

Some general feedback from our users is that they liked the aspect of not having to pull out their laptop to enter information on existing sites like Doodle Poll or When2Meet. One user suggested to have a feature for the chat bot to say when most people are available if not everybody is available at the same time. Our current version of the chat bot either responded for when everybody is available or that nobody is available. Additionally, one user said that looking for availability over multiple days may have been helpful, but we purposely set the scope to a single day because we thought scheduling across multiple days would be difficult for users to send in a text message as well as difficult for the wizards to process quickly.