Note: Since this a current program at The Eureka Room, I won’t go into spoilers but I did learn some important lessons in the creation of this program that I think are worth sharing.
House of Psych! 1.0
For the House of Psych I had a pretty strong vision of the experience. I was just coming off the poor reception of the Mindfultainment programs and had decided to focus only on the “tainment” part of the programs for these reason:
- That’s what I liked to create.
- That was what people enjoyed most about my work.
- It was way easier for me to create them because they were invented from whole cloth rather than a patchwork of Eureka Room sensibility and established science and practices of which I was not a domain expert.
I had just learned my lesson. I had my focus. And I dove right in.
All by myself.
Mistake.
The First Showing
I easily spent a couple hundred hours working on this program in what start-ups would call “stealth” mode. After many months it was ready to show. I invited everyone who I thought would support it the most.
Twelve people showed up at my house. I introduced the program and explained all the work I had put into it. I thanked my friends Matt and Vianca for their help with the live action parts of the video. I led them into the room. I turned out the lights and hit play.
The feedback I received I think could best be described as courteous warmth. Not totally bad. Definitely not good. So I guess that means some amount of bad. Considering it was my friends, I knew it was probably really bad.
Even while running the program (I stood in the room, behind the curtain and watched) I could see parts that were confusing, boring, repetitive, way too long, way too short. It wasn’t awful but it wasn’t the WOW that is the bar I set for the programs.
In the living room I tried to engage a discussion about the program but I was unprepared. Also the sheer number of people (and not everyone knew each other) inhibited some people from voicing their thoughts and I could sense a real missed opportunity.
A bigger more embarassing problem was that I had another smaller group coming in a few weeks and that group was not made up of my closest friends. I didn’t even know some of them. Uh oh.
The Second Showing
Over the next few weeks I made some major adjustments to the program that I thought would help. Since the program was already “done” it took many hours to rearrange and redesign things because the edits were so major that they upended the whole experience and story.
I also made a list of questions for post-experience discussion. If it wasn’t working well, then at least I could find out what people thought needed changed.
I won’t go into spoilers but the feedback I received from these participants was very valuable and insightful. And much of it… obvious in hindsight.
These wonderful and patient visitors were able to show me what needed trimmed and they highlighted some smaller things about the program that they really liked – these items actually became core to the program its final form.
I’ve Learned This Before
Before The Eureka Room and before I started Big Weekend Calendars, I worked for software companies. This is a simplification, but the two most popular ways to develop software are the waterfall method and the agile method. The waterfall method is when a group of designers design the full product and then hand it off to the engineers to create. Then they ship the product and see if the customers like it.
In Agile, the designers create a product that consists just of the core features, the engineers create it and then they show it to customers to test and get feedback from.
With waterfall you avoid the messy and scary business of having to actually talk to your customers. You have sole power of creation. Talking to customers will just slow things down, anyway, right?
With Agile, not only do you get to see if users like your product but they will tell you what they want to see that isn’t there. It’s painful to see that you were off the mark, but it’s better to find out now before you’ve got all the way down the path.
And after all, you’re not there to create stuff you think people you will like. You’re there to create things people actually want.
Agile is not just a software development concept. It goes by other names. In manufacturing it is often referred to as continuous improvement. In design there’s a similar concept called Design Thinking.
I’d seen the waterfall method fail again and again during my time in the software industry. But for some reason I didn’t realize I was making the same mistake with The Eureka Room.
And boy did I pay the price. So much had to be changed. And not just the program. My entire process.
The New Testing Process (Stage 1)
After learning (re-learning?) from my mistakes, I developed a much improved testing and vetting process. I’ve split it into “Stage 1” and “Stage 2”, but in reality it’s a bit more fluid.
Stage 1 of testing consists of these tasks:
- Figure out the core mechanics.
- Figure out what makes it fun, what they like to do, how much or little of it they like to do.
- Figure out the core moments.
This is what people remember from the Eureka Room. As Dan and Chip Heath write in their excellent book, The Power of Moments, people remember the start the end and a few things that are in the middle. The rest can be pretty mediocre and no one will care.
Where are the surprises, the wows, the heightened emotions? These are the pillars of the experience and the rest can be designed around it. You might think that this would lead to some forced plotting but actually I found that it gave me great creative constraints, resulting in a better plot.
Smaller Groups For Testing
One big fix was to have a small group of 2-5 people test it. It’s been pretty well established that more than 5 doesn’t give you much additional feedback. This was way better than 12 at a time because:
- It will take a long time to allow each person to share their experience, much less allow for the depth of sharing you might need.
- Some people feel uneasy speaking in groups.
- You get into group think.
- You don’t need as many testers (in my case, I had used almost all of them in the first showing and now had to dig up more.)
Know What You Are Testing
To figure out core mechanics and core moments, I then designed not an entire program, but multiple short (10-60 second) versions of ideas.
Instead of showing them random things I made to uncover what they liked, I designed the segments to test out hypotheses. For example, I needed to know which speed felt comfortable so I created the segment with different speeds.
I had specific questions for the testers that were open-ended (aka not yes/no) prepared in advance. They were also thought out and designed to test the hypothesis. I know it’s hard to not introduce bias and lead people, so I tried to word them as neutrally as possible and really emphasize that they should tell me whatever they thought, not what they thought I wanted to hear (strangers are always better than friends at this request).
I also set up a video camera to record their facial reactions as they watched the videos. This was somewhat helpful but I wasn’t quite sure how to interpret them beyond “no laugh” “looks confused” “looks delighted”. I think to make more inferences beyond that would be guessing. The videos didn’t have the highest ROI (and also might have influenced their behavior) but didn’t take long to set up so I’ll probably continue to do them until I determine they have no utility.
In the questioning I try to make it a request for “advice” not “feedback”. This tip is thanks to Robert Cialdini’s book Pre-Suasion. If you make people co-creators rather than judges and you’ll get them to think more deeply, care more, and give more detailed suggestions. Err.. I mean advice.
Other Considerations For Testing
Time of day. One person came from a really long day at work and was just exhausted. They just weren’t into it and I wonder if I should test more at the times of days (or at least states o f mind) that they would actually do this experience. The people who have come over for the established programs always seem to enjoy it more when they are after happy hour and before dinner (though, who doesn’t enjoy things more after happy hour?)
The testers should belong to the tribe you are trying to serve. More than a few times someone comes over to see the rooms existing programs and they are enthused. They bring their partner who is clearly just going along because they were asked to. As much as I try to weed out the party poopers, they do show up sometimes and there’s nothing I can do. Take their feedback with this in mind. Though sometimes if they are particularly not wanting to be there you might get a nugget of advice inadvertently. But often just eyerolls.
Visuals. If the visuals are not done then that can impact the experience. But the visuals do not have to be in their final detailed state. Instead do the 20% that makes them ok enough. Because getting them to perfect take so much effort, wait until the last step of the program’s finalization to add the polish. Resist the temptation to perfect things for testers! Lining things up, getting the right look, etc… NOT IMPORTANT at this stage. Good enough is way good enough here.
The New Process (Stage 2)
In Stage 1 I determine what seem to be the best Core Mechanics and Core Moments. Now, in stage 2, I try to piece the segments together into an overall narrative for a new round of testing. Ideally this round is tested with new people who have not seen the program before.
Stage 2 of testing consists of:
- Find points of Exhaustion
- Find points of Confusion
- Find points of Boredom
The fastest way to give some a bad experience is to exhaust them or confuse them mentally, emotionally, physically, or spiritually. (With the occasional exception of using these states for humorous effect).
Pauses and quiet moments are great to replenish energy but it is a balance. Too much time to refuel and it becomes boredom. Too little and exhaustion and confusion surface quicker during the more challenging moments.
I’ll save “how to write the plot/narrative” for another post and keep this one focused on the process of vetting ideas.
Lessons Leared
- Test frequently and often. The tribe wants to help and has great ideas.
- Be prepared to test: have hypothesis and have questions that test it.
- Groups of 2-3 people test things the best.
- Don’t perfect the visuals in the beginning because they will change until the end.
- First identify awesome moments and mechanics that can last. From there, assemble them into different programs to test.
- Ruthlessly hunt down and eliminate exhaustion and confusion.
- Carefully balance boredom and refueling