Note: This post was first drafted on 11/19/2020
At the beginning of the pandemic, it was clear that I wouldn’t be able to do The Eureka Room as planned. It was probably going to be at least a year until I could show it to people. Not only that, but I decided I didn’t want people to come test it due to Covid concerns.
Developing programs in isolation has proven to be terribly inefficient. See my post on the House of Psych. Testers make all the difference because I can present something that is not even half-baked and watch their reactions moment to moment to see what’s working before I’ve put a lot of time into the program. I can get great feedback and advice during and afterward. This process was incredibly helpful in developing Turkey Volcano.
Another project that had gotten shelved were the Curious Dinners. Clearly we weren’t going to be having 12 strangers and 10 hosts in a small living room anytime soon.
So I had some time on my hands and no projects to fill them.
The Curious Cam Idea Is Born
I knew a lot people were doing performance shows on zoom, but I hated the zoominess of it. I wanted to combine people into a single “room” and have an overlay of actors and special effects play on top of and behind them.
On the surface, this seemed doable. The pieces were all out there: background removal, special effects, live streaming, most people have video cameras on their computer. We already had the actors and performers wanting to do a dinner. Why not put it all together and do the dinners online?
This was also exciting to me because of the scalability. Our 12 person IRL dinners were not ever going to be very scalable. We’d have to charge $200 a plate just to break even for our efforts. And even if we found those high rollers, finding an evening all that all the hosts had available usually required planning months out. And those were just the obvious hurdles to scaling.
But with this new “Curious Cam” idea, I had some experience conceits that would allow us to have all the actors pre-recorded. The whole thing (in theory) could be automated and not require anyone to really be there other than someone available as customer or tech support. Suddenly thousands of dinners were possible – simultaneously.
Curious Cam Research
I started talking with different people who might want to work on it. Iain is a great programmer and friend and I asked him if he was interested. At the time, it looked like we’d need some front-end help so Wendy got involved as well. Rebecca and I made some test videos and a mockup.
The hardest part was finding the technology. We tried so many options for video conferencing and did so much research. We spent many frustrating hours just trying to figure out what technologies were out there, what they did, and how they worked. Iain probably frustrated the most. Since he is a much better programmer than I am (to say the least) he was usually the one trying to get some poorly-documented solution found on github to work for us.
We were stuck in the tech development phase far longer than I had anticipated. Eventually there wasn’t much for Wendy or Rebecca to do so they faded out. My friends Andy and Kevin both volunteered to help with programming but this was a daunting project and neither got past the setup phase.
I put a shoutout on facebook and piqued the interest of my friend Alan Watts, a maker extraordinaire who works at Meow Wolf. At first he was just chiming in answering our questions, then he jumped in and taught us a lot about the world of video and streaming.
But even with Alan’s expertise most of the ideas we tested proved they wouldn’t work for our needs. The hardest part about video conferencing is that it is done in real-time. Any processing, combining, or sharing of video streams has to be done lightening quick. This meant that the special effects we were trying to do, while minimal, were really, really challenging to implement.
The second hardest part (imo) was that video conference technology has not been a popular area in DIY software development. The big corporations have their proprietary ways of doing it, but there’s not a lot of mature open-source video conferencing solutions out there. OBS, which is what most professional streamers use, uses protocols leftover from the FlashPlayer era. WebRTC, the internet open standard for real-time communication was built with peer-to-peer in mind. That’s fine if you just want to see the raw video, but if you want to do any video processing, it requires an MCU server, which slows things down.
The third hardest part is that editing and mixing the streams is costly. Rather than just routing pipes here and there, we were trying to open the pipes up, see what’s inside, tinker with it, then close the pipe and send the data on its way. This is not only difficult, but it introduces latency so the experience of chatting with someone seems very unnatural and annoying.
We attempted many things. Here’s just a few technologies we tinkered with:
- Kurento (Owned by Twilio)
- OpenVidu (uses Kurento under the hood)
- Jitsi (built as a relay, as such the streams are not open to manipulation)
- OpenCV.js/opencv4nodejs (OWT).
- OWT This looked promising because it was made for openVINO stream processing. However the configuration was a nightmare.
- Green-screening on client and on server
- Combining streams on client and on server
- Applying effects on client and on server
- Local hosting, AWS, and Digital Ocean
We did eventually get a 4-person conference going fairly well with node-webrtc and were able to combine the streams onto a canvas on the server. It looked promising but the more we dug in the more we could see all the other parts we would need. Like sound. Like a way to sync the streams with the background video. And all the other parts we’d need to allow a user to sign up, pay, login at a specified time, test their mic and camera, etc, etc.
The project slowly faded out over time. I got busy with my calendar company’s busy season. Alan and Iain had other projects. Then last week two effective vaccines were announced. The end of the pandemic looks more in sight.
At this point in the pandemic, I think most people don’t want yet another video chat/call/show. Given the current rate of progreess and the mountain we now see in front of us, Curious Cam might not get done until well after everyone was inoculated. I think it’s time to shelve the Curious Cam or find someone else to take it on.
Another reason I feel ok shelving it is that I wasn’t able to do much of the actual work and I felt guilty about that. I would spend hours looking at code just to understand it and maybe suggest one thing. Iain or Alan could do that same thing in a few minutes. I just wasn’t adding much value other than organizing meetings and working on how we can work around technical limitations. Could I find more people to help them? Maybe. Could I get better at coding? Yes. Do I want to? No. I can be mediocre, but unfortunately it’s drudgery for me.
Other things making this upstream battle: the big guys are coming out with more and more video conferencing features. While they aren’t focusing on “entertainment conferencing”, I suspect they’ll get to where we could just use their tools instead of making our own.
It pains me to quit this one because I love the idea but it’s become clear the risks have grown and the finishline is still nowhere in sight.
Curious Cam, you’re on the shelf.