Singing Inside a Painting (2D in 3D)
In the last year, I’ve successfully performed live from my living room to audiences all over the world using open sources technologies for 2D live-composited video and 3D live avatar performance. However my workshops with universities and regional theater companies consistently revealed that my tutorials “skipped too many steps” thus preventing most people from wrapping their heads around these emerging workflows. As an actor and entrepreneur, the majority of my time is spent on “messaging” and “branding” to communicate and convert my value in the marketplace so I continued to release case studies to simplify and streamline my Future Stages work.
I was surprised to be tagged on Twitter by the scenic designer @kellylinhayes in a post with the iconic painting by Georges Seurat with an “Enter Here” sign, suggesting the painting could be experienced in virtual reality. Theater nerds know there is actually a Stephen Sondheim musical inspired by this very painting that often uses the imagery as scenic elements in the show. When I studied at The Lee Strasberg Institute in New York, they had us create a "living painting" to write and perform a monologue “inside” a famous work of art. I'd been wanting to use-case this exercise in VR and thought it might elegantly show off the potential for "pop-up" art in 3D.
I realized this was the perfect scaffolding to ease the audiences understanding of Extended Reality tools. The painting is a familiar touchstone from the physical world. The musical is a 20th century approach to immersive storytelling inside this exact painting. By performing a song from this musical inside the painting in three dimensional space, I can build on existing knowledge and artistry, rather than present an entirely new experience. And it worked. Friends, family and even business partners whose eyes had glazed over during explanations of my previous work, were suddenly calling me with their own ideas and reactions.
The first step was to take the original painting and separate all the individual assets - the people, the scenery, the background. The end of the video shows a timelapse creating a “cut out” of each asset as its own “layer.” These layers were stacked and arranged in Mozilla Spoke so that the painting looked complete from straight on, but had gaps between each character to walk around…and through.
Mozilla Hubs allows you to easily share your web camera, however there is no background replacement or removal filter. Recently, they integrated the ability to add PNGs or GIFs with transparent backgrounds so I thought I could record a video of myself singing, remove the background, convert to a GIF, and drop myself into the world as a “cardboard cut out” performing on a loop. This also allowed me to act as my own camera operator and fly around my performance to show off different angles of the experience.
A note about GIFs. Adobe Photoshop limits GIF imports to 500 frames/images in a sequence. So I exported my video in 500-frame pieces and stitched them together in Photoshop (a sneaky workaround) to export more than 500 frames for the final, uninterrupted GIF. There is a lot of artifact-ing due to my rough chroma key, but it got the job done and created a slight shadowing/point-cloud effect.
If the platform would allow a chroma key on the webcam...or volumetric stream...or even an IK rigged avatar…this would get exciting quickly. I'm especially intrigued by the idea that each of the characters in the painting could be their own Avatar, so the painting could come to life as an ensemble of virtual actors...or even the orchestra members...represented through virtual scenic elements. Extremely grateful to the shout outs from the XR Theater community, calling this “one of the smoothest on-stage donning of a HMD that has ever been captured in voxels” and hope you’ll suggest other works of art that have musical potential!