Written by: Christopher Llewellyn Reed | March 15th, 2023
VR, AR, MR … XR … so many acronyms to describe a form of enhanced reality that is very much still in flux in this now-third decade of the 21st century. Many film and technology festivals have, for at least about 10 years now, been showcasing the state of the art as its stands in the very moment. SXSW has now rebranded the section devoted to such work the “XR Experience” exhibition. Here at the 2023 festival, I had an opportunity to walk around and marvel at the enormous variety of offerings, though time prevented me from seeing more than three.
And there is the unfortunate rub, for now: these experiences take time to watch and require a special headset (or more, as you will see below), limiting the number of people who can participate at a time. Until we move beyond the complicated interface, all of these “R”s will demand lengthy queues and careful planning to slot them in. Still, what the artists and technologists can accomplish is impressive. Thanks to the folks at Jessie Cohen PR & Consulting (I appreciate you, Nate and Frida), I was able to see the below-described exhibits.
aespa VR Concert at KWANGYA (approximately 7 minutes)
Christopher Llewellyn Reed: Tell me about AmazeVR and this particular VR concert experience.
Patrick Esch: We were here last year debuting our Megan Thee Stallion experience. This year’s aespa experience has kind of the same premise. We shoot our artists by bringing them into a green-screen studio and then we take that footage and use Unreal Engine to create the virtual world around them, so you get the best of both worlds, with that realistic stereoscopic footage and then all the cool things you can do with the CGI environment. I think that’s what differentiates us from other people trying to do VR concerts. Our goal is to do a VR concert for every artist by 2023. We have a joint venture with SM Entertainment called Studio A, so we’re going to shoot more K-pop acts with them, but otherwise, we will be creating VR Concerts for all genres of music.
CLR: And since I don’t know aespa, how big a K-pop band are they?
PA: They debuted 2 years ago, so they’re not the biggest, but they are right now one of the most popular in Korea.
Forager (approximately 15 minutes)
This exhibit was a little more involved, in terms of how to experience it. In addition to a headset and headphones, there were two haptic controllers (to be used in just one section of the stream) and a beanbag chair which would vibrate in certain parts. The user is encouraged to lounger and get comfortable and move and rotate as desired throughout the project. It was simultaneously lovely and relaxing. Here to tell you about this work is technical director and co-creator (with Winslow Porter) Elie Zananiri.
Elie Zananiri: Forager is a project that is all about mushrooms, and is premiering at SXSW. There are many facets to the experience. Basically, the project started during the pandemic when my partner on the project, Winslow Porter, moved to Maine and started growing mushrooms, got really into it and was fascinated by how fast they grew and how gnarly they looked when they grew. He thought it would be really cool to capture this digitally to have different ways of exploring what this looks like. So, together we set out to build what is called a photogrammetry rig. What you do with photogrammetry is take many different photographs of an object from many different angles and then you run it through software which reassembles it as one 3D model. So it’s a 3D model that’s based on an actual representation of the thing itself.
If you look at our rig, you see that it has 5 cameras and a turntable with a mushroom growing on it. This thing runs on its own, and over the course of every 30 minutes it takes 200 pictures of the mushroom from different angles. This gets sent to a computer which runs a piece of software called RealityCapture that spits out a 3D model and then that’s one frame of our animation. So we do this over time as the mushroom grows and end up with 3D stop motion of a mushroom growing.
Once we figured that out—we tried it, thought it looked really cool—we thought that a great way to witness this and to be able to play with scale and time would be to try it in VR. We then set out to build a VR experience based on our captures, and for the past 3 years we’ve been trying to perfect this technology to make it look better and better and as accurate as possible. We try to put as much of the work onto the computer to make it a low-touch thing.
In the VR, we built a lot of prototypes, and while we were doing that we got really interested in mycology and the world of mushrooms. We went to the Telluride Mushroom Festival, met a lot of people there and found out that there is this whole subculture around mycology that’s really fascinating. So the experience itself is not just about the mushrooms growing but also about the entire lifecycle of the mushroom.
You start off as a spore, coming down into the forest after a rainfall. You then go underground where you become a piece of mycelium; mycelium is kind of like the roots of the mushroom, but also what connects different parts of the forest together to transmit messages and nutrients and that sort of thing. You come back up as the fruiting body, which is the mushroom that we are used to eating, and then eventually you decay, which is the final step and you turn back into a spore. So you experience the full lifecycle within the VR experience.
Temporal World (approximately 40 minutes)
Beyond the greater length of the piece, Temporal World is also the one of the three I saw that had the most complex interface (this is not a bad thing, just more involved). In addition to the headset and headphones, there is specially designed jacket that the user must wear that generates audio feedback and more. Rather than have me clumsily describe the process, here is yet another creator, director/producer Chloé Lee (who was assisted by VR experience developer Lucas Martinic). They have worked on it together since July 2022. And although there were more than a few technical glitches in my viewing, the experience was frequently beautiful and sometimes awe-inspiring.
Chloé Lee. The project is a haptisonic, VR memory world. It comes with a custom haptic-jacket component. People enter my memory world, which is represented by 3D scans from my personal experiences in different places that I’ve lived or have traveled to. People enter these and I guide them to slow down and move through the world in a different way. So, although they’re my memories and my experiences, it’s really an opening for people to see how they can interact with the world in a different way and reflect on how they relate to place. It’s very atmospheric and there’s a focus on the ephemerality of memory. So you see these landscapes shift as you walk through them, according to how you move and the speed at which you move, and everything is sculptable.
This also means that you generate your own sound. It’s very heavy in my own field recordings and my experiences from the last few years of my life, and you hear that, but it also means that you shift the haptics in the jacket. And the jacket was developed to replicate my vibrational experience and is based on sound frequency. The sound that’s generated in the world is translated into vibrations of the body. Every interaction you have in the world is felt in the body and you hear the change in the world.
But really, I was interested in how we remember and what should be remembered. For me, growing up in California, I would think frequently about migration and my family’s migration from China and places changing over time. So this started me thinking about place and how we attach to that even when that’s not there. That’s why I have different places in the experience: my family’s ancestral home; Berlin, a place I moved to to research memory and to see what it means to come to a place to build connection where I have no personal history; and my apartment in New York, which I called home for 10 years, and it was a very strange feeling when all of a sudden I didn’t feel like I related anymore and felt disconnected.
So in the VR world you see it’s actually kind of fragmented. I thought that this is a poetic way to experience that, like how do you express that feeling? I hope that people going through it can get something from it to reflect on their own experiences or at least it might change their way of thinking about how they move through spaces.