Six months ago, I decided to focus on mixed reality art. The accumulating evidence was clear. The sort of storytelling I had always wanted to do was becoming technically feasible. While researching software solutions, I came upon Adobe Aero and quickly dove in with both feet. A month later, when Adobe asked me to be part of the Augmented Reality Residency, I was over the moon. The opportunity to explore this new medium with support of an incredible team of artists and technicians was truly amazing. This is a summary of the work I create during the twelve week program.
Physical space is incredibly important in AR. Context plays a huge role in an artwork's meaning, as does proximity. Experiences that (seem to) happen in your immediate physical space automatically feel personal and emotionally potent. So, I started my residency by exploring ways of using the space around participants, specifically proximity triggers and spatial relationships (through scale and distance.)
CHERRY BLOSSOM BALLERINAS
EXPERIENCE: When you get close to a blossom, it detaches from the tree and begins to dance around you in circles, indefinitely. It’s like having your own private sugar plum fairies.
MUSIC: Arrival of the Geese by Chad Crouch
IMMERSIVE MANDALAS
Questions are always a good starting point for art. I found myself wondering what it would be like to be inside of an enormous swirling mandala, so I built one. I began my experiments with simple shapes, colors, and interactive controls.
EXPERIENCE: Three buttons allow the participant to control the mandala's spin.
MUSIC: Illumination by Kai Engel
Eventually, I started exploring storytelling possibilities by using figurative elements. This piece contains illustrations and animations that are inspired by the ocean.
MUSIC: Never Forget by Ketsa
Inspired by the biological kingdoms, this piece reminds me of 3d zoetropes where the traditional "illusion" has been replaced by a new one.
MUSIC: Song Sparrow Serenade by Chad Crouch
TARGET PRACTICE
EXPERIENCE: The animation is triggered by a timer. When you tap each of the rings, they begin to rotate starting with the ring you tapped.
MUSIC: Plasticity by Blue Dot Sessions
FEELINGS FEELINGS FEELINGS
This piece was inspired by the emotional overwhelm that many people (myself included) were feeling during the early days of the COVID-19 pandemic.
EXPERIENCE: When you are 2m away from a letter, it senses that you are there and begins to shake and move. When you are 1m away, it rises up to meet you. When you are 1/2m away, they glom on to you, orbiting around you indefinitely. If it all starts to feel like it’s too much, you can get rid of each letter by tapping it, but you have to deal with each one individually (just like feelings.)
THROAT FIRE
EXPERIENCE: One lesson I've learned over the last few months is that immersive environments feel more authentic if both active (branching narrative choices) and passive interactions (optional interactive details) are present. It makes the world feel more complete and deepens the sense of immersion.
So, I included both types of interactions in this piece. Some elements are triggered by proximity. Some are triggered by touch. Others are based on time. When you get close enough, the eyes open and follow you around (using the AIM function in Aero.) When you tap the woman, she climbs the ladder and jumps into the fire, where she is transformed into a butterfly that orbits around you indefinitely.
MUSIC: Headwaters by Chad Crouch
COVID-19 AR ELIXIR
The final weeks of the residency were devoted to a showcase project. When something like a pandemic lands in your society's lap, it's hard to ignore. Personally, I found it difficult to focus on much else, so I decided to put aside previously laid plans and create a project devoted to it. For more info, check out my Behance post about it.
EXPERIENCE: Participants interact with the environment through proximity and tapping. By using active interactions (direct participant choices) and passive interactions (optional elements) I was able to add depth and authenticity to the "world". Subtle ambient animations and sounds (like the breathing lungs) also contribute to the sense of place.
SOUND DESIGN: I created the composition using a combination of field recordings, piano, and my voice. The foundation for the entire piece is a field record of an actual hospital lobby (Courtesy of Tim Kahn.) The main percussive element is a heart rate monitor. I played the piano and sang the choral elements. When you tap the lungs, they play a recording of me breathing. The candles have a proximity sensor that plays individual choral tones, creating a "candle choir".
Choosing to pursue mixed reality has led to lots of new skills. I've been an interactive artist for several years - focusing on illustration, animation, and 2D game design, but 3D is completely new for me. It has presented some predictable hurdles. Thinking spatially is both challenging and exhilarating. For me, since I come from a theater background, it feels like set design on an individual scale. Looking back now, I can tell how much I've learned- technically, artistically, and personally.
LESSONS LEARNED
Getting comfortable with discomfort: Since this is a brand new medium, everyone is a newbie. Fumbling is part of playing and play is how we learn. (Plus, it's super fun!) Reframing “failing” is essential for innovation. When this residency began, I made failure an explicit goal. I kept a list of experiments by my desk and every time one failed, I would celebrate it with a little star sticker. It may seem silly, but I recommend it!
Trust: Over the years, and especially during this residency, I’ve learned the importance of trust: trusting the process and trusting myself. At many points along the way, I felt lost and unsure of what I was doing. But, I kept at it. Time + trying is the magic recipe. Returning to my roots and my traditional workflow (in my case, drawing first) always helped to put me back on course.
Discovery through doing: Experience has to be part of the augmented reality development process. It’s an experiential art form, so you have to live it to fully understand it. It’s best to test early and often.
Letting go: Part of creating augmented reality art (and interactive art, in general) is letting go. When you give a participant authentic choices, there is a certain amount of surrender involved. Very simply, you can't have complete control... and that can be a great thing. Allowing for surprises is part of the process. There is beauty in not knowing.
WHAT'S NEXT?
I plan on continuing my exploration of mixed reality. I'm excited to see how Aero continues to develop. In the past three months it has improved by leaps and bounds. The only way this medium will come into its own is if lots and lots of artists make lots and lots of work. Software that can create rich augmented reality experiences without coding is a vital step in making that possible. For that reason, I think Aero will be a game changer.
Buckets of thanks to the Adobe team that helped me through my residency! Special shout outs to: Silka and George for being brilliant and supportive mentors. Sarah Rothberg and Matt Lyon, my artist peers, for their support, excellent thinking, and inspiring work. Stefanie, Chantel, Alex, Eric, and the entire crew for their dedication, patient assistance, and technical chops. I'm deeply grateful for the opportunity to explore this new medium backed by such an incredible team.
To see more mixed reality artwork, follow me on Instagram.
