Kiki Yago Live a Realtime Face Capture & Musical Performance


Kiki Yago Live, a realtime face capture and musical performance that was exhibited in 5 different venues in Tbilisi. We explored live stage visuals in Unreal Engine 5, bringing a metahuman to life with the facial expressions of the live performingXenofontova Dasha.

Over the previous 6 months, Dasha and Codercat had been developing the Kiki Yago project. Kiki Yago is a poetess, who creates music and casts high speed spells to actively use the multidimensionality of our mind. She presents the feeling of being uploaded to the Metaverse and recieving every fragment of information in just a few milliseconds.

Clip from Kiki Yago's meditation series, wisdom from UE5.

We expanded on her story and desires by creating music videos and meditation clips. Dasha would record herself singing her tracks rapidly using a Face Capture application on her iPhone, and send the motion data via Telegram for us to then transfer to the metahuman Kiki Yago. If you are curious about the details, check out this blog post I wrote about our first music video here!

Dasha and Kiki meet in person for the first time, at Cyber Camp in Tbilisi.

After 6 months of collaborating remotely, we were overwhelmed with excitement when Dasha said she would come stay with us in Tbilisi Georgia, for a 3 month cyber camp! The idea was to spend 3 months living, creating, playing, and working together -- with a focus on developing the first live concert/performance of Kiki Yago!

We started planning for the show by creating a track list -- Dasha arranged the songs starting with the slow, vitalizing, birthing moment "Mother Taiga", transitioning into the becoming of self with "Ya Kiki", then into the rapid lyrical madness of "Siroiha" and "Koni", and ending with dance beats like "Utro" to make the crowd get up and (gabber).

Screenshot from our Unreal Engine live stage visual setup, we projected straight from play mode at the show.

Then Codercat matched this progression with a series of visual concepts from previous music videos and new works that could run in a realtime Unreal Engine scene at atleast 30FPS (this quickly eliminated our forest scene from "STR3AM").

After we came up with a scene list, we started to build a live VJ setup in Unreal Engine. We duplicated the chosen scenes from pre-existing videos and refactored them into scenes that could be used live. We built a few controls to VJ live during the show, such as switching scenes, changing camera angles, lighting scenarios, IK Controls, and shader effects.

One of our first rehearsals at F0RTHSP4CE. Testing sound, visuals, and face capture in different lighting settings.

At the end of this first iteration, we set up our first rehearsal in a local hackerspace, F0RTHSP4CE just a few minutes from the house. The hackerspace had an awesome basement with sound equipment and a empty white wall for our projection visuals.
This was our moment to rehearse the face capture in a dark club environment. We set up a selfie tripod ( which we somehow manifested on the streets of Tbilisi ), and attached some lights so that Dasha's face would stay lit and trackable. Originally we thought to have Dasha and Kiki both center stage, but weird recursive things happened when the face capture application would see the projection of Kiki and detect it as a face.. :)

The face of Kiki Yago when she looks at herself through the face capture application.

Our amazing musician friend and Kiki Yago Collaborator, She Monkey, joined the Team as a DJ / sound mixer -- and we ran through the whole set! With some iterations on placement, lighting, and network issues, the live face capture seemed to be working pretty smoothly and synced to Dasha's vocals! And of course, hearing Dasha sing these songs that we had been working with, live, was really exciting and energizing.

Testing testing testing.

After two weeks of rehearsals and building on our VJ setup, we were ready to hit the venues of Tbilisi:) We played 5 shows, some under bridges, some in basements, and some in strange party houses far away from the city. We cheers with some cha-cha, and brought our energy to each show. Best team ever, and we can't wait to reunite :)

Stage info from the organizor of Secret Place
Stage placement ideas for the organizor of Monkey Radio x WAP Point
Codercat wearing Kiki Merch, flying away after a beautiful 3 months with the Kiki Yago team.


@snayss - Realtime Unreal Engine and graphics dev

@xenofontova_dasha - Vocals, realtime face capture

@kif11 - Environment artist and on-set director

@destroyer4destroyer - DJ, sound mixer