[+]Origin

At some point during a fascination with the praying mantis, I came across a term called Sexual Cannibalism. It's a trait of female praying mantes and other carnivorous insects, who eat their mates prior to, during, or after copulation. There are a few theories as to why this behaviour evolved -- the theory of adaptive foraging, for example, suggests that the mate may be more valueable as a meal than a partner (lol).

I had also been watching Scavengers Reign at the time, and completely in awe of some of the wildly obscure and chained biological processes that the animation team had designed.

These themes eventually fused together in my head to create the concept of the Alien Seductress -- an alien that is sent to earth to assist her species in their modest goal of biological survival. In this universe, an alien species discovered codes in the human DNA that could be used to optimize their reproduction cycle.

They observed humans and the absolute power of female seduction on men, and devised a strategy to retrieve the human specimen: sending disguised females down to earth to seduce, capture, and feed males into their biological baby making machines...

This concept was probably pulsing in my head for over a year! I played with some storyboards and sketches, and talked about it a lot, but could never really move the idea along. Until Aiji came to me!

Aiji is a director, photographer, and body artist, from Nizhny Novgorod and with roots in Kyrgystan. We met in 2024 in Tbilisi, and I fell in love with her soft charm and whimsical vision of beauty and femininity. We were so in line creatively, but sadly did not have enough time to create together before her move to Belgrade.

In 2025, our paths aligned again in Tbilisi and I told her the story of The Seductress.

[+]PRODUCTION

Aiji was exactly what this story needed. She jumped in as the director, and brought life to the concept with a romantic eye and lo-fi look. I remember how inspired I was after she first showed me the treatment she made to convey our ideas to the Styling, Casting, and Makeup team.

We decided to have the Seductress's lair be at the stylist's apartment, as it conveniently fit the storyline and our production constraints! We shot the film in reverse with the apartment scene being first, to have the bodypaint be fresh. We experimented with shadow play scenes to bridge the gap between the lovely couple entering the apartment and seeing the victim in the machine, without having to do any VFX or complex acting.

We wandered around the underground passages of Tbilisi for the "night walk home" scenes, and ended with the bar for the "seduction" scenes, and a few Alkhanaidze's :)

[+]Post Production

Post went smoothly overall, yes the nodes in the screen capture below are normal. We had designed the story and shots in a way where VFX complexity was quite low. There was only one shot that needed compositing, which we shot on a DSLR to have cleaner footage for tracking. As always, designing the 3D element went through phases of creative blocks and brilliance. The final inspiration that made it click for me was strangely a drawing of the human pelvis in an anatomy book.

I could not have made this baby making machine without Leneeen and the "Sweet Child" that they kindly modeled and shared for free on Sketchfab. I used it pretty much as is for the alien baby, and it honestly made me smile everytime I thought about it (smiling as i'm typing this!).

[+]Credits

Aiji - Director, Camera, Body Art
Snay - VFX & Creative Concept
Ksusha - Styling
Nikita - Makeup, Body Art Assistant
Julia - Alien Girl
Vlad - The Victim
Viki - Second Camera
Solvani Jewelry - Jewelry
Leneeen - Sweet Child 3D Model

[+]About

Above is a demo video of our realtime NGINX log visualizer, created in the style of Defend your Castle. When connected to an NGINX log, the app dispays incoming requests as they approach the corresponding server.

This app development started when we noticed a stream of requests from Open AI bots crawling our website. We felt the need to fight back, and generally understand the invisible landscape of the internet around us. There is a whole bustling world of crawlers, bots, and attackers operating on the internet that somehow our modern way of browsing leaves us completely obvlivious to!

We aimed to use the spatial dimension to comprehend the log in a new way. Requests from the same IP appear from the same spatial location, IPs are parsed to country flags, device names are simplified, and avatar type signals the intent of the request. For example, most ethical crawlers and bots will let their identity be known in the sender URL. Possible malicious agents can be identified when you see requests going to urls such as .env, .git, or /admin.

Future developments include the ability to adjust your NGINX config directly through the web app, providing a real-time defence strategy:)

This app is made with Golang and ThreeJS frontend, compiled to a single binary for ease of use. If you would like to try it out on your own NGINX server log, you can download the latest release on Github.

[+]Credits

Snay - Go / ThreeJS Dev
Kif - Go Consultant

[+]CODERCAT HIGHLIGHT

[+]FLOWER BLOOMS

For this campaign, the team on site in Los Angeles created a floral lagoon, and our task was to bring it to life with a layer of 3D blooming flowers! Before even seeing the footage, I got started researching. I knew from looking at the moodboards that a procedural system would be the way to go, since the shots called for a variation in flower types and levels of detail.
After browsing several tutorials I created a custom procedural setup involving KineFX rigged petals, phyllotaxis, and vellum simulations. With this setup, I could change the flower petal shape and distrubtion to create a wide array of looks!

Some flowers got special treatment of course.

Once the footage came in, I started designing flowers that would match the beautiful arrangement they created on physically set.. but also give the scene a surreal twist. Finding that balance was really fun.

[+]LAYOUT

First step was a sketch to get ourselves and collaborators on the same page about the distribution of flowers and general flow of movement for each of the shots. Kif made these lovely sketches (in Blender by the way) to do so!

In parallel, Kif was devloping the pipeline for layout and scene development. Traditionally we would export the flower assets as alembics from Houdini, and load them in Blender for layout and timing. However for this project, it seemed to us that handling the animation of 100 or more flowers manually in Blender could be a nightmare.

Kif pushed for the bold move of setting up the scene with USD in Solaris. We felt that the instancing tools and procedural nature of Houdini would allow for smoother iterations.

I can't say for sure if we were right or wrong. There were MANY hurdles along the way. Big Con, the layout tools are terrible in Houdini, so bad that Kif actually made his own custom USD Export to Blender that allowed him to do the flower layout in Blender and import it back as instances in Houdini.

Big Pro, updating assets and materials was very smooth, and you can make... a procedural flower field... in just a few minutes...

[+]THANKS TO

Chevy Tyler - Creative Director x Director of Photography
Greg Swales - Creative Director x Director

{full production credits on their pages}

[+]Behind the Scenes

[+]Credits

Snay - Concept, Spine Modeling and Simulating, Sound Design
Kif - Environment Design & Lighting
Bito - Livecoded Music and Visuals in Orca

The above is a section from a short film that we worked on, directed by Nicola Rios. The film was shot in Puerto Rico, and about a night of rebellion, where a teenager confronts the disturbing truth that looms over her house in an effort to break free from her family's dysfunction. This trauma is represented by a giant spider living in the family's basement -- and that's where Codercat comes in.

[+]the Spider

The scope was to have the spider start out in it's lair, crawl out and attack the protagonist. To get us all on the same page before we started working, Nicola provided a pre-visualization animatic of the spider motion for us. It is amazing to me how much a few simple hand drawn lines could convey! It was enough for us to have an idea of the general spider framing in the shot, it's size, proportions, and movement style.

Kif then iteratively modeled, rigged, and textured the spider. I recall that we were in Montenegro at the time, and were able to get real spider reference shots right outside our door.

Kif used these scouted spider references to develop the look and add realism. He also individually groomed every hair on each of the eight spider legs. I will tell him to elaborate on this here at some point!

While the hero asset was being developed, I was setting up the render and compositing pipeline. Each shot had it's own individual blender file, with camera, lighting, and proxy geometry to match the plates. I also created a procedural spider web in Houdini that I could scatter around the room for a nest like atmosphere.

With every major iteration of the spider, Kif would send me his blender file with the character and it's materials, and I would append the spider to each shot.

[+]Compositing

I was the REnDeR WRanGLeR. Most of what I learned here was how to efficiently manage five shots simultaniously. I am very thankful for naming conventions, and also our Blender deploy script! With this script, I could send multiple shots to our remote machine, and batch render them (and their shadow passes configured with a blender script) on their own GPU's.

Everything ran overnight, on it's own, and far away from me. I slept in silence, without the sound of my laptop fan taking off. At some point I was even batch rendering the final composites from Nuke on command line:)

On the creative side, some compositing work went into the spider webs and how to add realism. On top of the render of the 3D web, I collaged in some real spider web textures from a great photo pack on art station.

A Day for Night shot was added to the list later -- breakdown below.

Thank you for reading <3

[+]Credits

This piece is set in a future where biotech researchers created Living Walls — engineered organisms that produce regenerative bathing fluids, much like the human body produces blood. These structures are sustainable in space, feeding on the abundant chemical elements. They grow, repair, and adapt, offering a healing Onsen far from Earth.

Inspired by our walk through Japan, and the many Onsens that healed us on the way. Specifially visually inspiring were the black water hot and cold pools in Nishi Magome!

[+]Behind the Scenes

[+]Credits

Snay - Director, 3D Modeling Interior, Edit, Color
Kif - 3D Modeling Exterior

[+]Behind the Scenes

[+]Credits

Anna van der Velde - Director
Florian Christopher - Director
Snay - 3D Simulations
Kif - 3D Environment

Full production credits on the Official Youtube Video.

[+]AERIAL WORLD

For this campaign, the team created a large physical set of giant holographic looking crystals. They wanted an overhead shot to reveal the entire crystal labrynthian world that Paris Hilton finds herself lost in, with a bedazzled flip phone. She quotes, "I THINK I TOOK A WRONG TURN".

That's where Codercat came in! Our goal was to extend and replicate the physical set in 3D for this overhead angle, including Paris Hilton herself.

We started off with developing the material and form of the crystal labrynth walls. This was an interesting task, because we needed to strike the balance between continuity with the physical set, and visual impact of a fantasy crystal formation.

Next task: create a digital clone of Paris Hilton on set. This was necessary because we did not have a plate of Paris walking from an overhead angle. Thankfully the 3D replica did not have to be 100% photoreal due to the angle and distance, but damn there was still a lot of work that needed to go into this to be convincing, even at this framing!

Kif was the true mastermind here, 3D sculpting her face and body, grooming her lovely blonde locks, reprojecting face & body textures from the reference plates, and rigging. I supported with a velour tracksuit made in Marvelous Designer and a walk animation.

Putting it all together for the final 5 second delivery was an iterative process with the team. Honestly so thankful for Chevy and Greg, our liasons and creative directors. They made the process so smooth, giving us just the amount of creative space we needed, pushing the visuals to their vision, while also appeasing clients! Enjoy the breakdown video below highlighting just a selecta of moments from the development process.

[+]EYE REFLECTION

This was a lighter (but still involved) task of adding a sparkling perfume bottle in the reflection of Paris Hilton's eye. It began with contacting the fabrication team for a production model of the perfume bottle -- rendering with a bit of rotation for that specular lens flare we all love -- and compositing onto Paris's eye! Enjoy the breakdown video below.

[+]LOVELY THANKS TO

Chevy Tyler - Creative Director x Director of Photography
Greg Swales - Creative Director x Director

{full production credits on their pages}

YOU AND ME TO INFINITY

Inspired by the concept of "Glanding" in Surface Detail, by Ian Banks. In this future, some of us chose to install specialized chemical receptors in our body that bind to designer drugs efficiently. We skip the digestion process by inserting the pill straight into the cavities of the body. The pill works its way through the body in a chain of natural processes, creating unique and long lasting effects.

Advancements in our understanding of the human body have led to this products development, allowing for new ways to experience our organic biological machine, would you get one?

Behind the Scenes Coming Soon...

[+]Credits

Lela Laperashvili - Actor, Co-Creative
Snay - Director, 3D Modeling & Simulations, Edit, Color
Kif - Camera, Tracking, Compositing
Revolution Bar - Venue
Abragi - Venue

[+]THE DECORATOR /Episode 2/

The Decorator will be here in a few minutes... and I have a party to attend in one hour. Should be more than enough time for a serpent's gaze.

[+]ZOLTAR /Episode 1/

The Zoltar Machine outside OMG Pizza in Brooklyn is generally disfunctional. But in this parallel universe, the Zoltar's fortune is a new look (swallow to download), for your night on the town.

[+]Credits

Snay - Half of Everything
Kif - Half of Everything

About

This is an interview with Kif & Snayss, collectively known as Codercat, a digital artists duo who have been nomadic for 6 years. Their worldly belongings mostly fit into a backpack, and they get around the world with public transit and a pair of Brompton folding bikes. They are some of the most creative, fun, resilient, and resourceful people I know.

I've been good friends with these two for a few years now and I've always been a huge fan of their work and their lifestyles. Finally, I had some time to sit with them and learn about their stories.

The Bike-packing

The bike community is one of the most wholesome and uplifting communities out there. Doesn't matter who you are or where you are, when bike-packers meet they just so naturally connect with each other.

Nico: What was your first long-term bike-packing journey, how was it and did it change your lifestyle in some fundamental ways?

Kif: We met in SF and both worked as creative technologists. We would go biking on weekends, sometimes we would take longer trips and bike from SF to Santa Cruz or something. So biking has been part of our lives for a while.

One year we went on world travel with no return date. Our original intent was to hitchhike. Mexico was the first destination where we landed and where I convinced Snayss to go for a Bike trip to Guatemala. It was not an easy decision for her.

We flew to Mexico City, bought used bikes for under 500 dollars, and biked to Guatemala. We would bike on average 5 hours a day, we stayed in tiny motels in rural places only camping where we could not find any place to stay for a bike day distance. We only took buses when it was physically impossible to bike (in some areas the wind was so bad even buses were shaken from wind)

The journey was hard but unforgettable, it really tested our friendship and physical abilities. We saw a lot of rural Mexico and came to appreciate the scale of the land. There are also parts of it that were extremely hard.

We stopped in a city called San Cristóbal de las Casas which is about 4 days biking from the Guatemala border, we went to the market and bought a piece of cheese from a local lady. The cheese was spoiled and we both spent 5 days in bed with fever and throwing up. We could not stay any longer and decided to bike. That was the most difficult four days because our stomachs can not process any food that our bodies needed so desperately. On the 4rth day when approaching the Guatemala border, I almost passed out from exhaustion. Luckily it was the end of the stretch and we rented a room right at the border where I went to bed immediately.

Snays: It was so physically and mentally exhausting. Most of the time your mind gets tired before your body does. Many times I thought I was ready to drop and could not continue biking, but then I realized it was just my mind giving up, my body actually could still push forward more.

The Bicycle Kitchen Community

After the Mexico trip they moved to LA. Over there Kif started volunteering at Bicycle Kitchen, a community bike shop. They learned how to fix bikes and met a really cool and awesome community there.

Snays made her first DIY custom bike. We eventually decided to get into folding bikes for international long distance travels since it was a lot easier than buying and selling bikes on different continents.

Fashion for a Minimalist

I’ve always admired Snays’ incredible styles and how resourceful she is with materials and objects found anywhere she goes.

Nico: Snays I know you are very artistic and you’re super passionate about fashion. Was it hard to become nomadic to get rid of belongings?

Snays: It was difficult! In college, I had a daily goal to never repeat my outfit. So I had over 100 different t shirts and around 30 pairs of shoes. Before setting off on our adventure, I somehow had to transition from 100 shirts to 2 that could fit in my backpack. I sold most of my clothing and gave the rest of it to a homeless person on the streets of SF. Having limited clothing and space was difficult at the beginning, since now any item I wanted to purchase came with a plethora of practical constraints. Can I wear this item everyday? Will it take up too much space in my backpack? How does it match with my other clothing items? Does it hold up in the weather? One big optimization problem. Overall now I find that the constant rotation of a few optimally selected items allows for a fluid and evolving style.

Kif: She’s very resourceful when we move through places. When we moved to a new country Snays would swap for items, and always find creative ways to outsource items that fit her style.

Nico: Once you think in that resourceful way you can realize that we live in such an abundant world. So many people have way more than what they need, and they don’t even have time to properly give these items away.

Snays: One day we were having a conversation and saying how it’d be so nice to have a monitor, literally Kif went to take a walk and found a monitor on the street, in great condition! Every time when we move, I assess all the items as they go inside in my backpack, making sure they truly deserve some of the very limited real-estate. It's been interesting to see how things that I once thought I couldn't live without, end up eventually being left behind. This constant re-evaluation of belongings is a moment to also re-evaluate my own goals and desires.

Creativity in Virtual Worlds

Snays: Creating digital avatars became one way to be expressive. I love that I have infinite imaginative space and materials to create in the digital world.

When we started to minimalize our lives we gained so much more mental and physical space to dedicate to our creative explorations... Our cost of living is a lot less, our needs and wants became very minimal, we make enough to cover basic survival needs and we work minimal hours on client work, which gives us a lot of time to delve deep into learning programs and making our own art.

Houdini changed my creative practice in the 3D Graphics world. I used to mostly do programming and stay away from actual 3D asset modeling, however with Houdini I could now use math and code to create abstract interesting shapes. It gave me a limitless vector to dive into asset creation.

Kif: I have a background in coding and visual effects (VFX). I didn’t care that it was difficult to ‘make it’, I just loved the craft of doing 3D. The fact it was so difficult really excited me, it felt like the Wild West. I didn’t want to do things that were easy or predictable.

It’s also great when we share the love for similar creative expressions. We really excite each other with our passion for our work. Working as a couple is challenging of course, but it’s an amazing feeling that you exchange love and you get more love back.

At first, we had a complementary skill set, I was doing a lot of Houdini stuff and Snays was doing VFX shaders. We constantly learn from each other and we are actually reversing our skills by doing what the other person would do in a project, which is kind of funny 😄

What's in the Future?

Codercat: Making and maintaining social ties on the road has been the hardest part for us. Mars College is amazing for this reason. We spent a few years looking for a community like Mars, and when we found it, it was exactly what we needed. There are so many creative people here, hackers, musicians, artists… it’s a great community.

Mars College is a great place to experiment and explore, but it doesn’t quite have a creative focus, we love it for that reason but we also want to create space for people to focus and get some concrete results. Hopefully, we can make that happen soon.

We had an idea to gather a few people together with complimentary skill sets, find somewhere low-cost to live for a month, and make a project together. So we did this with the KikiYago projects in Tbilisi, and we have plans to dedicate more to creating these kind of creative containers.

Hot Tips and Jump Starts

Kif: Finding community really helps, going out to the world in the void by yourself is hard. We spent 4 years finding communities, and we had each other which was amazing, but having a community really helps uplift your mental health, so anywhere you go, anything you do, do it with cool people you like.

Change is scary, there can be many fears, fears of traveling are real, like what happens to my friends, my clothes, my career, money…. All that is reasonable, but if you really believe in yourself and your ability to make things work, you have to forget about those fears and just do it.

Snays: When I left SF, my job and belongings, I knew that SF would always be there, that I can always go back. And it’s the same for anyone, wherever you are and whatever you’re doing, you can always go back. Knowing that made it a lot easier for me to jump-start this journey, funny thing is I never went back.

Kif: You should definitely save some money. Have a lot more runway than you think you need. Calculate the minimum amount of money that you need to survive.

For me for example it was $300 a month, and I would live in places like Thailand, and it would be fine. Most likely you would figure out ways to make money on the road, people do it all the time and it’s part of the challenge also part of the fun. But having your bare minimum covered for the rest of your mind is nice.

I would recommend a bag size of 30 liters, if you have a larger bag you would stuff it with shit you don’t need. Don’t get a larger bag. 30 liters is all you need.

See Ya Next Time

I had a lot of fun with this conversation. It also gave me some inspirations on ways to live my life, and I start to question a lot of decision to accumulate ‘stuff’ a lot harder now. My life has become a lot more minimal, not yet in a backpack, maybe it never will but hey that’s ok! The minimal mindset has been incredibly helpful.

We don’t always have the chance to think about our priorities, we grow up with a lot of these priorities defined for us. When we do question them, we often realize that a lot of the ‘must-haves’ can become options. Life is full of tradeoffs, and the ability to prioritize makes our lives richer or harder.

One big motivation for me to continue working on Grid Free Minds and Agartha is to create resources for people who want to ’try out the lifestyles outside of societal default’.

For all the dreamers who are itching to wander, hope you’ll soon discover a reason to do so.

Subscribe to see more ;)

Credits

Nico Shi - Writer / Interviewer

The Kiki Yago Replication Facility. Behind the Scenes Coming Soon...

[+]Credits

Dasha - Vocals, Face Capture
Snay - Director, Unreal Engine
Kif - Look Dev Consultant

[Bathhouse No.9] /Part One/

Tskaltubo is a town in the Imereti region of Georgia. During the Soviet period, the town was developed into a prominent spa resort. Lavish sanatoriums and bathhouses were constructed in the name of proletarian health and vigor. The magic lies in it's radioactive thermal springs which were believed to have healing properties. Just the optimal amount of carbonated radon to cure your bones.

[Her Landing] /Part Two/

After the Soviet Union brokedown, the thriving spa industry declined and Tskaltubo along with it. The grand sanatoriums and buildings however, still remain in their faded grandeur. A few buildings were made into shelters for refugees from the Abkhazia war, and a few esteemed guests have been delivered to scope out the property.

Part three coming soon... follow our Patreon for behind the scenes content!

[+]Credits

Snay - Director, HoudiniFX Simulation, Edit, Color
Kif - Camera, Geometry, Tracking, Compositing, Look Dev
Dasha - Actress
Detlaff - Drone Pilot

[+]Visual Concept

For the visual concept we were inspired by biological and organic images of membrane, alien tissues and embryos. The concept of birth is also very inspiring and prominent in our previous work that we want to iterate on. We were also inspired by our friend malgavochka and her nails :)

After conducting several discussions, we made the decision to create a sack made of fabric with an actress inside it. We were fortunate to discover a high-quality, transparent and stretchable fabric at a local store. Our team member, malgavochka, made the sack using this fabric. To test its look, we did a test shooting by placing Marina inside the cloth and spraying water over the fabric. As we lacked appropriate lighting options, we incorporated bike lights into our setup. The combination of the wet fabric surface and the light emanating from within the sack showed promising results.

[+]Approach and tools

Making visual effects on live action footage is a hard process that requires several steps such as camera tracking and compositing.

Camera tracking is used to estimate camera motion in a video using software like PFTrack, SynthEye, or 3DEqualizer. These programs track high contrast points on footage to estimate camera movements, producing a virtual 3D camera motion. This process is essential for adding computer generated elements to videos. The 3D camera's motion is then applied to synthetic elements making them appear to be part of the scene.

Compositing is another important step when making visual effects with live action footage. It is a process of overlaying synthetic elements on live action video in a coherent fashion. Compositing tasks might include green screen removal, camera tracking, rotoscoping, color correction, object removal and more. The full scope of work depends on the particular task.

We have done several experiments in the past where we incorporated some synthetic imagery over live action footage. It is a fun and interesting process but it takes a lot of time to make something that looks professional and real.

For the Cocoon project we wanted to take a different approach where we separate synthetic and live action imagery. This approach simplifies the work by eliminating the camera tracking and compositing steps.

By keeping things separate we were able to make full 3D environment in Unreal Engine without worrying about live action. In fact, we finished the first version of Cocoon fully in 3D before we filmed the live action.

We choose Unreal Engine for this task because we knew we wanted to render a couple minutes of full 3D environment. A couple minutes in VFX terms is a lot! 2 min of video will result in rendering 3000 frames! Unreal's ability to render beautiful looking light and shadows real time is game changer. Let's say we choose Blender Cycles for the rendering. I've done test with similar scene that took one minute to render a single frame which equates to 50 hours of rendering time in Blender! This exact fact hinders ones ability to iterate on the work, which has a big impact on creativity. On the contrary we can render the entire 3000 frames of animation in a matter of minutes.

For all of the geometries in the scene we use HoudiniFX. At that time Snay was experimenting with FEM (Final Element Analysis) and Vellum simulations which fit perfectly with the organic look we were going for. For transferring simulated meshes to UE, at first we tried to use VAT (Vertex Animation Textures) from SideFX Lab tool set, but it was difficult to work with. At the end we settled on Alembic caches since performance was not a big concern for us.

We put a lot of work into texturing and shader look development for all the assets because it was challenging to achieve that soft tissue look in real time rendering engine. We were excited to try generating textures in Houdini but that was harder then we first thought. After several failed attempts we ended up using Substance Painter and making PBR textures by hand drawing over a few procedural mask such as thickness and curvature.

[+]Filming

For shooting we needed a place where actress can lie on the ground, while positioning the camera above her. Behind our house, we discovered a small creek with a tree that the camera person can climb and shoot a subject on the ground. However, this location had a few disadvantages. The ground was somewhat muddy, and the presence of numerous mosquitoes posed additional challenges. Prior to the scout, we collected various dead branches, which we believed could contribute to the scene. We had the idea of constructing a nest of sorts for the actress to lie in within the sack.

To address the issue of the actress coming into contact with the muddy ground, we used a transparent cover from our kitchen table, which proved to be an ideal fit.

On the day of the shoot I equipped with a saw and an axe went and built a nest out of the prepared branches. I also climbed the tree and did a few test shots. Shortly, Marina came prepared with tools and supplies for face make up. She even brought her own mirror. After her make up was applied, Snay and I did a few test shots of her face to test the camera setting and bike lights. Snay experimented placing different color plastic bags over the light, some of them looking very interesting and similar to moon light. However, the power of the bike light was not enough to light our large scene.

Around 6:30pm (about 40 min before sunset), Snay and I descended to the creek, equipped with Lumix S5, bottle of water, fabric for cocoon, couple of bike lights and glass of beer Marina.

It was already dark in the creek even though the sun wasn't fully down. Since everything was ready we started shooting immediately. I climbed the tree with the camera. Marina wrapped herself in the cocoon on the ground directly below me, and Snay was on the side wrangling the light and directing. We ended up placing one cold blue light on a side tree branch, pointing approximately 45 degrees down on the cocoon. We set another light to red mode and gave it to Marina inside the cocoon.

The scene looked gorgeous on camera. We experimented filming various poses for about 40 minutes until it came time for Marina to rip the cocoon and escape into the darkness. We knew we will only have 2 opportunities at most to shoot the escape, since we only had one piece of fabric. After completing a few short rehearsals we decided to go for the final fabric rip and escape sequence. I stood over Marina with the camera and shot the whole sequence in one take. Marina did a great job performing and we were all happy with this single take result. We called it a wrap and went back to the house for hot tea and review.

While the girls were drinking tea, I loaded all the footage to DaVinci resolve for us to review on the TV panel we had previously set up in a dark room. I also played some ambient music to the the mood right while we were watching raw footage. Everybody was excited about the result. We called it a day and Marina went home.

[+]Putting it all together

The next day, we embarked on a two-week trip to Turkey from Tbilisi. I only brought my MacBook Air, while Snay brought her Windows machine. During these two weeks, we not only engaged in "parents" and tourist activities but also managed to incorporate our previously filmed materials with the 3D clip we made earlier.

One of the challenges we faced was ensuring a consistent look between the real footage and the 3D elements. While exploring Istanbul, we took a local ferry where I captured footage of the sea water swirling behind the ship's engine turbine. We then overlaid this footage onto the cocoon video using a "subtract" blending mode. The result was quite astonishing, prompting us to stick with this look regardless of any obstacles we might encounter. However, we soon discovered that this overlay approach presented some challenges. It did not produce favorable results for clips with significant motion, and the "subtract" blend mode made the areas outside the cocoon appear pitch black. After some trial and error, we were able to achieve the desired color balance and create a mask specifically for the cocoon portion of the video. This cocoon mask was created using the luma keyer tool in DaVinci Fusion, and remarkably, it worked well across all the shots with only minor adjustments needed.

For the sound track we sampled the Half-Life Alyx official sound track, from a 4 hour compilation on Youtube made by @SalimMuller. It worked very well creating proper suspense and emphasizing the fabric tear.

[+]Vlog

Credits

Marina - Actress, Makeup.
Kif - Camera, Layout, Light
Snay - Director, HoudiniFX Simulation, Edit, Color

[+]About

Maquette is a live interactive performance leveraging motion capture technology, avatars, and dance in an exploration of the parallel histories of averaging and idealism in art and society, written and directed by Lisa Jamhoury.

Reckoning. the third scene of Maquette, is an 8 minute spoken word and motion captured piece performed solo at New Inc's Demo Day 2023 at ONX Studio in New York. It's a moment of introspection given by movement and speech, visually exploring through the sub conscious space of thoughts on deconstructing the digital self and a reckoning with our physicalities.

I was the main Unreal Engine developer for this scene of Maquette, and it was such a fun experience to work together with Lisa and her team! Our collaboration was cross dimensional, moving fluidly from the physical and digital. We first met eachother at ONX Studio in New York, where Lisa had arranged a three day Maquette hackathon. We had the mornings for tech time, speed prototyping some visual effects and avatar manipulations in Unreal Engine. And afternoon, the four dancers were with us to get in the motion capture suit and play in the scenes! I really enjoyed these days, though they were intense and we didn't see the sun, the duration was perfect to get into the zone. We prototyped some really fun things with the four dancers, blending their motions together, or turning them abstractly into cubes.

After this hackathon, we spent the next month working together remotely over Parsec, Zoom, and two more physical rehearsals at ONX. Building out some of the concepts that we enjoyed when we were together, focusing on the moments where we both would look at the projection and go "wow" in unison. Lisa was really pushing for non linear mapping of movement from the performer to the avatar, to match the deconstruction in the text. We came up with some themes and visual concepts in Unreal Engine together -- my favorite being 100 bodies forming a weirdly organic structure by positioning them evenly on a sphere ( where I learned about the Fibonacci Sphere ) , and breaking up the avatar in Houdini and applying a hover effect to each limb in Unreal.

After reaching an interesting point with the visual themes, we laid out everything onto sequencers and matched the cameras and visuals to a spoken word track in the Unreal Engine sequencer. We broke it out into three sequences that were triggerable by key press, so that Lisa could easily sync the visuals with her speech.

I could not attend the actual show, but kif was in New York and I sent him as a codercat representative to film and enjoy the experience physically! From what it seems from the footage, the visuals, movement, and speech came together in a really interesting way. I loved the stage setup as well, creating an intimate space with a single spotlight and bean bags.

I'm very looking forward to continuing this collaboration, as there will be a full 45 minute showing of Maquette in Nov 2023, including all seven scenes and four dancers. Lisa has mentioned some interesting visuals and concepts for the scenes that still need developing, and I can't wait to explore these with her soon!

[+]Credits

Lisa Jamhoury - Director, Spoken Word
Sneha Belkhale - Unreal Engine Developer
Françoise Voranger - Choreography, Movement, Performance
ONX Studio, NYC - Venue

[+]About

Maquette is a live interactive performance leveraging motion capture technology, avatars, and dance in an exploration of the parallel histories of averaging and idealism in art and society, written and directed by Lisa Jamhoury.

Reckoning. the third scene of Maquette, is an 8 minute spoken word and motion captured piece performed solo at New Inc's Demo Day 2023 at ONX Studio in New York. It's a moment of introspection given by movement and speech, visually exploring through the sub conscious space of thoughts on deconstructing the digital self and a reckoning with our physicalities.

I was the main Unreal Engine developer for this scene of Maquette, and it was such a fun experience to work together with Lisa and her team! Our collaboration was cross dimensional, moving fluidly from the physical and digital. We first met eachother at ONX Studio in New York, where Lisa had arranged a three day Maquette hackathon. We had the mornings for tech time, speed prototyping some visual effects and avatar manipulations in Unreal Engine. And afternoon, the four dancers were with us to get in the motion capture suit and play in the scenes! I really enjoyed these days, though they were intense and we didn't see the sun, the duration was perfect to get into the zone. We prototyped some really fun things with the four dancers, blending their motions together, or turning them abstractly into cubes.

After this hackathon, we spent the next month working together remotely over Parsec, Zoom, and two more physical rehearsals at ONX. Building out some of the concepts that we enjoyed when we were together, focusing on the moments where we both would look at the projection and go "wow" in unison. Lisa was really pushing for non linear mapping of movement from the performer to the avatar, to match the deconstruction in the text. We came up with some themes and visual concepts in Unreal Engine together -- my favorite being 100 bodies forming a weirdly organic structure by positioning them evenly on a sphere ( where I learned about the Fibonacci Sphere ) , and breaking up the avatar in Houdini and applying a hover effect to each limb in Unreal.

After reaching an interesting point with the visual themes, we laid out everything onto sequencers and matched the cameras and visuals to a spoken word track in the Unreal Engine sequencer. We broke it out into three sequences that were triggerable by key press, so that Lisa could easily sync the visuals with her speech.

I could not attend the actual show, but kif was in New York and I sent him as a codercat representative to film and enjoy the experience physically! From what it seems from the footage, the visuals, movement, and speech came together in a really interesting way. I loved the stage setup as well, creating an intimate space with a single spotlight and bean bags.

I'm very looking forward to continuing this collaboration, as there will be a full 45 minute showing of Maquette in Nov 2023, including all seven scenes and four dancers. Lisa has mentioned some interesting visuals and concepts for the scenes that still need developing, and I can't wait to explore these with her soon!

[+]Credits

Lisa Jamhoury - Director, Spoken Word
Sneha Belkhale - Unreal Engine Developer
Françoise Voranger - Choreography, Movement, Performance
ONX Studio, NYC - Venue

[+]About

Within Touching Distance is an experience of Virtual Reality synchronized with human touch, written and directed by Persis Jadé Maravala, and run by the performance team ZU-UK. This piece is very special and I find it incredibly hard to describe in words, but essentially you put on a VR headset in a physical bedroom set, and are gently put to bed by your amma in VR, with synchronized touch by an actress in real life. You then are immersed into an interactive dreamscape narrated by a child, and finally experience a palliative care setting, again synchronized with the touch of an actress.

The human touch immerses you deep into the virtual reality, and for me it had the ability to bring me back to the memories of my childhood, and also somehow push me into the unknown feeling of being elderly. You experience a lifetime in 35 minutes, all from the difference in the way the actor is touching you, and worlds created for you in the headset.

For this project, I was invited to come for a residency in London, to be the resident Unreal Engine Technical Director until the premier of Within Touching Distance at the Sheffield Docfest. Before arriving on site, Codercat worked remotely on creating a 3D representation of the physical bedroom set for the VR part of the experience. The idea was to have a surreal moment while in VR, as you look down at your feet and you are in the bed with same dinosaur sheets, holding the same teddy bear, wearing the same pajamas, but with the hands and feet of a little indian girl.

You start from the first person perspective and lift up to see yourself from above.

We played with some fun tech to create this asset, using a modified Metahuman for the girls body, cloth simulations in Houdini for the pajamas and fabric, and even tried using AI to generate some of the textures from low res images of the real life dinosaur sheets we managed to find on Ali Express (These were quickly turned down by myself, and I simply asked the team on site to take a high resolution photo of the bedsheets :)

Keeping on the trend of cloth simulation and fabrics, Jadé asked me to make a coat rack full of children coats which could be lined along the primary school hallway. Instead of hand designing and simulating everything in Houdini, I found an awesome resource, the Marvelous Designer Asset Store. This store has tons of free patterns that you can load into Marvelous Designer! I found some simple coat patterns, and simulated / exported them straight from Marvelous. Of course there was some hackery that needed to happen to get them to simulate like they were on a coat rack:)

After the weeks of working remotely, I was so excited and ready to visit the team in London. It was quite a surreal experience to step into a room with the 3D assets I had been working on remotely, now physically in front of me... to touch the dinosaur sheets and plaid pajamas in real life! We had one week in Bathway Theater rehearsing with the actors doing the synchronized touch before we were off to Sheffield, and making changes to the unreal engine gameplay as different technical and aesthetic issues came up.

It was quite an intense week of 10+ hour work days... this experience hits right to the core of our human natures, so most participants would come out of the headset in tears...which also had an effect on the actors and team.

But through the intensity there were so many beautiful moments...

{Dinner with Jadé & Jorge & Family}

{Watching the actors get into character and how they gracefully handled the emotions of participants}

{Catching a train with Ross & a care package of mixed drinks}

{Trying to fill up a glass vase with water in blindfolds}

{CHAOS ensuing the moment Ross (the technical director) left}

{Enjoying the set and energy we thoughtfully created at Sheffield (over the days people would start crying as they enter the room before they had even started the experience)}ƒ

{Crying inside the headset when I did the full experience myself}

{Hugs as we break down}

{Love in the intensity}

{SPOONS}

{Preparing a speech for the award with Alice a bit too early:)..}

And so much more...

Still processing this whole experience, but I can't believe that this all started from an email to Jorge, we all followed the coincidences as a sign, and here we are! I've learned so much from this team, their attitudes towards art and creation and honesty. And now I will end on the note of kif's most beautiful work and star of Within Touching Distance, The Teddy Bear.

[+]Credits

ZU UK - Production Company
Sheffield Docfest - Premier Venue

[+]About

Within Touching Distance is an experience of Virtual Reality synchronized with human touch, written and directed by Persis Jadé Maravala, and run by the performance team ZU-UK. This piece is very special and I find it incredibly hard to describe in words, but essentially you put on a VR headset in a physical bedroom set, and are gently put to bed by your amma in VR, with synchronized touch by an actress in real life. You then are immersed into an interactive dreamscape narrated by a child, and finally experience a palliative care setting, again synchronized with the touch of an actress.

The human touch immerses you deep into the virtual reality, and for me it had the ability to bring me back to the memories of my childhood, and also somehow push me into the unknown feeling of being elderly. You experience a lifetime in 35 minutes, all from the difference in the way the actor is touching you, and worlds created for you in the headset.

For this project, I was invited to come for a residency in London, to be the resident Unreal Engine Technical Director until the premier of Within Touching Distance at the Sheffield Docfest. Before arriving on site, Codercat worked remotely on creating a 3D representation of the physical bedroom set for the VR part of the experience. The idea was to have a surreal moment while in VR, as you look down at your feet and you are in the bed with same dinosaur sheets, holding the same teddy bear, wearing the same pajamas, but with the hands and feet of a little indian girl.

You start from the first person perspective and lift up to see yourself from above.

We played with some fun tech to create this asset, using a modified Metahuman for the girls body, cloth simulations in Houdini for the pajamas and fabric, and even tried using AI to generate some of the textures from low res images of the real life dinosaur sheets we managed to find on Ali Express (These were quickly turned down by myself, and I simply asked the team on site to take a high resolution photo of the bedsheets :)

Keeping on the trend of cloth simulation and fabrics, Jadé asked me to make a coat rack full of children coats which could be lined along the primary school hallway. Instead of hand designing and simulating everything in Houdini, I found an awesome resource, the Marvelous Designer Asset Store. This store has tons of free patterns that you can load into Marvelous Designer! I found some simple coat patterns, and simulated / exported them straight from Marvelous. Of course there was some hackery that needed to happen to get them to simulate like they were on a coat rack:)

After the weeks of working remotely, I was so excited and ready to visit the team in London. It was quite a surreal experience to step into a room with the 3D assets I had been working on remotely, now physically in front of me... to touch the dinosaur sheets and plaid pajamas in real life! We had one week in Bathway Theater rehearsing with the actors doing the synchronized touch before we were off to Sheffield, and making changes to the unreal engine gameplay as different technical and aesthetic issues came up.

It was quite an intense week of 10+ hour work days... this experience hits right to the core of our human natures, so most participants would come out of the headset in tears...which also had an effect on the actors and team.

But through the intensity there were so many beautiful moments...

{Dinner with Jadé & Jorge & Family}

{Watching the actors get into character and how they gracefully handled the emotions of participants}

{Catching a train with Ross & a care package of mixed drinks}

{Trying to fill up a glass vase with water in blindfolds}

{CHAOS ensuing the moment Ross (the technical director) left}

{Enjoying the set and energy we thoughtfully created at Sheffield (over the days people would start crying as they enter the room before they had even started the experience)}ƒ

{Crying inside the headset when I did the full experience myself}

{Hugs as we break down}

{Love in the intensity}

{SPOONS}

{Preparing a speech for the award with Alice a bit too early:)..}

And so much more...

Still processing this whole experience, but I can't believe that this all started from an email to Jorge, we all followed the coincidences as a sign, and here we are! I've learned so much from this team, their attitudes towards art and creation and honesty. And now I will end on the note of kif's most beautiful work and star of Within Touching Distance, The Teddy Bear.

[+]Credits

ZU UK - Production Company
Sheffield Docfest - Premier Venue

[+]About

Oneness is a 360 immersive installation and performance, exhibited in the CSULB Innovation Space in collaboration with Behnaz Farahi. The piece envisions a future where an army of women wear robotic masks to communicate secret messages using AI-generated Morse code. The concept is brought to life by a physical performance, costume design, robotics, 3D graphics, and sound design, immersing you deeply into this powerful fictional environment.

I worked on the digital aspect of this project, creating the 360, 8K, 7 minute render to project on the dome during the installation. I'm really proud of the technical achievements during this project, specifically with the high resolution fabric simulations brought to Unreal Engine, compositing and rendering in 8K out of Unreal Engine.

And I'm also so happy to get the chance to work with Behnaz! I had been following her work for a years, she makes beautiful algorithmically oriented, robotic, 3D printed fashion, with a critical eye for design and story. A recent piece I saw from her was called Can the Subaltern Speak. Inspired by the Bandari women in Iran, Behnaz created a 3D printed mask with 17 mechanical blinking eyelashes. The idea is that the women wearing these masks are able to communicate with eachother through blinking their eyelashes in morse coded patterns. They develop their own language, empowered with their faces concealed.

This project blew my mind, first of all the intricacy of the 3d printed design, and then how it so critically brings technology into the social phenomena of the (male) gaze, and allows us to think of a future where this mask could be used as a protective element, a biological enhancement.

I reached out to Behnaz, asking her if she would like to collaborate, as I felt like we could be a harmonious blend of physical and digital creation. We chatted and I think we both could feel there was a good energy between us. She had access to a 360 immersive projection cylinder dome, and on some beautiful synergy moment we imagined how powerful it would be to digitally recreate her mask from "Can the Subaltern Speak", and create a scene with hundreds of women wearing the mask in chador, slowly walking towards you. And in the center of the physical space, there would be a real woman wearing the physical mask, and a long white chador that extended to the diameter of the room, with wind underneath creating ripples in her fabric. This was our fever dream giving us chills as we spoke it out loud, and I can't believe we made it happen, one step at a time.

We started this journey on the technical side, Behnaz sent me the 3D model for the mask, and I loaded it into Houdini to rig the 17 eyelashes, and then animate them in Unreal Engine. Of course this process reminded me of how Houdini can literally do anything.

Now that we had one mask digitally replicated, we could have hundreds! We started thinking about the aesthetics for the scene and how to shape it for the cylinder dome. We loved the idea of having hundreds of women surrounding the camera, and slowly walking towards it from all angles.

Behnaz has a strong love for the ocean, living just a few minutes from the beach and surfing every morning:) So she was keen on having an ocean in the background, for the surreal feeling of the women emerging from it.

On the technical side in Unreal, this could be implemented by mirroring one 4K video, however we quickly realized that mirroring places a large aesthetic restriction and creates various glitches in the shadows we would have to work around. This meant that we had to go all the way, create the full scene in Unreal Engine, and render all 360 degrees. Although this meant having a super heavy scene in Unreal Engine, I was happy to work towards this way because it would lead to freedom in aesthetic choices later. For example, now we could have lighting that affected each character differently for a dynamic and dramatic scene.

This also posed the challenge of how could to render this 8K footage out of Unreal on my little laptop with NVIDIA RTX2060! I downloaded some plugins that are meant for rendering out 360 video from Unreal Engine, but they simply ran out of memory trying to render the 8K that I needed. I had to build my own 360 renderer, essentially making a rig with four 90 FOV cameras placed at 90 degree angles. I rendered out the sequence individually from each camera, using a post processing material to do the cylinder projection warping (ask me about an interesting moment using ChatGPT to solve the projection mapping equation myself), then stitching each the four images together using FFMPEG.

It was quite a pipeline to figure out! But at the end of it, I was able to get an 8K video for a 360 projection rendered from my bb laptop.

Behnaz took some footage of the ocean with a wide lens, and I played with the lighting in Unreal Engine and compositing in DaVinci to match. We went back and forth with each other testing the different combinations, but eventually we converged to our love for black and white, moody and fog:)

The character AI for this piece was quite simple, as the characters had two motions, walk to the camera, speak to each other, and blink. The part that probably took me the longest was getting the cloth simulation to play nicely with all these states! I did a pretty deep dive into cloth simulation for this project, to solve the challenge of hundreds of simultaneous cloth simulations rendering real-time. I played a bit with the simulation tools in Unreal Engine but they were much to hard to control or get the level of detail I wanted. I decided to go with vellum in HoudiniFX, and exporting it out to Unreal using Vertex Animation Textures (VAT). There was a lot of hackery needed to be able to blend between two VATs (as I needed a walk and idle cloth state), as well as apply a bone transform for the head rotations in conjunction with the simulation.

However once this was solved, it's just one shader lookup per frame for the intricacy of a Houdini Vellum cloth simulation! And it seemed like I could not even hit a rendering bottleneck in Unreal Engine with 200+ characters!

We added some wind, offsets per character, slowed down the cloth simulation speed, and the result was powerful.

The digital side was coming together! With feedback and suggestions from Behnaz along the way, I was really happy in the direction it was going. Since I was making my way to Los Angeles, Behnaz and I decided to have a little 24 hour hackathon together. I was so excited to meet in person! We spent the day in the innovation space at CSULB, testing out different color corrections for the visual, playing with fabric to cover the carpet for the exhibition, getting to know each other, seeing the mask in real life (!), skating and biking around venice beach, looking at her 3D printed material experiments in her studio, and a sweet salmon & wine dinner :) In the morning we even got up at 7AM to catch some footage of the ocean in overcast (which is the footage we used in the final video).

The next week, we also went to the fashion district in Los Angeles, to see what kind of options we had for fabric for the chador that should span the width of the space. I didn't know but apparently most of stores in the area are run by Iranians, Behnaz was speaking to them in Farsi, and it was really fun for me to watch the interactions, everything became louder and more animated, I felt like we had entered a portal escaping America.

After this, I was off to New York. Behnaz spent the next few weeks sending me updates on the physical part of the installation, such as printing a new mask in white, the large white chador, working on the fan system to lift the chador like waves, and a new white flooring. Everything she does in the physical realm, to me is so thoughtfully executed and stunning.

We also brought my friend and musician from Athens, Kostadis into the project to work on the sound design. This was a very fun mixing of my worlds, as I had met Kostadis covered in sparkles at a rave the previous year, I also worked on a music video for one of his tracks Emerald. Of course he is an amazing musician and really it was such a smooth collaboration, he knew exactly how to bring the piece to life, adding elements we could never have imagined (listen for the shrimp sounds:).

The show was on June 10th, I wish that I could have been there in person to witness all the physical elements that came together. Behnaz said that it was a success, many people questioning whether the 360 projection was real footage :) She got a videographer at the event so I will update this article with the footage once I get it!

Overall, I'm so proud of us for making this dream a reality, I think we made something so unique and powerful, from the heart.

[+]Credits

Behnaz Farahi - Creative Director and Designer
Sneha Belkhale - Unreal Engine Dev
Kostadis Michail - Musician

[+]About

Oneness is a 360 immersive installation and performance, exhibited in the CSULB Innovation Space in collaboration with Behnaz Farahi. The piece envisions a future where an army of women wear robotic masks to communicate secret messages using AI-generated Morse code. The concept is brought to life by a physical performance, costume design, robotics, 3D graphics, and sound design, immersing you deeply into this powerful fictional environment.

I worked on the digital aspect of this project, creating the 360, 8K, 7 minute render to project on the dome during the installation. I'm really proud of the technical achievements during this project, specifically with the high resolution fabric simulations brought to Unreal Engine, compositing and rendering in 8K out of Unreal Engine.

And I'm also so happy to get the chance to work with Behnaz! I had been following her work for a years, she makes beautiful algorithmically oriented, robotic, 3D printed fashion, with a critical eye for design and story. A recent piece I saw from her was called Can the Subaltern Speak. Inspired by the Bandari women in Iran, Behnaz created a 3D printed mask with 17 mechanical blinking eyelashes. The idea is that the women wearing these masks are able to communicate with eachother through blinking their eyelashes in morse coded patterns. They develop their own language, empowered with their faces concealed.

This project blew my mind, first of all the intricacy of the 3d printed design, and then how it so critically brings technology into the social phenomena of the (male) gaze, and allows us to think of a future where this mask could be used as a protective element, a biological enhancement.

I reached out to Behnaz, asking her if she would like to collaborate, as I felt like we could be a harmonious blend of physical and digital creation. We chatted and I think we both could feel there was a good energy between us. She had access to a 360 immersive projection cylinder dome, and on some beautiful synergy moment we imagined how powerful it would be to digitally recreate her mask from "Can the Subaltern Speak", and create a scene with hundreds of women wearing the mask in chador, slowly walking towards you. And in the center of the physical space, there would be a real woman wearing the physical mask, and a long white chador that extended to the diameter of the room, with wind underneath creating ripples in her fabric. This was our fever dream giving us chills as we spoke it out loud, and I can't believe we made it happen, one step at a time.

We started this journey on the technical side, Behnaz sent me the 3D model for the mask, and I loaded it into Houdini to rig the 17 eyelashes, and then animate them in Unreal Engine. Of course this process reminded me of how Houdini can literally do anything.

Now that we had one mask digitally replicated, we could have hundreds! We started thinking about the aesthetics for the scene and how to shape it for the cylinder dome. We loved the idea of having hundreds of women surrounding the camera, and slowly walking towards it from all angles.

Behnaz has a strong love for the ocean, living just a few minutes from the beach and surfing every morning:) So she was keen on having an ocean in the background, for the surreal feeling of the women emerging from it.

On the technical side in Unreal, this could be implemented by mirroring one 4K video, however we quickly realized that mirroring places a large aesthetic restriction and creates various glitches in the shadows we would have to work around. This meant that we had to go all the way, create the full scene in Unreal Engine, and render all 360 degrees. Although this meant having a super heavy scene in Unreal Engine, I was happy to work towards this way because it would lead to freedom in aesthetic choices later. For example, now we could have lighting that affected each character differently for a dynamic and dramatic scene.

This also posed the challenge of how could to render this 8K footage out of Unreal on my little laptop with NVIDIA RTX2060! I downloaded some plugins that are meant for rendering out 360 video from Unreal Engine, but they simply ran out of memory trying to render the 8K that I needed. I had to build my own 360 renderer, essentially making a rig with four 90 FOV cameras placed at 90 degree angles. I rendered out the sequence individually from each camera, using a post processing material to do the cylinder projection warping (ask me about an interesting moment using ChatGPT to solve the projection mapping equation myself), then stitching each the four images together using FFMPEG.

It was quite a pipeline to figure out! But at the end of it, I was able to get an 8K video for a 360 projection rendered from my bb laptop.

Behnaz took some footage of the ocean with a wide lens, and I played with the lighting in Unreal Engine and compositing in DaVinci to match. We went back and forth with each other testing the different combinations, but eventually we converged to our love for black and white, moody and fog:)

The character AI for this piece was quite simple, as the characters had two motions, walk to the camera, speak to each other, and blink. The part that probably took me the longest was getting the cloth simulation to play nicely with all these states! I did a pretty deep dive into cloth simulation for this project, to solve the challenge of hundreds of simultaneous cloth simulations rendering real-time. I played a bit with the simulation tools in Unreal Engine but they were much to hard to control or get the level of detail I wanted. I decided to go with vellum in HoudiniFX, and exporting it out to Unreal using Vertex Animation Textures (VAT). There was a lot of hackery needed to be able to blend between two VATs (as I needed a walk and idle cloth state), as well as apply a bone transform for the head rotations in conjunction with the simulation.

However once this was solved, it's just one shader lookup per frame for the intricacy of a Houdini Vellum cloth simulation! And it seemed like I could not even hit a rendering bottleneck in Unreal Engine with 200+ characters!

We added some wind, offsets per character, slowed down the cloth simulation speed, and the result was powerful.

The digital side was coming together! With feedback and suggestions from Behnaz along the way, I was really happy in the direction it was going. Since I was making my way to Los Angeles, Behnaz and I decided to have a little 24 hour hackathon together. I was so excited to meet in person! We spent the day in the innovation space at CSULB, testing out different color corrections for the visual, playing with fabric to cover the carpet for the exhibition, getting to know each other, seeing the mask in real life (!), skating and biking around venice beach, looking at her 3D printed material experiments in her studio, and a sweet salmon & wine dinner :) In the morning we even got up at 7AM to catch some footage of the ocean in overcast (which is the footage we used in the final video).

The next week, we also went to the fashion district in Los Angeles, to see what kind of options we had for fabric for the chador that should span the width of the space. I didn't know but apparently most of stores in the area are run by Iranians, Behnaz was speaking to them in Farsi, and it was really fun for me to watch the interactions, everything became louder and more animated, I felt like we had entered a portal escaping America.

After this, I was off to New York. Behnaz spent the next few weeks sending me updates on the physical part of the installation, such as printing a new mask in white, the large white chador, working on the fan system to lift the chador like waves, and a new white flooring. Everything she does in the physical realm, to me is so thoughtfully executed and stunning.

We also brought my friend and musician from Athens, Kostadis into the project to work on the sound design. This was a very fun mixing of my worlds, as I had met Kostadis covered in sparkles at a rave the previous year, I also worked on a music video for one of his tracks Emerald. Of course he is an amazing musician and really it was such a smooth collaboration, he knew exactly how to bring the piece to life, adding elements we could never have imagined (listen for the shrimp sounds:).

The show was on June 10th, I wish that I could have been there in person to witness all the physical elements that came together. Behnaz said that it was a success, many people questioning whether the 360 projection was real footage :) She got a videographer at the event so I will update this article with the footage once I get it!

Overall, I'm so proud of us for making this dream a reality, I think we made something so unique and powerful, from the heart.

[+]Credits

Behnaz Farahi - Creative Director and Designer
Sneha Belkhale - Unreal Engine Dev
Kostadis Michail - Musician

[+]About

Above is a live motion capture performance set in a fantastical Unreal Engine world and using the Rokoko Smartsuit Pro. Leilani, I, and Kif were once again reunited at Bombay Beach and embarked on Part 3 of our digital journey together.

This time, the Mars college infrastructure has granted us the Media Cave. An empty wooden box with walls painted white for us to play. We found our favorite interactions and angles in the space. Thinking about lighting, physical touch with the metahuman, four body play with the shadows, projections, and physical body.

[+]Credits

@snayss - Unreal Engine Dev
@helloleilanifranco - Mocap and Choreography
@kif11 - Camera and Stage Manager
@mars.college - Venue

[+]About

Kiki Yago Live, a realtime face capture and musical performance that was exhibited in 5 different venues in Tbilisi. We explored live stage visuals in Unreal Engine 5, bringing a metahuman to life with the facial expressions of the live performingXenofontova Dasha.

Over the previous 6 months, Dasha and Codercat had been developing the Kiki Yago project. Kiki Yago is a poetess, who creates music and casts high speed spells to actively use the multidimensionality of our mind. She presents the feeling of being uploaded to the Metaverse and recieving every fragment of information in just a few milliseconds.

We expanded on her story and desires by creating music videos and meditation clips. Dasha would record herself singing her tracks rapidly using a Face Capture application on her iPhone, and send the motion data via Telegram for us to then transfer to the metahuman Kiki Yago. If you are curious about the details, check out this blog post I wrote about our first music video here!

After 6 months of collaborating remotely, we were overwhelmed with excitement when Dasha said she would come stay with us in Tbilisi Georgia, for a 3 month cyber camp! The idea was to spend 3 months living, creating, playing, and working together -- with a focus on developing the first live concert/performance of Kiki Yago!

We started planning for the show by creating a track list -- Dasha arranged the songs starting with the slow, vitalizing, birthing moment "Mother Taiga", transitioning into the becoming of self with "Ya Kiki", then into the rapid lyrical madness of "Siroiha" and "Koni", and ending with dance beats like "Utro" to make the crowd get up and (gabber).

Then Codercat matched this progression with a series of visual concepts from previous music videos and new works that could run in a realtime Unreal Engine scene at atleast 30FPS (this quickly eliminated our forest scene from "STR3AM").

After we came up with a scene list, we started to build a live VJ setup in Unreal Engine. We duplicated the chosen scenes from pre-existing videos and refactored them into scenes that could be used live. We built a few controls to VJ live during the show, such as switching scenes, changing camera angles, lighting scenarios, IK Controls, and shader effects.

At the end of this first iteration, we set up our first rehearsal in a local hackerspace, F0RTHSP4CE just a few minutes from the house. The hackerspace had an awesome basement with sound equipment and a empty white wall for our projection visuals.
This was our moment to rehearse the face capture in a dark club environment. We set up a selfie tripod ( which we somehow manifested on the streets of Tbilisi ), and attached some lights so that Dasha's face would stay lit and trackable. Originally we thought to have Dasha and Kiki both center stage, but weird recursive things happened when the face capture application would see the projection of Kiki and detect it as a face.. :)

Our amazing musician friend and Kiki Yago Collaborator, She Monkey, joined the Team as a DJ / sound mixer -- and we ran through the whole set! With some iterations on placement, lighting, and network issues, the live face capture seemed to be working pretty smoothly and synced to Dasha's vocals! And of course, hearing Dasha sing these songs that we had been working with, live, was really exciting and energizing.

After two weeks of rehearsals and building on our VJ setup, we were ready to hit the venues of Tbilisi:) We played 5 shows, some under bridges, some in basements, and some in strange party houses far away from the city. We cheers with some cha-cha, and brought our energy to each show. Best team ever, and we can't wait to reunite :)

[+]Credits

@snayss - Realtime Unreal Engine and graphics dev
@xenofontova_dasha - Vocals, realtime face capture
@kif11 - Environment artist and on-set director
@destroyer4destroyer - DJ, sound mixer

[+]About

Above is a music video for the single YA KIKI, by the Poetess//Princessa//Gadgetessa Kiki Yago. A Metahuman whispering cyber mantras to you from the digital realities of Unreal Engine 5. Come closer my friend with a sweet face, touch her feathers, she wants to tell you something about herself.

[+]Credits

@kiki_yago - Musician and Actress
@snayss - Unreal Engine Development
@kif11 - Editor
Unreal Engine 5 - Venue

Kirill and I (Snayss) recently made a music video in Unreal Engine 5, exploring the new advances in Metahumans, Lumen Lighting, and Face Capture technology. This was a very cool exploration in digital production, and left us feeling like we had just been "on set" for one month :) I want to go through some of the details of the project in this article, as I think it will be interesting to those with eyes on the metaverse!

THE METAHUMAN

Our drive to start this project actually emerged from an introduction to the Metahuman software for Unreal Engine 5. I had seen the amazing advances in Unreal Engine over the last few years, but could not convince myself to learn this beast of a program until I saw THE METAHUMANS. Metahuman Creator is a web app that allows you to create and customize an avatar. During this process, the app streams to you a video feed in order to avoid any client side render costs. When you are ready, you can export and load all the textures / models / shaders / blueprints for a functional metahuman in Unreal Engine.

It’s very mesmerizing to use this tool, blending between different faces and sculpting the face and textures to your taste. It’s really hard to look away from her, watching her animate shyly, looking at me, looking away…

We did the first pass at recreating Kiki Yago - we found this task quite challenging, as the tool always seemed to clamp the features we wanted to sculpt, for example we could not find a way to get the avatars eyes to be wider apart. Anyway, we sent this draft to Kiki and she said that it was cool but definitely did not look like her, and that she would give it a try.

I was skeptical that it would be possible to create an avatar that looks like yourself, since we all have some sort of self perception deficiencies and biases… but I was proven wrong! Kiki’s versions capture the essence of her facial features way better than ours, even with our restrictions that she must be bald:)

I am very curious now to create my own avatar, and think it could be a cool exercise in self perception for all of us, since it’s the first avatar creator that can actually photo-realistically look like you, but do you even think you look like you?:0

FACIAL CAPTURE

The goal was to get Kiki to sing and record her facial capture for the entire song, keeping in mind that Kiki was in Saint Petersburg / Russia , and we were in San Francisco / USA. We decided on using the FaceCap application, since our friend used it for her Little Martian project and had promising results, and all it needed was an iPhone 10 or above! Kiki downloaded it on her iPhone, and could send us the FBX over telegram!

It’s amazing with the technologies that we have today and the power of virtual production, we were able to come up with a face capture solution that did not require any studio equipment or even being in the same country!

Kiki sent us some short initial test recordings, and we spent a few days struggling through the technical details of getting the animation to map to the Metahuman facial rig -- this was mostly due to the fact that it was our first time using Unreal Engine for.. anything, and also that Unreal Engine completely changed their animation retargeting system in UE5.

However, thanks to watching this video, its beautiful elevator soundtrack, and the FaceCap morph target re-mapping data that the author generously shared, we were able to get an initial working demo!

This was super promising! Meta Kiki was alive! But obviously there was a lot of work to be done on adjusting the remapping of FaceCap to Metahuman Rig, specifically in the mouth and jaw area.

We asked Kiki to send us the FaceCap FBX along with a recording of herself from a separate camera and a WAV file, on Telegram, our Remote Face Capture Studio. This was a complete picture of the data, so that we had reference videos while adjusting the Metahuman facial rig remapping, and reference audio for aligning the face capture to the main audio track.

After we got a good pipeline going, we could focus on ways to improve the overall effect of the face capture. We found that eye acting and head rotations came across really well and was a good place to focus the performance efforts. It was also critical to over articulate words and press lips together, which was actually a difficult task for this song where Kiki is pushing her lyrical alliterations at 100 words/second:)

Kiki pushed forward on this, coming up with some crazy fast eye blinking action and creepy doll expressions. She split the song into 12 sections of around 20 seconds each, because we found that smaller recordings were much easier to process in Unreal Engine.

TO BE CONTINUED

Once we were confident in our Face Capture remote pipeline, we moved on to the environment design and visual effects! This is a big and slightly unrelated topic since most of this work was done in HoudiniFX -- so i’m going to split that into another article:)

We are really excited about this technology, and to take this whole experience to the next level. Since the metahumans are so realistic, we definitely need to put some work into the 3D environment, shader, and geometry styles that are harmonious. This will be an exploration that i’m very looking forward to, with some shorter concept video renders.

<3 CODERCAT

Credits

@snayss - Realtime Unreal Engine and graphics dev

@xenofontova_dasha - Vocals, realtime face capture

@kif11 - Environment artist and pipeline

Footnotes

A copy of this article also exist on mirror.xyz and medium.com

[+]About

TORN, a realtime looping mocap performance piece done at Mars College, exploring a looping system made in Unity, where our friend and contortionist Leilani Franco, was able to loop and playback her motions on avatars live while wearing the Rokoko motion capture suit.

We had just finished working on Neither Ever Nor Never and were ready for the challenge of using live motion capture data in the performance, instead of making a video using pre-recorded animations. After getting the basic setup of live streaming Rokoko motion to Unity, we started thinking about ways in which Leilani could play with the avatars and her own motions.

We decided on a looping controller, like the ones lone wizard musicians use, for looping motion instead of audio. When she gave me a hand signal, I would stop/start recording her motions into an array of bone transforms. Once the recording period ended, the avatar would get stuck in a loop of the recorded animations. We placed two of these avatars in the scene, one that she looped with her right hand, and one with her left. From there, she was able to separately control the two Avatars movements, and juxtapose it with her own live movements to create a dimensional story of conflicting internal emotions.

We had a lot of fun recording together in the workshop with this initial setup... and decided to make an entire Army for Leilani to control, playing with different army formations and delay effects.

I programmed a 6 minute sequence in Unity hooking up a soundtrack and some visual scene transitions, and after a few rehearsals we were so excited to play it live at the Bombay Beach Biennale. Our location was remote, on top of a shipping container, so we took about one week of figuring out logistics, making sure all devices were able to connect to each other without internet, and making sure we had our own reliable power supply, lighting, projector, and speakers at the location. At the end of all this preparation, we had a traveling Golf Cart that was fully equipped to project live performed mocap at a location near you:)

[+]Credits

@snayss - Realtime Unity and graphics dev
@helloleilanifranco - Dance, realtime motion capture and choreography
@kif11 - Environment artist and on-set director
Bombay Beach Drive-In - venue

GYROID

[+]SHELL

Gyroid formation carved into the skull. A few different designs available and on exhibit before implantation

[+]IMPLANTED

First when I exited cyberspace, I saw it in my shadows. Long and strangely elf like, metallic and cold. And then when I brought myself to look in the mirror...

MARINA

[+]BATHROOM TRICKS

We went to the bathroom at Fabrika together, always in twos. Admiring the red lighting filling and bouncing through the space. Maybe we spent too long inside because soon I noticed Marina's nails were..growing...

[+]THE WALLS

The walls started pulsating into us, overlapping, alternating, she stood up to dance between them.

[+]STRAY

Two stray doggies followed us from the highway to home, clearly enamored by Marina's new gown. They played with eachother viciously to get our attention, so that maybe we would take them home, adopt them into our care. I am surprised that this dress never seems to get too dirty

KAMAJI

[+]REACH

Prototyped for easy access to [whatever you can dream of within 6 feet], extending limbs to reach, gather, organize, fight, dance, in parallel.

[+]GATHER

[+]ORGANIZE

[+]DANCE

[+]INSTALL

Complicated human installation process, since part of the torso must be replaced. Interface of limbs to human brain must be added as a module, for simultaneous control of six limbs. Sally here seems to be doing excellent. Roses, Pills, Shrine, in reach.

ESCAPE

[+]TESTING IN VOIDS

[+]COME WITH ME

The escape was staged in advance. Lora knew what she had to do, and designed her chords accordingly. Prototyped with the strength and flexibility to grab her subject. Lora first created them in her metaverse, then tested them in the white voids, and then, Earth.

Go with her.

Go with her.

Go with her.

PHYSICAL

[+]TO BE LORA

Physically

Digitally

Physical being

Digital form

Physical or Digital

Phygital

No boundary

REGENERATE

[+]4 OF US

Sometimes it's healthy. Tear yourself apart to allow for a new biome to take it's place. Some use fasting and extreme temperatures to trigger the regeneration processes, but here is another non documented non tested approach.

[+]AND OF COURSE PROTOTYPE FIRST

PRINTS

[+]PHASE 1

We begin in the black voids, with infinite material and possibilities.

[+]PHASE 2

Once satisfied, layer by layer we bring it to the dimension in front of us.

[+]PHASE 3

Phase of pure Shock. Can you believe? We were staring at this in the void for hours and now we can TOUCH it. We can touch what Lora touched.

[+]PHASE 4

A debut at Bar Chatsubo. No longer humanoid, worthy of service.

[+]PROTECTIVE CASING

Repeat phases 1:4. This time with an initial 3D Scan for the satisfaction of a perfect fit when you have a flat skull.

[+]ADORNMENT

Repeat phases 1:4

[+]CLAWS

Repeat phases 1:4. Printed in multiple sizes, statistically shown to fit at least the thumb or pinky. Designed with utilitarian functions in mind.

[+]GEARS

She told me she thought about dollification, in the cyborg sense. What would this doll want from you, and for herself?

We developed a system to control, to place the gears where the functionality was needed. A special port that needed to be covered or extended.

Once outside the realms of black void, processes get much slower and tedious. Without much control.

And we imagined a martian wedding, gears locked and moving together.

[+]CYBORG FAM

And when the family comes together.

[+]About

Neither ever, nor never, a live performance piece done at Mars College, exploring story telling through motion capture choreography, motion graphics, and projection mapping.

We metLeilani Franco at Mars College, a 3 month long art/tech residency in Bombay Beach. Leilani is a crazy body contortionist who decided to bring herself into the metaverse with motion capture and her own Rokoko suit. We were so excited to collaborate with her, since we had been researching AI based motion capture and playing around with funky contortion avatars ourselves.

We wanted to create something together that would fit naturally into the architecture of Mars College, which is how the 5 room / 5 girl structure came about. Leilani and I wanted to tell a story of being trapped in emotional phases, shown through the choreography and visual effects. We were also super excited by the idea of her dancing live and interacting with the characters during the projection.

Leilani choreographed the rooms, and then recorded the 5 rooms separately into FBX files with her motion capture suit. She sent them to me, and I did some quick test renders for her so that she could see how they all moved together.

I was so amazed by how the individual sequences lined up perfectly, she said she just had timed it to the music :) I then took these files and added specific effects we had discussed together to each room, with some VEX wizardry in HoudiniFX.

We did a few iterations of motion capture recording and vfx touchups until we arrived at the final picture. Then I handed it off to Kif who beautifully designed the backlit lighting scheme and rendered the piece. It took 6 hours on both of our computers to render this 2 minute piece!

On the day of performance, we gave the video to Vincent and he mapped the video to the buildings surface. After staring at the visuals on my computer screen for hours that week, it was amazing to see them projected on the wall of our "Air" building, and Leilani's live movements emotional and perfectly in sync with the graphics.

[+]Credits

@snayss - 3D effects
@helloleilanifranco - dance, motion capture and choreography
@_the_s0urce_ - projection mapping
@drmbt - projection mapping
@kif11 - lighting and rendering
@mars.college - venue

Heya! So, lately I’ve been thinking about how the current state of media directs us to spending a lot of time focusing on global events and people. We are so connected with the entire world, that sometimes we lose sight of what is happening right outside our windows. Instead of trying to reach the largest audience possible by sharing events and ideas on social media, what if we steered our efforts towards our local communities? This could look like pinning up event posters outside, gathering for mutual aid community dinners, oor hosting a web page on an open local network…??

I had seen this idea in Berlin — Google was attempting to open a new headquarters in Neukölln and the people were not going to allow it. Alongside the protests, someone had setup a local network where Google was planning to build its office. When you connected to the network, it would pop up a page of anti-google propaganda that everyone should be aware of. It was a cool supplementary way to share information, localized to a specific area and idea.

I wanted to implement something like this here in NYC, and maybe take it a step further by including a local network chat to spark the sense of community along with some (abolitionist?) propaganda. I came up with a list of the most important features…

• The page should pop up automatically after you connect (Captive Portal).
• The local network should not need internet connection.
• The router should be able to run a small server for the chat API.
• All processes should survive a reboot.

Lets do iittt::::

The Hardware

We need a portable router for this project. This could come in many different forms, I even contemplated using my Raspberry Pi, but eventually settled on the GL.iNet Slate Router for it’s simplicity and specialized functionality.

This cute portable router comes with an installation of OpenWRT, as well as a Admin Panel web interface. I knooww we all want to be terminal purists sometimes, but this admin panel is actually really nice and helpful to get a visual representation of the router state.

To prep for this experiment, I used the Admin Panel to set up an open guest network named Starbucks WiFi (ask me about it :p) I also enabled a captive portal for the guest network using the interface. Depending on which router you use, you may have to figure out some of those things with commands, but I have been blessed with this interface and will enjoy its benefits:)

Setting up the Captive Portal

Now that the captive portal has been enabled, when connecting to the router, I receive a popup html page! Now to swap it out with our own.

OpenWRT comes with the NoDogSplash captive portal manager. Looking through the docs, I found that the html page that it sends to newly connected users is located in /etc/nodogsplash/htdocs/splash.html.

So all we need to do is copy over our html page with (anti-capitalist?) propaganda using scp from our computer.

Wonderful! Now every time you connect to the open guest network, it should bring you to your (anti-google?) propaganda page.

Allowing Captive Portal to Work Offline

By default, NoDogSplash captive portals will not work without internet. If the router fails to resolve the DNS request for the operating systems “connectivity check” URL (mac uses captive.apple.com), then NoDogSplash will return an error and deem the network/captive portal unusable.

I found this to be quite a blocking problem, since one of my goals was to take my local network chat to places where a WiFi connection was not guaranteed (NYC Metro, street protests, etc). Furthermore, since all data for the chat is local to the router, there really was no reason it needed WiFi..

After scrolling through some Github issues, I found a hack to basically configure NoDogSplash to resolve all DNS queries locally with the same random public address, making them seem to pass the “connectivity check” and allow NoDogSplash to continue.

Setting up the Chat Server

At this point, we basically have enough to recreate the Berlin anti google setup:) However I still wanted to see if it was possible to setup a simple chat server. The simplest chat ever needs a running server with one route for posting a message, one route for getting all messages, a super sophisticated .json file database, and an html page to display the chat.

The router model I have only has 128 mb of RAM.. so I thought about downloading Python-Light, but it was too light and did not have SimpleHttpServer. I decided to just pray and hope the device could withstand the vanilla Python installation.

First I installed Python on the router using the OpenWRT package manager.

Then copied over my server code and new chat html page from my computer ( I wont go into describing how the chat code works here, but its pretty simple and you can check it out here if you want :)

Create an empty “database” on the router to store our messages:

And run the server!

All is well…. but if we open our chat now we can see that requests to the Python server are failing with a connection time out, meaning that the port our server is listening on is not accessible from our chat page. If we remember, NoDogSplash is in charge of the captive portal which blocks most of the ports by default, so we can adjust its settings to allow traffic directed to our server port.

Now, if we run the server again and try connecting to our open network, our chat should pop up as a captive portal, happily GETting/POSTing requests to our local server :)

Survive the Reboot, the Final Boss

If you’ve gotten this far.. you should have a captive portal that automatically pops up with a chat window served by the simple Python server running on the router.

In order to reach our ideal of robustness, we need to make sure our processes can survive a reboot!

OpenWRT uses Init Scripts to configure daemons that run on startup, if we make our python server run as a daemon, it should hopefully be able to survive a reboot.

The OpenWRT docs give a pretty simple example Init Script, which we can modify for our purposes :

Init scripts use the template /etc/rc.common, a wrapper that provides its main and default functionality, such as start, stop, restart, enable, and disable. We need to override the default start and stop functions. If you want to learn more about the details of init scripts, check out the OpenWRT docs on the subject, they are awesome.

If you noticed in the script, our start function has a lot going on in the line:

First off, the process is running in the background — indicated by the &. I noticed that if I tried running the process in the foreground, the Admin Panel wouldn’t show up! I have a hunch that init.d start functions are blocking, so whatever you need to start needs to account for that by running itself as a background process.

Another mystery is that if I just ran (python /root/server.py)& without the sleep bit, the server would not start up properly. After killing the process and starting it again manually using /etc/init.d/chat start, everything would work fine! My hunch here is that something needed for python servers to initialize is not available at boot time… would love if someone has more information on that. For now, the sleep timer works :)

We can copy this file from our computer into the designated directory of init scripts:

And enable it! Enabling the process allows it to startup on boot.

Nowwww cross your fingers and lets reboot!!! To make sure your python server is running in the background after the reboot, you can run

and check if the server process survived :)

We made it! This article may not accurately portray the amount of mental breakdowns that occurred during this exploration, but hey thats linux so it’s implied :) I hope after an adequate amount of flipped tables you are able to get this to work too and we can spread the joy of LAN chats & local environment awareness :P

❤ Snayss

Fog is everything. Its a simple effect that effortlessly adds depth and cinematic qualities to a scene. It’s also has negligible performance costs, meaning it’s suitable for web graphics. Imagine a cube on a plane with a sky sphere.. seems pretty bland but then add FOG and BOOM! Masterpiece.

I think most 3D designers/nerds/whatever are pretty aware of the glories of fog, but from what i’ve seen, the effect is normally just taken at face value. The three.js default fog is definitely beautiful on its own, but I want to share some shader simple tricks on how to take the implementation to the ~ next level ~.

I’m going to be doing this exploration in three.js, but the basic ideas can be easily implemented in shaders everywhere. Also, if you are the kind of person that hates reading and is screaming “JUST SHOW ME THE CODE”, the demo and code is available @ https://stackblitz.com/edit/threejs-fog-hacks

···

So, let’s begin with the three.js fog demo as the starting point. This is the only mention of fog on the three.js examples page, yet it seems to be more about the terrain :(

The article requires a basic understanding of how the shader library works in three.js. If you are not familiar, I would really recommend reading this article on Extending Three.js Materials , its good stuff.

Also, i’m going to mostly just go over concepts, with code snippets here and there, so I advise that you have the full Stackblitz demo code open for reference :)

THE SETUP

So, our first goal is to get to a hold of the fog implementation and have it under our control. One way to do that would be writing a custom shader from scratch, but this would mean having to reimplement a lot of the material features ( present in THREE.MeshBasicMaterial, THREE.MeshPhysicalMaterial, etc. ) that you are already love and are using for your scene.

You could also just copy and paste the bones of the material shaders you are using, and overwrite the fog shader chunks with your own.. This is better, but since it is still a custom shader and of type THREE.ShaderMaterial , it will miss out on the integration with the rest of the scene properties ( including lights and fog).

For example, the standard way to control fog is scene.fog = new THREE.FogExp2(color, density) . For all built in materials, the fog parameters are automatically passed in by the library as uniforms with the correct shader #defines. However, they are ignored if the material is of type Three.ShaderMaterial .

So… what we need is a material that is a default type but also has our custom logic?? Don’t worry, there’s a third option! We can instantiate our material as whatever base material we want (so that it is registered and gets all scene properties), and just sneak in the chunks that we want to replace in the material OnBeforeCompile callback.

Yay! Okay, but let’s take a step back… how do we know which chunks we need to replace?

By studying the shader chunks on Github. No but really, the only way we can successfully use this approach is by getting familiar with what the included shader chunks do, what #defines exist, what varyings are provided, etc.

I recommend two things:

1. Go to Github and look at the shaders chunks that compose each material, and dive into the shader chunks you are interested in.
2. If you “break” the shader by, lets say, writing an invalid line like hellofdksaf #include <fog_fragment>: you will get an error message in the console with the full print out of the shader and all its chunks in one place! it’s nice reading material.

I feel like the setup is actually one of the coolest parts of this lil article, since it sets you up with the freedom to pretty much hack any part of the three.js shader system with little resistance.

Once you’ve done those things, you will see that the fog uniforms are presented in <fog_pars_vertex> & <fog_pars_fragment>, and the implementing is in <fog_vertex>, <fog_fragment>. You will want to start out with the default implementation in a javascript file , that we can slowly modify.

NEAR + APEX COLORS

Finally ready for our first hack! Who said fog had to be restricted to one color?? It may not be physically accurate, but it surely does give an artistic edge to be able to control fog colors that are near and at a distance.

At the end of the fog fragment chunk, there is a float named fogFactor , a 0-1 value that represents where the fragments depth value lies on the scale of no fog, to full fog. It is used to calculate the final fragment color like so:

When fogFactor is 0, the color remains equal to gl_FragColor. When fogFactor is 1, the fragment is at our max depth value and is fully in fog.

We can achieve a dual fog color system by having fogColor depend on the fogFactor as well. If a new uniform fogNearColor is the color of fragments closer to you, and fogColor is for those far away, we can mix the two depending on the fogFactor:

Of course this can also be extended to having horizon + sky fog colors, but thats something for you to try. Moving on.

NOISE

The three.js fog is fixed at every depth value, meaning the fog will look the same in every direction. From the example above it’s not too noticeable since the terrain we are working with is complex on its own, but if you have simpler geo, it becomes pretty obvious quickly.

Real fog has patches, areas that are denser than others. We can try to mimics that complexity by applying a noise function to the depth value.

Perlin noise will solve all your problems. -me

There is a cool resource of GLSL Noise algorithms that I tend to include in all of my three.js project, before even starting to work on the shaders, and they always come in handy.

For the purpose of fog, we are going to need a noise function that takes a 3D input, since we want it to be fixed and driven by the world space position. I’m going to go with Classic Perlin 3D Noise by Stefan Gustavson, but its fun to experiment with others if you want :)

The first thing we are going to need is to set up the world position value as a varying, to access it in our fragment shader. That includes changes in all of our replacement chunks : Declaring vFogWorldPosition in the headers, setting it in the vertex shader chunk vFogWorldPosition = (modelMatrix * vec4( transformed, 1.0 )).xyz; , and using it in the fragment shader. Another note to look at the stackblitz if you need more details.

In the fragment shader, let’s just start and see what the noise looks like by multiplying it with the fogDepth value. Since cnoise outputs a value between 0–1, this should be a kosher experiment.

Playing around with different values of fogNoiseFreq gives us some pretty interesting looks already! But since we are multiplying all fogDepth values equally by noise, we are losing the constant gradual shift to complete fog.

Scaling the noise by the ogDepth value, as well as adding a multiplier for artistic taste should help us maintain that gradual shift.

MOVEMENT

Ok maybe i’ve been staring at the fog too much ( can’t help it in Daly City.. ) , but fog is rarely static here. There’s always a gust or slow wind pushing it ( karl ) along its way.

In order to start making the fog move, our shader needs a time uniform ( reference codeif you are unsure of how to add that ). Then, all we need to do is add a direction to vFogWorldPosition, and the noise will smoothly transition with time.

PERFORMANCE

Having the noise function in the fragment shader looks awesome, but isn’t the most performative option. If you are tied on performance, there is always the option of losing some smoothness and calculating the noise in the vertex shader.

If you are looking for a more subtle effect, you can create a pretty similar effect by approximating with parametric surfaces.

I think I was browsing parametric surfaces for something else.. but when I saw this plot I was like ooooh looks like a blanket of fog!! Again, thats what living in Daly City does to you…

We can use the x & z values of our world position to calculate the surface, and then do a fade base on how close the y position is to the surface value — that way basically any point under the surface would have fog, and any point over would smoothly transition to no fog.

There is some hackery you need to do if you want to add movement to this fog, since this particular function does not look good when x & y move away from the origin.. I solved this by just moving the world position in a circular pattern, but i’m sure there are other ways :)

FIN

Anyways, I hope there were bits and pieces of that that would be useful to some of y’all threejs friends out theree! Feel free to reach out if anything was unclear, or if you come up with some other cool hacks :)

❤ snayz

Resources

Final demo @ https://threejs-fog-hacks.stackblitz.io/

Check out other related repos @ https://github.com/sneha-belkhale

Other VR/Graphics projects @ https://codercat.tk

And renders / screenshots @ https://www.instagram.com/snayss/

For those who are curious, Kif and I thought we would share some behind the scenes of our first WebVR demo, Monster or Friend.

This project was created using A-Frame and three.js in about one week. To be honest, I am generally not a framework person and try to stay as low level as possible while still humane (no openGL lol) - but using A-Frame was probably the right choice for this project.

A-Frame gives you a working controller in just one line, including the controller model, positioning, and highlighting user interactions on the model itself. Also, I was able to open this demo on Chrome, Oculus Go, and my Pixel 2 phone and the views / controls just worked, and this was really cool and something that I wouldn’t want to program from scratch. I also appreciate that A-Frame is an Entity Component System, which may take some getting familiar to if you are used to coding three.js examples in one html file.. but I think it is worth it for organizational purposes.

THE CREATURE

The creature was the starting point of this demo. I have been interested in procedural animation in the last few months, and contributed a little to this repo that implemented inverse kinematics for three.js. I wanted to play around more with this library, so I thought I’d make a 25-legged sea / space creature : )

Inverse kinematics is really awesome, it adds a complexity that would be very hard to manually animate. Every limb of this creature is one bone chain, with a target point ( the head ). Inverse kinematics reverses the forward kinematic equation to determine the bone positions and rotations such that the last bone reaches the target point at every step.

After playing around with randomized target point movement, a tweening library called tween.js, and the forever handy sin function, I found a pretty organic effect that brought the creature to life.

Making a shader for the creature was the next step, I blended some colors along the UVs, changed the colors over time, and used an opacity map to add complexity.

My creature was ready for the world.

THE COLORS

Colors are really hard, and also really important for the purely visual demo that we were building. At this point, we had a nice environment with a spacey sky sphere, little dust particles, and the creature. We decided to consolidate these elements with color themes of 7 colors: 2 colors for the sky, 1 color for the particles, and 4 colors for the creature (it transitions between two states).

We organized these colors into one color component that would be added to all elements of our scene, and then got carried away for a few hours playing around with the endless color possibilities. We used this color palette generator for initial inspiration, sampled some colors from images we like, and tried the colors from our website logo :)

THE CONTROLLER

I mentioned before that the A-Frame controller was awesome because a lot of the functionality was built in… but not all of it. Since movement based on controller input is not easily defined in VR, the A-Frame devs decided not to include it as part of the library. However, it was not too difficult to define our own movement schema, where the user can move forwards and backwards with the trackpad, and change color theme on trigger.

Another thing we thought it was important to add was a tutorial for the controller, to show the users how to move around the scene. We made this 3D Text in HoudiniFX, and added a subtle fresnel shader to the controller. This tutorial hovers in front of the camera until the users first interaction.

THE MUSIC

We felt like the creature might need some entertainment in its new home, so we decided to add in some music to its world. I was dreaming of something ambient and deep, and decided to create it in FoxDot, an awesome framework to make music using Python. I used simplex noise to drive the ambient sounds, and recorded it with SoundFlower and Quicktime. Bellow is a little clip from the process.

ADDITIONAL RESOURCES

The experience

Source code

Other projects