THE CITY

The City is an ongoing personal project of mine designed to continually explore and improve AI enhanced virtual production pipelines. It’s a series of videos and images with a common theme and aesthetic that allow me to have a creative north star when learning new tools and creating new pipelines.

So far I have used: SDXL, Flux, Diffusers, Stable Video Diffusion, LoRA, Controlnet, ComfyUI, AnimateDiff, ElevenLabs, Udio, Procreate, Unity and After Effects.

WORK FLOW

The main workflow is img2img or vid2vid on top of images and videos generated in Unity.

The combination of AI and real time 3D engines is a powerful addition to any artist’s tool kit. It provides rapid iteration, exploration, and the ability to create work that you would normally need a bigger team for.

Virtual Production

Each shot starts with virtual location scouting. This gives you a very natural way of choosing shots and camera angles. For this piece, I used a prebuilt asset as the setting. Within Unity, I was able to move props and set the camera to make the perfect shot.

Model Finetuning

By fine-tuning two separate LoRAs on two different datasets, I was able to create a unique style by blending them together. For this I used a mixture of Edward Hopper and 90’s Anime Background Art (Jin Roh and Ghost in The Shell). This gave the scenes a nostalgic New York City feel while creating a lot of detail in the backgrounds.

DATASET

OUTPUT

Rotoscoping AI

The actual animation of the character is a combination of traditional 3D animation and AnimateDiff via Comfy UI. To create the most consistency, I hand rotoscoped key frames in each shot, and fine-tune a model on those frames. This gave the model all extra information it needed to smooth out the generated frames.

HAND DRAWN BITS

It’s important to know when to use the right tool for the job. The same is true with AI. Therefore there are a number of elements in the animation that I tackled by hand.

Ongoing Art Direction

Since The City is an ongoing project, I have a lot of opportunity to explore new technologies and push the visuals. I use this process to flesh out the feel of the series, adding little back stories here and there. This continual process refines the visual language and theme. These images are made with SDXL and Flux via Diffusers. They are then taken into procreate to fix up details by hand. Finally they are post processed in Photoshop.