Topaz Gigapixel AI - January 2019

My background is in filmmaking and video post-production, so naturally my first entry into AI was in finding ways to enhance footage. Especially a decade ago and more, there were many limitations to shooting in digital. A lot of the things that we produced prior to the introduction of HD cameras did not age well, and I always wanted to remaster a few of my favorite edits. One of them was a short film my friend and now commercial director King Palisoc’s final student film for college, Mahal Ko Si Direk! (I Love My Director!)

This movie was shot on a Canon XL2. While that was the top of the line for MiniDV cameras at the time (2005!), ultimately the image resolution was a measly 720x480. I was able to recapture the original footage from tape, upscaled the footage (using software that I’ve unfortunately been unable to recall) and reedited/colored in 1920x1080.

Nowadays it’s easier to upscale footage, with many options online. I would recommend Topaz Video Enhance, but there are free options as well.

DAIN and RIFE - 2020

Another AI video enhancement technique I looked into was frame interpolation, i.e. creating intermediate frames in between existing frames to provide more motion data, aka synthetic slow motion.

Fast Style transfer via DeepAI.org - February 2021

Deep Dream AI by Google - February 2021

I also experimented with AI solutions to applying an image’s style onto another. As interesting as this approach was, I was looking for a video-centric solution, which brought me to a more dependable, widely used (albeit non-AI) option - EbSynth - Transform Video by Painting Over a Single Frame. Still, I wanted something that had I could run on my own hardware and give me more control.

Text to Image model @ RunwayML.com - February 2021

Just over a year ago I started playing around with Runway.ml’s text2image model. It’s interesting to see how far we’ve come in such a short span of time.

Wombo - January 2022

After a year we were getting even closer, with Wombo being able to generate more coherent images.

Midjourney v.1-2 - July 2022

When Midjourney first came out, my immediate objective was to see how far one could push the algorithm, prompting a variety of scenes and compositions. True enough, especially compared to the images above, Midjourney certainly produced images with a level of coherence we’ve never seen before. Of course apps from other companies have already been operating since before, but there was a certain unmatchable ease one got with Midjourney.

Using repeated iterations of midjourney same-seed renders in animation

Midjourney v.3 and beta - August 2022

Using AI generated assets in traditional animation

Midjourney —test and —testp - August 29, 2022

After a month of using the platform however I noticed that while Midjourney’s images were getting better and better every week, Midjourney seemed to heavily prefer certain styles and subjects. And of course, Midjourney didn’t do animation, so it wasn’t a viable long term solution for me.

Things changed when Stability.ai released stable diffusion to the public. Suddenly anyone could make applications to run on their open, publicly available model. In a few short days several options opened up to everyone, including me.

Stable Diffusion via Google Colab and VoC - September 2022

Disco Diffusion via Google Colab

Deforum Diffusion via Google Colab and VoC

Stable Warpfusion via Google Colab

Links:

Essentials

Gigapixel AI (topazlabs.com) - image upscaling

Video Enhance AI – Video Quality Software (topazlabs.com) - video upscaling

Rife-App 3.20 by GRisk (itch.io) - faster

Other ML apps

Fast Style Transfer API | DeepAI - Applying the style of one image over another.

EbSynth - Transform Video by Painting Over a Single Frame - Applying the style of one image over another (non-AI)

Deep Dream Generator - Google’s AI-driven style-to-image application

Runway - Online Video Editor | Everything you need to edit video, fast. (runwayml.com) - an all-in solution for editing video on your browser

Other text2Image

Dream by WOMBO