I generated a photorealistic music video with AI (GauGAN2)
995,335
Published 2022-03-02
To support these projects and include your name in the next video, here is my Patreon page: www.patreon.com/doodlechaos
Discord: discord.com/invite/7FCrWAzDY7 (I spoil what the next video will be pretty often here)
Twitter: twitter.com/doodlechaos
Official tiktok: www.tiktok.com/@doodlechaos
Shorts Channel: / @doodlechaosshorts2905
Instagram Shorts: www.instagram.com/doodlechaos_shorts/
Email: [email protected]
Here is a link to the web version if you'd like to try it yourself!
Here is a quick explanation of my process for this project:
1. Animate segmentation maps in sync with music in After Effects.
2. Export this animation as a series of images.
3. Convert the image sequence into landscape photos via GauGan2.
I was able to automate the conversion process. My first attempt used AHK to move my mouse and click to upload the correct image, convert, download, and name. This was sketchy and actually crashed my chrome browser somehow (I think due to downloading so many images and they were all being cached?). Attempt two was to run it locally. Unfortunately, the NVIDIA Canvas downloadable software doesn't allow png segmentation maps as input (only .can files). My final solution works by sending images directly to the gaugan.org server via python. If you're interested in trying out something like this, I'll make the script I used available for download on patreon.
Resources:
www.nvidia.com/en-us/research/ai-demos/
blogs.nvidia.com/blog/2022/03/01/what-is-gaugan-ai…
Music Credit:
Track: Lost Sky - Vision [NCS Release]
Music provided by NoCopyrightSounds.
Watch: • Lost Sky - Vision | Dubstep | NCS - C...
Free Download / Stream: ncs.io/VisionID
Shoutout to Winkey for giving me the idea to make this Bob Ross themed.
All Comments (21)
-
Now that was different! I loved the part at 3:20!
-
He's using his actual powers to terraform the Earth now
-
That was super original. I love seeing experimental ideas like this!!!!!!
-
WOAH. That visualizer totally caught me off guard. Now that is amazing.
-
This is such an interesting medium. At first I thought it was a little weird but cool. But actually this is super cool.
-
watch out guys, they'll be coming for you as an instrument next
-
i've seen this AI but wow i've never seen it put like this :O this is so cool, especially those sound waves at 3:14
-
It's really cool how the picture moves with the music the mountain moving remind me of my cat crawling under a blanket
-
What an interesting way to synchronise music!! Each frame is truly beautiful, I see how you fell in love with it! Nice job!! :D
-
Props to the AI for moving mountains, changing the terra and creating/drying ocean for this sick beat
-
Awesome! Personally I love the part at 1:32. Every frame looks like a real landscape.
-
This is such a creative idea. You even managed to seamlessly work your own logo in as an integral part of the video. And the little behind-the-scenes at the end!
-
It's crazy that an AI can be used to generate such realistic looking landscapes
-
As someone who’s been following AI image generation for the past several years, it’s such a pleasure to be able to see it utilized to this degree. Very innovative and beautiful!
-
As someone who has played around with GauGAN a LOT, I commend you for your skills. It is truly incredible and hard to get consistent and looking how you want it. Well done.
-
The beauty of nature + the beauty of music *chef's kiss
-
What a clever idea! It's so cool that you could pause the video almost anywhere, and there would be a beautiful landscape. And the way the landscape flowed and changed was amazing!
-
Wow! That was trippy and cool - amazing what can be done with today's software...and even more amazing, YOUR brain for coming up with these ideas and then managing to execute them too! This channel is a deep well of happiness for me and I'm so grateful you keep making things to share with us! Thank you!
-
Visualizer part was definitely the best, but generally the whole video just woah'd me!
-
This is so satisfying. The video. The music. These images. This is what AI should be used for.
