The big improvements over the first generation version of this from last year (
https://
app.eden.art/creations/65b1
5634c7e9dd341fd49258
…) are: 1) More distinct style. I got a bit tired of that AnimateDiff aesthetic on top of every film. The IPAdapter trick helps here. 2) Multi-stage writing pipeline is great
@genekogan
-
Improved AI Video Generation with IPAdapter and Multi-Stage Pipeline
By
–
-
AI-Generated Film Production: Storyboard to Music Composition
By
–
Last but not least, the music! A prompt is generated at the storyboard phase, which then goes to Musicgen to generate a backing soundtrack for the entire film.
— Gene Kogan (@genekogan) 8 septembre 2024
Videos, vocals, and music are all composited together, and voila. pic.twitter.com/EbngkyR2BzLast but not least, the music! A prompt is generated at the storyboard phase, which then goes to Musicgen to generate a backing soundtrack for the entire film. Videos, vocals, and music are all composited together, and voila.
-
img2vid Pipeline Creates 3D Parallax Animation Effects
By
–
Then each frame is animated using an img2vid pipeline cooked up by @xsteenbrugge that does some parallax tricks on the foreground and background to give it a 3d motion feel. pic.twitter.com/9sA1Rf8YAE
— Gene Kogan (@genekogan) 8 septembre 2024Then each frame is animated using an img2vid pipeline cooked up by @xsteenbrugge that does some parallax tricks on the foreground and background to give it a 3d motion feel.
-
IPAdapter Image Generation Shows Unique Style Bleed Effects
By
–
Another example, this style image: Generated these frames. There's some subtle content bleed from the IPAdapter, like the random tree branches everywhere, but I like it — feels more unique / idiosyncratic.
-
Using Style Images and IPAdapter for Consistent AI Visual Generation
By
–
Now the visuals. First, I generate a "style image" using the visual aesthetic from step 1 as a prompt. This image is used as an IPAdapter for generating the actual frames. I found this kept a consistent aesthetic across frames, and distinguished it from other stories. For
-
Converting Screenplays to Storyboards with AI Voice Generation
By
–
Then convert the screenplay into a storyboard. The storyboard is a sequence of images that retell the screenplay, along with character dialogue. All the characters are assigned matching voices from ElevenLabs, using their description as a prompt. Vocals are generated, giving us
-
AI Pipeline Generates Short Films from Text Prompts
By
–
I've been experimentinmg with pipelines for generating short films / trailers from prompts.
— Gene Kogan (@genekogan) 8 septembre 2024
The process goes roughly like this:
1) Synopsis: draft a basic story kernel and visual aesthetic.
2) Storyboard: Expand synopsis through multiple stages (script/cast -> screenplay ->… pic.twitter.com/OAYeGCgSCLI've been experimentinmg with pipelines for generating short films / trailers from prompts. The process goes roughly like this: 1) Synopsis: draft a basic story kernel and visual aesthetic.
2) Storyboard: Expand synopsis through multiple stages (script/cast -> screenplay -> -
GameNGen: Google’s Real-Time AI Game Engine Innovation
By
–
For context, GameNGen is a game engine developed by Google which runs a real-time playable version of Doom. Not quite text-to-game yet, but a step. https://
gamengen.github.io The prior work I referenced is World Models by Ha & Schmidhuber -
GameNGen: The Sora Moment for World Simulation
By
–
The more I read about GameNGen, the more I think it's groundbreaking. I mean @hardmaru @SchmidhuberAI were definitely the OG of this idea, but this is the Sora moment for world simulation. I think of simulation as the terminal goal of the entire field of generative AI. Every
-
Chat History Storage: Alternating vs Single Message Architecture
By
–
Slightly off-topic, but I'm curious if you've compared storing the chat history as a sequence of alternating User and Assistant Messages (the natural/expected form) vs just putting everything into a single User Message so the prompt is always just 1 System and 1 User Message.