Meta has announced Movie Gen, a new AI model designed to create high-quality 1080p videos with synchronized audio. The system enables instruction-based video editing and allows for personalized content generation using user-supplied images.
The core of Movie Gen is a transformer model with 30 billion parameters. Trained on large datasets of images, videos, and audio, the model can produce 16-second videos at 16 frames per second. It also integrates latent space management and flow-matching techniques, improving the generation of realistic video motion.
In terms of data, Movie Gen was trained using over 100 million video-text pairs and 1 billion image-text pairs, which enhances its capacity to generalize across various media tasks.
In benchmarks, Movie Gen outperforms previous state-of-the-art models and commercial systems like Runway Gen3 and OpenAI Sora in text-to-video synthesis, video personalization, and video editing tasks. While SDEdit maintained video structure but struggled with fine details, models like EVE and InsV2V performed well in generating realistic motion, though EVE relied on video captions for certain metrics. Runway Gen3 V2V had issues with preserving fine details. On the TGVE+ benchmark, Movie Gen Edit was favored 74% of the time over EVE and achieved state-of-the-art results in some metrics. It also outperformed Runway Gen3 V2V and SDEdit, particularly in maintaining video structure and detail across various tasks.
Users have reported high satisfaction with the generated content, particularly the quality and realism of the videos.
AI Evangelist Alex Volkov posted on X:
Meta steps into video generation to fill a big SORA void!
While director and photographer Ravi Vora shared:
This could change video forever.
Challenges and future directions for Movie Gen include improving complex scene understanding, implementing safeguards against misuse, and reducing resource requirements for broader accessibility.
The system's applications range from social media content creation to film production and personalized marketing campaigns. Meta plans to continue refining the model with future updates that may include expanded editing and personalization features.