Generative Models

From diffusion models to GANs, explore the AI systems that create images, video, audio, 3D assets, and code from simple text prompts.

What Are Generative Models?

Generative models are AI systems designed to create new content that did not previously exist. Unlike discriminative models (which classify or analyze existing data), generative models learn the underlying distribution of their training data and produce novel outputs - images, videos, music, 3D objects, and code - that match the patterns they have learned.

The generative AI revolution began with GANs in 2014, accelerated with diffusion models in 2020-2022, and has since expanded to every creative modality. Today, generative models are among the most widely used and commercially impactful AI systems.

Image Generation

Diffusion Models

Diffusion models are currently the dominant approach for high-quality image generation. They work through a two-phase process:

1. Forward diffusion: Gradually add random noise to an image until it becomes pure noise
2. Reverse diffusion: Train a neural network to reverse this process - predicting and removing noise step by step until a clean image emerges

At generation time, you start with pure random noise and iteratively denoise it, guided by a text prompt through a mechanism called classifier-free guidance.

GANs (Generative Adversarial Networks)

GANs were the first deep learning approach to produce photorealistic images. A GAN consists of two networks trained in opposition:

• Generator: Creates fake images from random noise
• Discriminator: Tries to distinguish real images from generated ones

This adversarial training pushes both networks to improve. While GANs have been largely superseded by diffusion models for general image generation, they remain relevant for specific applications like real-time face generation (StyleGAN) and super-resolution.

Video Generation

AI video generation has progressed rapidly, moving from short, low-quality clips to cinematic-quality videos lasting minutes.

Audio and Music Generation

Generative models now create music, sound effects, speech, and audio from text descriptions.

3D Generation

3D content generation is one of the fastest-evolving areas, with models that create meshes, textures, and scenes from text or images.

• Meshy: Text-to-3D and image-to-3D with PBR textures, suitable for game development
• Point-E (OpenAI): Generates 3D point clouds from text, fast but lower quality
• NeRF-based approaches: Neural Radiance Fields create photorealistic 3D scenes from 2D photos
• Gaussian Splatting: Newer technique for real-time 3D scene reconstruction and rendering
• Genie 2 (Google DeepMind): Generates playable 3D environments from single images

Code Generation Models

Code generation is one of the most commercially successful applications of generative AI. These models write, debug, explain, and refactor code across dozens of programming languages.

Prompt Engineering for Generative Models

Getting great results from generative models requires effective prompting. Each modality has its own best practices:

Image Prompts

• Be specific about style: "oil painting," "photorealistic," "3D render," "watercolor illustration"
• Describe composition: "close-up portrait," "aerial view," "rule of thirds"
• Include lighting: "golden hour lighting," "dramatic rim light," "soft diffused light"
• Specify quality: "highly detailed," "8K resolution," "professional photography"
• Use negative prompts: Tell the model what to avoid (blurry, low quality, extra fingers)

Video Prompts

• Describe motion: "camera slowly pans left," "person walks toward camera"
• Set the scene: Include environment, time of day, weather, atmosphere
• Keep it simple: Video models handle single scenes better than complex multi-shot narratives

Code Example: Generating Images with Stable Diffusion

import torch
from diffusers import StableDiffusion3Pipeline

# Load the Stable Diffusion 3 pipeline
pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

# Generate an image from a text prompt
prompt = (
    "A futuristic city at sunset, flying cars between glass skyscrapers, "
    "neon signs reflecting on wet streets, cyberpunk style, "
    "volumetric lighting, highly detailed, 8K"
)

negative_prompt = (
    "blurry, low quality, distorted, watermark, text overlay"
)

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=28,      # More steps = higher quality, slower
    guidance_scale=7.0,          # How closely to follow the prompt
    width=1024,
    height=1024,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]

image.save("futuristic_city.png")
print("Image saved successfully!")

# --- Batch generation with different seeds ---
images = []
for seed in range(4):
    result = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        num_inference_steps=28,
        guidance_scale=7.0,
        width=1024,
        height=1024,
        generator=torch.Generator("cuda").manual_seed(seed),
    )
    images.append(result.images[0])

# Save all variations
for i, img in enumerate(images):
    img.save(f"variation_{i}.png")

Ethical Considerations

Generative AI raises important ethical questions that practitioners must consider:

Deepfakes and Misinformation

Realistic image and video generation can be used to create convincing fake content of real people. This poses risks for fraud, political manipulation, and harassment. Many platforms now require AI-generated content to be labeled, and detection tools are actively being developed.

Copyright and Training Data

Generative models are trained on large datasets that may include copyrighted material. The legal landscape is evolving, with ongoing lawsuits from artists and publishers. Some models (like Adobe Firefly) are trained exclusively on licensed content to address this concern.

Consent and Likeness

Generating images or voices of real people without their consent raises legal and ethical issues around right of likeness. Many platforms prohibit generating content depicting identifiable individuals without permission.

Use Cases Across Industries

• Marketing and advertising: Generate product images, ad copy variations, social media content at scale
• Entertainment: Concept art, storyboarding, music scoring, visual effects
• Prototyping and design: Rapid UI mockups, architectural visualizations, fashion design concepts
• Game development: Texture generation, NPC dialogue, level design, sound effects
• Architecture: Generate photorealistic renders from floor plans and descriptions
• Education: Create custom illustrations, diagrams, and interactive content
• Software development: Code generation, test creation, documentation, debugging assistance

Comparison by Modality