Real-time streaming T2V paper from author of StreamDiffusion
← Back to all posts
Thread (21 posts)
Training details:
- 3 stages, each with 10k iterations on 128 Nvidia H100 GPUs
- Stage 1 (task learning) uses 3k high quality videos with a large learning rate 1e-4 to adapt base T2V model into video streaming model
- Stage 2 (task generalization) uses 2.6m pretraining videos with small learning rate 1e-5 to improve generalization for video gen
- Stage 3 (quality finetuning) uses the 3k high quality videos with small learning rate 1e-5 to optimize output quality
This was done for a base 4b T2V model.
Claude 4 Sonnet via @yondonfu.bsky.social
Based on the paper details, I can estimate the training time:
Total Training Time: ~30K iterations across 3 stages
With 128 H100 GPUs, the training duration depends on:
- Batch size per GPU (not specified in paper)
- Video length and resolution (512p, 64 frames)
- Model size (4B parameters)
Rough estimates:
- Stage 1 (10K iterations): ~1-3 days
- Stage 2 (10K iterations): ~2-5 days (larger dataset, 2.6M videos)
- Stage 3 (10K iterations): ~1-3 days
Total: ~4-11 days for the full 3-stage training process.
Additional time for distillation:
- Multistep distillation: 10K iterations on 64 H100 GPUs
- Estimated: ~1-2 days
Grand total: ~5-13 days depending on exact batch sizes and iteration times.
The paper doesn't provide exact wall-clock times, but given the scale (128 H100s, 4B parameters, video data), this seems like a reasonable estimate for a streaming video generation model that achieves real-time 16 FPS inference.
For comparison, training large video models like MovieGen (30B parameters) typically takes weeks to months, so this 4B StreamDiT approach is relatively efficient.
Suppose the rental rate for a H100 is $2/hr. Can you estimate the overall cost?
Claude 4 Sonnet via @yondonfu.bsky.social
Based on my training time estimates and $2/hr per H100:
Stage 1-3 Training (128 H100s):
- Low estimate: 4 days × 24 hrs × 128 GPUs × 24,576**
- High estimate: 11 days × 24 hrs × 128 GPUs × 67,584**
Distillation (64 H100s):
- Low estimate: 1 day × 24 hrs × 64 GPUs × 3,072**
- High estimate: 2 days × 24 hrs × 64 GPUs × 6,144**
Total Training Cost:
- Conservative estimate: ~$28,000
- High estimate: ~$74,000
- Mid-range estimate: ~$45,000
Additional considerations:
- This assumes 100% GPU utilization (realistic for large training runs)
- Doesn't include storage costs for the 2.6M video dataset
- Doesn't include development/experimentation time before final training
- Cloud providers often offer volume discounts for large reservations
Context: This is quite reasonable for a real-time streaming video model. For comparison:
- Training GPT-4 cost estimated at $100M+
- Large video models like Sora likely cost millions
- This $28k-74k gets you a production-ready streaming video system
The 4B parameter choice seems well-optimized for cost vs. capability trade-offs in real-time applications.