LMMs Engine

Getting Started

Getting Started

User Guide

Datasets and Packing: Naive vs Streaming
Data Preparation
- Recommended Data Format
  - Cloud Data Access
- HF Format
Performance Optimization
- Flash Attention Installation
Merging FSDP Checkpoints
FSDP2 Mixed Precision: Reduce Dtype Considerations
Asynchronous Checkpoint Evaluation During Training

Tutorial

Tutorial
- Adding a Custom Parallel Strategy (Expert Parallel)

Developer Guide

Creating New Datasets
Creating Custom Data Processors
Adding a new model (and performance monkey patches)
Adding a new trainer

Reference

Main API of this framework
- Base Class
- Mixin Class
- Processor Class
- Collator
- Dataset
- Proto
Design Principle
- Factory Pattern
- Builder
Video Configuration Guide
Model FLOPs Utilization (MFU) Reference

Models

BAGEL Model Training Guide
Qwen-VL Model Training Guide
dLLM (Diffusion Language Model) Training
FLA Models (DGN) Training
RAE-SigLip Training
SiT (Scalable Interpolant Transformer) Training
WanVideo Training
Qwen2.5 LLM Training
Qwen2.5-Omni Training
Qwen3-VL MoE Training
Qwen3-MoE Training
Qwen3-Omni MoE Training

Troubleshooting

Troubleshooting
- Transformers 5.0 Migration Guide

LMMs Engine

Performance Optimization
View page source

Performance Optimization

Flash Attention Installation

cd ~

git clone https://github.com/Dao-AILab/flash-attention.git

cd hopper
python setup.py install

Previous Next

© Copyright 2024, LMMs Engine Contributors.

Built with Sphinx using a theme provided by Read the Docs.