CAT 
Mixture of Experts (MoE) architectures for large language models (LLMs) have recently gained popularity due to their ability to increase model capacity and computational efficiency compared to fully dense models. By utilizing sparse expert subnetworks that process different subsets of tokens, MoE models can effectively increase the number of parameters while requiring less computation per […]
Originally appeared here:
Accelerate Mixtral 8x7B pre-training with expert parallelism on Amazon SageMaker
