Boost inference performance for Mixtral and Llama 2 models with new Amazon SageMaker containers
CAT
1 min read
In January 2024, Amazon SageMaker launched a new version (0.26.0) of Large Model Inference (LMI) Deep Learning Containers (DLCs). This version offers support for new models (including Mixture of Experts), performance and usability improvements across inference backends, as well as new generation details for increased control and prediction explainability (such as reason for generation completion […]
Originally appeared here:
Boost inference performance for Mixtral and Llama 2 models with new Amazon SageMaker containers
