The Mistral-NeMo-Minitron 8B is a miniaturized version of the existing NeMo 12B AI model. Based on the “pruning and distillation” process, the miniaturized version can run locally on laptops and PCs.
Nvidia and Mistral AI launch a small AI model, Mistral-Nemo-Minetron 8Bwhich can be run natively on laptops and computers with less computing power. The model is a miniaturized version of the NeMo 12B and performs well in benchmarks with similar small models. The success of this smaller type is a combination of two techniques: “pruning and distillation”.
“Trimming and distillation”
The new Mistraal-NeMo-Minitron 8B mini AI model is a miniaturized version of the existing NeMo 12B. The model has been completely “pruned” from 12 billion to 8 billion parameters. To ensure this mini model remains efficient, the developers used a “pruning and distillation” technique.
“Pruning shrinks the neural network by removing model weights that contribute less to accuracy. During distillation, the team retrained this trimmed model on a smaller dataset to significantly improve accuracy, which was reduced by the pruning process,” Brian Catanzaro, vice president of deep learning research at Nvidia, said in a blog post. Blog post from Nvidia.
Read also
Authors Sue Nvidia Over Mishandling of NeMo Training
The optimized language model was trained on “part of the original dataset,” saving significant costs in terms of initial computation. The model is suitable for running locally on laptops and PCs due to the amount of computing power saved, and is said to run nine language-based AI benchmarks of the same size.
“Packaged as an NVIDIA NIM microservice, the model is optimized for low latency, which means faster responses to users, and high throughput, which equates to higher computational efficiency in production,” the blog post said. Furthermore, Nvidia is offering its own custom modeling service, AI Foundry, so that Minitron 8B can also run on smartphones.
