MASS: MoErging through Adaptive Subspace Selection

Table of Contents

BibTeX Citation
@misc{crisostomi2025massmoergingadaptivesubspace,
      title={MASS: MoErging through Adaptive Subspace Selection}, 
      author={Donato Crisostomi and Alessandro Zirilli and Antonio Andrea Gargiulo and Maria Sofia Bucarelli and Simone Scardapane and Fabrizio Silvestri and Iacopo Masi and Emanuele Rodolà},
      year={2025},
      eprint={2504.05342},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2504.05342}, 
}

MASS is our novel, training free model MoErging method, that allows to recover up to ~98% of accuracy of finetuned models with only a two times increase in storage and computational cost.

We tested its versatitility across different domains (Vision and NLP), architetures (ViT-{B,L}-{32,16,14}, Flan-t5), and number of tasks (8-14-20 dataset) proving its increadible scalability at fixed overhead cost.

At 🤗 this page you can find all the checkpoints you need, while in the card below there is our codebase that contains detailed instructions to reproduce our experiments.

crisostomi/mass

Codebase for “MASS: MoErging through Adaptive Subspace Selection.”

Jupyter Notebook

Author

Alessandro Zirilli