Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks

Samuele Poppi^*,2,3, Zheng-Xin Yong^*,4, Yifei He⁵, Bobbie Chern¹, Han Zhao⁵, Aobo Yang^†,1, Jianfeng Chi^†,1

Findings of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), Albuquerque, NM, USA 2025 — Poster

Affiliations:^*,1Meta,²University of Pisa, ³University of Modena and Reggio Emilia, ⁴Brown University, ⁵University of Illinois Urbana-Champaign

📄 Project Page  |  🐙 GitHub  |  📝 arXiv  |  📚 bibtex

We show that fine-tuning in a single language compromises safety across all languages (cross-lingual generalization). We hypothesize that multilingual LLM safety relies on both language-specific and language-agnostic parameters. To study this, we introduce Safety Information Localization (SIL), which identifies the subset of model weights responsible for safety alignment. Our analysis finds that 20% of parameters encode most language-agnostic safety knowledge, with substantial cross-lingual overlap. Yet, freezing them fails to block attacks, revealing alternative learning pathways [3]. Finally, stitching these parameters into another safety-aligned model is enough to jailbreak it, confirming their effectiveness and transferability.

^* Work done during internship at Meta     ^† Equal advising

Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models

Samuele Poppi^*,1,2, Tobia Poppi^*,1,2, Federico Cocchi^*,1,2, Marcella Cornia¹, Lorenzo Baraldi¹, Rita Cucchiara¹

European Conference on Computer Vision, Milano, Italy 2024 - Poster

Affiliations:^*,1University of Modena and Reggio Emilia,²University of Pisa

Project Page  |  🤗 Hugging Face  |  GitHub  |  📝 arXiv  |  bibtex

We tackle toxicity in generative AI by aligning CLIP encoders with safety standards. We generate the the ViSU dataset, obtained by jailbreaking LLaMA2-7B-Chat, and it pairs quadruplets of safe and NSFW image-text examples. Leveraging the power of ViSU, Safe-CLIP applies knowledge editing techniques to remove unsafe associations in CLIP’s embeddings, thus ensuring outputs remain safe even with NSFW inputs, advancing responsible multimodal AI. These safe encoders can be attached and detached from any generation pipeline, due to their modularity.

^* Equal contribution

Unlearning Vision Transformers without Retaining Data via Low-Rank Decompositions

Samuele Poppi^1,2, Sara Sarto¹, Marcella Cornia¹, Lorenzo Baraldi¹, Rita Cucchiara¹

International Conference on Pattern Recognition, Kolkata, India 2024 - Poster

Affiliations:¹University of Modena and Reggio Emilia,²University of Pisa

📝 arXiv  |  bibtex

The GDPR and CCPA have spurred interest in removing sensitive information from pre-trained models without retraining. Standard unlearning approaches use a two-sided loss function with retaining and forgetting, relying on a forget set (D_f) and a retaining set (D_r), to preserve knowledge and ensure accuracy. These methods (1) face scalability challenges, (2) are resource-intensive, and (3) become impractical without access to retaining training data. This paper introduces a trainable low-rank decomposition to enable targeted information removal without a retaining dataset, significantly reducing computational and memory costs.