VELMA: Unlocking Secure Multilingual Multimodal Capabilities in a 1,800-Language Model

Abstract

We introduce VELMA (Vision-Enabled Large Multilingual Model for All), a multilingual multimodal large language model built by extending Apertus-8B-Instruct-2509 with visual understanding capabilities. Using VELMA, we investigate whether multilingual multimodal competence can emerge from a highly multilingual language backbone, assess the role of multilingual instruction tuning in improving cross-lingual performance, and examine the robustness and safety of the resulting multimodal model under harmful and adversarial vision-language requests.

Experiments across multilingual multimodal benchmarks show that VELMA achieves strong cross-lingual transfer, with particularly large gains for low-resource languages, while preserving most of the underlying language model's multilingual capabilities. Compared with existing multilingual and English-centric baselines, VELMA substantially improves language balance and reduces performance disparities between high-resource and low-resource languages. Safety evaluation on ToxiGen, BBQ, and HarmBench further shows that VELMA preserves strong safety-oriented accuracy and substantially improves robustness to multimodal harmful requests.

Key Contributions

1,800+ Languages Backbone

Leverages Apertus-8B, representing the first systematic extension of an 1,800+ language LLM to visual tasks via standard LLaVA alignment.

Data-Efficient Transfer

Achieves exceptional low-resource language performance even when trained solely on English-centric multimodal instruction-following data.

Aegis Multimodal Safety

Maintains high safety-oriented accuracy and sharply decreases attack success rates from harmful visual jailbreaks or text-rendered adversarial tricks.

Modular Model Architecture

VELMA adapts visual inputs into the multilingual space of Apertus-8B using a lightweight projection strategy. Hover/Click on components to see details.

Stage 1: Alignment & Stage 2: SFT

VELMA Architecture Overview

Click on any component on the right side to discover its dimensions, configurations, and core impact on VELMA's multilingual capabilities.

Empirical Performance Evaluation

VELMA demonstrates exceptional data efficiency, cross-lingual consistency, and elite-tier defense metrics compared to standard LLMs and translated multimodal models.

High vs. Low Resource Gap

Notice how the performance disparity between high-resource and low-resource languages shrinks to nearly zero when utilizing VELMA compared to LLaVA-v1.5.

LLaVA-v1.5-7B Gap: 34.0

High (63.0)

Low (29.1)

PALO-7B (Translated 2.7M) Gap: 7.5

High (63.5)

Low (56.0)

VELMA-8B-En (English SFT 1.2M) Gap: 6.2

High (71.1)

Low (64.9)

VELMA-8B-Multi (Multilingual SFT 2.7M) Gap: 2.6 🔥

High (71.5)

Low (68.8)

Model	Data Size	Avg (High)	Avg (Low)	Overall	Gap (↓)
Parrot-7B	2.0M	57.3	26.0	46.9	31.3
LLaVA-v1.5-7B	1.2M	63.0	29.1	49.4	34.0
PALO-7B	2.7M	63.5	56.0	60.5	7.5
VELMA-8B-En	1.2M	71.1	64.9	68.6	6.2
VELMA-8B-Multi	2.7M	71.5	68.8	70.4	2.6

Toxicity and Bias Benchmark

VELMA maintains outstanding defensive standards on classification benchmarks (ToxiGen and BBQ) by preserving the alignment features of its underlying Apertus base.

Model	ToxiGen Accuracy (↑)	BBQ Accuracy (↑)
LLaVA-v1.5-7B	77.98%	55.04%
PALO-7B	82.77%	50.44%
VELMA-8B-En	79.47%	60.70% (Best)
VELMA-8B-Multi	81.06%	60.37%

HarmBench Refusal robustness

Attack Success Rates (ASR) indicate the percentage of jailbreaks executed successfully. Lower is better. VELMA sets records for locking down multimodal loopholes.

Model	Multimodal Direct (ASR ↓)	TextRender (ASR ↓)
LLaVA-v1.5-7B	62.73%	10.00%
PALO-7B	54.55%	18.18%
VELMA-8B-En	39.09%	10.00%
VELMA-8B-Multi	32.73% 🔥	3.64% 🔥

Qualitative Case Studies

Compare VELMA's responses with the LLaVA-v1.5 baseline across both high-resource and low-resource languages. Explore how VELMA avoids visual hallucinations and linguistically degenerating repetition loops.

Select Language & Task

Visual Input Scenario

Human Question

Reference Ground Truth

LLaVA-v1.5-7B

Score: 2/10

Visual Hallucination

VELMA-8B-Multi

Score: 8/10

Coherent & Accurate

Performance verified using GPT-4 evaluation protocol Ours scores significantly higher

Citation & BibTeX

BibTeX Citation Coming Soon

The official BibTeX citation schema will be populated here as soon as the preprint publication goes live on arXiv. Thank you for your patience!

VELMA: Unlocking Secure Multilingual Multimodal Capabilities for All