RedSage: Cybersecurity Generalist LLM

Naufal Suryanto1, Muzammal Naseer1,†, Pengfei Li1, Syed Talal Wasim2, Jinhui Yi2, Juergen Gall2, Paolo Ceravolo3, Ernesto Damiani3
1 Khalifa University 2 Universität Bonn 3 University of Milan † Project Lead
OpenReview Hugging Face Python PyTorch

An open-source, locally deployable 8B model bridging the gap between general knowledge and domain-specific security operations via agentic data augmentation.

Download Models Our Model Showcase
11.8B
Pre-training Tokens
266K
SFT Dialogues
30K
Benchmark Qs
+5.6%
vs. Baselines

Abstract

Cybersecurity operations demand assistant LLMs that support diverse workflows without exposing sensitive data. Existing solutions either rely on proprietary APIs with privacy risks or on open models lacking domain adaptation. To bridge this gap, we introduce RedSage.

We curate 11.8B tokens of cybersecurity-focused continual pretraining data via large-scale web filtering and manual collection of high-quality resources. Building on this, we design an agentic augmentation pipeline that simulates expert workflows to generate 266K multi-turn cybersecurity samples for supervised fine-tuning.

To rigorously evaluate the models, we introduce RedSage-Bench, a benchmark with 30K multiple-choice and 240 open-ended Q&A items. At the 8B scale, RedSage achieves consistently better results, surpassing baseline models by up to +5.59 points on cybersecurity benchmarks and +5.05 points on Open LLM Leaderboard tasks.

Model Lineup

Model Name Type Best For Link
RedSage-8B-Base Base Domain adaptation, further fine-tuning HuggingFace →
RedSage-8B-Ins Instruct Multi-turn chat, step-by-step explanations HuggingFace →
RedSage-8B-DPO RECOMMENDED Chat (Aligned) Production-ready assistants, aligned behavior HuggingFace →
Benchmark Collection Data Collection

Quick Start

inference.py Transformers 
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "RISys-Lab/RedSage-Qwen3-8B-Ins"

tok = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

messages = [
  {"role": "system", "content": "You are RedSage."},
  {"role": "user", "content": "List three SSRF mitigations."}
]

text = tok.apply_chat_template(messages, tokenize=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=300)
print(tok.decode(out[0]))
bash vLLM Serving 
# 1. Install vLLM
uv pip install vllm --torch-backend=auto

# 2. Start the OpenAI-compatible server
vllm serve RISys-Lab/RedSage-Qwen3-8B-DPO \
  --port 8000 \
  --max-model-len 32768

# 3. Query via curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "RISys-Lab/RedSage-Qwen3-8B-DPO",
    "messages": [
      {"role": "user", "content": "Explain CTI."}
    ]
  }'

The RedSage Pipeline

From raw web data to a specialized assistant via multi-stage training.

1. Pre-Training

CyberFineWeb (11.8B Tokens) + RedSage Seed (28K Docs)

  • Filtered CommonCrawl
  • MITRE, OWASP, NIST
  • Offensive Security Writeups

2. Agentic SFT

266K Multi-turn Conversations

  • Planner Agent Strategies
  • Simulated Expert Workflows
  • General Instruction Mixing

3. Evaluation

RedSage-Bench + Public Benchmarks

  • 30K MCQs (Knowledge/Skills)
  • 240 Open-ended Q&A
  • LLM-as-a-Judge Rubric

Capabilities Showcase

Select an example below to view static model outputs (RedSage-8B-DPO).

redsage-8b-dpo — example_log — 80x24 READ ONLY
❯

Benchmark Performance

Base Models

RedSage-Bench
(MCQs)

RedSage-8B-Base 85.05%
Qwen3-8B-Base 84.24%
Foundation-Sec-8B 78.51%
Llama-3.1-8B 78.02%

Related Cybersecurity
Benchmarks (5-shots)

RedSage-8B-Base 84.56%
Qwen3-8B-Base 80.81%
Foundation-Sec-8B 76.90%
Llama-3.1-8B 75.44%

General Capabilities
(Open LLM Leaderboard)

Qwen3-8B-Base 70.86%
RedSage-8B-Base 69.23%
Llama-3.1-8B 61.15%
Foundation-Sec-8B 60.24%

Instruct & Chat Models

RedSage-Bench
(MCQs)

RedSage-8B-Ins 85.73%
Qwen3-8B (non-thinking) 81.85%
Foundation-Sec-8B-Ins 76.90%
Llama-3.1-8B-Ins 75.44%
Llama-Primus-Base 71.69%

*Evaluation on 30K domain-specific MCQs covering knowledge, skills, and tools.

Related Cybersecurity
Benchmarks (0-shot)

RedSage-8B-Ins 81.30%
Qwen3-8B (non-thinking) 75.71%
Foundation-Sec-8B-Ins 75.44%
Llama-Primus-Base 71.69%
Llama-3.1-8B-Ins 68.52%

*Mean accuracy across CTI-Bench (MCQ & RCM), CyberMetric (500), SecBench (En), SecEval, SECURE (CWET, KCV, MEAT), MMLU-CSec.

General Capabilities
(Open LLM Leaderboard)

RedSage-8B-DPO 74.33%
Foundation-Sec-8B-Ins 69.28%
Llama-3.1-8B-Ins 68.20%
Qwen3-8B (non-thinking) 65.92%
Llama-Primus-Base 64.82%

*Mean accuracy across MMLU, ARC-C, GSM8K, HellaSwag, TQA, WinoGrande, IFEval.

Citation

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Suryanto, Naufal and Naseer, Muzammal and Li, Pengfei and Wasim, Syed Talal and Yi, Jinhui and Gall, Juergen and Ceravolo, Paolo and Damiani, Ernesto},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}