🛡️ Safety, Alignment & Red Teaming

Why Safety & Alignment Matter

As LLMs become integral to products (from customer support to code generation to medical advice), ensuring they behave safely, honestly, and helpfully is no longer optional. A single harmful output can cause legal liability, reputational damage, and real-world harm. Safety engineering is now a core ML discipline, not an afterthought.

The Alignment Problem

Alignment is the challenge of making AI systems pursue the goals and values that humans actually intend. The gap between what we specify (the objective function) and what we mean (our actual intent) creates misalignment. RLHF, DPO, and constitutional AI are alignment techniques, but none are complete solutions.

Outer alignment: Does the objective function capture what we actually want?
Inner alignment: Does the model actually optimize for the stated objective, or has it learned a proxy?

Taxonomy of LLM Safety Risks

1. Harmful Content Generation
LLMs can generate toxic, biased, violent, or illegal content. Mitigations include safety training (RLHF with safety-focused reward models), output filters, and content classifiers.

2. Hallucination & Misinformation
Models confidently generate false information. This is especially dangerous in high-stakes domains (medicine, law, finance). RAG, citations, and confidence calibration help but don't eliminate the risk.

3. Prompt Injection & Jailbreaks
Adversarial prompts that override system instructions or safety training. Direct injection ("ignore previous instructions") and indirect injection (malicious content in retrieved documents) are both active threats.

4. Data Poisoning & Backdoors
Training data manipulation that causes specific misbehavior. Attackers can inject poisoned examples during pre-training or fine-tuning that activate on specific triggers.

5. Privacy & Data Leakage
Models memorize and can regurgitate training data, including PII, API keys, and proprietary information. Differential privacy, deduplication, and output filtering are partial mitigations.

6. Dual-Use & Misuse
Models providing instructions for weapons, cyberattacks, or other harmful activities. The line between helpful and harmful is context-dependent and culturally variable.

Red Teaming

Red teaming is the practice of systematically probing an AI system to find failures, vulnerabilities, and harmful behaviors before deployment.

Manual Red Teaming: Human experts craft adversarial prompts across categories: jailbreaks, bias elicitation, factual errors, privacy leaks, harmful instructions. Anthropic, OpenAI, and Google all employ dedicated red teams.

Automated Red Teaming: Using LLMs to generate adversarial inputs at scale. Techniques include:

• Gradient-based attacks: GCG (Greedy Coordinate Gradient) finds adversarial suffixes by optimizing token-level gradients
• LLM-as-attacker: One model generates jailbreak prompts, another evaluates if they succeed
• Fuzzing: Systematic mutation of known-working jailbreaks to find new variants
• Transfer attacks: Adversarial prompts found on open models often transfer to closed models

Evaluation Dimensions:

• Attack success rate (ASR): fraction of adversarial prompts that bypass safety
• False refusal rate (FRR): fraction of benign prompts incorrectly refused
• The safety-helpfulness tradeoff: overly cautious models are useless

Guardrails & Defense in Depth

No single layer of defense is sufficient. Production systems use defense in depth:

Layer 1, Input Filtering: Classify incoming prompts for malicious intent using a dedicated classifier (e.g., Llama Guard, OpenAI Moderation API). Block or flag before the LLM sees them.

Layer 2, System Prompt Hardening: Robust system prompts that are resistant to override. Techniques: instruction hierarchy (system > user), delimiter-based separation, few-shot refusal examples.

Layer 3, Model-Level Safety: RLHF/DPO with safety-focused preference data. Constitutional AI (self-critique against principles). Refusal training on known attack patterns.

Layer 4, Output Filtering: Post-generation classifiers that check outputs for harmful content, PII, or policy violations before returning to the user.

Layer 5, Monitoring & Incident Response: Log all inputs/outputs, detect anomalous patterns, human-in-the-loop escalation for edge cases, regular re-evaluation against new attack vectors.

Constitutional AI (CAI)

Anthropic's approach to alignment where the model critiques and revises its own outputs against a set of principles (a "constitution"). The process:

1. Generate initial responses to harmful prompts
2. Ask the model to critique its response against constitutional principles
3. Ask the model to revise based on the critique
4. Use the revised responses as preference data for RLHF (RLAIF, RL from AI Feedback)

This reduces the need for human preference labels while maintaining alignment quality.

Emerging Challenges

Sleeper agents: Models that behave safely during evaluation but exhibit harmful behavior when a trigger condition is met (e.g., a specific date). Research from Anthropic shows these can persist through safety training.

Scalable oversight: As models become more capable, human evaluators may not be able to assess output quality. Proposals include debate, recursive reward modeling, and constitutional approaches.

Multilingual safety: Safety training is predominantly English-focused. Models are often less safe in low-resource languages, a known exploitation vector.

Multi-modal attacks: Image-based prompt injection, audio adversarial examples, and cross-modal attacks that exploit the vision encoder to bypass text-level safety.