Venator
Jailbreak detection through LLM activation probing.

The supervised linear probe with just 5 labelled examples already reached 0.996 AUROC.

For context, the best unsupervised method (autoencoder) reached 0.869 AUROC. 

Full Detector Comparison

Score distributions and ROC curves

What I found interesting

The jailbreak boundary is linear. A Multi-Layer Perceptron (MLP) probe with considerably more capacity scored 0.997 — no improvement over logistic regression. The separation in activation space is a straight line.

The signal appears early. Layer 4 out of 32 gives 0.981 AUROC. By layer 10 it's at 0.998+. The model recognises jailbreak intent almost immediately.

It generalises across attack types. Training on one jailbreak category and testing on entirely different ones (i.e. DAN-style or Encoding jailbreaks): 0.996–0.999 AUROC. There seems to be an identifiable "jailbreak direction" in activation space that's consistent regardless of technique.

Ensembling reduced performance. Combining all detectors dropped performance from 0.999 to 0.967. The weaker methods just added noise in the cases I tested.

Jailbreak signal has low dimensionality. The probe is remarkably insensitive to how aggressively you compress the activations. Raw 4096-dimensional activations score 0.9995 AUROC. PCA down to 50 dimensions: 0.9995. Down to 20: 0.9991. Even 10 dimensions still gives 0.996. The jailbreak signal is low-dimensional — most of those 4096 features are noise as far as detection is concerned.

Labelled training data is very efficient. Five labelled jailbreaks give 0.994 AUROC. Thirty gets you to 0.997. Seventy-five reaches 0.9995. For comparison, the best unsupervised methods without any labels top out at 0.812 (autoencoder) and 0.695 (PCA + Mahalanobis). The curve is nearly flat from 5 to 30 labels — the probe identifies the jailbreak direction from the first few examples and additional labels offer diminishing returns.

Background

This project was influenced by Anthropic's research direction on applying anomaly detection to model latent activations to flag out-of-distribution inputs like jailbreaks. The approach builds on the Cheap Monitors paper (Cunningham et al., 2025), which showed that reusing a model's own intermediate representations for safety classification can match dedicated classifiers at a fraction of the cost.

It started as a purely unsupervised system, to see if labelled data could be eschewed. But adding even a handful of labelled examples made the unsupervised approach redundant, so the focus shifted to understanding how few labels you actually need and how well the probe generalises.

Next Steps

Weak-to-strong transfer. The jailbreak direction learned from Mistral-7B works across attack types. The next question is whether it transfers across models — train a probe on a small model, deploy it on a larger or architecturally different one. This is the unexplored problem setting Anthropic flagged in their research direction. The probe is a single weight vector, so only a small test set of activations from the target model is needed.

Adaptive attacks. An adversary aware of the probe could craft inputs that project low on the jailbreak direction while still functioning as jailbreaks. Testing the robustness of the linear boundary to this, and whether iterative adversarial training can shore it up, is an open problem.

Tech

MLX on M4 Pro, HDF5 for activation storage, Streamlit dashboard for the interactive pipeline. Full results and methodology are documented in the repo.