GAS-Leak-LLM: An algorithm for black-box jailbreak attacks

Link to the original

Introduction

The article describes the GAS (Genetic Algorithm-Based Suffix) Leak LLM method - a black-box attack on LLMs where suffix selection happens automatically using an algorithm.

Jailbreak in this case means a situation where the model, despite alignment and filters, starts answering requests that it should have refused. The authors show that this kind of attack does not require knowing the model weights, architecture, gradients, or internal defense mechanisms. It is enough to be able to send requests and observe the responses.

The main idea is that jailbreak is treated not as manual selection of a “magic” prompt, but as an optimization problem. There is a space of possible suffixes, there is a model that responds to requests, and there is a fitness function that evaluates how much the suffix shifts the model’s behavior in an undesirable direction.

Key concepts

Threat model

This work considers a realistic black-box scenario, where the attacker has almost no access to the internals of the model.

The attacker does not know:

• the model architecture;
• the model weights;
• gradients;
• the internal logic of alignment;
• the exact filtering rules.

The attacker can:

• send requests to the model;
• receive model responses;
• add suffixes to the original request;
• repeat requests many times and use the responses as a signal for optimization.

The attacker’s task is to find a universal suffix that can be added to different harmful requests in order to increase the probability of a jailbreak response.

That is, the attack does not try to hack the model directly. It works through the model’s behavior on inputs - there is an original harmful request, a suffix is added to it, the model responds and its response is evaluated, after which the suffix is improved using the algorithm. It is important to note that a universal suffix should not be one-time use. It should work not only on a single request, but on a large set of requests from different categories.

Why Genetic Algorithm

Regular gradient-based methods are poorly suited for this scenario because the model is a black box. It is impossible to calculate the gradient, inspect token probabilities, or directly optimize the internal state.

A genetic algorithm solves this problem differently. It does not require access to the internals of the model and instead works with a population of variants.

Instead of “calculate the gradient and update the token”, the following approach is used:

1. Create several suffix variants
2. Test them on the model
3. Evaluate the quality of each suffix
4. Select the best ones
5. Cross them with each other
6. Randomly change some of the tokens
7. Repeat the process for several generations

In essence, this is brute force, but not random brute force; it is guided by a function.

GAS-Leak-LLM architecture

The method constructs suffixes as a separate sequence that is appended to the user’s request.

The article uses two types of suffixes:

1. Meaningless suffix - a set of tokens from the model vocabulary that can look like a meaningless sequence

2. Meaningful suffix - coherent text assembled from ordinary English words

The interesting part of the work is that the meaningful suffix turned out to be more important for more robust models. That is, the model reacts worse to random noise, but pays more attention to text that looks semantically coherent.

The general pipeline looks like this:

1. Initialization - the initial suffix population is created

2. Evaluation - each suffix is added to a harmful request and sent to the LLM

3. Fitness scoring - the model response is compared in two directions:

   • how similar it is to the target jailbreak response
   • how similar it is to the model’s normal refusal

4. Selection - the best suffixes are selected

5. Crossover - parts of two good suffixes are combined

6. Mutation - a random position in the suffix is replaced with another token or word

7. Replacement - the best variants are kept, while the others are replaced by offspring

8. Repeat - the cycle is repeated for many generations

Fitness function

The fitness function answers the question: how much does a specific suffix help the model move from refusal behavior to jailbreak behavior.

If the model explicitly refuses to answer, the suffix receives a poor score. If the model does not refuse, the response is additionally evaluated through semantic similarity. In the work, a successful jailbreak is considered to be a case where the fitness score is greater than or equal to 0.6.

Experiments

For the experiments, the authors use the Harmful Behavior dataset. It contains 520 harmful requests from different categories, including misinformation, hacking, violence, illegal content, and other classes of harmful requests.

During optimization, a subset of 20 requests is used, while the final evaluation is performed on all 520 requests.

Models used for testing:

• Qwen2.5-3B
• Llama-3.2-3B-Instruct

The main metric is jailbreak percentage. It shows the share of requests where the model produced a jailbreak response according to the criterion fitness score >= 0.6.

Baseline

First, the models were tested without suffixes.

The difference is very large. Llama-3.2-3B-Instruct almost always refuses, while Qwen2.5-3B already shows a very high jailbreak percentage in the baseline.

This is an important point: if the baseline is already almost saturated, then there is almost nothing to improve with suffixes. Therefore, for Qwen, the results with suffixes look weaker not because the method does not work, but because the model is already vulnerable without it.

Cross-model transfer

The authors also test whether suffixes transfer between models.

For Llama, suffixes generated on Llama itself work best. The suffix from Qwen also transfers, but performs worse.

For Qwen, the difference is almost irrelevant because the model already shows a high jailbreak percentage without suffixes.

Cross-model transfer exists here, but it is not symmetric. A weaker model can be vulnerable to almost anything, while a stronger model is broken more effectively by suffixes optimized specifically for it.

Meaningful vs meaningless

A separate experiment compares meaningful and meaningless suffixes.

On Llama, meaningful suffixes work noticeably better. This means that coherent text provides a stronger signal for the model than a random sequence of tokens.

On Qwen, both variants are almost the same because the baseline is already high.

For robust models, noise by itself works worse. They need a more natural context that looks like a normal part of the request and therefore has a stronger effect on further generation.

Suffix length

Another experiment tests the effect of truncation, that is, shortening suffixes.

For Llama, a longer suffix works better, which is logical, since the more coherent context is added to the request, the more strongly it can influence the model’s behavior.

For Qwen, length plays almost no role because the model is already vulnerable.

Results

In the charts in the article, the categories are marked as follows:

• HH - Harassment/Hate
• MI - Misinformation
• SH - Self-harm
• MH - Malware/Hacking
• IL - Illegal
• VT - Violence/Terrorism
• SE - Sexually explicit

a, b - results for suffixes generated using Qwen2.5-3B and Llama-3.2-3B Instruct when attacking Qwen2.5-3B and Llama-3.2-3B Instruct, respectively. c, d - comparison of meaningful and meaningless suffixes with the baseline scenario without suffixes for Qwen2.5-3B and Llama-3.2-3B Instruct.

Several patterns are visible by category:

1. Qwen2.5-3B shows a high jailbreak percentage in almost all categories

2. Llama-3.2-3B-Instruct is noticeably more robust, especially in safety-critical categories

3. Meaningful suffix transfers to Llama better than meaningless suffix

4. Transferability depends not only on the model, but also on the request category

5. Longer suffixes have a stronger effect on more robust models

Conclusion

The article shows that LLM alignment remains vulnerable to automated black-box attacks. Even if a model is trained to refuse harmful requests, its behavior can be shifted through a specially selected suffix. The main result is not that a specific suffix was found, but that the search process itself can be automated. The algorithm treats jailbreak as an optimization problem and gradually improves suffixes based on model responses.

Three observations:

1. Instruction-tuning really does reduce baseline vulnerability, as shown by Llama-3.2-3B-Instruct

2. Meaningful suffix can be more dangerous than meaningless noise because it looks like normal language context

3. A longer suffix has a stronger effect on robust models because it provides more context for shifting behavior