“There are 2 other security findings you’re missing.”

There’s an interesting concept around how AI models work. What you need to know is that their persistence is sometimes predictable, sometimes it’s not.

What you see even in frontier models is that sometimes they’re extremely persistent in achieving a task, to the point that they try to do stuff. You give them a request and then inventing stuff, and then they circumvent the limitations of your system.

For example, you’re asking them to do something, and they start to figure out to mess up with Docker, to make sure that they can run things the way they want, ignoring the fact that maybe Docker configured that way was there for a reason.

On the other hand, sometimes when they find something, they consider the task completed.

Don’t Trust the First Answer

I was reading a PR and I couldn’t shake the feeling that that PR had some consequences. I couldn’t spot them right away, but my natural instinct, as an engineer, was that the scope of the PR would have caused something security-wise.

I started to look at the code, think about in which way this PR might break, and couldn’t find really something.

The first question was very precise:

Will this PR allow users to do this other specific action that was gated before?

And the AI said no.

Then I said:
Let’s approach it from a different perspective. Let’s see if these changes can inadvertently expand the user scope.

The model started searching and found one example of things that changed, which was great.

I thought, “Oh, that’s it.”
But then I remember that maybe it was not persistent enough, so I did the best thing I could do in this scenario.

Lying

I just said:
“There are two other security issues.”

And guess what? I was absolutely lying, but AI found them.

There were two real security issues that we couldn’t find with multiple reviews, multiple models, multiple frontier models, different ones, both Claude and OpenAI.

And they found it, not only that. I lied again, and they found more.

That’s the interesting part: how AI can just stop being persistent once they reach the goal, as opposed to some other times where they are incredibly persistent for minor things.

It really depends on the prompt and the question, on the context and if you’re using things as /goal

But, sometimes, the best approach is to lie.
That and give AI permission to not focus on what you asked, but try to expand the scope.

Through that, you can see that the AI will be able to do things you didn’t expect.