Recently me and a coworker had the unusual scenario where we were working on exactly the same bug.
He was using Claude, and I was using Codex, but what was fascinating was the PR result: The code output was incredibly similar. It was a small change, not a big refactoring, but they were so close, almost identical.
Different Models, right?
There is a lot of talk about all these AI models and why one is better than the other. This is true to some extent. Each model has different capabilities and skills, and some areas are trained differently.
The way the model will reply to you will be different. But in this case the results were so similar that I wondered if we were using the same model. Guess what? We were not.
Getting predictable results is, obviously, extremely valuable.
At the organizational level, you want your models to behave in a predictable manner, even though they are not predictable because they might hallucinate and do things that you didn’t.
The fact that these two things were extremely similar made me wonder: how is this possible?
Investigating
What led to this?
Well, for one, the scope for this change was small. The code base was well structured and very predictable, with a lot of old patterns used to inform the change.
The change was small enough to allow the AI to not digress or think too much. This kept the scope down and made everything easier to flow into the same result.
That gave me some ideas about how to create and model the way AI models work so that they behave in a predictable manner.
Three Things to help Predictable Output
The things I get out of this experience are three:
One: keep the change small. This is self-explanatory.
Two: consolidate on patterns.
I have not been reading a lot about consolidating patterns in codebases.There is a lot about “search for bugs and regression” as a skill or an automation but there is very little on: look at the codebase and the patterns that are emerging. Then consolidate so that the codebase already provides a way of doing things.
Third: give it context that is meaningful.
In this case, the codebase already had enough context. The change was small enough and did not need too much guesswork.
Every time I interact with AI, I try to see how that could be improved.
Some interactions are frustrating. Some are mind-blowing. You are in between this rollercoaster of emotions while using it, but the good news is that we can always adjust and steer it (even outside of the single conversation) so that it can converge more.
Bonus: “The Simplicity Prompt”
One prompt I’ve been using lately when coding is this, I love how it forces AI to rethink its work.
Review current additions and look for ways to make it even simpler and less code heavy. Aim for beauty and simplicity. Removal of unneeded code is ultimate beauty
Leave a Reply