I’ve been using Raptor Mini and Codex to randomly check bug-reproduction code posted in Crystal compiler GitHub issues. They can often point out where the “bug” is.
In the process, I came to understand that the bugs remaining today are more likely to be undecided edge cases in language design than pure bugs. Whether a given snippet passed to the Crystal compiler should work as valid code, or be rejected with an error, depends on human preference. So when creating a pull request, a small statement of what Crystal should be slips in and makes it meaningful.
There are other kinds of bugs that might not even reported. For example, improving the API documentation is one of those, IMHO.
I think AI-assisted coding is probably here to stay, even if there are still many who prefer to write code themselves. Both views are completely understandable, but I still believe the human element is required even for AI-assisted code.
If you’re good at crafting clever system prompts, you can achieve quite a lot with AI-assisted code writing — provided you’re always working with fresh context and keeping good track of project changes through a CHANGELOG.md and a README.md. Both can provide decently fresh context each time you begin a new coding session.
Some specialized prompts worth using are ones designed to act as experts in best-practice syntax, structure, and design choices that can scrutinize your codebase. Have it always reference the most current documentation and related sources, with strict “don’t guess or make things up” guardrails.
While not always perfect, this can catch a surprising number of bugs and inaccuracies if performed multiple times with fresh context during each review. You can even take it a step further and have another specialized prompt designed to act as an expert at identifying and “de-AI-ifying” code to at least make it look more human.
Having it roast code works well for when I want to find bugs, it’s kind of mean, but it seems to do well. Burnt out senior dev roasting code is one of my favorite prompts.
It’s definitely good at roasting. ;D
If this were a year ago, I probably wouldn’t be using AI (GPT + Codex) so proactively, but now I really love it.
I’m not sure if my approach counts as “vibe-coding,” so I’ll break it down into two parts:
- Focusing on small, well-defined problems. (Except for the very first, highly detailed prompt， there are often leads to a lengthy conversation too)
This is how I mostly coexist with GPT: we dive deep into specific issues, narrowing them down to the smallest possible scope, removing any ambiguity, and guiding the discussion correctly. In most cases, there’s a particular aspect where I’m not strong, and GPT’s insights from other languages, combined with its broad knowledge base, often deliver surprisingly good results—sometimes even exceeding expectations.
Regardless of whether the problem gets fully resolved, I usually end up having many follow-up conversations with it, and I always come away with valuable insights.
- Building new features using Codex.
I typically ask it first to explain how to approach the task. If the explanation seems solid, I give it plenty of suggestions—often emphasizing what not to do rather than what to do. Only after confirming it fully understands my intent do I ask it to actually start coding (yes, writing the code directly). Then I review every single line: deciding what needs refactoring, what could be improved, and asking it to explain anything I’m unsure about. The whole process feels relaxed and enjoyable, and I’ve actually learned a ton from it.
Like in the first part, I constantly need to curb GPT’s tendency to over-explain or drift off-topic. That does take some time upfront, but overall, it’s far more efficient than implementing the same feature myself (which I might not even be capable of doing). The final code often looks quite different from the initial idea, but the quality is excellent—in fact, in handling certain details, it sometimes outperforms what I could write myself.
AI’s role here isn’t simply to “outsource work to it”; instead, GPT-5.4 feels like a teacher and pair-programming partner who’s incredibly aligned with my MBTI.
Of course, there’s one thing I almost never get involved in (I’m even too lazy to do a code review): I let Codex handle it entirely. I don’t change anything myself, and if anything goes wrong, I just let it fix itself. That one thing is the spec.
Yes, I’ve stopped writing specs!!
(Translated with qwen3.5:35b)
I admire your patient and detailed approach. Indeed, one can learn a lot from it as well.
While it all sounds sensible and thoughtful, but at this point:
I want to scream “YOU’RE DOING IT WRONG!”, just as every time I hear “and then I make the LLM generate the unit test”.
You’re completely right that you’ve stopped writing specs, because you’re auto generating mindless regression tests.
The point of a spec isn’t to write a test that’ll break whenever you change the tested code, the point is to make it break if you change the existing expected behaviour. That’s why TDD proscribes that you (or the LLM) write the test first, in order to focus on “what is it I want it to do” rather than after where the resulting tests will most likely test “what it happens to do”. The latter usually breaks on any change of the code, while the former ensures that your change doesn’t break existing code.
That’s why we’re calling it specs instead of tests. It’s a specification of what we want from the tested code.
But if you’re not interested in the specification, don’t deal with it and just tell the LLM to fix it, it’s worthless, it doesn’t specify anything, it just test stuff, and you can’t be sure that what it tests is actually correct, so why bother at all?
You’re absolutely right. The reason I let AI write my tests is precisely because they require zero effort from me to write or maintain. These tests might end up being completely useless, but honestly, I don’t care.
In fact, I sometimes take a bit of pride in calling myself a TDD expert. Throughout my decade-plus career, I have genuinely been the person who writes a massive amount of tests, and I do them well—whether it’s data-driven development, stubbing, mocking, or integration testing.
However, my approach to testing has some personal quirks:
-
I don’t necessarily start with the tests. I only write them after I’ve experimented with the logic and thought deeply about the requirements, and only if I’m certain the tests will actually aid my development. (For example, I might build datasets to better use test-driven approaches to validate business logic, rather than just testing a complex database query and having to manually check it in a browser like most programmers do.) The key point is: tests don’t always come first. I might write them early, I might write them late, and if I feel a feature is important, but no test, I’ll come back and add the tests later.
-
I tend to favor redundancy over “DRY” (Don’t Repeat Yourself) principles in my tests. I believe DRY is great for production code, but it makes tests too brittle. I often copy and paste large chunks of test code, even though I almost never do that in production code.
-
Since moving from Ruby to Crystal, I’ve drastically reduced the amount of testing I do. When I do write tests, they are almost exclusively for the purposes I mentioned above. As I said, “I’ve stopped writing specs!!”—meaning I’ve stopped writing those types of tests which not so necessory when I start to use Crystal, You’re right, the tests GPT generates for me might be unnecessary because I might ensure correctness through other means, such as black-box tests I wrote myself using bash scripts. But if they happen to be useful (e.g., I might revert production code but run the tests to see if they work as regression tests), then it’s a win-win, right?
(Translated with gemma4:26b)
Well, don’t write specs then. Sounds like they serve no purpose and provide no value.
Whatever you do in your own projects is of course up to you, but I’ll put forward the argument as for code shared with others, no specs are more honest than LLM maintained specs.
I just think that the presence of a spec directory in a project signals that someone has had an opinion on the shape of certain things. A signal that’s undermined if it’s entirely LLM controlled.
That’s a great point. As for specs, my rule is simple: if it’s useful, I keep it. (In fact, the specs written by codex 5.4 are actually pretty solid, I often intentionally delete key pieces of production code just to see if the spec fail as expected, it’s did!)
Anyway, even though I’ve been writing spec for a long time and could be considered an expert, but my approach to testing isn’t all or nothing.
(Translated from Chinese using gemma4:26b)
You can always prove the test suite value using mutation testing.
But what does that prove? It proves that the test breaks when you change the code.
But that’s not what you want, you don’t want tests that break when you implement new features, you don’t want tests that break when you fix a bug.
You want the test to break when you break expected behavior.
If you leave specs up to the LLM, how can you be sure it’s not actually testing and passing buggy behaviour?
You are setting up a strawman and asking for guarantees nobody ever asks for.
If you leave specs up to your coworker, how can you be sure they’re not actually testing and passing buggy behaviour?
If you leave specs up to you, how can you be sure you are not actually testing and passing buggy behaviour?
If your “bug fix” is not breaking an existing test, then the buggy behaviour was just not tested and the test suite is irrelevant to any discussionn regarding that “bug”.
What I said was “prove the test suite value”. I have seen plenty of human-written suites that failed to break under mutation testing. Those had no value.
Speaking of strawman, why would you do that? I know, in the old days, and probably still alive and well in the enterprise segment, you had developers and you had testers. But in those kinds of arrangements you have a specification which the developer tries to implement and the tester test up against, so that’s an entirely different ballgame.
If you just lob a bunch of code over to a co-worker and say “write tests for this” without further instructions, you have a pretty dysfunctional development style. But that’s essentially what people do with the LLM.
Because I write the test first? Save for the occasional brainfart, the spec is the manifestation of the vision I have for the code. The LLM doesn’t have a vision, it just sees that multiply(2, 2) returns five and writes a test for that.
Again, prove what? Mutation testing can prove that tests are insufficient to catch all possible changes. But not all changes are bugs and some changes are bug fixes.
Which leads us back to the “specification” part. It’s not about testing that the code haven’t changed, it’s about checking that the expected behaviour haven’t changed. If your specs break on every change, they’re worthless.
I’m not saying one can’t use LLM to help write tests, it’s the idea that tests are second class citizens we’ll just let the LLM fix for us. The tests sole purpose is to prove that your other code works as intended, in my book that makes them even more important to get right. And i general you don’t get stuff “right” by just leaving it to the LLM.
Why would you leave specs to a coworker? Because the project is large enough that you delegated something.
But you know what? Whatever. Not replying anymore to you, it’s completely useless.
While Vibe Coding gets attention, I am more interested in AI as a tool for exploration and investigation.
Using AI to investigate large data like a detective will become important.
In programming, the target is the ever-growing codebase. The challenge is coverage: where across the codebase is a given function called? In Crystal, despite being statically typed, call resolution can be complex enough that you largely have to rely on search to find out. Crystal’s characteristics are generally well-suited to AI, though this may be one weak point. (Or perhaps grep is enough.)
As for specs, I think they have always carried another meaning: creating a human-readable comprehensive index of the codebase.
This discussion about TDD, BDD, and writing tests gave me a lot to think about in terms of how it relates to my current project. My project is an existing tool written in Bash with a Python version as well, which I’ve chosen to port to Crystal. I’m sure there are cases for spec/ in my project. However, my project depends heavily on deep knowledge of Linux internals and how the system works as a whole.
My source of truth for testing are retired vulnerable machines with a plethora of writeups all over the internet where escalation paths were found. As long as my tool detects them without flooding the user with noise and false positives, it works. To Xen’s point though, logic bugs still exist. They can and WILL emerge if you’re not testing your code for expected behavior before writing out new features or continuing to build it out.
I’d be open to suggestions, thoughts, or feedback on how I might be able to incorporate testing into my project. Currently I’m about 85% of the way to completion, at which point I’ll be shifting my focus to improving the UX and how the findings summary is displayed.
Please keep it polite 
You can just say “But you know what? I’ll leave it here.”
Maybe I sounded less polite than I intended. I meant it’s useless to argue because the discussion reached a point where positions are based on feelings and not amenable to argument.
Sorry for any offense, was not intended.