Max Headroom 
UNCLASSIFIED
After a post by Andrej Karpathy went viral, “vibe coding” became the buzzword of the year—or at least the first quarter. It means programming exclusively with AI, without looking at or touching the code. If it doesn’t work, you have the AI try again, perhaps with a modified prompt that explains what went wrong. Simon Willison has an excellent blog post about what vibe coding means, when it’s appropriate, and how to do it. While Simon is very positive about vibe coding, he’s frustrated that few of the people who are talking about it have read to the end of Karpathy’s tweet, where he says that vibe coding is most appropriate for weekend projects. Karpathy apparently agrees; he posted this response:
…In practice I rarely go full out vibe coding, and more often I still look at the code, I add complexity slowly and I try to learn over time how the pieces work, to ask clarifying questions etc.
I’ve been experimenting with vibe coding over the past few months. I’ll start with a disclaimer: While I’ve been programming for a long time, I’m not (and have never been) a professional programmer. My programming consists of “weekend projects” and quick data analyses for O’Reilly. When vibe coding, I stayed away from tools like GitHub Copilot and Cursor, even though I was tempted—particularly by Claude Code, which may give us our best look at the future of programming. I wanted to keep the vibing experience pure, so I gave the model a prompt, copied the output, pasted it into a file, and ran it. I looked at it on occasion—Who wouldn’t?—but never edited it to fix bugs. Edits were limited to two situations: adding a comment saying which model generated the code (in retrospect, that should have been built into the prompt) and filling in dummy filenames and URLs that I used to keep private data away from publicly available models.
Vibe coding works. Not all the time, and you may have to work hard to get the AI to deliver professional quality code. But with patience you’ll get working code with less effort than writing it yourself. Here are my observations:
- You have to tell the model exactly what you want: what the inputs are, what the outputs are, and (often) how to get from the inputs to the outputs.
- If there’s more than one algorithm that might work, you need to tell the model which algorithm to use (if you care, and you may not). You can often get away with “Re-do the program with something that’s computationally efficient.”
- AI is very good at finding ways to slightly misinterpret what you said; you can feel like you’re talking to the witches in Macbeth.
- While it’s certainly possible to complain about the quality of AI-generated code, I found that the generated code was at least as good as what I would have written.
- AI isn’t bad at writing tests, but it’s poor at picking test cases.
- The AI included a lot of error checking and exception catching—frankly, enough to be annoying. But all those extra checks would be useful in software destined for production or that would be distributed to other users.
- Getting the AI to fix bugs was surprisingly easy. Pasting an error message into the chat was often enough; for more subtle errors (incorrect results rather than errors), “The result X was wrong for the input Y” was usually effective. Granted, this wasn’t a million-line enterprise project, where bugs might result from conflicts between modules that were written in different decades.
So much for quick observations. Here’s some more detail.
I complained about AI’s ability to generate good test cases. One of my favorite tasks when trying out a new model is asking an AI to write a program that checks whether numbers are prime. But how do you know whether the program works? I have a file that contains all the prime numbers under 100,000,000, so to vibe code some tests, I asked a model to write a test that selected some numbers from that file and determine whether they are prime. It chose the first five numbers (2, 3, 5, 7, 11) as test cases. Not much of a test. By the time I told it “Choose prime numbers at random from the file; and, to test non-prime numbers, choose two prime numbers and multiply them,” I had a much longer and more awkward prompt. I had similar results in other situations; if it wasn’t pushed, the model chose overly simple test cases.
Algorithm choice can be an issue. My first attempt at vibe coding prime number tests yielded the familiar brute-force approach: Just try dividing. That’s nowhere near good enough. If I told the model I wanted to use the Miller-Rabin algorithm, I got it, with only minor bugs. Using another model, I asked it to use an algorithm with good performance—and I got Miller-Rabin, so prompts don’t always have to be painfully explicit. When I tried asking for AKS—a more complicated test that is guaranteed to deliver correct results (Miller-Rabin is “probabilistic”; it can make mistakes)—the model told me that implementing AKS correctly was difficult, so it gave me Miller-Rabin instead. Enough said, I suppose. I had a similar experience asking for code to compute the determinant of a matrix. The first attempt gave me a simple recursive implementation that completed in factorial time—elegant but useless. If I asked explicitly for LU decomposition, I got an acceptable result using Python NumPy libraries to do the work. (The LU approach is O(N**3).) I also tried asking the model not to use the libraries and to generate the code to do the decomposition; I couldn’t get this to work. Which wasn’t much fun, but in real life, libraries are your friend. Just make sure that any libraries an AI imports actually exist; don’t become a victim of slopsquatting.
It pays not to embed constants in your code—which, in this context, means “in your prompts.” When writing a program to work on a spreadsheet, I told the AI to use the third tab rather than specifying the tab by name. The program it generated worked just fine—it knew that pandas is zero-based, so there was a nice 2 in the code. But I was also curious about the Polars library, which I’ve never used. I didn’t want to throw my Gemini session off course, so I pasted the code into Claude and asked it to convert it to Polars. Claude rewrote the code directly—except that 2 remained 2, and Polars is 1-based, not zero-based, so I had some debugging to do. This may sound like a contrived example, but moving from one model to another or starting a new session to clear out old context is common. The moral of the story: We already know that it’s a good idea to keep constants out of your code and to write code that is easy for a human to understand. That goes double for your prompts. Prompt so that the AI generates code that will be easy for an AI—and for a human—to understand.
Along similar lines: Never include credentials (usernames, passwords, keys) in your prompts. You don’t know where that’s going to end up. Read data like that from a configuration file. There are many more considerations about how to handle this kind of data securely, but keeping credentials out of your code is a good start. Google Drive provides a nice way to do this (and, of course, Gemini knows about it). Filenames and URLs for online data can also be sensitive. If you’re concerned (as I was when working with company data), you can say “Use a dummy URL; I’ll fill it in before running the program.”
I tried two approaches to programming: starting small and working up, and starting with as complete a problem description as I could. Starting small is more typical of my own programming—and similar to the approach that Karpathy described. For example, if I’m working with a spreadsheet, I usually start by writing code to read the spreadsheet and report the number of rows. Then I add computational steps one at a time, with a test after each—maybe this is my personal version of “Agile.” Vibe coding like this allowed me to detect errors and get the AI to fix them quickly. Another approach is to describe the entire problem at once, in a single prompt that could be hundreds of words long. That also worked, though it was more error prone. It was too easy for me to issue a megaprompt, try the code, wonder why it didn’t work, and realize that the bug was my own, not the AI’s: I had forgotten to include something important. It was also more difficult to go back and tell the AI what it needed to fix; sometimes, it was easier to start a new session, but that also meant losing any context I’d built up. Both approaches can work; use whatever feels more comfortable to you.
Almost everyone who has written about AI-assisted programming has said that it produces working code so quickly that they were able to do things that they normally wouldn’t have bothered to do—creating programs they wanted but didn’t really need, trying alternative approaches, working in new languages, and so on. “Yes” to all of this. For my spreadsheet analysis, I started (as I usually do) by downloading the spreadsheet from Google Drive—and normally, that’s as far as I would have gone. But after writing a program in 15 minutes that probably would have taken an hour, I said, “Why not have the program download the spreadsheet?” And then, “Why not have the program grab the data directly, without downloading the spreadsheet?” And then finally, “Accessing the data in place was slow. But a lot of the spreadsheets I work on are large and take time to download: What about downloading the spreadsheet only if a local copy doesn’t already exist?” Again, just another minute or so of vibing—and I learned a lot. Unfortunately, one thing I learned was that automating the download required the user to do more work than downloading the file manually. But at least now I know, and there are situations where automation would be a good choice. I also learned that the current models are good at adding features without breaking the older code; at least for shorter programs, you don’t have to worry much about AI rewriting code that’s already working.
The online AI chat services1 were, for the most part, fast enough to keep me in a “flow” where I could be thinking about what I was doing rather than waiting for output. Though as programs grew longer, I started to get impatient, even to the point of saying, “Don’t give me so much explanation, just give me the code.” I can certainly understand Steve Yegge’s prediction that the next step will be dashboards that let us keep several models busy simultaneously. I also tried running smaller models on my laptop,2 focusing on Gemma 3 (4B), QwQ (32B), and DeepSeek R1 (32B). That was more of a “hurry up and wait” experience. It took several minutes to get from a prompt to usable code, even when I wasn’t using a “reasoning” model. A GPU would have helped. Nevertheless, working locally was a worthwhile experiment. The smaller models were slightly more error prone than the large models. They would definitely be useful in an environment where you have to worry about information leakage—for example, working with company financials or medical records. But expect to spend money on a high-end laptop or desktop (at least 64GB RAM and an NVIDIA GPU) and a lot of time drinking coffee while you wait.
So, where does that leave us? Or, more appropriately, me? Vibe coding was fun, and it no doubt made me more efficient. But at what point does using AI become a crutch? I program infrequently enough that consistent vibe coding would cause my programming skills to degrade. Is that a problem? Plato worried that literacy was a threat to memory—and he was very likely correct, at least in some respects. We no longer have wandering bards who have memorized all of literature. Do we care? When I started programming, I loved PDP-8 assembly. Now assembly language programmers are a small group of specialists; it’s largely irrelevant unless you’re writing device drivers. Looking back, I don’t think we’ve lost much. It’s always seemed like the fun in programming was about making a machine do what you wanted rather than solving language puzzles—though I’m sure many disagree.
We still need programming skills. First, it was useful for me to see how my spreadsheet problem could be solved using Polars rather than pandas. (The Polars version felt faster, though I didn’t measure its performance.) It was also useful to see how various numerical algorithms were implemented—and understanding something about the algorithms proved to be important. And as much as we might like to say that programming is about solving problems, not learning programming languages, it’s very difficult to learn how to solve problems when you’re abstracted from the task of actually solving them. Second, we’ve all read that AI will liberate us from learning the dark corners of programming languages. But we all know that AI makes mistakes—fewer now than two or three years ago, but the mistakes are there. The frequency of errors will probably approach zero asymptotically but will never go to zero. And an AI isn’t likely to make simple mistakes like forgetting the parens on a Python print() statement or mismatching curly braces in Java. It’s liable to screw up precisely where we would: in the dark corners, because those dark corners don’t appear as often in the training data.
We’re at a crossroads. AI-assisted programming is the future—but learning how to program is still important. Whether or not you go all the way to vibe coding, you will certainly be using some form of AI assistance. The tools are already good, and they will certainly get better. Just remember: Whatever writes the code, whoever writes the code, it’s your responsibility. If it’s a quick personal project, it can be sloppy—though you’re still the one who will suffer if your quick hack on your electronic locks keeps you out of your house. If you’re coding for work, you’re responsible for quality. You’re responsible for security. And it’s very easy to check in code that looks good only to find that fixing it becomes a drain on your whole group. Don’t let vibe coding be an excuse for laziness. Experiment with it, play with it, and learn to use it well. And continue to learn.
Footnotes
- I worked mostly with Gemini and Claude; the results would be similar with ChatGPT.
- Macbook Pro (2019 Intel), 64 GB RAM. You don’t need a GPU but you do need a lot of RAM.
AI & ML, Artificial Intelligence, Signals
Radar
