Max Headroom 
UNCLASSIFIED
If your company is building any kind of AI product or tool, congratulations! You are now an AI company.
Yes, you’re still a retail company. Or a bank. Or a CPG operation. You’re that plus an AI company—let’s call this an AI as Well company (AIAW)—granting you a license to tell sales prospects and investors that you’re “doing AI.”
That license also puts you on the hook for new responsibilities. They’re easy to skip over at first, but you’ll hold yourself back from your true AI potential if you do. And maybe take on needless risk exposures in the process.
If you AIAWs want to make the most of AI, you’d do well to borrow some hard-learned lessons from the software development tech boom. And in return, software dev also needs to learn some lessons about AI.
We’ve seen this movie before
Earlier in my career I worked as a software developer. I quickly learned that any company building custom software—no matter their core business—had to learn the ropes of running a professional software product shop.
Which was all well and good, except that they had no experience running a software product shop. Executives’ decisions were based on a surface-level understanding of custom software—mostly, “throw some developers into a room and tell them what to build”—which was enough to get started but nowhere near enough to succeed.
If you’ll pardon the well-worn “iceberg” analogy, most of what they needed to know about custom software existed below the waterline. That’s where they’d find things like “how to build a team.” (Remember the misguided job postings that required a computer science degree?) Then there was “the need for separate dev, QA, and production runtime environments,” each of which called for their own hardware. That led to “we need to hire people to do QA and manage ops.” The subsurface knowledge also included legal concerns like intellectual property (IP), which dovetailed with open source licenses… And so on.
That was a lot to learn. And yet, it was just enough to get the initial product out the door—a sizable achievement but one which is said to run just 20 percent of a software project’s total lifetime cost. The time, effort, and money required for long-term maintenance came as a triple sticker shock.
(The bonus lesson here is that the so-called “overpriced” off-the-shelf software they were trying to replace wasn’t so overpriced after all. But that’s a story for another day.)
There were plenty of strategic matters below the waterline too. Companies weren’t just adding software to their business; that custom software changed how the business operated. The ability to run certain processes 24/7/365 created new efficiencies and risks alike. The efficiencies were double-edged: Automating one process might overwhelm downstream processes that were still done by hand. Managing the new risks required everyone to exercise new discipline. One person forcing a hasty code change could upset operations and lead to sizable losses.
These concerns still hold today, but they’re mostly invisible, if not laughable, because software development has matured. Company leadership is well-versed in industry best practices. (In part, because many of today’s tech leaders are former developers who learned those best practices firsthand.) But back then progress was measured in hard-learned lessons, based on short knowledge horizons. Each step revealed more of the custom software picture, showing leaders that their previous understanding was oversimplified and underpriced.
Some leaders retained expert help to protect their investment and accelerate their efforts. Others stubbornly pushed through on their own and eventually figured it out. Or they didn’t figure it out and suffered downtime incidents, high turnover, and project failures.
We don’t have to relive that same movie
A similar story is playing out in the AI space. (For brevity, I’ll lump all of data science, machine learning, and GenAI under the term “AI.”) Like early-day custom software development, today’s AI opportunities bear the price tag of new approaches and new discipline. You can’t just cram a bunch of data scientists into an office and cross your fingers that everything works out.
Plenty of companies have tried. They’ve stumbled through the dark room that is AI, bumping their shins and stepping on spikes because…I don’t know why. Hubris? Ego? A love of pain?
Today’s newly minted AI as Well companies, like their earlier software counterparts, have to address operational matters of this new technology. But before that, AIAWs must perform prep work around strategy: “What is AI, really? What can it do in general, and what can it do for us in particular? How can incorporating AI into our products harm us or our customers or unaffiliated parties who just happen to be in the wrong place at the wrong time?”
Answering these higher-level questions requires AI literacy, and that starts at the top of the org chart. A leadership team that appreciates the full scope of AI’s capabilities and weaknesses is prepared to make realistic decisions and surface meaningful use cases. They know to involve the legal, PR, and risk management teams, early and often, to limit the number of nasty surprises down the road.
And there are plenty of surprises to go around. Most stem from AI’s probabilistic nature: Models may exhibit a sudden spike in errors, either because they’ve hit some weird internal corner case or the outside world has changed. And that’s if you can even get them to work in the first place. Like a financial investment, AI can bring you 10x return or eat your money or anything in-between. You can influence that outcome, but you can’t control it—no amount of shouting, cajoling, or all-nighter sessions can force a model to perform well.
Then there are the new risks AI brings to the table. The models will inevitably be wrong now and then; how do you handle that? How often can they be wrong before you find yourself in hot water? Are you licensed to use that training data for this specific commercial purpose? Are you permitted to operate that model in every jurisdiction where it interacts with your end users?
Expect some of those legal questions to be in flux for a while. You might win by sitting in the gray area of regulatory arbitrage, but only if you’re prepared for a fast pivot when those boundaries shift. And that’s just the court of law. You also face the court of public opinion. AI practices that are considered creepy or invasive can trigger a public backlash. (Hint: You may want to steer clear of facial recognition for now.)
You’ll notice how much ground I’ve covered before any talk of hiring. Bringing AI into a company means you have new roles to fill (data scientist, ML engineer) as well as new knowledge to backfill in existing roles (product, ops). Companies that begin their AI journey by hiring data scientists are skipping a lot of prep work, at their peril.
Capping the list of lessons for AIAWs, there’s vigilance. AI is a changing landscape. There is no viable “set it and forget it” approach. Roles, strategy, and execution all call for periodic review and adjustment.
A strong weak point
AIAWs that run strong software development shops are, counterintuitively, poised to learn these lessons the hard way.
That software strength doubles as their AI weakness. Since application development and AI both involve writing code, they overestimate the overlap between the two. We know Python. All this AI stuff is Python. How hard could it be?
These firms adopt AI the same way some developers move to a new programming language: by clinging to the mindset of the old. Their code may pass the Python interpreter, but it’s all Java constructs. Java-flavored Python is hard to support and doesn’t make the most of what Pythonic Python has to offer.
So what is software dev-flavored AI? It’s the CEO who assumes that, by using a popular LLM API or other AI-as-a-service (AIaaS) product, they won’t need any AI expertise in-house. It’s the product lead who announces AI-backed features before the models have proven themselves. Or expects software’s consistency of behavior once the models are in service. It’s the CTO who is so dead set on getting their AI efforts to conform to Agile that they never look for AI-specific best practices. This person high-fives the lead developer, who believes their model is ready for prime time because they’ve followed the TensorFlow tutorial.
Overall, it’s the company that moves forward on AI at high speed, driven by a self-confidence that overshoots their horizon of knowledge. This arrogance injects needless frustration and risk exposure into their AI efforts.
The funny part is that this crew might actually get an AI product out the door. But they will not realize the harsh truth: Just because it runs doesn’t mean it works.
At least, they won’t realize this until after the AI-enabled app is interacting with customers and driving business processes. The inevitable problems will be far more difficult to address at that point.
The future is the past all over again
I actually sketched the first part of this article several years ago. Back then it focused on companies getting into custom software. It was unsettling to come across the old outline and see the same story playing out in today’s AI world.
Ironically, the AIAWs that are best at creating software stand to learn the most. They first need to unlearn certain software practices in order to build AI solutions using an AI mindset. But once these companies truly embrace AI best practices, they are also positioned for the biggest wins. They already understand ideas like uptime, deployment, version control, and monitoring, which is everything required once you move the model out of R&D and into production.
On May 8, O’Reilly Media will be hosting Coding with AI: The End of Software Development as We Know It—a live virtual tech conference spotlighting how AI is already supercharging developers, boosting productivity, and providing real value to their organizations. If you’re in the trenches building tomorrow’s development practices today and interested in speaking at the event, we’d love to hear from you by March 12. You can find more information and our call for presentations here. Just want to attend? Register for free here.
AI & ML, Artificial Intelligence, Research
Radar
