Max Headroom 
Join Steve Wilson and Ben Lorica for a discussion of AI security. We all know that AI brings new vulnerabilities into the software landscape. Steve and Ben talk about what makes AI different, what the big risks are, and how you can use AI safely. Find out how agents introduce their own vulnerabilities, and learn about resources such as OWASP that can help you understand them. Is there a light at the end of the tunnel? Can AI help us build secure systems even as it introduces its own vulnerabilities? Listen to find out.
Check out other episodes of this podcast on the O’Reilly learning platform.
About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.
Points of Interest
- 0:00: Introduction to Steve Wilson, CPO of Exabeam, O’Reilly author, and contributor to OWASP.
- 0:49: Now that AI tools are more accessible, what makes LLM and agentic AI security fundamentally different from traditional software security?
- 1:20: There’s two parts. When you start to build software using AI technologies, there is a new set of things to worry about. When your software is getting near to human-level smartness, the software is subject to the same issues as humans: It can be tricked and deceived. The other part is what the bad guys are doing when they have access to frontier-class AIs.
- 2:16: In your work at OWASP, you listed the top 10 vulnerabilities for LLMs. What are the top one or two risks that are causing the most serious problems?
- 2:42: I’ll give you the top three. The first one is prompt injection. By feeding data to the LLM, you can trick the LLM into doing something the developers didn’t intend.
- 3:03: Next is the AI supply chain. The AI supply chain is much more complicated than the traditional supply chain. It’s not just open source libraries from GitHub. You’re also dealing with gigabytes of model weights and terabytes of training data, and you don’t know where they’re coming from. And sites like Hugging Face have malicious models uploaded to them.
- 3:49: The last one is sensitive information disclosure. Bots are not good at knowing what they should not talk about. When you put them into production and give them access to important information, you run the risk that they will disclose information to the wrong people.
- 4:25: For supply chain security, when you install something in Python, you’re also installing a lot of dependencies. And everything is democratized, so people can do a little on their own. What can people do about supply chain security?
- 5:18: There are two flavors: I’m building software that includes the use of a large language model. If I want to get Llama from Meta as a component, that includes gigabytes of floating point numbers. You need to put some skepticism around what you’re getting.
- 6:01: Another hot topic is vibe coding. People who have never programmed or haven’t programmed in 20 years are coming back. There are problems like hallucinations. With generated code, they will make up the existence of a software package. They’ll write code that imports that. And attackers will create malicious versions of those packages and put them on GitHub so that people will install them.
- 7:28: Our ability to generate code has gone up 10x to 100x. But our ability to security check and quality check hasn’t. For people starting, get some basic awareness of the concepts around application security and what it means to manage the supply chain.
- 7:57: We need a different generation of software composition environment tools that are designed to work with vibe coding and integrate into environments like Cursor.
- 8:44: We have good basic guidelines for users: Does a library have a lot of users? A lot of downloads? A lot of stars on GitHub? There are basic indications. But professional developers augment that with tooling. We need to bring those tools into vibe coding.
- 9:20: What’s your sense of the maturity of guardrails?
- 9:50: The good news is that the ecosystem around guardrails started really soon after ChatGPT came out. Things at the top of the OWASP Top 10, prompt injection and information disclosure, indicated that you needed to police the trust boundaries around your LLM. We’re still figuring out the science for figuring out good guardrails for input. The smarter the models get, the more problems they have with prompt injection. You can send prompt injection through images, emojis, foreign languages. Put in guardrails on that input, but assume they will fail, so you also need guardrails on the output to detect kinds of data you don’t want to disclose. Last, don’t give access to certain types of data to your models if it’s not safe.
- 10:42: We’re often talking about foundation models. But a lot of people are building applications on top of foundation models; they’re doing posttraining. People seem to be very excited about the ability of models to connect to different tools. MCP—Model Context Protocol—is great, but this is another vector. How do I know an MCP server is sufficiently hardened?
- 13:42: One of the top 10 vulnerabilities on the first version of the list was insecure plug-ins. OpenAI had just opened a proprietary plug-in standard. It kind of died out. MCP brings all these issues back. It’s easy to build an MCP server.
- 14:31: One of my favorite vulnerabilities is excessive agency. How much responsibility am I giving to the LLM? LLMs are brains. Then we gave them mouths. When you give them fingers, there’s a whole different level of things they can do.
- 15:00: Why could HAL turn off the life support system on the spaceship? As I build these tools—is that a good idea? Do I know how to lock that down so it will only be used in a safe manner?
- 15:37: And does the protocol support secure usage. Google’s A2A—in the security community, people are digging into these issues. I would want to make sure that I understand how the protocols work, and how they’re attached to tools. You want to be experimenting with this actively, but also understand the risks.
- 16:45: Are there lessons from web security like HTTP and HTTPS that can map over to the MCP world? A lot of it is based on trust. Security is often an afterthought.
- 17:27: The internet was built without any considerations for security. It was built for open access. And that’s where we’re at with MCP. The lesson from the early internet days is that security was always a bolt-on. As we’ve gone into the AI era, security is still a bolt-on. We’re now figuring out reinforcement learning for coding agents. The opportunity is for us to build security agents to do security and put them into the development process. The last generation of tools just didn’t fit well into the development process. Let’s build security into our stacks.
- 20:35: You mentioned hallucination. Is hallucination an annoyance or a security threat?
- 21:01: Hallucination is a big threat and a massive gift. We debate whether AIs will create original works. They’re already generating original things. They’re not predictable, so they do things you didn’t quite ask for. People who are used to traditional software are puzzled by hallucination. AIs are more like humans; they do what we train them to do. What do you do if you don’t know the answer? You might just get it wrong. The same thing happens with LLMs.
- 23:09: RAG, the idea that we can give relevant data to the LLM, dramatically decreases the probability that they will give you a good answer but doesn’t solve the problem entirely. Understanding that these are not purely predictable systems and building systems defensively to know that will happen is really important. When you do RAG well, you can get very high percentage results from it.
- 24:23: Let’s talk about agents: things like planning, memory, tool use, autonomous operation. What should people be most concerned about, as far as security?
- 25:18: What makes something agentic? There’s no universal standard. One of the qualities is that they’re more active; they’re capable of carrying out actions. When you have tool usage, it brings in a whole new area of things to worry about. If I give it power tools, does it know how to use a chain saw safely? Or should I give it a butter knife?
- 26:10: Are the tools attached to the agents in a safe way, or are there ways to get into the middle of that flow?
- 26:27: With better reasoning, models are now able to do more multistep processes. We used to think of these as one- or two-shot things. Now you can have agents that can do much longer-term things. We used to talk about training data poisoning. But now there are things like memory poisoning—an injection can be persistent for a long time.
- 27:38: One thing that is quite glaring: Most companies have incident response playbooks for traditional software. In AI, most teams don’t. Teams haven’t sat down and decided what’s an AI incident.
- 28:07: One of the OWASP pieces of literature was a guide for response: How do I respond to a deepfake incident? We also put out a document on building an AI Center of Excellence in particular for AI security—building AI security expertise inside your company. By having a CoE, you can be sure that you are building out response plans and playbooks.
- 29:38: Teams can now build interesting prototypes and become much more aggressive about rolling out. But a lot of these prototypes aren’t robust enough to be rolled out. What happens when things go wrong? With incident response: What is an incident? And what’s the containment strategy?
- 30:38: Sometimes it helps to look at past generations of these things. Think about Visual Basic. That presented a whole new class of citizen developers. We wound up with hundreds of crazy applications. Then VB was put into Office, which meant that every spreadsheet was an attack surface. That was the 1990s version of vibe coding—and we survived it. But it was bumpy. The new generation of tools will be really attractive. They are enabling a new generation of citizen developers. The VB systems tended to live in boxes. Now, they’re not boxed in any way; they can look like any professional project.
- 33:07: What I hate is when the security gets on their high horse and tries to gatekeep these things. We have to acknowledge that this is a 100x increase in our ability to create software. We need to be helping people. If we can do that, we’re in for a golden age of software development. You’re not beholden to the same group of megacorps who build software.
- 34:14: Every year I walk around the expo hall at RSA and get confused because everyone is using the same buzzwords. What’s a quick overview of the state of AI being used for security?
- 34:53: Look for the places where people were using AI before ChatGPT. When you’re looking at things like user and entity behavior analytics—inside a security operations center, you’re collecting millions of lines of logs. The analyst is building brittle correlation rules trying to find needles in haystacks. With user and entity behavior analytics, you can build models for complex distributions. That’s getting to be pretty robust and mature. That’s not large language models—but now, when you search, you can use English. You can say, “Find me the top 10 IP addresses sending traffic to North Korea.”
- 37:01: The next thing is mashing this up with large language models: security copilots and agents. How do you take the output out of user and entity behavior analytics and automate the operator making a snap decision about turning off the CEO’s laptop because his account might be compromised? How do I make a great decision? This is a great use case for an agent built on an LLM. That’s where this is going. But when you’re walking around RSA, you have to be aware that there’s never been a better time to build a great demo. Be deeply skeptical about AI capabilities. They are real. But be skeptical of demos.
- 39:09: Many of our listeners are not familiar with OWASP. Why should our listeners listen to OWASP?
- 39:29: OWASP is a group that is more than 20 years old. It’s a group about producing secure code and secure applications. We started on the back of the OWASP Top 10 project: 10 things to look out for in your first web application. About two years ago, we realized there was a new set of security things that were neither organized or documented. So we put together a group to attack that problem and came out with the top 10 for large language models. We had 200 people volunteer to be on the experts group in the first 48 hours. We’ve branched out to how to make agents, how to red team, so we’ve just rechristened the project as the GenAI security project. We will be at RSA. It’s an easy way to hop in and get involved.
AI & ML, Artificial Intelligence, Generative AI in the Real World, Commentary
Radar
