Max Headroom 
UNCLASSIFIED
As a pandas core contributor and early Parquet adopter who built AI data pipelines at streaming company Tubi TV, Chang She saw firsthand why the traditional data stack breaks down for AI workloads—and founded LanceDB to fix it. Chang joined Ben Lorica to explain why vector databases are too narrow a solution for modern AI data needs, and what a true multimodal data infrastructure actually looks like. Chang and Ben get into why the Lance file format is quickly becoming the open source standard for multimodal data, how the rise of agents is exploding data infrastructure demands, why open-weight models are the enterprise cost shift to watch in the next 12 months, and more. “Trillion is the new billion,” Chang says, and the enterprises that set up their data infrastructure now for that scale will be the ones that succeed.
About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.
Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.
Transcript
This transcript was created with the help of AI and has been lightly edited for clarity.
00.35
All right, so today we have Chang She, CEO and cofounder of LanceDB, which you can find at lancedb.com. Tagline is “Build better models faster.” So Chang, welcome to the podcast.
00.49
Hey Ben, super excited to be here.
00.52
All right, we’ll jump into the core topics, but a bit of a background there for our listeners who may not be familiar with you. You worked on pandas—you were a core member of the pandas team. You were very early on with Parquet as well. And at some point, you became convinced that for AI workloads, these former tools that you worked on—Parquet, pandas—were not enough. So what was the moment of realization for you that these traditional tools that were foundational for analytics were lacking?
01.33
Absolutely. So I worked at a company called Tubi TV, which was video on-demand and streaming. So movies and TV. And it was there that I ended up dealing with a lot of I guess what I would call AI data. So we had to have embeddings for personalization, video assets, image assets, audio, text for subtitles and all of those things. All of those did not really fit into the traditional data stack—you know, pandas, Spark, Parquet, and even Arrow. So that was sort of the inspiration for me to start LanceDB.
02.15
And Chang, at this point, do you think that more people are aware of this disconnect between those tools and the kinds of tools they’ll need moving forward?
02.30
When I talk to data infrastructure folks who are building and managing that stack for dealing with this kind of data, there’s broad recognition that something has to be done, that the existing stack is just not sufficient to deal with this data. And what’s more interesting is that this data is also becoming a lot more valuable because of AI.
02.52
So obviously, before you came on the scene, there was this wave of vector stores or vector databases which were optimized for retrieval. So let’s say I’m a listener and all I have is text. Do I need anything beyond the vector database?
03.17
Even if you just have text and you just have text embeddings, the creation of those embeddings and then the management of all of those data assets—your metadata, the actual documents, how to serve that—a lot of that falls outside the purview of a vector database. The vector databases tend to be very narrow solutions for a very narrow problem, whereas something like LanceDB takes a broader view of, “When you have AI data, what are all the things you need to do to it throughout that life cycle of application development or model development? And how do we build a tool and a system that allows you to simplify your life by having one system to do all of the major workloads throughout that life cycle?”
04.13
And by the way, for our listeners, there’s LanceDB and then there’s the open Lance file format, and I wanna ask you about this file format in a second, but you mentioned something about vector databases and you were kind of saying that, you know, they’re not great at creating the embeddings. But Chang, the vector database people, they never really positioned themselves as responsible for creating the embeddings, right? So they just assume that you’ll show up with embeddings.
04.47
That’s right. But even if you take that narrow view, what we find in enterprises today is a lot of folks have an offline generation process in the data lake itself, where they chunk up the documents, then they generate the embeddings, then they have what they call an offline store, then they have to copy-paste that data into a vector database for serving. So there’s a lot of data syncing [and] data movement, so it creates expense and there’s a lot of complexity.
And so that’s the. . . Even for just text-based workloads, even just for pure vector search, that tends to be a big pain point. And then two is vector databases, a lot of times, don’t pay as much attention to the overall retrieval stack, right? If you remember, the task for users is I want to find the right data in my dataset, and vector search is just one technique. You have many different kinds of techniques, full-text search, or even just outside of search. You might have SQL queries that you want to run, filters, regexes, all of that goes into a rich and very accurate retrieval process. And vector databases, in general, do not expand beyond just that simple semantic or vector search.
06.10
So I mentioned the Lance open file format, which. . . I guess the shortcut that people use is like Parquet for AI, but it’s actually both a file and table format. So maybe give our listeners, Chang, a high-level description of the Lance format and why it’s become so popular.
06.33
Lance is what we call a lakehouse format. It is quickly becoming the new open source standard for multimodal data. And what I mean by a lakehouse format is that it spans a couple of different layers. So you mentioned in the beginning a file format. So this is the equivalent in the stack to Parquet, where we would talk about “How do we lay out the data in a particular file?” And at this layer, the innovation in Lance is that it is really, really good for random access without sacrificing any speed and scans. And our files are actually smaller than Parquet for many AI datasets.
The next layer is usually what we call a table format that is occupied by projects like Iceberg and Delta and Hudi today. And [the] Lance format comes in at this layer. We have much better designs, more optimizations for machine learning experimentation, so doing backfills easily, doing two-dimensional data evolution, being able to handle really large blob data like videos and images, and then just being able to do a branching strategy that supports true sort of Git for data semantics that takes the best of Parquet and Iceberg.
And then finally, there’s a third layer, which is about indexing so that you can have fast scans, fast searches, fast queries. So when you put all that together, that’s what we call the Lance lakehouse format.
08.11
I described Lance as open. Can you kind of clarify what that means, because I actually don’t know?
08.19
Number one is Lance format is open source. It’s Apache 2.0 license. You can find it on our GitHub. We have community governance; [we] have PMCs that are from lots of external contributors. And then I think beyond that, there’s open source and there’s open source, right? I think what Lance format is designed for is a true open architecture as well. So not only is it open source; it also plays really well into the rest of the data ecosystem.
So for example, when people compare us to Parquet and Iceberg, well, we’re not designed as a head-to-head competitor with Parquet and Iceberg. We will slot into the same Polaris data catalog, or you can have one unified view on all of your datasets, but then under the hood it can be Parquet/Iceberg for BI data and Lance for your AI data. And then Lance itself plugs in natively to Spark and pandas and Polars and DuckDB and any sort of open data tooling that you’re already used to.
09.31
So operationally then, Chang, if I’m a data architect, should I think of Lance as, “OK, so I have Parquet and these table formats like Delta and Iceberg for my structured data. And then if it’s nonstructured, which could mean video, audio, and also text, right? So then I have to bring in this other format, Lance.” Is that operationally what happens in practice?
10.07
Yeah, often what the data infra folks and data engineers we talk to interact with is the tooling, right? So they’re looking at their data pipelines, they’re looking at maybe their Spark jobs or their search applications, and then those are the jobs that actually interact with the underlying storage, for example. And so instead of. . .
And that data transfer process is actually really easy through Apache Arrow. And most of the time, it’s really just one line of code change. It’s the same Spark code, for example. Instead of writing to Parquet, you’re writing to Lance. And it simplifies your overall data pipeline by bringing all of your tabular data and metadata along with your multimodal data all in the same place and also embeddings.
11.05
And then in terms of workload, you alluded to the fact that the previous-generation vector source, they excelled at something very specific, maybe retrieval. So is Lance equally specialized in the sense that, “All right, Lance is great for X, and X might be, I don’t know, analytics, but it doesn’t excel in other things”? Describe the kinds of workloads that teams that are using Lance are using.
11.39
So very high-level, the summary is LanceDB, our enterprise data platform, excels at helping our customers manage really large-scale AI data. So embeddings for search, adding new, adding new features and extracting new, new columns, enriching their dataset, doing data curation and exploration, and then feeding that to GPUs really quickly for distributed training jobs so that they can get as high GPU utilization and as high auto-flops utilization as they can.
12.20
You’ve used the word multimodal a few times, and I’ve always been a proponent of people really making sure that their data infrastructure is positioned for this multimodal world. But sometimes I question this assumption in the following sense, right? Is multimodality a Bay Area bubble thing? In other words, if I go to the East Coast and talk to, I don’t know, Goldman Sachs or an insurance company, are they still grappling with legacy systems that are mostly structured data? What they want to do is be able to do all this fancy AI stuff now with agents, but still using the old-school data that they have.
13.12
I think when we talk about multimodal data, a lot of times what comes to mind first is video generation, image generation, all of those. Self-driving cars. . . So there’s a lot of high-tech, cutting-edge applications that are multimodal. But I think if you look at more traditional enterprises, they already have a lot of multimodal data.
So you just mentioned insurance: They have millions of documents and PDFs and contracts lying around. Insurance especially will have top-down views of houses and boundaries so that they can figure out and assess risk a little bit better. The way I think about it is before AI, it’s just really hard to get value out of that data. They just really haven’t paid as much attention.
So it’s kind of like when I clean up my house, what I like to do is just like move all the mess into a back room or storage. And so then I don’t have to think about it, right? My wife yells at me all the time. She opens up the storage and everything kind of falls out. And so I feel like with multimodal data, this is kind of what traditional enterprises have done: They didn’t know what to do with it. They stuck it in some directory in SharePoint or something like that and kind of just like leave it there for storage. But there’s actually a tremendous amount of value and AI is helping them unlock all of that. So I think in the next few years, especially, we’re going to see a lot more attention paid to, “If we can get a lot more value out of this data, how do we actually manage it? How do we work with it? And how do we combine it with the rest of our data stack so that it’s governed within a single entity?”
15.06
The hot thing a few years ago in data infrastructure was the lakehouse, right? Great term we introduced. [laughs]
15.18
I wonder who came up with that one. [laughs]
15.22
Yeah. So you folks are starting to use the term multimodal lakehouse. So compare the status of the lakehouse. . . [The term] is I think now widely used, right? And then now you’re introducing the multimodal lakehouse. So where is the multimodal lakehouse now kind of mature, and where does it still need to do some work?
15.50
Just for the audience who’s not as familiar, the really, really simplified way I think about just a lakehouse is you have all your data in one place in the data lake, and then you have a combined data warehousing layer on top that provides structure, tables, and structured ways to run workloads on all of that data.
Now, the way we think about multimodal lakehouse is in a couple of different ways. One, the data changes so that you go from purely tabular data or maybe like clickstream data to now all sorts of multimodal data. So from embeddings to all of your multimedia types. So that changes a lot about how you can read and write data efficiently, how you manage that, how you synchronize that with metadata.
Number two is the workloads also are multimodal. You’re not just thinking about running SQL and analytics workloads. You’re now thinking about search. Now you’re thinking about training. Now you’re thinking about feature engineering and “How does your lakehouse interact with GPU clusters?” and all of those things that traditional lakehouses are not very good at.
And then I think the third layer, where the meaning “multimodal” comes in, is traditional lakehouses tend to be good only at batch offline processing. And then if you want to do serving, online processing, you probably need to introduce a sort of an OLTP kind of database or some system that’s primarily for serving. Well, with LanceDB, because of the innovations in the format, you can actually do both at the same time. So the online-offline scenario can also become multimodal in this sense.
17.44
So if I understand what you’re saying, you’re multimodal in multiple senses. So multimodal data types, multimodal workloads, and multimodal kinds of operations. So right now, in the Databricks world, they have—I don’t think they used the word multimodal. If anything, they go back to that HTAP kind of thing, so [a] hybrid transactional analytics kind of processing engine. I think through an acquisition, now they are very good at Postgres. I forget what they call this. [Chang: A lakebase.] So they have the transactions, and they have the analytics. So what you’re saying is that your vision of the multimodal lakehouse has that hybrid transactional analytics, multimodal types of data, and then multimodal workloads. Is that a fair summation? Surely, Chang, certain aspects of what you just described are more fleshed out than others, right? So what areas do you anticipate you folks will be working on hard, in terms of multiple notions of multimodality?
19.16
Number one is actually scale. Scale is actually the biggest driving factor late last year and this year. And a lot of that has been the rise of agents. Because of the rise of agents, data volume and scale, query throughput and scale, and performance and latency requirements, all of those things have just kind of been exploded. And that’s the thing that we find we’re uniquely suited for. And that’s something that we’re pushing a lot on. Oftentimes when we talk to customers, really what we think about is like, trillion is new billion. And we have folks who probably are operating at a thousand times the scale that they were just a year ago or two years ago.
20.22
I guess the hack that people will do for some of these things, Chang, is just let’s put the files in S3 and then use a database somehow. So are you still seeing a lot of people kind of try to do this?
20.39
Yeah, I mean, I think there are a few attempts that [are] doing that. And I think there’s generally a trend because of the data scale, like object storage is kind of the only sort of cost effective and scalable storage backend for a lot of these newer data storage systems. I think where the challenge lies for data infrastructure providers is “How do you actually have scalability and high performance and maintain the cost advantages of S3 and object store?” That is, I think, the difficult challenge. And so we actually have a recent blog article talking about how we do that at 10 billion-vector scale.
At smaller scales, that’s actually really easy. You just slurp up all the data from S3 into some caching system. You can serve it from there in any in-memory system. That’s a really easy problem. There’s tons of open source projects, Lance, for example, that can help you do that pretty effectively. And then the challenge is really at scale. If you have 10 billion vectors, pretty much, your only cost-effective solution is to store that on object storage. Then, you know, imagine the query times if you were just targeting S3 directly. So then indexing challenges and search and caching and all of that, that becomes a big distributed systems problem. So that’s what we solve.
22.16
Like you said, many data engineering and data infrastructure teams are trying to think through, “So what does our infrastructure look like in a world of agents?” right? So imagine—this isn’t happening yet—the equivalent of OpenClaw in enterprise, where a single employee might have 10 of these AI delegates or AI assistants. Some of the things that come up: One, identity management, so access control, identity management. Secondly, maybe some of these AI agents and AI delegates don’t really need anything permanent. They just want something ephemeral. So stand up a LanceDB for a minute and then make it go away. Are these some of the things that you are starting to think of?
23.14
Yeah, so for our cutting-edge customers, that’s already the reality. We specialize a lot in infrastructure for model training, for example. So if you think about features, like a researcher might have, “Hey, I have a feature idea. There’s two input features, each with 10 variants. And then I have some output feature that combines the two.” Well, now I’ve got 100 different variants. So before, there was a limited [number] of variants that I can test as an individual researcher manually. But now I can use agents to run all of that automatically. And I can just go to sleep and it’ll run. Well, now humans can go to sleep, but then the agents are presenting a lot of load on the underlying data infrastructure. This year we’re talking about going from hundreds of queries per second from plain RAG a couple of years ago to a hundred thousand queries per second in this land of agents.
And then when it comes to security and compliance, there’s a lot of churn in the stack about sandboxing and ephemeral systems. And when we talk about object storage, this is actually a big, even a bigger challenge, right? So if your source of truth is on object store, that’s actually the only way you can make this ephemeral workload work out well so that when you have hot data, you cache it, you serve it for a time, and then that can go away. And then the cache can expire it [to] be replaced by the next hot workload. And you can do that without having to pay for really expensive memory and NVMe for all of your data.
25.04
So the other thing, Chang, that comes up with agents right now, the hot thing that it seems like there’s a gazillion people working on is this notion of memory. So I guess my question to you is, if I have a bunch of agents and then I have a multimodal lakehouse. . . I have a lakehouse and now I have memories. So I have three different systems that I have to maintain. What’s your what’s your guys’ take in terms of agent memory?
25.42
LanceDB open source is actually the main memory plug-in for OpenClaw and a number of other agents like Crew AI, for example. And for a lot of these agent frameworks and harnesses, there’s a couple of different requirements. Number one is just lightweight, super easy to use. LanceDB is the only one where it supports hybrid search; it supports reranking, all these fairly sophisticated retrieval mechanisms, without having to maintain a service.
26.20
Before you continue. . . All right, so this notion of lightweight, right? On the one hand, there’s the notion of multimodal lakehouse and a lakehouse is never lightweight, right? But then, it seems like you folks are positioning yourself also in the DuckDB kind of very lightweight SQLite world. Can you clarify what you mean by lightweight when you are supposedly a lakehouse, right?
26.49
So what I mean by lightweight here is that if you think about it from an agent perspective, it simplifies a lot of things if you don’t have to connect to another service and talk to another system in order to get access to your memory and to retrieve from memory. So that’s what I mean. So the open source, the. . .
27.15
But then you’re large-scale infrastructure. . . So then if I’m a lightweight agent, how can you… This is where I guess I’m a bit confused. Can you clarify, why am I bringing along a big piece of infrastructure if I’m a lightweight agent?
27.37
Right. LanceDB open source is actually very lightweight. So there’s no heavy infrastructure involved. This is why it’s perfect for memory. Because a lot of times, memory is very ephemeral. So you just interact with a session and then when that session is gone, you want to retain all of that. At most you might want to compress some of it and then retain it for downstream historical processing. But most of the time, it’s just gone. You don’t have to think about it. And so that’s what I mean by lightweight. So there’s a version of that.
And then for large-scale retrieval, you have a large historical corpus, if you’re working in a corporate environment, if you have an agent that’s searching through patent history or something like that, right? And then that’s where the infrastructure comes in. Well, if I have a petabyte of data out there that I need to search through, the embedded library is not going to do. So you need to have a scalable system out there, but it needs to be easy to use. And from an agent perspective, it’s the same interface. So from the agent perspective, it’s just as easy, but there is a scalable system for that large amount of data that’s kind of hidden beneath the surface there.
I think for agents, that’s sort of just one of the requirements. The other one is having more sophisticated retrieval so that agents can find what they’re looking for. And different agents will want to look for data in different ways. So being able to support all of that without having like a million different plug-ins to do each modality, I think that’s also something very important for agents as well.
29.28
By the way, I was playing devil’s advocate there because I actually use LanceDB every day on my laptop. It can be something that you can use in your laptop just in-memory.
29.42
Yeah. So I think what we find is that when you make it really easy for agents to actually use it, that’s when scale really takes off. The way we’re looking at it is agents are kind of like an ideal gas that if you make it easy for them to use, no matter how much compute you have, no matter how much data and infrastructure you have, agents will expand to fill all of that that you have, right? So what we’ve seen is. . . We talked about growth and creep throughput. And then because of complex agents, there’s compression and latency. Your agents want a hundred-millisecond or like 20-millisecond latencies now. And then we also see a lot of proliferation of data.
One of the largest users in LanceDB told us they’re now managing something like a billion tables. Just because they have so many agents and so much data that they have to manage, like that number of tables within their system. Any computational and data management dimension you can think of, agents will expand to however much capacity you give them.
30.59
So this is a two-part question. Our listeners may not be aware, but for some reason, LanceDB kind of blew up a little more during the launch of OpenClaw. So I guess my two questions are one: How did this OpenClaw community land on Lance? And have you heard back from them, and have they told you what they liked about Lance?
31.32
Yeah, I mean, a lot of that is what we just talked about: It’s lightweight; it’s easy to use the model.
31.39
But how did it happen? How did they land on Lance? Do you know?
31.43
So my recollection was that originally it was a recommendation from Claude or something like that. And I think [Lance] was the only one out there that met the requirements, was embedded, lightweight, sophisticated retrieval. And it can do both in-memory on NVMe local and also on object store.
32.11
Interesting. So since then, has this kind of marriage [with OpenClaw] continued?
32.20
Yeah, we continue to see engagement from the open source community. Our open source continues to grow. I think at the latest, we’re at around 14 million downloads a month across our open source projects. And we’re super excited about working and supporting the open source community on that. What we see now is demand for a more filesystem-like interface. It’s easier for agents a lot of times to interact with a filesystem interface.
Now, I’m choosing my words carefully. I don’t mean a filesystem. I just mean an interface. This is something that we’re looking into—trying to see what it would look like to put a filesystem interface over a LanceDB or Lance format. Based on the usage patterns that we see from agents, this is fairly straightforward to do. So I think if you’re listening and this is something interesting, we’d love to have early users come check it out and test it out with us.
33.29
It’s interesting, actually, as you were talking there, it just dawned on me that this notion. . . These various notions of multimodality that you described earlier actually might be another reason why people landed on Lance. Because there are other vector search systems that you can run in-memory or embedded. If you want to build agents that are more capable moving forward, then the various notions of multimodality that Chang described earlier might come in handy, right?
34.06
Yeah, yeah, absolutely. I will say that like, I’m sort of a. . . There are AI maximalists. I’m sort of a multimodal maximalist. So my prediction is that in five years, multimodal won’t even be a word anymore. It’ll just be data, and it’ll just be multimodal by default. People will just say data, and it’ll be inclusive of all the different modalities. And when we think about data engineering, there won’t be multimodal data engineering. It’ll just be multimodal by default when we say data engineering.
34.37
Interesting, which actually. . . As we’re winding down here, I was going to ask you, If I’m a CxO or an architect at an enterprise, what data infrastructure decision do you think I should bear in mind? Or I guess to put it negatively, what are some of the decisions I can make right now that potentially can hurt my team moving forward in the next year?
35.08
Right, right. So I think we’re already. . . For a lot of early adopters, we see big pain points around new AI data silos. So one pattern, I wouldn’t call it an anti-pattern, but one I would say pain point is if you’re a CIO or CDO or something like that, chances are a lot of your teams within the enterprise have charged forward with their own AI applications and AI stack. And so now the centralized data platform team are faced with maybe like 10 different vector databases that they have to support and maybe five different ways to store the AI data, some in images and some just embeddings and others, many different modalities. So that becomes a big pain point going forward, right? So as companies go from “Let’s try out AI in this particular area” to, I guess, AI transformation, having large swaths of the enterprise be AI-assisted or AI-native, that becomes a big pain point.
I think if I were a CIO or a CEO or CTO at a larger enterprise, I would be looking forward a little bit to think about how do I set up all of my teams across the enterprise for success so that one, “How do I allow them to charge forward very quickly and iterate very quickly without presenting this crazy, untenable challenge on the central platform team?” So that’s what I would be thinking of. That’s actually. . . At LanceDB, that’s what we’re building for.
37.05
If your thesis is multimodal data matures over the next few years, and so do agents and everything that comes with agents, including memory, what does the data stack look like in a few years?
37.22
In broad strokes, the base layers are not going to change all that much. I think the infrastructure layer stays roughly the same. There’s going to be object storage. There’s going to be a storage layer. And then the compute layer will start to change.
37.49
Ray. [laughs]
37.52
What I think we’ll see is that the middle layer of data tooling will start to melt away a little bit because of agents.
38.04
Define data tooling.
38.07
I don’t want to name names, but I think there’s a lot of [what] I would call developer middleware for data where it’s neither the infrastructure layer nor is it the layer that’s interfacing with agents and users directly, right? That middle layer, I think will melt away a little bit or at least be very much refactored. So there’s going to be a lot of churn in that. It’s going to be interesting to see what shakes out. I think what will happen is that agents will continue to push that layer down, and agents will want to get as close to the base layer as possible.
If you look at this middle layer, there’s really two things that they’re providing. One is a precanned data model for how their users think about the problem, right? So they built that on top of the base infrastructure. So they would build that on top of LanceDB, for example. And then the other thing that they have in this middle tier right now is user interaction, right? The combination of the two is how they capture user workflows. And that’s the core of that. I think what happens in the future is that that UI workflow layer will largely go away and be replaced by agents.
But useful data models will still be useful, and they’ll still stay. Yes, you can have agents directly talk to random bits on S3, but why waste all that intelligence? It’s not worth the token cost. A well-formed data model is the right base layer for agents to interact with. And so I think that’s what we’ll see, is that melting away and reformatting of that middle layer. And I think this is something when I talk to data builders and AI infrastructure builders today, I think we’re all seeing that all at the same time.
40.22
What I describe to people right now as kind of the forward-looking stack has two main parts: So one, you have the multimodal lakehouse built around Lance, LanceDB, and the Lance format. And then you have the AI compute layer, which I call the PARK stack, so PyTorch, AI foundation models, Ray, and Kubernetes. So PARK stack here, and then your lakehouse will be around Lance and the Lance format. I see that quite a bit actually. I definitely see the PARK stack, PyTorch, Ray, Kubernetes. And now I’m starting to see more and more people talking about Lance and Lance format. Do you think of these as complementary or what?
41.16
Yeah, yeah, absolutely. I think we have close relationships with Ray and Spark and really like native-level integrations. And also PyTorch, right? I don’t think that’s going away. Those are either like. . . PyTorch is essentially interacting with developers directly, whereas Spark and Ray are very much infrastructure layer, so I don’t think those things are going anywhere. Kubernetes is definitely still around.
41.51
Yeah, yeah, yeah, yeah. And so what big trend are you paying attention to right now that we haven’t yet talked about? This is how we close.
42.08
What’s been really interesting that we didn’t talk about is the rise of open source models. And I think that’s going to have a big impact, maybe starting next year or even the remainder of this year. Enterprise AI. [Ben: Open weight.] Open-weight models. That’s correct. Yeah.
42.35
Who’s the source? Because right now the main source is China for the better ones. And I still see a lot of hesitation for enterprise teams to adopt such models. I actually just wrote a short post about this. Basically the perception seems to be that while the open-weight models from China are closing the gap, there is still a gap, and there’s structural reasons why there’s a gap. So one is the Chinese seem to be benchmaxxing. You know, they’re optimized for the benchmark, so not real workloads. And then secondly, there is a compute challenge, which makes iteration for them more challenging. So whereas the labs here may update their models every three or four months, the Chinese have to wait six months. And then finally, the data pipelines and the investment in data pipelines is just not the same as you would see at, for example, Gemini, Anthropic, and OpenAI. They’re licensing data from all over the place. The Chinese labs tend to do distillation, which means. . . When you’re doing distillation, your cap is basically the model you’re distilling from.
And then there’s the flywheel—OpenAI and Anthropic and Gemini have a lot of users, so therefore they get better as more users interact with them. . .
44.20
That’s right. Don’t forget the open-weight models in China are also. . . [cross-talk] Here’s the way I think about it, right? So I think as AI adoption grows exponentially within enterprises, they are going to be extremely motivated to invest in their own inference on open-weight models, right? Just because there’s such a drastic cost in tokens.
Because of that economic incentive, I think there’s going to be a lot more incentive for companies to create better open-weight models. If you look at the open-weight models in China, one, the fact that they can create open-weight models of this quality on really limited hardware is really telling. So a team in the US theoretically should be able to create much better quality open-weight models because of that.
Number two, I don’t think the distillation argument is actually true. If you look at the report that Anthropic threw out, right, like if you look at the numbers of how much distillation they accused DeepSeek of doing, it’s actually not that much. It’s basically negligible, right? Like MiniMax is a legit big offender, but DeepSeek, basically, didn’t really do that much. I don’t think distillation is a big factor in the quality of open-weight models anymore.
So then there is a remaining gap in quality. Maybe there’s a three- to four-month gap between open-weight models and SOTA. But what’s interesting is the experiments that people have done is, open-weight models, one, are cheaper, and they’re much faster. So if you have a coding agent task, you can do a one-shot with SOTA models or you can do multiple rounds and iterations on an open-weight model, which gets you the same quality, still lower total costs and tokens, and you finish around the same time, or you actually might finish faster. So then I think a lot of that is lack of familiarity and a skill gap, where if you have to do a few shots, that complexity is way more than what people want to think about right now.
So the pattern today is you go into production with SOTA models, then you reach some cost-prohibitive moment where you say, “OK, what are the areas where there’s not requirements for really heavy intelligence but still have a lot of token costs, and then I can replace [them] with open models?” And I think that will happen more and more across enterprises. So I think that’s going to be a big trend to watch this year and next.
47.18
And actually, as you mentioned, my conversations are a product of the fact of the stage of adoption, which is basically [the] early stage of adoption. I will deploy with state-of-the-art models because I’m early. And then as my agent or my application gets used, then I start paying attention to cost, latency, and all these. And then I can worry about swapping the models then. And hopefully, we will have some Western labs start cranking on open-weights models again, right? It seems like Meta is off the table. The Gemma folks produce models, but they’re meant for on-device, I think. Maybe there’s an opening there for someone to start up something that…
Especially as people become more clever in terms of training and tools like LanceDB make training more affordable somehow. We’ll see what happens. And with that, thank you, Chang.
48.24
That’s right. Thank you, Ben.
Generative AI in the Real World, Podcast
Radar
