Max Headroom 
UNCLASSIFIED
In this episode, Ben Lorica and Chris Butler, director of product operations for GitHub’s Synapse team, chat about the experimentation Chris is doing to incorporate generative AI into the product development process—particularly with the goal of reducing toil for cross-functional teams. It isn’t just automating busywork (although there’s some of that). He and his team have created agents that expose the right information at the right time, use feedback in meetings to develop “straw man” prototypes for the team to react to, and even offer critiques from specific perspectives (a CPO agent?). Very interesting stuff.
About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.
Check out other episodes of this podcast on the O’Reilly learning platform.
Transcript
This transcript was created with the help of AI and has been lightly edited for clarity.
00.00: Today we have Chris Butler of GitHub, where he leads a team called the Synapse. Welcome to the podcast, Chris.
00.15: Thank you. Yeah. Synapse is actually part of our product team and what we call EPD operations, which is engineering, product, and design. And our team is mostly engineers. I’m the product lead for it, but we help solve and reduce toil for these cross-functional teams inside of GitHub, mostly building internal tooling, with the focus on process automation and AI. But we also have a speculative part of our practice as well: trying to imagine the future of cross-functional teams working together and how they might do that with agents, for example.
00.45: Actually, you are the first person I’ve come across who’s used the word “toil.” Usually “tedium” is what people use, in terms of describing the parts of their job that they would rather automate. So you’re actually a big proponent of talking about agents that go beyond coding agents.
01.03: Yeah. That’s right.
01.05: And specifically in your context for product people.
01.09: And actually, for just the way that, say, product people work with their cross-functional teams. But I would also include other types of functions, legal privacy and customer support docs, any of these people that are working to actually help build a product; I think there needs to be a transformation of the way we think about these tools.
01.29: GitHub is a very engineering-led organization as well as a very engineering-focused organization. But my role is to really think about “How do we do a better job between all these people that I would call nontechnical—but they are sometimes technical, of course, but the people that are not necessarily there to write code. . . How do we actually work together to build great products?” And so that’s what I think about work.
01.48: For people who aren’t familiar with product management and product teams, what’s toil in the context of product teams?
02.00: So toil is actually something that I stole from a Google SRE from the standpoint of any type of thing that someone has to do that is manual, tactical, repetitive. . . It usually doesn’t really add to the value of the product in any way. It’s something that as the team gets bigger or the product goes down the SDLC or lifecycle, it scales linearly, with the fact that you’re building bigger and bigger things. And so it’s usually something that we want to try to cut out, because not only is it potentially a waste of time, but there’s also a perception within the team it can cause burnout.
02.35: If I have to constantly be doing toilsome parts of my work, I feel I’m doing things that don’t really matter rather than focusing on the things that really matter. And what I would argue is especially for product managers and cross-functional teams, a lot of the time that is processes that they have to use, usually to share information within larger organizations.
02.54: A good example of that is status reporting. Status reporting is one of those things where people will spend anywhere from 30 minutes to hours per week. And sometimes it’s in certain parts of the team—technical product managers, product managers, engineering managers, program managers are all dealing with this aspect that they have to in some way summarize the work that the team is doing and then shar[e] that not only with their leadership. . . They want to build trust with their leadership, that they’re making the right decisions, that they’re making the right calls. They’re able to escalate when they need help. But also then to convey information to other teams that are dependent on them or they’re dependent on. Again, this is [in] very large organizations, [where] there’s a huge cost to communication flows.
03.35: And so that’s why I use status reporting as a good example of that. Now with the use of the things like LLMs, especially if we think about our LLMs as a compression engine or a translation engine, we can then start to use these tools inside of these processes around status reporting to make it less toilsome. But there’s still aspects of it that we want to keep that are really about humans understanding, making decisions, things like that.
03:59: And this is key. So one of the concerns that people have is about a hollowing out in the following context: If you eliminate toil in general, the problem there is that your most junior or entry-level employees actually learn about the culture of the organization by doing toil. There’s some level of toil that becomes part of the onboarding in the acculturation of young employees. But on the other hand, this is a challenge for organizations to just change how they onboard new employees and what kinds of tasks they give them and how they learn more about the culture of the organization.
04.51: I would differentiate between the idea of toil and paying your dues within the organization. In investment banking, there’s a whole concern about that: “They just need to sit in the office for 12 hours a day to really get the culture here.” And I would differentiate that from. . .
05:04: Or “Get this slide to pitch decks and make sure all the fonts are the right fonts.”
05.11: That’s right. Yeah, I worked at Facebook Reality Labs, and there were many times where we would do a Zuck review, and getting those slides perfect was a huge task for the team. What I would say is I want to differentiate this from the gaining of expertise. So if we think about Gary Klein, naturalistic decision making, real expertise is actually about being able to see an environment. And that could be a data environment [or] information environment as well. And then as you gain expertise, you’re able to discern between important signals and noise. And so what I’m not advocating for is to remove the ability to gain that expertise. But I am saying that toilsome work doesn’t necessarily contribute to expertise.
05.49: In the case of status reporting as an example—status reporting is very valuable for a person to be able to understand what is going on with the team, and then, “What actions do I need to take?” And we don’t want to remove that. But the idea that a TPM or product manager or EM has to dig through all of the different issues that are inside of a particular repo to look for specific updates and then do their own synthesis of a draft, I think there is a difference there. And so what I would say is that the idea of me reading this information in a way that is very convenient for me to consume and then to be able to shape the signal that I then put out into the organization as a status report, that is still very much a human decision.
06.30: And I think that’s where we can start to use tools. Ethan Mollick has talked about this a lot in the way that he’s trying to approach including LLMs in, say, the classroom. There’s two patterns that I think could come out of this. One is that when I have some type of early draft of something, I should be able to get a lot of early feedback that is very low reputational risk. And what I mean by that is that a bot can tell me “Hey, this is not written in a way with the active voice” or “[This] is not really talking about the impact of this on the organization.” And so I can get that super early feedback in a way that is not going to hurt me.
If I publish a really bad status report, people may think less of me inside the organization. But using a bot or an agent or just a prompt to even just say, “Hey, these are the ways you can improve this”—that type of early feedback is really, really valuable. That I have a draft and I get critique from a bunch of different viewpoints I think is super valuable and will build expertise.
07.24: And then there’s the other side, which is, when we talk about consuming lots of information and then synthesizing or translating it into a draft, I can then critique “Is this actually valuable to the way that I think that this leader thinks? Or what I’m trying to convey as an impact?” And so then I am critiquing the straw man that is output by these prompts and agents.
07.46: Those two different patterns together actually create a really great loop for me to be able to learn not only from agents but also from the standpoint of seeing how. . . The part that ends up being really exciting is when once you start to connect the way communication happens inside the organization, I can then see what my leaders passed on to the next leader or what this person interpreted this as. And I can use that as a feedback loop to then improve, over time, my expertise in, say, writing a status report that is shaped for the leader. There’s also a whole thing that when we talk about status reporting in particular, there is a difference in expertise that people are getting that I’m not always 100%. . .
08.21: It’s valuable for me to understand how my leader thinks and makes decisions. I think that is very valuable. But the idea that I will spend hours and hours shaping and formulating a status report from my point of view for someone else can be aided by these types of systems. And so status should not be about the speaker’s mouth; it should be at the listener’s ear.
For these leaders, they want to be able to understand “Are the teams making the right decisions? Do I trust them? And then where should I preemptively intervene because of my experience or maybe my understanding of the context in the broader organization?” And so that’s what I would say: These tools are very valuable in helping build that expertise.
09.00: It’s just that we have to rethink “What is expertise?” And I just don’t buy it that paying your dues is the way you gain expertise. You do sometimes. Absolutely. But a lot of it is also just busy work and toil.
09.11: My thing is these are productivity tools. And so you make even your junior employees productive—you just change the way you use your more-junior employees.
09.24: Maybe just one thing to add to this is that there is something really interesting inside of the education world of using LLMs: trying to understand where someone is at. And so the type of feedback that someone that is very early in their career or first to doing something is potentially very different in the way that you’re teaching them or giving them feedback versus something that someone that is much further in expertise, they want to be able to just get down to “What are some things I’m missing here? Where am I biased?” Those are things where I think we also need to do a better job for those early employees, the people that are just starting to get expertise—“How do we train them using these tools as well as other ways?”
10.01: And I’ve done that as well. I do a lot of learning and development help, internal to companies, and I did that as part of the PM faculty for learning in development at Google. And so thinking a lot about how PMs gain expertise, I think we’re doing a real disservice to making it so that product manager as a junior position is so hard to get.
10.18: I think it’s really bad because, right out of college, I started doing program management, and it taught me so much about this. But at Microsoft, when I joined, we would say that the program manager wasn’t really worth very much for the first two years, right? Because they’re gaining expertise in this.
And so I think LLMs can help give the ability for people to gain expertise faster and also help them from avoiding making errors that other people might make. But I think there’s a lot to do with just learning and development in general that we need to pair with LLMs and human systems.
10.52: In terms of agents, I guess agents for product management, first of all, do they exist? And if they do, I always like to look at what level of autonomy they really have. Most agents really are still partially autonomous, right? There’s still a human in the loop. And so the question is “How much is the human in the loop?” It’s kind of like a self-driving car. There’s driver assists, and then there’s all the way to self-driving. A lot of the agents right now are “driver assist.”
11.28: I think you’re right. That’s why I don’t always use the term “agent,” because it’s not an autonomous system that is storing memory using tools, constantly operating.
I would argue though that there is no such thing as “human out of the loop.” We’re probably just drawing the system diagram wrong if we’re saying that there’s no human that’s involved in some way. That’s the first thing.
11.53: The second thing I’d say is that I think you’re right. A lot of the time right now, it ends up being when the human needs the help, we end up creating systems inside of GitHub; we have something that’s called GitHub spaces, which is really a custom GPT. It’s really just a bundling of context that I can then go to when I need help with a particular type of thing. We built very highly specific types of copilot spaces, like “I need to write a blog announcement about something. And so what’s the GitHub writing style? How should I be wording this avoiding jargon?” Internal things like that. So it can be highly specific.
We also have more general tools that are kind of like “How do I form and maintain initiatives throughout the entire software development lifecycle? When do I need certain types of feedback? When do I need to generate the 12 to 14 different documents that compliance and downstream teams need?” And so those tend to be operating in the background to autodraft these things based on the context that’s available. And so that’s I’d say that’s semiagentic, to a certain extent.
12.52: But I think actually there’s really big opportunities when it comes to. . . One of the cases that we’re working on right now is actually linking information in the GitHub graph that is not commonly linked. And so a key example of that might be kicking off all of the process that goes along with doing a release.
When I first get started, I actually want to know in our customer feedback repo, in all the different places where we store customer feedback, “Where are there times that customers actually asked about this or complained about it or had some information about this?” And so when I get started, being able to automatically link something like a release tracking issue with all of this customer feedback becomes really valuable. But it’s very hard for me as an individual to do that. And what we really want—and what we’re building—[are] things that are more and more autonomous about constantly searching for feedback or information that we can then connect to this release tracking issue.
13.44: So that’s why I say we’re starting to get into the autonomous realm when it comes to this idea of something going around looking for linkages that don’t exist today. And so that’s one of those things, because again, we’re talking about information flow. And a lot of the time, especially in organizations the size of GitHub, there’s lots of siloing that takes place.
We have lots of repos. We have lots of information. And so it’s really hard for a single person to ever keep all of that in their head and to know where to go, and so [we’re] bringing all of that into the tools that they end up using.
14.14: So for example, we’ve also created internal things—these are more assist-type use cases—but the idea of a Gemini Gem inside of a Google doc or an M365 agent inside of Word that is then also connected to the GitHub graph in some way. I think it’s “When do we expose this information? Is it always happening in the background, or is it only when I’m drafting the next version of this initiative that ends up becoming really, really important?”
14.41: Some of the work we’ve been experimenting with is actually “How do we start to include agents inside of the synchronous meetings that we actually do?” You probably don’t want an agent to suddenly start speaking, especially because there’s lots of different agents that you may want to have in a meeting.
We don’t have a designer on our team, so I actually end up using an agent that is prompted to be like a designer and think like a designer inside of these meetings. And so we probably don’t want them to speak up dynamically inside the meeting, but we do want them to add information if it’s helpful.
We want to autoprototype things as a straw man for us to be able to react to. We want to start to use our planning agents and stuff like that to help us plan out “What is the work that might need to take place?” It’s a lot of experimentation about “How do we actually pull things into the places that humans are doing the work?”—which is usually synchronous meetings, some types of asynchronous communication like Teams or Slack, things like that.
15.32: So that’s where I’d say the full possibility [is] for, say, a PM. And our customers are also TPMs and leaders and people like that. It really has to do with “How are we linking synchronous and asynchronous conversations with all of this information that is out there in the ecosystem of our organization that we don’t know about yet, or viewpoints that we don’t have that we need to have in this conversation?”
15.55: You mentioned the notion of a design agent passively in the background, attending a meeting. This is fascinating. So this design agent, what is it? Is it a fine-tuned agent or. . .? What exactly makes it a design agent?
16.13: In this particular case, it’s a specific prompt that defines what a designer would usually do in a cross-functional team and what they might ask questions about, what they would want clarification of. . .
16.26: Completely reliant on the pretrained foundation model—no posttraining, no RAG, nothing?
16.32: No, no. [Everything is in the prompt] at this point.
16.36: How big is this prompt?
16.37: It’s not that big. I’d say it’s maybe at most 50 lines, something like that. It’s pretty small. The truth is, the idea of a designer is something that LLMs know about. But more for our specific case, right now it’s really just based on this live conversation. And there’s a lot of papercuts in the way that we have to do a site call, pull a live transcript, put it into a space, and [then] I have a bunch of different agents that are inside the space that will then pipe up when they have something interesting to say, essentially.
And it’s a little weird because I have to share my screen and people have to read it, hold the meeting. So it’s clunky right now in the way that we bring this in. But what it will bring up is “Hey, these are patterns inside of design that you may want to think about.” Or you know, “For this particular part of the experience, it’s still pretty ambiguous. Do you want to define more about what this part of the process is?” And we’ve also included legal, privacy, data-oriented groups. Even the idea of a facilitator agent saying that we were getting off track or we have these other things to discuss, that type of stuff. So again, these are really rudimentary right now.
17.37: Now, what I could imagine though is, we have a design system inside of GitHub. How might we start to use that design system and use internal prototyping tools to autogenerate possibilities for what we’re talking about? And I guess when I think about using prototyping as a PM, I don’t think the PMs should be vibe coding everything.
I don’t think the prototype replaces a lot of the cross-functional documents that we have today. But I think what it does increase is that if we have been talking about a feature for about 30 minutes, that is a lot of interesting context that if we can say, “Autogenerate three different prototypes that are coming from slightly different directions, slightly different places that we might integrate inside of our current product,” I think what it does is it gives us, again, that straw man for us to be able to critique, which will then uncover additional assumptions, additional values, additional principles that we maybe haven’t written down somewhere else.
18.32: And so I see that as super valuable. And that’s the thing that we end up doing—we’ll use an internal product for prototyping to just take that and then have it autogenerated. It takes a little while right now, you know, a couple minutes to do a prototype generation. And so in those cases we’ll just [say], “Here’s what we thought about so far. Just give us a prototype.” And again it doesn’t always do the right thing, but at least it gives us something to now talk about because it’s more real now. It is not the thing that we end up implementing, but it is the thing that we end up talking about.
18.59: By the way, this notion of an agent attending synchronous some meeting, you can imagine taking it to the next level, which is to take advantage of multimodal models. The agent can then absorb speech and maybe visual cues, so then basically when the agent suggests something and someone reacts with a frown. . .
19.25: I think there’s something really interesting about that. And when you talk about multimodal, I do think that one of the things that is really important about human communication is the way that we pick up cues from each other—if we think about it, the reason why we actually talk to each other. . . And there’s a great book called The Enigma of Reason that’s all about this.
But their hypothesis is that, yes, we can try to logic or pretend to logic inside of our own heads, but we actually do a lot of post hoc analysis. So we come up with an idea inside our head. We have some certainty around it, some intuition, and then we fit it to why we thought about this. So that’s what we do internally.
But when you and I are talking, I’m actually trying to read your mind in some way. I’m trying to understand the norms that are at play. And I’m using your facial expression. I’m using your tone of voice. I’m using what you’re saying—actually way less of what you’re saying and more your facial expression and your tone of voice—to determine what’s going on.
20.16: And so I think this idea of engagement with these tools and the way these tools work, I think [of] the idea of gaze tracking: What are people looking at? What are people talking about? How are people reacting to this? And then I think this is where in the future, in some of the early prototypes we built internally for what the synchronous meeting would look like, we have it where the agent is raising its hand and saying, “Here’s an issue that we may want to discuss.” If the people want to discuss it, they can discuss it, or they can ignore it.
20.41: Longer term, we have to start to think about how agents are fitting into the turn-taking of conversation with the rest of the group. And using all of these multimodal cues ends up being very interesting, because you wouldn’t want just an agent whenever it thinks of something to just blurt it out.
20.59: And so there’s a lot of work to do here, but I think there’s something really exciting about just using engagement as the meaning to understand what are the hot topics, but also trying to help detect “Are we rat-holing on something that should be put in the parking lot?” Those are things and cues that we can start to get from these systems as well.
21.16: By the way, context has multiple dimensions. So you can imagine in a meeting between the two of us, you outrank me. You’re my manager. But then it turns out the agent realizes, “Well, actually, looking through the data in the company, Ben knows more about this topic than Chris. So maybe when I start absorbing their input, I should weigh Ben’s, even though in the org chart Chris outranks Ben.”
21.46: A related story is one of the things I’ve created inside of a copilot space is actually a proxy for our CPO. And so what I’ve done is I’ve taken meetings that he’s done where he asked questions in a smaller setting, taking his writing samples and things that, and I’ve tried to turn it into a, not really an agent, but a space where I can say, “Here’s what I’m thinking about for this plan. And what would Mario [Rodriguez] potentially think about this?”
It’s definitely not 100% accurate in any way. Mario’s an individual that is constantly changing and is learning and has intuitions that he doesn’t say out loud, but it is interesting how it does sound like him. It does seem to focus on questions that he would bring up in a previous meeting based on the context that we provided. And so I think to your point, a lot of things that right now are said inside of meetings that we then don’t use to actually help understand people’s points of view in a deeper way.
22.40: You could imagine that this proxy also could be used for [determining] potential blind spots for Mario that, as a person that is working on this, I may need to deal with, in the sense that maybe he’s not always focused on this type of issue, but I think it’s a really big deal. So how do I help him actually understand what’s going on?
22.57: And this gets back to that reporting: Is that the listener’s ear? What does that person actually care about? What do they need to know about to build trust with the team? What do they need to take action on? Those are things that I think we can start to build interesting profiles.
There’s a really interesting ethical question, which is: Should that person be able to write their own proxy? Would it include the blind spots that they have or not? And then maybe compare this to—you know, there’s [been] a trend for a little while where every leader would write their own user manual or readme, and inside of those things, they tend to be a bit more performative. It’s more about how they idealize their behavior versus the way that they actually are.
23.37: And so there’s some interesting problems that start to come up when we’re doing proxying. I don’t call it a digital twin of a person, because digital twins to me are basically simulations of mechanical things. But to me it’s “What is this proxy that might sit in this meeting to help give us a perspective and maybe even identify when this is something we should escalate to that person?”
23.55: I think there’s lots of very interesting things. Power structures inside of the organization are really hard to discern because there’s both, to your point, hierarchical ones that are very set in the systems that are there, but there’s also unsaid ones.
I mean, one funny story is Ray Dalio did try to implement this inside of his hedge fund. And unfortunately, I guess, for him, there were two people that were considered to be higher ranking in reputation than him. But then he changed the system so that he was ranked number one. So I guess we have to worry about this type of thing for these proxies as well.
24.27: One of the reasons why coding is such a great playground for these things is one, you can validate the result. But secondly, the data is quite tame and relatively right. So you have version control systems GitHub—you can look through that and say, “Hey, actually Ben’s commits are much more valuable than Chris’s commits.” Or “Ben is the one who suggested all of these changes before, and they were all accepted. So maybe we should really take Ben’s opinion much more strong[ly].” I don’t know what artifacts you have in the product management space that can help develop this reputation score.
25.09: Yeah. It’s tough because a reputation score, especially once you start to monitor some type of metric and it becomes the goal, that’s where we get into problems. For example, Agile teams adopting velocity as a metric: It’s meant to be an internal metric that helps us understand “If this person is out, how does that adjust what type of work we need to do?” But then comparing velocities between different teams ends up creating a whole can of worms around “Is this actually the metric that we’re trying to optimize for?”
25.37: And even when it comes to product management, what I would say is actually valuable a lot of the time is “Does the team understand why they’re working on something? How does it link to the broader strategy? How does this solve both business and customer needs? And then how are we wrangling this uncertainty of the world?”
I would argue that a really key meta skill for product managers—and for other people like generative user researchers, business development people, you know, even leaders inside the organization—they have to deal with a lot of uncertainty. And it’s not that we need to shut down the uncertainty, because actually uncertainty is an advantage that we should take advantage of and something we should use in some way. But there are places where we need to be able to build enough certainty for the team to do their work and then make plans that are resilient in the future uncertainty.
26.24: And then finally, the ability to communicate what the team is doing and why it’s important is very valuable. Unfortunately, there’s not a lot of. . . Maybe there’s rubrics we can build. And that’s actually what career ladders try to do for product managers. But they tend to be very vague actually. And as you get more senior inside of a product manager organization, you start to see things—it’s really just broader views, more complexity. That’s really what we start to judge product managers on. Because of that fact, it’s really about “How are you working across the team?”
26.55: There will be cases, though, that we can start to say, “Is this thing thought out well enough at first, at least for the team to be able to take action?” And then linking that work as a team to outcomes ends up being something that we can apply more and more data rigor to. But I worry about it being “This initiative brief was perfect, and so that meant the success of the product,” when the reality was that was maybe the starting point, but there was all this other stuff that the product manager and the team was doing together. So I’m always wary of that. And that’s where performance management for PMs is actually pretty hard: where you have to base most of your understanding on how they work with the other teammates inside their team.
27.35: You’ve been in product for a long time so you have a lot of you have a network of peers in other companies, right? What are one or two examples of the use of AI—not in GitHub—in the product management context that you admire?
27.53: For a lot of the people that I know that are inside of startups that are basically using prototyping tools to build out their initial product, I have a lot of, not necessarily envy, but I respect that a lot because you have to be so scrappy inside of a startup, and you’re really there to not only prove something to a customer, or actually not even prove something, but get validation from customers that you’re building the right thing. And so I think that type of rapid prototyping is something that is super valuable for that stage of an organization.
28.26: When I start to then look at larger enterprises, what I do see that I think is not as well a help with these prototyping tools is what we’ll call brownfield development: We need to build something on top of this other thing. It’s actually hard to use these tools today to imagine new things inside of a current ecosystem or a current design system.
28.46: [For] a lot of the teams that are in other places, it really is a struggle to get access to some of these tools. The thing that’s holding back the biggest enterprises from actually doing interesting work in this area is they’re overconstraining what their engineers [and] product managers can use as far as these tools.
And so what’s actually being created is shadow systems, where the person is using their personal ChatGPT to actually do the work rather than something that’s within the compliance of the organization.
29:18: Which is great for IP protection.
29:19: Exactly! That’s the problem, right? Some of this stuff, you do want to use the most current tools. Because there is actually not just [the] time savings aspect and toil reduction aspects—there’s also just the fact that it helps you think differently, especially if you’re an expert in your domain. It really aids you in becoming even better at what you’re doing. And then it also shores up some of your weaknesses. Those are the things that really expert people are using these types of tools for. But in the end, it comes down to a combination of legal, HR, and IT, and budgetary types of things too, that are holding back some of these organizations.
30.00: When I’m talking to other people inside of the orgs. . . Maybe another problem for enterprises right now is that a lot of these tools require lots of different context. We’ve benefited inside of GitHub in that a lot of our context is inside the GitHub graph, so Copilot can access it and use it. But for other teams they keep things and all of these individual vendor platforms.
And so the biggest problem then ends up being “How do we merge these different pieces of context in a way that is allowed?” When I first started working in the team of Synapse, I looked at the patterns that we were building and it was like “If we just had access to Zapier or Relay or something like that, that is exactly what we need right now.” Except we would not have any of the approvals for the connectors to all of these different systems. And so Airtable is a great example of something like that too: They’re building out process automation platforms that focus on data as well as connecting to other data sources, plus the idea of including LLMs as components inside these processes.
30.58: A really big issue I see for enterprises in general is the connectivity issue between all the datasets. And there are, of course, teams that are working on this—Glean or others that are trying to be more of an overall data copilot frontend for your entire enterprise datasets. But I just haven’t seen as much success in getting all these connected.
31.17: I think one of the things that people don’t realize is enterprise search is not turnkey. You have to get in there and really do all these integrations. There’s no shortcuts. There’s no, if a vendor comes to you and says, yeah, just use our system, it all magically works.
31.37: This is why we need to hire more people with degrees in library science, because they actually know how to manage these types of systems. Again, my first cutting my teeth on this was in very early versions of SharePoint a long time ago. And even inside there, there’s so much that you need to do to just help people with not only organization of the data but even just the search itself.
It’s not just a search index problem. It’s a bunch of different things. And that’s why whenever we’re shown an empty text box, that’s why there’s so much work that goes into just behind that; inside of Google, all of the instant answers, there’s lots of different ways that a particular search query is actually looked at, not just to go against the search index but to also just provide you the right information. And now they’re trying to include Gemini by default in there. The same thing happens within any copilot. There’s a million different things you could use.
32.27: And so I guess maybe this gets to my hypothesis about the way that agents will be valuable, either fully autonomous ones or ones that are attached to a particular process. But having many different agents that are highly biased in a particular way. And I use the term bias as in bias can be good, neutral, and bad, right? I don’t mean bias in a way of unfairness and that type of stuff; I mean more from the standpoint of “This agent is meant to represent this viewpoint, and it’s going to give you feedback from this viewpoint.” That ends up becoming really, really valuable because of that fact that you will not always be thinking about everything.
33.00: I’ve done a lot of work in adversarial thinking and red teaming and stuff like that. One of the things that is most valuable is to build prompts that are breaking the sycophancy of these different models that are there by default, because it should be about challenging my thinking rather than just agreeing with it.
And then the standpoint of each one of these highly biased agents actually helps provide a very interesting approach. I mean, if we go to things like meeting facilitation or workshop facilitation groups, this is why. . . I don’t know if you’re familiar with the six hats, but the six hats is a technique by which we declare inside of a meeting that I’m going to be the one that’s all positivity. This person’s going to be the one about data. This person’s gonna be the one that’s the adversarial, negative one, etc., etc. When you have all of these different viewpoints, you actually end up because of the tensions in the discussion of those ideas, the creation of options, the weighing of options, I think you end up making much better decisions. That’s where I think those highly biased viewpoints end up becoming really valuable.
34.00: For product people who are early in their career or want to enter the field, what are some resources that they should be looking at in terms of leveling up on the use AI in this context?
34.17: The first thing is there are millions of prompt libraries out there for product managers. What you should do is when you are creating work, you should be using a lot of these prompts to give you feedback, and you can actually even write your own, if you want to. But I would say there’s lots of material out there for “I need to write this thing.”
What is a way to [do something like] “I try to write it and then I get critique”? But then how might this AI system, through a prompt, generate a draft of this thing? And then I go in and look at it and say, “Which things are not actually quite right here?” And I think that again, those two patterns of getting critique and giving critique end up building a lot of expertise.
34.55: I think also within the organization itself, I believe an awful lot in things that are called basically “learning from your peers.” Being able to join small groups where you are getting feedback from your peers and including AI agent feedback inside of the small peer groups is very valuable.
There’s another technique, which is using case studies. And I actually, as part of my learning development practice, do something called “decision forcing cases” where we take a story that actually happened, we walk people through it and we ask them, “What do they think is happening; what would they do next?” But having that where you do those types of things across junior and senior people, you can start to actually learn the expertise from the senior people through these types of case studies.
35.37: I think there’s an awful lot more that senior leaders inside the organization should be doing. And as junior people inside your organization, you should be going to these senior leaders and saying, “How do you think about this? What is the way that you make these decisions?” Because what you’re actually pulling from is their past experience and expertise that they’ve gained to build that intuition.
35.53: There’s all sorts of surveys of programmers and engineers and AI. Are there surveys about product managers? Are they freaked out or what? What’s the state of adoption and this kind of thing?
36.00: Almost every PM that I’ve met has used an LLM in some way, to help them with their writing in particular. And if you look at the studies by ChatGPT or OpenAI about the use of ChatGPT, a lot of the writing tasks end up being from a product manager or senior leader standpoint. I think people are freaked out because every practice says that this other practice is going to be replaced because I can in some way replace them right now with a viewpoint.
36.38: I don’t think product management will go away. We may change the terminology that we end up using. But this idea of someone that is helping manage the complexity of the team, help with communication, help with [the] decision-making process inside that team is still very valuable and will be valuable even when we can start to autodraft a PRD.
I would argue that the draft of the PRD is not what matters. It’s actually the discussions that take place in the team after the PRD is created. And I don’t think that designers are going to take over the PM work because, yes, it is about to a certain extent the interaction patterns and the usability of things and the design and the feeling of things. But there’s all these other things that you need to worry about when it comes to matching it to business models, matching it to customer mindsets, deciding which problems to solve. They’re doing that.
37.27: There’s a lot of this concern about [how] every practice is saying this other practice is going to go away because of AI. I just don’t think that’s true. I just think we’re all going to be given different levels of abstraction to gain expertise on. But the core of what we do—an engineer focusing on what is maintainable and buildable and actually something that we want to work on versus the designer that’s building something usable and something that people will feel good using, and a product manager making sure that we’re actually building the thing that is best for the company and the user—those are things that will continue to exist even with these AI tools, prototyping tools, etc.
38.01: And for our listeners, as Chris mentioned, there’s many, many prompt templates for product managers. We’ll try to get Chris to recommend one, and we’ll put it in the episode notes. [See “Resources from Chris” below.] And with that thank you, Chris.
38.18: Thank you very much. Great to be here.
Resources from Chris
Here’s what Chris shared with us following the recording:
There are two [prompt resources for product managers] that I think people should check out:
However, I’d say that people should take these as a starting point and they should adapt them for their own needs. There is always going to be nuance for their roles, so they should look at how people do the prompting and modify for their own use. I tend to look at other people’s prompts and then write my own.
If they are thinking about using prompts frequently, I’d make a plug for Copilot Spaces to pull that context together.
Generative AI in the Real World, Podcast
Radar
