Max Headroom 
| This post first appeared on Aman Khan’s AI Product Playbook newsletter and is being republished here with the author’s permission. |
Let me start with some honesty. When people ask me “Should I become an AI PM?” I tell them they’re asking the wrong question.
Here’s what I’ve learned: Becoming an AI PM isn’t about chasing a trendy job title. It’s about developing concrete skills that make you more effective at building products in a world where AI touches everything.
Every PM is becoming an AI PM, whether they realize it or not. Your payment flow will have fraud detection. Your search bar will have semantic understanding. Your customer support will have chatbots.
Think of AI Product Managements as less of an OR and instead more of an AND. For example: AI x health tech PM or AI x fintech PM.
The Five Skills I Actually Use Every Day
| This post was adapted from a conversation with Aakash Gupta on The Growth Podcast. You can find the episode here. |
After ~9 years of building AI products (the last three of which have been a complete ramp-up using LLMs and agents), here are the skills I use constantly—not the ones that sound good in a blog post, but the ones I literally used yesterday.
- AI prototyping
- Observability, akin to telemetry
- AI evals: The New PRD for AI PMs
- RAG versus fine-tuning versus prompt engineering
- Working with AI engineers
1. Prototyping: Why I code every week
Last month, our design team spent two weeks creating beautiful mocks for an AI agent interface. It looked perfect. Then I spent 30 minutes in Cursor building a functional prototype, and we immediately discovered three fundamental UX problems the mocks hadn’t revealed.
The skill: Using AI-powered coding tools to build rough prototypes.
The tool: Cursor. (It’s VS Code but you can describe what you want in plain English.)
Why it matters: AI behavior is impossible to understand from static mocks.
How to start this week:
- Download Cursor.
- Build something stupidly simple. (I started with a personal website landing page.)
- Show it to an engineer and ask what you did wrong.
- Repeat.
You’re not trying to become an engineer. You’re trying to understand constraints and possibilities.
2. Observability: Debugging the black box
Observability is how you actually peek underneath the hood and see how your agent is working.
The skill: Using traces to understand what your AI actually did.
The tool: Any APM that supports LLM tracing. (We use our own at Arize, but there are many.)
Why it matters: “The AI is broken” is not actionable. “The context retrieval returned the wrong document” is.
Your first observability exercise:
- Pick any AI product you use daily.
- Try to trigger an edge case or error.
- Write down what you think went wrong internally.
- This mental model building is 80% of the skill.
3. Evaluations: Your new definition of “done”
| If you haven’t checked it out yet, this is a primer on Evals I worked with Lenny on. |
Vibe coding works if you’re shipping prototypes. It doesn’t really work if you’re shipping production code.
The skill: Turning subjective quality into measurable metrics.
The tool: Start with spreadsheets, graduate to proper eval frameworks.
Why it matters: You can’t improve what you can’t measure.
Build your first eval:
- Pick one quality dimension (conciseness, friendliness, accuracy).
- Create 20 examples of good and bad. Label them “verbose” or “concise.”
- Score your current system. Set a target: 85% of responses should be “just right.”
- That number is now your new North Star. Iterate until you hit it.
4. Technical intuition: Knowing your options
Prompt engineering (1 day): Add brand voice guidelines to the system prompt.
Few-shot examples (3 days): Include examples of on-brand responses.
RAG with style guide (1 week): Pull from our actual brand documentation.
Fine-tuning (1 month): Train a model on our support transcripts.
Each has different costs, timelines, and trade-offs. My job is knowing which to recommend.
Building intuition without building models:
- When you see an AI feature you like, write down three ways they might have built it.
- Ask an AI engineer if you’re right.
- Wrong guesses teach you more than right ones.
5. The new PM-engineer partnership
The biggest shift? How I work with engineers.
Old way: I write requirements. They build it. We test it. Ship.
New way: We label training data together. We define success metrics together. We debug failures together. We own outcomes together.
Last month, I spent two hours with an engineer labeling whether responses were “helpful” or not. We disagreed on a lot of them. This taught me that I need to start collaborating on evals with my AI engineers.
Start collaborating differently:
- Next feature: Ask to join a model evaluation session.
- Offer to help label test data.
- Share customer feedback in terms of eval metrics.
- Celebrate eval improvements like you used to celebrate feature launches.
Your Four-Week Transition Plan
Week 1: Tool setup
- Install Cursor.
- Get access to your company’s LLM playground.
- Find where your AI logs/traces live.
- Build one tiny prototype (took me three hours to build my first).
Week 2: Observation
- Trace five AI interactions in products you use.
- Document what you think happened versus what actually happened.
- Share findings with an AI engineer for feedback.
Week 3: Measurement
- Create your first 20-example eval set.
- Score an existing feature.
- Propose one improvement based on the scores.
Week 4: Collaboration
- Join an engineering model review.
- Volunteer to label 50 examples.
- Frame your next feature request as eval criteria.
Week 5: Iteration
- Take your learnings from prototyping and build these learnings into a production proposal.
- Set the bar with evals.
- Use your AI Intuition for iteration—Which knobs should you turn?
The Uncomfortable Truth
Here’s what I wish someone had told me three years ago: You will feel like a beginner again. After years of being the expert in the room, you’ll be the person asking basic questions. That’s exactly where you need to be.
The PMs who succeed in AI are the ones who are comfortable being uncomfortable. They’re the ones who build bad prototypes, ask “dumb” questions, and treat every confusing model output as a learning opportunity.
Start this week
Don’t wait for the perfect course, the ideal role, or for AI to “stabilize.” The skills you need are practical, learnable, and immediately applicable.
Pick one thing from this post, commit to doing it this week, and then tell someone what you learned. This is how you’ll begin to accelerate your own feedback loop for AI product management.
The gap between PMs who talk about AI and PMs who build with AI is smaller than you think. It’s measured in hours of hands-on practice, not years of study.
See you on the other side.
AI & ML, Commentary
Radar
