12,000 attendees with AI on their lips, Robots moving with others on the exhibition floor, DJs playing amazing music and parties in Las Vegas
I went to Ai4 2026 in Las Vegas for real-life AI case studies – A presenter taking me through what they actually built, what it cost them, and what functionality broke. Most sessions were twenty minutes, and that time is only enough to state a conclusion.
So I walked out of a few sessions. They never got past the summary slide.
I spent most of the conference talking to CTOs and managers of managers from various industries: Transportation, Logistics, Manufacturing, etc. Almost none of them were from technology organizations
Every one of them was asking the same question: “What problem are we trying to solve?”
Uber’s CTO said a very important part was: I have seen so many AI product companies fail.
“You can always slap an AI into a product, but the question is, is that useful?”
(Ai4 2026, “Building an AI-Native Company at Global Scale,” Praveen Neppalli Naga with Nathan Bomey of Axios, Hall A Keynotes, Thu Aug 6)
Right now, AI is a solution in search of a problem. I did not expect to be writing that sentence in 2026.
My PM students ask me: What problems do we solve with AI?
What I stopped believing
I went in assuming that the speakers and attendees alike had two or three years of production experience behind them and we were all comparing notes.
That is not where we are. This is 1996 (Internet Age – Gold Rush), not 2010.
Doug Villhard said it better than I can, in a session where he graded startup pitches live in front of the room:
“Thanks a lot, AI. Thanks a lot for building things quickly. If there was accounting software, now there’s opportunity for AI software. If there was hair salons, now there’s opportunity for AI hair salons. Thank you for all of that, but everybody at this conference has access to all of that.”
“Push yourself to be different than everybody else, and do not be the AI for whatever. That’s not a world. Everybody ever heard of pets.com? I’m going to be the website for pet food. That’s not a thing.”
(Ai4 2026, “The Billion-Dollar Blueprint: Stress-Test Your AI Startup Idea with Professor Doug Villhard,” Tue Aug 4)
It is a gold rush.
While browsing the exhibit hall, I found many AI product companies still looking for their first credible customer. Twenty percent had genuine AI knowledge. The other eighty percent were there to figure out what to do in their own organization
Villhard’s advice to that room was the oldest advice in product management, and it landed because nobody wanted to hear it:
“If you really want to know if you have a good idea, it actually has nothing to do with what I’m going to teach you today. It has everything to do with actually going out and talking to potential customers about it. And the sooner you talk to them, the better.”
“Anybody can build anything. Relationships with customers- this is how we make money, and how I know for sure it’s a big problem.”
The sessions that I attended.
- “TCO of Going Open Weight: A Framework.”
- “The Hidden Tax on AI: How Connectivity and Identity Unlock Real AI ROI.”
- “Why Agentic AI Dies in Pilot.”
- “Managing Through the AI Cost Crisis.”
- “ROI Maxxing Case Study: The Alchemy of Shipping Faster, Proving AI Value, and Controlling Token Cost.”
I counted around twenty sessions across three days whose titles are about cost, returns, or why AI pilots don’t deliver.
On Thursday, there were three keynotes in the morning
1. “What’s the ROI on Your AI?”
- “There Is No ROI of AI”
- “Why AI Agent Pilots Stall, and What Actually Fixes Them”
Two of those sessions argue that ROI is the wrong question.
By the closing keynote on Thursday, Praveen Neppalli Naga, CTO of Uber coined the word tokenmaxxing for what happens when you hand engineers frontier AI tools and no ceiling. Uber spent its entire 2026 AI budget in about four months. His answer was not to cap anyone:
“First thing is simple solutions like caching. That’s not something new to the engineering industry. So prompt caching really helps us a lot because everybody is prompting similar stuff. And then it is about using the right models for the right things, and open-weight models.”
This is best described in the abstract for the PayPal keynote:
“AI capability is at an all-time high, but for most organizations, the hardest problem isn’t getting access to it. It’s turning that power into tangible business outcomes. That gap between AI’s potential and real-world impact is the last mile, and closing it is entirely about removing friction, not adding features.”
(Ai4 2026, “The AI Gap: Turning AI Capability into Outcomes,” Srini Venkatesan of PayPal with Ryan Donovan of Stack Overflow, Hall A Keynotes, Wed Aug 5)
Venkatesan put it in one sentence on stage:
“The problem with AI is not any more clever innovation, it’s all about adoption.”
I turned all the knowledge learned from the sessions into 18 AI PM interview questions
Part 1: Strategy and Value
1. Why do most enterprise AI initiatives fail?
According to the AI4 2026 Program guide
“Everyone says agents are a digital workforce. If agents are labor, the business hires them, pays for them, and fires the ones that underperform. Half your company is already building AI that was not approved.”
(Ai4 2026, “Stop Governing Your AI. Start Hiring It.”, Mark Abramowitz and Jed Dougherty, Dataiku, Hall A Keynotes, Tue Aug 4)
A panelist on the product management track shared something different
“Many times, when developing AI products, organizations feel that it’s a technology problem, or it’s a data science problem. But in my view, it’s neither a technology nor a data science problem. It’s a problem for the business, where the business wants to invest and see where the value is.”
(Ai4 2026, “Product Managing AI Products”, Tue Aug 4)
That panelist also cited the MIT figure from the stage. MIT’s NANDA initiative published “The GenAI Divide: State of AI in Business” in July 2025, finding that 95% of organizations were getting zero return on GenAI.
2. How do you decide which AI use case to fund first?
From the same panel, on why uniform governance fails:
“Not every AI use case [requires] a similar governance process. Summarization, reporting, recommendation. Do we want to go through, for each and every such use case, a very rigorous governance process? That’s a food for thought for all organizations and as a product leader.”
And the practical answer another panelist gave:
“Based on how a user completes that end-to-end process, a certain risk level is assigned. Based on that risk level, there are certain things that they need to do before a tool or agent is actually deployed into production.”
On what the intake process is actually for:
“All of the approvals, all of the ownership, the accountability, all of the testing validation, it’s all stored within this one centralized place, which makes governing tools or agents a lot more simplified. We’re not slowing people down, we’re not becoming a bottleneck, but we’re giving them the tools that they need to succeed.”
(All three: Ai4 2026, “Product Managing AI Products”, Tue Aug 4)
3. A competitor can rebuild this feature in a weekend. What’s your moat?
Doug Villhard, grading a pitched AI marketing assistant live in the room:
“All what’s called thin wrapper AI stuff gets an F. Because basically ChatGPT can do it. There has to be something about you that’s different than ChatGPT.”
What he wants to hear instead:
“I need to hear that every customer you get, it’s learning something new that’s proprietary to you. I need to hear that it’s integrated.”
“If somebody can make what you do in a weekend, or if you made it in a weekend, I’m not funding it. I am looking for unicorn companies, billion dollar companies. I am looking for monopolies in new spaces.”
And his closing line:
“I always say three T’s. Team, tech and traction. And the greatest of these is traction.”
(All: Ai4 2026, “The Billion-Dollar Blueprint: Stress-Test Your AI Startup Idea with Professor Doug Villhard”, Tue Aug 4)
Jasper raised $125 million at a $1.5 billion valuation in October 2022 on roughly $75 million ARR.
By September 2023 Jasper had cut its 2023 revenue forecast from $140 million to about $100 million. This marked its internal valuation down about 20%, and its founding CEO was replaced. All this because ChatGPT was launched.
4. Describe your product’s problem in one word.
Villhard runs this on companies everyone knows:
“One of the beauties is if you can figure out what problem you’re solving down to one word, you can actually tell if you’re on strategy or not. So I tell my students all the time, you get an internship at McDonald’s corporate, do not pitch them pizza.”
On Netflix, after the room guessed content, personalization and recommendations:
“Why do you watch it? All you do is ask yourself why three times. Boredom. Escapism.”
On Starbucks:
“Go get you an internship there, my MBA students. Pitch them something that doesn’t have caffeine. You’re out the door.”
And on what the test actually is:
“You know what my test really is? It’s really the definition of a monopoly. So all you have to do is run your idea past the concept of a monopoly, and it will tell you what’s wrong with your idea. And that is what you will work on the whole rest of the time that you own it.”
(All: Ai4 2026, “The Billion-Dollar Blueprint”, Tue Aug 4)
His published framework has five parts and he is explicit that it is aspirational, not pass/fail. McDonald’s, Starbucks and Netflix do not hit all five.
5. Leadership says “we need an AI strategy.” What are your first 90 days?
Praveen Neppalli Naga, Uber’s CTO, on what did not work:
“My CEO said we need to pick up and engage in AI. But what does that mean? Everybody started using AI to write emails, fix the grammar and write some documents. But even in the engineering world, taking a task and asking the agent to write the code for it is not going to have a huge impact. That is what is called a task.”
What did work:
“You need to think of a workflow, automating the entire workflow, because that’s where we spend a lot of time. You get buried in organization boundaries and then you get decision making. That workflow, when you automate, we started seeing results.”
“We really took our best AI engineers, 30 of them, paired them up with functional experts in finance or operations team or marketing team. Engineers understand the workflow and really put an agent in it. It’s incredible the results that we saw. Our CEO and I pushed for it, but the innovation came from the people who know [the work].”
(All: Ai4 2026, “Building an AI-Native Company at Global Scale”, Praveen Neppalli Naga with Nathan Bomey of Axios, Hall A Keynotes, Thu Aug 6)
Part 2: Software Development LifeCycle
6. How does the AI product lifecycle differ from a normal SDLC?
From the product management panel:
“Traditionally we have used a [software development] lifecycle, but building an AI product itself is a different lifecycle altogether, and needs to be embedded with the required guardrails, and that goes right from the start, not like when you are shipping the product and then you are evaluating its state.”
(Ai4 2026, “Product Managing AI Products”, Tue Aug 4)
The counter-argument is that front loading is slower and can be fatal in a startup. Here you need to share that guardrail work has to scale with exposure
7. When do you decide not to use AI?
A great article I recently read from the MIT Sloan School about this is https://sloanreview.mit.edu/article/when-not-to-use-ai/ where he talks about AI’s use as a Manager
If you are using it as a product architect and designing AI agents, check if the problem can be solved with a deterministic rule then use a rules engine.
As an architect try these three tests on your business rules:
- Can I write the rule down?
- Does the cost of a wrong answer exceed the value of a right one?
- Is there a deterministic path to 80% of the value at 5% of the cost?
8. Where does your AI agent feature live?
Will Bryk of Exa on where agent performance actually comes from:
“The performance of your agents is not just about the intelligence of the model. The agents are already smart enough for a lot of tasks.
“Whether you’re building a market intelligence agent or a financial agent, they’re smart enough, and they’re bottlenecked by the information they have access to.”
“A lot of companies are using cheap open source models or cheap frontier models and combining them with a super high quality information engine, and you get the same quality as if you were using a bigger model.”
(Ai4 2026, “The Future of AI Depends on Perfect Search”, Will Bryk, Exa, Hall A Keynotes, Thu Aug 6)
What survives at the application layer is your data, workflow, permissions model, integrations, and evaluation harness. The reasoning is rented.
9. How do you write acceptance criteria for a non-deterministic feature?
“We have acceptance criteria, and this is written in a domain-specific language called Gherkin. The idea is to be able to express acceptance criteria, or what a system should do, in a really natural language format, so it’s easy for anybody to read.”
“When we get to the developer stage, we’ll be able to implement these tests. Anything that gives the AI a chance to validate the results of its own action and check that it’s done the right thing, it’s going to give you better results.”
And the step at the end that most teams skip:
“We actually go back and validate the implementation, and this creates proof artifacts. That’s actually a video of the system walking through the checkout, making sure that it does what it’s supposed to. We’ve got test reports and everything. These are ways to keep AI on the rails and let it see the results of its own actions.”
(All: Ai4 2026, “Building an AI-Native Product Workflow”, Thu Aug 6)
10. Define hallucination for your product and set a launch threshold.
According to Wikipedia: a hallucination or artificial hallucination (also called bullshitting,[1][2] confabulation,[3] delusion[4], or mirage[5]) is a response generated by AI that contains false or misleading information presented as fact.[6] These terms draw a loose analogy with human psychology, where a hallucination typically involves false percepts.
For AI summarizing large text, it is generating a summary that is not part of the text; for a retrieval agent, it is a citation link that does not resolve; and for a Financial Services assistant, it is any number that is not part of a system of record.
Chloé Bakalar, on making a value decisionable:
“You can think about guardrails versus targets. Guardrails block, targets guide. Which ones are appropriate given a particular value in a particular context?”
“It’s all good and well to say that we care about broad values like safety and fairness. That doesn’t do much good until you can actually crisply define what it is that you mean by those terms. Then figure out: how are we going to measure them?”
11. Latency and quality are in tension. How do you decide?
Bryk on why agent latency is a different problem from human latency:
“Google was made to be half a second, because that’s what humans like. But these AI systems, they want faster than Google.”
“Some use cases in AI agents that people build require the fastest possible search. But other AI agents, they don’t care about latency at all. We have customers who want to run crazy batch jobs over the world information. And they don’t care at all about the latency. They just want the highest quality.”
(Ai4 2026, “The Future of AI Depends on Perfect Search”, Thu Aug 6)
Pick the latency budget first, then buy the best quality that fits inside it.
12. How does your product get better as people use it?
From the marketing agent session:
“Previously, a lot of company proprietary information, in terms of how decisions are made, is lost. Someone might decide to discount a certain product and then launch it, and you see how it performs. That information is mostly [lost]. In this new world where we see agents and humans working together, and agents learning from experts in terms of what to do and not to do, this decision, what we call the system of decision lineage, becomes more and more important.”
“Once an agent works with you through an entire workflow, it’s able to see what works, what doesn’t work, learn, and keep optimizing. All of this information will now be stacked in your company’s internal memory. And every time you have a new recommendation, it learns from what worked or did not work in the past.”
(Ai4 2026, likely “The Unfair Advantage: Why a 2 Person Team Can Now Outmarket a 2,000 Person Brand”, Thu Aug 6)
13. How would you measure ROI on an internal agent?
Pick the cycle time the agent is supposed to compress and instrument it before you build.
From the self-healing pipeline agent session, on the measured outcome:
“What we have seen is this has reduced our RCA [root cause analysis] time significantly.”
(Ai4 2026, likely “Self-Healing Connectors: AI Agents That Fix Production Breakages Autonomously”, Wed Aug 5)
From PayPal, on the workflow they targeted:
“Before [the assistant], you would be fighting your roadmap. It would take weeks, if not months, depending on your roadmap.”
(Ai4 2026, “The AI Gap: Turning AI Capability into Outcomes”, Srini Venkatesan, PayPal, with Ryan Donovan, Stack Overflow, Hall A Keynotes, Wed Aug 5)
Part 4: Agents and Autonomy
14. How do you get from copilot to agent autonomy?
Srini Venkatesan, PayPal’s CTO, walking the ladder:
“Autonomy is not a single step. It takes a while to get into it. Even at PayPal, I would say about a year ago, my engineers were very skeptical about AI. They were like, I’m not sure whether this is going to be useful.”
“What we did is, we started out with code assist. That’s it. And then after maybe three months we started getting into loop engineering, and that’s when we started to really focus on the validation pattern. Because once you have the validation pattern then the engineers are more comfortable. It might do two, three tries but it will get to a consistent answer.”
“If you want to get into autonomous and you want to believe it, have a great validation suite, certification suite, and then let the agent work it out while providing that context and that expertise to them.”
On the humans in the loop:
“We still have the human in the loop, but we have the reviews that are happening. But the key is, you let the agents have the first say. And you get 98% of the work done in less than a few weeks.”
And the problem:
“The problem with AI is not any more clever innovation, it’s all about adoption.”
(All: Ai4 2026, “The AI Gap: Turning AI Capability into Outcomes”, Hall A Keynotes, Wed Aug 5)
Ryan Donovan of Stack Overflow shared the numbers
“From the previous Stack Overflow developer survey results, we found that about 46% of developers out there actively distrust AI.”
15. What can an agent execute without human approval?
From the marketing agent session:
“Agents are great at knowledge synthesis, reading many different data sources and being able to inference on that. As an analyst, or a competitor monitoring, or information tracking tool, they are quite good. They can be running while you sleep. I run my agents overnight.”
On the tiers above that:
“As we start moving more towards the output side, customer-facing sort of aspects, this depends more on a case by case basis. For things that are low stakes, for example creating an advertising campaign with a new creative, these are things where you want agents to execute, but with defined guardrails.”
“For things that are more high risk and more long term, like positioning or creating a new product, this is something where a marketer or a business owner does what matters the most.”
(All: Ai4 2026, likely “The Unfair Advantage”, Thu Aug 6)
Automate what can be automated and design a clean handoff to the human for everything that cannot.
16. How do you keep an agent’s context and provenance straight?
From the AI-native workflow demonstration:
“We assemble all of this context into what we call our context tree. It’s like an index. It’s got all the information about what was found, and links back to the source, so you’ve got this provenance leading back to the original data.”
- A meeting transcript
- A shelved 2024 feature document in SharePoint
- An existing web accessibility policy
- The clinic portal code
- The billing service
- An old pull request
“In this story, we have a context tree attached. So we’ve got all of that context that will be available to whatever system is executing this.”
(All: Ai4 2026, “Building an AI-Native Product Workflow”, Thu Aug 6)
17. What’s your security posture for agents?
Neppalli Naga, on the tension:
“This is the problem of being a CTO. We’re excited, we want to use the agents and we push the teams, and which means that you have to give access to all these tools. Nobody knows what they do. The other side of the CTO, which is my CISO, is really scary. The one thing that keeps me up at night is security.”
On the controls:
“We don’t give the access to the [sensitive] data to the agents. If we do, we contain no internet connection in the host so that it’s not leaked at all in any way, and then we are able to use the ADR to monitor what’s going on.”
On why monitoring has to be selective:
“It’s impossible to monitor everything, every session. It’s just a humongous world, it’s really expensive. So that is where some kind of [model] can actually say what are the malicious or benign sessions and monitor those things.”
On building rather than buying:
“You can just buy it. I don’t think it’s mature enough, available outside. The thing for us is, as we start building these things, we will learn a lot.”
(All: Ai4 2026, “Building an AI-Native Company at Global Scale”, Hall A Keynotes, Thu Aug 6)
Uber open-sourced the monitoring component as ADR, Agentic Detection and Response, at github.com/uber/ADR in August 2026. It collects agent telemetry including prompts and tool calls and flags credential exposure, prompt injection, data exfiltration and policy-violating tool use, across a reported 50,000-plus agent sessions a day.
The Ai4 program framed the general problem in one line:
“Every agentic action is a routing decision, a trust decision, and a telemetry event.”
(Ai4 2026, “Intelligence, Cost and Control: Infrastructure for Agentic AI”, Jeetu Patel, Cisco, with Matt Egan, CNN, Hall A Keynotes, Thu Aug 6)
Part 5 – Data
18. What is synthetic data, and where does it actually earn its place?
Mikayel Khachatryan of Wirestock, on the categories:
“It could be direct model outputs, AI generated content or data. It could be using AI models to edit or enhance real world or human created data. Or it could be used for annotation, curation, and all sorts of other use cases. Depending on which one of these types of synthetic data we’re talking about, there could be different pros or cons.”
On where it runs out:
“As the models get better, they need more and more complex input from humans. So it would be the same humans creating the data, but it would be much more complex and nuanced data. Simple visual data, for example stock photos, might not be as needed.”
“Models improve and improve and then there are diminishing returns on how much these models can improve. So you need to go back to the well of real data to refine the model again.”
Aniket Tushar Ajagaonkar of Airbnb, on trust and safety, which is the best line of the conference:
“Humans are really creative in how to abuse policies and platforms, and you get to learn what people are thinking. Synthetic data is limited by our imagination about people, like how people could abuse. The capacity for abuse in humanity is much more widespread than you could imagine as an individual.”
On the technique that works anyway:
“What has worked is not using real data to just generate synthetic data directly, like these are 10 samples, generate 1000 samples for me, but rather getting a level above and trying to understand the human behavior, extract attributes from it, and then pass those attributes to the model to generate synthetic data. That way you get generalization.”
Karl Hanson of Tonic.ai, on proving it works:
“You could train a model on synthetic data and then do your test and eval on small sample real data. If there’s a public benchmark you want to see if it improves the model on, you can go try it out. If it hill climbs properly, then you have a pretty good reason to think that your synthetic data [worked].”
On buy versus build:
“It’s really on the vendor to prove that it’s faster, better, and cheaper.”
(All: Ai4 2026, “Can Synthetic Data Replace Real Data?”, Thu Aug 6)
If you liked the article, please let me know in the comments below and I love to share more from the conference and invite presenters to my podcast.
Share your thoughts in the comments below.
Citations
- MIT NANDA, “The GenAI Divide: State of AI in Business,” July 2025. 95% zero return, based on 52 interviews plus 153 survey responses. Widely criticized for counting non-attempters.
- Stack Overflow 2025 Developer Survey: 45.7% total distrust, 32.7% total trust, “highly distrust” 7.9% to 19.6%, n=33,244 on that question, 84% using or planning to use AI tools. No 2026 figures published yet.
- Jasper: $125M raised at $1.5B valuation Oct 2022, roughly $75M ARR peak, 2023 forecast cut from $140M to about $100M, valuation marked down about 20%, founding CEO replaced Sept 2023.
- Uber: entire 2026 AI budget spent in roughly four months; adoption of frontier tools quadrupled since start of 2026; cost per token down; total cost not flat. Fortune, Aug 7 2026.
- Uber ADR, github.com/uber/ADR, open-sourced August 2026, 50,000+ agent sessions per day.
- Exa: $250M Series C at $2.2B valuation, May 2026. 5,000+ companies, 400,000+ developers, sub-200ms search, 500 billion+ URLs.
- Doug Villhard, Professor of Practice in Entrepreneurship, WashU Olin. Sold Second Street to Upland Software for $30M, January 2021. Litmus test published at source.washu.edu.
- Karl Hanson, co-founder and CPO, Tonic.ai. Mikayel Khachatryan, co-founder and CEO, Wirestock.
- Stanford HAI AI Index: 59 US federal AI regulations in 2024 vs 25 in 2023, from 42 agencies vs 17 in 2022. The 2026 Index describes a deregulatory federal turn while state obligations expand.
