Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon
Descripción del Episodio
Aishwarya Naresh Reganti and Kiriti Badam have helped build and launch more than 50 enterprise AI products across companies like OpenAI, Google, Amazon, and Databricks. Based on these experiences, they’ve developed a small set of best practices for building and scaling successful AI products. The goal of this conversation is to save you and your team a lot of pain and suffering.
We discuss:
1. Two key ways AI products differ from traditional software, and why that fundamentally changes how they should be built
2. Common patterns and anti-patterns in companies that build strong AI products versus those that struggle
3. A framework they developed from real-world experience to iteratively build AI products that create a flywheel of improvement
4. Why obsessing about customer trust and reliability is an underrated driver of successful AI products
5. Why evals aren’t a cure-all, and the most common misconceptions people have about them
6. The skills that matter most for builders in the AI era
—
Brought to you by:
Merge—The fastest way to ship 220+ integrations: https://merge.dev/lenny
Strella—The AI-powered customer research platform: https://strella.io/lenny
Brex—The banking solution for startups: https://www.brex.com/product/business-account?ref_code=bmk_dp_brand1H25_ln_new_fs
—
Transcript: https://www.lennysnewsletter.com/p/what-openai-and-google-engineers-learned
—
My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/183007822/referenced
—
Get 15% off Aishwarya and Kiriti’s Maven course, Building Agentic AI Applications with a Problem-First Approach, using this link: https://bit.ly/3V5XJFp
—
Where to find Aishwarya Naresh Reganti:
• LinkedIn: https://www.linkedin.com/in/areganti
• GitHub: https://github.com/aishwaryanr/awesome-generative-ai-guide
• X: https://x.com/aish_reganti
—
Where to find Kiriti Badam:
• LinkedIn: https://www.linkedin.com/in/sai-kiriti-badam
• X: https://x.com/kiritibadam
—
Where to find Lenny:
• Newsletter: https://www.lennysnewsletter.com
• X: https://twitter.com/lennysan
• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/
—
In this episode, we cover:
(00:00) Introduction to Aishwarya and Kiriti
(05:03) Challenges in AI product development
(07:36) Key differences between AI and traditional software
(13:19) Building AI products: start small and scale
(15:23) The importance of human control in AI systems
(22:38) Avoiding prompt injection and jailbreaking
(25:18) Patterns for successful AI product development
(33:20) The debate on evals and production monitoring
(41:27) Codex team’s approach to evals and customer feedback
(45:41) Continuous calibration, continuous development (CC/CD) framework
(58:07) Emerging patterns and calibration
(01:01:24) Overhyped and under-hyped AI concepts
(01:05:17) The future of AI
(01:08:41) Skills and best practices for building AI products
(01:14:04) Lightning round and final thoughts
—
Referenced:
• LevelUp Labs: https://levelup-labs.ai/
• Why your AI product needs a different development lifecycle: https://www.lennysnewsletter.com/p/why-your-ai-product-needs-a-different
• Booking.com: https://www.booking.com
• Research paper on agents in production (by Matei Zaharia’s lab): https://arxiv.org/pdf/2512.04123
• Matei Zaharia’s research on Google Scholar: https://scholar.google.com/citations?user=I1EvjZsAAAAJ&hl=en
• The coming AI security crisis (and what to do about it) | Sander Schulhoff: https://www.lennysnewsletter.com/p/the-coming-ai-security-crisis
• Gajen Kandiah on LinkedIn: https://www.linkedin.com/in/gajenkandiah
• Rackspace: https://www.rackspace.com
• The AI-native startup: 5 products, 7-figure revenue, 100% AI-written code | Dan Shipper (co-founder/CEO of Every): https://www.lennysnewsletter.com/p/inside-every-dan-shipper
• Semantic Diffusion: https://martinfowler.com/bliki/SemanticDiffusion.html
• LMArena: https://lmarena.ai
• Artificial Analysis: https://artificialanalysis.ai/leaderboards/providers
• Why humans are AI’s biggest bottleneck (and what’s coming in 2026) | Alexander Embiricos (OpenAI Codex Product Lead): https://www.lennysnewsletter.com/p/why-humans-are-ais-biggest-bottleneck
• Airline held liable for its chatbot giving passenger bad advice—what this means for travellers: https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know
• Demis Hassabis on LinkedIn: https://www.linkedin.com/in/demishassabis
• We replaced our sales team with 20 AI agents—here’s what happened | Jason Lemkin (SaaStr): https://www.lennysnewsletter.com/p/we-replaced-our-sales-team-with-20-ai-agents
• Socrates’s quote: https://en.wikipedia.org/wiki/The_unexamined_life_is_not_worth_living
• Noah Smith’s newsletter: https://www.noahpinion.blog
• Silicon Valley on HBO Max: https://www.hbomax.com/shows/silicon-valley/b4583939-e39f-4b5c-822d-5b6cc186172d
• Clair Obscur: Expedition 33: https://store.steampowered.com/app/1903340/Clair_Obscur_Expedition_33/
• Wisprflow: https://wisprflow.ai
• Raycast: https://www.raycast.com
• Steve Jobs’s quote: https://www.goodreads.com/quotes/463176-you-can-t-connect-the-dots-looking-forward-you-can-only
—
Recommended books:
• When Breath Becomes Air: https://www.amazon.com/When-Breath-Becomes-Paul-Kalanithi/dp/081298840X
• The Three-Body Problem: https://www.amazon.com/Three-Body-Problem-Cixin-Liu/dp/0765382032
• A Fire Upon the Deep: https://www.amazon.com/Fire-Upon-Deep-Zones-Thought/dp/0812515285
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email
Episodios Recientes
Más podcasts de Sociedad y Cultura
Ver toda la categoría →
Cuentos e Historias Infantiles
By shows
Cuentos e Historias Infantiles es un podcast creado con amor para todos los niños de habla hispana. Aquí encontrarás cuentos, fábulas, poesía e historias maravillosas que fomentan el gusto por la lectura y la comprensión lectora. Un espacio donde la imaginación cobra vida a través de la narración. ¡Acompáñanos en cada aventura!
La Crónica Oscura
By shows
La Crónica Oscura es un podcast donde hablamos de true crime: asesinatos, desapariciones y misterios que se quedarán dando vueltas en tu cabeza mucho después de escucharlos. Me interesa ir más allá del simple titular: entender qué pasó, quiénes eran las personas detrás de cada historia y por qué estos casos siguen dando de qué hablar tantos años después. Si te apasiona el true crime contado de forma cercana y con un punto de misterio, este es tu espacio. Aquí nos sentamos, analizamos y, sobre todo, intentamos comprender.
Julia en la onda
By shows
Julia Otero te trae toda la información y el humor de los fines de semana de Onda Cero
Pablo Torre Finds Out
By shows
<p>Follow Pablo down the rabbit hole, beyond the game — every Tuesday, Thursday and Friday — on his Pulitzer-winning video podcast, part of The Athletic Podcast Network.</p><br><p>You’ll get a deep-dive "talkumentary" episode, reported by Pablo and Curiosity Correspondents like Emmy-winning comedian Wyatt Cenac and Oscar-winning director Ezra Edelman, answering urgent questions such as: Are you smarter than an NFL quarterback? Does Giannis intentionally miss free throws to win free chicken for fans? And why does the FBI director play so much hockey? This storytelling has been honored by the Pulitzer Prizes, the Emmy Awards, the Peabody Awards and the Edward R. Murrow Awards. (Journalism!)</p><br><p>Plus: Pablo hangs out with handpicked figures from sports, culture and politics like Sue Bird, Action Bronson and U.S. Senators… plays Share & Tell with friends like Domonique Foxworth, Mina Kimes, Katie Nolan and Dan Le Batard (aka his boss)… and unveils the secrets of capitalism with David Samson and John Skipper on The Sporting Class.</p><br><p><br></p><p>Follow PTFO…</p><br><p>https://www.youtube.com/@PabloTorreFindsOut</p><br><p>https://www.instagram.com/pablotorrefindsout</p><br><p>https://x.com/pablofindsout</p><br><p>and subscribe to Pablo's newsletter: https://pablo.show</p><hr><p style='color:grey; font-size:0.75em;'> Hosted on Acast. See <a style='color:grey;' target='_blank' rel='noopener noreferrer' href='https://acast.com/privacy'>acast.com/privacy</a> for more information.</p>
Monólogo de Latorre
By shows
El director informativo de Más de uno expone su reflexión sobre la actualidad del día
El Larguero
By shows
El deporte por dentro, el deporte como forma de vida, con Manu Carreño. Sin tremendismos, con humor y, a la vez, con rigor y pluralidad: escucharás distintos enfoques y puntos de vista. En directo de lunes a viernes a las 23:30 y a cualquier hora si te suscribes.
Nayo Escobar Podcast
By shows
Aquí tendremos historias asombrosas de personas que han logrado vivir feliz haciendo lo que le apasiona; tendremos también música para el alma, relatos de crecimiento, críticas culturales y discusiones sobre temas que, no nos atrevemos a hablar pero nos encanta escuchar.
Sastre y Maldonado
By shows
Una charla entre José Luis Sastre y Miguel Maldonado puede derivar por cauces imprevistos, derroteros impensables y argumentos que nunca imaginarías. En el piloto, Sastre le preguntó a Maldonado qué tendría este podcast que no tuvieran los demás: Que en otros podcasts no estamos tú y yo. Y es verdad. Maldonado está en otros podcasts y Sastre en otros programas. Pero juntos, sólo en este. Que se sepa.