Building Netflix With Elizabeth Stone: Entertainment At Scale, Personalization, And Pushing Beyond Film and TV

Speaker A: On a Friday night, the thing we show you might be different than what we show you on a Monday morning. And so it sounds so simple. Different time of day, different device. You're on TV versus mobile. But those were less relevant when we had film and TV series than when we start to add some of these other content types. We have just as many pixels on the homepage. We have, hopefully, more moments and times of day and across more devices to try to be the right thing for a member in that moment. But. But the product should feel like that's effortless.

Speaker B: Netflix is one of those rare products that feel incredibly simple from the outside. You open the app, you scroll to find something you like, you press play, and you're ready to go. But to create something that simple, everything behind a product has to be world class. The technology, the operations, the content, the culture, and the craft itself. And now that experience is evolving. Netflix is no longer just about movies and TV shows anymore. It's about live sports and games and podcasts and vertical video on mobile. And it's no longer just about AI implicitly powering accommodations in your feed. It's about AI becoming more active and more explicit in your experience. Leading this work at Netflix is my guest today, Elizabeth Stone. She is the CTO and CPO of the company. She has one of the most unusual paths into product and technology. She has an economics PhD. She worked in Wall street, healthcare, transportation, and now entertainment. We talked about how Netflix builds, how content and product work together, the tension behind expanding the product while keeping it simple and delightful. We also talked about the future of entertainment and what it means to become more personalized, more immersive, and more interactive for the user. This evolution is what makes the conversation so much fun. With that, here's my conversation with Elizabeth Stone.

Speaker C: Elizabeth, it's awesome to have you on the show. Thank you so much for joining me.

Speaker A: Thank you for having me. Great to be here.

Speaker C: So we always start on the personal side. Um, and like many guests on the show, you had a very unusual career. You did economics, PhD, Wall street, healthcare, transportation, and now entertainment. As the CTO and CPO of Netflix, when you look back, like, do you see a thread that connects the whole thing? I was trying to find it, and then I'm thinking when you started your PhD in economics, your career turned out different than what you had in mind.

Speaker A: So when I went to do a, uh, PhD in economics, it was because I wanted to teach. I learned very quickly that the thing that I would actually be focused on would be research versus pure teaching. And so I decided I would switch into an industry, economic consulting, which is sort of at uh, the intersection between industry and academia. That ended up being a sweet spot for me, which in some ways is not dissimilar to leading data science teams in healthcare and transportation and entertainment, which is find a way to bring science to a business problem. And that has ended up being common ever since I was in economics.

Speaker C: What did you have to unlearn? Because Netflix works very differently that when you came in you're like, okay, I'm used to working display, but Netflix just builds differently.

Speaker A: So one is the culture is unusual and that gets reflected in the way things are built. So um, much less top down than many other companies. So ideas get generated more organically from the teams in many cases. There's a lot of decision making and agency that individual contributors have on the team and there's a sense that responsibility and decision making sits very deep in the team. It's a tougher role for leaders in a way to find like what's your role? If it's not always determining exactly what we should do or exactly how we're going to do it. You have to be comfortable leaving space for the teams to have their own ideas to lean into risk, to understand that bets are going to be made and some of them will fail and to relinquish some of that control that you might have as a leader in other places. But do that because we get better outcomes. A company like Netflix or a technology company is like conducting an orchestra of lots of different areas of expertise. So you have engineers and product managers and designers and data scientists and so on and how to get those teams to work together. I think companies like Netflix or other technology companies have more diversity and skills and the thing you learn as a builder is how to get those things to be very complimentary and powerful together. We're both entertainment company and technology company so I had to learn about how do you conduct and orchestrate a bunch of different product and technology roles in a company that also is very strongly oriented towards content and brand and things related to that that has to come together and be the whole is greater than the sum of the parts across technology and entertainment.

Speaker C: Very few companies in the world have world class content operations and world class product operations and R and D operation at the same time. And that's something that Netflix is very, actually goes all the way to the co CEO relationship on the top. How does that work day to day?

Speaker A: So at the overall Netflix level there's a sense of like, what are we trying to do better at or how do we further improve for consumers across what Netflix is very good at already? But it was more of an optimization problem. One of the things I've seen at Netflix, which has been a ton of fun, is standing up some of the newer areas, like live programming, like games, like now podcasts, where then you're not in an optimization, you're in like, we're trying to stand up something totally new that requires both a content perspective on it. So which live events should we program? What partnerships do we want to have, and how do we build the product and technology to do that? Well, and that goes back to the earlier days of Netflix, where you have to have a weekly standup. You have to talk about, like, which deals are we thinking of signing, what would be the requirements for that, what's going to be easy or hard? And we've had to relearn how to work really, really closely together to make sure that these new efforts are more effective than they would probably be if we were working in parallel or not connected. And through these new efforts, like live and games and podcasts, we've also gotten better at working together in the mature areas of the business. So what would be exciting that we could do, like, we redesigned TV, UI to think about how to better promote content and how to do a better job with recommendations for members. We're rethinking mobile now. Those things have been in much tighter partnership with the content team, I think because we learned a little bit about what we were missing when we were doing more local optimization versus if we partner together. And that's where the it. It sounds cliche, but it's true. Like the magic of, uh, Netflix, like, what could only Netflix do would be to bring really creative content ideas together with the product to make it very successful.

Speaker C: If we just go back to the core streaming capability, is it that, um, there's two roadmaps and you guys meet quarterly to align. You know what's coming, they know what you're building, and you're trying to make sure they're in concert with each other. Is it that they give requirements for product? Product gives requirements back. And you try, like, how is that coordination working?

Speaker A: The role of definitely people on the product and technology team is to let the content organization know what is the direction of the product, what features do we have today? What features will we have in this quarter or the next year? But if we're just talking about film and TV content, that content is not dependent on new features. It's more what is the product doing to be even better for consumers. And the content team might have some requests of could we find new ways to surface this type of content, or can we think about merchandising differently on top of personalization? So there'll be ideas that we consider as part of our roadmap, or learnings that we've had based on experiments that we've run, that we work with content to decide which of these features are we more excited about rolling out. But they're not both roadmap oriented teams. It's when a new content type comes, like when our chief content officer says, we signed a deal with NFL, then we say, well, then our product roadmap needs to be really prepared on a set of features that are critical in order to deliver live content. So that's where the roadmaps have become more aligned because of some of the newer business areas that had special feature requests. Otherwise, I would say that the content and product teams were in a constant state of like, let's always be improving and communicating with one another about what we're doing to improve. You know, there's still problems to be solved and things we want to do better, things that come with scale or things that come with change in consumer expectations, or when we have a very large content catalog, we have to be excellent at personalization.

Speaker C: So you move from one product to multi product. It's always a hard transition. It's never easy because, you know, your house was so organized before and everything had their places to be and, um, the product was seamless and cohesive and people knew it really well. How did you think about that transition from one product to multiple products?

Speaker A: It's one of the biggest challenges we're facing now to figure out how all of these new, I'll call them, like content types come together in one product. I hope not to create multiple products, to have one that has a broader set of types of entertainment or content, but that knows exactly what is the right thing to recommend for you in that moment. So the more it feels like we're able to broaden out what we're offering, but it doesn't feel more complicated or like it's multiple products to navigate for a consumer, the more successful we will be. We can't do that out of the gate necessarily for a couple reasons. So first we have to introduce to members that we have something new and generating awareness of something. There's a tension relative to personalizing because we're basically announcing. We have games, we have live, we have podcasts. It's learning who's interested? It's learning about things like Live is very different in a lot of the joy and value value from Live comes in anticipating it and then watching it in that moment when it's live. So when you go a layer deeper into things like what does personalization or the product need to be able to do to understand the difference between a timely piece of content and a not timely piece of content, and how much anticipation should we build that doesn't cross the line into this is annoying for a consumer. In addition to on a Friday night, the thing we show you might be different than what we show you on a Monday morning. And so it sounds so simple. Different time of day, different device. You're on TV versus mobile. But those were less relevant when we had film and TV series than when we start to add some of these other content types. So we have just as many pixels on the homepage. We have hopefully more moments and times of day and across more devices to try to be the right thing for a member in that moment. But the product should feel like that's effortless for a consumer. So I think that's the core challenge.

Speaker C: Anything you tried that you're like, okay, that's not gonna be the path or any. Or anything you're seeing from others that you're like, okay, this is a good principle.

Speaker A: We underestimated how much the different content types need to be visually differentiated. So we're introducing liver games and we presume that's obvious to a member, but we need clearer visual cues. So even things like icons or different presentation on the page to signal this is something different than the rest of the content on the page was something that we've learned. We saw confusion around people are clicking on something and then realizing it's a game or seeing a row of things that are podcasts, but not realizing that it's something different. We also have learned some hard lessons about how much to put how much real estate to use for the new content types versus the more mature content types. And the trust buster for a member would be doing too much that it uh, like we have to find that sweet spot of not over promoting. So whether that's in the navigation at the top of the page, we switched it from the side to the top so it would be prominent so that it would allow members to explore the catalog more easily if they don't see what they want on the front page. And then we're experimenting with something now that is just below the billboard which lets you hop into directly. Podcasts, games, different offerings we have. So that creates a visibility of the new offering without saying we should show title by title or game by game on the homepage. Exactly the specific things. Because if we get it wrong, then there's still a reliable pathway where people start to realize we have more to offer, but it's more subtle.

Speaker C: So in a way, there's the awareness, discovery side. And then when you come in, how is there more of, uh, a comfortable or knowledgeable pathway? And then once I build habit with it, I know exactly where to go. I'm not trying to find my game instead of the experience space.

Speaker A: That's right. I mean, things like we hear from members too, like if we move, continue watching, that can be a very frustrating experience. Like, I'm just trying to find the title I was already watching, make it easy for me to find that. And it's one of the things you can see in the data and, um, the patterns of how members are exploring, of when they're going to search right away that they're either looking for a specific title that they've not yet watched, or they're looking to explore the catalog in a way that we're not offering on the homepage. There's other times continue watching, going straight to continue something. And there's other times which are more like, explore. I want to see a whole bunch of things. And then you can see that across the homepage. Part of that is, okay, that we see members, you know, they're on one of those paths when they come in, but I also believe we should be able to anticipate that better. And so that's related to, like, how does it make it feel like the Netflix product anticipates very well which of those modes I will be in? And if we don't anticipate it well, how can we quickly get signal from you that you're in one of those modes so it doesn't feel like we're trying to read tea leaves and making it more difficult. And that problem has gotten more challenging as we think about across devices or times of day or content types, because there's more to anticipate or there's more to figure out. How can I get an explicit signal from you in that moment so that we can reset the recommendations we're making based on the context that's relevant for you?

Speaker B: Yeah.

Speaker C: I'm curious when you think about, um, the true Norths you're putting or have you rethought, like, what are you optimizing for or what are you holding as a true North Metric, same true north.

Speaker A: Uh, this is actually a positive and a strength. I would say at Netflix that the true north has been consistent on how much value are we delivering to members,

Speaker C: which is time spent.

Speaker A: More complicated than that because I think time spent can always be immediately translated to value. Um, so without revealing the details of the metrics that we track, things that we think are strong proxies for, are we delivering great value to members? And we have the benefit of being able to ask members too, through our consumer research of, um, how much value are we delivering relative to the price? How do you think about what you're getting from Netflix versus competitors? So there's ways we can look at both the data of how people are behaving and then the feedback we get from our members that has persisted not even in spite of the new content types, but the new content types were motivation for driving more value. So the reason to go into games or live or podcasts is because we think there are ways to drive more value for members. It was in service of ultimate member value and that members feel like they're getting more from Netflix. The mindset that gives everyone is there's going to be trade offs here. And as we measure what's the right moment to recommend this film versus this game, we're all in service of the thing that we think maximizes the value for that member. So as long as we're competent in the measurement of that and understand the trade offs there, sometimes that's gonna fall in the favor of recommending the movie, sometimes in favor of recommending the game. But we're doing the right thing for Netflix members overall.

Speaker C: Was duration also part of it? Like, for example, if I introduced Live, and I'm, um, I can see that being okay now with Live and other aspects, we need to give more time to see the effect because now I have to sign up for the live. You're taking prime real estate, the billboard. So that used to be a great way to kind of go and watch something, but now I'm m signing up for something that might take away from

Speaker B: an action I wanted to do.

Speaker C: But really the value will come a bit later. So there's like more patience that you have to build into the system that you had before or not.

Speaker A: Yeah. M. When you first said duration, I thought of a couple things. Okay, so the first is different content types, like a game or a podcast or a live NFL game or a TV series actually have different duration for engagement with that, uh, content. And they're not apples to apples because a minute playing A game could be as exciting and valuable as 20 minutes watching this other thing. So we can't just index on time spent for that reason. Like there's different value attribution. The second thing is definitely in the early days of live games. Now for podcasts, you don't get success out of the gate. You get do. We see signs of life and signs of promise here and we have to be willing to take a headwind for a period of time because of the opportunity cost. So like, I could have for sure shown you a movie that I think you would have loved in the Billboard. Instead I showed you the live title because we're trying to understand interest in it and see what viewership looks like and whether that's helpful for members. But then I took the trade because we're basically making a bet as a company that if we want to be an amazing entertainment company, we've got to have more than film and series. And so we're going to have to be willing to take some of that trade off and headwind in early days to learn the toughest thing from the builder perspective and from the company strategy perspective is are we learning enough fast enough to be confident that we're on the right path and these trade offs will be worth it. So that creates an urgency in the teams of, uh, we're not going to have forever to understand how this drives value or whether it drives value or whether we have the right strategy. So let's come out of the gate as strong as we possibly can. Fail fast, learn quickly, because at some point we're going to need to say that trade off was pretty big.

Speaker C: Now when I think about my family's TV, there's two apps next to each other. It's Netflix and YouTube. And um, they're both heavily used and um, they're both kind of unique in their own way. Right? Like Netflix has this like TV, laid back, premium content. YouTube is UGC, a lot of vertical content. Put aside YouTube for a second. Just entertainment in general. Do you think of Netflix as more. Hey, there's concentric circles of entertainment and it's more progression of, you know, we

Speaker B: started with On Demand.

Speaker C: Nowadays live, there's games, tv, there might be UGC in the future because there isn't really a reason why not. Or you see that QT was just a very different island from what you're doing.

Speaker A: I think the entertainment landscape has gotten a lot broader because there's a lot more types of entertainment now. But we're still, there's still the same number of Hours in a day. And when Netflix started DVD too, streaming over the Internet. But also the way of thinking about content at scale that we could distribute all around the world, that's a great baseline to then say what are other entertainment types that we should integrate into what we're offering to consumers? And that's where live and games and podcasts come from. Realizing that in order to entertain the world, we have to have more than just a narrow set of content types. But I don't know that I would take that to the limit to say we have to have everything in order to compete as long as the sets of things like concentric circles that we're expanding into are very high value for consumers and make them excited to keep Netflix. I don't know that Netflix has to be the one stop shop of everything in order to be really compelling as an entertainment offering. So I think where we sit now is we know we want to have more than film and TV series, be best in class at film and tv, but expect expand beyond that. But I think it's, it's still a valid question of like, do we have to have exactly the same things that every other entertainment competitor has? I don't know because I don't think the end game is there's not tons of competition for entertainment time. It's that we have the right combination of things and we deliver them to our earlier conversation really seamlessly. From a product perspective, um, I want

Speaker C: to shift a bit to AI, uh, because before this LLM era, a lot of it has been about personalization and I sometimes call it matchmaking.

Speaker D: Right.

Speaker C: Is that still how you think about the core promise of AI for Netflix? Or now you're in this mindset of like, personalization is just one aspect. There's so much we can do.

Speaker A: So definitely for personalization, as you're saying, also in things like figuring out how best to localize content to many other languages, there were lots of AI tools that were used for things like visual effects, things on studio production. And then there's also a lot of AI that goes into how we optimize content delivery, to say it in the simplest way. So how do we actually shape and direct content delivery across our open connect? Our CDN all around the world has

Speaker C: a lot of AI as optimizations, basically

Speaker A: as optimizations for how we encode, um, content and how we actually shape and deliver content.

Speaker C: I'm curious for you if you want to determine five to ten years from now, uh, like what are you envisioning would be very different or potentially what Would not change at all. And you're kind of looking at that as the vision for the industry.

Speaker A: The words that I have as a headline in my head are more personalized, more immersive, more interactive. So that can take different formats that could be, um, to make it more concrete. Interactive can mean I'm able to change the outcome of what I'm watching. So today that looks like live voting for some of our content. How does that evolve over time, where the consumer actually shapes the outcome of the title? Um, games is a way of thinking about immersive. Like, I'm in the world. I am. I'm experiencing that IP or that title and the characters myself. So that would be one thing that I think more entertainment beyond just games will go towards and personalize is not just the matchmaking, but how content ends up really suited just for you. And so that could be because we have a greater breadth of content and we know how to match it in that moment to what you're looking for, or the content itself becomes more personalized. I think all of those things are going to be shaped, uh, by Netflix, among others. But it feels like the direction of travel across a whole bunch of things, plus layering in the social, the community, the connectedness, which I hope is a very important part of it, especially if I think about entertainment. It's a human endeavor. It's something that's part of the human identity.

Speaker C: I love that, uh, what is a show that you love on Netflix? But people underappreciate.

Speaker A: I'd have to go with Unchained. Um, so I am a big Tour de France and cycling follower. The last season we did of Unchained was last year. So it's not continuing anymore. But I loved it. It revealed the humans and the how team oriented cycling really is, which I don't think is well understood by people who are close to it. And so maybe selfishly, I was hoping that more people would watch it so they would understand why I'm so obsessed with.

Speaker B: It's a great. For everybody.

Speaker C: Like, this is a great, uh, show. It's just remarkable. And it will get you excited about cycling.

Speaker A: Yeah. And there's flavors of that for golf that we have for Formula One. But I was personally drawn to the cycling.

Speaker C: Elizabeth, thank you so much.

Speaker A: Thank you. Thanks for having me.

Speaker C: It was fun.

Speaker B: I very much enjoyed my conversation with Elisabeth. Her background and thoughtfulness came through very clearly. Here are a few takeaways from our conversation. The first one is whenever you think about expanding your product into multiple formats or multiple offerings, Be careful to not ship that complexity to the user. One of my favorite laws in design is the Tesla's law. It basically says that in every system, there's an inherent amount of complexity that cannot be reduced. The only question is, who bears that complexity? The user of the product or the builder of the product. Elizabeth is fully aware of this dynamic. Netflix is expanding into many new experiences live in games and and podcasts and mobile and more interactive formats. But Elizabeth's ambition is not to simply add more tabs or more surfaces inside of the app. It is to make Netflix better at knowing exactly what is right for you at every given moment, so that the product feels cohesive and coherent whenever you use it. This is a very hard product challenge. And the lesson here is extremely key. As companies expand, the enemy is not adding complexity. The enemy is letting that user feel that complexity. Second, as your product evolves, the measure you measure, value has to evolve with it. When Netflix was just about movies and TV shows, time spent might have been a good proxy for value. But as Netflix expands into new formats, that proxy of value starts to break down. A minute playing a game is not the same as a minute watching a movie. When you anticipate a live event, it has value well before the event started. When you listen to a podcast in the background, that's different than sitting down on the couch to watch a show. If Netflix measured all of those experiences in the exact same way, it would force every new format to behave like the old format. And this is where many companies get stuck, because they spend all this time in the past to understand the legacy metrics, but they're trying to force new

Speaker C: experiences into those metrics.

Speaker B: But new experiences often create new kinds of value that has to be measured differently. And if you cannot measure that value well, your product will eventually stop showing that value. Also. Third, AI lowers the floor, but raises the ceiling. This is true far beyond Netflix. As we can all see, AI makes it easier for everybody to get started. It's easier to prototype, it's easier to write code, it's easier to generate images. It's easier to create content. It's easier to launch something that actually looks real. The floor has come down dramatically, but that doesn't mean that the bar for a great product is lower. It actually means the opposite. When everyone can create more, the average thing becomes less valuable. More software, more content. M more feature equals to more noise. So the ceiling goes up. What will matter even more is taste, craft, judgment, originality, distribution, deep customer understanding. The ability to turn raw capability into something people actually really, really love. This was one of my favorite moments in our conversation. In the AI era, starting is easier, but standing out is much harder. I'm Tomer Coyne. I hope you enjoyed this episode.

Speaker C: Keep building.

Speaker D: You've been watching Building One. Our show is hosted by Tomer Cohen. Building One is produced and edited by Mason Cohen and the team at Coastal Production Works. This episode was mixed by Tim Boland at LinkedIn. Our team includes Rachel Karp, Sarah Storm, Dave Pond and Alicia Mann, with support from Alex Kuznetsova and Mujeeb Mehrdad. Until next time, Keep Building.