
AB Testing · 2026-07-10 · 49 min
Key moments - from our scoring
Substance score
56 / 100
Five dimensions, 20 points each
Brent Jensen, recently transferred to a new Microsoft team focused on LLM agents, introduces Jevon's Paradox to explain why AI-driven code automation paradoxically increases rather than decreases demand for software solutions. He frames this through three principles: the end of ownership (Microsoft must fight to retain enterprise position), the dissolution of business moats (customer trust becomes the only defensible advantage), and the irrelevance of duplication of effort in a commodity-code world. Using historical examples like the loom and electricity production, Brent argues that as code becomes cheaper to produce, previously unprofitable ideas become viable - diagnostic tools, debugging utilities, log analysis - creating a long tail of software problems that companies never bothered solving. Alan pushes back on whether this hiring rebound is actually happening yet, distinguishing between Gartner predictions and observable market reality. The discussion reveals tension around who builds these new solutions: traditional developers, domain experts using Claude Code, or data scientists tuning specialized models. Brent suggests the frontier model market is shifting toward locally-hosted, fine-tuned open-source models for specific tasks rather than generalized giants, positioning developers to focus on scalable architectures while data scientists handle model specialization.
Jevon's Paradox describes how automating production of a commodity doesn't eliminate jobs but increases overall demand for that commodity, leading to job growth. In code, as AI makes development cheaper, previously unprofitable software ideas (diagnostic tools, debugging utilities, log analysis) become viable, creating more opportunities than were eliminated.
Brent operates under the principle that even Microsoft's strong enterprise position via Office and existing contracts is under attack and cannot be relied upon; the company must continuously fight to maintain position rather than assume defensibility.
Customer trust is the only moat that will persist long-term, as code becomes a commodity and all other competitive advantages erode due to rapid disruption and the ability for competitors to build similar solutions overnight.
Alan argues it's not yet observable in the market despite Gartner trends; Brent claims Gartner predicts many non-hyperscaler companies will rehire at least 50% of laid-off workers by 2027 as they discover new software opportunities.
Developers will focus on building scalable architectures and systems, while data scientists move into fine-tuning specialized open-source models for specific tasks, and domain experts may use AI coding tools to solve industry-specific problems.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode contains some substantive ideas, particularly around Jevon's Paradox and its application to AI-driven code commoditization. However, the conversation is heavily padded with banter, personal anecdotes, and meandering asides that dilute the insight density. The core insight about how automation of code production creates new opportunities rather than eliminating jobs is valuable, but it takes 20+ minutes to get there and the supporting analysis is somewhat repetitive.
code is a commodity...things that no one bothered coding before, there's now going to be a return on that investment
there's not only a long tail, but a very long tail of things that could be improved by code that haven't been because the cost is too high
The application of Jevon's Paradox to modern AI and code production is somewhat novel in framing, but the underlying economic principle is not new. The discussion about quality as a system property, domain experts creating code, and the transition of roles is thoughtful but not particularly contrarian or first-principles thinking. Much of this echoes existing conversations in the software development community.
Jayvon's paradox is this phenomenon where for certain types of goods, the elimination or the automation of how we produce that good actually doesn't end up removing jobs
the purpose of the role will persist, the tools will change
Brent is an experienced practitioner at Microsoft with significant depth in AI, testing, and organizational dynamics. He brings real operational experience and has tangible insights about team transitions and LLM limitations. However, this is a casual conversation between two hosts/former colleagues rather than a formal guest interview, which reduces the value of his caliber somewhat. He has domain credibility but limited external validation of specific accomplishments.
we have to very rapidly adapt and move in a different direction or we're not going to be here in 6 to 12 months
I had built a really sweet...dashboard...very responsive
While Brent references Gartner reports and mentions specific examples (the Power BI dashboard anecdote, the college debate group project), most claims lack concrete numbers, timelines, or named companies. The Gartner statistic about rehiring 50% by 2027 is cited but not detailed. The dashboard story is concrete but brief. Most of the discussion remains at an abstraction level without sufficient grounding in data or specific case studies.
Gartner claims it is and that they further predict that by 2027 many...of the companies other than hyperscalers...are going to end up hiring back at least 50% of the people they laid off
I had used LLM...very responsive dashboard...bringing in signals near real time
Alan frequently plays the role of skeptic and pushes back (e.g., 'Because it's my job...this of course it's inevitable. But is it happening? No.'), which creates some productive tension. However, much of the conversation devolves into tangential banter about pickleball, microphones, and personal jokes. Many of Alan's follow-ups are genuine but the episode lacks tight questioning structure. The hosts seem more interested in riffing than systematically exploring claims.
Why are you being a dick?
Because it's my job. Because you're...talking like this has already happened and it's not
Computed from the transcript - who did the talking, and the words that came up most.
In this episode, Alan and Brent explore the rapid evolution of AI, the implications of Jaevon's paradox, and how organizations can adapt to the changing technological landscape. They discuss the future roles of developers, the importance of critical thinking in education, and the ongoing shifts in the software industry.
Transcribed and scored by The B2B Podcast Index.
Speaker A: Welcome to AB Testing Podcast, your modern testing podcast. Your hosts Alan and Brent will be here to guide you through topics on testing, leadership, agile, and anything else that comes to mind. Now, on with the show. Welcome to the A B Testing Podcast, the bed Best podcast in the world that's named A B Testing. I am speaking by or Alan. And I am here with another special guest host. Uh, someone who's been on the show before. Sometimes, uh, is intelligent. Sometimes is just flat out not. Please welcome Brent Jensen to the podcast.
Speaker B: Woo hoo. Thanks for inviting me back after the.
Speaker A: And. And speak some more. Speak some more. Say something, something.
Speaker B: These are words. Okay, Check out the new picks up. Freaking everything. Mike folks. Everything. So as long as I stay still.
Speaker A: I just begged for Brett not to play with the chord. We were talking and now he's doing it just to try and make my blood pressure rise.
Speaker B: As long as I stay still. Uh, with which Alan, you know, he's asked me before we started, like multiple times. Okay, are you good? Are you done messing around? I'm like, alan, it's me. We both know the correct answer to
Speaker A: that question on the podcast. When I told you, maybe you weren't here. Maybe I mentioned it to one of the guests. Other guests is I use Whisper AI to transco the transcode. That's not the right word. Um, transcribe.
Speaker B: Yeah.
Speaker A: Every single podcast into text that we've ever done. Stop it. And then put all that text into Notebook LLM. And now I can go to Notebook LLM if I want and ask, in what episode did Alan yell at Brent about eating? In what episode did Brent. Did Brent's microphone sound like crap? And it'll tell me all of them
Speaker B: isn't the LLM is great.
Speaker A: Yeah, it's fun. So. But, uh, what do we. It's good to see you. Uh, it is summer in Seattle. It's kind of warm. Uh, it's World Cup. It's, uh, controversy. World Cup. We're not going to talk about that one on a sportsball podcast, but it's happening. So that's what I've been doing. I'm watching soccer games, playing some pickleball, getting pretty good. I'm good enough now. I think I mentioned this before. The. The older retirees I play with, they'll cheat a little bit. And I let them because it's funny. And, um, yeah, that's my day. What have you been up to? How's it going? What the hell are we talking about today?
Speaker B: I asked you that just before.
Speaker A: Sorry, that was. That was too Hard of a question. Um, by the way, Brent, this is a podcast.
Speaker B: Oh.
Speaker A: And we're going to pick up right where we left off, like six years ago the last time we talked. Talking about AI and AI and AI principles. Like, like something about that.
Speaker B: Yeah.
Speaker A: Is that where we should start? No, no, we have to ease into that. That sounds like. That sounds like hard.
Speaker B: Yeah, the. Like, I, I didn't even go back and look at. Like, that would be too much pre work on air. I would say. Let's, let's, uh, let's ease back into it. Things are advancing, uh, the biggest thing in my life. Like, we did re. Record. I don't remember if.
Speaker A: If it's been a month.
Speaker B: So had I changed teams yet?
Speaker A: You had changed teams. You're on a new team.
Speaker B: Yes. Okay.
Speaker A: Um, and I forget, are you able to talk about how much the old team sucked right now, or is that for later? Uh, you know, for job security and stuff? We can. We can save it.
Speaker B: Yeah. Yeah. So. So, um. Yeah. Uh, not great. Old. Old team. It's. It's one of the places where once in my career I was. I was the golden child on the team. And then I, I started on the team. They're like, oh, my God, Brent's gonna save the world, blah, blah, blah. And then I started executing against that, and they're like, oh, wait, wait, wait, wait a minute, wait a minute. That's not what we wanted. We wanted you to just do what we said. Um,
Speaker A: a job like that, it wasn't very fun.
Speaker B: Very rapidly, uh, the way I described it is right. The. The fall from heaven is much harder than, say, you know, the fall down the stairs. Um, the team I'm on right now, it was, it was absolutely the reverse. Right? Um, the. So it was a, uh, I'm on a team now that I've actually been a partner, a very close partner with for the last 10 years. So there's no ignorance. They. They know what I can do and what I do well. Uh, they. They feel like they really need it. Um, they're. They're happy to have me aboard, like, like in a month. I haven't had so many hugs from people I work with, like, literally in my entire career. Right? They're like, oh, my God, we're so glad that you're here. We're so glad that you're here. Um, been doing a bunch of off sites, uh, this whole, uh, LLM agent, right. Part of, Part of a huge transition because there, There's a, A reality check that. That is pervasive throughout Microsoft. I'll, uh, I'll be honest on that one. But different teams are handling it differently and on this team it's, we have to very rapidly adapt and move in a different direction or um, we're not going to be here in, in six to 12 months. Right. But in this particular case, like other teams, the way that gets rolled out is more of a, adding more fear, uh, to the already afraid. Right. Which creates um, self preservation behaviors, creates defensiveness. And in, in times of trouble people have to band together because it's not, it's, it's not the world. Like it's not the case where you can, you can um, run, run faster than the other guy and the lion will eat that guy. No, it's, now we're bound together. If the lion eats any of us, it's eaten all of us. Because the lion's not internal, the lion's external, lion's market forces. Lion is how the culture is changing. Um, and this team acknowledges that. And it's more around, hey, we got a band together as a team. And that's what's going to be important first. Because if we don't nail that, the rest of it isn't going to happen. And. Right. Um, I'll just say, uh, the last Org is, is like much, much, much more on the opposite side.
Speaker A: Interesting.
Speaker B: Yeah.
Speaker A: Um, very, very abstract. We'll leave it there. I think, I think the industry has seen like even from outside this stage. I've been gone from Microsoft now for almost 10 years, believe it or not.
Speaker B: Yeah. And you missed, you missed. I mean those 10 years would have been good. You would have been proud of those 10 years. I absolutely believe so.
Speaker A: Yeah. I would have died. Uh, but what I see is, yeah, there's been some ups and downs and now, uh, I hear, you know what my network tells me is, uh, it's, it's almost like. And this, you don't have to even comment on this. It's almost like you're not in front as much as Microsoft has been in front. You know, maybe parts of the. Org are. But there's a lot of, there's a lot of external forces, um, like more competition in some areas than you had before that the, we don't, we don't have to go. This is.
Speaker B: No, no, no. I mean, I'm going, I'm going to say it a different way. Right. So the hyperscalers all face this, right. Whether you talk about Microsoft or Meta or Amazon or gcp, like they're all facing the same Sort of thing. Um, the issue is as companies like, let's say anthropic. Who. Who I think, quite honestly, from my point of view, they are the ones that are staying, uh, laser focused on what their mission is. Right. And that is how to enable more of task completion with this tool they've built and a series of tools that they've built. Right. Fable, um, is fantastic. Um, Claude code. Claude coworker. There are fantastic. Um, and it's. It's. Right. It's all just tools now. And the. I've been given a lot of speeches lately about Jayvon's Paradox. Are you familiar with that?
Speaker A: Not familiar with. And probably none of our three are. Or one of them are. Congratulations. Whoever that is.
Speaker B: Uh, and. Right. It's code is a commodity. Or so I am. Uh, I'll just say a couple of principles that I am currently.
Speaker A: Then eventually we get to the paradox.
Speaker B: I will.
Speaker A: Okay. Um, Just checking. Because you never know.
Speaker B: Valid. There are three key principles that I am living in today, even though none of them are necessarily true. But what I find is if I just live as, uh, if they are true today, uh, when they actually become true, I'm going to be more prepared than most. Okay. Number one, there is no such thing as ownership anymore. Right. You. You are. If. If, uh, what would you say is a key differentiator for Microsoft versus the other hyperscalers?
Speaker A: I don't know. Um, people who refuse to quit even though they don't know anything anymore.
Speaker B: No, no, no. I. So market side,
Speaker A: like, why would, uh, businesses. You have all. You have all of the big business contracts for office.
Speaker B: Enterprise. Sure.
Speaker A: That's what it's called. Enterprise. I've been out for a while. I've been playing pickleball. I forget the words.
Speaker B: So, um, Enterprise. Certainly, uh, there's a lot of people that share that opinion. Um, so I am moving forward as if Microsoft no longer owns Enterprise.
Speaker A: Good.
Speaker B: Right. We have to fight to keep our position there. I, um. Another one. So we were just talking about Microsoft. Right. Many people are gonna, um, disagree with me on that one. And I'm just saying it's fine. But I choose to live as if that is under attack. Right. And anything like. You can't rely on any moat. Do you know the definition of a moat? Okay. From. For the three emote in business language is that thing that protects a key part of your business. And I'm moving forward with the idea, um, that I think is eventually going to be the case, um, that the only moat that exists is customer trust. Right.
Speaker A: Sure. And we'll come back to that. I get. I'm just, I'm on the edge of my seat waiting for the paradox. I'm not sure if we're any closer or not.
Speaker B: Fine. Um, the other one I'll say is, um, there's no such thing as duplication of effort anymore. That used to be a big philosophy within Microsoft, but it wasn't.
Speaker A: It wasn't. But go on. I'm going to fight.
Speaker B: But it doesn't exist anymore. And the reason why it doesn't exist is because you can't stop it. Because disruption is everywhere. Because you don't own anything anymore. So if I have a good idea today, code's a commodity I can build. Or, uh, if you have a good idea today, and I go, oh, Alan, it's a good idea, but, you know, you didn't go far enough. I can build the new thing by tomorrow.
Speaker A: Right.
Speaker B: Uh, it's, it's you, you. You don't get to lick the cookie anymore. And because I can just build a new cookie overnight. Does that make sense? It.
Speaker A: In a sickening way, yes.
Speaker B: Okay.
Speaker A: Still waiting for the paradox.
Speaker B: So the paradox is.
Speaker A: Where's my applause Down. I don't have it. Anyway, go on.
Speaker B: All right, so Jayvon's paradox is this phenomenon where for certain types of goods, the elimination or the automation of how we produce that good actually doesn't end up removing jobs. It actually creates more jobs in that same space. Okay, so, um, uh, an example of, of this might be coal or electricity. Like, we, we,
Speaker A: we,
Speaker B: we figured out how, uh, new and wonderful ways to make it cheaper to produce electricity. But because electricity is sort of, um, a basic resource, right now that we have more electricity available, um, there are more things that we could do that previously we had to prioritize.
Speaker A: Okay.
Speaker B: Okay. Um, one typical example of Jayvon's paradox I remember is like factory workers on a loom, like, they would have a loom and, and, uh, uh, the old, the old loom. And a guy would sit there and they, let's say they could, they could loom together, uh, one sheet a day. And then the next innovation, and this part's all true, the next innovation is they change the design of the loom. And so that same one person can now do three sheets a day. Well, back when it was one sheet a day, really only the rich could afford buying raw fabric because it was a very limited resource. But now that the same amount of effort boosted the productivity, um, then they started having a glut of that resource, which for those who haven't taken economics, um, the more you have a supply, if the demand isn't there, then the price of the supply falls, falls, which then also has a tendency to increase the demand. Right. And so now you had, as we produced more and more fabric, you had um, fabric being exposed to say the middle class. Right. And then factories came along, massively produced it. Right. Um, now you have, oh, I just need a little bit of fabric because I could now, you know, patch my clothes or whatever. Right. It. The way Jayvon's paradox works is essentially. Yeah, it will eliminate, it will eliminate, um, jobs and the, the old way of thinking. Um, but because the demand then goes way up because it's a lot easier to produce these uh, things.
Speaker A: Let me clarify understanding because it's been a windy road and so I'm going to just draw it straight to AI. That's what you're talking about here. The fact that today we have more efficient chips and we've lowered the cost of computer, so now we run far more compute intensive stuff, so we still don't have enough commute and we're still drawing more energy, we're still using more water, all those things. That's the paradox. Is that correct?
Speaker B: That's part of it. But here, what I was talking about is code. Code. So previously code. In order to produce more code, you needed more people who went through significant training.
Speaker A: Yes, you did.
Speaker B: Right?
Speaker A: You need, uh, that part makes sense. Keep going.
Speaker B: But now, but now we have the ability to produce code at a whim. Nearly everybody.
Speaker A: Yeah.
Speaker B: Right. Which means, um, things that no one bothered coding before, there's now going to be a return on that investment.
Speaker A: Yes. And I think you and I have talked about this. I talk about this all, all the time when I talk about AI. I, um, was talking to a company where I may or may not work and I think I scared one of their test leaders away when I talked about, um, all of the diagnostic and debugging tools that we never had a chance to write now become so easy to write. I think there's such a open opportunity for that. I remember working on graphics in early versions of Windows and spending weeks and weeks writing something that would allow me to analyze, uh, fonts at a minute level and see where gray scaling was happening, things like that. And those things could get whipped out very, I mean, I should say weeks, months probably.
Speaker B: And there are, there's. We both could go back in our career. There's a large number of decisions I know I made, uh, in the past where I'M like, that's going to take forever.
Speaker A: Testing. I don't want to use AI for anyone. Devs, testers, PMs, I don't care who. I don't want people to write more tests with AI. I want them to write all the tools they never wrote because they were harder to write than the tests.
Speaker B: I'm fine with them doing both.
Speaker A: Okay. They can do both. It's just as easy because code's just an easily creatable artifact.
Speaker B: Right?
Speaker A: It's. Yeah.
Speaker B: Right. But now what we. What where, where there's an opportunity in.
Speaker A: Um.
Speaker B: I may have shared this before. Let's just agree, much like the examples that we talked about, that there's not only a long tail, but a very long tail of things that could be improved by code that haven't been because the cost is too high.
Speaker A: Yes.
Speaker B: That's all now possible. That's where, that's where opportunity definitely exists. Right.
Speaker A: Um, agreed. What do you do with it?
Speaker B: And so what Jayvon's Paradox is essentially saying is because it's not only because code is a commodity, but what code represents, which is the software based control of some action or activity. Right. Which, which if you think, if you think of code that way. Right. We have, we have far from saturated the market on using code for that purpose.
Speaker A: Yeah.
Speaker B: There's a lot of opportunity. And so what, by bringing the price down, we now open up the ROI for going after that opportunity.
Speaker A: Right.
Speaker B: I think everyone knows the need for more builders, not less.
Speaker A: Yeah. It creates the need for more ideas, for sure.
Speaker B: Yes.
Speaker A: And so let me talk about two things in there that I think are important to mention.
Speaker B: And actually I don't even think it creates the need for more ideas because there's plenty of ideas that already existed that, that people have that, that they. Because it was too expensive.
Speaker A: Yes and no. And uh, I'll talk about lying in corporate America as well. A lot of companies and everyone knows this as well. None of these are secrets. A lot of companies are laying off people due to AI and it's in zero case. Is it because AI is replacing what people used to do? Because there are a virtually infinite number of ideas for products for that within a company with new companies. We have not exhausted the idea, the, the well of things that could be solved with software, especially now that they. The investment side of ROI is so incredibly cheap to create things. The layoffs in tech have happened not because in some cases they're trying to save, trying to save money to spend more on data centers. I don't Even think that's it. I think a lot of companies over hired during the pandemic and it's a big freaking messy adjustment. AI has something to do with it. But I am waiting for. There's going to be a rebound at some point when companies slowly start to figure out what you and I are just talking about. That there are so many things that can make lives easier for so many people in so many different ways that had never been built before because it was too hard and it's just not difficult anymore.
Speaker B: That's already happening.
Speaker A: Where um, uh, Gartner uh, did back with extra tricks.
Speaker B: Gartner just did a report on this for several companies and they actually I'm
Speaker A: saying as a consumer looking what's out there, I'm not seeing it.
Speaker B: What, what even further so that they're noticing this as a trend. It's at the beginning of the trend.
Speaker A: Well then.
Speaker B: But I think at the beginning of
Speaker A: the trend doesn't mean it's happening. It means we both all know it's going to happen. So I'm glad it's starting.
Speaker B: Uh, okay.
Speaker A: It's not there yet.
Speaker B: Why, why are you being a dick?
Speaker A: Because it's my job. Because you're, you're, you're, you're talking like this has already happened and it's not. I will guarantee you at least two of our three listeners are on my team and say yeah, this of course, this of course it's inevitable. But is it happening? No.
Speaker B: I'm going to claim that Gartner claims it is and that they further predict that by 2027 many uh, of the companies other than hyperscalers, other that they're not flagging companies like Microsoft but companies that um, bought the bullet and said ah, we're going to do this. Not tech companies that they're going to end up hiring back at least 50% of the people they laid off.
Speaker A: Yeah. Which is good. They absolutely should. Let me give you an example. Uh, just to walk from. And again this may be happening. In fact it probably is happening in the tech world already. So like let's have agents analyze our logs and tell us what's going on versus trying to parse through them. I'm sure that's, that's much fat. We never had the time to write those. That's great. But let me give me. And I'll make something up here. Um, real estate. Real estate websites suck. Um, they're basically shitty. Um, I don't need to hire. The question I have is there are a bunch of different Things, there's a bunch of innovation happened there, a bunch of things to solve for both buyers and sellers and um, um, agents, all those things, all kinds of problems that could be solved there. I don't know that software developers need to be solving those problems because real estate agents who have all the domain knowledge, given a little bit of coaching on how to prompt uh, claude code in the right way can probably solve those problems for a lot of realtors just as easily. So ah, do we, do we need to hire, what do the developers do in the future? Because I don't know if they're the best people to be creating code with. Agents.
Speaker B: Um, I believe, um, very similar to what, what we've been saying around um, testers, right. The, the purpose of the role will persist, the tools will change. Right? There is still going to be a need to build scalable systems. Um, there's still going to be a need to validate ah, customer quality. Right. But the tools will change. Okay, right.
Speaker A: Uh, similarly, are you saying AI coding tools can't create scalable solutions?
Speaker B: Not today, no.
Speaker A: I bet you there's a Gartner statistics. There's a trend that says they are. But you get what I'm saying, right? That that's not, that's not a reason that we'll need developers in the long term. I'm not saying we don't.
Speaker B: I just think that the biggest thing I would say is right, um, AI still has a massive cognitive limitation. There are a bunch of trends that are, that are coming. Um, there's several of them that I'm, I'm absolutely fascinated by. Like um, there is this because the price of, because the price of the frontier models is, is surprisingly going up, not down.
Speaker A: Mhm.
Speaker B: Right. Um, uh, it's created a market for uh, these engineers that go okay, how can I, how can I do local hosting of models, right, using open source models and how can I tune the hell out of those? Um, or use different techniques to get near ah, frontier quality or near for quantity quality for the tasks that I have at hand. And I'm like yeah, go innovators. Because, because the thing is LLM, in my humble opinion LLM is going to become a commodity itself sir. Whether it, it's done because of, of like what I was talking about an individual in their garage building taking an open source model and creating a marketplace for finely tuned smaller models for specific focus. Right? We don't need, you don't need a generalized model for, for all of your tasks.
Speaker A: So what you're saying is it Makes sense. Um, and Gartner would say already moving there in the long run, move away from the large general models we see today towards the three or four or ten we have to a whole crap ton of tuned for specific task models.
Speaker B: Yeah.
Speaker A: I extrapolated and expanded based on what you said and I like that idea. And you're saying that's what developers do is they build new custom LLMs.
Speaker B: I don't know if developers.
Speaker A: I don't know if I have an answer for this. I just think that. And maybe one model.
Speaker B: I think so that last market is something I think data science moves into.
Speaker A: Sure.
Speaker B: Um, what developers will be freed up certainly to build up more um, scalable architectures I think.
Speaker A: And you sound a little bit like M. Ever see that cartoon of this is like from 15 years ago, Alan.
Speaker B: And we just move on. You don't have to figure out your metaphor.
Speaker A: It works at web scale. Um, yes, I hear you say scalable. So maybe a hybrid of that is in the short term, um, because I think you just need. The only thing you need to build scalable software From M. What LLMs do is a tuned LLM that works on building scalable software. That could be. Someone could go solve that soon. So but in the short term maybe what happens is I, I really like the idea of domain experts creating much more of the software. Like here's I want real estate to come to the ID department and go like okay, here's uh, we created this thing. Can you make it deployable to our whole fleet? And then maybe that's, that's what the devs do. But I don't know like how can. I don't know that a dev can prompt ah, something outside of their domain expertise better than that person in that domain can. So I, my prediction is I think those domain experts are creating a whole crap ton more of the code than they are today. Yes, I know they are. There's going to be more but there's still a role.
Speaker B: Let me give you a very concrete example that even with the current models I'm not certain is fixed. Okay. Um, this, this, this happened in my last team a couple of months ago. Um, I had used LLM. I had built a really sweet like um, I could care less about Power BI anymore. Right. I'm a data scientist. I do a lot of presentations. Okay. But building the Power BI because of the DAX language, the DAX is very complex Flex, they don't have copilot integrated into that in my view the correct way. And it's just as easy for me to tell ask my buddy GitHub copilot, hey, now that we've done this, can you build a dashboard? Right. Just as easy or uh, it's even easier than diving into power bi so I did, I had a sweet dashboard running, bringing in signals, near real time, very, um, responsive. And then so this ic, um, basically, uh, a couple years into her new career, hasn't done anything with, with data science or uh, with development. No training in development at all. And all of her training on data science she actually did in my team. And I said I, I'd like you to take this over and I want you to extend it this way. Okay. And she did. And she's like, hey, Brent, I want to show you I did it, I integrated, I added these new things and I, I looked at it. Uh, absolutely. She showed that she had achieved the what. What I had asked her to do. But the whole app was sluggish af and I'm like, I feel like without you really noticing it, that the LLM added this job to the main thread.
Speaker A: Yeah, we've heard the story. What's the main thread?
Speaker B: Exactly? And this is a place where, where right. LLMs still are at a point where they give you what you ask for, not necessarily what you need. And so you're still, as a, as a builder sitting down, you're still limited by knowing what you know.
Speaker A: And this is. And yes, and I want to build on that because the challenges if you and I. This is the same thing. Your other employees, a great example. If three developers are all given the task of implementing something, the, the way you prompt, the way you give context through the chat interface, the LLM, we will get three different solutions.
Speaker B: 100% true.
Speaker A: Which is uh, and again, the um, the real estate person, they're not going to know what the main thread is, but they're. But. And so yeah, that there's some architectural things they won't know. But the LLMs can be taught to take care of those over time. But the ability, the critical thinking, the judgment, the creativity. I absolutely abhor the term prompt engineering, but I will say that everyone creating code, not necessarily a developer, everyone creating code needs to be able to, um, need to prompt in a way that gets answers. I'm actually, I'm going to back it up from code. Uh, there is like, there are ways to prompt an LLM M about anything that will give you better answers or worst answers. And I want to take us all the way back away from computer software. Who gives a crap about that? Just like let's talk about schools or education. Don't tell kids to avoid AI. It's not going away. Teach them how to use it. Teach them how to ask. You ask questions. Help. Help them learn how it works by prompting it in different ways, understanding what comes out.
Speaker B: I uh, am of, I am of a mixed emotion on that one, but
Speaker A: there are so many. This is the same like people would come to me years ago and have me search for. They, they'd say I can't figure out how to do blah, blah, blah. I said, you search the Internet. I said, yeah, because could. I said, why you coming to me? Because you search better than I do. And.
Speaker B: Yeah.
Speaker A: And um, and it's, it's that times 10 with chat interfaces to LLMs when it comes, you have to be careful
Speaker B: when it comes to kids when it comes to education. Okay. There is a, there is a trend that I find disturbing and I think educators, uh, I'm going to do my part to think through this, but I think educators have not struck the right balance around teaching it, the tool as
Speaker A: a tool because the teachers don't know.
Speaker B: And that might be it. So let me explain the problem. Um, there's a growing trend where students are not learning except for how to get the LLM to give the answer.
Speaker A: Yeah, well that's just poor teaching.
Speaker B: It's not poor teaching.
Speaker A: Yes it is.
Speaker B: It, it's poor testing. It's not teaching at all.
Speaker A: I disagree.
Speaker B: It's the one, one example I, I read about just a, uh, couple of weeks ago was uh, around. There was a group project. It was, it was written up by one of the members of the group and, and she knew the majority of the other groups of her, of her own group were using LLM to produce slides. So they had, it was a debate class and they had to. The teacher split uh, the class into this side supports, this side refutes. Okay. And this was a college level class and this person was writing out and said the majority of the other slides that came out of my team, they produce the slides and they were able to talk through them. Okay. But they didn't understand them even to the point of not uh, failing to understand that they were arguing for the wrong side. Right. And so, so I am absolutely concerned about the construction of intellectual laziness.
Speaker A: It's, it's uh, teaching. This is what I'm talking about. This is why the answers are so easy. Just like codes, easier to create than ever before. The answers are easier to find than ever before. We need to teach critical thinking and Judgment and evaluation and those skills, those help with using LLMs, those help understand those things. Those things aren't happening.
Speaker B: Um,
Speaker A: and using chat interfaces on LLMs, it is a fantastic opportunity to teach those things, but the teachers don't know how to take advantage of that either.
Speaker B: I'm going to take it further, and I don't even care if it's the teachers that do it. This is something that I've talked with all of my kids. Um, we'll pretend you're one of my children. Alan, what is the job of a teacher?
Speaker A: And the. The. The, uh, easiest answer to give is to teach.
Speaker B: Correct.
Speaker A: And it's an incomplete answer at best.
Speaker B: It's. It's not only incomplete, it's wrong. Okay. The job of a teacher is to get their students to learn. I don't need. I. I don't care if you teach it. But your job isn't to teach. You're you, you, you. If. If you gave up, went up there and gave a brilliant presentation and none of your students learned anything, you may, as a teacher, go, well, I taught. They're just morons.
Speaker A: Yeah.
Speaker B: And I'm like, it's a little. Not your job.
Speaker A: I. While I agree with what you're saying, it is a little semantic because they're just not teaching. In that case, they're just talking anyway. Go on.
Speaker B: That. That. No, to me, that. That is not semantic. It's actually the single most important thing. It's the same thing to me. Like, what is a leader? How do I know if you're a leader? Well, you have to have. Because you have followers. You. You cannot be reorged into leadership.
Speaker A: Well, and there's. And there's. This has been a much better podcast. We just talked about the difference. Like, just because you're a teacher doesn't mean you can get your students to learn. Just because you're a leader doesn't mean you have followers. You. Those are both things you need to develop and earn with. With your team or your students.
Speaker B: Teacher,
Speaker A: can we start over and talk about that?
Speaker B: I got. Well, no. I'm supposed to take my daughter to an errand. I'm like, I got nothing else.
Speaker A: Let's get to the part of the podcast. What's the takeaway? What's the point of this discussion today? What did we accomplish in this slight debate? Discussion around.
Speaker B: AI Believe. I absolutely believe I sufficiently covered Jayvon's
Speaker A: paradox that was in there.
Speaker B: Yeah. Um. Uh, I showed that my new microphone is awesome.
Speaker A: Yeah. And if you're listening right now, wonder why his voice sounds better is because it's actually within a foot of the microphone versus sitting back in his chair.
Speaker B: Really? Is that noticeable?
Speaker A: Oh, yeah. It's much better than last time. People are. There will be. There will be positive comments about your microphone. Yeah. This is what I've created. I've created Brent sitting back, leaning forward in his chair, doing a. Doing a part of a Gregorian chant.
Speaker B: Uh, what else are we taking away then?
Speaker A: I want to preview next episode where we have a little bit more sensible topics in order. But, like, what do you want people to take away from this? Javon's paradox. What's happening with AI?
Speaker B: Uh, lots of things. The thing. The thing. The thing. I think that's important and feeds into what it. What I hope will become, you know, the modern.
Speaker A: The postmodern testing principles.
Speaker B: Right. Or, uh, whatever the hell we name it, but. Right. It's very similar to when QA started to disappear.
Speaker A: Yeah.
Speaker B: Right. It's. No, you gotta head. You gotta understand what has actually been done and make the changes that are necessary. Right. Um, back then, it was testing as a tool for assuring quality is better suited under Dev. Right. But quality, if your identity is around. My purpose is to deliver quality, like Alan and I. You agree that quality is defined by the customer, not by you. It's not craftsmanship, it's utility. Right. Where we're in the same sort of transition, like we're just changing the tool set.
Speaker A: Yes. And I'm a big believer you hear me talk about this or write about it, at least. Quality is a system. It's produced by the system, which is everything, including not just what the agents and the people are doing, but what's rewarded, what's incentivized, what, uh, what. All the little things that make up the system of delivering software. And that system is changing a lot. So when you change one part of a system, you have to be aware of what it's doing to the other parts, and that's going to be something else good for us to talk about as we get into whatever postmodern testing is.
Speaker B: Absolutely.
Speaker A: And I look forward to that. So are, uh, we on a regular schedule now? Do we talk every two weeks?
Speaker B: Yeah.
Speaker A: So we. We are back on the regular podcast thing. I, uh, still have some. I might, I might. I can't decide if I want to subject guests to branch or not do on my own, but we, uh, might get some of those in there, too. But I am excited to m. Talk about.
Speaker B: Things are changing and. Alan, I honestly don't care.
Speaker A: I know you Don't. Yeah, I know you don't. And you be you boo. And maybe to recap and when we first started this podcast it was because testers saw agile happening and didn't know what the hell they were supposed to do.
Speaker B: And we saw a path and we
Speaker A: saw a path and a pattern that was happening industry wide. We began talking about that and expanded on that. And the first time I Talked about this 10 years ago at um, a test conference, uh, I thought I was going to get booed off stage, but I, I did not. And people come to me and said thank you for giving a name to what we've been doing. So it wasn't. We're not trying to tell describe some new way we're inventing. We're trying to describe what's actually happening. And I think what we'd like to, what we're moving into, if I understand and I think we're on the same page on this is we're going AI shit's changing even faster. We'd like to talk about how teams, not just testers teams and, and I wouldn't see not even just engineering teams. I would say how organizations adapt to how the world is changing with AI. And I can bring the quality of the system part to it and you can bring the knowledge about AI that I will make fun of part to it. And we may get somewhere that's kind of exciting. Call it the AI Testing. What's your middle name? Start with an I?
Speaker B: No.
Speaker A: Okay.
Speaker B: Yeah.
Speaker A: Maybe you can legally change your name. We may need to be the AI Testing podcast. But just something, just a random idea there.
Speaker B: Um, the ah ah, abort. Right. For the software industry, I think this is my view. I think this is our last major shift and what comes after that is going to just be continuous path changes from here on out. Once the engineers and participants in software make this shift, then it's just going to be a continuous shift. If it's not already there.
Speaker A: It'll be a fun ride. Get on the roller coaster.
Speaker B: Woo m. Yeah.
Speaker A: Wait. Get on the roller coaster.
Speaker B: I'm trying to remember. There was a movie.
Speaker A: Thank you. Thank you.
Speaker B: I forget the movie, but there was a movie where this, this grandmotherly lady talks about there are two kinds of people. Those who like the roller coaster and those who like the merry go round. Right? And the merry go round is nice and safe and comfortable. But the roller coaster is exciting.
Speaker A: I'm so bored on the merry go round.
Speaker B: I think software industry, like if you're a merry go round person, it's so
Speaker A: you want me to be.
Speaker B: You're in for a lot of anxiety going forward, I think.
Speaker A: All right. I think it's important to adapt. And we. Adaptation, um, is more important than ever been. So is a bunch of other things. We're kind of out of time. We got to go. Like, um, do you have more wiring to do in Satya's house? You all the GPUs installed?
Speaker B: No.
Speaker A: I'll go with. Okay, well, you might gotta. Brent has to head back over to Satya's, install GPUs in the attic. It's for a big lighting thing he's doing for Christmas. It's gonna be a lot of fun. Um, but it's good to see you again. And, uh, I am, um, Alan or spy.
Speaker B: And I'm, uh, Igor. Voice.
Speaker A: And see your face in my dreams.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.