Credit Anxiety: When Nobody Knows What AI Really Costs

Speaker A: Foreign

Speaker B: what happens when the entire software industry gets repriced on the cost basis of AI? When AI, uh, procurement agents are pitting your product against five competitors in real time speed trials and when every project management tool builds the exact same agent platform? Welcome to Unsolicited Feedback where we dig into the messy realities of building in the age of AI. I'm Brian Balfour, Founder and CEO of Reforge and joining me today is a good friend and Aaron White. Aaron is the former founder of Price, intelligently and blissfully acquired by Vendor where he served as CTO for a couple years. I love talking to Aaron as he is in the thick of building AI tools and his company in a completely different way for the AI era. He always has a forward looking take. Today we're pulling back the curtain on a massive shift happening right now. The entire industry is scrambling to shift from all you can eat seat based subscription models to credit based models that few consumers actually understand and all of the secondary effects that emerge from that. We also tackle the launch of Notion's Agent platform and how it feels like every project management tool from Jira to Glean to Notion to Monday has the exact same strategy. Grab your coffee, keep listening and let's turn these insights into your unfair advantage. But before we dive into the full episode, there's more than just the AI market that's been on fire. That includes the reforged product team. We just launched another new product, Reforge Research, which combines an AI user interviewer with AI surveys to completely change how you capture capture net new insights from your customers. Product teams typically face an impossible choice when they want to learn new things from their customers. They either need to blast out surveys to get shallow responses, or they can run human led interviews which are super time consuming, expensive and complicated and have really long cycles. And with both of those, you're still stuck manually synthesizing all the results, trying to join it with all your other data and trying to make it actionable for other teams. Research just ends up being a huge bottleneck. Well, with reforged research we've completely changed that with our AI surveys and AI user interviewer. Our AI uh interviewer generates a discussion guide from scratch based on your learning goals. That discussion guide guides the AI interviewer and it runs a dynamic conversation just like a senior user researcher would. Your customers, they can take it on any time, any schedule they want to and conduct it in any language. Especially for those of you with international customer bases, they can do all of this without any coordination hell required. And finally, all of those results automatically synthesized and analyzed in real time, joined with all of the other important business

Speaker A: metrics and data about your users.

Speaker B: Completely change the way that you do user interviews and customer research. Open up new use cases that were completely not possible before. Go to reforge.comresearch to find out more.

Speaker A: I feel like this is going to be a, uh, never ending Fareed and I touched on it on a previous episode which is just uh, the actual costs of AI, how that's translating to monetization models which then flows through honestly to everything else like acquisition models. Because this year's been a whirlwind of where we started and where we are now. I am personally feeling it on the Claude side of things. I had to spin up two Claude Max accounts just to get the work I was doing done. Uh, because I was getting cut off so quickly. But it's just not just them, but it feels Claude chatgpt cursor all the AI prototyping companies have gone through massive big pricing changes around this and so there's multiple ways that we can attack this topic. But as you've seen all of this play out, I'm just interested from your perspective, Aaron, kind of what, what's going through your mind.

Speaker C: This hits a real sweet spot for me because the last two companies I was involved with, one was helping SaaS figure out pricing, which is a hard

Speaker A: problem for some reason. It always just like doesn't compute for me.

Speaker C: But yeah, but it's like, you know, there's, there's cost based, there's competition and there's sort of value based and everybody's

Speaker A: moving to cost based right now.

Speaker C: Like that's moving cost based, which is like not great but also maybe required. And then the other company was helping figure out where the hell you're spending all your money on software, which to your point, by the way, how many people in my org am um, I outgrading because they exhaust their limits and I'm like, this is the best $200 I'm going to spend all week. I'm going to do it anyway. But boy, am I getting boiled alive by, by. I, I think it's really interesting because there's real utility in AI, uh, real utility top to bottom, but it is currently expensive and it's wildly unpredictable. Right.

Speaker A: Like the expense is unpredictable. Yeah. The distribution of your users that Faree talked a lot about, this is massive. And that used to not matter, but it really matters now because to Claude's point, like their top 5% were costing them so much Money, Right. That uh, because they were doing things that they weren't expecting like running Claude code in the background 247 just consuming tokens nonstop. So.

Speaker C: And you know, why not, right? If you give me an all I can eat buffet pricing, I'm just like sure, Claude, spin up 4 million experiments and then you know, choose the one that happens to work rather than me thinking a little more thoroughly up front. Like seems like a good trade. I think we're in like the tweener, no, no man's land stage of this. We've moved to. Okay, well if we can't figure out a distribution across our customers that prevents us from getting eaten alive on the back end, then we, we, we don't have enough data to predict it or it's changing too fast, there's no formula to look at, then we must move to cost based pricing. And that seems to be where a lot of folks are going. At best they have a hybrid model. But a lot are going to cost based pricing. And I think the problem with that is it is really opaque to an end user what their costs are going to be.

Speaker A: Yeah.

Speaker C: So let me give you, let me give you the like the best examples and the worst examples. I love making AI videos, uh, for my friends or for advertising. It's just a lot of fun. My vibe coded up pipelines to do this. It's really predictable. I know what it's going to cost me per second of AI video and you know, the length of a video is uh, a pretty useful metric. I don't know how many iterations going to take me, but I know what I'm getting each iteration. Now you contrast that with something really useful like a Manus which can go out and do a bunch of deep research and project work for you. I have no idea what the token costs are going to be upfront when I ask it to do a thing. And yet I'm being charged on a to a credit basis. No clue. I have no clue. I have no hope of knowing. And the AI frankly has no hope of knowing what it's going to cost either. Right. And you kind of get back to the same problem which they don't have nearly enough data to even help predict what the cost is going to be. So as a consumer of these credit based systems, I'm left either massively overpaying for credits. And I know it. I'm constantly being nickeled and dimed or you know, boiled alive like a frog pulling more credits out of me. And I just feel frustrated. And the worst one in the middle for both the product builder and myself is the one where I'm hesitant to use the product because I don't know what it's going to cost me. And then I don't engage as much and you lose your, you lose understanding of your customers providing them value. And I certainly lose the value I would have gotten. I find myself in that messy middle bucket very frequently with these credit based products. I don't have a good solution for this. I'm not sure anybody does. I think the best are probably these like hybrid methods. Yeah, but I don't know, I like it's a tough spot.

Speaker A: Where do you think it ends up? Do you think things get more stable, the underlying costs get more stable and therefore more predictable and as a result we can move away from this credit based pricing? Or, or do you think this is kind of a permanent fixture of, you know, the next decade of software?

Speaker C: Let me go on the record and say I think it'll be a permanent fixture of the next phase of software, but in this in between, we're going to try to fight like hell to return to models that we like. Yeah. And let me justify why and why it's going to be complicated in the middle. I mean, we even talk about the house of cards dynamic that somebody was tweeting about the other day, which I love, which is like, wait a second, so VCs are subsidizing companies that subsidize token costs to customers, but the VCs are subsidizing the people, you know, serving the tokens up, which are, you know, paying for graphics cards, which are being propped up by public investors expecting more. And it's like what happens when we reach a point where all that subsidization gets collapsed out? Do costs skyrocket? Do we thread the needle? It might be bumpy, but here's, here's my thought process. I think human beings, we don't have a lot of time, we don't have a lot of cognitive capability to devote to pricing. So we're looking for the fastest path to smart decisions. Right. It's like the reason things, we price them flat monthly in a lot of ways or based on the packages or the utility. The value metric, which is probably the most useful is it's very easy for us to say, okay, I'm always getting a good deal here, let's go for it. Right. The cost plus pricing requires me to understand your operations and then compare them to mine, which is a huge cognitive load. So I think we're going to fight like hell to not do cost based pricing because it's just super unnatural as human consumers. And look, one of the reasons we don't do that today is because compute is so effing cheap, right? Generally speaking, compute is so cheap these days with modern techniques. It's just that this new graphics card based LLM compute is particularly expensive right now.

Speaker A: Right.

Speaker C: That said, ultimately, as a consumer of AWS or any infrastructure platform, you are paying for units of compute and units of storage and units of bandwidth. They have boiled it down to the cost like entirely. And we are okay with that in those scenarios. And so I think what's going to happen is we're going to fight like heck that if humans are doing the purchasing decisions to not add this cognitive overload. But at some point in the future it will come down to it is a intelligence, is a utility, like electricity. Like I actually agree with Altman there on that one. And then we're going to find ways to just have the AI itself thread the needle better for us on the back end and just do sort of utilization. Pure utilization.

Speaker A: Yeah. The other side of this equation, there's costs in how the com. In how you price your product. The other side of this equation is willingness to pay. And that's the other thing that I've been trying to wrap my head around because even in just this year, you watch the willingness to pay for these products drastically shift. Like if we were looking at Reddit threads of Cursor or one of these other products communities in Q1, the Reddit threads are dominated by things like, my God, like this thing for 20 bucks completed, this thing that would have taken me two weeks and you know, maybe 10 grand of developer time and they're like mind blowing.

Speaker B: But you look at all of the

Speaker A: Reddit threads now and Q3 is littered with just angry customers of like, oh my God, this thing cost me 10 credits and it like shouldn't have. And it's an indication of how the commoditization of uh, this starts Amazing, like for con, for consumers. But over time I think there's this whole narrative around how, you know, it's going to tap into human labor budgets and people are going to constantly compare like all of this AI to what it might have cost them in human time. But you can already see that that is actually probably not true, that just in a matter of 3/4 for something like cursor, the people who are using cursor have lost all grasp of, of thinking about, oh, this thing would have cost me two weeks and you know, 10 or 10 grand of developer time to it automatically the price pressure being pushed down to thinking of it more like true software. And I think people just value software differently. And the more that AI becomes more and more commonplace, AI starts to look more and more just like how we think about software, not how we compare it to what it would've cost us in human time. So that's the other side of this equation. And, and so like to your point, if costs just skyrocket at some point, if prices just go up at some point, then there's like this whole question of is the willingness to pay still there for some of these use cases or not? And maybe some of these use cases just implode.

Speaker C: I, I, I want to push back in a way on this, but I, I, you see a lot of complaining. The Internet's great at servicing complaining. Fair enough, right?

Speaker A: It's especially Reddit. Especially Reddit.

Speaker C: So uh, especially Reddit. And, and that's where I go to complain after X. I try to complain in multiple mediums. The so you see that on the other hand their revenue numbers are astronomical. So it's not like, you know, it doesn't seem to actually be reflective of true behavior. I know, I remember when Devin came out and they said they're priced at 500 bucks a month and people were like that's insane. And I'm like all day, every day, give me the thousand dollar a month plan. Like that is so cheap when you consider that the time it takes a human even just to fix a copy change on an app is like measured in minutes, you know, plus following up on the deploy like no brainer, but we are dumb monkeys at the end of the day and we're on the hedonic adaptation treadmill. And so when you, when you're used to like okay, well I can just generate up a ton of software using these agents now. But also the bar of what expectations for software is going up. The bar is going up for how quickly I can get it out the door. Suddenly it just becomes your background condition and you start complaining about these, these costs. But to me this just comes back to, it's not that the costs are out of whack so much as people have no expectations that they can anchor on because credits are a really weird way to measure productivity. I don't know what, I still don't know what that is. Yeah, yeah, you know, I, I would not be able to, to predict. It's like doing stupid um, uh, you know, estimate poker for agile. Like I, if we're doing credit poker for agents to bid and have, you know, agents bid back to me. I think I could do this for a hundred credits. Maybe that's the model. I don't know because I, I have no ability to, to, to guess this. But what I, I, I have enough data from my career to suggest that we are saving a ton of time and we're doing more. I think the back pressure will come the other way though, which is like, yeah, you can complain about how little you're getting relative to your new expectations, but what's your alternative? Are you going to go hire another person? It's still vastly cheaper than that, I guess.

Speaker A: I just think it's using Devin as your example. Right. I just feel like in three years, maybe even less time turn it, that alternative that you're comparing to just completely disappears. Right.

Speaker C: Like you will not the human alternative.

Speaker A: I'm not saying humans disappear. I'm just saying in your mind, the alternative that you compare that to will not be humans making copy changes and it will be to some other piece of software, some other AI software and

Speaker C: it'll be a race. So, so I think it could be, you know, we could, if you're going to interview agents, we know how to interview humans, sort of. Not really. We're actually bad at this, but we're terrible at this.

Speaker A: It's a deeply fundamentally flawed process. And even though people.

Speaker C: It's totally flawed.

Speaker A: Yeah, yes.

Speaker C: But, but let me posit how it might go for agents. I don't know this is actually going to happen. But let's say we go to the future where we would never consider a human doing code changes because like, are you kidding me? You're going to have like this thing that moves, this glacial pace, things that happen, you know, instantaneous. So how would I compare agents? I would marshal out a, ah, task to all of them and see which ones meet the mark and how many credits they spent.

Speaker A: And so it's going to be already happening 100%. Uh, we see this in the AI prototyping space. People will take the same prompt, put it in lovable bolts, reforge, build. Right. And see what, Compare the outputs. Yeah. And compare essentially the consumption.

Speaker C: Yeah. So you can imagine AI procurement doing this automatically.

Speaker B: Right.

Speaker C: Across many different dimensions. But that's why I think it's going to get, come back, Brian, to that utilization pricing because it's a race to the bottom for all those folks to prove that they can deliver the same quality or better for less. For the same or less. Right. And so it's sort of a race to the bottom of the on this stuff. I mean AI ought to be massively deflationary for things we're doing today. And I think this is exactly how you get there.

Speaker A: Yeah, I hadn't, I hadn't taken it that far to just thinking about how a procurement agent would literally just you basically procurement agent. Like a uh, few evals, a few tests.

Speaker C: Yeah. Right.

Speaker A: Then the agent goes and gives it to the five competing products in the software space and probably has some eval of cost, speed and quality of output and essentially makes a recommendation and decision.

Speaker C: And the moment, the moment you automate this, you can constantly be recalibrating it for very cheap. You want to recalibrate every quarter. We never do that historically. But now.

Speaker B: Uh, you.

Speaker A: Yeah, yeah.

Speaker C: But how much do you want to pull that in? Because if you're doing like there's this new.

Speaker A: There's this new player. Yeah, that's. That's promising. You know, just run this on. Yeah, let me click this button and see if that's true or not. Oh my God.

Speaker C: And if it looks promising, take your old workload, run some percentage of it through and if it's statistically significant, move over to them. We will have massive flows of compute shooting around to the best new flavor of AI prompting and infrastructure techniques weeks. It's going to get wild. I just don't. What we classically think of as stable is not there. Because look, and I can say all this because I was working in procurement for a couple of years but like it. The reason software was sticky was because it was hard to move your data and it's hard to retrain people. Yeah. AI needs no retraining. It will not be hard to have AI adopt a new tool and everything's going to be API ified. Because if AI can run a browser, I can reduce your product to an API at the end of the day.

Speaker A: Yeah.

Speaker C: And so there's me a lot of just dollar flow whiplash.

Speaker A: All the old. Yeah. All the old benchmarks and kind of mentalities towards LTV and churn rates are just going to be completely irrelevant in that type of world.

Speaker C: So if the efficacy is measurable. Right. And if it's not, I don't like what is it? Five based, you know.

Speaker A: Yeah.

Speaker C: But yeah, I uh. Yeah, I think it's going to be weird. That's what I'm saying.

Speaker A: So I think terminal use a massive value compression and these types of.

Speaker C: Well, what did you expect with AI though? Right?

Speaker A: Yeah, yeah.

Speaker C: Like I think that was that's the thing we're forgetting. I, I don't buy the folks that try to say that, you know, it's, it's, it's just gonna be like 10x more of the same but 10x more I think it'd be. No, I think it's going to be like most things that were software or intellectual labor are going to end up at the price of electricity. Again. Altman's insight here I think is correct. We don't know when you know, we don't know exact path but I think the, the, the physics of it seem to indicate that.

Speaker A: Yeah, on this note, I'm blanking on the name but I've seen this um, something that was developed by some YC company or some YC person which was a leaderboard of all these products in the AI prototyping space who basically have just created a public version of this. They have some set of prompts in the background and they grade them and it's like this leaderboard that updates, I don't know how often updates, but probably like on a daily or, or weekly basis and then they're just like the rankings are changing of like who's doing a better job at these things. So yeah, yeah, your G2 crowd, human based reviews is that were incentivized and bought basically by the company are essentially

Speaker C: via, via coffee Starbucks prides. That's over.

Speaker A: Yeah, yeah, that's, that's, yeah, that's right. And yeah, it starts to look, gosh, it starts to look way more like a uh, actual leaderboard of like uh, some sports league or something like that.

Speaker C: And, and it comes down to can you or someone else come up with a reasonable grading? But here's the thing, here's what made grading like metrics have always been tough because it's hard to come up to quantify things. And one way to quantify things that people don't like. We like counting, we love counting. It's pretty objective. We don't like quantifying of like hey human, you're our internal evaluator. Rate this on a scale of 1 to 5 and we will trust your judgment. No one ever trusts that person's judgment. Right. We just, we just don't. But if AI can turn the qualitative into the quantitative and we're at least quantifying the model that's doing that and the prompt we hold those things fixed it. There's no like, it's not like a judge just before lunch who makes people go to jail more often because they're Hungry and angry. It is a repeatable thing if we've pinned down the model and the prompt. So we can actually. I do think that even some of these things that are qualitatively judged we will be able to quantify in ways that we'll at least agree on enough to move forward.

Speaker A: All right, let's transition to our next topic. So Notion last week launched their agent platform. Simply put, uh, allows you to create basic agents on top of all of the knowledge that you have stored in Notion demo. One that they basically show is creating a bug tracker, right? So it'll automatically create a Notion database. It might absorb, uh, certain things from like a Slack channel, automatically log the bugs in that Notion database for you to organize something that you might have done, you know, a PM might have done manually before. The interesting thing that, um, and I texted you about this over the weekend is it feels like the 12th Agent platform that looks exactly like. They all look the same. Whether it's them or Atlassian launching that or one of these project management tools, like a, ah, Monday. They all feel like they are enabling the exact same thing, which is. I've got some form of project management coordination work. I'm um, going to layer this agent platform on top of it with very little distinction in between.

Speaker C: Yeah, well, I think it's, I think it represents a massive identity crisis and confusing transition and a fundamental devaluation of a lot of these companies underlying platforms. And so without having like a specific concrete thesis of where I think this is headed, let me at least outline what I think the problem is. These tools were all invented to help humans coordinate around consistent data and run a consistent process with some good ux. And in the age of AI, where essentially intellectual labor is now let's call it free, I mean, you know, there's token costs and we're gonna get into that a little bit later, but it's not, it's dramatically lower than, you know, human costs. All of these platforms have an opportunity, which they're gonna be forced by their competitors to take to layer on AI agents that work with that data to do more for you. But if you think about what's going on with that, those agents are fundamentally skirting all those investments in user experience, in collaborative features, and just sort of doing the underlying thing that needs to be done. And I think if you like, take a step back from this, and I think I said this on, on, on this podcast a while ago, like, what do I think the terminal state is for agents in a company? It's like, you as CEO of a company, have a trusted layer of people who do things on your behalf and shy of reports, you really don't get into the collaborative workflow of what you're asking them to do. Like I think that's kind of where this is all headed. And in a world where you've got bug filing agents and document agents that look stuff up and contribute docs back and just manage all of that collaboration and context management for you, none of that UI UX matters anymore and it's really just what's the pathway to the agents. And there I think it just opens the door for a great flattening between all these platforms. It opens the door for entirely new platforms that can rapidly generate that context. It pulls people like glean like straight into the mix as well who are leveraging that context to do similar things.

Speaker A: Yes, Lean was another one that felt like same exact agent platform. Yes, that was another one.

Speaker C: Yeah, they're all gonna, they're all gonna head there. Because all of this work to coordinate humans around shared state with a shared UX is being undermined by the fact that agents don't care about your UX and can adapt really quickly to any UX and could be just as good at coordinating context on a file system as they would be in a notion directory.

Speaker B: Mm mhm.

Speaker C: And so I just think there's this incredible leveling that's coming and there won't be a lot of differentiation at the end of it.

Speaker B: I mean the thought that goes in

Speaker A: my head too is like who's then best positioned? Like is this just a transition into the new world? Right, because these folks have all of this knowledge and context stored in their tools. They just kind of see this as an intermediary step to the future of the world. Or does like some new tool specifically designed for context store in the AI age emerges because look like, you know, yes, like connecting ChatGPT or Claude to your network. Notion is partially helpful, but all of the knowledge like stored in Reforge's notion as an example was not designed to optimize for that specific use case. If you were to build a context store for AI, how different does that look and do you interact with that like fundamentally differently than people are interacting with notion? And is that just a bridge too far for notion Atlassian glean, whoever it is to get to?

Speaker C: I think it's a bridge too far for them. I just think it's going to be um, you'd have to be very strategic to stay ahead of this. And I think you want to focus on new accounts. Like, what is the experience for new accounts in this world that we're in today? And if you don't have a solid answer for that, the game is up because nothing else is really going to matter. Before I answer that, let me say, like, I actually don't think it's a problem that reforges notion was not tuned for AI because I just think that AI is going to require less and less tuning. Right. Like, I don't know if you've seen what Devin can do for code bases. I actually think this is spectacular. You can point it at a code base, it can do PRs, but it will also go through your entire code base. And for its own purposes that happen to benefit the team, using it will generate a wiki of all the architecture, the patterns, the locations of things. It is not inconceivable that an agent even built by these platforms could look at your content and internally organize it or reorganize it for you in such a way that makes it better for humans and agents. And I would argue like what's good for the goose is good for the digital gander as well. So I don't think there's going to be much distinction there. So I don't worry about your organization. What I, what I will say though is that like, are those PMs thinking about that? Because as humans we'll put contradictions in the content all the time and AI will get tripped up by that. We won't as easily because we have slightly more prime memory of like, oh, that's an old pattern. We don't do that. Right. But you need your AIs to kind of know that as well. I go back to like, I think everything's up for grabs and I would think first principles of how knowledge gets captured in the first place. If you had a tool, let's assume you new company starting today. If you had a tool that took in all the raw data from what your team was collecting, it could automatically be organized into a context library for your organization. So what is that? Emails, phone calls, you know, meetings, zooms. Yeah, whatever it is. Right. And so like, and keep in mind call recording, email parsing. This is so commodity these days. Right. Everybody has all this and it's only getting better. So it's, the door is certainly open for someone to do an incredible job of coming up automatically with your corporate context.

Speaker A: Yeah. So it's almost like I almost think about it in um, two places. It's all these project management types of platforms live in this weird middle where uh, on one end you have like the root source of where this context is captured, like you said, the conversations, the meeting recordings. And there's a bunch of tools there. You've got the otters and the fathoms and the granolas. And then the other side of it is actually where users are shifting their daily habits to which I would argue is more in the places like Cursor, ChatGPT, Claude and a few other places where they might actually trigger these agents, type in commands of what they want to do. And I feel these project management platforms are in this weird middle where I don't. The use case I think of is not triggering an agent from one of those platforms. And when I think about that, I think about going to Claw, chatgpt, Cursor, one of those other environments. And so it's interesting, like they are

Speaker B: neither at the root source nor are

Speaker A: they at the place of habit. Except, you know, you do see some of these folks like notion they now have the meeting transcriber. So you do see them pushing in those directions. But it's certainly not where my habits are. My habits are still granola to capture meetings and those other platforms I'm talking on and like where I would type in these commands.

Speaker C: Yeah, well, I think it's smart that they have meeting capture now. Right? Yeah, I think, I think you're exactly right. I think if you are, again, I go back to like what was defensible in legacy SaaS and we talked about this before, system of record and like workflows to coordinate humans habitually. That's just not nearly as valuable as

Speaker B: it used to be.

Speaker C: And that's why I think like the foundation has liquefied under these companies. They're doing a little bit of a wily coyote. I mean, don't get me wrong, tons of revenue, tons of customers, tons of habituation, but sort of the future's coming fast here. So if they don't move out to those edges either, like doing the work in a really meaningful way itself or capturing all the context in the first place to enable themselves or other tools to do the work and just be that sort of central repository, it's going to end up being a no man's land. Is that two years? Probably not. Is it? You know. Yeah, I don't know. Um, I'm like bad, I always bet, you know, several years too early.

Speaker A: But it's not a text coming. Yeah, it's not a technology thing. It's how fast we change these habits. I mean this Kind of puts into light some of Atlassian's aggressive acquisitions. I wrote an article on the acquisition of the browser company because I thought it was an interesting one tech Twitter did its thing and memeified the whole thing. But I took the approach in that article of, well, there's some really smart people, right? Let's assume, you know, let's assume that for a second. What might be some, you know, reasons to actually make that acquisition because it just like didn't make sense to a lot of people. And one of the core reasons I, I thought about, which was essentially we do have these new points of entry into our work where our habits are shifting. I mentioned a bunch of them and, and a lot of them feel like that game is Atlassian, can't win that game. Atlassian is not going to go up against ChatGPT or Claude. It feels like, uh, the browser as an entry point uh, of one of these habits where we might fire off agents to do work is another one of those potential entry points that still hasn't a hundred percent been determined yet. There's a lot of people making bets. There's rumors about ChatGPT working on it and a couple others. And so kind of realize this, okay, agents, existential crisis, this thing that I've owned for 20 years and monetized and built my whole strategy off of, of like acquiring adjacency to jira, not as valuable anymore. Like let's start to take some bets on like pushing towards, pushing towards these edges. They did the other acquisition, dx, which I get why it fits their customer base. I don't know how it fits the broad future of AI.

Speaker C: I read an article on DX that was essentially saying it would start giving them the metrics to understand how well certain agents or coding output, uh, of agents was performing and that would give some sort of like feedback loop on efficacy. I don't, I don't know either. You know, if you're listening Atlassian, come on, divulge your grand strategy. Let's tear it apart. But I think you're right, I think it makes sense to some degree to me that they should be seeking out as high upstream into the workflow as they can get to get ahead of as much as they can. And maybe they need to start buying, you know, the granolas of the world as well. Right. The other end of it I think is interesting but complicated. I mean, I personally think, you know, despite my computer science background and degree that coding will be dead, dead within short number of years here. But it would make sense in the meantime to get as embedded in the world of what's going on there where sort of like the frontier of agents in a lot of ways is right now. It's like the work agents are doing stuff but there's English adapters for automations, 99% of the cases. There's some really cool agentic ones, but they're much rarer. Whereas coding agents are a uh, very real frontier right now that you can see market improvement from one release to the next. And so it makes sense to try to get closer and closer to that, um, at the tip of the spear.

Speaker A: Okay, random question for you. You're still a super early stage company but, but let's say you were just starting a company from scratch today. You're like a year in a year and a half in something like that. Um, and maybe you're already doing all these things. How would you want, how would you be setting up for the future of AI? Like thinking about like this uber context store. Would you be documenting things in the same way as notion or like what, what kind of new best practices is to like store all of this context knowing, feeling like the trajectory of these companies?

Speaker B: And I know some companies, every meeting

Speaker A: is recorded unless it's like an HR meeting. All those meetings go to some central repository. It doesn't matter who's participating. It's exactly how we build our startups going forward that's different than history.

Speaker C: There are spectrums of this. By the way, I've seen companies leaning in incredibly aggressively to this. Yeah, there's definitely a point of no return where it's or diminishing or even negative returns. But let me tell you at least what I would do and what we're doing that I think is compatible with that. So number one, we're a fully remote organization at appy. We got people all over the world. The same thing I do to keep good hygiene for a remote distributed org, I think actually matters for the AI because they cannot sit in the same room with you. So in some sense we're almost better positioned to solve this problem because you don't have any of that off digital conversation happening. Everything's digitally mediated. We have a chance to record everything.

Speaker A: Yeah.

Speaker C: So number one, all of our meetings get recorded like full stop. And we use where do they go in our case they go to day. AI is the one recording them all. Um, and you can actually parse over that database using their agent framework to do quite a bit. Plus it distributes meeting notes after the fact. Now what we don't have is like turning that into knowledge context to push into a directory or an ocean. But we could do it after the fact. It's all mineable, which is interesting.

Speaker A: Yeah.

Speaker C: The second thing that we do is a rubber ducking, uh, policy. It's like every person the organization's expected sort of live blogger day in Slack. And that's a way to externalize your thought process into digital form so that your colleagues can, can keep up to date and help when they can sort of see around a corner for you. But that works just as well for our agents which can read over Slack. So that's how we're solving it. But there are companies that are like going even further. So like that's just called like basic AI level hygiene. Record everything on some level and get people to verbalize more and don't have meat space conversations that don't get right. Char stuff aside.

Speaker A: And before you wait, go on. Are there the spectrum like we did, we had this summit with about a hundred product leaders and it was all about what people are trying internally. And the uh, VP product from Carvana came in and talked about this thing that he calls baselining. And what they did is um, to your point of like thinking out loud is they went around the team and had everybody essentially brain dump how the parts of the product that they work on actually work both from the user perspective and from behind the scenes. And then they took that and they structured it all into essentially what they call a baseline document context. It actually ends up just being a massive structured Google Doc at the moment. But that way it was getting all of these things that people would typically call uh, institutional knowledge out right. In a structured way. And they use that to feed it into all of the different AI tools. And of course now the question becomes how do you maintain it and update it.

Speaker C: Yep. Um, there's also terms like best known method documentation, similar type stuff. But I think that the meta insight is that all work has sort of become meta work in a way, which is if you're just turning the crank once, you're failing. If you can turn the crank once, as long as you follow up and say here's how to turn the crank.

Speaker A: Right.

Speaker C: And why it's important to turn the crank and what a good crank turning looks like. And that document is the actual new endpoint for now you've really done the work because now we can train an AI agent the moment they become capable enough to do that work. You know, 90% of stupid ops work probably can do a chunk of that. Today, not all can. I think eventually more will fall. So the folks that I think are really leading the way here are doing that documentation, but they have a site on. They're always building the machine of the organization. If you're ever doing something by hand, it's an experiment, it's not work. And if you think it's work, it's the precursor to the real work, which is why were you doing, what were you doing? How would you know what succeeded and how did you do it? So we could train the agent on it. Everybody's always a, uh, process designer now. They have to understand what good outcomes are. They have to understand what good metrics are, if possible, or what good judgment is, good heuristics. And writing up that as though they're about to hand that job off to someone else, that raises the bar considerably. And I think this is actually like part of the challenge for, you know, the, the junior cohorts coming in to companies to work is that you kind of have to like build a lot of that muscle. Whereas right now it's like there's an opportunity if you can write that stuff down and be a good manager and writer. You actually have a ton of leverage these days. So I think the, the new school of companies are really leaning into that heavily. And that's where I think, by the way, the diminishing returns can come in, which is, I believe that will be the future of work, is that we sort of front run things either with a copilot or with a goal. But ultimately the end goal is not having done the thing, it's that output. But you can get into a situation these days where like, yes, that is the future. But maybe if the AI isn't just there yet, you'd be beating your head against a wall, training an agent and getting no returns back on it. Because also, how often are you doing that? There's a classic XKCD that shows like how often you do a task versus what the automation will save you. And I don't know many people that can mentally do that calculus on the fly, especially when it's uncertain if it could be automated today eventually, but we don't know.

Speaker A: Well, the funny thing is, is like the more often you do something, that just means the deeper the habit is and the deeper the habit is, the harder it is to break. So.

Speaker C: Right. And the more blind you are to it. I mean, I, uh, you know what I haven't seen yet, and here's a product for, for some, someone ambitious out there, right. Is like some Tool that watches my work and tells me what's repeatable and is good fodder for being agentified.

Speaker A: Well, so wait, uh, that actually triggers. This is an aside, but like there was this article from the Information over the weekend about how all of the foundation models and all these data labeling companies have now started to build custom software to just basically record specialists and experts doing work on top of all these cases to train these models. The other end of there we have

Speaker C: a dearth of like useful data streams of people doing work. We have like what people twitch, stream or put on YouTube. But actually watching people do prosaic work day in and day out is a massive useful corpus to feed the AI. And by the way, Meta is going the other direction with their AR glasses, which is like, how do we get just regular physical life? More footage.

Speaker B: Mhm.

Speaker C: Google probably has one of the largest corpuses of that Tesla certainly amassing theirs. The robotics company. Yeah, but anybody who's got AR glasses, now you've got a massive new corpus of like, here's how the world works.

Speaker A: Yeah.

Speaker C: So you know, that's the other, the other runaround here.

Speaker B: Okay.

Speaker C: Uh, all right.

Speaker A: I think that's a good place to pause for the day. Or if you have any other hot takes you want to pen to the end here.

Speaker C: So many hot takes, man. I mean I just like, what a time to be thinking about software in business right now during the single biggest massive shuffle up. Like, I just, I cannot shake how wild and weird and cool it is to be here right now, whether any of us win or lose or God knows what happens. Like what an intellectual exercise.

Speaker A: Yeah.

Speaker B: All right, thanks man.

Speaker A: Thanks for joining today. All right everybody, thanks for joining us. I'm Brian Balfour, founder and CEO of Reforge.

Speaker B: If you are looking to level up on AI, we have an upcoming cohort of reforged courses all around AI, including AI Foundations, Mastering AI Productivity, AI Strategy, AI Leadership, and the course that I just completely rewrote, AI growth. All of these are baked in timeless foundational principles of how to tackle these topics, but includes all the new ingredients that we have around AI. You go incredibly deep, incredibly fast. Each of these courses is the equivalent of four to 500 pages, about 20 hours of video. And they're led by expert operators, people that are actually on the front lines implementing the things that they are talking about. Go to reforge.com courses to check out all of the courses that we're offering in this upcoming cohort starting October 14th.

Speaker A: It.