AI Adoption in Enterprise Beyond Writing Code - Ivan Bilan

Speaker A: Welcome to our event. This event is brought to you by Datadogs Club, which is a community of people who love data. Uh, typically I have a slide here, but I thought I just skip it. So, like the video, subscribe. We have an amazing Slack community. The link is in the description. Uh, so just skip that part. Uh, but the important thing is if you have any questions today, uh, to our guest Ivan, there is a link, a pinned link in the live chat. Click on that link, ask your questions. And we will be covering these questions during the interview. And it's my, uh, big pleasure to invite Ivan again. Ivan was here already, I don't remember, two, three years ago, but it was a very lovely conversation. And then one day I was coming home from a gym I think, and I was thinking, NLP has changed so much over the last years and probably it would be nice to get somebody, um, on our podcast where we can talk about how NLP changed. And the first person I thought about when, uh, thinking about this was you, Ivan, because you had this nlp. Um, what was the name?

Speaker B: That's uh, Computational Linguistics. That's what I studied. And it's kind of like NLP thing. Oh yeah. And I had. But the repo. Yeah. NLPendext. It's basically like a list of all things NLP.

Speaker A: So usually people just uh, call this project, uh, awesome. Something like Awesomeness. But you decided to call it differently, right? And it was. Can you say it again? What's the name?

Speaker B: Uh, pendect. It's NLP pendect. It's on my GitHub if you search for my name.

Speaker A: Yeah. What does it mean? Pandect.

Speaker B: Hey, it's like uh, I think it's an old Greek for encyclopedia, so.

Speaker A: Ah, uh, okay. And you had this Greek letters, uh, all over the place, right?

Speaker B: Yes, yes, yes. I just wanted to make something different to like list because there are so many awesome lists.

Speaker A: And um, with time you got promoted, you started working as an engineering manager. And I think this is what the um, podcast we had was about, right? Getting from like individual contributor to an engineer manager. Anyways, I reached out to, to you thinking uh, whether you can talk about nlp. But then I real, there is something more that we can talk about which is now everyone is obsessed, myself included, with all this, uh, generative AI and what it can do to help us. But I keep hearing from people that expectations from the team members and also from managers because of that are growing. And I thought, uh, Ivan, you um, would be the best speaker to talk about that. So here we are talking about this. So it's very nice to have you back.

Speaker B: Yeah, happy to be here. Thanks for inviting me. Uh, yeah. There's a lot to talk about. Right. I think also we kind of planned this, like, maybe a month or two ago, and I think it's like a light year since then. So many things have changed.

Speaker A: Um, but I need to formally introduce you. Right. So I'll say. Um, so. Hi, everyone. So Ivan is a senior engineer manager at personio, leading multiple teams in the identity and access management domain iam. So every time I see this im, three letters. I have PTSD from my time, like, working in a corporation and dealing with all this IM stuff in aws. Uh, yeah. Fun times. Well, it's a very important thing. So. Previously Van served, uh, um, as a data Science and Engineering manager at trustu, where he led NLP and infrastructure groups to optimize massive ETL and ML pipelines. And with master's, uh, in Computation Linguistics from LMU Munich, and background in cdtm, I have no idea what is that? Um, Ivan bridges the gap between deep technical NLP research and senior leadership. So can you tell us what CDTM is?

Speaker B: Oh, yeah, yeah.

Speaker A: That.

Speaker B: That's, uh. That's actually part of technical, uh, University in Munich. It's, um. It's a technical leadership, uh, technical management degree that I got.

Speaker A: I see.

Speaker B: Like a additional thing. Yeah, so, yeah, I studied computational linguistics and at the same time doing, um. It's like an mba, but more like, uh, engineering specific.

Speaker A: Are you still based in Munich?

Speaker B: Yeah, yeah, I'm still in Munich.

Speaker A: Okay. I'll be in Munich in a couple of weeks would be nice.

Speaker B: Nice.

Speaker A: Okay.

Speaker B: Uh, um, thanks for the intro. Yeah, thanks a lot. Um, yeah, just sort of. I've, uh, worked a lot in AI before. I mean, I, um. My master thesis was on the Transformer model. I was doing some tweaks to it way back when it first released. Um. Yeah. But then I moved a bit to a different area. Right. But I think you cannot escape AI. Uh, even in IM, you still have MCPs now. Right. So that's authentication for AI agents. That's there. Uh, and then obviously, everyone is now using AI for so many things. Yeah. Um, so, yeah, we can talk a bit about, uh, how it's going on now. Like, industry adoption of AI, talking to a lot of people, um, working in different orgs of different sizes, trying to understand what's happening. Um, as I said, it's a lot. If you look at the news, like, every week something new comes up. Um, from security breaches to crazy IPOs and whatnot.

Speaker A: Yes, we'll definitely talk about all that, but I'm really curious to know more about. I mean I already know a few things about you, but maybe for our guests you can tell us about your career journey so far. We briefly touched it when I was uh, talking about your uh, biography so far. But maybe you can give us some more proper introduction of what you have done and how you ended up doing what you do right now.

Speaker B: Oh yeah, sure. Um, yeah, as I mentioned, I've studied computational linguistics, which is basically data science and sort of NLP based. Then I was working for a while as a data engineer building ETL pipelines. Uh, people will remember Hadoop Spark and stuff like that. So that was a lot of fun back then. Um, then sort of moved.

Speaker A: Goosebumps.

Speaker B: Yeah. You remember how fun it was to do and build those ETL pipelines? Yeah, Doing then data science. Specifically I was doing um, sentiment analysis, summarization and a few other things in nlp. Um, I worked on a multilingual system. We supported like 20 languages back then. Um, so that was quite interesting. Working a lot with Asian languages as well. So NLP for Asian Arabic languages is super complex. Um, yeah. And then kind of moved on more into technical leadership, started working with different kinds of teams and then a great opportunity came up to move into um, something more security focused. So always um, was interested in that. And been five years I've been working in identity and access management space, doing access rights and login. Um, yeah, and as I said now, uh, it all comes back to AI again. Now AI agents need accessories checks now AI agents need, uh, proper MCP authentication, so on and so forth. So there is no matter where you work now, there's no escaping AI I would guess.

Speaker A: Yes. Uh, especially now for developers. Right? Because I went to a friend recently, um, with my son. I have a son. And um, my son is like other kids, he's very inventive. So he has uh, a pair of pants and there was a hole on this and he fixed it with a scotch tape. Right. So he just put scotch tape on the pants and then it was the fix. And then I saw, okay, you are growing, you are growing up to be a developer. We developers like you know, this kind of patches.

Speaker B: Nice, nice.

Speaker A: And he was like, no, no, no, like I don't want to be a developer. And then we were at my friends and my friend said that actually developer is um, kind of endangered species right now. Right. Because, uh, you know, with all the AI stuff. And my son said he wants to be, he wants to work as a mechanics mechanism like with uh, you know, gears and stuff. And it seems to be like a safer job these days. Right. What do you think about this? Like, are developers going to be needed like in five years?

Speaker B: That's an interesting one. I think like end of last year that was like a take. We're gonna, you know, chill, uh, and AI will do the work for us. But I think it's completely different. Right. If you talk to people working in the industry right now we actually have more work than we had before. Uh, we now have to manage AI agents, uh, put everything they produce together. And then what's also happening in the industry is that um, now that you have these AI helpers, you also work a lot in parallel things, right? Maybe before that you had like one work stream, focus, flow and work on that. But now you try to focus on one thing and then in the background 10 things are happening, uh, because your AI agents are doing something and then you actually need to go back and be a bit of a, uh, even as a software engineer you need to do a bit of management now, but you're managing AI agents basically instead of people.

Speaker A: For you, it's something you're used to. I guess when you were an engineering manager you had to switch contexts all the time.

Speaker B: Yeah, yeah, exactly. For me, I mean engineering managers, the, what we do. But I think for engineers that's a, ah, new kind of change, right? A bit. Um, and so yeah, it also takes time for engineers to get into that flow and then it also, we need to give everyone enough time to experiment because there's just so much to learn, even for engineers because there's so many new contexts and so many new things. Right? So we went from basic prompt engineering, now talking about context engineering, talking about um, teaching AI agents to be effective, to use less tokens, whatever whatnot. There are just so many things everyone needs to figure out right now and there's a lot to learn. Um, and again as I mentioned, things are changing almost all the time. Right?

Speaker A: Yeah. So you mentioned Hadoop, right? So probably you remember old days where everyone was doing things differently, but over time we kind of converged. We as industry, we converged to using Spark. Nobody was writing like my previous jobs anymore. Now people don't use Spark anymore, so it's considered legacy now. What I'm trying to say is like maybe right now things like, okay, how do we teach our agents to be more effective and use less context or fewer tokens this is like early days of Hadoop where people were thinking like, okay, how do we, uh, I don't know, do MapReduce in a way that, um, you know, uh, we can actually manage this complexity. But over time things became easier and easier and all we needed to do was to write a spark job and spark would create all these mapreduce things itself. Um, we wouldn't need to think about this. So do you think this is where we are also going with AI?

Speaker B: Probably. And I think sort of bigger, um, companies that have a bit more budget are maybe more like have a headway because they have a bit more time and can. So like there's a big decision for companies now. Do I pay quite a bit for ready made models or do we start doing something local? Right. And so some smaller companies would need a lot of time to ramp up. Right? You need to set something up, um, set up some local models, some sort of orchestration around it. And so, um, that takes a lot of time. And then bigger companies, um, they can sort of go in, okay, let's pay for a subscription. It's expensive at the start, but everyone can jump right in. Right, so you're using Claude Codex, um, cursor, whatever. It's much easier than waiting for your infra team to set something up with, uh, I don't know, Quan or Deep Seq or whatever. And so a lot of these bigger companies are really pushing now and I think a lot of standards will come out of that. And you can kind of see that already, right? Big companies are starting to share more what they're doing. Right. And we're kind of looking at them. I think a good example is, uh, Amazon, right? They had their problems in December where they tried something, you know, AI does everything and then it backfired to some extent and now they changed the policy. Now AI code needs to be reviewed by a senior engineer. Um, you know, so, you know, these big companies try out these massive experiments and they kind of in the forefront of, you know, giving us the guidelines to some extent. Um, yeah, but then also it's on all of us to, you know, keep experimenting and things will be changing probably as well.

Speaker A: Do you think software engineering is a safe career choice for kids these days?

Speaker B: I'm pretty sure, yeah. There's just so much work. It's actually interesting. I have, um, some friends who are freelancers. Their business is booming because everyone white coded something and then it doesn't scale, it doesn't work and they just have so much work. I mean, my friend who's A freelancer has never had that much work in their life.

Speaker A: Okay, so for us it's good, right? Because the code that AI generates is, uh, we all know is sloppy, doesn't work scale, and uh, there are many problems with that when engineers use it. I think we, as engineers we tend to produce code that is at least I want to think this way, that the code we produce through agents is probably more scalable. I don't know how through it is that.

Speaker B: I think there's a lot like, um, I think we kind of went from this idea of AI slop to now figuring out how to actually, uh, through mainly context engineering for AI agents to make actually really good output, right? So if you give all the right context and if you have everything lined up for AI agents or the model that you use, the quality is actually pretty good. But you need to invest a lot of time to build that context, to maintain it. And you need to do some pre work, right? So maybe like make sure you know, your code documentation is actually next in the repo, not in some Google Doc or whatever, right? So there's a lot of work to be done. And I think there's a shift now where we're moving away from this idea that oh, AI does sloppy work. AI is actually doing pretty good work now if you invest the time into making sure it can actually do the work right. And that costs time and you know, and it actually, the output is pretty good after that. Um, but then again, you still need to manage all of that, right? And then as I mentioned, many things in parallel are running. You need to manage that as well. You need to constantly sort of maintain that. And then I feel like also AI cannot do everything yet either, right? So AI is pretty good at fixing bugs, doing migrations, fixing um, tech debt, but sort of building something from scratch, um, as a POC is great, but then you still need to deep dive into, um, you either go and spend hundreds of agents to fix all of that and then manage that, or you actually just do it yourself probably within the same amount of time. So that's the trade off.

Speaker A: Uh, sometimes, uh, there are cases where I just need to stop the agent and go check the call and I would just find a few files, delete them all and say, okay, now re implement like everything is deleted. Don't even try to check the git history. Now we need to reimplement it and this is how we should do this. And then I explain like almost line by line and then now after that it works. But these cases are I must say, uh, more rare now these days where I need to nuke the code base and say now you need to rewrite this part from scratch.

Speaker B: Yeah, yeah, Nice. I mean it also depends on the org setup. Right. If you work in a larger org where there are just many microservices, then AI can get that context also faster than you and help you contribute outside of your domain. Right. Uh, but yeah, if you're working just on one repo or you're building your own project, then ah, it's pretty easy to handle as well.

Speaker A: What do you do as a senior manager?

Speaker B: Good question. You ask a manager what they do.

Speaker A: You manage managers, right?

Speaker B: Not uh, right now. Not right now. It depends. It changed a bit. I mostly work with more senior people on staff, engineers and technical leads. We're basically um, I do some stuff hands on as well. Uh, mainly like I took over some data analytics for my team, um, because we want to understand the data we get, uh, as well. Um, and then basically work on sort of um, what we're doing on company level as well to some extent. What are we building and then how my team can support and then there's sort of um, hands on work as well for the features that we have. Building new features.

Speaker A: Okay. The reason I asked this question is not to you know, put a manager in an uncomfortable position and try to make them explain what they do. I mean I also was a manager and then like what do you do? And it's like at the end of the day you're like what did I do today?

Speaker B: Right, yeah, context. Rating. Context. Yeah, constantly context.

Speaker A: Well, so the reason I had this question is I wanted to lead with this question to another question is how much depends on work you do now?

Speaker B: M. Mhm.

Speaker A: I don't know. Everyone is talking about December last year as that time when agents became useful for me, I started using them earlier. So uh, for me December was just a usual month, but still like well, let's take December. Right. So half a year ago now compared to, I don't know before. Is your work different? Do you do more hands on stuff?

Speaker B: Oh yeah, it depends. Right. I think smaller things. Uh, yes. Um, and that's kind of what I hear from the industry as well now. Um, you saw it back two years ago and everyone tells you for two years there's no capacity to fix it. So you just go ahead and do yourself now. Right? You can spin up AI agents and fix that and release it. So yeah, I've been doing a lot of that as well. And uh, um, there's uh, with that there's an interesting thing developing. Like should, uh, everyone be sort of uh, contributing code now? Right. So any role. And I think, yes, to an extent, if it's possible, if the sort of the company set up allows for that. Right. Interesting thing that happened is, um, uh, now when people actually try that, um, they find out that writing code is just like one of the 10 things you need to actually do to push something to production. So AI agent can code something for you. Uh, but then tests fail, CI CD fails, uh, rollout breaks, uh, something breaks. Right? And that's where actually most of the complexity is, right? So, um, I've seen some companies realize that. And then you have two options, right? You either, uh, teach, um, everyone, then these sort of all of the concepts, not just coding, but how sort of be able to fix cicd, be able to fix tests, uh, where to get that context. Right. Uh, or you just say like I think what Amazon did, right, Senior, uh, engineer needs to review and nothing goes to production if, you know, no engineer actually use it. Right.

Speaker A: So maybe there is also middle ground, right? So then everyone can deploy your, their code, but to some sort of sandboxed environment. M. And then like there you don't need any approval from senior engineers, but if it's like production production, then of course you need like a real experienced person to take a look at this and approve for reject.

Speaker B: That's true. And that also generates more work for engineers. Right? We were talking about like, uh, the workload grows, you know, and the workflow also grows. Because now you're getting Mrs. And you have to help, right? And you have to not just review, but sometimes just really, you know, invest the time to help. Uh, so that next time when the person contributes can do it on their own. And that's the upfront investment that, you know, if you want to allow that in your org, you need to set up some time for that upfront investment, I would say.

Speaker A: So right now I have two questions. The first question, how do you actually do this? But then before that, um, so there is this guy who created OpenCloud, you know this from Vienna. What's his name?

Speaker B: Yeah, I forgot the name, but I know who you're talking about.

Speaker A: Yeah, but like every time I open Twitter, I see this person and he's a kind of humble bragging or I don't know how to call this, but he's saying like, look, I spent like uh, 1 million on tokens today or something like this, all these merch requests, right? But, um, okay, yes, his scale is unprecedented. Other people don't do this. But the thing is he has a process which there are agents that are working on these PRs. So the agents are closing the PRs, the agents are reviewing the PRs, the agents are rejecting PRs and these PRs are created by other agents. Agents. Right. So it's a very interesting situation that at least for his project that he needs to uh, work on. Right. So, but he has a setup for that. And coming back to uh, where we started and why I thought about this, you mentioned that we need to educate people and have the proper M setup for that. Yes, we create more work for engineers. But how do we manage this work? Right. So um, this person found a way to manage the work right, with uh, OpenClow and like all these Codex agents that he is using. But like in your case, and maybe people you talk to, like what is the typical setup to make it possible for developers to actually bear with this load?

Speaker B: Yeah, good question. I think, um, again it's important, I think to make the first of all to make the LLMs or AI agents that are actually useful, you need to go in uh, and do context engineering, give them access to everything they need, have um, clear instructions. Because otherwise you're going to be spending time on fixing stuff. You're wasting so much time on fixing stuff after the AI agent. So we need to spend time upfront. Um, and then you kind of go, you can go level higher. Um, I see a lot of interesting setups that are open sourced where you have different types of agents. Right. So you have like a, there's a project on GitHub called Three, uh, Man Team. And so one person is a developer agent, the other one is a reviewer agent and the third one is like a product agent. Right. So reviewer, architect. So they all have like separate roles. Right. So yeah, and one creates the code, the other one solely focuses on reviewing it, making sure it fits the architectural design. And the third one cross checks it was product requirements and so on. And so um, that's just one example. Right. And I think that's where we are now. We need to keep um, understanding how to make AI more efficient through these more elaborate setups. Right. Um, at the same time making sure we don't overdo it and don't have like a huge uh, memory problem in terms of the AI forgets, uh, context management, uh, so on and so forth needs to be there. Um, yeah, so there's quite a lot. And then for um, sort of on. Org level I would say uh, really what's important is, um, you need the time to experiment, right? So if you're adopting AI, don't expect it to be like immediately next week. Everyone is using it, right? Um, everyone needs the time to experiment, play around, around with it, try different tools, try different setups, um, and then sort of eventually converge on something. Um, yeah. So there needs to be that support.

Speaker A: Why do you think, um, it doesn't work this way, that you just handle everyone, cloud code, subscription, and they don't, uh, all of them don't start using it in a week. What's preventing them from using it? Why not?

Speaker B: They can use it, right? I think will the output be useful first of all? Uh, and then, um, what will probably also happen is that everyone will start building their own plugins and things where you could actually unify that, right? You can, uh, make sure everyone is converging on similar tools, similar plugins. Um, yeah, because then everyone wastes time building their own thing and then they need to maintain it, right? And then there's also varying quality. Um, so someone did it better. The other one maybe has lower quality. Uh, so you need to actually, like, eventually, if you're using stuff like flood code or whatever, the setup, um, you need to work on unifying that and maybe creating like a, uh, general strategy for the company.

Speaker A: So a more pragmatic or wise solution in this case would be to just get a few senior engineers and give them a task like, okay, now you use this for a month, you figure out what kind of infrastructure we need in order to scale it to the entire company, right? And then they go deeper into this and then they come up with some sort of like, not necessarily a platform, but the set of best practices, right. And it's unified. And then, um, oh, I come from like what I was doing before, um, I run an MLOps team. This is what we were always doing. So we want to, uh, introduce a new technology. So now we need to actually understand how people would use this technology, uh, what kind of things they would need on top of that. Like not, um, just, you know, you lock yourself, uh, in a room for a month and then you go out and say, okay, this is how you should use this. But you constantly talk to people and get their feedback and, you know, improve. So probably this could be a way when it comes to using organization.

Speaker B: Yeah, that's also true. That's also true. I mean, yeah, there's just so many ways and I think like, um, so many companies do it in different ways, right? That's what I meant like these bigger orgs, um, they can experiment and then you either wait for them to share how they did it and what actually worked, or you go in and experiment yourself and see what happens.

Speaker A: And uh, what's the role of a manager or a senior manager, uh, in this AI adoption?

Speaker B: I think it's uh, basically that it's actually useful. Right. So first of all, uh, making sure, uh, again what I mentioned before, there's sort of unified approach to it, right? So that um, we're not in a state where everyone builds their own thing on top of whatever LLM you're using. Um, and then the whole team does stuff differently. Uh, it's more like unifying things, making sure the team is aligned and then ideally also aligned with company direction and how that is set up and using what the company is using as well. Um, and then basically I think for managers, um, I think that's a big question now. Is it actually useful? Is it actually doing something? So how do you track that? So how do you, uh, check are we actually bringing more value to customers? A customer is even noticing anything changed. And then for, I think for internal developer experience, that's what I meant before is AI is good for fixing small things, tech debt migrations. So you kind of then see where can you apply that, right? I'm pretty sure there is no team in the world that doesn't have a tech debt backlog. Uh, so, yeah, you see how you can apply it there. Can you start fixing some things there that you just never really had time for.

Speaker A: Um,

Speaker B: where is the usefulness of AI? You need to check that. You need to apply it in the right place. Uh, and then converge the team and org to use the same tools.

Speaker A: How do we measure if it's useful or not? Because, um, there is a story probably also heard from Uber, right? So they burned through their entire year budget and now they ask themselves like, was it actually useful? And my understanding from what I read online that they couldn't give a satisfying answer. Yes. Right. So they, okay, yeah, we kind of spent all this money, but like, what did we do?

Speaker B: Yeah, yeah, I think it's um, you know, these big companies also, um, they went away of, um, there's no limit. Or even worse, there's a leaderboard, all right, who spends more tokens, right? Uh, and then people just read stories online. People just run LLMs in recurse, do uh, some random stuff, right? Just to generate, to burn more tokens so that you look better on the leaderboard. That's horrible, right? Yes. And I think we're finally going away from that. There was an experiment, a very expensive experiment for Uber and the others. Uh, but I think kind of everyone realizes that now that that was not a great idea. Uh, yeah. And so basically how do you measure? Right. I think that's hard. I mean ideally, um, there's nothing easier to change. You should ideally already have measurements of, you know, the quality of your product from customers, you know, NPS or whatever. You're tracking quality, um, of the satisfaction of like developer experience from engineers. I mean we all should be tracking that ideally already. So if you already have all of that in place, then you actually should see the impact in those metrics.

Speaker A: Right?

Speaker B: Um, but then the question is how good are those metrics in your case in your company? Um, are they measuring the right things? Um, but I would say yeah, customer satisfaction obviously, and then a satisfaction of developer experience inside the company.

Speaker A: And uh, do have actually because this coding agents on such scale is a uh, relatively new thing. Right. So we didn't have everyone talking about using teams of agents a year ago, right? It started recently. Do we have actually enough data to say that? Okay, we did this experiment and it led into the improvement of nps. NPS is like Net Promoter score, right?

Speaker B: Yeah, exactly, exactly.

Speaker A: Like, um, do we have enough um, data to actually say that? Or like maybe these experiments don't need to be long running so we need something in a week. We ship the feature, customers are happy and we continue shipping. Or maybe we should stop and think about things like, well, what's, what's the, like, I don't know. What do you, what do you think about this?

Speaker B: Yeah, that's a good one. I think like everyone is trying to figure out it now and so some companies are still in the mode where they just let's throw money at it and see what happens. And then for the others, for smaller ones, that's not an option. So they need to be more careful with that. They need to make sure either they went and set up fully some sort of local setup that they manage and tokens are cheaper for them, or they are really checking what's the progress, what's the impact, uh, and then adjusting accordingly. Um, and yeah, that's the thing. I think it's like, um, um, I don't think there is a clear answer yet because we're still in that hyper adoption modus of like everyone is in that mode right now.

Speaker A: So you mentioned token consumption and ah, leaderboards and this is ah, best possible metric for that. Worst possible metric for that Sorry. Uh, it's like measuring the output for developer in lines of code. Right, Right. We all know that it doesn't work, but do you know any other metrics that works okay at tent? Yes. You. The only thing that matters is, I don't know in case of businesses revenue, where the revenue is growing, how exactly you are affecting this revenue through some other proxy metrics. Right. Like NPS is one of these metrics and then the team satisfaction is another thing. M. Are there any other metrics that are um, let's say closer to maybe this token consumption. Right.

Speaker B: So they.

Speaker A: Because like in order to measure how exactly you affect the revenue, it's a long journey. Right. So it's not. We might not have enough data to do this, but there are some other metrics that could be useful like number of pull requests or number of merge pull requests or things like that. Which kind of metrics? Like developer specific metrics. Um, have you seen work better in practice than token consumption?

Speaker B: Good question. I think we're kind of going back to discussion of this developer experience metrics, DORA and similar metrics. So you mentioned Mr. Throughput and so on. Um, I'm also not very convinced those are uh, so helpful. I think it really depends on what you're doing, what you're focusing on. So if you say you're going to throw AI into fixing tech debt, then a good measure would be, I don't know how long your CI CD pipeline runs, how long does it take from creating an Mr. To actually releasing it? Right. So are you actually improving that time? Is it faster? Is it better? Uh, is there less code complexity now to deal with? Because AI fixed all the tech debt, hopefully.

Speaker A: Are there metrics to measure code complexity? I think there are. Right?

Speaker B: There are, there are quite a lot, but I haven't seen them used really in the industry.

Speaker A: Yeah. When I was a developer, a Java developer who had sonar and sonar would show some cryptic metrics which I had no idea how they actually work. But like when it goes down it's like good job. I guess it's one of those, right?

Speaker B: Yeah, yeah. There's this ATS thing in Java where it measures complexity of code, but I, I've only seen it in research. I did some research on that as well, but I've never really seen it used in industry. I think in the industry it's more like going back to the DORA metrics. Right. So, um, um, it's a, it's a. I think it's a developer experience. Um, uh, it was like a few years Ago, um, a research done by, don't remember which company, probably ThoughtWorks, uh, where they did research. What are the best metrics to measure, uh, sort of output, uh, of code. And I think most of those metrics are like what you mentioned, like uh, Mr. Throughput. So how many misters you get per week or whatever. And then measuring, um, your start, you literally create the Mr. And you measure from that time. How long does it take until it lands with a customer? So um, it's a cycle metric. Basically it just tracks how long does it take for everything like test to run, um, the team to review the mister, how long it takes in cicd, how long it takes in uh, delivery canary releases whatnot. And so you measure from start to finish. Worst case it's like a week or two. Best case it's an hour. Right. And that's one of the metrics that I think we should be optimizing for, uh, because nobody wants to wait.

Speaker A: The regressions we had like number of that are piled as a follow up. That's also part of, that's also true or.

Speaker B: I think so. I think so. I mean, um, I'm checking it uh,

Speaker A: right now, but it's a very long document. Maybe I should have asked ChatGPT, but there's uh. Yeah, so the, the, the idea.

Speaker B: Right, you're right.

Speaker A: Is you um, want to have uh, software delivery performance metrics. Right. So how good your delivery pipeline is.

Speaker B: Yeah, exactly. So lead time, how long it takes M to prod. Um, then you mentioned this like failure rate that's there as well.

Speaker A: Right.

Speaker B: So how often or how many bugs happen after that? Right. So how much time do you need to spend on fixing stuff afterwards? And that's an interesting one for AI agents. An uptime as well. But for AI agents you deliver something. Um, what's the failure rate of code generated by engineers versus failure rate of code generated by AI agents? That's a good metric to track. So I haven't really thought about that. But as we were talking about that, that's a good one.

Speaker A: But um, now with um, a lot of AI code being generated, we see uh, I don't know if it's correlation or not just correlation or causation, we uh, don't know. But now with all these AI tools, many major um, services like GitHub, they experience a lot of outages. Right. So if you go to their status pages, you don't see this uh, 99.999% deal anymore. It's in the 90s. Like it's. I Don't know. In some cases it goes to like 80s and I mean, for GitHub, they're in a very tricky situation right now because all these agents, where do they publish code? Of course, on GitHub, where all these, uh, CI CD jobs run on GitHub, where do they store all these artifacts? On GitHub. Of course, it's very tricky for them. Right. Um, but there are other providers. Like there was a story that the rsync maintainer discovered, uh, cloud code recently and now like, with the newest release, like this thing stopped working.

Speaker B: Uh, yeah, yeah. I mean, some companies just, you know, you know, it's a balance. Right. M. Some companies just want to go fast. Right. And failure rate goes up and, you know, but they still delivering new things. So I, I don't know. I don't know. I. I've seen that about GitHub. Yes. But the same thing happened with Amazon, right. There were so many outages and then now it's kind of under control. So maybe that's, you know, GitHub enforces more code checks and that helps. We'll see.

Speaker A: So, uh, essentially what I heard from you, you repeated multiple times, is the big companies, they pave the way, so they experiment. They are really on the frontier of all these things. They have the money, they have the, um, workforce, like they have the people, a lot of people, and they experiment or sometimes they are forced to be a part of the experiment, like GitHub, I guess. Uh, so, uh, and then what we can do, we, as people who don't work in these big companies, you can learn from them because usually this information is public or something.

Speaker B: Yeah, M. I think that's a good source. Yeah. So, and thankfully they put out a lot of, you can read the postmortems as well, and they put out some blogs as well, so it's interesting to see how that develops.

Speaker A: Um, so let's say you join a new team and your task is a senior engineer manager to, um, set up, um, have this framework, have these processes in order for people to use AI as effectively as possible. How would you approach this? Let's say the company, the team, does not use AI yet. I mean, the individual developers, they are trying to use this, but there is no standard, uh, way of doing this.

Speaker B: I think the first thing I would do, and this is probably controversial, is to say, um, you have two weeks to do anything with AI, there is no expectation to deliver anything. The expectation is that you experiment, learn tools, and then at the end of that two week, we run like A knowledge. So again, there should be no expectation that you actually build something because it's just so complex. Ideally you do, but just give the team the space to try out things without fearing anything. Right. So, oh, I need to build something in two weeks and I have no clue how. Right. So they actually start and they have the free time. And then what's important is continuous, um, knowledge chain. Right. So I've mentioned this before. Some engineers will do it in one way, others, uh, in the other. And this is how it was also at the beginning. Uh, we all like, learned things and we all learned things differently. I don't know how that happened. Right. We follow different people online, we read different blogs, uh, we try different things. And so, um, talking to my engineers, they did stuff completely differently. They use completely different tools for context management. Uh, I use a different tool just because I found it myself or someone recommended it to me. And then you need to work on converging, right? So you need to work on knowledge sharing. And then ideally you agree on one way of using AI, if possible, with some variation.

Speaker A: How do you do this knowledge sharing sessions? Is it like you just all get together in a room, everyone presents a demo. So then. And everyone knows what exactly you did in these two weeks?

Speaker B: Yeah, demo.

Speaker A: Ah.

Speaker B: Or just like. Yeah, exactly. Show what you learn, show the tools, uh, and then have a conversation.

Speaker A: Right. Like what you liked, what you didn't like could have been better.

Speaker B: Like obstacles and then also like collect feedback. Like people will say, oh, uh, I could have done the same thing at the same time myself without using AI.

Speaker A: So there are expectations for people to uh, have like a deliverable at this point, like the expectation is they just try it and they share what the.

Speaker B: Yeah, I would propose that, I think that would work really well for the first two weeks and then maybe after that you can actually say, okay, now we have, uh, better understanding of it. Let's do a hackathon and actually build something. So as a follow up on that.

Speaker A: Mhm. So hackathon is like for a couple of days.

Speaker B: Oh yeah, yeah, yeah.

Speaker A: Mhm. Okay. So then everyone builds a hackathon and then what, I mean, space in a hackathon and build something.

Speaker B: Build something, yeah. And then, you know, we'll see. It depends on the. Org again, if, I mean, ideally you build something that is part of the product, right? You build some new feature and then you actually like, you would test how close are you with the tools that you have to actually like pushing something that's not just a prototype, but something that can be delivered to customers. Right. So that's a good measurement of are you using the right tools. Right. So maybe using some LLM, that's not good. Maybe you need to use a different one or you use different ones already as a part of the experiment and then you actually measure uh, you know, what's the, you know, the hackathon project output. Right. This is just something that will never go to production. Then maybe you need to go back and you know, check your tools, check what happened or something that's working. And hey, we're going to release it after some cleanup to prod as well. Right. And that's a good indicator. Oh okay. The tooling is good, it's working, it's right fit for the team. The team figured out how to do it together as a team and then.

Speaker A: Mhm. You'll probably need a few iterations of that.

Speaker B: Yes, definitely.

Speaker A: And after you iterate, what's the good state? The state we want to be eventually?

Speaker B: The state we want to be is basically it's part of regular uh, workflow. Right. So it becomes regular thing for the team to use the tools. Um, and ideally, I mean ideal outcome is that um, it becomes sort of invisible. Right. Whereas you still produce code, it's still good quality code. Um, but you might be able to produce a bit more uh, depending on what you're using. But ideally the quality shouldn't go down. Right? That's what I mean was it should be invisible, the quality should stay. So you need to invest in that. Um, but you're just using AI then And then the trade off of that is what we talked about is the workload of the whole team will then increase because there's more stuff to review, there are more stuff to manage. So I think there needs some work needs to be done. Then in terms of team processes, do we change the team processes? Um, you know, how, how much throughput can we actually handle as a team? Right. If someone does 10 PRs per week and that's just not sustainable for the team, then we need to scale down or scale up or whatever.

Speaker A: And then um, yes, the goal is the tool is invisible, uh, the code, uh, the code we produce is still good quality. Maybe we will do a little bit more. But with this invisibility there is one uh, like asterisk that the cost. Right. So then as a company you pay more and sometimes it's a lot more like uh, you know, considering what happened now to GitHub Copilot, uh, so how do then you decide uh, whether it's actually a thing worth keeping.

Speaker B: I think that's sort of for every company to figure out their budget for this and as I said um, probably like set some limits as well. Uh, or while you know a good strategy would be um, you know you, you pay for a subscription but at the same time your AI engineering team works on something local, fine tuned for your org and less expensive. Right. So I think that's a good, good approach. Right. We don't know like uh, what we've seen in the last half a year. The costs are uh, just like go up, up, up, up. The features go down. Right. So like you know I think lot removed like uh, from, from Mac subscription, decoding one or something like that. So like the features go down, the cost goes up uh and we don't know how far that cost will go uh because it's increasingly more expensive to run those things.

Speaker A: And then another thing I wanted to ask you is the dependency but I think you answered that because like as we start using it and incorporating in our flow we become dependent on the provider. And this happened to me personally. I was, I got excited about cloud code so much so I started using it a lot. And then at some point um, they had some problems like so they, it was a few months ago where you would hit the weekly limit in like one session or something. And then I was like okay, what do I do now? Like uh, all my workflow is I am so dependent on this tool right now.

Speaker B: So and I do.

Speaker A: And then I started thinking about like okay, there's codecs, there's other providers. How do I make my work independent? Like how do I not depend on the tool? Yeah. And this is just, I'm a solo uh, how to say solar uh developer. But as a team you have risks, a lot more risks. Like so I'm flexible, I can just go ditch cloth and start using codecs.

Speaker B: Right.

Speaker A: Of course it will require some readjustment for me but I'm more flexible as a single person. Um, but when you talk about a team of people or when you talk about like entire organization, you can just tell everyone hey, today we use codecs. And then in one week you say I know you can actually we go back to code. Right Is the answer having this AI engineering team working on the local solution

Speaker B: or maybe because what's happened for me, I had the same problem. I've been playing around with this stuff and so what I'm m doing myself and what others I see do is that um, for day to day stuff you use open Source models and there are so many open source models and there are many tools you can use for that. Ah, and then when you converge on something, you want to actually build something like production ready, then you go and pay your, you get your subscription and use it. Um, and I don't know if that's the answer for big orgs, right? But I would assume so. Uh, if you want to save on costs, uh, you need to in parallel start building something yourself or something that's just cheaper. You know, you could even use like AWS Bedrock and then you can use open source models there. Uh, so the orchestration layers there so it's easier to do something yourself. Um, and it's much, much cheaper. And again we don't know how more expensive the stuff will get. Uh, but I think it's uh, like with any tool, it's not just AI, we have the same problems as other tools. How do you manage your dependency? You basically what you can build in agnostic way, you build in agnostic way. So let's say you're building the um, CLAUDE MD or whatever. Uh, ideally it's hopefully would be reusable in another LLMs, right? You build it in a way that's not specific to CLAUDE or whatever you're using.

Speaker A: So in my case, uh, the instruction I have in ClaudMD is Go Read Agents MD. That's the only line in all my Claude MD files. Um, so now um, another question for you about juniors. So now we know that getting hired as a junior is much more difficult compared to a few years ago. Right? Um, and AI is one of the reasons. Right? So now you can get your senior engineer a cloud code subscription and then the senior engineer would produce, uh, ideally when we figure these things out would be more effective. So now there is no reason for us to hire juniors. But then eventually these engineers, they will, you know, some will decide to leave their uh, software engineering job and focus on, I don't know, carpentry. Uh, some become managers. Like eventually, you know, we need fresh blood. So the companies probably also realize, but I don't see a big uptick in junior positions still. So why do you think, uh, we don't hire juniors anymore and what to do for juniors? Like how can they get hired?

Speaker B: Yeah, yeah, sure. Um, yeah, uh, that's a big problem, I think. And you know, all this reminds me of, um, you know, there are people who just like a few people who can maintain like prologue systems and they get brought out from retirement just to do that, uh, for a lot of cash. I Hope that doesn't happen to us. Right? So it's important to, um, get juniors to uh, onboard them into software engineering. And then there's also an interesting thing happening right now. You kind of jokingly see people saying, um, it's cheaper to hire a junior engineer than run Claude or Codex, right? Because the costs are higher than what you would pay an actual human. Right? And again, we don't know how far the cost will go up, but there might be a tipping point where just say, all right, this is too expensive. Let's just hire people again. Yeah, but, but it's important. I think it's really important. I think every company is to think about that, um, start hiring juniors. Um, one thing I can say about my current company, Personio, and it's public info, we started like a drive to hire junior engineers. Um, so we have open positions now, uh, which is great. And I think other companies need to think about that as well, uh, because that knowledge will go away. And I, um, think people still have this, um, thought that, you know, AI is there and it can do the work. But, um, AI, ah, doesn't have the context, right? You need someone to provide the context. You need someone to fix the problems. You need someone to orchestrate AI. Um, and then, you know, if in 20 years we all retire, who's going to do the job?

Speaker A: Right? Yes. Yeah. And another thing I was thinking about we or senior engineers, um, so we learned coding when we didn't have AI systems, right? So now like, even if, like, I hope it doesn't happen, but like now if I don't have code subscription, I will not like it. But if I am forced, like if the circumstances forced me, I'll be able to go and modify the code, update the code. I still have the skills. They are rusty now.

Speaker B: Oh yeah.

Speaker A: Compared to before, right? Because I don't do this anymore. I just give instructions. But still, I have, still have the skills, right? So I have some muscle memory versus juniors who never did this at the scale as, um, you know, older generation. Uh, so what are the risks of hiring juniors with this kind of setup right now? Like, uh, should we think about this now or we just should hire juniors anyways and then see what happens.

Speaker B: Um, I mean, I think we should, like, I, I think people also, um, um, don't use current LLM models enough to learn, right. I, for example, my, like, I, I love doing that, right. Instead of just coding something, I tell, oh, uh, I ask, like a prompt, explain everything in detail. What are the steps? Right? So I learn A lot through that. You know, 10 years ago I'd go to stack overflow, post there's be called an idiot first and then someone would actually answer it. It's a part of the experience, right?

Speaker A: You have to live through this.

Speaker B: Exactly. Uh, yeah, but now you can just go and ask an LLM for it. And I think the other problem is to what extent will LLM keep giving you the right answers? Right. We have this idea, uh, of model collapse as well going around. Right. So at one point there will be not enough training data, uh, for LLMs. Um, so to what extent will they be helpful? I don't know. Uh, but I think just because of that it's easier to start as a junior, right? Because you no longer have to. You know how we did it. I, uh, would come up to some more senior person and say, uh, I need help. Sit down next to them and just plug them. Right. But now you can just, you know, open Codex or whatever and start asking 100 questions there. Right. So it's easier to onboard yourself with LLM's presentation.

Speaker A: M. And this is a comment that I just see right now. So there's a theory that juniors catch up faster with AI. Um, so this is exactly your point, right? And it also gives more kind of breath room for senior engineers who now can actually do the focus on what they do and not to be bugged by people, uh, constantly dropping them.

Speaker B: Yeah, um, yeah, so, yeah, just sort of bottom line, I think, you know, we should keep hiring juniors. Um, but let's see, let's see what happens. It's going so fast, we don't know how to interview.

Speaker A: Have time for one more question?

Speaker B: Yeah, sure, go ahead.

Speaker A: Okay, so there's a question about uh, people who want to get, uh, to start a career in AI engineering as a software engineer. Um, maybe you have some recommendations for them, like how. What's in your opinion? Um, you probably as a manager, you saw people making these transitions. Um, so what you would recommend to them right now in 2026.

Speaker B: Good. I think, um, it depends. I think we still have quite a broad spectrum of AI roles, right. We still have the sort of regular MLOps. Right. Someone needs to orchestrate everything. Uh, build like, um, if your company is doing uh, reg, so like retrieval, um, then someone needs to build that. That's basically like we have ETL pipelines now it's reg. Uh, so, um, you can go that way, right. You can learn the tools there. And the good thing is there are a lot of open source tools. Like there's Llama Index, LangChain, Haystack and many others. So you can just go and build a toy project yourself? Um, of course, if you have a gpu, that's great, you can rent something online. Um, and then there's uh, I feel like there's another role forming in the industry where it's more. You're focusing on context building, right? So you're actually the one uh, writing very efficient, um, instructions. You're building an infrastructure around um, re engineering prompts, uh, re engineering context, how it's given to the models. Um, so that's another way you can go. Uh, and I'm pretty sure that will stay for a while as well. You can learn the best practices there. Um, and again you can experiment locally. There are just so many tools right now. I'm really happy to see open source is still alive and well in AI. Uh, yeah, exactly. It's catching up. There's amazing tools like anything LLM, it's on GitHub, you can plug in any LLM model there, play around with all the steps you need.

Speaker A: Um, so yeah, and this context building and context engineering, you mentioned it so many times. It feels like uh, I need to find somebody who would talk about this thing for an episode.

Speaker B: It's pretty new I think. Um, like I've heard the term uh, of concept engineering like maybe a month ago. Right. So it's kind of developing now because there's this idea I've heard, right. We're going from, you know, we assumed AI is just going to keep generating slop, but that's not a problem with the model, it's the problem with orchestration around the model.

Speaker A: Right.

Speaker B: So you need to spend more time improving everything around the model. You know, prompts better prompts better context. And that's, that's where context engineering comes into play. And if you have everything set up well then your, the power of your LLM is much, much higher and the quality is better.

Speaker A: And there is a comment uh, about specification driven development and I think this could be exactly that. Right. So then you give context, you give your agent the context. How you should approach a, like there is a, let's say a product manager who is scoping or grooming the problem. Right? So there is a software engineer who is uh, working on implementing this. There is a tester who is testing this and then again pm accepting or rejecting this thing. At least this is the setup I arrived at and this is what the setup I use. And you mentioned this uh, too. So I think this is, we are

Speaker B: probably talking about the same thing. Right. You know, different people call it different names. Uh, or it's part of. Of the whole thing. Right. It's, you know, building blocks of proper context engineering that still being figured out. Right. I think I mentioned this before. Um, we're kind of in this research state where we don't know yet what's working best.

Speaker A: Yeah. And to me, it's also about giving your agents ways to, um, understand your code base faster. Right. I don't have an answer for that, but this is how I understand it. Like all this. Co agents, MD files, things like that. So you want to say instead of doing like, grep on the entire code base, like, if you need to make a change here you go here, Right. Maybe you have index on top of your code base. I think Corso is doing that.

Speaker B: Exactly. Yeah. That's really helpful. And at the same time, also dangerous in terms of costs. Right. You need to be very careful. Then it needs to be very precise. Right. So that. Because every. Every single time you do a prompt, uh, LLM will load that context and span.

Speaker A: Right.

Speaker B: Tons of tokens and processing it.

Speaker A: Okay. Ivan, thanks a lot. Um, pleasure to talk to you as always. So thanks for staying a bit longer with us. Really nice talking to you. Um, I really enjoyed this conversation and thanks, everyone for joining us today, too, and asking questions. Um, so it was really nice.

Speaker B: Same here. Thanks a lot for inviting me. Uh, thanks for being here.

Speaker A: Hello. Uh, I'll drop you a line because maybe we can meet in Munich.

Speaker B: Oh, yeah, definitely.

Speaker A: Okay, so then, uh, speak soon. Bye.

Speaker B: Nice. Thanks.