Engineering Management and AI - Part 1/3 - Value

Speaker A: This is the Effective Engineering Manager podcast. In today's episode, Adam and Slava begin to unpack the impact of generative AI on software teams and the role software engineering managers have to play. Welcome to the, uh, Effective Engineering Manager podcast.

Speaker B: Hello Adam, it's your time today. What would you like to talk about?

Speaker A: Hello, Slava. Uh, we're going to do something slightly different today and we're going to kick off a, uh, small series on uh, a topic that's on, I'm sure, a lot of people's minds. Um, the adoption of AI, specifically generative AI and how, providing some guidance to engineering managers. Uh, we're going to talk a little bit about, uh, up front about where, what we think of what's going on in the trends in the industry. We're uh, going to talk a little bit about where we see it kind of going and then adapt it to how engineering managers can effectively, uh, approach it going forward.

Speaker B: Good stuff. And um, um, I'm super excited because you, uh, and I, we just talked. Uh, there are so many things to cover and uh, there's a possibility that we are going to have actually, uh, not just one, but maybe two or three episodes on this. Super excited. Let's do it.

Speaker A: Absolutely. All right, so, uh, let's, you know, I think since this is our first podcast on the topic, uh, I know we've touched on it slightly in others, but first major dedicated, uh, episode to this, uh, let's start talking about what we mean by AI, generative AI, and what is its role within engineering. Um, Slava, do you want to kind of kick us off with where you see things are at and kind of talk about this wave over the past two years or so?

Speaker B: Yeah, two plus years. Yes. And um, I think AI makes an amazing, Just an amazing, uh, marketing and sales and hype term. It's just amazing. Right? I've never seen it before and it feels like even Internet is not as big as AI these days. Um, and I think for engineering managers, especially those who are still considering or haven't tried it yet, or don't fully understand what they're dealing with when we say AI, at least in the current state of development of things, which is in our case, um, the advent of large, uh, language models, what AI means is a large scale, uh, knowledge base, um, in LLM terms is called um, corpus, large scale base of human knowledge, which is accessible by using normal, everyday language. So the language of access, the programming language, is the language all of us speak. And that is just huge, number one. And number two, the output of those um, knowledge bases um, which are called uh large language models is also minimally language that we all can understand and with minimal effort. Also um, structured data and uh, all providers of large language models these days do provide ability to output requests in form of structured data. And it just means that you can tell what the structure must be and um, you're going to get an answer. Um so essentially large scale um base of human knowledge accessible um by using normal language, a natural language and getting results in form of the natural language. So I think if we keep that in mind when we talk about AI and application to the engineering and what it means to engineering managers I think it's going to be helpful. We'll keep saying AI, we will interchangeably use LLM as a term but we should keep in mind this large scale knowledge base, uh, that's my take. What do you think?

Speaker A: Yeah, I think that's right. You know that's exactly um where we're at. And you know the, the whole concept that really kicked off this whole new wave is the fact that um, you can generate new unseen things before right? Generate new content, generate new whatever it may be, putting a resume together, generating new code for software, um you know generating images. Right? Every single area of kind of create creative um spirit uh, right is embraced by this whole generative movement. And we saw that when ChatGPT AH came uh, you know came out with its 4.0 model or 4 model and it was much more powerful than it had been and had larger depth of knowledge and ability to do a lot of these really cool things. And then that kicked off the race for all these uh other competitors and all the big players to kind of race to see who's going to have the best offering. And then over the past couple years that's gotten tossed up and it's sort of settled around the big three out there with um, of course OpenAI and Microsoft's partnership and then um, Amazon's offering partnership with Anthropic and uh now and Google with their um enhanced offering now with Gemini and the um LLM families that they support and really the big players and of course Meta has something and then all of them are competing to kind of win this race or dominate the race. And um, they're all sitting on all these uh, uh Nvidia based um machines that are, that are doing all the processing and so they're at every layer of the technology from the hardware to the uh, to the platform to the application. I mean it's just kind of a race to see this. And what that has created as a formula is uh, the next huge generation in the technology industry, right? It's a large enough wave that uh, it's clearly being seen as the next uh, uh, big wave in sort of industrial revolution, um, of technology and computing, um, not seen since event of the World Wide Web and all the growth in the Internet, etc. So I think uh, this is for real. Um, we'll see if it fizzles out a bit over the next several months and years and kind of normalizes, but it's not going away. And we've seen industry, um, all the businesses and software providers and SaaS, companies and others recognize um, they need an answer for this because their competition is doing uh, that. And so now we've got every company saying, oh, I got to have an AI offering, I got to have a generative AI offering for my product, whatever that is. And that has create. I think that's created a lot of stir and chaos for um, engineering teams to come up with um, effective solutions. And um, it certainly has pushed the notion that uh, the technology is going to be strong enough to even replace individuals and jobs that were previously done by humans. Um, and so there's a concern there. And so I think all of these things together is really where we're at. And coming back to our purpose today is really to start to look at what kind of guidance we can provide to engineering managers. Because one thing I think we're both pretty clear on is that uh, engineering is not going away. Engineering, uh, individuals, engineers are not going away. Certainly engineering manager leadership is not going away anytime soon. And so how do you position yourself for this next week wave when all these things are coming together, um, and it's like the perfect storm of everything that also threatens to potentially um, eliminate some jobs or cause you to have to position yourself to do new jobs. So as engineering managers and leaders and thought people in companies, um, we want to dive into what we think is some really good guidance for where we're at right now and how you can be effective in um, supporting your company's need to, or desire to embrace generative AI, but also being practical and uh, able to actually execute on it at the same time. Um, what do you think about that, Slava?

Speaker B: Well, I think if we spend, um, um, if you create an episode for each of the sentences you made, it's going to take us a year to cover. And these are such rich topics. But I think we'll try to be uh, concise, uh, but also create Value for our listeners. And speaking of value, I think let me just say I fully agree with you and it's just a gigantic topic. Um, my personal view from 64,000, 64,000ft high, uh, even the current state of um, AI which is LLMs and essentially just as someone said, very smart parrot, um, who can talk to you and sometimes be even smarter than you, uh, even this is huge. And to me personally this is new Internet. So when someone says oh my God, what's going to happen to the jobs? Look what happened with Internet, right? How many software developers did we have before the Internet? I know 100,000 and now it's like 30 million. So um, when someone says the jobs are going to go away, uh, we'll cover the jobs. But uh, I'm just completely opposite of that. I think it's just, it's huge, is the next Internet. And uh, I'll get on, let's go, let's do it right. It's huge. And um, my personal note is that I thought hey man, uh, you know, 1960s, 1970s, it was heydays, um, we missed the boat, nothing fun is happening. And then guess what, we're right in the middle of it now and it looks like everyone is prepared. A lot of people are prepared to build amazing things and we love building things. So as a personal note, um, I'd like to cover or continue the topic of the value. Right where is the value right now and where is the value next? And I believe there are two uh, levels to look at it, what an engineering manager and his team can get out of LLMs today and the second, how we as companies and businesses are going to be creating value using LLMs. And these are almost like they're related because first one enables the second one. But for the first one I uh, believe that the value is that now you can access this gigantic um, planet sized body of human knowledge within milliseconds before it would take you. Depending on the task it will take you maybe from hours to days and weeks. Researching, reading, searching, composing, aggregating, trying to understand. Now you can talk to this knowledge and get answers within like tens of milliseconds. Just think of how much of value just in that the value is in where access to knowledge within minutes and seconds and milliseconds, not hours and days. Right. So it means that if you're an engineering manager and if your team is not using any of the uh, providers these days like you mentioned, OpenAI, ChatGPT, Gemini, Google, uh, meta, AI, uh, anthropic and a couple of other folks, you know, the big guys. And by the way I believe it's going to continue to be the big guys because small guys just don't have resources to buy those GPUs from Nvidia, um, and you know, other providers. So it's going to continue like that and it's a good thing because it's a very healthy ecosystem. Everyone's competing, we are benefiting. Your team must have one of those um, um, chatbots, um, continuously open in their browser. Don't know about you Adam, but in my case I'm a big fan of ChatGPT. It works. I have like 4 windows open at the same time and I didn't Google anything for like four months I think or maybe two. Right? Uh, uh, uh but I periodically use um Gemini because um, sometimes I need a second opinion and um, so I use that. So that's basically the immediate value and the immediate value, right? Coding, right? You stuck with a problem. You forgot how API works. You don't know how API works. Type it up. Ask a question. Ask the question about API. If you don't understand something, ask it to write the code. Will it write the code? That is going to be a production grade? Uh yes. If it's a pojo in Java, that's going to be production grade, even documented. But uh, uh, beyond that, uh, maybe not. But it's a source of inspiration. It's a huge time saver because you don't have to Google anything and try to parse through um, uh developer websites like Stack Overflow, uh so immediate access to um, software, um, development knowledge, even on some degree, uh, access to uh, management knowledge because you can have a conversation, um, you can use ChatGPT or any LLM as a source of inspiration or as a source of um, bouncing ideas. But the thing is very simple. You just ask a question and see where it goes. Sometimes it doesn't go anywhere, sometimes it's complete bullshit and um, it hallucinates um, and you ask wrong questions, you get wrong answers. So it's a skill but you have to do it now and you start building those skills with yourself, with your team now and the second part on the value and then I'm m going to shut up after this for a bit. I'll give you the microphone back. Creating the value on a large scale, creating the value in form of products that use LLMs, AI, right? That is going to be huge because now we can access that knowledge, um, we can provide that knowledge, we can use that knowledge, we can Convert company's data into knowledge and access it within milliseconds and seconds before you had run. You should run the, the um, um LLM, um, pipelines for days to get something meaningful out of them. Right? Not anymore. Right. And you can use RAG to put in your data. You can train your models on uh, ground truth of your company, like from even financial reports to detailed knowledge of how projects and everything else. Right? But there's like, there's knowledge and there is data, right? Knowledge is the ground truth. Data is what's happening in real time. So we need to be separating but and on the value and um, we'll get to the jobs. The value. To create value using LLMs and AI, you need software developers, right? If you are a, on any level and if you try to get something out of ChatGPT in a meaningful way fast, it's still going to take you hours and days sometimes, right? Even though it's like faster than month and years now. Uh, and I'll just give an example. I mean I needed to understand something. I just dropped in a PDF document and asked what was interesting to me and got an answer within two minutes. How would you do this before? Could you do this before? And I know that there are companies which are trying just to wrap, uh, OpenAI, whatever, um, Gemini APIs to simple APIs to a few calls and charge some money. This is going to go away very fast. Right? Because everyone can do it. Uh, because everyone can do it. There's not much value but the complexity of existing processes, existing data, uh, existing interactions with the external world. This will require very heavy engineering work. Right. Which is just beginning and people are asking, well it's been two years, where's the value? Right? Uh, where's the value? Well, guess what? It takes time to build products that use that new amazing tool. Right. And I believe, my personal belief, 2025 we are going to see the emergence of those companies that took it seriously, that took time to build great products that work and now they're showing the value and it's going to be mind blowing I think. So that's my take on value.

Speaker A: I think you're spot on on all of that and I think, you know, tying it back to value on both fronts, uh, you know, the team and then the business side is exactly the right way and I think that's what we're trying to guide here. Uh, just to respond to a couple things, I think just walking back when you say, you know, jobs aren't going away. Couldn't agree more. Um, Jobs, the people will adapt as the industry adapts. Right? I think that's what's going to happen. Uh, yes, there will be, there will be certain tasks and, and roles that are obsoleted because of the technology, but we will find new opportunities. Right. Those engineers that previously were doing X, that don't need to do X anymore, can now work on other things that um, that may have been uh, too ambitious previously. So I 100% agree with you that there's always going to be a role. And that's, I think one of the reasons why we want to talk about this because there's clearly guidance that um, engineers can take from this. Now walk backward from there a little bit and you say, okay, well if engineering teams are going to need to be successful with this, what do they do? And uh, you're absolutely right. Start to use the tooling, embrace that. I've seen companies where they kind of like almost started or have started to force feed some of the tooling down their developers throats to say, hey, start using this. And it's like, okay, you know what, that's great but what do we do with it? And to me one of the best responses to this, um, to this wave in my opinion has um, come from Google. Um, I think how Google has approached platform, platform engineering as a whole, you know, and of course they've been, they've been in the AI business for a long time. You know, they, they've, they tabled a lot of their work on that because they wanted to be maintained themselves as the leading search provider, which as you point out is starting to shift. Um, but nevertheless, you know, from their platform they've really started to bake in um, AI into the existing platform and infrastructure components that people use on a day in, day out basis with purpose. And therefore the engineers and engineering teams that use those products, um, are able to absorb the new wave of technology and the benefits of it while also still maintaining their core responsibility of keeping their current systems operating. Right. And I think that hits that sweet spot of okay, how do we be practical with this but also embrace this wave. And I really liked how Google has sort of embedded Gemini very strategically into its database engine and its compute engine and some of their other services that they offer to make it easier for you to um, uh, pivot in your journey and maybe eventually start to, down the road build meaningful apps. Because as you point out, it's not something you just build overnight. Yes, you can build a really cool app app, but does it have business value? Is it being purposeful? Is it being responsible? Um, all those things take time to really evaluate. And you know, some of these things require large, uh, investments in learning as well. So building the models to learn, not just the individuals. So I think that's a good example. So, you know, does that mean that everyone has to go use Google's tooling? No, I just, and I'm sure the other providers are, are, you know, have their own offering, but I really like how they presented that. I really like how they created a path for going from where we're at today, or where we were yesterday to where we're at today and then to kind of start to pivot to where we want to go tomorrow. And that just reinforced for me that, yeah, technology is not going anywhere. The need for engineering and engineers to engineer is not going anywhere. And I think when you look at using AI as a thread in your platform, in your tooling, um, you're going to be greatly successful in the long run because you're going to pivot without being, without knee jerking your way through this. And I think that's the first point where we can come to what an engineering manager can do. I think the first thing is, number one, embrace the reality of where we're in, embrace what tooling is out there and what it can and can't do for you and really find that sweet spot for your team and your platform where you can start to embrace the AI in a way that's going to be productive, going to enable you to still meet your business needs and support the business requirements that are being fed your way. Uh, maintain your current systems, but start to enhance them little by little at every layer in your stack. Not just building a nice little chatbot that can do like as you point out, slava wrapper on top of, um, OpenAI. That's a great starting point. Right, but it's not a product. And then I think that takes us to the second question of, okay, how do we build business value? And as engineering managers, we are always responsible for business value whether we like it or not. We can't just go off and just make. Just because the industry seems to be pivoting doesn't mean that we in our day lives are going to be able to pivot that quickly. And we shouldn't starting to embrace the tooling purposefully and starting to feed it into little things that enable your engineers to be more productive. But at the same time meeting your business needs is going to be super critical. Um, you shouldn't be force fed to use two's tooling just for the sake of saying you're generative AI compliant, but using it purposely, encouraging your team to understand the responsibility behind it. Uh, because you know, you also can't just put something out there on the market that's that, you know, is this great new generative AI product, but is highly irresponsible with the data or inaccurate. Right? Because now your customers are um, or your consumer base is going to suffer and we don't want that. So it takes time, you're right. And it certainly takes an investment and it takes uh, a strong leadership to be able to look at the tooling, understand the state of things and find the pieces that you think are going to be most effective for your team and find the models that are going to work best for your application. And then, um, pivot start, uh, to pivot and then go to your business and say, look, here's what we can do with this. Now we have the team that is aware of how to start doing this and they're getting trained and using some of these things and here's the incremental path that we're going to be able to provide for the business to continue to add value. Um, what do you think about that, Slava?

Speaker B: Yeah, good stuff. Um, um, I agree. So I think the model sort of beginning to what we are pitching today, the model is beginning to shape up and maybe today's episode, we just rotate around, um, and walk around, uh, and you know, take little pieces, take little bites of the value and maybe focus later, focus on things like jobs and um, um, um, what's next? Uh, because we should talk about Genai, maybe spend a separate episode what it means. Um, so in terms of value and let's talk about business value because I think in terms of engineering, it's pretty clear, right? You have to use, must have, you know, go, go. Um, um. And really what I mean to me personally, the question is what do you do if you work for a company that is completely scared And I've worked for, at a company that said absolutely no way, I do not touch it. And in fact all those Sites like Gemini, ChatGPT, OpenAI, Anthropic, they all blocked. You cannot work, you cannot call it from the, from the working computers and it's like, hey, it's Internet. But we are going to burn soft our software on CDs and ship them over USPS, right? And send catalogs with forms, other forms that actually, yeah, you go, how long are you going to survive? Right. That's the sort of like a backwards uh, attitude and um, I'm um, thankful that I'm not participating in this anymore. Uh, um, on the value, particularly on the business value, uh, it is unlikely unless you are AI first startup. It is possible and it's possible that many companies will come and ask, but how do you start, how do you start creating value as a company? As uh, a company that is pro AI, pro LLMs, right. And the value always comes from a simple thing. You ask your customers, what is your problem? How can I help? How does a solution look to you? Or if a solution existed to a problem, how would you like to see it? So basically always comes from the customer telling you what the problem is. And I suggest that the first step, there are two steps in this, um, in beginning to create value for your customers who pay you money for your work. Number one, pick a simplest, if not a trivial use case. Maybe it's already solved in your system, maybe it's not, but it has to be something very simple. I don't know if you're an email provider, maybe figuring out if it's a spam or not, right? Solve that. Or if you're selling furniture, solve a simple question of what product should I show first on our website? That's it. And solve it with this new amazing tool we've got now. And then once you solved it, you're going to learn the APIs, you're going to learn uh, uh, prompts, you're going to learn uh, data formats, you're going to learn what comes out if you ask wrong questions, you're going to learn how to ask right questions, you're going to learn how to know you've got the wrong answer, right? So you're going to learn all those things. And once you've done this first, like a, uh, day one iteration of doing very simple things for your customer. What I propose, and I encourage everyone who is listening, pick the hardest unsolvable, unsolved and unsolvable problem and go and solve that. It's not going to be easy, but it's going to be challenging. But because you have this new tool, you're not going to be thinking in the old terms anymore. When you are going to approaching uh, those problems, you are going to be asking yourself how can this problem be converted into a natural language that model understands how I am going to feed the data. Is it the first principles, is it the ground floor truths or is it the operational data? And then your thinking is going to be, you're not going to be thinking about nlp. That takes Days to crunch through the text. You're not going to be running Hadoop clusters anymore, right? All that stuff goes away. You just. That complexity and hardness and impossibility of dealing with natural language, poof, it disappeared, right? But if you do not take on the hardest problem in front of you, it's going to take you years and tens of years to get where you want to be. And by that time, you're going to be solid and your competitors have done it. You're done. You are done up. So when I'm saying that, do the simplest one, create a minimal possible value or maximum value out of the simplest use case, and then pick the hardest unsolved and unsolvable one and solve that. That is my proposal for creating, uh, value for the customers and, um, being paid for that.

Speaker A: I think that's also spot on, Slava. And I think, you know, there is, I think, a lot of value to looking at what the most challenging thing is and going after it. Um, especially if you have some good fundamentals in understanding what we talked about previous to that, the tooling and the space and the requirements. You know, I think it sets yourself up to be able to, um, go after some of those larger problems. Um, so if we were to kind of sum up what we're saying here in this initial talk on AI, um, I think the guidance that we're providing here is, number one, recognize the environment that we're living in. Recognize a little bit of the trends and where we started to see things start to converge. Understand this is a, um, pardon the pun here, but really a generative generational change in industry and technology. Um, it's a huge pivot. And to approach that pivot successfully, number one, understand the tooling that's available to you, become comfortable with it and use it responsibly, especially, uh, for the use case that is applicable to you. Uh, don't just take whatever's out there and try to build a chatbot, for example. Um, understand the tooling, understand how it connects to your existing platform. Understand, uh, some of the threads that are being baked into the platform components that you already have with some of the vendors that are out there. Then define what is going to add business value for you, for your company, for your product. Um, communicate that, embrace that, and connect how you're going to help pivot your engineers, um, to embrace the new technology, because as we recognize, engineers aren't going anywhere. Um, but how we pivot is important. Recognize how they can approach and adapt and support the business value. Define the business value and then lastly, as you mentioned, go after some of the most challenging, uh, ambitious opportunities out there and take time to do it. Right. Um, what do you think about that, Slava? Did it capture what we were saying?

Speaker B: Yeah, I think this is a good step. Tough. Uh, and it's a good start. And, um, um, I think we really covered the value well today. And, um, uh, in the next episode, I suggest we, um, talk about, uh, the impact on the economy and on the jobs, how we see it, what it means for engineering managers, and then see, uh, where it takes us. Where it takes us.

Speaker A: Yeah, absolutely. Um, thank you, Slava. So more good stuff is coming. As always, we encourage our listeners to share this episode if you like it. As always, we are looking forward to any feedback and suggestions for future topics or letting us know how you're using some of the guidance that we are providing here. We'd love to hear about your story and, uh, we'd like to mention that we do offer customized training on effective engineering management to help companies. Companies accomplish more with less. You can reach us@contact effectiveem.com by email or reach us, find us on our website, www.effectiveem.com to learn more.