Our Cybersecurity Mission: AI In Development

Speaker A: We believe if you're going to do it, the audit should be worth it. The problem is audits are hard. Yet We've issued over 20,000 reports to 2,000 clients worldwide.

Speaker B: If you want to experiment with how something, one specific feature could look right, you can potentially write up a decent prompt for it and you can have it maybe working in 20, 30 minutes.

Speaker A: Cybersecurity and compliance will no longer be a mystery. Hey, everybody.

Speaker C: Welcome back into our cybersecurity mission. My name is Ali Krings. Today we're sitting down with a repeat guest, Robert Clark. How are you?

Speaker B: Good, how are you?

Speaker C: I'm great. Senior team lead on the development team. Is that right?

Speaker B: Yep.

Speaker C: I mean, this is so fun because last time you were here we talked about vulnerability management. We've had you on another podcast. You can find those all@kirkpatrickprice.com podcast. But today we're going to talk about AI and development. Before we get there, tell me what you're working on right now. What's going on?

Speaker B: Um, we've actually got a lot going on right now. Um, funny enough, we've kind of accelerated some of our development by utilizing AI. So we've kind of been working it into our workflow and I think that it's caused us to have a really good quarter, um, just because we've been able to move so much quicker.

Speaker C: Are you surprised, like, looking back now, because, I mean, it wasn't like all that long ago that we were in college. Are you surprised now how the industry has adapted into AI?

Speaker B: I, uh, think it depends on which part of the industry you're working in, who you're working with. I think we've done a good job at career pressure, price of being ahead of the game and kind of adopting a lot of this new AI, um, stuff early on. Um, but I think it really depends.

Speaker C: It's such a fascinating topic because I think so many people are interested in it, they're intimidated by it, they're not sure how they can implement it to make their work smoother or faster. Or on the flip side of that, maybe people are also nervous that it will take away from the quality of their work. Were there conversations that you had within your team before you adopted some of the AI tools?

Speaker B: Um, I think we've been continuously having them, um, because it changes so often. So we've had discussions around potentially, um, slowing down sometimes to make sure that we're not going off, like, moving so quickly because we're just able to generate stuff so quickly with AI. Um, we've also had conversations about how to use it better as a team and kind of use the same tools as a team so that we're all working with the same tool set and we're all kind of getting the same results out of the AI. Um, so yeah, there's a couple different pieces there.

Speaker C: So let's start from the very beginning when Kirkpatrick Price was like, hey, let's implement some AI. What were the first things that you wanted to bring in to the development side?

Speaker B: I think one of the biggest pieces was just having an established standard for everyone.

Speaker A: Mhm.

Speaker B: So we've done a few iterations of this, but basically finding one tool that the whole team is using, that's an AI tool obviously. Um, and uh, then kind of sticking with that and then kind of building on top of that single tool.

Speaker C: What tools did you guys try out and what tools did you end up liking?

Speaker B: Um, we've tried a couple different tools, different ones. So there's ah, on the Microsoft side there's Microsoft Copilot, which is um, built in with GitHub directly integrated there. Um, we've used that one a good amount. Um, we've also experimented with on the anthropic side, Claude code and uh, some other tools there, agent tools on that side as well. So we've kind of tried both sides of the market.

Speaker C: So on my side where I primarily work in like a marketing facet with Kirkpatrick Price, um, you know, I use Copilot from time to time. I don't have any experience with Claude or the other one that you mentioned. What did you like, what did Copilot maybe offer to you but lacked in other ways that you were seeking other tools?

Speaker B: Um, I mean there's been a lot of little pieces.

Speaker C: Yeah.

Speaker B: And it's kind of complicated, uh, to break it down into one single thing. There's kind of just the overall experience of your using them, um, trying one and it's really just finding what you and your team likes. It's also the extensibility of them like being able to extend it and add additional pieces. Um, that one maybe supports better for your team than the other as well. And yeah, it really depends. I don't want to make a direct recommendation for one or the other. I just would say play around with them and try them out with your team.

Speaker C: So what did you like about Claude when you started using that one?

Speaker B: Um, just the fact that you could extend it more easily to add your own custom skills, your own custom things inside of it and it would kind of, and then you could, um, leverage those as a team as well, like share them with your team. That's pretty useful piece.

Speaker C: And did you mention there was a third one that you said?

Speaker B: Um, not that we've experimented with. There are a bunch of other ones out there that are, um. Yeah, there are other ones out there, but, uh, I haven't experimented personally with any of the other ones.

Speaker C: Are there any that you're, like, curious with or are you happy with the work that Claude and Copilot can help you with?

Speaker B: We've been quite happy with what it can do for us.

Speaker C: Yeah. Was there any pushback on. For many members of your team, thinking, why should we try out this new technology? And I think for a lot of people, it's also, if we do this, are we cutting team members? Did you have any conversations like that?

Speaker B: Um, yes. I, um, think that there's.

Speaker A: There's.

Speaker B: There's been different levels of, um, uh, I guess concern throughout the team on that. I think in our case, um, it's. Once we've kind of had a standard and brought everyone in and kind of shown everyone what. What it can and can't do, I think everyone start to feel more comfortable. There's still. There's still very much a need for a human in the loop on a lot of this.

Speaker C: Oh, totally.

Speaker B: Um, so that's why we haven't really had a discussion about needing to cut people. If anything, we have more work now that we're able to tackle as a team. And so it's not. There's not been a need to reduce anything. It's just helped us all move faster.

Speaker C: So can you give me maybe a couple specific examples of where introducing the use of AI has made your life as, you know, both an employee, but maybe as like a family man? It's made your life better because maybe you can get things done faster and you're feeling less stressed.

Speaker B: Sure. Yeah. So, I mean, one of the neat things is if you want to experiment with how something, one specific feature could look right, you can potentially write up a decent prompt for it and you can have it maybe working in 20, 30 minutes. Um, not production ready by any means.

Speaker C: Sure.

Speaker B: Um, and that's where the human comes in. But you can definitely get really far, really quickly.

Speaker C: What do you mean by that? Um, if you were explaining this to a first grader. I'm the first grader. Uh, how would you explain this? You're giving them a prompt to create code to.

Speaker B: Yes. So, um, let's say that you have a certain new button you want to have on your website.

Speaker C: Like, I want to have a button that says, um, click here for the podcast.

Speaker B: Sure.

Speaker C: Ok.

Speaker B: Right. So, uh, previously we would go in and we would manipulate the code manually to add that button. We'd try different styles of it.

Speaker D: Sure.

Speaker B: We'd say, oh, do we want it in green? Do we want it in blue? Do we want it in red? Do we want it to be this big, this small? Um, all these things. Now, instead of going into the code and doing that, I could write a prompt up that says, I would like a button on the top right side of the page that is potentially. I could even say, let's play with a couple different sizes and it could mock up all three of them on the page. And I can go back and forth, say, I don't really like any of those. Let's try making it skinnier or taller, or move it to the left instead. Right. Um, and I can do all of that without actually having to manipulate any of the code directly. We can just have a conversation about it and then I can see how it looks Interesting.

Speaker C: So how long, let's say you don't have the access to AI in this scenario. How long would it take for you to create that button? And, uh, maybe after the third try, you finally get the color and the size right. What's the time consumption on something like that?

Speaker B: So in the case of a button like that, it wouldn't be very long. But where it really comes in handy is for more complex features. You click the button, another page comes up, you can, can fill out a form, you can do all this other stuff. It's when you start chaining those things together and you kind of want to see the whole flow of that. Um, it becomes a lot more valuable to just be able to say, I want this workflow to happen. And then I can kind of visualize it, see if it's even a direction I want to go. And if it's not, I can throw the whole thing away and not feel that bad about the fact that I did not spend two hours building this out just to see it and not like it. I spent 15 minutes on a prompt.

Speaker C: Yeah, that's incredible. I think we've all been in situations where, you know, you work on something so hard and it is so time consuming all to find out later that you've done something wrong or it just wasn't the vision that the team was looking for. So this sounds like a really nice shortcut. With that being said, it's really important, especially in you know, in our firm, where it's a cybersecurity company, to keep everything safe for one. So how have you maybe implemented some controls to ensure that the AI does what you want it to do and not what you don't?

Speaker B: Right. So when a developer puts up, um, code for review, the expectation we have is even if they used AI for it, they still look through it themselves.

Speaker C: Okay.

Speaker B: Um, and then we still require, uh, reviews by a human, um, on that code as well. So a separate person, not related to the one who actually originally had the conversation with the AI to build it, has to come in and also look through it and approve it as well. Um, so we still require human in the loop there. And then on top of that, we've also begun using some AI review agents to also help with that.

Speaker C: Oh, interesting.

Speaker B: Um, so, uh, on top of the human having to review it, uh, right away, when someone puts up any sort of code, uh, AI automatically reviews it and gives feedback on it. Um, there's definitely been some good feedback from it.

Speaker C: Really?

Speaker B: Um, yeah, we've had some times where we've read it and said that definitely that's something we should fix. Um, there's been other times where it gives bad feedback, and that's a whole separate issue.

Speaker C: But, um, when it gives bad feedback, how do you know it's not giving you, how do you know it is bad feedback?

Speaker B: Uh, in our case, we either test what it says and it doesn't work as it's saying, or we just, um, are more familiar with the code. Since all of us have been working in this code base for a long time, um, we all have a sense to say maybe that isn't right. So I guess intuition sometimes,

Speaker C: um, have you. So obviously this, like, peer review or having another, um, team member from the development side check over your work, your code, that's not something new that you've implemented, it's just the addition of the AI. Has that peer review process become now full, faster and more efficient, or is it still the same amount of time because of the just due diligence?

Speaker B: Yeah, I'd say it's still on that side. It's the same amount of time, um, and it's almost a problem because we can generate a lot more on the front end. And now the burden is on the back end of actually reviewing all of this code that's being proposed.

Speaker C: Interesting.

Speaker B: Um, that's kind of where the AI agents doing some review has helped cut down the amount of feedback during the first pass of a review that's needed Just because it does catch some stuff upfront without a human having to be involved yet. So that's one way we've been trying to mitigate too much burden on the developers. But, um, yes, it definitely is an issue.

Speaker C: What other controls, um, do you have in place to ensure that AI is doing what you want it to do?

Speaker B: Yep. Uh, so we have, uh, well, we have limitations on what it can and can't do. So, um, there's only certain commands that it's allowed to do.

Speaker A: Like what?

Speaker B: Hands off. Uh, just, um, ah, like running system commands. It's only limited to certain things that it can access without prompting someone to say, I need this, and then us having to approve it or deny it. Um, so we have a default allow list of things that are allowed and then everything else has to be manually approved at that, beyond that.

Speaker C: So one thing I do want to talk about as well, because again, AI is such a hot topic. Um, one of the things that people are, I think, nervous about is the fact that these, you could be inputting data that they could be learning from. So how can you be sure that the AI tools that you're using are keeping your data sensitive and not, um, you know, saying, hey, Kirkpatrick, Price is using this really great code. Everyone should try it.

Speaker B: M. So that's one of the important things about actually having an established, um, tool for the company that is the allowed tool to use. So that way whoever's handling the procurement of the tool can make sure all of that is established and the individual users don't have to deal with that.

Speaker C: Yeah.

Speaker B: Um, otherwise you're in a situation where it's. Every single person has to get their configuration right and that's not going to happen. So you definitely want to have it be a managed tool for the company.

Speaker C: Are you seeing in the industry, um, and maybe not our industry specifically, but I think about like ChatGPT and how so many, you know, my siblings and I, we all work in different industries, we all love reaching for ChatGPT. When I'm doing something specific for work, I'm always using copilot. Um, but are you seeing people use ChatGPT and maybe not realize that it is becoming now public information?

Speaker B: Yeah, um, that's one of the big. Yeah, definitely. That is the big issue. Right. So I do have heard stories of people putting private information into public chat.

Speaker C: Do tell, do tell.

Speaker B: Um, I mean, I know it's happening. So, um, it's definitely. If you're working somewhere and you don't have an approved chatbot, that Is I'm just, I guarantee you people are using it on the back end, just doing shadow it, doing it themselves. Right. So that is a huge concern. Um, so, yeah, definitely having an established

Speaker C: tool, very important how, you know, if somebody's listening to this and they're saying, hey, we haven't quite made that first step toward getting our company, ah, approved tool, what are some considerations that they should think about before implementing a certain one?

Speaker B: There's a lot of options out there right now. You mentioned Microsoft Copilot, right. So that is, um, an easy one for people to get started with, just because it is kind of included with everything Microsoft it feels these days. So, um, if someone is just concerned about getting started with something, it might be a good option as, um, a starting point if, uh, you're already in Microsoft, right, With like all of your users and uh, devices and management for all that. Um, and then, um, if you're looking to branch out, I mean, there are team plans for each of the team and enterprise plans for all of the, uh, other companies as well. OpenAI has a team enterprise plan and, uh, Claude has an enterprise plan as well. So, um, really just making sure that as you're shopping around that whichever ones you're using allows you to turn off training on your data, uh, and allows you to, um, kind of manage, make sure that people are actually going through the proper channel, um, to get to it. So, for example, there's with Microsoft, there's the one inside of your company. There's like the company version, and then there's the personal one that's still kind of linked all in the same place. And we actually have that personal one completely banned. Uh, you may not have noticed it

Speaker C: because I had no idea.

Speaker B: Um, but it's a mistake you can make. And if you go to that one, it does actually, I believe, still train on that data or it's less isolated into our corporate environment. Right.

Speaker C: Yeah.

Speaker B: So, um, just making sure that you have controls in place that direct people to the correct version of it. And then, um, just. Yeah, actually having an approved one are really my two biggest takeaways.

Speaker C: Were there any, um, you know, when your team decided, hey, we're adopting this tool and we want to see how it can make our life, I mean, easier.

Speaker A: Right.

Speaker C: Um, were there any pushback on which tool should be used or which tool shouldn't be used? Or was everybody kind of like, okay, yeah, like, I'm ready to try Copilot?

Speaker B: Um, we had a lot of discussion around it.

Speaker C: Yeah.

Speaker B: Um, we tried a couple different things before we landed on where we're at now. Um, and definitely there were pros and cons of each. I think right now there's been, at least in some circles, there's definitely been winners in the market for sure that people have kind of gravitated towards. But yes, there's absolutely been discussion around do we go this way, do we go that way? Um, and it took us a while before we landed on where we're at.

Speaker C: What was it that made one seem better or more intuitive or more easy to use than another?

Speaker B: Um, I mean, I think it really depends on preference a little bit.

Speaker C: Sometimes

Speaker B: there's not really necessarily one thing for every person where I would say, oh, this is the thing that made it better. Um, in our case, some people like using more text based tools, some people like using more graphical based tools. Um, that was kind of at the beginning a decision we were grappling with. Like, which do we like more? Um, there. Um, but yeah, I mean, it's all moving so quickly. I don't really want to say this one. If I say one thing, it's like it could be different in three months.

Speaker C: So, um, which is exciting. But is that also maybe, um, intimidating to be like, are we in it for the long haul with this tool or will we find something better?

Speaker B: Yeah, it's very, it's very possible. Yeah, we don't, we don't really know. It's all moving very quickly.

Speaker C: So when it comes to development, what kind of prompts do you put into Claude? And um, can you maybe just like walk us through what writing a good prompt looks like and also what writing a bad prompt would look like?

Speaker B: So I think, at least for me, the prompt kind of depends on how, um, kind of what I'm looking for. Right. Okay. So in the case of that button styling thing we were talking about earlier, um, I potentially would just make it a very short prompt. Just say what I said earlier. I need a button that is here and just see what happens. Right. Um, I don't really like to write super in depth prompts until it's failed at a very short prompt. Um, and then as it's kind of failing on things, I'll start making it longer and longer, um, and eventually get it to be potentially this whole paragraph of stuff. Um, but yeah, really I just start short and just see where I can get and then kind of iterate from there.

Speaker C: What were maybe like the, some of the first projects or questions that you had when it came to using AI tools?

Speaker B: Some of the first projects were um, really just seeing how far I could get with building out a feature. So like describing a feature and saying we want to have this workflow.

Speaker C: Mhm.

Speaker B: And just seeing what happens. Um, and early on it's gotten so much better in the past six to nine months honestly. Um, I'd say maybe a year ago trying to do that you'd get really varying levels of results. Now it can really crank through making something that works more or less um, first try, um, even my personal uh time I've experimented with some basically like Windows app little projects just to see how it could do. And I've got one that I've been playing with that since it's for work. I haven't even looked at the code for it and it's built this, this whole thing um, that I've just been kind of just adding stuff to and as I add stuff it's still just, I haven't looked at the code at all and it's just building it out. So it's kind of funny to see stuff like that now.

Speaker C: What's your education like by the way?

Speaker B: Uh, I have a bachelor's from USF in computer science.

Speaker C: So do you think that the curriculum or education um, in computer science degrees, do you think it's going to shift because of this AI and the revolution that we're kind of seeing?

Speaker B: I mean I think it's going to have to a little bit. I don't think that. I think that even from a, from the perspective of testing someone on what they know and giving them a take home project.

Speaker C: Yeah.

Speaker B: I don't even have a. I feel a little bad for teachers. I don't know how they can, how you can really give someone a take home project right now and not, not and grade it knowing that they potentially just asked an AI build me this whole project and it could probably do any school based project uh, in coding.

Speaker C: Man, it would have made our school work so much easier.

Speaker B: Yeah.

Speaker C: You know, but it's a conversation that we've had amongst other cybersecurity professionals is now people are coming out of school and maybe don't have the right training or knowledge or support. Um, you know if they're so. But however we know that Gen X is it Gen Z, Gen X, whoever, whoever they are who is graduating, um, they are some of the first adopters of AI tools. Do you think that's going to widen the gap of them not having maybe some of the right foundational um, information that they need?

Speaker B: Definitely. I think that if you don't, if you haven't had to struggle through some of it. It might be a little difficult when you get stuck with a problem where you are hitting that AI can't solve.

Speaker A: Um,

Speaker B: I don't know if me or the education sector at this point has an answer for how to still give people that foundational knowledge when, um, a lot of stuff can just be answered with an AI without really having to learn the steps to get there. Um, yeah, that's a complicated question.

Speaker C: Well, I'm just thinking about how you've now written this great code with the help of Claude, and you're asking your peer to like, hey, just double check this. If that person doesn't have just the foundational groundwork, um, that would be a really challenging task to do. Right.

Speaker B: For sure. Um, and I mean how you can leverage AI to talk to it about the code and say, what is this doing? But once again, that's still not just you knowing and being able to read through it yourself. So that is definitely a difficulty.

Speaker C: So we're going to still have to learn, I guess. Um, to somebody who's thinking about implementing AI and implementing controls to ensure that their data is kept safe and kept private, what are maybe some words of encouragement that you have?

Speaker B: There's just a lot of information out there right now. So, ah, definitely just read up on what other, uh, people are posting, uh, about this online because everybody is kind of going through this together at the moment. Um, so definitely, uh, find peers that are providing their shortcomings or things that they've run into, um, online about. That is definitely a good place to start reading on that.

Speaker C: Um, are there specific areas that you like to look like? Are you seeing, um, your peers post on LinkedIn about things they found? Are you looking on Reddit sites or are you seeing actual publications come out with information about this? What tools do you like to use online?

Speaker B: I mean, I wouldn't say that it's necessarily big publications, but people tend to have their own blogs where they will write stuff out. Um, and yeah, it's really all over. I mean, like you mentioned, LinkedIn has it going on there as well as just other social sites online.

Speaker C: Okay, can we, I'm gonna challenge you. We're gonna come back to this in one year, we're gonna have this same conversation. What do you think, what do you predict one year from today that this AI tool is going to be able to do for you?

Speaker B: Um, I mean, considering right now it can already more or less build out whole features.

Speaker C: Right.

Speaker B: My hope would be that it is fine tuned More so that it requires a little bit less hand holding on our side to get it to the right place. Um, so just really improvements in code quality would be amazing. Um, and then, um, just improvements in security reviewing and uh, just the regular reviews that it can provide on our code would be fantastic. I would honestly take just incremental improvements at this point, um, because it really is pretty good right now. But if it could just get a little better at some of those things, it would be, it would be great.

Speaker C: Awesome.

Speaker B: Um, I don't know how we could have a whole nother complete revolution again, um, on top of this currently, but I could be wrong.

Speaker C: And that's what I think is so exciting about it is 10, 15 years ago. This all sounds like a weird sci fi book that you would read and be like, well, that's never gonna happen. Um, so Robert, I want to say thank you so much for sitting down with us. I love hearing your perspective and you know, uh, as a senior developer developer, a team lead and we're going to have more podcasts before one year comes back. But in March of 2027 we're going to come back, we'll check in with you guys and see where we're at with AI and how you've adopted it. So thank you so much Robert, I really appreciate it. And my friends, we'll be back again soon on our cybersecurity mission.

Speaker A: When you work with with a Kirkpatrick Price cybersecurity auditor, you work with someone who's been in your shoes. Let's see what the next expert is ready to share to support you in your cybersecurity mission.

Speaker D: Hi, my name is Jason Kent. I'm a senior information security auditor with Kirkpatrick Price. Um, take a minute and just imagine with me your IT systems as a medieval castle. Um, a formidable fortress with high strong walls but with a old wooden door, um, and secret passageways to the perimeter. System hardening is like going and finding these secret passageways, closing them, going to the door and reinforcing your doors and your gates. Um, this reducing the attack surface makes it much harder for bad actors to gain a foothold in your systems. Excellent resources for um, finding hardening information. Um, would include the center for Information Security, CIS benchmarks and also um, vendors best um practices for security and configuration of their products.

Speaker A: Today's episode highlights the need for secure application development to achieve our challenging compliance goals. Integrating security and development in your business pipeline will strengthen your applications from attack. Are you ready to join a community of 10,000 people who are working together to elevate the standards for cybersecurity and compliance. It's free to Sign up@kirkpatrickprice.com podcast or check the show notes and achieve greater levels of assurance today. Thanks for joining us on our cybersecurity mission.

Speaker B: It.