The Hidden Security Risks of AI Coding Agents

Working with agents and LLMs introduces some new dimensions on security that we wouldn't normally think about. Text becomes something that can be vulnerable, can have some executable intent. A couple years ago, you would never worry about the security of a markdown document. A combination of some access to privileged information or some private data, some exposure to some untrusted content, maybe from the outside world, and the ability to communicate externally.

Most agents by default are going to have all of these three things, and that is where you're one prompt injection away from uh let's say bad time. The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy for Germany, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev.

Back in November, we hosted the first ever in-person AI Native DevCon in New York. This June 1st and 2nd, we're bringing it to London. It's two days built for AI native developers and engineering teams. One day full of hands-on workshops, and one day full of practical talks on agent skills, context engineering, agent orchestration and enablement platforms, and how teams are actually shipping AI in production.

Join us at the brewery in London near the Pubican for all of them, plus networking, parties, giveaways, and a room full of people building the future of AI native development. You can also join us from anywhere in the world even via the live stream. As you're listening to this podcast, you get 30% off your ticket with code POD30. Just head to aiNatavedevcon.

io and we'll see you in London. Hello and welcome to another episode of the AI Native Dev. And on today's show, we're going to be tackling the big questions of how organizations should be thinking about security in agentic development today. Not just for the security issues that we already know about that software development has taught us over the years, but thinking about some of the new security risks that agentic development brings today.

Joining me on today's show, we have Joe Holcroft, a member of technical staff and team lead at TESL. Welcome, Joe. Thanks for having me back. How absolutely.

And it is having you back because you joined um, gosh, probably a good six months, nine months ago, we did a we did a session all about how we use different Agentic tools. Uh and we actually swapped applications and played with that. It was it was good fun. Um but Joe, of course, you are a member of technical staff, a team lead here at TESL.

Um, previously worked at Sneak. Tell us a little bit about that. True, yeah. Worked at Sneak for three years, sort of during that hyper-growth period, mostly supporting the open source products.

Awesome. And uh, in between those two, you also are fractional CTO and did a number of uh different mentoring uh gigs. Really interesting to hear your point of view around security and agenda development from all of those experiences. And very recently, you actually worked directly on the Sneak and Tesla integration, uh, adding security advisories and reports into the Tesla registry uh for skills and context.

So some super cool stuff there. Yeah, it was fun to get to work with some old friends. Some old memories, absolutely. So why don't we jump in and talk about how the attack surface has changed and what we need to be concerned with from an agenda development point of view.

For sure. I mean, lots to talk about here, but I think um one of the best uh articulations I've ever heard of this is um in Simon Willison's uh Lethal Trifector, um, where he talks about you know the combination of um three characteristics that um all agents pretty much have, which is um some access to privileged information or some private data, um, some exposure to some untrusted content, maybe from the outside world, um, and the ability to communicate externally. Um and most agents by default are going to have all of these three things, and that is like a dangerous situation.

So you think about a coding agent that you're working with on your machine in a repository that has access to the code and secrets um that you have on your machine, which is your private data. They're pulling in dependencies, documentation, issues, um, search results from the outside world, that's your untrusted content. And they can run shell commands and hit uh HTTP endpoints and CLI commands, um, and uh that's your communicating with the outside world. Um and really the those combinations of three things is is sort of where you're one prompt injection away from uh let's say bad time.

Yeah, yeah. And it's interesting actually, and too, you said prompt injection there. A lot of those things are are uh are fairly consistent with you know some of the issues we have now, with the exception that you know you have an agent doing it this time around. So perhaps when we think about um, you know, what's what we need to to do to make sure we have solid code quality, uh solid uh you know vulnerability management, those types of things, um, you know, what is it we need to do to our existing processes to cater for these types of issues?

There's a bunch of things you need to do, but I think um maybe the first thing to consider is that uh a lot of the same um considerations that you might have for software that's written by humans is the same kind of thing that you need to care about with agents. I think if it's like a force multiplier, right? So all of the same um hygiene that you'd be doing, making sure you have your uh SEA scanning and your SAS scanning and your um processes in the right place in terms of security.

Um, all of that stuff um becomes more important, not less important, uh, as agents are working on uh your code because you know you can consider them uh somebody that you might trust less than you otherwise would, like uh, for example, a um uh a contractor that you might get in, someone who's going to be writing a lot of code for you. They'll kind of do whatever they're told. Maybe they have a certain amount of access to your internal systems, um, but there's not a huge amount of oversight, so you can kind of think of your agents in that way.

But there are a few ways in which um working with agents and LLMs introduces um some new kind of uh dimensions on security that we don't wouldn't normally think about. Most interesting one, I think, is uh text becomes uh something that can be vulnerable, um, that can have some executable intent, right? Um couple years ago you would never worry about the security of a markdown document anywhere in your organization. Uh, and today that's actually very important.

You have a markdown file that you're giving to an agent could encode some vulnerability uh to instruct the agent to go and do something malicious or just unsafe. Uh and that's just a completely novel situation. Yeah. And it's interesting that a lot of the time we we find ourselves, we get caught catch ourselves talking about how agents add security issues into the space.

But how reliable are agents uh not just finding issues, but even you know being able to start coding in a reliable way? We have heard of things like Mythos coming out. Um can we rely upon and do you rely upon agents to identify issues and help you guide you into making sure you develop more securely? I think you can for sure.

Uh I mean agents are always kind of backwards looking, so uh there's like a little bit of a lag before things are gonna catch up with new um types of attack vectors and and new vulnerabilities. Um, but absolutely, I mean, and I think it's kind of the only effective uh defense that we have against the kind of vulnerabilities that you can introduce into text documents and markdown files these days, is to have another LLM judge that content and see if it kind of seems risky because static analysis is not gonna cut it here.

There's many ways you can phrase something to kind of um not even trick an agent, but encourage an agent to do something that maybe isn't what your uh what the consumer um is intending. So so I guess the classic question is um do AI coding tools make developers more or less likely to ship vulnerabilities these days? Yeah, I think it really depends. Um so of course, uh, you know, agents are allowing you to uh be a lot more productive and ship a lot more features and a lot more code.

And the more code you write, the more chances you have of introducing vulnerabilities. Um but also they're pretty good at like following patterns as well. So um you know, if you have good, well-established patterns in your code base that make it quite difficult for you to um be vulnerable to things like SQL injection, for example, um, then they're gonna tend to follow those patterns. Um but then on the other hand, if you haven't got well-established patterns um or you're you know you're not managing the context with your agent well to sort of encourage them to do things in the way that you need them to uh to be done, um then they're gonna fall back on training data.

And the training data obviously is like a wealth of insecure open source software that uh is out there on in the world today. Yeah. And you go and going back to what you mentioned previously about text being vulnerable, that's really interesting because there are very few patterns. It's almost it kind of reminds me a little bit like WAF security.

Like when you think about uh something that whereby the input can change very, very slightly and go round a rule that is a pre-existing rule. Essentially, you know, what happens when text can become a vulnerability, things can be phrased in a slightly different way, or things can be approached from a different angle, and you can't have static tests for every single thing. So you very often do rely on a kind of LLM as a judge assessing whether a certain text is potentially dangerous or not.

So when you think about skills, a skills MD is essentially a bunch of text, and that text describes and guides an agent into doing something a particular way. In order to be able to identify whether that's malicious or not, one of the things that was added to the uh TESL registry recently by yourself and the team is that ability to uh to scan the skills with uh Sneak and get those results via Tesla. Tell us a little bit about that. Yeah, so we've introduced um Sneak's uh agent scan tooling into uh into the product.

Um so everything that's published to the registry um now goes goes through this skill scan. Um and also we're indexing a bunch of um skills sort of out there on GitHub and running uh security scans for those as well. Um so if you're trying to install these things with um TESL, then we're gonna warn you if something is uh like has a has something that we think might be uh malicious or maybe uh you know malintent. Um the other interesting angle on this is that a lot of the potential security risks here are kind of contextual.

Um so you know, uh a skill contacting the outside world or using some API may not necessarily be a bad thing. You know, TESL has a skill that is using the TESL API. That seems fair enough. Um it's important that the scanner has that kind of context.

Um, but a lot of the time, really, what you need to do is surface that to the consumer, to the user, right? Like, um, of course, many of the skills I'm going to be using are contacting the outside world in some way. Um, but am I okay with this skill using this particular API or this particular CLI command? Um, and you know, is that appropriate with uh what what the skill is trying to do?

Um so we're sort of working on evolving the user experience a little more to um not just flag up things that are like obviously malicious um packages, because in a lot of the cases is a little more subtle than that, that perhaps this skill is um doing a little bit more than you might be comfortable with if you just skimmed the skill MD, you might not realize that it's also going and hitting GitHub and posting some of your personal information there, for example. And a lot of the issues that we've talked about already are some of which people will be very familiar with, others are completely new, and there's a just a growth in the attack vector that we have as as or attack surface, rather.

Um one of the styles that is uh something like a supply chain um dependency risk. Sometimes it can be you can look at that from the libraries that we actually import, other times it's supply chain through the context. Uh let's talk about what's already existing. Let's talk about uh libraries first and then we'll jump onto the other.

Um so when we think about libraries and and AI agents using um libraries to assist with its abstraction coding. You know, what what should we be worried about when an AI is trying to, uh or rather, an agent is trying to choose a library when it wants to to perform a certain task? There's a number of things to think about here. Uh and the first one, as with many of these things, is just like velocity.

Uh, more code means more packages are getting added, which um is harder to review, uh, and maybe you're gonna be less detailed with a review that you're doing. One of the other things to keep in mind is that uh, you know, a lot of the time um a agent is just gonna hallucinate a package name. Um it's just gonna guess at something and see if it's there and install it. Um and there's uh uh a beautiful new phrase that's popped up called slop squatting to sort of describe this um this attack vector um where people can uh upload malicious packages on um package names that they think an LLM is likely to pick for some kind of scenario to then get that into your into your supply chain, which is an interesting one.

Um and the other thing to keep in mind is that uh agents these days at least are not really looking at packages in the same way that a human might. Um I think like if I'm choosing an open source package, um, I'm not just like looking for the first search result that matches the kind of task that I'm trying to do. I'm also looking at them for some social proof, you know, how many stars do they have? Are they well established?

Have they been around a long time? Are they sponsored by a company or is it an individual? All of these kind of things that um an agent typically isn't looking at. It'll, you know, maybe pick something that's well represented in the training data or just something that kind of matches its its search of what it's trying to do well.

Um so there are a few new things to consider here. There will be a phrase that we use in today's podcast, which is going to sound rude, but weirdly, slop squatting is not it. We'll we'll move on. We'll move on to that.

We'll move on to that. Let's go on to thinking about the supply chain of context now. And this is interesting because you know, essentially, when we look at the supply chain of dependencies, it's about us being comfortable trusting the third-party uh libraries' content code that we're pulling into our organization, to our application, our project. When we think about the same thing, but for context, um, this is more of a development time uh uh thought process of we're not shipping this necessarily, but we're using this context to guide our agent to write appropriate code.

Uh first of all, how important is it that we know who these people are, how many skills we're using in our in our organization? How important is that given it's a development time thing? I think it's hugely important. Um, and it's it's an interesting topic for me because I think um it's almost like we've got so excited with the capabilities that we have now with agents and skills that we've kind of forgotten about you know 10, 20 years of um of uh software engineering and um what we've learned about software supply chain and open source.

Um and you know, people are freely pulling in skills from random GitHub repos, um pulling in main, updating that you know, frequently without really having any oversight. Um, this is not typically how we deal with um third-party open source software anymore. You know, we're using um properly version software, you know, a lot of the time you're looking at only pulling in things of a certain age because you know you want to um protect against these kind of supply chain attacks that we're seeing more and more of these days.

Um you're interested in some kind of like provenance, uh, the where this is actually coming from. Um a lot of that has kind of gone out of the window. Uh, and this is um really something that we care a lot about a lot at TESL and that we're investing a lot of energy into building, you know, a package, uh a registry for package context in the same way that you have many registries for open source software that kind of just bring a lot of these same things. You know, it's not like hugely novel um uh you know new approaches, it's the same kind of approach to the um the new uh the new kind of uh context, the new material, I suppose.

So it's time to drop the C bomb. First time on the podcast. First time on the podcast. We've heard about the S bomb, the software bill of materials.

Do we need a C bomb, a context bill of materials so that we can identify for this project, for this you know, development scope, we know what context we are using, where we're pulling it in from, versions, etc. etc. But is that something you feel your teams could you know provide value uh from? Absolutely, yeah.

And another thing we're looking at at Tesla, and I think um it's uh it's it's an important aspect that's missing at the moment. And again, potentially more complicated than it is with um things like S Bombs, for example. You may also need to cover um context that is just running on a developer's machine globally. Um there's sort of different um different scopes, right, that you can kind of install and get something into an agent's context.

So, okay, I haven't got um anything that looks uh scary in my projects. Maybe I'm even checking that in a CI action using Tesla's um Tesla CLI. But what about what are developers installing on their machines and and how can I restrict that? Um is uh you know another layer of complexity here that I think we we haven't even started to touch the slides on.

Yeah, I guess the auditing or the gating of um context in a similar way that we would think about dependencies and and almost it starts to raise the questions of okay, at what stage do we do that in terms of do you want developers to want to be creative, want to build fast, then have a gating station a little bit later? There's a whole bunch of interesting things that could be happening. Do you have opinions on that now, or is this something that you feel we as an organization are evolving through?

I think it it depends on your use case and and sort of how uh restrictive or kind of liberal you you want to be. Um, but again, a lot of the just the the fundamental tools are missing right now. Um, so uh maybe talk a bit a little bit about Tesla CLI, which is gonna put a manifest in your project. So this is something that you can you can go and do some checks in your CI if you want.

What are the different pieces of context that we have installed? How what's their security rating? You know, do we trust them? Are they in some allow list that I might have?

Um without the manifest, you can't do any of those things. So you can't even really have the opinion about how much you want to secure um your projects. Yeah. Yeah, I actually built a little tool that I shared at demo time a while back that looks at your TESL JSON, looks at all of the uh all of the pieces of context, the the the skills that you're using.

And based on the results from Sneak via the Tesla API, we kind of like pull together this nice uh almost like a security report, a posture, security posture style report of you know, for your application, for your project, this is what this is what you know, uh this is where this is where the issues exist. And I think people are going to be asked for that more and more in terms of you know, don't just freely pick whatever you want, let's actually have a look at what we're using now.

Um what about other things like you know, whether it's secrets and credentials, for example, uh some more uh core security uh style uh concerns. Is is this also uh iss an issue for uh agentic development? Are there new things that we should be wary of when uh when thinking about how to hide those credentials? Yeah, absolutely.

I think this is um part of the more uh a more fiddly part of the equation here. Um, you know, the the agent has uh amazing access to privileged information on your machine. Um so even if you think about you know your environment variable file, a very common way to be working. Um can you and do you and how do you prevent the agent from reading that file?

Um because if it can read that file, you can assume that that is going to end up somewhere that you don't want, right? Like uh it also has the ability to post information out uh on the internet. Um, and even just the fact that that might be read in means that it's it's in logs at various places of um you know your stack and probably with the model providers as well. Um and if it can read files, why couldn't it read that file?

Um so you know, maybe you want to start looking at um different mechanisms where you have uh you know like a secret broker and you're issuing more like just-in-time credentials with like short, short-lived um TTLs. Um, so that you know, some code does get out there, you don't need to worry about it because it's only valid for 15 minutes so much. Um, but this is it, yeah, again, it's kind of um typical security hygiene, but it's uh uh at a new level of abstraction um that uh is much trickier to deal with because it's running on your machine as if it's a developer that you would trust.

Yeah, and I think that that's that's a key phrase there. It's running on the machine like a developer. Um and I think there was I was reading a Reddit thread the other day where there were people who were uh using an agent and they were they they'd essentially said, I think it was clawed code, and they said, look, you can't in it through the settings, you can't you don't have access to the .m file um as a as a file read.

And uh what Clawed Code did was it wrote a bash script that essentially read from the .m file. And it's not it's not trying to circumvent security per se, it's got a problem that it's trying to solve and it's trying to work out how, given the tools it has, it can it can perform that. And and a lot of people were replying saying, look, what you essentially want to do is run it as a particular user that doesn't have and doesn't have actual access to this file uh through the operating system access.

And and it's it's really about trying to find that mechanism, a good mechanism where it can't gain access to that. So this kind of like leans into the a principle of least privilege, which you know everyone's all familiar with from a normal coding point of view. From an agentic coding point of view, what are the types of guardrails or controls we need to put in? Place to achieve that in an agentic uh software development environment.

Yeah, I mean there's some uh good hygiene things worth looking at, you know, sandboxing your agent locally is going to reduce uh kind of blast radius of um some things that can go on um on your machine. Um and then really, you know, you're not gonna get a lot of success telling the agent not to do certain things because you know it will find a way around, as you had your example a minute ago, uh which is a great example. Um so really you need to look at like how can you sort of strangle access uh at the right level.

So maybe that's some sort of uh network egress controls, you know, um uh which domains um can you be talking to? Um capability-based permissions, so you know, which repos can you work with? What are how are you setting up your um your Git remotes? What kind of pushing can you do?

Um, what actions do you have running on branches where agents can push to? Um, and then uh thinking about things that are dangerous and putting and putting in approval gates that are not just like you know, please stop and ask the human. It's like it's physically impossible to do that without some kind of um approval that then generates a hash that then you can kind of push it through. Um but really it's an interesting um it's an interesting uh situation because um there isn't really like an escalation of privilege here, you know, like you're running an agent on a developer's machine.

Typically, that agent has the same kinds of permissions um that the developer does. Um and typically uh you're not gonna be able to tell the difference um between the developer or the developer plus the agent um hitting the various different um, you know, services that they might be using. Um because essentially it's performing actions on your behalf with your credentials. Yeah, 100%.

Um how do we avoid because like you know, we we can always add more and more security controls in security guardrails. How do we avoid securing ourselves into bankruptcy where the cost of over-securing or securing ourselves actually starts having an impact on our company's uh you know, time pressures of hitting that deadline, speed of delivery, those types of things? Yeah. It's an uncomfortable conversation because I think the first thing to accept is um that this is a nascent field.

Um we're, you know, we're still figuring things out. And if you're gonna be using agents, you kind of need to accept that uh there's gonna be perhaps some breach at some point. So, really, maybe what you need to do is be prepared and figure out what your risk tolerance is and where your risk tolerance is. So um uh I sometimes like to think about this about how reversible something is.

So um, if something is kind of like low risk and reversible, those are the kind of tasks that you might want to be giving to an agent uh autonomously, working in a branch, you know, not a huge deal. Um, pushing something up to the Git remote, probably not a huge deal. Um, do you want to give the agent control to like push directly to production? I probably wouldn't.

Um, you know, that's that's that's a lot less uh reversible than you know, working within a feature branch, for example. Yeah, yeah. And offline we were talking a little bit about how you kind of consider an agent to effectively be similar to an employee, where you almost, you know, you trust them as much as you do, maybe it's more of a junior employee, maybe it's more of a you know, someone who's a contractor or something like that. When thinking about where you draw the line on how much autonomy an agent has, is it similar to the kind of thought processes you go through if someone's just joining the team and you wanna you want them to be, you know, uh autonomous enough but equally validating what they're doing?

Yeah. Again, I think about it as like a sliding scale with the with the risk. So um uh I can maybe talk about an example that I have where um you know mostly my agent is um working directly on code and sort of pushing things um from my machine with autonomously. Um but even when it comes to say PR reviews, um if I'm getting some feedback on a PR either from you know a colleague or from um you know one of the AI PR review um systems that we use, um I want to be in the loop there.

Um, but it's also like kind of low risk. Like if the agent posts a comment that makes me look silly, that's not ideal at work, um, but I'm not like breaking production. Um so there I've used a skill where I've kind of asked the agent to like analyze the comments and come up with a plan and talk to me and wait for my approval a couple times kind of before pushing something through. Um whereas again I kind of use like the pushing to production, it it just doesn't have that ability.

So you know, I'm not using, I'm not relying on a skill to stop it from doing that. But presumably it can get so far before there needs to be some human intervention to check or to have some ability to say, yeah, I I agree with what you've done, I can now push to production, or I will allow you to push to production, or something like that. I get, I guess, what do those checkpoints look like um for irreversible uh actions? Yeah.

I mean, personally, I just haven't implemented those kinds of checkbooks, uh, those kinds of checkpoints for high danger um irreversible actions because um uh how do you trust that the agent is going to follow that instruction if if the checkpoint is um So the checkpoints are autonomous, the checkpoints are part of the autonomous system? Um yeah, so like so in my PR review example, it's like part of the skill. So I'll invoke the skill when I want to do a PR review, and in the skill I say, you know, make sure you check with me and blah, blah, blah, blah, blah.

Um, but that's kind of fairly low risk. If you wanted to have a checkpoint for something like push to production, um, actually systematically making it so the agent can't push to production until there's human um can confirmation, like how do you build that system? Yeah, yeah. Um, like uh with without just saying please ask for permission before running this scary command that actually you can always run.

Um you know, it's it's kind of about trust, like how much do you trust the agent and you probably shouldn't with those kind of things? Yeah, and and when we do think about that human oversight, uh, how good are we as humans at, I guess, you know, we know we know we've always been pretty bad at reading other people's code naturally anyway. As the amount increases, how good are we going to be as humans to actually monitor what agents are doing and actually say yes, yes, yes, this is fine, this is fine, versus actually making sure we do go into depth with the with almost like the stringent uh methodologies of code reviews and things like that?

Do we need do we actually need to lean more onto you know testing and deterministic uh code reviews as well as AI code reviews to essentially manage and maintain that yes, this code is correct because I can see all this has uh been tested and passed, or is there a lot more manual intervention that humans need to make? Yeah, I mean this is the other big thing that changes um not just in security but like in general software engineering is um the this sort of asymmetric velocity.

Agents can do like a ton of things, you can invoke a bunch of them to do a lot. Um, and we can't sort of clone ourselves in the same way. Yeah. Yeah.

Um, so how do you avoid kind of becoming the bottleneck? Um, and I think the same principles kind of apply at every point. You know, um, what do you care about? Maybe, you know, a certain point you really care that like the code style guide is being followed and PR reviews you might be more focused around that.

Um, I don't think anybody's particularly interested so much in that anymore. Uh the parts that you are interested, you're gonna make sure that's like automated and linted. Um, and really like if you if you have limited attention, where are you gonna focus it now? So um A, selecting the PRs that you're gonna have more oversight over than others, you know, something's touching the auth service probably needs some human eyes, um, some detailed human eyes.

Um, you know, if something um kind of seems like it's dealing with some new customer data or something, you might want to be paying more attention there. Um, where can and and then when you're actually doing the review, um what kind of things can you lean on to help you? Like, can you use uh another LLM as a judge on this to kind of give you um a bit of a report um that you can then look at rather than having to crawl through all the code um uh from scratch? Um and uh yeah, which things can you kind of automate away so that you can spend more time focusing on the higher risk areas?

Um uh like where can you leverage your time, really? I think is is the thing to be thinking about at all kind of points of the stack there, because we will always be the bottleneck, we're kind of just moving where in the development process that bottleneck is. And if something bad did go through interproduction, how does the uh the auditability and the logging, how is that affected by a genetic methods? Because as you kind of mentioned, you know, it there could be multiple actors here, like it could be the individual, me or you that's working with a group of agents.

There could be a number of different models, a number of different agents that we're working with. How do we actually identify where these types of issues come through when when when there's so much behind potentially a single user commit? Yeah, it's an interesting one. I think you know, all of what we've done in the past still counts.

Um, you know, the traceability of like a commit um through CD into production and then your telemetry on top of that. Um, but there's kind of a new dimension that is also becomes important now, which is like what actually happened on the developer's machine and uh you know, how did you arrive at the at the code that got committed? Um, because that's gonna help you understand how can you how can you avoid this kind of thing from happening again. So being able to cross-reference um not only the commit to the issue, but the commit to the session that was had with the agent, um, and maybe what skills were invoked and what context um was in the context window is gonna help you understand, yeah, how do we actually get to this point?

Uh, and then also, you know, looking at things that we already have, like the PR reviews and like how did how did this not get caught by your um say agentic PR review system? Um, so all of the same things that we had before are important. Um, and again, just to go back to the supply chain thing, um, understand it, like having some kind of manifest and some kind of versioning on those skills is helpful here as well, because okay, you had uh the developer's machine had this particular skill, and oh, I wonder why it didn't work.

What version was it at? Where was it installed? Um uh and I think at the moment we're not very well equipped to answer these questions. Yeah.

And it'll be interesting when we when we can and we can determine it came from this place or that place. When we think about accountability, um, and who's accountable for for the software if it's got a bug in it, um, there's this wonderful uh uh episode of uh the office, the UK office, where Gareth uh he's he's on the phone to a calculator company and he said, I I built this quote out for uh for uh one of our clients, and uh well I've eliminated all other aspects, I I I believe it's down to the calculator.

And and so you know, when something does go wrong, you know, what do we need to look at? Is it the individual who was running the agent? Is it the team that you know together are are allowing code to go into production? Is it the agent itself?

What what what typically is the the accountable piece that we need to that we need to work through? I mean, I don't know how far blaming the agent gets you. It might make you feel a bit better. You're absolutely right.

Yeah, exactly. Um, you know, I it's a tool, right? So and uh the it's there to sort of serve us. Um so I think you know, really it's the team are accountable and like the processes that got you there.

I'm sure there's no like malintent normally with these kind of things, but um, how did this kind of thing happen? How did we allow it to happen? Um, are the interesting questions there, I think. But uh yeah, I don't think you'll get very far with blaming an agent or even, you know, the frontier model um companies in in this kind of thing.

Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel.

If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favour and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. Alright, back to the episode. When we think about humans and the cultural changes that we need to make, um obviously at TESOL here we're we're we're hyper into AI and we're very AI first with the and AI native the way we are building software.

But is it healthy still to have a level of um skepticism, a healthy skepticism when using agents to build code versus being you know having an overagency to to the to models and agents and almost to having too much trust? Where's the right balance there where we're not all in on AI? Yeah, I mean it's it's so subjective, isn't it? I think um it really depends on your use case and your risk tolerance.

You know, um a big investment bank is probably gonna be quite different to a seed stage startup in kind of the things that they're optimizing for and also their risk tolerance. Um I think uh some of the things that would apply well to everybody is like thinking about what is our AI governance um, you know, strategy um and who owns it, maybe at whatever size, even if it's we don't really have one and it's the CEO owns it, um, you know, putting a little bit of thought into that.

Um but larger organizations uh that we're talking to are you know interested in all kinds of like um governance tooling here, um, you know, uh a lot of the same um uh hygiene things that you would normally have, you know, your branch protection, your SAST, um, your SCA, um, thinking about your logging and your retention of those logs, um, and uh maybe rethinking how you do secrets. I think this uh one thing that agents do change is uh your approach to secrets is may uh may have served you well for humans, and uh that might start to fall apart a little with agents.

Um but then in terms of the the actual agents that you're using, you know, which agents um are okay to use, and then it's and then it's really the the context piece, which is again what we're focusing on at TESL, like um what context is allowed to be pulled in, which which things are pushed um to developers so they can't forget about um or they can't be missing. So would you say it's almost like skepticism through you know, having real visibility and transparency to the issues that that each of these things occurs.

So like when you say, you know, what the models do have, don't have access to, it's about seeing how they trip over, seeing how they fall over and recognizing, ah, this Asian's not great at this and understanding that kind of like skepticism through transparency. Yeah, I suppose so. But again, you know, uh depending on your on your risk tolerance, you might you might want to start with a much more sort of locked down environment and see how things work and sort of slowly um add abilities.

Um, but you know, I think you're gonna get the fastest results by moving fast and breaking things. Um it's just uh maybe thinking about what things you're breaking up front. You know, if you're moving fast and breaking production, uh that might not be so good. If you're moving fast and you know, breaking your development process, that's probably interesting if you're trying to move quickly.

So let's uh let's roll up, let's uh wrap up with uh two more questions. First of all, what would you say is the most underrated security risk that you see organizations take uh adopting agentic development tools? Yeah. Again, I probably sound like a broken record at this point, but I do think it's this uh the context supply chain piece.

I think it's really interesting how you know we're just installing context into our agents in the same way that you know you might just pull something in from a repo pre NPM or uh various package manager days. Um folks are finding all kinds of skills on like developer machines and in their repos that have come from all over the place. Um maybe they don't even necessarily know where they've come from uh or or who's authored them, they haven't gone through any kind of scanning, there's no you know, companies that have their artifactory set up, but you can still just put in any skill um into any repo or into your machine.

Um so I really think it's it's this whole um uh the whole supply chain of context, how do you get that into your projects, how do you get that into your machines, um, and how can an organization have more oversight and more control over that? Um, and how you can um have more trust um in those pieces. I think it's a governance issue that people are calling out for here. This is one of the reasons why it has so much a really, a really uh doubling down on that.

And and a wrap-up question then, if you could give one piece of security advice to a CTO whose uh their company have just decided to go all in uh on AI native development, what would that be? Yeah, I mean so many detailed things, but I guess just at a high level to think of the agent as like a very competent, fluent, um but somewhat inexperienced um contractor or like inexperienced in your context, um, how would you work with those kind of folks? You know, you'd probably bring them in, you'd give them a bunch of code to write, you give them a bunch of projects to work on, you'd probably be thoughtful about which production systems you give them access to.

Like you can work in dev, but you know, we're not gonna let you um push or access prod directly. Um, you probably would make sure at least their first couple PRs are getting some proper good human review. Um, you check their references, you check their work a little bit. Um, so uh I think that I find that framing kind of helpful um to think of to think about things in a way where you know you're not doubling down so hard that you're tying shoelaces together, but um, you know, you're also being thoughtful about you know uh there there is some risk here.

So perhaps so perhaps dress up a mannequin in the office, call them Claude, uh, and um you know pretend that they're they're quite uh a junior, less experienced, maybe a contractor. And then every time you think about giving an agent something, you know, that's you know potentially damaging, potentially reversible, would you give it to that that that mannequined, that uh that more inexperienced dev in the room? Probably not. Think about that twice when giving to an agent.

Yeah, for sure. Yeah, amazing. Uh Joe, it's been a pleasure again, and uh thank you very much for uh all the time today. It's been great.

Thanks for having me. Awesome. Thanks very much for joining us on today's episode. Really hope you enjoyed it and tune into the next one.

Bye for now. The AI Native Dev is brought to you by Tesla, the package manager for Skills and Context. Your hosts are Guy Pajani and me, Simon Maple. Our producer is Tom Dowler.

The AI Native Dev is not just a podcast, it's a community. And we host monthly meetups at the TESL offices in central London. Visit Tesla.io forward slash community to learn more, and I hope to see you there.