The Trouble with Certificates with Charlie Al-Batty (Episode 123)

Speaker A: Hey, it's Steven here. Just a quick note. Before the episode today we had some technical difficulties in the recording of this episode, so there is no video. Uh, I'll be uploading audio only to YouTube, uh, and everything else will be audio only as well. Sorry for the inconvenience and I hope you enjoy the episode. Welcome to Slight Reliability, the show about human beings and their experiences in SRE observability and technology leadership. I'm your host, Steven Townsend. Welcome back to Site Reliability. I'm Steven Townsend and this is the show about SRE observability and technology leadership. Today I'm joined by Charlie Al Batti. He's a cloud operations Engineer with over 10 years experience architecting resilient cloud infrastructure and driving organizational transformation toward cloud native organization automation, first operations. He serves as a technical leader, a DevOps advocate, cloud champion and security custodian across enterprise infrastructure. How are you going today, Charlie?

Speaker B: Hi Stephen.

Speaker C: I'm good, how are you?

Speaker A: Yeah, I'm pretty good. Uh, it's a Friday at ah, 2pm in the afternoon so uh, it doesn't really get much better in terms of timing. Uh, so today we were going to talk about this challenge that I've been having and that you have had solutions for in the past and I thought it was pretty cool. We have a lot of applications and services, lots of B2B and many of these services to communicate with each other securely. They have certificates and one of the challenges is making sure that certificates are renewed and applied uh, effectively in that context. Uh, and uh, we were talking about this a couple of weeks ago, maybe two or three weeks ago. My first question really is though, uh, have you in your career seen challenges managing certificates and to the point that they might cause incidents?

Speaker B: Uh, yeah.

Speaker C: So uh, throughout my whole career I've had the pleasure of working with outages from certificate management and having to uh, manually create certificates and then deploy them across different environments and having to manage different uh, expiry dates and going through the manual toil, uh, that comes with that. In my current role I uh, did work extensively on um, automating the whole certificate process end to end. So from creating the certificate to storing the certificate to then utilizing the deployed certificate. And a cool thing about Microsoft Azure, which is the platform that uh, I work on, is that Azure has uh, native integration for offerings into uh, Azure Key Vault. Once the certificate ah, rotates you don't have to have manual intervention there.

Speaker A: Right. Okay. So are you like purely Azure for all of your um, infrastructure and applications? So is it all in the one place.

Speaker C: Yeah. So I'm Azure native, that's the current platform that I work in and that's the platform that I've built expertise in through study and upskilling.

Speaker A: I think my situation might be a little more complex because we've got like CA certificates that are managed through multiple platforms including GoDaddy. Uh, we've got applications which are sort of in data centers, we've got a lot of aws, we've got some Azure and there's all kinds of different situations where sometimes the like in the cloud, cloud aws where a lot of the certs are quite hands off they just automatically renew and apply and there's no problem at all uh, especially with the infrastructure. But some of the more off the shelf applications uh, where you've got these certs and they actually you have to manually apply them and stuff like that for me that's like a scary territory for uh, missing things for ah, incidents to occur. Um, and I'm not expecting you to have any answer but do you have any advice for someone working in that context how they could keep on top of not just the renewal of certificates and making sure we don't miss any, but also the application of those and applying them to the right places at the right time and uh, safely.

Speaker C: Yeah. So on that um, the one tool that I find very helpful is uh, Ansible M awx. So Ansible can be used uh to if you're manually going and uploading the certificate to our uh, service. So say you receive the certificate from GoDaddy and then you go and try and upload it to your service.

Speaker B: One thing that I found really helpful

Speaker C: is understanding the process of what you're doing and then translating that into an automation template using Ansible that doesn't just apply to certificates, it applies to a whole range of things. In my day to day my thought process is usually I don't want to do something more than once manually. So understanding how I did it the first time and then how using tools I can automate that, uh, you want

Speaker A: to understand the manual process first before you start trying to automate something. I think that might be a step that a lot of people maybe don't pay enough attention to. You know what I mean? Like actually understanding this is what I'm trying to achieve and here are all the steps to get from step A to step Z. Is that something that you have to slow yourself down for to sort of really. You're almost value stream mapping in a small context.

Speaker B: So I do find Like a lot of people don't take the time to do that. They would think that, oh, uh, if we've done something a certain way, we'll just continue doing it that way. And another thing is in quite a few managed services organizations that I've worked in, they've been very keen on documenting process. So I find people like to follow process and do things manually. But having that mindset shift where you think about what am I doing, do I really need to be doing this and can I be doing it more efficiently? Has really been helpful.

Speaker A: And does having good documentation of a process help you automate it though?

Speaker B: Not necessarily so I found everyone has a way of doing something differently. So as long as you understand the core process of what you're doing once you're automating it, Ah, one thing that I found helpful is adding notes like treating your ah, automation script as documentation.

Speaker A: M. Let's say in my context, my team and I, the first thing we're doing right now is that we are actually just making sure that we have a clear central record of every certificate in the organization and we know when it expires and we know where it's managed and we know where it's applied. So that's, I think that's the first step, uh, um, before you can really do anything else. Then you're like, okay, how can I make the automation of this work? Even then though, like really successful organizations still experience outages because of certificates, right?

Speaker C: Yeah.

Speaker A: Why is it so hard? Why are certificates such a hard thing to get? Right.

Speaker B: I, uh, honestly think it's because it's so finicky. It's like you have a bucket with 50 different objects in it and you need to know where each object goes. And managing that manually is, is just a nightmare. Like I think it would add maybe weeks of operational toil to the team in outages and manual work. So yeah, I think it's because it's so many different objects connected to so many different systems. So having the database of where everything links to is good.

Speaker A: Okay, two part Christian, how do you give others confidence about that? Because if it's just invisibly in the background doing all this stuff, how do you give other stakeholders confidence that this stuff is renewing and applying safely? Have you got any tips for that?

Speaker B: Okay. Also in regards to how to validate the automation that uh, you're building, one thing that, that I found good is to have a mindset of logging. So if something goes wrong at uh, any section of your pipeline, understanding what has gone Wrong pulling the error message and sending it to uh, team. So either via like a webhook into Slack or teams or an email.

Speaker A: Interesting. And now I'm probably going to go a little bit off topic here. So this is one of the things I've been also challenged with is that having alerts that go to, in our case teams, chats or channels, it's very easy for the team to miss them. Whereas so we've been rethinking alerts uh, in terms of we kind of want them to go to our uh, ticketing, uh, system so that there's an actual alert ticket created and it gets so that, and if it's serious enough that the on call engineer actually gets called for that. Is it like with Slack channels if you've got a lot of alerts, do you have to like train yourself and your team to continually check how does it work for you?

Speaker B: Okay, uh, so I personally also use Microsoft Teams and we face the same struggle where if uh, alert triggers someone has to manually go and check that and if they don't check it then it can cause an outage. One thing that I am exploring is looking into using, we use uh, Azure DevOps for ticketing. So one thing that I'm exploring is using the Azure DevOps API to create trigger a runbook that will create a DevOps backlog item with the alert name and alert details. So that way we would have a log of what's triggered and what we did to resolve that.

Speaker A: That's cool. That's kind of what we're doing um, with JIRA Service Management. Uh, JIRA Service Management has this built in concept of an alert. So you can actually create this alert object which you can then still handle and notify people if required. Uh, um, and it has different priorities that you can assign to it but it means that you're not necessarily constantly creating incident tickets for things which aren't actually incidents yet. This is like. Do you think there's a difference between something which needs a human to look at, it needs attention versus an incident? And that's what my thinking's like at the moment.

Speaker B: So from like an uh, from an SRE perspective, do you guys uh, have like a def. Because I was listening to site reliability one of your first. I think it was one of your recent episodes where you mentioned that just because an alert has triggered the business value to that alert may not be significant. So do you guys have a way of managing the legitimacy of an alert that's come through in an automated way?

Speaker A: Probably not in A sophisticated way other than when we configure alerts, we decide at the time that we configure that alert, uh, how impactful that alert probably is. So there are certain events that occur or a thing fails to happen, we know that it is a major incident and we need to immediately call the on call engineer because it requires attention straight away. But no sort of smarts in the systems necessarily that are looking and going actually I'm going to assign this level of severity or anything. Do you have some options there, some tools there?

Speaker B: Uh, not really, but one insight that I recently gained, uh, I attended an AI conference recently and they were talking about uh, AI integrated operations. So that got me thinking to. I haven't explored it yet, but I might explore it in the future, uh, using like your monitoring and telemetry data and sending that through to an AI agent via API and getting it to determine the actual impact of the alert based on the application that it's uh, referencing to. So that was something that I thought was really cool, but I haven't personally had the opportunity to explore that yet.

Speaker A: Wow. So I'm just trying to play that out of my head how that might look. So let's say if you're continually sending telemetry to an AI agent, which is essentially telling the AI agent this is what normal operations looks like. And then if there is an alert, you can then cross reference has the telemetry changed in a way and then it might be able to pick out, oh, uh, yeah, it looks like 100% of your customers are uh, currently impacted. Um, maybe even this is losing you five, fifty grand a minute or so. I don't know. Is that the kind of thing you're thinking about?

Speaker B: No. So I was thinking more so like around uh, understanding what uh, component is currently broken, um, what, what functionality of the application is it currently impacting. And um, yeah, that was more my thought process. What you're saying makes, makes sense. Like, but I think a lot of organizations nowadays are quite hesitant to add monetary values when it comes to AI. I don't know if you've seen that as well. Um, but that's been a kind of a trend that I've noticed is uh, maybe some people uh, may not want to get AI to say like a dollar saved figure because that may not always be accurate.

Speaker A: At the first when you said that, I was thinking, um, we don't want to train AI to understand how profitable we are. But you're saying, I mean AI is notoriously bad at providing accurate mathematical figures. Uh, is that what you're talking about?

Speaker B: Um, maybe, yeah. Uh, and I think it's just also maybe a lack of confidence. I don't think that we're at that stage.

Speaker A: One of the things that I really like about the work that you do is in true SRE fashion. And honestly I've met, I've had a lot of guests on the show. I've been dabbling in SRE for years and now I'm finally getting in there. I don't meet many people who build as much software as you do to solve problems and I think that's absolutely awesome. And uh, one of the things that we talked about last time is that you've recently built some open source tooling for doing service level objectives in Azure. So can you tell us a little bit more about that? Why did you choose to build it in the first place and how does it work?

Speaker B: Yeah, sure. I built a tool called Azure SLO Guardian. And uh, the reason why I built the tool was to address a gap in uh, Azure where I don't think Azure has a native SLO function. So the open source tool essentially connects to Azure Monitor and it runs SLO and SLI objectives against Azure Azure native PaaS applications. So I saw that there was a gap in current tooling. The reason why I built it is so that organizations that don't currently have an SRE capability can have that easy uh, open door into implementing something because something's better than nothing I think.

Speaker A: And this, this tool that you've built is open source and it's available to people if they wanted to use it right now.

Speaker B: Yeah. So it's currently open source and hosted on GitHub.

Speaker A: That's cool.

Speaker B: I'll be sharing the URL with you

Speaker A: for those listening in. I'll put a link to uh, that in the description. But that's cool. But Leah, like I said, I love the way that you write code to solve problems. I'd love to do more of that but something I am curious about is how valuable is uh, using AI tools to help you write code? Because I know a lot of people who end up in more the operations space um, aren't as confident as a seasoned developer to write software. But uh, do you use um, LLMs to help you write your code? Are you finding it helpful?

Speaker B: Yeah. So uh, when it comes to software, I'm not a software developer. Ah. I just wanted to say that um, I do build a lot of automation pipelines in Powershell and also uh, ansible module pipelines. So that's more so where like my automation comes in. But in regards to uh, LLMs, I think that they're really really powerful if people use them in a way where they're reducing operational toil. So as an example if you have one uh, thousand records in a spreadsheet and you need to organize them into another spreadsheet in a certain way, previously you'd have to manually go and do that, whereas now with LLMs you're basically saving hours worth of work. So I think that it's going to enable people to become super developers I guess like people who are scared of it, I think they should be using it and I think that we shouldn't slow down because it enables someone to save I'd say 10 or 20 hours per month. If they're using it effectively, one person could output double the work that they're currently doing.

Speaker A: I like that. Do you have a particular, I mean I'm uh, not trying to promote anyone over the other. Do you have any particular LLMs or tools that uh, you find work for you and your use cases?

Speaker B: Yeah. So I really love uh, Claude. So I use Claude Opus. Um, it's really efficient and helpful.

Speaker A: That's the one uh, a lot of our developers have been using as well for uh, quite interesting use cases and I'd love to get myself and my team more involved in it. I'm like, I'm terrible. I mean I haven't been very hands on engineering for a little while but I'm about to, I'm slowly heading back in that direction. But there's nothing the same as this. But I have to at the moment, at this stage of my role, I have to define a lot of um, guidelines, uh, approaches, strategies, visions, uh, that kind of thing. And I do have to say in the past I would probably sit in front of an empty document and sit there for four or five hours just like figuring out how I want to structure it. Just being able to go to M even Microsoft copilot and say hey, here's a bunch of raw notes I've taken, here's what I want to create. Can you give me some ideas of how I might structure this? That probably saves me two or three hours for each of those things that I write. And even uh, that's huge, right?

Speaker B: Yeah, exactly. I see the same productivity gains as well.

Speaker A: It's not like you have to get AI to write everything for you, but just having someone to rub a duck with to give you some feedback instantly about very complex things that another uh, person, it might take them Just an hour to digest what you've written.

Speaker C: That's quite powerful.

Speaker A: Yeah, so, I mean we're coming, um, close to time, but, uh, I just wanted to ask, like, what are you, Are you working on any particularly interesting challenges at the moment or things that you're excited to try out?

Speaker B: Currently, I'm using Ansible as a native tool set and I'm working on a way to build state management into Ansible. Uh, that's been one of the biggest challenges that I faced, uh, with the Ansible tool is having state management, uh, because it's not built in, uh, like Terraform has state management built in.

Speaker A: What do you mean by state management? Sorry for my uninterrupted views.

Speaker B: So when I say state management, I'm talking about what's in the cloud and what's reflected in your code base. So when you're using infrastructure as code, you would write a template, uh, manually and then as people get introduced into your cloud environment, they may manually click buttons and change settings. And then if you want to delete that environment and reprovision it, uh, it could be missing settings or variables. So state management is basically maintaining the state of the current environment based on what you have deployed in your cloud.

Speaker A: That sounds amazing. Sorry, I'm obviously completely new with this, so is it able to then understand? Because this is a problem we have as well. Everywhere has it, right? You've got infrastructure as code and you deploy a thing and then someone changes something manually and they feel like you've got a console somewhere and then it's no longer in sync. So using state management, would it update the config to reflect the state, so

Speaker B: it updates your ah, IAC templates to reflect what's currently deployed so that you have peace of mind before you go and delete your environment and then come to reprovision it.

Speaker A: Sorry, you just blew my mind. That is a. I had no idea that was possible and you just made me very, very happy. And now I've got something to go back to m my developers and my team and say we should be doing this because we're using Terraform, right. So I should be able to do it. Is that right?

Speaker B: I. Yeah, because Terraform has state management inbuilt into it, so it has a state file which uh, pulls your current environment. But Ansible is missing that functionality.

Speaker A: Uh, amazing. So you're building that out. Is that a challenge?

Speaker B: Yeah, it's quite a challenge because it's a lot of load, but it's not like a proprietary system. Uh, it's just PowerShell scripts. So yeah.

Speaker A: Something else I wanted to talk about is the idea of certifications in technology, um, the value that they have or don't have. And I was just curious, what are your thoughts on certifications? Like ah, a cloud certification of this or a uh, certification for a framework. What do you think of them?

Speaker B: Okay, uh, so my thoughts on certifications are uh, uh, they're not the be all or end all. They don't make a great uh, engineer. Just because an engineer has a thousand certifications doesn't mean that they're going to be a unicorn of their job. But one thing that I do like about certifications is I enjoy the process of learning because as I go on the learning journey I pick up knowledge that I can then translate into how can I improve my current environment. As an example, if you complete a uh, security certificate, uh, say like an Azure security certificate, and say you had no prior knowledge of Azure security offerings that could enable you to then go to your current organization and implement features that you wouldn't have known about beforehand. So yeah, I think certifications are good for people who want to learn, but I don't think that they validate who you are as an engineer and your capabilities. What are your thoughts?

Speaker A: I think the last time that I was really hands on doing stuff every day, um, I was working mostly in kubernetes, did a lot of Grafana stack and we wrote a lot of PowerShell code and there's nothing there. Which I thought a sort of certification really would help me in. And I've always thought that certificate I mean uh, I agree. I like what you said because a lot of the time like especially cloud, cloud, public cloud vendors, they release a lot of new stuff and unless you're actively going and seeking out the new things that they have implemented, you might miss out on things which will make your life easier or save you your organization money or more secure whatever it is. Uh, that's exactly right. There's value there for sure. But the other, but in a lot of other areas you're going to learn the vast majority of your skills on the job, solving the really messy problems in the real world which you'll never see in any training ever because they uh, by the nature of training have to be clean and contained and something everyone can understand.

Speaker B: Yeah, that's so true. Being in the trenches and getting your hands dirty is the number one badge of good engineering.

Speaker A: So this has been a fantastic conversation. Charlie, thank you so much for uh, reaching uh, out and coming on the show and sharing your experiences. And I admire that. What you're doing in terms of building software to solve problems and automating things and just being so unapologetically make your own life easier by applying engineering principles. And I think, uh, a lot of people who have the title SRE maybe don't do it enough. And there's so much more that we could be doing, myself included. So.

Speaker C: Yeah.

Speaker A: Thank you for the show.

Speaker B: Thank you. Thank you so much, Stephen. I really appreciate it, and it's been

Speaker A: great speaking to you and to everyone else. I hope you have a wonderful week and I'll see you next time. Enjoying slow reliability rate review and follow. Wherever you listen to pods, it helps more than you think.