DOP 362: Feature Flags vs Canary Deployments

Speaker A: So it's more of a configuration management problem in that context. And that doesn't really give you what I was referring to before as far as runtime control and being able to flip a flag in production in seconds. If flipping a flag still requires a full CI CD pipeline and a full redeployment, is it really a feature flag in our opinionated view as an unleash? We don't think that's the way to use feature flags because you can. That's more like an environment variable, right, that you can configure. The application reads it at startup time and that's it.

Speaker B: This is DevOps Paradox, episode number 362, feature flags versus canary deployments. Welcome to DevOps Paradox. This is a podcast about random stuff in which we, Darren and Victor pretend we know what we're talking about most of the time. We mask our ignorance by putting the word DevOps everywhere we can and mix it with random buzzwords like kubernetes, serverless, cicd, Team, productivity, Islands of Happiness, and other fancy expressions that make us sound like we know what we're doing. Occasionally we invite guests who do know something, but we do not do that often since they might make us look incompetent. The truth is out there and there is no way we are going to find it. Yes, it's Darren reading this text and feeling embarrassed that Victor made me do m it. Here are your hosts, Darren Pope and Victor Farsek. There's a new word floating around. We've got new words floating around all the time. This one is feature ops. At least they didn't try to make it devsec, uh, thin feature ops, because that would have just been more silly. Is Feature ops really a real discipline or not? Or is it just a marketing wrapper around yet another thing? Today we're going to be talking with Alex Casaboni from Unleash. Alex, how you doing?

Speaker A: Hey, guys. Glad to be here and thanks for inviting and yeah, let's talk about feature ops today.

Speaker B: Let's talk about feature ops because there is a site for feature ops and just we'll go ahead and tell you up front because it's featureOps IO feels very agile manifesto to me. Is that sort of what you were going for is like, Here are the 95 theses of how features should be used.

Speaker A: Pretty much that's based on what we see with many of our customers, but also many of the experience of, um, the founders of Unleash and many other people that work at Unleashed today. And the main point is to be super vendor neutral, super referenceable. Because it's a problem that basically every company has, whether you are in a financial industry or a big retailer or whatever you're doing. Managing feature management and release management is a big problem. And handling that when you have a thousand engineers across 25 teams at scale, and especially when if you're down for an hour, it's millions of dollars and reputational damage and all those issues, you know, how do you handle this process at ah, scale that that's what the manifesto is for really.

Speaker C: I feel that we need to backtrack for a moment what this feature of. Because I was, before you started speaking, I was imagining my PRDs that I passed to my agent. This is the feature I want. I'm assuming that's not where this is going.

Speaker A: Yeah, let's do a step back. Let's take a step back. I think the main, first of all, why Ops? Because there are some operations involved and most of the other discipline like DevOps and all the cousins pretty much help you until the deployment. You can do all the things in the local environments, you can do all your CI cd, you can do all the static check and all the GitOps magic to make sure the environment is aligned to your configuration, blah, blah, blah. But basically a lot of the, you know, 90% of the approaches you ship to production and then um, if something happens in the production environment, you need to go through another cycle of that. You need to go through a hotfix, you need to go through another CI CD pipeline running for 20, 30, 60 minutes. It depends. And so what uh, the future Ops methodology is trying to address is can you apply the same principles but after the deployment is done, when the application is running in production? And the better term to understand what I mean is runtime control. Can you change application behavior at runtime in production to reduce the blast radius of an incident? Things like a kill switch, things like something that uh, you can enable or disable or change the behavior in seconds instead of waiting for a patch from the development team and then a full CI CD pipeline running for half an hour and then you get the fix in production. That's like the main reasoning behind uh, Core.

Speaker B: So what I really lacked when I was setting this up is we're talking about feature flags, but from a little bit different angle. What we're used to with feature flags, and we've talked about feature flags on the show many times, is okay, back in the old days it was kill switch, right? Or turn something on, turn something off. But then it became more important. We also have to do maintenance of those flags because flags eventually go stale, become useless. And that's really, to me, sort of the part of it is the governance and compliance part of this is how do we deal with turning those flags on. It's like it's easy to turn a flag on. You click a box, you run a cli, you do something flags on. But how do you make sure that you have the 75 different approvals to turn the flag on?

Speaker A: That's a very good question.

Speaker B: And that's hard, right? That's the hard part.

Speaker A: It is hard. It's hard. I think if you look at this problem space, you can look at it from three different angles. One is the technical angle. Like, yeah, how do you do the thing, how do you integrate the tooling into their product, how you do, how do you make sure it's up and running and you have great availability and uh, all those technical things. The second angle is the people. How do you coordinate people? How do you handle role based permissions? How do you give access to the right people, to the right environments and applications and projects and whatever you have? And the third angle is compliance and regulations and how do you make sure that this all lands well in your risk management processes and all that fun stuff that enterprises need. So we can look at it from these three angles and talk about what's actually hard in these three spaces. I think what you were referring to mainly is the second, like how do you coordinate, how do we have processing around cleaning up after a release is actually completed? And how do you coordinate all of these effort in a, uh, maybe in a centralized place? Maybe you're a platform team and you can enable all the development teams to use this thing as a centralized service instead of everybody reinventing the wheel. There are many conversations across these three angles and which one you think is more relevant or more challenging.

Speaker B: They're all challenging and we need to solve all three within the next 45 minutes. So how do we solve that?

Speaker A: First of all, there are different ways to do feature flagging. I've talked to many people that only think of flags as something that lives in their configuration, which means it's maybe read statically when the application starts up and then it doesn't really change ever. So it's more of a, um, configuration management problem in that context. And that doesn't really give you what I was referring to before as far as runtime control and being able to flip a flag in production in seconds. If flipping a flag still requires a full CI CD pipeline and a full redeployment. Is it really a feature flag in our opinionated view as an uh, unleash? We don't think that's the way to use feature flags because you can, that's more like an environment variable, right, that you can configure. The application reads it at startup time and that's it. But it doesn't give you the runtime control that a proper future management solution should give you. And then, um, some people actually do it at build time. You know, I was reading some articles from the companies of the decade, like Anthropic and some of the big ones that are now pushing the boundaries. And for example, they do it uh, at build time. They read a flag if the feature is not enabled. They don't even include the code in the CLI bundle or stuff like that. So that approach to feature flagging is build time or startup time. It's not about runtime control. And there are reasons for that. Right? You want to reduce the bundle size, you want to not release unfinished code into a uh, CLI that people might reverse engineer and blah blah, blah. That makes a lot of sense. But what we see when we work with companies every day is that uh, enterprise companies, especially every day, is that they need a mechanism to toggle stuff in production in seconds. Because it's not probably going to surprise you, but many companies have like uh, a 12 to 24 hour round trip between when they discover a problem and when the fix can go to production during the full release cycle. Even for a hot, for a hot fix. So it's a big problem.

Speaker C: If you're talking about seconds to feature something, we are really assuming that you're talking about total duration, not seconds after I click a button.

Speaker A: Yes, seconds between when you click the button and when the behavior in production changes. That's what I mean.

Speaker C: But that could be. Then that could mean actually basically hours or days.

Speaker A: No, no, it's the seconds. Like if you're using features in a total duration. What do you mean?

Speaker C: Okay, it's been a day since we deployed this. A day, uh, later we discovered it doesn't work. We click a button and two seconds later it's fixed. That's one day and three seconds duration.

Speaker A: But usually, uh, you measure between when you find the problem and when you can release a solution to the problem. Right. And so if you find the problem 24 hours later, some companies will have a fix in production for the problem another 24 hours later. So it's two days. So I think yeah. Does it make sense?

Speaker C: So kind of basically you're not dropping 50%. That's quite something, right?

Speaker A: For some companies it's six hours, for some other companies it's one hour because you're smaller and leaner and you're more like a uh, small medium business or a startup and you can push a hot fix in minutes or like less than an hour between when you find and identify the problem and when you can push the fix to production.

Speaker C: Where I'm um, going with this question is basically if you measure the time for real, not from the point that is convenient after clicking a button, then the real bottleneck is how do we detect an issue and fix that issue right away. I'm talking metrics, I'm talking traces, I'm talking logs, and so on and so forth. So I'm assuming that feature toggles of some form or another, once we've solved the button problem, there must be solving that problem in a similar way that canary deployments are solving it, but through in case of canary deployments that would be rolling back the whole binary. Right? And in case of feature flags you're rolling back a feature and then you can actually talk about seconds and not days plus seconds.

Speaker A: I think most companies measure mean time to remediation or mean time to whatever is again the mean time from when you find the problem, not from when you deployed the problem the last time. But that's another problem. Like how do you find the problems as early as possible? That's not a problem that feature flags could or could not help you. But to your point, yes, there is an advanced, let's call it more advanced way to use feature flags in the sense that they can act more as a automated mechanism, similar to like an automated kill switch or a circuit breaker where you're not just waiting for a human to toggle the flag, but the feature flag system itself knows what's going on and we call these impact metrics. So basically what you can do with the unleash SDK is ah, that not only you use it to check whether a feature is on and off, but you also use the SDK to send back metrics to the system so that if an error rate crosses a threshold or some other CPU whatever crosses a, uh, defined threshold, you can automatically switch off that feature without a human intervention. Because you're right, if you look at this from the second angle we were talking about earlier, usually the bottleneck is not the clicking a button, the bottleneck is okay, I figured out there is a problem, but what can I actually do to remediate it as quickly as possible. If it takes a full cycle, it might take hours. But still, if it takes 25 million approval checks and a lot of internal bureaucracy and all those things, there is still a process or bureaucracy or human bottleneck between when you know what to do and when you can actually do it. So that's exactly where we are investing a lot on automating this concept too and integrating the business metrics or the technical metrics into the system to do the work for you in seconds.

Speaker C: Again, but if the problem is not technical, then the solution is not technical, right? You said that we have a bunch of approvals and what's not right, those are not technical problems.

Speaker A: It's uh, often a compliance problem because a lot of large companies need to track everything because flipping a flag in production is a change in production and a lot of regulations require you to have audit logs and to know who decided what and when and what exactly changed at what time. So what we see with our customers that maybe they have like an um, in house feature flagging system that doesn't really provide that auditability because everybody can go to the database and run a query and flip a flag in production. And that's a big compliance black hole because how do you prove what happened, who did the query, at what time, what exactly changed and when? And so that's one of the enterprise features that we definitely provide at Unleash, where you have an unmutable event log that serves you as an auditing mechanism, which includes if things happen automatically or if you have a uh, sophisticated change request process where you need for ICE principle and you need at least one or two more people to approve the change in specific environments like production. So absolutely yes. It's not only a technical problem. That's what we were talking about at the beginning where technically it's everything is possible and the SDKs and some of the features help you there. It's also a pupil problem because sometimes it's not an engineer that wants to flip a flag, sometimes it's a product manager, sometimes it's a release manager in the bigger companies. And it's also a compliance problem because you have your ticketing System, you're using ServiceNow or whatnot and you need to make sure that everything makes it there as well. And you know, there are ways to automate that part as well.

Speaker C: It's a bit confusing because do companies assume that all toggles that they have are safe, uh, to toggle or they make assumption that actually it's not safe to toggle some toggles because I can only assume it's latter, right? Because if they would say it's those toggles are safe to change, they wouldn't be asking for approvals. So basically you're saying that they treat every toggle as essentially a new release, not from the duration perspective.

Speaker A: And that's the point of feature ops. The main advantage you get from this approach is that you can finally decouple the act of deploying the application from the act of releasing a feature to your users. You know you can deploy your code anytime, multiple times a day, automated with your continuous delivery, continuous deployment. But that doesn't mean you are releasing the features when that happens. That's a business decision that should happen when the business and the customers are ready to actually start using the feature. To your point, yes, that's the major difference. And it's not only when, it's also who decides. As an engineer, you have written the code, you have written the functionality, you have written the whatever. But it's often not you as the engineer who decides, okay, we are launching this next Tuesday at five in the uh, morning or six in the afternoon.

Speaker C: And in that case you don't need approvals except the person who is a single person, right? Because it's not anymore we have approvals because we don't know whether it's safe. But now we're saying this is safe by definition.

Speaker A: Let's talk about what a feature flag can do, because there are many ways to use it, right? If it's like a release flag where like you are deploying a new feature and you have to decide when to enable that new feature, that's something. But it could be a kill switch that disables an API call or disables a, ah, third party integration. Or it could be a permission toggle that decides who can access what because of their subscription tier or something like that. It could be an AB testing experiment where the feature value is not just true or false, but the feature value is test A, test B or test C. And there are different things you're showing or different things you're doing. Like you're a B testing an AI model or you're a B testing uh, visuals functionality or the copy on your homepage. So it really depends is it safe? What is the tog building act going to do in production? And that's part of your feature ops discipline, like who is doing what, who is responsible for what, and also what's the outcome of toggling that flag? That's super important. And you can have internal limitation. The feature flagging platform, you can can somehow help you with that because for example Unleash, um has different feature flag types that you can keep track of and you have descriptions and you have all the things that help you understand what happens when you toggle that flag. But to your point, a lot of organizations just need approval 100% of the time or some kind of change request mechanism because that's a change in production and any change in production needs to be tracked and need to be auditable and needs to have the four eyes principle applied. So no single human can say this is changing in production. Now, because I decided somebody needs to review it and approve it doesn't mean that every flag is safe or unsafe to toggle. That really depends.

Speaker B: I'm going to stick with technical for just a minute. You said something a few moments ago. Immutable event log. I have to call BS on that. There is no such thing as immutable data.

Speaker A: Fair. I'm not going into the blockchain world because I'm not an expert there, but. But that's not what we're using. So I'm with you there. The idea is data, uh, that nobody has explicit access to like in an easy direct way. Whether it's like a lockdown table somewhere or some object storage where the application and the keys you are delivering only have read sorry, only had write access to in append only mode, whatever implementation. That's a technical detail but. Or whether it's a ticketing system somewhere. That's what most compliance frameworks require. That there is an audible ticket or slack conversation or something in ServiceNow that shows who requested what, who was the approver and when the change actually happened. Can it be changed or tempered, uh, after. Yes. And that's why there are a lot of products that try to help you in avoiding that. But yeah, that's something that we provided unleash. I'm lacking the internals details about how that is implemented, but I'm happy to share some details later.

Speaker B: No, that's fine. It's because I hear immutable just spouted by people and it's like now, hang on a second. To an extent. Blockchain could also be become mutable as well, but that would take a lot harder work versus what you were just saying. Oh, something goes into a bucket. Well, I can change the keys and get access to the bucket. So that's not immutable. We were talking about uh, lots of other acronyms too Today in the US, we've got SOX and FedRamp, but in Europe y' all have something called Dora. No, not that Dora, but the other Dora, the digital operations Resilience Act. Is that causing more problems than help, as most regulations do?

Speaker A: I'm not an advisor here. I know that DORA mostly applies to financial institutions. If I recall correctly, I remember it does apply to resiliency and uh, making sure your application, your infrastructure, your dependencies are resilient, blah, blah, blah. I don't have enough knowledge of the details here. I don't know if it's causing more trouble than actually helping companies, but it's not the only regulation. And it's all there is gdpr, there are many more, there's nist, there is more. And I think the important thing to remember if you are a technical guy like me or like us, trying to navigate this space, is that what's on you is usually the technical implementation and the tracking and making sure that the thing is resilient. I'll give you an example. A lot of feature flagging, whether systems, whether they are in house built, a lot of companies just built in house because sometimes makes sense or like an external service, they rely on external API calls for the actual evaluation of the feature flag. Now that has two main issues. One is obvious, one is performance. Because if your server side application or your front end has a dependency on an external API call, and maybe you need to make three, four, five or ten of them to render a page or to serve an API request, that means additional latency. And that's a, uh, problem because a round trip back and forth to any API will almost never take less than 10 or 20 milliseconds. M and if you do it 10 times, that's a lot of additional latency for just checking a toggle. But the other problems that maybe some engineers wouldn't think about immediately is the data privacy problem. Because most feature flags, yes, they give you a, uh, true, false or some kind of, uh, variation if it's an A B, testing as an output. But we almost never talk about what's the input of the evaluation. The input is not just the application name, usually the input is the user context. Because a flag can be true for a user in France and off for a user in the United States, or a flag can be on for a user that has a premium subscription. The targeting rules of your feature flagging system will usually have some targeting logic that depends on the user attributes or some kind of context. That's the keyword here. And so what's included in that context and where that context needs to be in order for the feature to be evaluated, that affects your privacy and your compliance. Right? If you need to send all the user context, including possibly PII and personal data and all those sort of sensitive information in general, that means you are sending the data out of your perimeter, out of your infrastructure. And that's a big problem. So that's why most mature, robust, resilient feature flagging solutions have some sort of local evaluation mechanism where first of all you don't need to do an API call every time you check a flag. You get some configuration at startup time and then you keep it fresh every few seconds or minutes and the evaluation happens with some local process, local logic, and with the same mechanism, the data stays where the application is running. You don't need to send the context out. So that's why one of the examples for if you're building it in house, for example, that can get really tricky because maybe you start simple. You don't want to add a lot of complexity to your SDK or to your local evaluation logic, but that means you're adding latency and you're sending your data, uh, out somewhere. Maybe it's not a problem at the beginning because everything's running in the US and in your data center or on AWS or somewhere, but then you scale out and you add additional regions and maybe you don't also scale out the service. So the data needs to go back to the US if the app is running in Europe and then that creates problems. Right? So yeah, that is one of the issues we help our customer with. From the technical and compliance angle, I'm

Speaker B: thinking about one of the things you said there. You brought up config and I'm going to sort of twist it a little bit. To me, it becomes a smell when a feature flag basically becomes a permanent configuration switch. Think pricing tiers, customer entitlements, things that were potentially A B or a B, C, D, E, F things, and you've picked one. Is that really a problem? Am I just overthinking it? Or is that just the next evolution of what flags were meant to be?

Speaker A: I think it's real and there is a lot of evidence around this and also previous incidents that prove that if you have too many stale flags over time, the um, technical depth really adds up and compounds because it adds more overhead to every code review. It adds more complexity to adding new features. If there are too many, if a, uh, switch or different brands to even think about in your brain, that's something that uh, in my opinion every feature flagging solution needs to help you with. Which means for I'll give you an example when you are on leash and you go through the, you know, setting up the flag, integrating it with your, the code and then switching on in production at some point, there is a uh, time in the future where you can call this release or this experiment concluded and Unleash gives you a wonderful dashboard that shows you like your instance technical debt and so it helps you. Okay, we have seen this feature flag in production for a few weeks and, and then it stopped being evaluated or it's always evaluated as true 100% of the time. So it's probably ready to be cleaned up. And we have also run a few POCs with some customers where you know, when you click Mark as completed on Unleash, you spin up an automated process that will go and clean up your code automatically because you have the feature flag name, you know, what's the outcome of that experiment or that release. For example, we have an SCP server that will help you automate the cleanup process based on the specific language, the specific framework. And so what we set up with this customer is basically a GitHub action kind of workflow that will automatically create a PR for you using the Unleash MCP server to clean uh, up your code of a released feature flag. I think that's one of the ways you can tackle the technical debt problem where you don't have to wait 10 years. I was talking to a customer a couple months ago and they told me, yeah, we want to migrate from our in house feature flagging solution to something and the oldest feature flag we have is from like 2012. Yeah, that shouldn't happen and we want to help you with that.

Speaker C: Do we have this? You just mentioned MCP M, right? Agents, what's it not? Do we have the same problem now with which flags cleanups as we have with reviews that okay, now whatever you had before now, it's multiplied by infinity right? Now for every pull request you got, you had before now you have 50. Is it the same thing with feature flex? Multiply by 50.

Speaker A: It depends how you think about it and how quick you are with cleaning up. What we suggest is to have a very intentional internal guidelines about what requires a feature flag, what domains, what subfolders in your app. To put it simply require a feature flag whatever is changing in that folder, for example. But not everything requires a feature flag. Like if your default is every change requires a feature flag, it definitely becomes a problem for example, what our NCP server provides is a bunch of tools that are not just API calls, it's not just an API wrapper, it's domain specific guidance that helps the agent do something that it wasn't trained to do. Like most LLMs were not trained to do feature flagging or some of this stuff. So what the SAP gives the agent is opinionated guidance about what to do. And you can of course customize the logic. There is for example a evaluate risk tool. It doesn't do any API call, but it does take a look at your code base. It does take a look at uh, the, the change that's current in your git history and whatever you're doing in the code. And it can give you an evaluation of a risk level of the change that's currently being developed. And that's one of the things that you need as a developer to understand, to speed up and to even automate the process. That's something we see because to your point, a lot of people now are using LLMs to write their code. And so unless you integrate all of these feature apps methodology they were talking about into their cloud or whatever best practice that they are using, it just becomes an afterthought and that's also a problem. But yeah, review fatigue is real. I don't want to deny it, I don't think feature flags are specifically solving that problem. But if you can automate the building the feature flagged feature automatically and if you can automate the cleaning up afterwards, you are already halfway there and you don't end up with 10 years of technical debt to clean up.

Speaker C: Does any of the uh, users you work with do automated cleanup? Kind of like I trust it if it's true for a year, Just don't ask me questions.

Speaker A: The short answer is yes. Longer answer is if you asked me two, three years ago people had static analysis tools, people who do uh, you know, regex matching, trying to find the things to clean up in the code and apply static rules. Now I think in my opinion it has become easier because an um, AI coding assistant already understands your code base enough to be able to understand what to remove, how to clean up, how to clean up the imports and how to clean up some tests that are not anymore, you know, not just the if logic. So based on my personal experiments and the uh, POCs and the things we are running with our customers, that is now a lot easier. And so you don't really have an excuse anymore for having 10 year old or 5 year old or even 10 months old flag in your code base because it's just making the life of your developers harder without no real reason.

Speaker B: If you had to guess a number, what percentage do you think in a customer's system? Pick an app, pick a whole system of the flags that are there are actually stale, 50, 60, 70%. Is that higher than we want to admit?

Speaker A: That's a great question. Thank you. I think I did see a statistic recently from a, uh, research paper. If I remember the statistic correctly, I'll correct myself maybe in the notes somewhere if it's wrong. But the statistic I remember is that there is an order of magnitude difference between how many flags a company creates every year and how many flags they clean up every year. So I don't know the percentage, but if you create 10 flag and only clean up one over time, that, yeah, that creates a lot of troubles. And if I had to be honest, that's what I. The main pushback that people have about feature flags. Oh, it complicates the code. Everything complicates your code. The best way to not complicate your code is to not code. But, uh, if you need it, we want to help you make sure it doesn't complicate your code in the long term. So let's collaborate there.

Speaker B: I'm going to go back to using AI and the MCP server and stuff to rip things out safely again. I don't trust it. Excuse me? I use AI on a daily basis. If I could have an hourly basis, almost a minutely basis. Is that a word? I trust. The AI I don't trust, especially knowing flags and having worked with flags since the early days, I don't trust it because I, uh, know the Cartesian of flags. It's like, okay, we're going to take this one flag out. Okay, that one flag down a happy path. Okay, that will be easy. There's a simple kill switch. Get rid of it. Because that kill switch was for a system that we no longer integrate with. Great, that one's easy. The problem is when we get back to the. Okay, this is one flag going back to your user context, uh, AI context, but actually, okay, what amount of data do we need in order to decide if this flag needs to be on or off? I don't know how we deal with that behavioral side of things. It just doesn't seem safe to me.

Speaker A: To me, I would describe that as discipline. If it's about tribal knowledge, like, yeah, that team, that engineer knows only that team and that engineer knows what this feature is doing in Production, you are counting on avoiding human error. While if you try to encode those rules and that knowledge into playbooks and now AI agent documentation and prompts and things that can help automate some of this stuff as well, I think that becomes a bit easier because you're not counting on the 12 year old flag that's still there. It's probably because somebody left and nobody wants to touch it. Right? That is a typical problem of companies that have a lot of legacy stuff around. And so how do you solve it is usually a knowledge sharing issue or a documenting things issue. I don't think AI is less trustable in that sense. I think the potential of solving the problem is more is higher than the risk of doing it with AI instead of doing it manually. But this gets religious really quickly and political really quickly. So I completely understand if you don't trust doing some of this stuff with AI. Ah, and you shouldn't trust, you should verify anyways. That's another argument, right? Don't just trust it.

Speaker C: Is there a collision between feature flags and canary deployments in terms that do we need both?

Speaker A: That's a good question that I talk about this with some of my colleagues almost every day. So thank you for asking. So first of all, there are situations where you need both. But if you use canary deployment for enabling or disabling things like similar logic to a feature flag, I think you can avoid it. Especially if you know, doing a canary often means or like a canary or blue green or other techniques that are similar. If you often requires that you have different clusters or different environments running and you can switch them easily at uh, DNS level or some other technique. And so if you can do that canary or blue green kind of logic directly in the application, it means you can do everything in one environment directly in production you need to do a lot of switching. But also it's about the complexity of what you can do if you do a blue green or a canary. Usually it's about one one thing and then when you do the switch, you're switching one thing. If you want to have three A B tests running and 10 different flags or behaviors that you want to change, that's really become too complicated to do with a canary or with a blue green deployment. So my shorter answer, sorry for digesting my shorter answer is if it's about infrastuff that doesn't require using user context, that doesn't require, you know, application level knowledge, but it's just something that you're switching like uh, you're changing Database, you're switching to another API vendor, you're switching to another host name or something. That still makes a lot of sense to do things at the infrastructure level with your canaries and your blue green deployments. But if you were using those techniques for application level flagging or behavioral changes or feature flagging, I think feature flags are a much better tool so you can use them at the same time. The reality is that different teams at different layers of the stack will probably use different techniques.

Speaker C: Does that mean that everything I'm, um, coding needs to take into the account all the permutations of the current code? So let's say that I'm building a new feature and there are already 10 features that are under Feature Flex. That's, I don't know, ten hundred permutations already that my new code, new. My new code essentially needs to have hundreds of picture flags in itself.

Speaker A: That's the theory. Like combinatorially, it gets really complicated really fast. The reality is that most teams try to test. First of all, if we assume, and that's a good, reasonable assumption in my opinion, that a feature flag shouldn't stay there for 12 years, and that if you're doing a release that will take a few weeks, not a few months or a few years, which means once something is released and you decide that the feature flag is on for everybody after 3, 4, 6, 12 months, you can clean that up and you really have no good reason for switching that off after the new functionality is available to everybody and not just a subsegment. So I wanted to start there because if you assume everything can be toggled on or off at all times. Yeah, that's a problematic assumption. I think a more reasonable assumption is some feature flags only go from off to on and then stay on forever or until you clean them up. That's like the typical workflow of a feature flag. Some others, like a kill switch, they will stay off most of their life, and you only switch them on for an incident or for a specific situation where you need to kill that integration, you need to kill that code execution path. So those are the two extremes, and then there are some shades in between. But so the reality, long story short, is no, you don't need to check all the possible combinations of all the feature flags, but there are some that are more likely to happen. So what you want to start from is what's the current state in production? So, so maybe you have 10 flags. Seven have been on for the last three months. One is a kill switch. That's Always been off, never been turned on unless you actually need it. And a couple of those are ongoing, uh, a B testing experiments that are in like 50% rollout or whatever. And so yeah, you can test the major from the production state. What happens if you toggle on and off any of those and then try to test the major branches, not all the branches combinatorially. So I don't give you, I won't give you a number, but if you have 10 flags, you can probably test five or six cases, not a hundred. Reasonably. But yeah. There are also ways to be a bit more strict and with the numbers and with the math is the number

Speaker C: of seven, ten theory or practice?

Speaker A: I just made an example because the practice is you don't have 10 flags. The practice is customers usually have hundreds. And some of those are, you know, land in different categories and have different history. It's also true that most flags explicitly never interact with each other, meaning they do not cover the same code path. And so if there is a situation where a flag depends on another flag, there are also ways to, for example, do hierarchies or you know, parent child relationships. That's something you can do in Unleash. So for example, you want to make sure that a flag can only be turned on and off if another flag is on. That flag is parent. So you can handle those dependencies explicitly in the tool that you are using without counting on again, documentation or tribal knowledge or humans remembering what are the dependencies. In the general case, though, I think it's fair to say that it's a best practice to avoid feature flagging interacting with each other. I do remember some white papers saying that even if you do it, there are still chances that some feature flags will interact with each other. And so you need to remember it, you need to document it, or you need feature flagging platforms that allow you to track and link and create your relationships between flags so the platform is aware of it and warns you if you try to do something wrong.

Speaker B: We've had flags and experiments. We had flags longer than experiments. But the whole feature flag as a marketing thing, uh, 15, 20 years, it's now 20, 26. What's new in feature flags in general? Just still feels like it's the same thing over and over again.

Speaker A: What's new in feature flag? That's, that's an interesting question. I think one thing that the thing that we talked about at the beginning where that's what we call autonomous feature management, like the idea that you can feed actual production data into the system so that the system knows if something's wrong and can do the work for you of enabling a kill switch and doing automatic circuit breakers or automatic pausing error load if the current cohort is experiencing high error rates and things like that. Because yes to Victor's point half an hour ago, if the bottleneck are the humans realizing there is a problem and figuring out which flag needs to be toggled off. And if that takes hours, you're still hours have issues in production. So can the feature flagging system help you automate that? So you can call it automated or autonomous? We like autonomous because it's a bit more AI adjacent as you can imagine. But the idea is, can you build a smarter system that helps you remove the human or the bureaucracy or the procedures bottleneck from how long it takes to remediate an incident in production?

Speaker B: What's one thing or a couple things that you wish people would stop doing with feature flags?

Speaker A: Ah, uh, that's a good question. It might be a repetition because we talked about this a lot in the last hour or so, but treating feature flags as something that you can forget after you ship the code, that's the main challenge that our largest customers face, where you have a thousand engineers and they all now push tens of PRs per day as you do, and then they forget that there are now a billion feature flags in the system that needs to be cleaned up. So what should I stop doing or what should I start doing is stop forgetting about it, stop treating it as uh, somebody else's problem in the future and start including the cleanup process into your definition of done. So that in a couple of weeks either you automate it or you remember because there is a ticket about it and you go and clean up whatever needs to be cleaned up. And we can help you with that with reminders and dashboards and automated, uh, workflows that start when you mark something as completed. But uh, it's still part of your internal culture and processes to just not forget about it and not think, uh, that somebody in the future will take care of it. It's still your responsibility as a developer or as a product manager or as a release manager to take care of the cleanup. So please do and please let us help you tackle that.

Speaker B: I think the next feature that you do need to build within your product is a self destructing feature flag. Sort of like mission impossible. You know, you listen to the message and it's message is over and now it's gone.

Speaker A: That's great marketing by the way. Thank you. You try to use it?

Speaker B: Yeah. That'll be a $5,000 recurring fee every time you use it. So we don't want people to forget that they're there. That's the problem. Couldn't we argue that if there's such a pain, and now we have billions, your word a moment ago of flags in the system. Why do we even bother doing flags at all?

Speaker A: That's a good pushback, I think. I. I have this pushback many times in the last three months, and usually it's from someone that is telling me, okay, but I've been trying to shift left everything in the last 10, 15 years. You know, I've been trying to solve the problem as early as possible and to avoid bugs, making it all the way to production with tests and static analysis and, uh, QA and all the things you can do before production. And that's a very valid argument. I think shifting left is something you should do. And so the question they usually ask me is, why should I shift right and just handle this stuff directly in the production environment? Isn't that going to create a culture where you don't really try to shift left because you can always fix it directly in production? And I think that's a false dichotomy. That's a false. You know, it doesn't necessarily mean you either do the shift left or you have a proper runtime control in production. I think you need both, because one is a safety mechanism to make sure you have the smallest number of issues in production, or while the other is, uh, emergency mechanism that makes sure that when the bug or there's something wrong makes it to production, you have a quick way to fix it instead of waiting hours and hours, if not days. So you can think of it as shifting left is your prevention mechanism and shifting right is your, I don't know, emergency box where what do I do when there is a problem in production? You still need both. You kind of one or the other. It's not either or. It's an end, in my opinion.

Speaker B: You can find out more about Unleash, um, at Get Unleash IO. As a reminder, FeatureOps can be found at FeatureOps IO and all of Alex's information will be down in the episode description. Alex, thanks for being on with us today.

Speaker A: Thank you, guys. It's been, uh, a pleasure.

Speaker B: We hope this episode was helpful to you. If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are@devopsparadox.com contact if you subscribe through Apple Podcasts, be sure to leave us a review there that helps other people discover this podcast. Go sign up right now@ah, devopsparadox.com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.