Guiding, not ruling: Architecture Governance Approaches

Speaker A: Welcome to a new conversation about software engineering. Today it's, uh, me and Heinrich. Heinrich, how are you doing?

Speaker B: Doing great. I'm excited.

Speaker A: End of the year, we are relaxed and it's a good opportunity to talk about, uh, architecture and technology governance. I think a very important, uh, topic. We look at it, um, a governance, not as hard rules, but rather, ah, as an uh, enabler for team autonomy. Right, so we want to have team autonomy, uh, but we also want to make sure that everyone runs in the same, that every team runs in the same direction. So therefore you need some sort of guidance. We see, um, four plus how you can do governance. Most of the blocks we already have an episode which we link to, uh, block number one is architecture principles. So we did an episode five years ago with Birgitta, uh, Berkeler from thoughtworks, um, the paved road service template Golden Path. We did an episode with uh, Chris Richardson about that one. Uh, the architecture advice process. That was the last episode with Andrew Hamel Law and the technology radar. We have no episode on a technology technology radar yet. Okay, so, uh, since we had all those episodes, um, what I would do is I would give a brief overview what, um, this governance tool is all about. And yeah, then we have a brief discussion, um, about our experiences with it or we think how good or how bad it is. I think it's good to start these days with the paved road, the golden path, um, because in my opinion, uh, this one has the most potential to get significant improvement with, um, AI assistants, famous last words. But I think that's really the one which helps a lot, um, where AI helps a lot. So I think that's quite interesting.

Speaker B: Good.

Speaker A: So, brief overview. What is actually this thing, Paved road service templates? What is it and what is the motivation to have it? So basically a paved road is an orientation for us how to do things. So, um, you are in a company and you want to, uh, write a new service. Uh, you have to think about a lot of things. How do you do you need a docker file and customize or helm, you need a CI cd, uh, pipeline, you need SBOM vulnerability scanning, policy checks, logging, tracing metrics, some security stuff like static analysis, dependencies, uh, secret handling. How do you, uh, deal with stuff like timeouts or retries, documentation, um, how do you do remoting, integration with other applications? What's actually your standard folder layout? Do you package by feature? Do you do hexagonal architecture? Um, and more. So there is a lot of stuff you have to think about before you even start. You Know you haven't written one line of code which um, gives the business any, or your company any uh, value. But everyone has to think about those things. And the idea is that um, companies just provide a template which is some sort of a dummy application. There is no business logic in it, but it has already everything you need. There is all the things I mentioned. You can even deploy it. You know, it doesn't do anything except hello world. But all the heavy lifting is done.

Speaker B: Yeah.

Speaker A: What are all the benefits? Well, one thing is obviously time to market, right? So you can start very quickly. Um, it also reduces cognitive load. Um, not everyone is an expert in all those things. So it's good if you have experts who provide already working code you can use. You know, how to use customize for example or how to do secret handling things. Um, like that. Not everyone is an infrastructure expert. Um, also why we often introduce it additionally is we want to onboard new team members faster. If you say, hey, look at this service template, it's very easy to get a good idea what the whole infrastructure is about. Um, yeah, maybe one last thing. Uh, it also helps with alignment and kind of enforcing rules. Enforcing rules may be too hard because it's a template. You can always change it, but it provides excellent defaults. Usually there is always a reason to change it, but not 100%, not 50%, maybe 10 or 20%. Uh, you change it to what you need. But um, I think it has usually excellent defaults. Heinrich, you wanted to comment on it?

Speaker B: Yeah, I think uh, it's a very important vehicle. Uh, um, but the way you described it is really like it's a kitchen sink. Right. Um, in this approach. So you basically put everything you can potentially think of into this and then you strip it down for the actual application that you need to write. And, and with very featureful platforms. I feel like this is kind of getting out of hand uh, very easily where you just have kind of tons of stuff in there and then if you release a new feature or let me kind of get the change the service template. Um, and so like how do you think about this? Um, do you think in terms of archetypes that you maybe have different kind of application types where you have different templates and there was an older idea. Um, uh, I think Ruby on Rails has this CLI tool which allows you to kind of put together a web application relatively fast. So I haven't actually used it, but my understanding is it's kind of a little bit like a toolbox where you can kind of uh, add features to it like an off layer uh, or some pages to it with a CLI tool. So I was always like okay, what does a really compelling kind of golden path, um, package look like? And such a kind of toolkit to build out. Um, very opinionated company specific applications. Um, for me that was always an interesting um, maybe endgame ish thing. So if you really have a really sophisticated platform those days, it's likely a chatbot that is uh, the end game. But um, uh, like last year I would probably have said or two years ago it's like can we get close to this kind of feeling of I'm putting together larger boxes instead of kind of getting started with a large set of kind of boilerplate.

Speaker A: So I never used the Ruby on Rails. There was something similar in the Java world or in Spring World, Spring Ruby. Um, you can, There are also something like Jhipster, uh, is another tool. It helps a lot with bootstrapping applications with the general stuff. So you need a website, you need a database, uh, you need configuration, all those things. Those tools help. So a lot. Um, I think that's already a step in the right direction. But what's missing is what is very specific to your company and also all the, I would say the infrastructure stuff. So they usually have not. They have no pipeline, uh, they have no quality insurance like code scanning, security scanning. Um, they are uh, opinionated. For example you said standard folder layout, things like that.

Speaker B: Yeah.

Speaker A: So that's good.

Speaker B: Um,

Speaker A: one thing they also don't have is if you need it, a very specific thing which is only required in your system. So for example I once worked on an integration system in a bank and basically everything was done with uh, uh, messaging and we used a specific uh, framework to handle the messages and uh, the post processing of those messages. That is something Jhipster or Ruby on Rails, uh, cannot provide. They can provide something but not everything. In that specific case the company came up with an archetype with a maven archetype I believe. Um, but uh, that was at the time I was already gone. I'm not sure if it was a big success but yeah, it would be an option I think. Uh, um. So I would say those kind of archetypes are better than the quote unquote, normal um, bootstrapping because the normal bootstrapping can only do whatever is necessary in your. Ah, sorry. For every, for every application but not specific to your environment. And another question you had I believe was types of applications. Um, I was never in that situation but I know companies who are. But I never worked on the golden path for those companies. So for example they say uh, we have application services but we also have data products which uh, with a specific data transformation pipeline and therefore we also offer templating. Um, so those are the things I

Speaker B: know but pragmatically in practice this would typically be one repository. It can have a template repository um, which is deployable as an independent application and then you are specializing and kind of molding it your way. Right. This is the typical implementation.

Speaker A: Exactly. You check it out, um, and then you have already working code and then you put it, you copy it in a new repository and then you can kind of immediately deploy it and then you have a running hello World application which, which you can use.

Speaker B: Yeah.

Speaker A: And yeah, I think in if you have a microservice environment, um, uh that's usually a good idea to onboard uh new teams but also onboard new new co workers. Because if you have a big service with lots of quote unquote pollution with business logic, it's hard to understand what's the infrastructure code, what are uh our cross cutting concerns in the sense of how do we do certain things like configuration, also authentication for example authorization. Um, cross cutting concerns are also part of the templates. Um and it's easier to onboard in a new team if you can just look at simple code which shows you uh, the most important parts of technology choices and architecture.

Speaker B: No, yeah, I think it definitely has its place. Right. Um, I think of it more from an educational perspective than um, of a. Well this is kind of a load bearing stuff um, in our kind of production pipeline. Um, in the sense of it's a little bit like copy pasting code uh, which is literally involved. Right, exactly. It looks a little bit like prototype based in Inheritance where you start with an object and then start mutating. Um, so you want to really have no powerful abstractions. So the actual code you have to write is very lightweight. Maybe you have nice libraries or nice APIs that abstract uh complexity so you are kind of building more on a platform but for just. I think this is working at a large company there's so much kind of friction and knowledge required to get started and get to know the ecosystem because there are so many features and so many ways to do it and um, having kind of minimal examples for this just to get started and learn how it's done. There is a tremendous value in having simple examples readily available.

Speaker A: Yeah. So um, one thing I haven't mentioned or I briefly mentioned, the cross cutting concerns. So usually if you have cross cutting Concerns, you know, how do we do logging metrics, authentication, uh, remote calls, you name it. Things which you cannot locate in one package which are uh, in all the places across your code. But it needs to be consistent. You want certain things which cannot be located in one place. They are cross cutting. You want them consistent. What you would, what you do without such a template is you have a documentation and you explain how it should be done and why it is, you know, what were the other options. So still I think it's worth to have a documentation which says that's how we do it. And uh, this is actually the rationale behind it. We do this instead of that. Right? Uh, but here's a link. Just look at the code and it works. You can execute it locally. Everything is already there. Um, so um, that's a big advantage. Another thing which I like is, which is I think also part of this template is setting up a local development environment. You don't have all those descriptions. It's like you check this thing out and then okay, there are some automation scripts but everything is already there.

Speaker B: Right? So

Speaker A: you need to read documentation but you have additionally I would say executables which guides the documentation. I think that's a big uh, advantage.

Speaker B: Uh, can we go a little bit of a tangent here? Sorry, can we go a little bit off a tangent here, uh, about local environments. Um, because uh, just one thing I want to mention is uh, in my recent kind of coding I've been so happy with my local development environments. Um, first of all the way UV in Python makes things just very very easy to set up locally and precisely control. But for a long time I've been also a big nix user and um, this kind of dear end uh environments together with the nix flake which allows you to kind of use CD into your directory and you just have a ton of tooling which is just installed on demand with locked versions in language independent and then like sequence injected through N variables through like this N4C setup. It's grown on me. It's so convenient and you kind of are uh, relatively machine independent. You just CD into the directory to just snap. Everything is there and it's like, I don't know. For me uh, that kind of setup feels extremely smooth. I know I'm probably a little bit um, outside of mainstream with the nix usage. I'm not really sure I would recommend that for a big environment because the edge cases are relatively sharp and it doesn't always work and kind of bending in your way is a slippery slope. Um but when it works, the happy path feels great. Feels really nice and slick.

Speaker A: Yeah, yeah. Um, if. Yeah, I'm just thinking of, um, of options here, right? You say this is great for you, but in the larger environment it might be not the best thing. Um, but when I think of optionality in service templates, we had once the case, now we have it again. Just as an example, I recommended not to use test containers in my service template. I just said, use your database in a normal container inside. Every solution has pros and cons. In our template we just say, do it like this. Big problem, right? So a lot of people came, ah, you know, how can you do that? The different camps showed up and one camp said, yeah, that's how we do it anyhow. And the other camp said, we are pissed because we love test containers. And I said, you can use test containers, it's just not part of this template. But they were pissed anyhow because it was not the standard. And I think this kind of problem you have all the time. We chose to have this template in actually every case. Uh, so I had like three projects where we used service templates and it was always spring boot based. And in one environment no problem because everyone was doing spring boot, there was no other technology. But in the other two environments they were very open. Uh, one environment said, we are quite open with the technology people choose, but we start with spring. And in the other environment we had already a zoo of technologies. And then the question is, do you offer a template for all the technologies? Right? Do you offer a GO template? Do you offer a node JS template? Do you offer a closure template? Well, you can't. I uh, think you can't because at that time, that's like five years ago, I saw a presentation from Netflix and they said, oh, we only had jvm, our service template, or they call it the Paved road. Paved Road is only JVM based. And now we try to catch up with nodes and I think they also try to catch up with go. But even if a company like Netflix in 2020 cannot do it, you know, we were like a five, uh, hundred people development organization at that time. We just said, okay, we can only offer one technology. Um, but I think it's okay if you have now, I think why decide, right? So if, if there is a, if there are two options which you would think they're equally good and you have the time to, to maintain it, then, then just do it. You know, just say if, if you want to use nix here is it because we think it's the best thing, but it's not for everyone. You know, it doesn't scale across the company. Here is, here's another way. So I, it would work, but the template. Sorry, uh, I just have to finish. The template is a product, it's an internal product. You cannot. In the beginning I just wrote it and I gave it to everyone and then I disappeared. And then I saw, you know, what happened to it. You know, everyone tweaked it, people had better ideas. It was like, oh shit, you know,

Speaker B: um,

Speaker A: if you treat it as an internal product, you need to have some sort of product management. And then it's always a hard decision. What are you offering? What can you maintain?

Speaker B: Yeah, we're getting to the heart of the topic again, which is governance and that is actually closely related to support. So if you are uh, kind of following the golden path, there should be somebody supporting you, you are doing it the proper way, support. So the company either has uh, the know how to do it along those lines and has informal support structures in the way. Okay, you can ask the colleagues who have already done it, but they may also have a platform and there may be a, uh, kind of sanctioned official way to do things. And there's even a team which is uh, responsible for helping you if you are along that road. So I think for every engineer, if you have a very specific problem where a specific technology choice gives you a huge upside, you should be able to make that technology choice, but you're also paying the price for it. So I don't know if you are the only RabbitMQ user and then you have the pleasure to learn MQP and you have the pleasure to operate an Erlang technology bit, and then you may have to become an Erlang expert. But if you are willing to pay the price as a team to say, we are confident that this is the right way, we made the business case for it, go your way. But will there ever be kind of a RabbitMQ template for the company? Very unlikely. So I think golden, uh, path is for an organization, a very good way to kind of decide, like to mark what is um, a supported way. So something where you can get help and the further you are staying away from it, the more unlikely it will be that you get support. And um, for a platform, I think one of the most important bits for the platform is to draw the boundaries and clearly let the developers understand what is supported, uh, functionality, which are supported paths and which are unsupported paths and then you can make a choice.

Speaker A: So the supported Path, I think, um, you need to do that. But, but my experience is also, we, um, sometimes teams just don't know what it means to choose a technology. You know, they just say they can always explain, oh, we are doing AWS Lambda and all those things, right? And we do it with Python and blah, blah, blah, blah, blah, because we just like to do it. And then they do it, but then all the problems come, you know, then they are in production and, oh, we need observability. Yeah, well, can't do that for you because it's not the supported way, you know, or, um, you know, all those things. And then they made a decision, but it was not really an informed decision because they are too inexperienced. So I'm not sure how to deal with that. So once we said there is a chief architect and this is the bottleneck, if you want to do something else, you have to go through the bottleneck. You have to talk to this person. Problem is, of course it's not good, you have a bottleneck. But on the other side it's. If you make this decision, it's an informed decision.

Speaker B: No, um, now the, uh, chief architect is informed.

Speaker A: Well, I mean, that was not. It was not a large organization, right? So it, I mean, it was, that was. It's my banking example. Um, not. It, it wasn't. They, they, they had tiny services, maybe 70 or something, but not too many. It was not like 300 people were working there. It was really t. But, um, yeah, that was one idea. Uh, I think now we had in one of the last episodes, the architecture advice process. You could use that one, right? Uh,

Speaker B: yeah, exactly. I think it brings us to this. Um, helping teams with decision making is definitely a good idea. And I think the dynamic should be that you have defaults which are articulated and understood, and you deviate from the defaults if you have proper reasons. And what constitutes a proper reason, I think should be ultimately in the, um, decision making of a team or a developer because they are in the end on the hook for doing this. But how to enable people to do the right decisions and how to also have a leadership structure around it. Right. If you have an engineering lead that is technically deep or a principal engineer who is involved, I would expect them to kind of be uh, guides or decision makers here, which are kind of having a little bit of a larger horizon. But, um, yeah, I think it's facilitating. Enabling more than outsourcing decisions. I think if you are letting other people make these choices, um, not always sure that you'll get better outcomes Having policies and guardrails. Yeah, I mean having a policy. We are not writing C libraries. I think I can get behind that. Or like everything is on the JVM M. Or if not, you need a kind of high level business, uh, stakeholder, ah, to write you an exception. Uh, they do it this way. But having a single architect who kind of overlooks everything, that um, looks very micromanaging.

Speaker A: Yeah. I mean I want to say it depends on the company. Right. Uh, but for sure, I mean we had uh, once a situation, there were 10 teams, which is not too much I think for uh, you would think 10 teams is not too much for elite architect to make a decision, but it is. So you wait for months and it's a disaster. Right. So I think it works for smaller environments. So the banking example, I'm not sure how many. I was only there in the bootstrap phase. We uh, were maybe, you know, in the beginning we were five, six people and I think all the core team. But then uh, probably overall they had like 25, 25 people that, that works. But already 150. Then the bottleneck is uh, is not

Speaker B: a good idea, I think. I really never think it's a good pattern. I think you naturally get into this position if you have a strong technical founder that wrote the first generation of services, that understands the business end to end and he naturally is the chief architect and he understands this the best. And he's also very quick. But that kind of lasts only so long. Either you become too large or you become too old. But if you have dedicated people who are just architects and just review stuff, they get like, they lose touch. Took technology, uh, the industry and also to the code base. And it's really tough to make informed decisions if you are not inside the code so much. So you're not actually seeing what the developer sees. And even if it's kind of okay for a few years maybe or for a few months that it will detach. And having the doers and the deciders separated, that's, that's never a good dynamic. Yeah.

Speaker A: Um, I mean the thing is we have this golden path where we say that's the way how we do it. And the question is, how often does it happen that um, you really have to deviate from the golden path in a way that you have to talk to someone, uh, up the chain. Yeah, I'm not sure how often that happens.

Speaker B: All right. We talked about uh, the golden path a little bit about ADRs, and we didn't really talk about it. Right. Uh, the architecture, um, decision process or design process or architecture advice process.

Speaker A: Exactly. So I mean, that's our freshest, uh, episode. Would you like to proceed with that one or should we talk about the technology radar? Uh, first?

Speaker B: Um, I'm not sure I can adequately describe the architecture advice process off the top of my head, but I think since we touched up in it, we should kind of give a little bit of a tldr if you feel comfortable in giving that.

Speaker A: Yeah, of course. Um, so the architecture advice process, uh, is based on a book Andrew Hamel Law wrote, Facilitating Software Architecture. Highly, uh, recommended reading. Um, but most of you do probably something similar. That's also his experience. So what is this? So the question is, you want to, what we discussed, um, a couple of minutes ago, you just want the decisions made, uh, from the people who feel the decisions most. Right. So you don't want the big architects and an architecture review board where you go with your decisions and those people make a decision for you, although they have no idea about your specific context and your code. So usually this doesn't work out well. I, um, mean, it can work out in smaller organizations, as I said, but usually people are detached from the code. It just doesn't work and you have to wait forever. So what is the alternative? What can you do? So Andrew proposes, uh, the architecture advice process. So what does that mean? You have a regular, um, place. It's called the advice forum. It's a place to meet. It's an, maybe a weekly meeting, for example. And in that meeting you can seek advice from other people. So who are the people who are in that meeting? Usually, um, people who have knowledge about the decision you want to make. So if you want, as a team, you want to make a decision, you have to find out who are the people in my company who, ah, who have specific knowledge in this. For example, I'm not very good in security topics. So whenever I have to make a security decision, I try to gather,

Speaker B: uh,

Speaker A: as much as possible security experts. Uh, that's maybe a bit broad, but people who know the topic. Right. But also what you need are the affected stakeholders, usually people from within your team. But also, you know, sometimes you have to make a decision which affects also people outside your team. They should also know about it. Very important, you know, you cannot just make a decision and uh, ignore affected people outside your team. And then, then you go, you, you have all those people and I mean, you have to organize this meeting. As I said, um, it can be a weekly place where people can subscribe to and you go there and you have an artifact which is the architecture decision record. So that is the shared decision making document. So the architecture, decision maker architecture decision record. Uh, it can be a markdown file or confluence page which describes the problem. Uh, in short, you know, what's the problem? What are the options you can think of and eventually what is the decision? Also other things you can put in additionally this, you know, all the stakeholders which are affected. Uh, what are the priorities you weighed the decisions on? Sorry, the options, you know, options have pros and cons, but it depends on your, on your priorities, how to wait. Uh, those pros uh, and cons. And that's basically the document you, you, you, you, you work on, right? So the, the, you seek advice from people who have knowledge and they can bring in the expertise. They can say ah, ah, you missed this option or this option you're describing is not perfect, it's not 100% right. You miss this downside or this upside. And basically when you do that, and I can highly recommend it, we didn't do it exactly like uh, Andrew did it. But generally if you go public with your decision, it's usually a much better decision because you're forced to have this document. This document. The ADR is also better written if you go to an advice forum because there are people who have sometimes no idea on what you are working domain specific. And if they read it they can say I just don't understand what you're saying. It's only, maybe you and your team understand it or it's too long, it's too short and in the end you have really a nice document, um, uh, uh, a clear document with all the options and you can make an informed decision. Um, so uh, I think one uh, thing uh, which Andrew says it's you seek advice. You know, you can get the advice but you can ignore the advice.

Speaker B: Yeah, that's really good.

Speaker A: So uh, I think. But you say it's really good but it hurts for people who give advice, I must say because I often give advice and I think you shouldn't, you really shouldn't do it. You know, I say, you know, you just use, we have an internal service, you don't have to buy it, uh, you don't have to develop it for example. Uh, but then people do it anyhow and I think ah, why you know, not invented here syndrome or they, I know I uh, had this case once, you know, I, I knew that they wanted to use a certain technology and it just doesn't matter what comes out of the advice process, they just are interested in, in this technology because it's resume driven design kind of. Right.

Speaker B: I mean it has downsides. So the problem is, I think if you are giving directions, um, in part of this review process, um, that A, you are reintroducing this kind of architect role who maybe just the decoupled from the consequences of the decision, uh, and B, uh, you are incentivizing people to just swipe it under the rug. There's a lot of things which don't need to be necessarily documented or on the surface. They will just happen, they will be checked and they will be deployed. And it's just, then it's that way. And then half a year later people realize, oh, we are doing it this way. Sure, this is how it's been. So that is worse. Um, because there's no transparency and I think uh, having this uh, incentive to be very um, to be, to having been like having the ability to be transparent and getting as a developer positive return of investment if you are already sold on this is the technology, um, to go to. Um, I mean you at least want to know did I overlook something? Right? Is there something that I'm not aware of? And so you can get it from this, this process without feeling your autonomy being um, being taken away. So I think that that dynamic is positive. So in the end you end up with a process where you heard the arguments and you as a stakeholder, you're, your voice is also on the record and you can enable this learning process. So I think this kind of cycle, okay, I'm resume driven or I'm kind of victim of the technology hype cycle. That is something that's also related to seniority. Um, and you should kind of within a team have enough voices of reason. But ultimately, I mean, what do you want to do?

Speaker A: Yeah, yeah, true. I mean uh, it's. You said something uh, very uh, very good, which was you make at least an informed decision and you know that there is no catastrophic downside because that was exactly the case we had. There were some, some um, advice givers who were extremely negative about this technology choice and they came up with lots of reasons why this you know, brings the whole uh, the whole development organization to a standstill. Which uh, is valid. You know, you can say that, you can say it's because this and that and this and that. We write that in the architecture decision record. We believe that the following disaster will happen. Yeah, and it was quite a junior team. So I said okay, let's pick those parts and try to get an answer. Is this really true? Because if. Imagine some very important person asks you why you still made the decision, or, uh, although those people said you shouldn't and you have no, no answer, you know, they say the following disaster will happen and then the disaster happens. Then you need. You cannot just say, yeah, uh, but we just. We just didn't feel to use another technology or to, you know, to. We just failed to ignore the advice. You have to have a good answer. So we spent a lot of time finding good answers, solid answers to those questions. And I must admit, though, I wasn't very happy with, uh, the technology choice so far. Three years. We are three years in now or four years in. Almost no disaster. Some small problems, but no disaster.

Speaker B: So,

Speaker A: um, yeah, I think, especially if you have very critical decisions, it's good to go out.

Speaker B: Yeah, for sure, for sure. If you cannot really foresee all the consequences, and if this is a thing that a lot of people will integrate against or care about, then this is very critical. And the upsides of getting this right are also huge. I mean, if you have a central component that is well architected, that can mean a whole lot for speed of development down the line. Um, but yeah, I mean, it's a social, technical systems thing. Right? So it's not about getting the technical decisions right. This is not what makes successful products or companies. It's having everybody on the same page pulling into the same direction with functional processes that converge to positive outcomes. So you have to allow people to make mistakes. You have to give people enough motivation. And agencies, you have to think about a collaboration model much more than about getting the decisions. Right?

Speaker A: Um, yeah, actually at that company, when I started there, at that client, um, so I did a whole lot of stakeholder interviews. And, uh, then I would say almost every team said, yeah, you know, now they are telling us we are autonomous. They tell us this autonomy for a long time, but we are all. And we are allowed to make our own decisions. But if the decisions are not of the taste of, you know, some more important people, then they just overrule the decision. So, you know, whatever. So that's also the thing, right? So if you want autonomy, you have to swallow this frog. Uh, you have to say, okay,

Speaker B: m.

Speaker A: You could even say, um, we are not talking yet about, um, architecture principles, but it could be a principle architecture or leader principle that teams make their own decision. No matter what, no one intervenes. It's the team's decision and the team's, uh, responsibility.

Speaker B: Yeah, I think at the very large you need some form of higher governance, how you want to do integrations, how your data flows are going to look like. Certainly in higher sense, certainly. And then the art is kind of cutting out boxes which have enough autonomy for meaningful, also localizing solutions. Right. You don't want to like, you don't want to live in communism where everything kind of is kind of architected from the top. You want to federate this decision making and federate the architecting. And so that's the art. I mean I wouldn't say um, I'm super knowledgeable about it, but I think on a high level you need both, you need kind of people who understand the end to end, but you need to enable the teams to move with as much autonomy as possible. And that's critical for speed and also for motivation and to balance this. I think that's the quality of a great cto.

Speaker A: Yeah, I mean that also brings us to uh, um, uh, other uh, governance uh tools. So if you have the advice process, I mean I'm saying you make your own local decision but of course in your, let's say in the advice process. But there is of course you have guardrails usually. I mean one guardrail is you have to go public, right? So you have to go public in a sense that you have to be in this advice forum. But there are other guardrails. So one guardrail is you have the service template Golden Path paved road, however you want to call it. You know, that gives you already some orientation. Um, if you have good reasons you can deviate from it. Uh, but um, there are also uh, architecture principles. Uh, there is also technology radar, I just say, I just say technology radar, but technology choices you can do. And um, something we haven't talked, which I didn't a list uh, today but it's something which is probably very innocue specific. It's called a macro architecture. So um, uh, what is a macro architecture or uh, what are architecture principles? So they are kind of similar I would say. So an architecture principle is a high level statement that describes how you want your system to behave and grow.

Speaker B: Yeah,

Speaker A: just um, have a few examples here. Uh, from companies I work with. Uh, data is a shared asset, for example, it usually has a catchy title. Or we have autonomous cross functional teams. We want to minimize technology variation. You build it, you run it, even is a delivery principle, uh, uh, which is always to a lot of uh, um pain is like the company is the company good is higher than the team good is higher than the individual good, which is very Often of course, uh, ignored because we make locally optimized decisions. But still you have those principles. And those principles they come. You have, uh, you have company goals from those company goals you derive architecture requirements from those architecture requirements to derive principles like those. And then if you make a decision, you can always orient the decision process on those principles. Right. So, um, simple example. You know, our goal is we are bank and we want to be in all kinds of countries. Uh, we have a banking product and we want to sell it to all kinds of countries. The problem is banking is heavily regulated so uh, we don't know all the regulations. So you could have a principle which just says, um, we don't use cloud provider specific technology. Right. That's a principle. You can make a decision. You know, if you want to use Kafka, well, you can, but don't, sorry. You can use Kafka. You could even use Kafka on aws. If you go to China and there is no Kafka on aws, well then it's not too hard to have another solution. But if you decide to use SNS from AWS and then you go to China and there is no sns, then it's probably more expensive to, to change something. Just as an example. Um, so that is, I think those principles, uh, they help to make consistent decisions across teams. Um, also align choices with business strategy. You know, whenever someone asks me, why did you do this decision? I always have the traceability. I can also always say, well, this decision orient itself to. At the principal. And the principal is part of this requirement and requirement is part of the organizational goal. Right. Um, so that usually uh, that helps M. Um, and also it reduces debate because we have some sort of shared decision framework which I, which I like.

Speaker B: Yeah.

Speaker A: Um, however, I think it's very hard to come up with those principles. I was always failing and then I, because I, you know, I looked what, what are uh, other companies doing? I think Zalando also has architecture and decision delivery, uh, principles which are publicly available. Yeah, exactly. Now you, you make this, uh, maybe so you, you don't know. And, and that's, that's the problem. I, I also experience, you know, you come up with all those princip this catchy title. You have a short description, you have a longer description. You know, you. But nobody knows about it. That is.

Speaker B: Yeah, I mean I think there are many overlapping things which are like resonating with principle. Um, like on the one hand it's kind of guardrails which are really easy. Like you are not Writing in rust. Um, or you are multi cloud. Okay, well that's, that's just like a guardrail or constraint that is really strict. And then there is this kind of very soft or kind of goal oriented things uh, like deri or um, you build it, you run it maybe in some sense. But for example M, it's maybe like tenants. Um, so what do you value more? Um, maintainability or performance? And um, so one organizing principle is like we are here to sell socks. So we don't optimize for performance before we kind of we optimize for the product. And so this is for me I think a principle like how we value every software System has like 100 different optimizations, directions and to have a sorting of those directions. If you make decisions, what should be your priority? Uh, do you kind of how much speed do you want to have? How does observability or operability rank with regards to I don't know, maintainability, product velocity, um, coherence. Um, I don't know a lot of different uh, attributes that you are desirable properties of a system. Right. And so yeah, I think those work different and those are important and those are um, documented to varying degrees. Um, for example, when we built uh, our observability SDKs at Zalando, um, which is actually a set of libraries that has evolved with a larger community, we spent quite some time to specify this ranking of how important is performance when building this. Uh, like how important is these. Are we allowed to throw exceptions ever from observability code or not? Right. Um, how simple should these things be to use versus how transparently are we allowing people to use Autel native APIs? Mhm.

Speaker A: Yeah. Um, I want to come back to your. Uh, maybe we have those um, uh principles in a smaller organization. So just say we have 100 developers I would say, or even less then I think it's easy to come up with those principles because it's easy to communicate. But I'm working for example with a company, they have 3,000 people in the IT and they are digital department, however you want to call requires really a communication campaign because people know, oh yeah, those things exist. Like 20 other things exist. And you know, how should I know? Um, and why should I care? Um, yeah, I think it's hard to communicate. Um, if you look at most of those, when I look at Solando or Scout24 there are a couple of companies who make them public. They have lots of those principles of those guiding principles. And in the episode with Birgitta, um, she uh, Said, yeah, you should actually start really small so you don't have dozens. Uh, maybe have five to start with, maybe even three. And that feels something like you can do even if it hurts, that you think, uh, we need more. Probably you don't. Right, so. Because if you have 10, nobody knows the 10. But if you have three, you have a good chance that people at least know three. And those are the decision, those are the architecture, uh, guidance, which is the most important one. And what we once had was at some point, which is maybe two years, you're two years in. Uh, some architecture principles are kind of, uh, institutionalized. You know, nobody really thinks about it anymore. So if you start at, at this company, then nobody. You just cannot do anything else than principle one, two or three. You know, just give an example. You build it, you run it back. Then, you know, it's like 10 years ago it was a big thing and now, I mean, now nobody thinks about it anymore. You know, if it's like the

Speaker B: version control.

Speaker A: Yeah, it's version. Yeah, it's like version control. You don't have to mention it. Yeah. Um, now you can introduce other things. Right? Um, they had also like a mobile first or API first. Uh, um, so, yeah, it's version control. Exactly. And then you can come up with new ones. Maybe one thing, because there is this thing called, uh, uh, the Amazon Leadership Principles. I don't know if you heard about those. People usually laugh about them. You know, it's like, ah, uh, this is only bullshit. And I don't know, uh, because they are so simple, they are so obvious, but I think, yeah, um, uh, they are obvious, but on the other side they are now explicit. Right? So, um, no, everybody knows what, uh, um, Everybody knows what's important and what's not important. All right, maybe take another 10 minutes on, um, the technology radar. Um, I must admit, I'm on very thin ice. Uh, it's time to make a full episode on this one. Um, so the technology radar, uh, is, uh, or has been popularized by ThoughtWorks 15 years ago. Um, so the technology radar is a lightweight visual tool that shows which technologies, tools, practices, you name it, it's up to you. Uh, an organization is evaluating, adopting, holding or phasing out. Um, uh, ThoughtWorks, for example, they have this, they have an adopt ring. You know, if your context is appropriate, you should seriously consider this technology. It's proven and mature to use. And the other side, there is something like hold, you know, don't use it, we have it, but forget about it. A trial is something like ready to use uh, but not proven and assess is. Yeah that's interesting stuff but do uh, not try yet. Except you have a really good case and they have in their radar uh, which comes out I think four times a year or two times a year. They have four quadrants, techniques, tools, platforms, languages and frameworks. But you're really, you're free. You know I think Zalando has other uh, uh quadrants. Um, it really depends on you. How can it help with architecture governance? To be honest, I don't. No, I know theoretically I can say I also tried it but it was not very successful. I can talk about it but the idea is it should create a shared guidance without rigid standards, align teams on preferred uh and discouraged technologies for example support decision making. But it also encouraged innovation uh, while managing risks uh, and it reduces the fragmentation of technology across projects. I think that's uh, the most important uh, parts.

Speaker B: Yeah, I can talk to this a little bit. Um, I look at the ThoughtWorks Tech Radar a little bit like the Gartner Quadrant. Um, it's a opinionated kind of perspective on kind of where different technologies are in terms of hype and maturity and how seriously to take them. But I think it's just a large part of marketing here and um, just having something to continuously talk about. Um, I mean it's interesting but it's a data point like the Gartner quadrant is. But I think there's a lot of kind of market forces putting on this and um, it has a different role. Um, Zalando is um. I don't know how many other companies have implemented the TechRadar or the TechRadar process but I think it's for large organizations it's an interesting way to solve the problem of kind of technical diversity. And I mean I was having the ops eyes on but this is, I think maintainability and operations go hand in hand here. So the basic problem is if you have too many different technologies you don't have the operational know how you run into problems with scaling or with keeping things alive or just with know how of your developers. Um, and some of those technologies require deep knowledge like Apache Flink for example. Yeah, you just don't set it just up and it kind of runs. It's. You have to know your way around, have your kind of network and the uh, developer community ideally um, or postgres. Right. Um, so you have certain um, certain technologies where you have sufficient knowledge inside the company to really run them well and build on top of them well and ideally kind of Evolve the product in a way that helps you um, or continues to support your use cases. So that's the ideal case, right? And there's something you have in the middle and um, the takeaway that can be leveraged in a way that it's kind of a process that is helping you with this form of governance. And um, it can be soft but it can be also pretty hard in the sense that you are not using any technology that is not in the tech radar period. Uh, you may do a prototype or kind of a time scoped experiment with the technology that's not there. But if you want to use it you have to pitch it for adopt or to assess. Um, so you have to get it onto the radar and there's a process for this you need a ah, champion that is curating the technology. You need internal support, challenges, you need documentation. So there's kind of a bar the technology has to meet and there's a process but in this case all the principal engineers around it who are making those decisions, which kind of moves on and which kind of stuff is phased out or voted out. And um, it's a source of friction because if you think this is great and we should really doing it that way, it takes a lot of kind of politics and words to convince enough peers to get it on there. But it's also a really great way to stop a lot of discussions that are just okay here I'm resume driven or I'm hype driven, um, because you always have um, this place in it. And for Zalando what's very interesting is the public um, engineering, branding. Um, I think a lot of people know Zalando as a tech organization through the radar. I don't know, I think it's even more popular than the blog, the engineering blog. So I think more people look at the radar. If they are just assessing technologies to kind of get Zalando perspective on it, um, then um, maybe read our tech blog. Right. So um, from this perspective I think it's interesting. Um, I think it's for very large organizations and there are a lot of ways to hold it and I think I'm not really sure that anyone is right or how to exactly do it, but I think there's value there.

Speaker A: Yeah, I mean you mentioned something which uh, we did, did certainly wrong because we thoughtworks they have this thing build your own radar and there is software and you can just set up your own radar and then uh, you can make uh, pull requests uh, for if you want to introduce, if you want to Use a new technology or assess a new technology. But the problem is if you don't set a bar, you know, I think, um, uh, uh, Financial Times, for example, Sarah Wells, she says at the Financial Times, you need to write a two pager, you know, for every technology you want to page. Uh, the problem you have and how this technology will help you, what are the costs, what are the alternatives and all those things. And they say if you are not willing to write a two pager and answer a couple of uh, uh, mandatory questions, then this pull request is just, you know, it will just be ignored. Right. So, um, because, and that was our problem. We were just flooded by ideas and then you're like, I just cannot even, you know, the ideas after ideas after ideas, you know, what is this technology for? I don't know, you know, what can I do? It was actually we, we were a bit of stupid, to be honest.

Speaker B: I mean it's a learning experience, but at least it's adopted. Right? So the default is you launch something new and nothing happens. Nobody cares.

Speaker A: Exactly.

Speaker B: It's the first step.

Speaker A: Yeah, you end up and you see, oh, there are uh, Clojure services and only two people can program in Clojure.

Speaker B: Now let me say, I think AI is changing a lot of the dynamics here. Um, for example, like shell scripting, I'm doing a lot of shell script in Ruby these days. I cannot do Ruby, I've never learned Ruby, but I feel like it's looking a lot nicer than Shell and then Python as well. I just like the aesthetics of it. I like reading it a lot. And also for the first time I understood the lineage of the name. Right. So you have Shell, then you have Perl as kind of a mix between Bash and awkward, uh, from the lineage. And then Ruby is kind of object oriented Perl with nice syntax and it's like, okay, it was made for this. And so you wipe code, um, stuff in Ruby, I don't know, I think for shell scripts it's largely fine. Um, but I think language wise, the dynamics of how much knowledge do I need, how costly is it to switch to a different language? This is shifting fast. So I also think the kind of truth about governance in the sense of this is how strict I need to be in an organization of that size to be fast and effective. Um, this is how much kind of many technologies I can use. This is changing quickly and also for a developer. And how many technologies do I need to be deep? Um, I feel a lot freer to use a lot more different languages um, because I don't no longer kind of have to type them up and need to know the libraries. I need to know kind of what the high level library ecosystem looks like, what the engine can do, the runtime can do. Well, um, how packageable it is. Like Python is so nice for Shell script, but then you cannot deploy it anywhere because nobody has figured out how to deploy Python. Um, it never works. And so it's like, then we write go, uh, and then the deployment is fine. And usually it's like, no, it's not going to happen because I cannot write go good enough to do that in a productive way. Well, if it doesn't just needs to be okay and it's not so large. Well, just do it and go. Um,

Speaker A: yeah, true. I really wonder how that, uh, how that unfolds. I mean, for me it's the same. I'm less. I do not do so much production codes these days, actually none. But trying something new is. So the bar is now very low. Um, it's great actually. Um, but still, I must say, uh, I mentioned closure Clojure or Lisp. They are probably, uh, you know, there are languages which are easier to adopt. I would say the problem is, for

Speaker B: me at least, the reading of Lisp is extremely unnatural. Um, and I have a blog post, that infamous blog post was hated on Hacker News, but it was called the Problem with Lisp. And I was arguing that you never know if it's a function call or a macro, which is the core problem, because as a mathematician I kind of know how to parse a formula inside out. And it looks like this is the same thing in Lisp, but it's not because you have all the special forms which kind of. You have to know that this one changes the evaluation order or just a transformation on the things. So you have to kind of parse outside in and inside out at the same time. And then it's also crazy functional, uh, land. So there are so many quirks that you kind of have to kind of get through in order to be able to read it. While it's actually from the interpreter perspective, it's almost exactly Python. There's nothing special about it. There are so many, uh, things that are called, uh, uh. It looks very functional, but you can set variables, you can have side effects. This is like it's all perfectly possible in Clojure and all the list variants, most of them. Um, so you can write exactly the same code in Python and Lisp. It's just like the syntax is so horrible, um, or so different. And, uh, I think it's kind of a pity. Um, maybe. I don't know if there is a really better way. Um, but I, uh, think for a lot of use cases, specifically shell scripting little tasks would actually be a good language. But, yeah, not going to happen.

Speaker A: Yeah. Reminds me, Rich Hickey, most successful episodes case podcast episodes with, I don't know, 20,000 downloads or something. All right.

Speaker B: On which topic?

Speaker A: I think it was problem solving.

Speaker B: Uh, okay.

Speaker A: It was. Yeah, it was, um, I forgot the English terms, but he said a few very nice things on how to solve problems. Right. Uh, you know, lying on the couch, have enough time for deep thinking.

Speaker B: Yeah. Yeah, that helps. Going for a walk.

Speaker A: Exactly. I think Hancock driven development. That's how he called it. Is it? All right. Yeah. Thank you so much. Um, actually, I want to split it in two episodes now it's one maybe. Let's see what we can do. But, uh, we have one. We have all the four. I forgot about the macro architecture, but yeah, next time. Next time. Exactly. All right, so, uh, thank you for everyone who made it so far. And, uh, yeah, Happy New Year.

Speaker B: We are recording this 19th of December. I don't know when it will be released, but I hope you had a good New Year's.

Speaker A: Exactly. Cheers.