How DATEV Scaled API Governance Across 4,500 Engineers

Ikenna: Hi everyone. Welcome to the API Governance for Skill podcast. My name is Ikenna Ikenna-Wiwu. I am Principal Consultant at Ikenna Consulting where I give engineering leaders ⁓ evidence-based independent assessments of API governance and delivery risks.

Today I have Andreas with from DatF. How are you Andreas? Andreas Reetz: ⁓ I'm fine. Ikenna: Good, good, good, good.

It's been good meeting you, Andreas, and just hearing about all the great work you do at DatF. In our talks before this, we mentioned how very few people, maybe outside Germany, may have heard of DatF. So today, I'd like to talk about how you've been able to scale API governance at DatF and just maybe learn some lessons from the great work you guys are doing there. So maybe, first of all, just start introduce let our audience hear a bit more about yourself you know tell us a bit more about yourself and maybe how you started your career yeah Andreas Reetz: Yeah, ⁓ okay, my name is Andreas Reetz.

⁓ in the area of API management ⁓ since 12 or 15 years. Something ⁓ before I've done ⁓ a APIs, I have done a lot of stuff in the soap area ⁓ and my Q and job role, I'm ⁓ the architect and lead for the API management and governance at DATAF. from my education I'm a mechanist. Before I studied computer science I really worked on hardware machines and so on.

and so on and so ⁓ in business since 94 or something like that. ⁓ Ikenna: Wow. Really interesting, really interesting. And tell us a bit about DatF.

It's this huge company that many people don't know about. Tell us a bit about that. Andreas Reetz: Yeah, is also not very known in ⁓ Most the people know DATEV from ⁓ their checks because there is a stamp from on ⁓ it. ⁓ ⁓ make software for ⁓ text accountants.

And we are also owned by tax accountants. So we are a corporative and we have over 90 % of the market in this area in Germany. So all the stuff which has to do with tax and payrolls and so on, it's mainly done in Germany by DATF. ⁓ So we are owned by 40,000 members roughly.

number changed ⁓ but ⁓ the number. And we are looking on who are using our software, we are somewhere in the millions. Ikenna: Interesting. Andreas Reetz: So really a lot of people are using us and we have 9000 employees and from that we have roughly 4 or 5 thousand software engineers.

So that is quite big. Ikenna: Wow. So that's a huge scale of engineers. say 45,000 engineers that you, that software engineers that build software at F.

Sorry, go on. Andreas Reetz: Nick. We're 4500 not 40,000 yeah Ikenna: 4,500. Yeah, yeah, I know, yeah.

So 4,500 engineers at DatF. What does API governance look like at that scale on a day-to-day? What do you typically do in your role as you kind of head the API management, API governance function, API platform team? What does day-to-day involve for you?

Yeah. Andreas Reetz: ⁓ Mainly we do empower the teams ⁓ themselves ⁓ use our environments and to build up good APIs. We ⁓ tools ⁓ and so to pick up ⁓ the developers and the requirements engineers so on. where they stand.

So we try to introduce or integrate tools into the ⁓ of our customers. And our customers are ⁓ mainly the developers ⁓ develop APIs and use APIs. That's our main focus on that side. So there ⁓ have several educational lessons which do frequently can be booked ⁓ by ⁓ our and also we a Ikenna: Hmm.

Andreas Reetz: consulting package and so on and so on. On that side you can't look on every, on each part you have to delegate to the teams. Ikenna: Yeah, yeah. this is strikes me ⁓ what struck me when I first spoke to you, Andreas, is like you guys seem to have a really mature API platform, really mature API governance set up in you approach it, which frankly, lots of ⁓ organizations I to, don't have the level of maturity ⁓ that you ⁓ at ⁓ So us a bit about Andreas Reetz: Yeah.

Yeah. Ikenna: how you got there. I'm sure it wasn't all like that in the beginning. Probably had some point where there was lots of maybe chaos perhaps, but along the line you've put in all these, you you have a platform, have a trade, have, you know, trade, you mentioned trainings on how to use the platform, which many organizations don't have.

And you mentioned providing almost like a consulting aspect and seeing the engineers as customers. Again, not, all organizations have this in place. So tell us about a bit about that journey of how it first looked and how you were able to, what were the key things you did in scaling governance to that, to where you are. Andreas Reetz: Okay, we started with API management at DATAV at 2020.

In the very first beginning, we built up the infrastructure and the automation at the same time. And that's quite important because most of the API management solutions which we find on the market has no automatization at all especially no automatization which fit into ⁓ your environments in the journey of developers. And that was ⁓ one of the very point which we really delivered and we started with ⁓ only one big application ⁓ and then scale it through more and more. we in our API management over 300 teams.

The goal ⁓ is every make developing teams as independent as we could by our own. And that's really a goal which tried to do at the start. So, yeah. Ikenna: Yeah.

See, I just love what you said there, Andreas, around making feature teams, the development teams as independent as they can. So enabling them, giving them all the, you know, tooling, support everything to them independent. But that was the philosophy, you know, and that's really a platform team kind of philosophy, you know, to support and make them independent as they can with the right, you know, support, guardrails, controls, everything. ⁓ And I have talked to teams where it's not always like that.

for some teams they're like no no no you know we can't let you guys manage yourselves we really need to control this bit or we really need to make sure ⁓ and that ⁓ I'm there must be a lot of as well as the enabling and providing all the tooling also a lot of trust and support that you're providing the teams to make that happen at that scale. ⁓ Andreas Reetz: Yeah. Yeah, yeah, and that's really sometimes also a problem because the ⁓ skills in development teams ⁓ are equal. ⁓ If are going from team to team, ⁓ you find ⁓ different sets ⁓ and sometimes you also see that ⁓ our they have a We have done it ⁓ every ⁓ on way and you really have to change their ⁓ so ⁓ they that the API is not something technical, it's really a product ⁓ that's quite important our platform and so on.

But that's quite challenging. Ikenna: Mm. Andreas Reetz: ⁓ Every week I got one or two talks through teams where try to transport this of behavior. But it's not so easy because the development teams always have pressure to deliver.

and that's something which comes on plus. Ikenna: Yeah. Yeah, yeah, absolutely. And you know, that whole idea of APIs as products, APIs as you know, not just a technical interface, but actual the interface to a service and making sure that it's seen that way.

It's not just like a, you know, necessarily just the data. Andreas Reetz: Yeah. Ikenna: transfer thing is so important Just to talk about this now, ⁓ was something as well when I was speaking to you before this, you talked about how you offer different of governance, right? Obviously with that F, you have external APIs, you have ⁓ your APIs, partner APIs and things like that.

Yeah, how do you ⁓ differentiate governance? Andreas Reetz: Yeah. Ikenna: Do you govern them all the same way? Do you apply the same standards and processes to all these APIs?

do you have different tiers ⁓ governance for these APIs? Andreas Reetz: Okay, we have different tiers, costs on... It's ⁓ quite... Many APIs the teams provided are just ⁓ provided for ⁓ And I think, okay, we give them the information ⁓ which standards they have to exceed.

And there are also some of the rules which also were a must for them. ⁓ But ⁓ makes, my point, hard for them to understand ⁓ that an API ⁓ go through the and ⁓ sometimes they are not alone anymore. But sometimes to give the teams more freedom to do their things, we have less rules for the teams which are. ⁓ having which provide the APIs not ⁓ for themselves.

On second tier we have APIs which provided through other teams in DATAV ⁓ and there it's important that the quality exceeds some standards. There we have the costs of using the API over several teams. And so in order to reduce the cost and the time ⁓ ⁓ delivery, time market, ⁓ it's important that they also use this common standards. really get ⁓ on using APIs.

And the tier is the APIs which we provide to third parties. And there we really ⁓ manual reviews ⁓ where our team on the APIs and look on what's good and what not and make a deeper consulting ⁓ that. Of that's a phase to outside and it can damage the brand by themselves. that's mainly our focus on that.

Ikenna: Mm-hmm. Yeah, yeah that makes sense. makes sense. And just move on to talk about like AI, how AI supports know, previously we've talked about this and how as you have different Andreas Reetz: Yeah.

Ikenna: tiered, know, different APIs, different categories of APIs and the different tiers of governance that you apply to them. told me you've also looked at using AI in some areas to assist some your governance practices. ⁓ Can tell the listeners a bit more about that, your experience with ⁓ AI, that worked and where that hasn't worked? Andreas Reetz: Yeah.

Okay, currently we rolled out ⁓ review assistant. That helps us mainly for the B2B API. And if the teams are using this tool, we get review mainly in one done. They ⁓ fulfill the standards, we will find some things and we just do on the review some cherry picking.

So ⁓ ⁓ names of ⁓ resources change it because it's not so and so on. But if a team do not use that, In the past we normally have 4 to 5 runs. ⁓ ⁓ the point that ⁓ the to market is better ⁓ because are faster. an appointment ⁓ we can ⁓ those talk together.

There are days between because we are quite good booked and so that's much better if they do it by them own and that really helps. And the newest thing where try to do is to help the developers with the thing which comes from linting and so to give them advices how they ⁓ could solve the points the linting will find. And that's also a point where we see that will improve the speed of the developers. Ikenna: Okay, that's cool.

So AI picking up from the issues that Linta found and helping the dev resolve those issues. Yeah, that's good point. Andreas Reetz: Yeah, yeah, yeah. And we also try to do is ⁓ ⁓ teams a tool where they can design an API ⁓ help of an AI, but ⁓ in the very, very beginning.

⁓ a... ⁓ Ikenna: Yeah, yeah. just to summarize, know, is looking at these three areas. One is AI as an AI design review assistant.

So an AI assistant for design reviews. The other side is AI for resolving issues picked up by a linter, an automated linter does a static check on API contract description file, finds issues, and then the AI assist the developer in fixing those issues. And then the third part is AI from the design side of designing the Andreas Reetz: Yeah. Yeah.

Ikenna: API following your guidelines. Now, ⁓ you you mentioned an interesting stat there that way, a statistic that where you're API for like, sorry, AI for design reviews, go from like, I think you said some like four to five meetings to just one meeting because it, you know, it, it, it, yeah, it just, ⁓ and a huge improvement, isn't it? To go from four to five meetings to one. And tell us a bit about cost, right?

Because in terms of, ⁓ Andreas Reetz: Alright. Ikenna: When we talk about AI, we can talk about AI model usage, we cannot not talk about the cost in terms of tokens. What's been your experience with the cost of using AI in these scenarios you've mentioned? Andreas Reetz: Yeah.

Okay, you see, if we just prove everything which we have in our catalogs, for example, in production with AI, that will cost something about 500 to 600 euros. And so you must... Ikenna: Did you just say 500 to 600 euros only or thousands? Is that thousands or just...

Andreas Reetz: Yeah, 100, 100, 100. But that's only for one run. so you must really take care where you're using AI and for which things you are using it. And especially if you are proofing an open API document with AI.

Ikenna: Cool. Andreas Reetz: that's quite expensive if you see the time. It's one course, not so expensive. But ⁓ you, for ⁓ example, do automatic ⁓ stuff, do every day, then ⁓ the cost will up and you will go ⁓ in areas where you can buy a car or a house in our area.

Ikenna: Yeah. Andreas Reetz: That's always ⁓ to have in mind what you are doing there. And especially if you ⁓ do checks... example, at the AI assistant, ⁓ we combined that with ⁓ linting ⁓ results.

So a of stuff... can you not find by just using AI. You have ⁓ to combine that to have a look on what's on in your company. Normally ⁓ enough if you just do the linting stuff go through all the catalogs or inventory.

Ikenna: Mm. Andreas Reetz: and look what's happened there and see where the problem heat maps are ⁓ where you have to consult the teams how to do it. That's something which we will do roughly ⁓ in the next do more of those ⁓ the existing inventory. Ikenna: Okay, okay.

That's interesting. If you were to to, you know, for maybe engineering leaders or people in charge of API governance who are looking see how, whether kind of assistance in terms governance adds value and justifies the where you think is the highest value place ⁓ ⁓ mentioned a few instances, right? But where have you found is the highest value place to first look for in terms of applying AI assistance ⁓ your governance? Andreas Reetz: Okay, the highest improvement you will get at the creation of an API and also to improve the quality of an API.

Because... Ikenna: Okay. Andreas Reetz: point is always to get as early in the process of creating ⁓ an If you find their problems and fix it, it ⁓ costs not so much. That's really something which we learned because we introduced linting ⁓ much too late.

Ikenna: No, no, no. Andreas Reetz: and that really will cost our company a lot of money. ⁓ So you are going ⁓ the early stage and ⁓ developer or the designer ⁓ the team sees the problems very early, they can avoid that problem ⁓ so ⁓ Ikenna: Mm. Andreas Reetz: the development stuff which ⁓ behind is expensive.

So that's my point on that. Ikenna: Yeah. Yeah, that makes a lot of sense. And that's a really good point.

know, if we can catch these things as early as possible, right, the place to get the standards. Andreas Reetz: Yeah. Ikenna: right in the first place or the place to get conformant APIs is right at the beginning, right at the beginning while designing it. So we don't need to do any kind of rework.

that makes a lot of sense. Let's talk about, let's move on to talk about developer experience. One of the things about API governance is we want to avoid API governance becoming like a bottleneck, right? Or too much bureaucracy.

And it looks like you guys have something really efficient going on. So tell us a bit about how you've avoided, what are the key things you've done to avoid API governance being a kind of bottleneck to API delivery? Andreas Reetz: Yeah. Okay, the very first point is we do not ⁓ introduce a point in the development where we are required in a point that somebody has to look on it.

⁓ We have only ⁓ the B2B APIs for internal usage we have ⁓ everything and the only point where they get in where they really get in ⁓ touch with us in the very first when they are starting that we create a space for the team and to onboard the team. That's the only point which we do by a ticket system because we don't want that we have too much spaces and so that our system is to frequent it. But adds, for ⁓ example, go live with a new API. That needs at ⁓ dataf about minutes.

⁓ Ikenna: ⁓ wow. Andreas Reetz: That be done by the team by themselves and there's no ⁓ step has be done by ⁓ our it's ⁓ all and they can do it all by them own. ⁓ So Ikenna: Mm. Andreas Reetz: That's a very important point because in the past I've seen things somebody from the infrastructure team has to do something to get something live that do not work if have so much teams because you're all over the day just putting something live.

It makes no sense. ⁓ Ikenna: Yeah, yeah, yeah. Andreas Reetz: Yeah. Ikenna: So you've really built in self-service into the whole process, you know, with all the controls, but almost self-service, you know, apart from maybe the initial onboarding when you touch points with the teams.

And I was really struck as well in our previous conversations about, you talked about your internal API store and how you kind of describe it. It was almost one click self-service, right? You can't really deploy an API without it being in the internal Andreas Reetz: Yeah. Yeah.

Ikenna: and everything is automated about that. ⁓ Andreas Reetz: Yeah. Ikenna: Tell us about all the capabilities your internal API store gives you. Now, the reason I want to just focus on this, Andreas, is when I talk to lots of organizations, this is the place they fail.

You know, they have a very poor API inventory. They know who are consuming their APIs. ⁓ If have an inventory, it's incomplete. There's no automation around putting things in the inventory.

So tell us a bit about what you have in ⁓ DatF. Andreas Reetz: Yeah. too. Okay, we have ⁓ is that we have an inventory and the ⁓ portals, we two portals, all ⁓ consume only from the inventory.

So... The first point is to get something in the inventory. we are using GitHub actions to ⁓ do the stuff. It just builds ⁓ part of the development deployment pipeline which the teams use by themselves.

So ⁓ there's click at all. ⁓ They fire the events to run the pipeline and all stuff is done. That's the first On the other point, you are looking on consumer side, ⁓ we have also two ways for the... to find the right APIs.

We have the API store there they will find all the information, can also download the and so on and to with that. That's one point. There they can also ⁓ build an own for the all the team members are still in there. They also give them the rights and so on and they also create their applications which they are ⁓ for for the using of an API.

That's all synchronized with our standard IDPs. So the client IDs and the client secrets are spread over the IDPs which we are providing and that's the point. On another point we also ⁓ creation of an app. Probably just that they create it, they ⁓ do subscription work, which they can also do in the store, but they can do it also automated.

and so they're both ⁓ ⁓ users ⁓ are not frequently working with API, they are using the store and for teams who are highly automated all their stuff they are using some automatization stuff which is also included ⁓ in their ⁓ Ikenna: Yeah, yeah. think this is very impressive because your organization has really invested ⁓ this, You've really invested in your API platform. You've really invested in these automations. You've made it a priority ⁓ to invest in this.

And I think this can't be said for all organizations. Andreas Reetz: Look. Ikenna: be interested to know, know, know, interested in knowing how or what drove that kind of in that need to in this make sure that you have a solid governance, solid platform there. I think you did you mention your actual governance ⁓ function, those of ⁓ who oversee governance, just a handful of you or is it a large, large governance like how?

⁓ Andreas Reetz: Okay, okay, we are splitted in three teams. One team just do the developer experience. They all provided the tools, the DevPropel, ⁓ automatizations and so on. They roughly seven, eight people.

And we have ⁓ an ⁓ team, we have a lot calls and ⁓ everything. must running are six people and are four people ⁓ no was three poor poor plus two no three external people ⁓ but just consulting ⁓ through through teams and also ⁓ ⁓ team also for all the guidelines ⁓ and Ikenna: Mmm. Andreas Reetz: governance, the rules which have to be applied for the EPRS. Ikenna: Yeah, Andreas Reetz: That's also quite an important point.

you just do ⁓ rules ⁓ to use or how to design APIs and you do not consult the teams, you do not get feedback. ⁓ And then have something which is more the ivory tower leak ⁓ and is on the ground. Ikenna: Hmm. Andreas Reetz: So for me quite important to speak with the teams and see their needs and where ⁓ can ⁓ ⁓ new rules or maybe can establish ⁓ good ⁓ Ikenna: Yeah, yeah, so important, so important.

So as we round up, Andreas, I just want to ask maybe one or two last questions. So the first one is, where do you see API governance? What do you, ⁓ how you see API governance looking like in the next, you know, three to five years for data? How do you see, yeah, API looking like?

Yeah. Andreas Reetz: Okay, what we are currently seeing is that we are... getting more and more ⁓ customers which are using MCP or bit demands on other APIs. for us, it's quite important to have one face to the customer.

that's also the point which we introduced this. start next year the same ⁓ we have for the Synchron APIs, for the OTHINQ APIs. ⁓ also ⁓ are thinking about ⁓ to go MCP ⁓ and have ⁓ the inventory also the one-phase tool as a customer. ⁓ So ⁓ goal ⁓ always that developers or the designer find their ⁓ one point.

And also for the automatization it's quite important that it looks the same because if you make for every part the tool chain completely different your developers get not on speed because the cognitive load goes higher and higher. Yeah. Ikenna: Absolutely, absolutely. Yeah.

If there was thing and only one thing, last question, if there was one advice you would give an engineering leader who's trying to scale up API governance in his organization, in his or her organization, what is the one thing you would tell them to look at first? Yeah. Andreas Reetz: Okay, find the right people for that. Ikenna: Hmm.

Yeah. People first. Andreas Reetz: Yeah, ⁓ I'm looking a technical aspect, a platform which is only ⁓ on ⁓ paper, didn't You have to make something really you can touch, where you can use, and so on. That's the point.

So think in papers. Start with right people. Ikenna: the right people. Absolutely, absolutely solid, solid, solid, solid advice.

There's a solid place to start. It always starts with the right people. Look, Andreas, thank you so much. It's been good.

Look, I could talk to you for another hour. Really, there's so many, you know, I find it so interesting what you guys are doing in that effort. It's, you know, highly super mature. And even the way you're evolving your platform is really interesting.

So thank you so very much. And yeah, maybe want maybe some other time maybe. Andreas Reetz: Yeah. Yeah.

Ikenna: knows we might have another conversation but thank you very much. If people want to find out about you perhaps what you do can they find out you on LinkedIn or any with the best Andreas Reetz: Yeah. Yeah, yeah, the... You will find me on LinkedIn.

Just send me a notice. I always glad to speak with people because it's always a good point to ⁓ experience because we have a lot ⁓ wrong ⁓ and ⁓ we figure out that ⁓ we are on the wrong track ⁓ and it's ⁓ you see that and you have to change it. ⁓ Ikenna: Absolutely, All right, thank you, Andreas, and speak to you later. Bye bye.

Andreas Reetz: See?