How do you update a network without downtime?

MICHAEL BIRD Okay, Sam, let's say you are about to buy a new home. What are the things that you look for in a new house? SAM JARRELL uh, price first, probably location and for me a big one is walkability. MICHAEL BIRD Oh, oh yeah, yeah.

I'm with you on that. Do you think about internet speed? I know you and I both do a bit of working from home. Does internet speed factor into it?

SAM JARRELL I generally assume because I live in the Bay Area, that I'm going to be able to have good internet access, But if I were not here, if I were like out in more rural areas, then I could see that being a huge part of the conversation. MICHAEL BIRD so I get my internet through overhead lines, and I've got quite a slow internet connection. Sam, you'll know when we do these recordings sometime, because, uh, 'cause….. SAM JARRELL That was a perfect impersonation.

You started to become quite robotic right as it happened too. MICHAEL BIRD You know, what does it take for us to stay connected? Have our communication systems constantly updated or while keeping our data safe? Sam, in this episode, we're gonna be finding out more I'm Michael Bird...

SAM JARRELL ...I'm Sam Jarrell... And welcome to Technology Now from HPE. MICHAEL BIRD Sam, we are all used to software updates, right?

You know, your phone or your computer gives you a little popup and then spends the next five, maybe 10 minutes, maybe an hour. restarting while everything updates behind the scenes. Often it's why you're in the middle of something. SAM JARRELL Oh, for sure.

It has to be only when you're doing something very important that is due, like an hour ago and. That downtime short though it is, can be incredibly frustrating if it happens at an undeal time, like when something is due an hour ago. MICHAEL BIRD That's always when it is. Yeah.

Anyway, that's just your personal device. Imagine if you had to update a system that millions and millions of people use, including the emergency services. So you need to update with no downtime at all. SAM JARRELL Yeah, that sounds like quite a complicated thing to do.

MICHAEL BIRD Yeah, absolutely Sam and one person who knows this better than anyone is Franz Seiser. Franz is head of the Data Tribe at Deutsche Telekom, a telco who recently upgraded their infrastructure which I wanted to know all about, but before we could talk about the upgrade, I wanted him to give me a bit more insight into what the infrastructure had looked like before the upgrade. FRANZ SEISER So classical telco networks were comprised of multiple silos, sitting next to each other.

So in order to provide, the network services to your customer, there are multiple systems involved. And in the past, each and every system was its own platform. So its own hardware, its own software, its own management solution. interconnected by protocols, which are.

highly standardized by an entity called 3G PP, which is producing worldwide standards. and they exactly define how the systems are interconnected. And then you have multiple of these, systems and platforms around, each run by a dedicated team, which own the hardware, software, vendor relationship, all the lifecycle, everything done. And multiple teams sitting next to each other.

MICHAEL BIRD Okay. So you, you mentioned silos. can you just talk through the silos what they are and why they exist. FRANZ SEISER a typical, teleco network consists of three big parts, the access, the transport, and the core networks.

if you talk mobile access is the base station, so the antenna, what you typically see, and some hardware. Then of course, transport networks 'cause packets need to go from A to B, and then you have the core network in the center. So if you zoom in a little bit into core networks, you have at least three big blocks. one is the packet core.

So this is where all the packets are processed and moved from A to B Then you have the voice system itself, which is called IMS. And the third thing is then the customer database. this has the subscriber information so that the tariff is defined there, how much speed you get, what kind of services you subscribe to, all that kind of stuff. Each of these things again, is comprised of multiple subsystems.

MICHAEL BIRD And, how was the communication between those silos, the individual silos? MICHAEL BIRD so the contentwise, I said highly standardized interfaces. So it was very clear what has to happen on that interface. team wise, very limited.

'cause teams knew exactly that is my system, that's the interface. I need to comply with these specifications, and if I do that, everything is fine. So if you really wanted to do something bigger that needed changes over multiple systems, you had to set up a pretty. Big exercise.

You need to start making teams talk to each other. We did not talk to each other that often. So this is really cumbersome, exercise. MICHAEL BIRD yeah, so Why did having individual silos make things more complicated?

FRANZ SEISER Because of complexity and, and need to align between the silos. 'cause you have from time to time do things where work is required in all the silos. What you also then saw, of course, you have engineers being in charge and engineers always have a certain ambition to optimize their own silo. So what you saw then is that, of course each silo was done to its local optimum, but we all know the sum of local optimiz is most likely not the global Optima.

Right?So all these things simply make you slow, make discussions pretty long and, inefficient and moving forward. It's slows you down. And over time, of course, more and more things were invented, so more silos added.

So the overall complexity was growing and growing like crazy. And,as of today, we have more than 50 of, of these type of systems in, in our network up and running MICHAEL BIRD And you, and you mentioned slow, like how do you quantify, the difference in speed that, you know, from a, from a siloed way operating to FRANZ SEISER typically in a silo, one software upgrade per year, roughly. Some may be a little bit quicker. So they're really, really, really good ones.

Managed maybe two or three. But the real big ones had one per year or maybe one per one and a half years. So this was really, slow speed as you can, MICHAEL BIRD and, and, and we live in a post AI world where everyone's expecting changes to be made yesterday. So.

that means the network isn't able to sort of keep up with technological changes or demands from customers FRANZ SEISER if we would still have that architecture. Exactly. MICHAEL BIRD So let's talk about the, the new system then. how have you gone about upgrading and what's changed?

What's different? FRANZ SEISER we have fundamentally different demands regarding speed, keeping up security wise with the environment resilience of what is a customer today expecting from a really good network. So we really had to think differently. And of course we looked What adjacent industries are doing, and you then rather quickly came to the concept of cloud and much more software.

and that required a fundamental change. the only thing that remains like before is that the network is still providing the same set of services to the customers, but the way we build and produce is. 90 degrees different 'cause you go from vertical to horizontal. So, we completed dismantled silos and defined more or less three distinctive layers.

One is infrastructure, one is the application, and the server one is automation. MICHAEL BIRD change, like impacted the way that the organization works now? so how has the change impacted the way the organization works now? FRANZ SEISER So when you're in the past, had one dedicated team per system owning everything.

Now you have three categories of teams. So you have one bigger team that. Deals with all the infrastructure. That includes, of course, all the hardware, all the servers, all the networking, all the storage then you have multiple application teams.

but they only look at the application and of the behavior of the application. And then again, you have one, common team that is dealing with all the management and automation and orchestration that sits on top. Was some of the applications tricky to migrate away from dedicated hardware? 'cause mobile technology, cellular technology has been around for decades.

billing systems probably have been around for decades, So I mean it systems, they are probably tricky because of legacy. But in Syria IT system can be moved rather quickly 'cause it systems are typically built like that and most of them evolved over time. Much more tricky are the application that really deal with network traffic and there you have a very clear pattern. The older they are from the standards perspective, the more difficult they are to migrate.

'cause in the past, of course, things like cloud were not part of the design at all. So you see typically point to point protocols and you don't do point to point in the cloud 'cause you don't know where this instant in the cloud is sitting. Yeah. So you need to get to, a stateless protocols, and allow dynamic, dynamic routing and all that kind of stuff.

More and modern architectures, were much easier. So 5G core, which the so-called service-based architecture that has been standardized from the very beginning. To be deployed in a cloud environment and to live with this very flexible environment. 4G is somewhere in between.

Some parts are easy, some parts are a little bit more difficult, Has every thing been fully solved today? So everything behaves totally Cloud native? No. And that most likely will not happen because standards are there and if the standard contains a point-to-point protocol, nobody is reopening 4G standards to replace that protocol by something more modern.

So you need to find ways to work around can be done is a, is a little bit of additional effort. So you are not a hundred percent cloud native. You're maybe only 80% cloud native, but you still can do it. MICHAEL BIRD because I guess the interesting thing about telco infrastructure is that your infrastructure sort of has to talk to your competitors' infrastructure and to some extent, and you can't just be like, wow, let's just update to something that's a bit more modern, because presumably that won't necessarily communicate correctly with other networks in other countries.

FRANZ SEISER Correct. But I mean, interconnection is anyhow very well defined as well. There are dedicated interconnect points. You very much agree on the, on the interfaces, you agree on all the information that has been exchanged.

You also put a lot of security functions there, so there are of course firewalls. there is so much fraud going on. you do a lot to, to defend yourself as. Especially on the border of the network to talking to other, networks.

But of course also on the borders to the internet, so to say.you have DDoS protection machines and all these kind of things to make sure your network cannot be killed from the outside. This is really one of the big topics. The attacks are getting more sophisticated.

They're increasing in numbers. They're also increasing in volume. So you see DDoS attacks with incredible number of traffic being generated. which also means we need to be able to react really, really quickly if it's super, super urgent, we would even be able to do it within one day.

Yeah. But typically you have at least a few days completely impossible in the old world. MICHAEL BIRD And, the other thing that is maybe particularly unique to your industry is, the fact that it's critical national infrastructure. you are providing service for emergency services.

that cannot go down FRANZ SEISER Exactly. Yeah. That makes this whole thing even more important that you protect it. Just think about it, just one minute.

How many lives may depend on that? That's something we are very, very aware and that's one of the reasons why we take all the efforts and really, really make sure we do everything that you can think of that. This is not going to happen. MICHAEL BIRD So from a migration perspective, you know, there must have been a moment where you had to switch a new system on and switch the old system off So how did you manage that?

Was that quite a challenging process? FRANZ SEISER Actually no. Because in fact, it's rather similar than doing an update. So what do you do?

You build a new system in parallel, of course. You test and test and test again. 'cause you wanna be really, really sure this thing is, is working properly. then you start first, you bring a few customers, you do pilots, you see how this is working.

and step by step you, you increase it. You know, the good thing is you are not on the phone all day long and as soon as you hang up your phone, if I switch, then in the back, the, the anchoring where your phone will talk to if you dial, you don't even recognize, and next time you suddenly go instead of that system over the other system. Same, same for data. Typically at nighttime you may not consume that much.

So if you disconnect you for a few seconds in nighttime and, bring you back with a different IP address you don't recognize. This switching over can be done relatively easily, but it needs tremendous amount of planning. but that's a manageable problem. MICHAEL BIRD I guess avoiding downtime is, is really important.

the new way of structuring your system, like, does that help to avoid downtime? FRANZ SEISER Well, it doesn't help to avoid downtime 'cause telcos always took pride in having this famous five nines availability. We also did it in the old world. The new system though, helps people like me who are in charge for certain part of the systems to sleep a little bit better.

Because you have more resilience in the network, right? there's always something going on and you, we do tremendous number of updates and changes in the network all the time. And if in a classical setup you only have two or three sites and one of the sites is going down, you're suddenly having two or maybe even only one leg. In a cloud environment that's fundamentally different.

let's take a simple example. You have one network function. You need 20 instances to deal with the capacity. If you have all 20 running on one location, or you have on five locations, always four doesn't make a difference.

the cloud is just located at different places. these four instances are running anyhow, independent of each other. So if, if I then lose one of the five. I still have four left.

And if you go to something like 10 sites, it's even better. So you need to do over capacity. If you go to 10 and one or two failure, again, you do the same, but you need to build less. So the more sites you have, the less money you need to invest in additional hardware to keep pace.

If you lose a certain amount of sites that you still can deal with the traffic, so it's even making it cheaper. to, To, to run your network in that setting. And that's one of the big benefits of this whole architecture if a single instance fail, you will not recognize, 'cause the whole thing is done in a way that it automatically is taken over by others. And here you go.

MICHAEL BIRD when it comes to automation, how much the new system is automated compared to the old one? FRANZ SEISER That's very easy. The new one is 100%. So in, in our new architecture, automation is part of the design.

we call it the zero, one and two. So preparation, first rollout and regular, operations. The testing is automated And even the rollout, it's all completely automated. So no manual intervention at any point of the system.

in the past, you do the critical maintenance you do at nighttime, and if this poor guy starts to work at midnight and is doing configurations all night long, the probability that you do something wrong is increasing because you somehow get tired. Right? If this all is done fully automated and everything is prepared and checked, this is not going to happen. MICHAEL BIRD And have you seen the roles that, um, that pe the people are doing, uh, have, have changed since automation has come in?

FRANZ SEISER Yes, of course, of course. We are still observing. you still have people on duty that watch very carefully what if it's going right? if you see things are not going as expected, you can stop it.

Of course we also make sure we can roll back quickly so we see upgrade is going wrong, but the click of a button, so to say, you, you roll back to the original situation to also be, as much safe as you can. you can easily compare that with the production side, which is highly automated still. You have a few people in the command center making sure the system is running smoothly MICHAEL BIRD Where do you see the future of the, the telco industry, I guess like relating from a technological perspective?

There's gonna be 6G and 7G, and those continual changes, like how is that going to impact your job, the, the wider industry as well? FRANZ SEISER I don't know if my glass pole is so much better than yours. but but, but, what we see, of course, this route going down, everything is fully ized. Everything is automated operations, of course coming into play much, much more.

All these AI topics we see is going to, to take a bigger and bigger role. The good thing is, again, our architecture is helping here tremendously because one of the big benefits is these whole systems are generating data in real time or near real time. Which again is a big change. in the old days, you had one data point roughly every five minutes.

With one data point. Every five minutes an AI system cannot do anything. 'cause in the five minutes a lot can happen. the more data they have, and the more accurate the data is, the better they can act upon this data and that's clearly what we going to see happen.

they can also see if things start to behave strangely, that the network itself can start to heal itself. MICHAEL BIRD I guess the thing with, Deutsche Telecom's, network, you know, you have presumably, mast and edge infrastructure, dotted around the country. and with the rise of AI and the rise of sort of edge computing and latency being super important like do you see a role for telcos in the sort of the ne the next wave of ai and as, as it's starting- to Impact us as, as humans FRANZ SEISER At the very minimum, you need connectivity.

Yeah. Whatever kind of AI have you need con and this is where we are coming from, right? So that there's clearly a role, how much we need to put ai, uh, distributed and, and how, what's the, what's the right edge. I think that discussion is going on for years.

Let's see if AI changes that to a certain extent or not. 'cause we always need to keep in mind, uh, power consumption and you need to utilize the hardware. Otherwise, it's, it's a bit wasted effort. So that's future will tell where, where this is going down the route.

But as a minimum, you need to have really good. Reliant secure connectivity, and that's not going away anytime soon. Rather the opposite. That's becoming more and more important as everything is, uh, connected going forward.

MICHAEL BIRD Amazing. Franz, thank you so much for joining. I really appreciate it. FRANZ SEISER Big pleasure.

Thank you for having me SAM JARRELL Oh my. Well, what did you think about, that very shocking stat that before in the siloed environment, folks just did one upgrade per year and now it's. constant. Right.

Did that shock you As much as it shocked me. MICHAEL BIRD So having worked in a IT department about 10 years ago, upgrades weren't that frequent and it was like a big thing, but you know, the world we live in today, upgrades are quite, frequent. I think technology just changes so much faster. but.

telcos are massive. the infrastructure is critical infrastructure. Like it cannot go down, SAM JARRELL Yeah, I'm also thinking about is automation for something super duper critical? does it always reduce risk or does it just change where humans need to pay attention to?

MICHAEL BIRD That's a really good question, Sam. I think from my perspective, we talked about this a lot, like let humans do the thing that humans are good at and let computers do the thing that computers are good at. SAM JARRELL Yeah, and I think, his examples about like emergency call reliability sort of resonated With me. And then even the complexity around like preventing DDoS attacks, I'm wondering how like if the average person has any idea, or even like the average employee at even a tech company like ours understands how much goes into making sure that, a minute long outage doesn't happen behind the scenes.

it's kind of crazy. MICHAEL BIRD Yeah. I think it's a testament to just how resilient our networks are because there was an outage from one of my phone networks last year and I was completely lost. 'cause I was like, how do I know if this is me?

How, I dunno if it's just me having an issue because I can't actually check. Because I can't get on the internet. It was a slightly stressful day. Maybe that also showsjust how critical that infrastructure is and,There are so many things like that, like public transport or phone networks or food supply chains or, or things like that where you don't really think about them until they stop working.

anyway, to bring it back to networks. I liked when he said, the way that this infrastructure is structured helps me sleep better at night. and I think, that's a good thing. That's tech technology is doing a good thing if it helps you sleep better at night SAM JARRELL That's true, but what did you think about the organizational change from vertical to horizontal?

that's quite a major restructuring. MICHAEL BIRD Yeah, massive restructuring. Massive restructuring. But I think it's a response to just how, complicated our mobile networks are.

I I got my first mobile phone when I was. I about 11 or something. They weren't smartphones. there was no 3G or 4G or 5G.

but if you think nowadays what we're expecting of our phone networks, like, how many different services and options and, that infrastructure has got more and more complicated. And I think it's about making sure the infrastructure is fit for the task at hand. And if it was more simple, silos might have been the perfect way to do it. But I think he said something like 50 systems.

and so that, that means 50 silos and so. Obviously it makes it slow and it makes discussions longer and it makes upgrades longer because all of those systems are integrated, but they're sort of separate. So actually if you can sort of start from scratch and be like, okay, if we are gonna design this in 2026, how would we design it? And that's how they did it.

Like it's. That does make sense. SAM JARRELL That's true. And I mean, you raise a good point, like when he talked about teams optimizing their own silo instead of the whole network, that it does make sense local.

Optimization doesn't equal global optimization. and the, horizontal layer approach does feel a little bit more modern and scalable. And I think also as he mentioned, it mirrors what cloud industries have been doing for years. And so I imagine like operators and, um, it organizations are probably more familiar with that, to be honest.

Um, and it does allow for a little bit more of those like. Continuous safe and scalable updates that we mentioned with having more and more things automated. MICHAEL BIRD Yeah, exactly. Now, um, now Sam, obviously we're all quite familiar with telcos.

Um, we've got phones in our pockets, but I, I thought it was important for France to tell us in his own words why everyday people should care about what's going on behind the scenes at telcos like Deutsche Telecom FRANZ SEISER As long as you want to have good, secure, reliant connectivity, you should have a good feeling and know that companies like ours, we are really pushing and driving That this is, is dealt with in a meaningful manner. 'cause you wanna be sure you can rely on emergency call, all that kind of stuff.

That's why you, you, should care and, and feel safe. Your personal data are protected. All that kind of stuff I think is super important and that's something where really customers need to feel safe and have the right trust in the network they're using. SAM JARRELL Okay that brings us to the end of Technology Now for this week.

Thank you to our guest, Franz Seiser And of course, to our listeners. Thank you so much for joining us. MICHAEL BIRD If you’ve enjoyed this episode, please do let us know - rate and review us wherever you listen to episodes and if you want to get in contact with us, send us an email to technology now AT hpe.com and don’t forget to subscribe so you can listen first every week.

Technology Now is hosted by Sam Jarrell and myself, Michael Bird This episode was produced by Harry Lampert and Izzie Clarke with production support from Alysha Kempson-Taylor, Beckie Bird, Alissa Mitry, and Janessa Ayache. Our theme music was composed by Greg Hooper. SAM JARRELL Our social editorial team is Rebecca Wissinger, Judy-Anne Goldman and Jacqueline Green and our social media designers are Alejandra Garcia, and Ambar Maldonado. MICHAEL BIRD Technology Now is a Fresh Air Production for Hewlett Packard Enterprise.

(and) we’ll see you next week. Cheers!