Milan Radia: Liquid Cooling and the Future of AI Data Centres

Welcome to the Fintech Times News and Views podcast. Established in 2016, the Fintech Times is a global multimedia news outlet centred around the world's first leading fintech newspaper. We report on the latest and brightest ideas from the fintech world. Follow the conversation using hashtag TFDnewsandviews and follow us at The FinTech Times.

general awareness in the market. Fantastic. We're here in the Middle East. It's really become kind of a hotspot for data centers.

But give us a little bit of an understanding of what do you actually mean by data centers? Because there is actually a bit of a difference in what we're looking at in these days compared to what traditionally has been built, isn't it? Yes. Well, I think the complexity has grown.

There was a time when I had a battle on my hands as an investment banker trying to persuade investors to look at data centers because they thought of it as glorified real estate. And they were coming at it from a very kind of powered shell perspective. And that was, I think, probably true in some senses because the rack densities, the amount of power that you were feeding into a rack was minuscule compared to what we're doing today. One hundredth or one three hundredth of the type of capacity that we're now looking at deploying.

But over time, I think that that legacy perception has been blown away. People now understand that these data centers are actually fundamental building blocks of everything we do in our daily lives, whether it's as an individual or as a company. So complexity has grown. And actually, some of the rules have already changed.

It's all about the connectivity that comes in and out of those data centers that really makes them useful in the kind of wider world. Yeah, I guess that's kind of come about because obviously, you know, everyone's heard of the cloud and moving things to the cloud. So it was very much about capacity. You talk about or we talk about hyperscalers, which some people will know what that means and others won't.

But they're the big boys like the Googles, the Amazons, etc. So they're all taking space because they're offering these cloud services. But we're kind of transitioning with the advent of AI into a kind of a new area of data centers, aren't we? Where it's not just the data that's stored somewhere.

It's the processing power that's used in these centers as well. That was always the case. Look, in the end, what you're doing is with the data centers, you're taking computers that could live inside a server room in a company or in your computer, right, on a hard drive that's connected to it. and you're moving it to another location, right?

And then you're moving a lot of the processing capability, the applications and so on into the cloud. I mean, what the cloud is all about was taking applications and running them remotely, doing away with the need to have a whole bunch of servers sitting in your office, providing greater resilience, and the ability to have a distributed capability. You could log into those applications anywhere. It removed a lot of restrictions from the traditional way of doing things on-premise.

But also then what Amazon and Google and Microsoft and the other hyperscalers that you talked about brought in a way of capabilities that were beyond the internal capabilities of any company. I remember working for UBS, one of the last financial trading companies or banks to move to the third party. But they kept it in-house for as long as they could. And then eventually even they gave up because they realized that what was being offered by specialist vendors and specialist capabilities, analytics and real-time intelligence and so on, they couldn't keep up.

So that's when they couldn't move it. Now you mentioned a very good point AI is resulting in another explosion in terms of activity And that I think really comes about from the massive amounts of computing that are being devoted to training AI the machine learning developing those large language models that really underpins AI. And now with this transformation in terms of agentic applications. I mean, actually, I was at a big summit in Abu Dhabi last week and a very eminent professor based in Copenhagen said AI minus BS equals software, right?

And I have a lot of sympathy with that view where actually what you're doing in the end is developing software applications that are very intelligent, can predict and take on a lot of the workloads for you. But at the end of the day, it's just software. We need to have that reality check sometimes as well. I think there's a lot of tendency to get carried away in the sci-fi of the idea.

And definitely for an average consumer, they're thinking of a Skynet or this sort of stuff and machines taking over. But hopefully we're a bit of a way away from this at the moment. Yeah, but we're loving it, aren't we? I mean, if you've used Gemini or ChatGPT, it becomes a little bit habit forming.

Because actually getting them to check some report you might have written or students beware, of course, there are lots of potential issues associated with that. But it becomes a way of life if you sort of get ingrained in that. And even for kids, you know, at school they're being taught how to leverage NanoBanana. Now that actually has massive implications in terms of compute.

and bandwidth. So many of the networks, for example, in Africa, Meta today is hitting capacity constraints and if networks, having built out these massive subsea cables coming from Asia, traversing through Africa and running up to Europe, they found pretty quickly, much earlier than expected, that those networks were hitting capacity constraints. So they're building new ones, even bigger. Without the consortium members, they want to own those networks end to end to feed and support this kind of activity on the line.

It's a bit like building roads, isn't it? You build a four-lane highway, and then by the time you actually get cars down it, you realize you should have built 12. You and I are Dubai dwellers. Who'd have thought that Sheikh Zayed Road within six lanes on either side would start to hit constraints and see massive amounts of traffic, right?

So absolutely. So these super information highways, subsea cables, which carry what, 98% of the IP traffic around the world running over subsea cables, highly, highly strategic. And actually, we're in a nice region in that regard, because a lot of these subsea cables coming from Asia pass through the Middle East. And you're seeing these incredible hubs being formed, you know, on their way to Europe.

Well, that kind of leads me into what I was going to ask next is why? Why is the Middle East becoming a hub? And is it just because of the subsea cables or is it access to power? Is one other thing that jumps to mind immediately, given the large amounts of oil that's available?

What's the reasoning? No, look, there's a number of different factors. I mean, I think, first of all, the economies in the Middle East, whether it's UAE, Saudi, Bahrain to some extent, they're firmly open for this type of activity. the regulatory structures, the ease of doing business, setting up corporate entities, the way to actually, as you mentioned, access large amounts of power in a very structured way, in a reliable way, not the day-to-day reliability of the feed, but actually the reliability of when someone says you're going to get power on this state, you get it, right?

What we've seen in Europe, for example, is the national grids and so on saying, yes, you'll have power. And then saying, actually, you know what, we've decided it's going to be five years later. That is destructive to any kind of data center business model. But it's a general ethos around it as well.

Look, I mean, the UAE, Saudi, a lot of AI investment going in, a lot of sponsorship of these initiatives, very, very large scale companies being created under the auspices of IHC and so on, which are the big investors in Abu Dhabi and so on I mean that is propelling a whole ecosystem in itself within that So you are seeing quite a lot of major businesses being created in the UAE And of course you got these bilateral deals as well US to UAE GPUs are becoming available in the region, which are feeding into a very, very large data center project.

So I think the connectivity is one part, but there's an array of other elements that are feeding that. And how does the, there's a term that I came across called on-soil data centers, and is there a drive for governments within the region to demand a certain amount of data stays on soil within their regions? And is that driving this as well? Well, this whole sovereignty issue, which is what you're alluding to there, is a big one.

And the simple fact is, it's becoming clear that actually a lot of data is deemed to be quite confidential. And so what you're going to see is proprietary AI models servicing that. We've just had a little bit of controversy in the US where came to light that quite confidential data for the US government was being fed into ChatGPT. Now, as many people will understand, whatever you feed into ChatGPT, you know, in the open models becomes effectively part of the public data set, right?

So not to say that ChatGPT is feeding that into any specific outputs, but it could. If asked to write prompts, you know, that data becomes part of the data set that will be analyzed and fed out to other users. So more and more you will see proprietary instances of LLMs. Remember, a lot of what's in the LLMs is available open source.

So people are building their own LLMs and building them for government or very large enterprises for their specific consumption. But the whole data sovereignty discussion is definitely leading to more data center capacity being delivered for the usage of a particular nation to keep that data within the boundaries of that country. So then the other thing that I'm sort of thinking about, we just liken building data centers to building, sorry, the cabling to building roads. Data centers don't get built very quickly, do they?

And so we're in a bit of a, let's call it a hype phase with AI. Do we know we're building the right things? And is there a chance that in like a year's time, we're going to have to rip some of these things down and build them again? That's an excellent question.

So look, I think you're spot on in many respects. Let's take rack densities. So today people think, you know, air cooled, not liquid cooled. Air cooled, air is not the best conductor, so it's tend to use for lower rack densities, traditional applications.

People thought 20, 30 kilowatts of rack was great stuff. It would keep you at quite a long time. That assumption is being challenged because NVIDIA's GB300s, which are becoming commonplace now, require 150 kilowatts a rack. And that's where you start to use direct chip liquid tooling.

Liquid is a much better conductor of needs. already we're starting to see those traditional data centers of which many many hundreds of megawatts into the gigawatts have been built in the US and elsewhere so if I was an investor in some of those traditional data center models I would be a little bit concerned and many of the private equity firms have done that so you've got this kind of rack density capacity moving up and up and up and where do NVIDIA think we're going over the next five years one megawatt per rack and that actually comes with high voltage DC so they're turning the kind of power infrastructure within the data center campus into a new direction.

Now, we'll see how that is adopted, whether the technology vendors keep up, quick in-up, and so on. But the risk of technology obsolescence for large-scale data center investors and operators has, in my view, never been greater. And then, of course, you've got the shift away from LLM training to usage, inferencing of artificial intelligence technologies. That's actually quite latency sensitive.

You and I are having a conversation with Gemini. So we expect it to be real time. And if we ask for an image to be created or a video to be created, we don't really want to wait 10 minutes because actually we're not going to go away, come back. Conversation flow is broken.

Of course there really be a lag but the latency parameters of data centers you inferencing how distributed they need to be is also a shift on So you know some major data center operators are telling me that in that pipeline 70 favor of infancy, some cases 80-20 in terms of the requirements and RFIs, RFPs that they're now seeing. So it begs the question, you know, we talked about the cables and building a four-lane highway instead of a 10-lane highway. What's to stop people building data centers now that you might have to rip them down in years time if things take now it's an excellent question this whole issue of obsolescence i think is now becoming real again for a long time people thought rack densities the amount of power it load that you provide to an individual rack 20 30 kilowatts would be fine right so even at the upper end of what these hyperscale build-to-suit facilities were targeting many many hundreds of megawatts of that type of capacity has been put in place especially in the us and and some other regions as well.

What we're actually seeing is some dramatic shifts in rack densities from the likes of NVIDIA. So they're now talking about GB300s becoming the mainstream GPU kind of in adoption this year and into next year. That requires 150 kilowatts a rack. So the whole air-cooled notion is giving way to direct-to-tip liquid cooling, liquid being a much better conductor of heat.

Now, if I'm an investor in some of this already legacy, large-scale capacity, I'm a little bit concerned because it isn't easy to retrofit liquid onto an air-cooled infrastructure. It's very cumbersome. And actually, the hyperscale customers don't really like that. They want dedicated builds that adopt the designs that are now putting forward.

And what people don't realize is a lot of what's being built for the hyperscalers is built to their designs. You can't just randomly build a facility and expect them to adopt it in most cases. They will require customization to their own requirements. So we've got this paradigm shift.

And NVIDIA is already now talking about one megawatt racks and high voltage DC, which is changing the way that the architecture of the data centers is provisioned. So that's actually a material set of risks now coming through for these hyperscale data centers as have been built to date. At the same time, you've got this big shift happening in the usage of artificial intelligence. So a lot of what was being done before was training, large language models.

I think you'll understand that. We're now shifting to usage, inferencing. Now, that's actually quite latency sensitive. And I used to have this debate with some of the investors and they would say, well, no, actually, that kind of AI thing is not latency sensitive.

People are willing to wait. Well, not really, actually, because what's now emerging is if you ask a question to Gemini chat GPT. Not only are they using their own LLM and the training to do that, they're also searching the web. They're also using third-party data sources in that instant to deliver you your perfect response in their view to that question.

Highly latency sensitive. Now, that doesn't work in the countryside where a lot of these AI data centers have been built with minimal connectivity and no real regard for how you connect to all of these other avenues of information. So you're going to see much more distributed air-density data centers emerging, not necessarily small. The notion that these are going to be edge data centers, 5, 10 megawatts, I dismiss.

I think actually they're going to be 50 to 100 megawatt data centers, but distributed and benefiting from very high levels of density and connectivity. And they'll also be suitable for online gaming. They'll also be suitable for video and Teams and our peer-to-peer conversations on video and so on and so forth. But they will be a little bit more localized.

And today, in the region where we are today, doesn't really exist in that way. That infrastructure is yet to be built out. In Africa, it doesn't exist really at all. So we've got a long, long way ahead of demand for data set of capacity, but maybe not quite what's been built in the past.

Thank you for speaking to me at the FinTech Times. Thank you for joining me at the FinTech Times. It's been a pleasure. Thank you so much for having me.