The Hidden Cost of Slow Autoscaling, with John Ford

A forced operating system migration is usually the kind of thing you just try to survive. But at Scout24, it turned into something very different. In this episode of KubeFM, we're digging into how a Kubernetes platform running hundreds of services use that constraint to rethink autoscaling, node provisioning, and cluster efficiency. Moving from a polling-based model with over-provisioned capacity to an event-driven setup with faster node startup and better bin packing.

We're talking clusters at real scale, where two minutes to provision a node is already too slow and idle capacity quietly eats your budget. The result? About a 30% reduction in infrastructure without breaking production. John Ford joins us to break down what changed, what broke, and what actually made it work.

From Carpenter to Bottle Rocket to the trade-offs you can only see once you hit scale. This episode of Cube FM is brought to you by LearnCube. Since 2017, LearnCube has been helping engineers all over the world level up their Kubernetes skills through training courses. Courses are instructor-led, are given online and in-person, to individuals as well as to groups.

Courses are 60% practical and 40% theoretical, and students have access to the course material for the rest of their lives. For more information about how you can level up, go to learncube.com. Now, let's get into the episode.

Hi, John. Welcome to Cube FM. So, what's new for you? What are some technologies that you've been checking out lately?

Yeah, thanks for having me. We're working on right now moving from on-demand with our Kubernetes cluster into using a spot-based instance setup with many different instance types and across different available solutions. It's our main thing that we're doing right now. All right, great.

And for people who don't know you, can you tell us a little bit more about what you do and where you work? Sure. I'm a staff platform engineer at Scout24. We have a few different companies that we own.

One of them is ImmobilienScout24. And if you're in Germany, you've definitely heard of this. It's the primary portal for finding places to buy, to rent. We also are expanding into more valuation.

And we recently acquired a portal in Spain. I work with the platform engineering team on developer productivity and cloud infrastructure and reliability topics. Very good. and how did you get into cloud native?

So before I moved to Germany, I was in the Valley and I was working at a client software company and we were doing a lot of release engineering topics and mobile operating systems. So I started working on a continuous integration platform, kind of a competitor to Jenkins. And in order to actually have the capacity we needed for that, we needed to start working with cloud vendors because we couldn't purchase the capital necessary for that. So I started working with a lot of the AWS APIs from a more application point of view.

And then as I did that, I learned more and more about the cloud native side of things, infrastructure as code. And then when I switched to Scout24, I started working full time on cloud native topics. Very good. Now, cloud native topics, the ecosystem, it moves very quickly.

How do you keep up to date? Do you, is it blogs, tutorials, videos, what works best for you? I'd say it's a combination of news sites. So I try to avoid a lot of push type notifications.

So I like to go to places like Hacker News, Reddit, different blogs I follow. Also, So in particular, a lot of the hyperscaler blogs, I think, have a lot of really good information. The open source community has really good places to go as well between different blogs primarily there. Excellent.

And John, if you could go back in time and share one career tip with your younger self, what would it be? So I think being able to tell a story about what you're working on is a really important thing for your career development. I think that it's something that I struggled with. I often would focus on just completing the task, getting the technical information done, but then sometimes not necessarily finishing everything up at the end and not really having a story to tell.

And so what ends up, I think, happening sometimes is that it's great that you do the technical work, but if you haven't actually benefited the team, the company, it doesn't necessarily matter. And I think being able to keep that in mind is really important for development. And here we are today with a story that was covered by some of your colleagues titled Infinity Transformation, How We Turned a Forced OS Migration into a 30% Infrastructure Reduction. So we want to get into this topic a little bit more deeply with some questions.

And starting out with the fact that Scout24 is one of the biggest digital platforms in Germany. And before getting too much into the migration, can you give us a picture of what the compute platform looks like? And what are you actually responsible for as the infrastructure team? Sure.

So we call it Infinity. It's based on Amazon's EKS. It's our second version. We used to use ECS, but we switched to Kubernetes a couple years ago.

We have multiple clusters. I think we're up to maybe seven clusters now. Amongst those, we have about 700 services and over 4,000 containers at peak. That is the majority of our stateless container environment We do have other compute platforms but they are not what we recommend for our product developers The product team at Scout is responsible for basically making infrastructure easy to use for our product developers, as well as being reliable and observable and have insights into how people are using the platform.

Infinity is really key to that because it gives us a standardized way to deploy, to scale and also to offload the cognitive load of maintaining a complex cluster. So hundreds of teams deployed to this platform every day without worrying about what's underneath. That works until something forces you to change the foundation. And your team was recently in a situation where Amazon gave them a deadline to move off the operating system.

Your nose were running. What were the options? Yeah, so we had a few different options. One was to potentially stay on Amazon Linux 2 as our host operating system, which would be insecure, not a good idea, or potentially look at alternative support options.

We looked into maintaining our current architecture with Amazon Linux 2023. There's brief consideration on other operating systems. and we worked with the security team to look into using BottleRocket as a potential alternative to Amazon Mix 2. But we had a really hard time with actually doing the migration with the architecture that we currently had.

Combining migrations sounds like it could go either way. It could simplify things or it could blow up. How did you think about the risk and what convinced you that this was the right call? Yeah, so I tend to agree.

I think it's generally bad practice to do too many things in one go. The reason why we wanted to do things together is that the end goal was to replace Bottle Rocket. But we found that actually creating a separate cluster autoscaler was actually going to be a very complex task. we would have to duplicate certain infrastructure we'd be making major changes and it's going down a pathway we didn't want to necessarily go so we basically decided with the with the security team that like bottle rocket was definitely something we wanted to do so it's definitely a pathway we want to go down and we we saw that carpenter would be a way for us to actually much quicker and much easier and much more safely make that transition.

So we basically broke it down into two separate projects that were part of the same overall deliverable. So we first moved to Carpenter, and then once we had finished Carpenter, we then had the bottle rocket transition happen. And the Carpenter transition basically fully unlocked our ability to do that migration much, much more safely. To understand why that sequencing matters, let's go back to how scaling worked before.

When a service needs more capacity on your cluster, something has to provision new nodes. How did that work in your previous setup, and where did it struggle? Yeah, so the old cluster autoscaler was doing polling. So it would check periodically and see if there was pending work, and then spin up new pods as necessary.

The real problem for us is that it would take up to about two minutes. and when we have as many services as we do, traffic spikes would cause an issue where in that two-minute period, the service would become overwhelmed before the new nodes would come up. As well, we basically weren't able to consider using spot instances because the time to spin up a new node was so long. So, you know, two minutes, as you said, is a long time when traffic is spiking and pods are sitting there waiting.

Most teams find some kind of a workaround for that. What was yours? We basically created a pool of over-provisioned capacity, around about 25% of our pool at any given moment. So these were nodes that we would be paying for 100% of the price that were sitting idle, waiting for load to come.

What ended up happening is we were spending about 25% more than we really should have on nodes that were just sitting around doing nothing. but for us this was an absolute requirement and it was still cheaper than just buying our peak amount of nodes 24 7. So you're paying for that buffer around the clock regardless of whether you need it. When you moved to Carpenter did that change?

It did substantially so we were no longer using a polling model everything became event driven. We didn't need to do this over provisioning. We started with Carpenter having the over-provisioning still and then slowly brought it down to reduce the risk of causing issues with scaling. But basically, we were able to very quickly scale nodes up.

We also found with the switch to Bottle Rocket that we did in parallel with this work, we found that the machines would boot up significantly faster due to the benefits of Bottle Rocket. And so this two minute period turned down to around 30 seconds. And with that amount of time we were able to safely eliminate the over For the operating system itself you had a choice between Amazon general purpose Linux and BottleRocket What made you pick one over the other At the lower level it starts up faster It has a much more, a much reduced attack vector, sorry, attack surface.

Basically, there's less there. There's less to configure. There's no ability to do modifications to the operating system itself. So everything is done in a more immutable way.

And as a result, the nodes were able to start up faster, a combination of the Docker image being much smaller, as well as the startup process being much quicker. You mentioned sequencing was critical, doing the autoscaler change before the OS change. Can you walk us through how that actually played out? And what did you do first?

And how did the second part build on it? Sure. The Carpenter migration took us about three weeks to do. Why we did Carpenter first before the operating system is, as I mentioned earlier, we previously had a very static node concept.

So it was an auto scaling group with just a bunch of EC2 images. I'm sorry, EC2 machines. what we would have had to do without Carpenter is essentially duplicate all of that logic, which would have caused us to have to make a lot of changes, not just duplication. We'd have to put different key values so that we could duplicate the infrastructure.

And we felt that at that point, we're going to be doing a lot of really custom work that's completely self-maintained. And basically, we felt that Carpenter was something we wanted to go to anyway. We had a lot of benefits from there. And so combining that and doing the Carpenter work first would allow us to have an easier transition.

Now, that is a lot of moving parts at the end of the day. New autoscaler, new operating system, workload shifting between node configurations. With 700 services running on this platform, I imagine not everything went smoothly. So what was the first thing that broke?

Probably the metadata service for the EC2 instances. So we were previously relying on the metadata service that's provided by EC2 for credentials, IAM rules, those kind of things. But Carpenter actually disabled that. So what we were finding is that applications that were not updated to work with the newer mechanisms for getting credentials started to get a bunch of errors.

um we we did re-enable the endpoint for a while uh to to figure out how to move forward um but then over time we have disabled that again um and one of the main causes there is that we had a bunch of services using old sdk versions in particular um they didn't work and the the way that we uh have a shared responsibility model for infinity we don't necessarily have a huge view into the software that's running on the cluster. So it was hard for us to know whether there would be services that would have this complexity.

So that's a security default in the new autoscaler catching old assumptions in your application code. Once you worked around that, what was the end of it? Or did the new operating system have its own surprises? Yeah, so the difference between Bottle Rocket and Amazon Linux 2, there's a new version of C Groups there.

As that's exposed through the file system, we were having issues where different file-like objects were not available on Bottle Rocket that were expected on Amazon Linux 2 and vice versa. Basically, we would check both paths. And if we saw the C Groups v2 path, we would use that. If we saw the C Groups v1 path, we would use that.

Two different issues, two different root causes. One from the autoscaler and one from the operating system. Both of those sound like the kind of thing you'd expect to catch before production. Did your staging environments flag either of them?

They didn't, unfortunately. So as I alluded to before, we have the shared responsibility model. So we present a contract to our users and we don't really question them as to what is going to be going on to the service. So when we make a change to that contract, we have to basically find and fix those issues.

One thing that made this a little bit more complicated for us and why we didn't catch them at the beginning is that our development cluster for our Infinity product is only testing the development of Infinity itself. So in general, we have an approach where our development environments for platform tools are for developing the platform, not for running the company's development environment. So basically what happens is our customers, they put their development and staging systems on our production environment along with their production services.

So it was a lack of visibility into these particular types of applications. and it's also very difficult to deploy the entire set of production services into our development cluster. So I think what we really want to do in the future is rethink how we do the development and staging environment and potentially move more towards a model where our staging environment is actually running the staging environment of our products After all of this a new autoscaler new operating system two production issues resolved, what did the before and after look like?

Node provisioning, as I mentioned, went from two minutes to about 30 seconds. We had a 95th percentile pod scheduling time of about a second and a 99th percentile, So pretty much the worst case scenario of about 27 seconds. We went down in the number of nodes by 30% because we no longer had to deal with over-frigioning. And during the whole migration, we actually didn't have any major downtime.

That 30% reduction is striking. Is that purely from removing the buffer or is there something else compacting things further? The bulk was definitely from removing the over-frigioning. So that was about the 30%, sorry, 25% of the reduction.

Carpenter's consolidation also helped us by more efficiently packing the containers onto the nodes. And because we have everything stateless in infinity, we're able to start and stop services without a lot of warning. And that basically lets us to do the consolidation, which gives us better utilization. So that's the extra 5% roughly that we saw.

From the developer side, hundreds of teams deploying every day. Did anything change for them? No. The whole point of this was that they didn't have to do anything, provided they were using recently recent versions of AWS SDKs and all those kind of tools.

But with the over-positioning, that was completely invisible to our developers. We've not had any issues with that. It didn't cause any downtimes. And for us, a big measure of that is whether we broke the website and we didn't break the website.

So it's a big success there. Congrats on that. One of the interesting things about this project is that it started first as a compliance deadline. You had to migrate, but it ended up being a modernization effort.

So what advice would you give to other platform teams out there who are staring at a similar forced migration? We worked with the team that we have internally, our security team who does a lot of the compliance. We basically, we took a step back and sort of looked at it from a different angle, not just what is the quickest way to close the security ticket. We looked at what do we want to do with Infinity and can we make it work that we improve security, we improve compliance, and also improve future compliance issues.

So for us, Carpenter itself does none of those things. What Carpenter did is enable us to do things like move to Bottle Rocket. So our security team and our team have been in discussion a little bit about Bottle Rocket, and they like the reduced security surface. So we actually decided in concert with our security team that even if it potentially might be a little bit more complicated, that this was definitely the right way for us to go.

and now that you have this new foundation in place what does it let you do that you couldn't do before a lot of things um we have different types of nodes that we'd like to use uh so our our data team would like to be able to do machine learning uh topics so that requires gpus but as i'm sure everyone's aware gpu nodes are quite expensive and so we can basically have a pool of gpu nodes for any workload that has GPUs. We're also looking into using ARM instances for the various benefits.

Some workloads are faster on ARM, some workloads are cheaper on ARM. And this lets us have many different types of instances at the same time. And with Carpenter, it's really easy for us to do that. It also unlocked our ability to do spot a lot easier, which is a direction that we would really like to go.

We're in the process of tuning our spot usage, but at this point, And we've actually switched almost, I think, not quite 100% of our workload in Infinity to Spot instances, but a large majority. I think it's above 90%. With that in mind, what's next for you? I think Spot is the main thing that we're pushing forward right now.

We are thinking about how we want to do purchasing for our EC2 instances. We have considered Fargate in the past. It's something that we're always evaluating as things change with PowerGate. But basically, Spot is our next main thing that we're working on in this space.

And if people want to get in touch with you, what's the best way to do that? I'd say LinkedIn and GitHub are probably the best ways to get in contact with me personally. I'm not a big social media person, but we actually do have some openings for this team. so you can get in contact with us through our jobs portal if that's something that's interesting to you.

For folks out there that may be looking for a new role or a new challenge, seems like a good place, very dynamic with lots of interest in new technologies. John, thanks so much for joining us and sharing your knowledge today. Really appreciate it and look forward to hearing about what's going on in Scout24 in the future. Take care.

Thanks for having me.