#060 - Beyond ELK: Elastic's 10-Year Evolution, Open-Source Licensing, and the AI Frontier with Philipp Krenn (Elastic)

Hello, everyone, and welcome to another episode of the Kubernetes for Humans podcast. Today I have in the show, Philippe. Philippe, do you want to introduce yourself? Yes.

Hey, thanks for having me. I'm Philippe. I work for Elastic. I feel like in the Kubernetes ecosystem, many people still know us as Elk.

And I've been surprisingly almost, I've been at Elastic for 10 years at this point. So I've seen a lot of deteriorations and things we have done. And I've been kind of like in different ecosystems over that time because we do a lot of observability. But I also spend a lot of time on search nowadays and the AI world like everybody else.

So that's what I'm doing. and today I'm in Vienna where I'm from but normally I'm based in San Francisco but I came a bit closer to you in time zones so it's a little easier that's cool, that's super cool so 10 years in Elastic maybe walk us through I don't know, like Elastic share a bit about your history what led you to Elastic and maybe also it would be interesting to hear your experience first handed from Elastic growing like you said from a Kibana dashboard to what it is today?

Yeah, so I was the small startup at my university back in the day, though. They did well. I think they reached 100 plus people and were sold to an American company, so they did well. I didn't kill them.

But there I had the opportunity to try out lots of data stores and we were using back in the day, like 15 years ago, it was the NoSQL days and we were using almost any NoSQL data store that was out there. We had reddit and mongodb and elasticsearch of course and we used mysql and activemq and we i i had a full circus of tools that i was using and then i started doing more meetups and conferences around that and at some point i felt like um that was maybe too much for my role because i was basically the main person managing the data stores and doing all the devops work i i did aws for the company but i was the only person and they were only one of their co-founders and me we were the only ones on the pager even when we were like a becoming a bigger and more relevant company and then I decided that maybe it's time to to do this more professionally that the conference world and that's how I came to Elastic because I knew somebody from from meetups and then I saw that they were looking for somebody to do conferences and other things for Elastic and that's how I ended up there and it started a long time ago and now 10 years later I'm still there I feel like things of course change um 10 years ago Elastic was like 300 people now it's 4,000 in the early days it was mostly conferences or I feel like I always say the world before COVID and after before COVID like it was like way more conferences still or at least for me I was traveling 200 days a year and who was just out there telling everybody what we're doing and what we can do and then during COVID I stepped a little more back now I have a larger team and I I always jokingly say I spend 80% of my time management and 80% IC work though a lot of it is is also internal facing like I get to review a lot of things so I would always jokingly say that um you know the the movie ratatouille when they try to poison the rats at the beginning and there's the rat that can smell very well and i feel like that's sometimes my role in the company that people hold up a piece of content to me basically and say like um is this good for developers or you spend so much time with the community like is this what we should be saying and then it's like in the movie it was this smell test it's like is something poisonous or not for us it's not poisonous but um uh it's more about like is this a good fit or not The main problem, though, is it's not like our content necessarily is poisonous, but it's often like, no, we should do something better.

But then it creates a lot of work to actually say how it should be better and what needs changing. So those are the things where I spend a lot of my time today. So, but, you know, like you're talking about today, you guys are huge. You guys are big.

Walk us through like 10 years ago, 300 people company. very like open source like mindset right and maybe share a bit about like how elastic started how did it like evaluate it did the evolution and maybe like you can also talk a bit about the license changes or or we shouldn't talk about it yeah so so it's up to you yeah so i at this point, I think 16 years ago, Shai started Elasticsearch. And Elasticsearch wasn't even the first implementation. It was basically the third implementation.

Three is maybe the charm. I don't know. There was Compass 1 and Compass 2. And then the third implementation, I didn't call it Compass 3, but Elasticsearch and that one stuck around.

And at my previous company, actually we were using Compass 2 in another project. So that was my very first experience with a search-related project in that area. And so he started Elasticsearch. And then over time, there were these other components that were started by the community that joined, I don't know, the Elastic portfolio or family, which was mostly Kibana and Lockstitch at first and later on Beats.

And I feel like Elasticsearch or the lore of how Elasticsearch started was that Shai wife was a chef and she had a lot of recipes and she needed to search her recipes and he wanted to build a system to help her search her recipes she's still waiting for that recipe search today he got a bit sidetracked starting a company and actually doing that but that's how elastic search as a search engine started and then it turned out that logs were also kind of a search problem because it was about like what errors did we have in the last hour or like how many errors did we have or what is the average latency.

All of those are kind of like search and retrieval problems that you store a lot of data and then you need to find something relevant in that. And then with the combination of Kibana for the visualization and Logstash, that just naturally grew. And I'm not even sure we had that plan to be a logging solution or observability solution early on, but it almost happened and it just worked too well not to do it. And then we started going down that path.

And I feel like at first it was almost accidental. But now today, of course, it's a more intentional area. We're also one of the biggest. I think we're always in the top three of OpenTelemetry contributors at this point.

So I feel like the observability story or observability has just shifted from like Logstash had its place and still has its place for many of our users. But OpenTelemetry, for example, is a very big part of how I feel like forward-looking, almost everybody looks at observability. And that's why we're heavy contributors, but also heavy users of that. So we have a managed OTLP endpoint in our solutions nowadays that you could just use.

And then you just point your applications at that. And you can still pull and parse your log lines, but you don't necessarily have to if you just integrate into OpenTelemetry. So there have been a lot of the changes over the years. Like initially, I feel like we were small and we were just doing stuff that kept working.

Nowadays that we're larger, I feel like it's more intentional efforts to some degree that observability is an official solution besides search and security, what we do as a company. But that we have kind of grown into that. And then, yeah, as you mentioned, we've had that license change. was um i'm always tempted to say like unfortunately involved in that because it was a kind of like a painful experience um for everybody involved but the the thing turns out that if you have a successful open source project there is a cloud provider that likes to take your work and sell it to others um and the problem is um i think providing something as a service is fine but like The ones doing the work don't make any money out of that, but somebody else does.

It kind of like strangles the product in the long run. So we tried to force them into their own path and we picked our own. So we have made the decision through the licensing. By the way, while a lot of people know that we relicensed to back then SSPL and Elastic License version 2, a bit over a year ago we added the AGPL as a license option back so Elasticsearch is open source again or you can get at least a large part of that source code under the AGPL license again again that big cloud provider does not want to touch AGPL normally so that gives you a safe enough path of where you can go and yeah we did the license, the first license change was in 21 and then a little while ago maybe I think it's almost two years ago at this point that we did the AGPL auction time is passing so quickly but we have been open source for two years again that's cool and I know there were a lot of back and forth there was the ready story and like I feel now that I see more and more this elastic license in a lot of open source projects yeah it's I feel like it's complicated because like everybody likes open source, but at the end of the day, you also want to have like a sustainable project.

I agree. The sustainability means that I also want to get paid at the end of the day. And most of us need to get paid. And if you are a big cloud provider, then you have a very large distribution channel and then your world looks a bit different.

But if you're an open source project, your distribution channel is more complicated or you need to find a solution. solution um so yeah i i think we but also mongodb redis uh hashicore we've all it's it's fascinating by the way i think that every one of these solutions has kind of like a slightly different approach that everybody had a slightly different problem they tried to solve or um who were the competitors or what was the problem in the space that's why everybody's solution or approach looks slightly different um but the direction of what everybody was trying to do is the same that everybody tried to find a sustainable business model for it or for their project.

Maybe now let's talk a bit about, like, you know, licensing is indeed like a sexy topic, but let's talk a bit about like the real sexy thing in our days and age, which is agents, agentic and so on. Give us your take, the elastic take. Do they like compete? Do you guys love agents, like what's your take as a, you know, someone who lives the, like the Ratatouille of developers maybe.

And like, what's the Elastic take on that? Yes. I think, um, if you don't love agents, then your CTO or whoever makes sure that you will love your agents at some point. Uh I feel like there is a big drive Um and while people individually maybe have like mixed feelings about like code generation and everything So from our perspective where agents mostly come in and I'll focus on the observability side, but it will apply to security to a large degree as well is that ideally you can get a bit away from like doing boring work and get to the more interesting work faster.

So I think the scenario that we like to think of is like You get paged in the middle of the night. And while you wake up, and maybe you make your first coffee of the day then to investigate, in the background, the agents hopefully start preparing the case already. That, you know, when there is an outage, normally you get an alert, and then you would go to the alert, and then you start pulling, I don't know, you open a dashboard, or you start searching with a query to find what happened.

And ideally, an agent can start doing that work for you. So by the time your coffee is done and at your laptop, it actually shows you a bit more of like what is actually going on that you don't just start clicking around in dashboards, especially if you don't know a system that well. But then it can actually do some of that background check already and that the agent then can give you like a three cent summary of like what is going on. And it will tell you, I don't know, we have a latency spike on the load balancer and like some connections are timing out.

And then it will potentially figure out like it is because this one component has more load or has a higher error rate or whatever. And then it might even know like, oh, this was recently deployed in this part of the system changed. Any of these are not super interesting investigations on their own. And I don't think we will be able to replace humans completely.

But some of that background and preparation work just to let you pick up the interesting tasks and then figure out what to do. maybe even have automatic remediation depending on how much you trust the agents that you have built around it but to give you that tooling that is the idea and we have now built something into kibana that is we call workflows where based on specific things either as a cron job or based on an alert or an anomaly you can actually take automatic actions as well so you could generate specific reports with that or there are other reports that are built into the solution.

But you could build your own reports. You could build your own remediations or rules around it. It could include something that you have an error rate that is higher than something and you could automatically update your status page. Because we all know people often forget to update their status page or it's very delayed and then everybody complains that you're not transparent or it's not visible what is going on with your system.

So I think there is a lot of things where agents can actually do very interesting and helpful things, especially when you wake up in the middle of the night and you're not fully awake yet. And they just help you do all the common steps that you should do and also give you some options and just make your work easier. I don't think we completely replace the humans, but agents actually hopefully allow you to get rid of some of the boring and repetitive work and just do more of that for you.

No, that makes sense. And let's talk a bit about business models for Elastic. Is it good? Every day is a good day for you guys.

Is it a bad day? Everyone is asking themselves, are we using AI? Are we enabling AI? What's the Elastic take on that?

I mean, I feel like right is a very fleeting term. or I think there's this funny quote about people saying like, oh, I'm currently unemployed and I'm so glad that I'm unemployed. Otherwise, I wouldn't be able to keep up with all the AI changes anymore. So I think it's important to keep in mind that it's a process.

It will just keep changing over time. And whatever we do today will evolve. I don't think anybody has the final answer yet. And so the right amount of AI we'll have to see or I don't want to be too evasive here but I feel like there's a lot of like Twitter is very polarized today and everybody's saying like this is the only way to do something and you need to do this otherwise you're doing it wrong and I'm not sure I subscribe to that I think skills for example we use skills a lot both to do certain things or fix things in a certain way there's a lot of things that you can codify or help LLMs to do in a certain way that is like your business standard or company approach from content creation but you can also have skills about like researching an issue or just to start a new project or how to instrument your applications there is a lot to that but it's again it's an evolving thing of like how to do that sometimes I wonder like skills are like this very free-flowing text which feels very I don't know chaotic to me like when I think like code is very deterministic and that's nice skills are often like yes you put something there and then it works or it doesn't work and then you need to have evaluations how well it works I feel like we're still learning a lot of the things but when I look at like three years ago when you started using JGPT maybe the evolution of how far we have come and how much the systems have improved but also how much better the integrations got.

It is fascinating and we made a lot of progress all over the place. So I think there is it's a process. We're not at the end yet and I feel like a lot of the products are also still working on integrating or fixing the AI story. But we are working on that and I think we have a lot of ideas We also doing a lot of prototypes internally and our SREs use it internally a lot as well But everybody is still trying to figure out what will the end result be?

And I don't think we're quite there yet. We're still in this growth phase of where agents will take us for observability. That's quite a cool take. How maybe...

maybe like so how do you envision like elastic a couple of years from now like in this like new world do you think the observability which was the i feel once it was like the main core and now i also use by the way like elastic is a vector db for some of my agents right like it's also this capability and you guys also have this security arm right so it's like security observability normal database like what do you think is is the core or is it all of it or maybe on those different capabilities or pillars that you guys have?

Yeah. I mean, we want to grow in all of the areas. And I think that where you see some of that strength, for example, is if you have a knowledge base or you have your past GitHub issues, if we can pull that into the same engine as your observability data. And then, for example, when an agent does this investigation, you have a single system where you say like this is the error rate or the latency rate that we have but it can also search for like past issues or it could search the commits that you have and then figure out like why did you have a certain change so there is definitely a strength to having multiple of these signals or data types combined and then security is becoming a fascinating field with all the supply chain attacks like I feel like every other week there is a new bad NPM package somewhere or there's some vulnerability that somebody finds.

So I think there is, well, as a company, the approach is very much like it is a strength that you can combine all of this into one engine and platform. So I don't think any one of those is going away. But you are correct. I think the first few years of Elasticsearch was very much like search.

And then it was elk with logging. and then it kind of like I want to say it almost exploded because then like vector search and AI is one big pillar today that is not going anywhere and then observability grew up from logging to more open telemetry and being all the signals and then we added security and I think also from an implementation point of view for us the integration of like security and observability it's like you have a large amount of data and you want to find what is relevant and it depends a bit on what is relevant and how you define relevance for the specific use case.

But there's a lot of shared tooling under all of that because for security, you might look for a specific hash or action or chain of actions that a user or account is taking. But a lot of that is kind of similar to how an observability system collects errors and then you have root cause analysis of like why something is failing or why something is slow. So on the tooling side, there's a lot shared under that hood, even if the solution at the top looks a bit different. So I don't think any of the solutions is going away.

For us, it's really a strength that you can have that observability and security data and then have the AI search world and can combine all of that to make a richer experience. Okay, no, that makes total sense. I think with that, maybe we will close today's episode, But before that, maybe final remarks, thoughts on the future. Where are we going?

What's going to happen? What do you want to say? Well, so I think that the future is fascinating. We're at this very interesting point where all the LLM providers are trying to go public and we'll see how cost will, for example, evolve.

I feel like nobody's quite sure. Has it been heavily subsidized? Will things become cheaper, faster, more expensive? what will the actual usage look like and will there be a divide in the board like how expensive will be llms be or junior developers suddenly become a thing again because the llms become more expensive so i think it's it's a fascinating world and everybody is trying to hedge their their bets to some degree like you want to be ready for a full ai world but maybe this is also are not happening so it's a fascinating time and every time i'm in in in the bay area i feel like everybody's very rushed to find figure out the next step and where to go um so it definitely it's not like a relaxed time in tech i feel like but it's very a very interesting time even though many people feel very pressured or rushed because everybody feels like it's we need to find the right thing now or we're history, which is definitely exciting, so it's not a boring time.

I think that's great. Okay, that's super cool. And with that, we will close today's episode. I wanted to thank you, Philipp.

I know that by the way this call was rescheduled a couple of times, so I'm super happy that we were able to get you all in good bites. So thanks a lot. Thanks for having me. Yeah, bye-bye.

Kubernetes for humans.