Holding Kafka Right: Product-Friendly Streaming with TypeStream

Tobias Macey: Hello, and welcome to the Data Engineering Podcast, the show about modern data management. This episode is sponsored by Data Driven dot I o, the free data engineering interview prep platform built by data engineers for data engineers. Have you ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill separate from the job.

Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL only or Python only practice, datagerman.io covers the full interview loop. Star schemas, slowly changing dimensions, grain and fact table design, item potency, watermarks, dead letter queues, change data capture, and back pressure.

Every question comes from real data engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to data engineering podcast dot com slash data driven today to start practicing. Your host is Tobias Maci, and today I'm interviewing Jevan Meltay about the challenges of building a reliable streaming system with Kafka. So, Jevin, can you start by introducing yourself?

Jevin Maltais: Sure. Hey, everyone. Yeah. Thanks for having me, Tobias.

My name is Jevin Malte. I'm a here in Ottawa, Canada. I'm a fractional CTO staff engineer type of person, usually, like, early stage companies. So lots of experience like Zapier or Clio, but also, like, early stuff.

Yep. That's that's me. Don't don't necessarily come from, like, a data engineering background, pharma product development, but I've learned a lot from the product the data folks. So excited to be here.

Tobias Macey: Yeah. And it's been increasingly the case where no matter where you're working in the stack, data becomes Jevin Maltais: a bigger and bigger portion of what you have to do even more so with all of these AI powered features that we're jamming into products willy nilly. Yeah. For sure.

I mean, data at DataJudaires, you know, they're they're in the background and and not necessarily appreciated, but it's it's gonna be going to become increasingly like the the moat, right, of, having access to maybe customers' data that you can run AI on because it's so easy just to I mean, it's far easier, I should say, for building those AI products, like building products with AI instead. But the data, of course, that's gonna be really the proprietary stuff that is gonna be hard for other competitors to build.

Tobias Macey: And so digging more into that, how did you first get started working in this space of data? Jevin Maltais: Yeah. So I I started I got recruited into Zapier as an engineering manager on the data team. And I think liked my profile, like moving fast, moving scrappy.

And so getting embedded there, learned a lot about just really the ETL, like just building traditional ETL pipelines with like Airflow and just where the purpose is really for BI. But of course at Zapier, their whole thing is just moving data around. And so I ended up moving to the growth team and we had a lot of problems with just trying to enrich our data for doing leads and managing sales. And so that's really how I got started into thinking about data pipelines and how we do this and why do we do this only at midnight every day?

Why can't we do it in real time? Is there a better way? So that was kind of my first touch with it. And then I moved to a company called Humi, which is a payroll company up here in Canada.

I was leading the payroll engineering team. We were moving about $20,000,000 a month just in payroll, and we had acquired a bunch of these different systems. And so trying to connect up our payroll, like our front end with our backend, where we were tracking all of our finances. Then we acquired like a time tracking company for like restaurants.

And so trying to sync all these things up was like really challenging when you have two different databases at a front end that wants async stuff. So how I started thinking about like, yeah, how do we work with data coming from the product side? And then kind of gig more recently over at Clio where I was managing a staff engineer on this documents team, where we had like 1,800,000,000 documents. We had to kind of manage the metadata or move these documents around between the different, a lot of different systems.

And so, yeah, so I had to start looking into how to do that. So I didn't go into it to be like, I want to, yeah, just move this data around in the backend, but really coming from product requirements. And so that's where I started coming to the school of learning all these data people. Tobias Macey: And so now you have started building TypeStream, which is focused on addressing some of these streaming use cases, particularly for people who are more product oriented and don't necessarily have or want to develop a lot of deep knowledge of things like Kafka.

And I'm wondering if you can just talk through some of what it is that you're building there and why and how it is that you decided that this is where you wanted to spend at least some portion of your time and energy? Sure. Jevin Maltais: So most places where I've worked, they they had Kafka already. So I mean, three places that I in two of the three places I mentioned, we already had Kafka, which I thought was cool.

You know, I'm learning about this technology, but I I started looking into it and understanding all of the functionality that it has that we can be leveraging. I realized we weren't really using any of the cool stuff at all. In fact, I think the reason why we use Kafka is people were just thought it was the right thing to do because it sounds like it's a smart queuing system. It's a queuing system for people who have to move a lot of data when in fact, it's not that at all.

In a lot of the cases it was moving. We were using it because we wanted to move data from one database to a different database. We would do either synchronous requests from one to the other at a certain time, or maybe you do it asynchronously. And I was like, Man, why aren't we just using a simple queuing system for that?

But I started looking into it. Was like, Wow, why don't Kafka's got this really cool thing called K tables or interactive queries where it can actually be the source of truth that could As you pass these events through, we don't need a database at all. We could just query these little endpoints with the particular optimized and indexed query that I want. Like, wow, that's great.

Why are we moving this data at all? Or why don't we just send a notification of a new user signing up into Kafka and then all of our systems react to that instead of having to asynchronously copy it to a different database and then check these things. So I'm blabbering a little bit, but as I was just exploring Kafka more to understand the technology from first principles, I saw that we just weren't using it properly. And there was a lot of functionality on top of it that could that I thought would be really applicable to our team, despite, you know, it being a very heavyweight product to be able to go and deploy and maintain.

So we can get to all of those, but that's kind of what led me down this path of wanting to build TypeStream is enabling both data teams, but also product teams to leverage the best that Kafka has to offer without having to be a super heavy data engineer, or even if you are a data engineer, to build this for your product teams who are requesting it, you know, trying to enable those teams. Tobias Macey: Talking through some of the product engineer challenges around some of these streaming use cases, as you pointed out, you had Kafka already, so there were certainly somebody who at least had some familiarity Jevin Maltais: with what it can provide.

But what are some of the situations that you experienced that led to the problem of you're holding it wrong? Yeah. Sure. So I can talk about two examples kind of at Clio.

So the first one was we had kind of a central model of database that was huge, and we acquired a different company that was really a different product that was very marketing focused. So in both cases, we had like these user entities. So we had users users tables, organization tables, you know, the first two things that you probably built. And two these are two very separate products.

And so we wanted to make sure it's seamless with SSO and and everything. And so so what we do is you'd have, you know, a sync that would happen periodically or worse yet, a user would have to log in and press the sync button from one platform, and it would start just to dump all of the changes from the previous timeframe to the other database. So I think the way I was looking at this is why don't we just instead, from both systems, have a singular event that's schema'd called create user.

And we just have this that goes on to this Kafka topic or a channel or, you know, if if you're not as familiar with the Kafka space, and both systems just listen to that. They just create it in their own database if they want. So that would be like the simple place. Simple thing is regardless of where you sign up, you could just have the single place that would just submit it, and you can robustly go and know where to go to to get this.

That would trigger an event. And then we could have we could take it further. We didn't. So that was just first thing is let's try to admit these events in a singular place and have a single source of truth.

And the other is like, why don't we just build like a central customer database where it would be updated in real time? And so trying to build out like just what's called a K table, where as you have these events come through, it will like materialize these tables that you can go and query using, you know, gRPC or HTTP or whatever to get the most recent up to date information. So you can get a picture of users and you can combine stuff from different topics. So, you know, if they're joining an organization or whatever.

So I think this is a far more robust way to try to pull information from different places. The other major project that I think was pretty interesting for this use case was what we called Clio desktop. And this is our own version of a Dropbox type of competitor where, if you're a law firm, you've got your easily 5,000,000 documents that you have stored with us and you want to sync those to your computer, but you don't want to always have them all on your computer. You want to have them maybe cloud synced.

Anyways, free product that we were giving away kind of as a lead gen. And we were having a hard time just trying to keep both sides up to date. You have to make a change maybe on the web UI, or you make a change on the client UI, or heaven forbid, you try to do two things within the same few seconds and you get some sort of conflict. And so that's where it would be it was a great use case.

Was like, hey, let's be able to whenever you add a file, whether it's local or on server, you have a centralized place for a source of truth saying like create file and both both systems downstream would consume that both on the server and your own desktop client would have, like, an endpoint, and it would just be able to stream in all of those changes. Far better than trying to have two big data stores that you have to do reconciliation between. So pretty interesting for, like, a very large scale product.

Tobias Macey: With the use of Kafka, just the Kafka infrastructure itself can be very heavyweight. It requires a lot of operational overhead, particularly if you aren't using some of the newer builds that have given you a way to drop at least some dependency on Zookeeper. There's the challenge of how much data you have to keep resident on disk for Kafka if you're not using the automatic offload to things like object storage. And so for teams who say, yeah, this is great.

I want all of these streaming use cases. I wanna be able to query things automatically, but they don't necessarily have that operational capacity or necessarily the budget to go to a Confluent or an AWS managed Kafka. What are some of the ways that they should be thinking about the trade offs of Kafka operationally and architecturally versus some of these other approaches for building streaming systems, whether that's something like an AWS Kinesis or there have been a number of competitors who are coming and nipping up the heels of Kafka through effectively ground up rewrites that are protocol compatible.

So I'm thinking RedPanda, AutoMQ, and Pulsar being the main contenders there. Just how much of it is the interface and how much of it is Kafka specifically that is truly beneficial? Jevin Maltais: Yeah. That's a great question.

So I actually don't know the difference, the underlying difference between too many of these, like Kafka compatible ones. I mean, for me, it's like, I don't care. Just as long as it's Kafka compatible, it should be swap like swappable. My my key that I you know, I'm interested as data as a product engineer is like, what are those things that I can build on top of it?

Does it still work with all of those great libraries? Kafka Streams, Kafka Connect has all the stuff out of the box that I can connect with. So I actually don't know the difference between a lot of these. I'm maybe a pleb in that respect.

But I think the interesting, again, for product engineers is the competitors with like temporal and ingest, which are doing some of the similar things that are far more like orchestration and workflow oriented. So the case that we talked about, say with, you you have a user sign up, right? You could have a temporal workflow where you have user sign up event that's submitted and then you have the workflow that will maybe it'll do the dual right for you to your two different data stores.

But Tempora will also manage all of the retries and all of the fallbacks and all of the logic around that, which I think goes a long way to be able to help you build out these very event driven workflows. And so that's where I see like the biggest competitor to something like a Kafka adoption, which I think is actually a very positive thing, right? So they're giving a lot more tooling out of the box to enable people to build these very durable workflows instead of, you know, the traditional thing we'd go and build, or like, I'll go build a cron job or I'll use like a Redis store to manage a cache and just pray it doesn't go down because I didn't really think through about, you know, item potency and things.

And so, yeah, that's kind of where I'm seeing things going from from my side. I know it's very I mean, I think I love the idea that I think AutoMQ came up with of, let's just use s three for, like, the backing instead. It's like, I don't know what the the folks at Confluent thought of when that first came out. It was like, oh, shoot.

That's a great idea, you know, probably. Tobias Macey: Yeah. There's definitely a lot of second mover advantage in terms of all of these competitors where they saw Kafka and said, that's great, but here are all the things that make it harder than necessary because of the fact that it was engineered quite a while ago and very much in the, I guess, middle point of Hadoop where everything was Java, everything was focused on the Hadoop ecosystem and how you can work with that. And before s three really came and ate the lunch of the entire HDFS and then led to things like Spark and MapReduce being largely obviated.

Jevin Maltais: Well and, I mean, I don't I know those words, but I don't never used any of this stuff. Okay. Just to just to show, like, my bias of where I'm coming from here. Yeah.

And and so I struggle, like, at at actually all three places to try to advocate for getting Kafka set up was a burden. Right? It's like, okay, I think the two places we already had it, they had Kafka, but they didn't have like the schema registry, which I would probably argue is like the most important thing that you need. When we were at Zapier, we had a data governance team and it was made of like, I think three people and one person full time, she was amazing.

She went and wrangled all of the different teams of all different things that they were sending as just raw JSON onto the Kafka topics, like just unmarshaled. And so like, okay, these are the 123 different events that we have. I made them in the schema registry. Could you all please go and migrate to that?

And of course they're like, no, there's no benefit to me doing it. I've already done it. But yet like, but yet there was downtime all the time because we'd have an event that would you know, that would be changed upstream and downstream. They were expecting things a certain way.

And so that's the other piece is, so we got the Kafka. Kafka itself is like a big thing you gotta deploy. The schema registry is another thing that you have to kind of understand and then want to have some, this idea of type safety and then deploy that. And then you probably wanna have some config as code around that.

So, you know, if something goes down, you wanna have all your topics rebuilt in a cert with a certain configuration that they want. So, yeah, it's a lot, it's a lot to manage. You know, we, with TypeStream, the idea there's this lovely thing called the Bayesium, if you're familiar with this and probably a lot of people are, but in a bunch of places, I've talked to, they're like, yeah, Kafka sounds cool. They're like, I just don't see the need for it.

I'm like, listen, let me just expose all of your events for you. So within TypeStream, we have it bundled where you could just connect up to your Postgres or your MongoDB from within the project itself, and we'll stand up a durable Debissium for you, which will basically just listen to all of your changes in your database, all of your tables, and put them onto their own respective topics. So you're you're immediately getting all of your you're baking your database asynchronously.

You can respond to your database asynchronously out of the box. So that's a pretty common way that I've seen people do it is just throw Devisium on there with your schema registry and put that all start reacting to your database. Tobias Macey: Talking through a bit more that product focus where Kafka is this powerful substrate, but is very easy to shoot yourself in the foot if you don't know exactly what you're doing that provides a challenge of how do you build an appropriate abstraction layer that makes it more approachable and easier to incorporate in these product focused features without inadvertently going down a path where maybe you have too many topics and not enough consumers or you haven't charted it appropriately ahead of time and how you can deal with some of those early decisions that can have long term consequences.

Jevin Maltais: Yeah. The one that that I was it was hard to get my head around is this whole idea of partitions, which probably is like a very newbie data engineering thing to to get their head around. But like, it's no, I just want to have one partition. Like I can understand that.

I just want to take all But of course you don't want to do that. You want to be able to support simultaneous consumption of your topics. So, yeah. So this idea with TypeStream is even before that is like, I discovered this thing called Kafka Streams, which is a very powerful Java Kotlin library for Kafka, where you can write kind of these minuscule little Java snippets, and it will allow you just to pass data.

You can react to whatever Kafka topic, but you could do transformations in line and you can connect up all these different Kafka streams together and then drop them into say a topic. And then you can have that topic being monitored by a Kafka connector, which is effectively dropping the data into whatever type of sync that you want, S3 bucket, database, elastic search. You know, there's lots of these built in. And so when building I these these things out of these companies that we talked about, I was trying to deploy this and they're like, Java?

Jevan, are you serious? Why are you trying to deploy Java? We're a Rails shop. We're Rails people.

We are literally Java refugees. That's why we started with Rails because we didn't want to compile things or do this massive configuration. So that was a big barrier. And so the path that I took with TypeStream is like, Hey, let's just do config as code.

You can write your JSON. Everything is strictly typed within kind of the, within all the topics that we built, have TypeScreen built on top of. So you can be really confident that that when you build your little JSON defined definition for your whole pipeline, that everything's fully typed inside of it. So it's very composable and you can have all the niceties of tooling around building your, doing your code generators for your clients on top of your topics and stuff.

And so I think that's far more digestible for people somehow, if even though it gets compiled down into Java, it's less offensive than if, you know, we just have this JSON file that defines the entire pipeline. And so, yeah, that's kind of the idea is like, let's take the best of Kafka. Let's use Kafka streams. Let's use K tables.

Let's have these gRPC endpoints. So I don't have to build a database for every service that I want to build instead of define in JSON and just have this deploy for me. And so, yeah, so the vision is like, yeah, if I'm a data engineer and people are asking me to do this, or if I'm observing that the product teams are like, man, they just want to put up another database for this little service. Like, Hey, here's a tool that we can give you.

Use Kafka tables. We've got TypeStream set up for you. And you could just instead query this endpoint and just define the query that you want to have index for. I think as far, will just, will allow us to scale far further and we could, but we can keep that nice scent like source of truth of Kafka in one single place.

And we don't have to manage 50 databases across, you know, 70 services. Tobias Macey: One of the challenges that inevitably happens whenever you are trying to build this more approachable facade on top of some underlying infrastructure is that every abstraction is going to leak. And so you have to determine, okay, where do I put the escape hatches so that if somebody does need to reach underneath and twiddle some knobs to get exactly what they want out of it without having to disrupt everything that's built on top.

How do I do that in a way that doesn't feel terrible and actually feels as though it's part of the overall intended experience and becomes intentionally designed rather than an accidental hack that you have to retrofit in in there afterwards. I'm sure you could talk through some of the process that you went through to figure out what were these appropriate interfaces and escape hatches to build into this more product focused layer on top of Kafka. Jevin Maltais: Yeah. I think so because the heart of it is really, let's just expose all of the greatness that has that Kafka has to offer.

I'm gushing a little bit. I am a Kafka fanboy. There's no doubt. But let's let's expose what it has to offer to people to make it as approachable as possible.

And so with that, the layer that we have on top is quite thin. So really what we have is the compiler that will go and take your JSON, build a graph from that, calculate the types, and then configure your topics in the appropriate way, and then deploy Kafka streams at in a single process there. So that's that's really that's really it. You can do it through a CLI or you can do it through JSON.

And so it's it's quite lightweight. The nice advantage of that is you can use your favorite Kafka observability tools on top of that. With using Kafka Kaf Bat, which is like the the successor to Kafka UI, you can go and observe the topics right away. And so you can just the data is not hidden and the abstraction's pretty thin.

So and all of that's kind of linked within the TypeStream UI there. But I so I totally agree. And that's what we've really been thoughtful about is like, what is what's the right level of abstraction here? Where if you're like a data engineer who know what you're doing, you know, you can build this JSON, you can go copilot, deploy it, go observe how the topics work in your favorite tool.

Or if you're just a product engineer who wants to go and deploy this thing, you don't care, you know, exactly what the underlying thing is. But if something breaks, yeah, you can go take a look and observe the topics, look at the consumer groups, see what's being consumed, and all of that kind of maps to the the TypeStream interface to see, you know, so you can see it all there. So anyway, it's something I was struggling with. That's why why I'm babbling a bit is I totally hear where you're coming from.

And Tobias Macey: so for teams who do have the opportunity to take advantage of some of the niceties of TypeStream where they don't have to do all of the schema registry and manage all of the partition mappings and determine the appropriate topics, etcetera. They say, I just want to be able to get data from over here to over there as fast as possible. What are some of those typical use cases that you're seeing people use TypeStream or even Kafka Sans TypeStream for in a typical application led architecture and design?

Jevin Maltais: Yeah. So common ways for talking about moving data between databases and there's other use cases, but I think that's that's a very common one that I've seen and a popular one. So say you have like Postgres and you wanna transform something into Elasticsearch. You want to maybe do some full text search for documents that people have uploaded, for example.

And so in Postgres, you've got your row that has the file, it's like the file name, and it's got the reference to S3, for example. And you don't have, let's say you don't have Kafka set up at all. So the first way would be like, let's connect to Bezium, which is a, you know, does the change data capture that can sit on top of Postgres, listen for all the changes. And anytime that what that particular table changes, it will emit an event into Kafka topic.

So great. Now we've got an asynchronous list of events that come through. So we've got But it's still got all of the It's just the raw metadata around the file. So the next thing to do is you want to be able to go and extract.

Let's get that content from the file itself. So typically what you'd have to do is make a little Java type Java Kafka stream thing that would connect it to the topic and it would have to have the Google or whatever it is, the library to go and download that file. You'd have to parse it. If it's PDF, you got to go and read that.

And so great, now you've got the content. And so you'd emit that into a So you take that Let's say simple case, you take that raw content, emit it into a new event or a new topic. Here's the extracted stuff. And then what you could do is connect up what's called Kafka Connect, which I didn't talk about it much, but these are just libraries that exist, most of them free, that connect to all of your favorite data sources that you like.

You can pull from data sources or you can drop them. So in this case, we're talking about Elasticsearch. There's a great Kafka Connect for Elasticsearch there. So you could stand up a process that would do that, that would listen to the topic, and you do a little bit of configuration to say, okay, here's from this metadata, you can go and drop it into this particular type of row or whatever it is, the document in Elasticsearch.

So it's a very common path of trying to do that whole thing. So with TypeStream, it's a bit simpler. You you could just have open up your TypeStream interface. You can type in your Postgres database and that will launch Debezium.

It'll start pulling the tables that you've selected. And we have a bunch of kind of pre configured internal nodes that will like say download file or extract text from file, do OCR, and it'll expose that into another topic. And then you just drag and drop the Elasticsearch sync on there, and it will go and launch the Kafka Connect for you. So that's like one very, very common example, I think, of how people are using it today or should be using it instead of doing like a cron job or a Redis or something to to manage that pipeline for you.

Tobias Macey: When you're talking about Debizium and change data capture, it's definitely great when it works, but there are also lots of edge cases and failure modes that can be very difficult to deal with, particularly when you have a large initial sync that you have to do to be able to populate the current state and then get to the point where you can catch up and just replay with the write ahead logs. The biggest problem is usually that all of a sudden, Kafka or Debesium goes down and fails to consume the logs for a substantial period of time, and so you have to reset the replication state and do the whole sync all over again.

And so how do you try to mitigate some of those challenges for people who are dealing with having to run these complex CDC jobs? Or are there cases where you're saying, we recommend it for maybe databases of this scale beyond that, kind of good luck, figure it out. Here are some useful pointers. Jevin Maltais: Right.

This comes back to the question of how much abstraction are we building on top of it? And there's people that are just way smarter than we are in the doing the Debizium stuff. And so we just we're just deploying Debizium vanilla with basically the customization that they can compatible with ours. And so when we're getting into like the complexities or edge cases with Debizium, we're pretty like, I'm pretty clear with the product.

Like, listen, you gotta, like, you just gotta make this work with Debizium. Most of the time, this is not the issue, especially people who are just getting started with it. Like they're not going to be depending on these events immediately, right? Like they're just starting up for the first time.

Certainly downstream. I'm hoping that if people get excited about just using Kafka and they're actually really like leveraging the events that they're emitting downstream from Debesium, then yeah, this is like, you need to really make it production ready or, you know, think about the different edge cases of what can happen, but I'm pretty upfront. It's like, Hey, like Debesium is a whole of the beast to itself, probably very specific expertise that I don't know too much about, but I can stand it up multiple times.

I've done that and we could set that up for you super, super easily. But I hear you that there's yeah. There's definitely edge cases around things going down and having delays in consumption, but I I don't touch that. Tobias Macey: Fair enough.

To that point of streaming being very beneficial, but also potentially very complex, one of the ways to make it more approachable is something akin to what you're building with TypeStream of make it friendlier APIs, things interfaces that are more familiar to application developers. But there is also that risk of once you adopt something and you integrate it tightly enough, then you're committed and you have to be able to operationalize it and manage it for all those edge cases.

And so for people who do come to TypeStream and say, oh, hey, great. I don't have to figure out Kafka anymore. What are some of the ways that you manage some of that expectation setting of this is the easy on ramp, but the, the on ramp goes much farther than maybe as far as you're willing to go. And so some of those ways of mitigating the expectations and, making sure that people who do go down that path, whether it's TypeStream or some other interface, are appropriately committed to the progressive level of complexity that they're going to have to deal with?

Jevin Maltais: Yeah. So I think the phrase that the way that we're framing the products, we're calling it Terraform for Kafka streams. And so it's pretty narrow in terms of what we want to enable. We want to enable you to not have to build a microservice for every single thing.

We want you to leverage the data that you already have in Kafka or get the data into Kafka. So you have that network effect of being able to leverage a single source of truth and not having to move data around all the time. And so we really help you on ramp into that. And we have some of the niceties for, you know, beginners ways on ramp, including dBzium and stuff.

But I think when you get to a certain scale, some of these things are just gonna have to manage yourself. Debesium, you should have your own instance of that's connected to the database. So, I mean, Shopify got started. They just connected Debesium to everything is what I understand when I was interviewing there.

And they just like exposed everything. It's like, cool. Now we have everything there. And then from there they had to learn how to be able to work through all the edge cases and just have a team that's dedicated on just the data, data emission.

So I think as people get as teams, as they get more experience or they get larger and they start to have some certain bottlenecks, there's certain things that you should be able to take over yourself. TypeStream, again, because the abstraction is pretty straightforward, you can go and build your own Kafka streams that can connect two different topics, say within the TypeStream pipeline, if you want. Again, because it's all perfectly visible and observable, see how that works.

But that's probably with most things, whereas you can start easy. And then once you My son, he's trying to learn to code. I asked a guy who's a coworking space. I'm a heavy AI guy.

He was like, How would you recommend it? He's like, you know, Do you start with Claude code and just like build these things? Or do you just like learn the old HTML CSS stuff ground up? And you can go either way.

So my son's like, Yeah, I wanna do AI to be able to go and rebuild ArcRaters. I'm like, cool, you can try that. And my hope is that if, as he looks at this abstraction and he's like, okay, now I have to figure out how to integrate like an AI library for these little guys moving, he'll start to go deeper and get some experience into these kind of deeper nuanced things. So that's what comes to mind here is like start with this abstraction.

And then as you need to get more experience or you're hitting edge cases, gain more experience with the lower layers down so you can see all the different nuance there. Tobias Macey: One of the other details that is very useful when you are building when any of these data focused layers is being able to understand how do I appropriately model these systems, what are the proper structures for the objects, how much information do I need to pack into this particular event to make sure that I have it available, how do I manage things like schema evolution, And what are the cases where maybe I'm putting too much information in and I should actually split that into two separate topic streams?

I'm just curious how you're seeing people tackle some of those questions, particularly from an evolutionary context of, I just wanna get started. I'll just jam everything in there. Oh, shoot. That blew up, and now I need to figure out how to decompose it a little bit and just some of those data modeling questions about how to appropriately manage the semantics and expectations about those discrete events?

Jevin Maltais: Sure. First, I know I mentioned before, but like get that schema registry because you talked about an example with Zapier is like, no one will ever try to be excited about trying to add types to their events later on. Implement that right away and do your marshaling if you can still convince people to use it after that heavy lift. Thankfully, the schema registry does have a bunch of niceties in there, including forward and reverse compatible for your evolution of your events.

And so that helps quite a bit where as soon as you're emitting your events upstream, you don't have to necessarily worry about having your downstream consumers immediately able to adapt and consume that if you've thought through the evolution piece there. So that's like a major, major advantage by having the schema registry is you can do some of this evolution and have your two systems decoupled from each other. Hopefully they're not. Or the other thing too with Kafka is you can have it set up where as soon as there's an error consuming, just stop consuming from the pipeline.

That buys you time, whether by design or not, to be able to go and do your upgrade and then set it off again and it'll be able to reconsume again. That's where I think there's just so much power in this decoupling, especially when you have this type safety. But as as as yours, you know, yours you mentioned, like, how do we think about this as we go? I'm a pretty scrappy guy.

So I'm kinda like, just have to do do your do your day design or your half day design, trying to think through this data model, especially when we talk about windowing and having to combine data across topics. So you have to join a user and an organization, maybe with a file upload, you have to sync through what are the race conditions there? How do I join all that data up? Which you can still do all in Kafka streams and stuff.

But it's like most things, sometimes we just have to put it out there and try. But before we put it out there, is this a one way we have to think to ourselves, is this a one way door, two way door to be able to see how much risk we we have when we actually go deploy this thing. And Tobias Macey: so for people who are adopting TypeStream, what does a typical workflow look like and some of who is the typical persona who actually goes and says, hey. That thing solves the problem that I'm trying to deal with right now and brings it in, and then just the overall process of getting it integrated and deployed?

Jevin Maltais: Yeah. So the primary group who really get us and it resonates with are people who are using Kafka and already have an idea of what Kafka streams are. Two major benefits for them is they can just have it config as code, makes it a lot easier for them to deploy and test their workflows. And the second thing is when those who are really kind of in this enabling role, where they're trying to help their team move faster, where the alternative is going to have to build like a bunch of infrastructure, like build a microservice here and a database to support it, or a Redis instance, you know, whatever.

Or it's like a high risk thing where they want to explore a new database because that's maybe a lot to stand up. Those are the people who really resonates with. Folks who are more product engineers like myself, unless they've really encountered the same problems that we've described, they just are generally fine with having a SQS queue or using Redis or having a number of different databases around. But I think the primary people are those who are like, yeah, I've used Kafka, but maybe, you know, or they're hearing today, like I had no idea.

It can do all those crazy things of like these materialized tables and replays so I can build something two months down the road and it can save all, have all my data there. Those are really the people I think are benefit the most from it. We have the Debussy and thing built in, which I think is pretty exciting for a lot of folks, but to stand it up, it's really straightforward. We can deploy with or without the schema registry.

We need it, it needs to be there, but we can deploy it with you. If you have a schema registry in Kafka, you literally just connect TypeStream to that and that's really it. We can manage the processes within our own We use Kafka to manage all the processes. We use a compacted topic if you're really geeking out on the Kafka stuff.

And so, yeah, it's really straightforward to be able to deploy. Tobias Macey: And for teams who you are working with and who bring in TypeStream to solve their data replication challenges, particularly if they don't necessarily have a dedicated data team to work with, what what are some of the most interesting or innovative or unexpected ways that you've seen it used? Jevin Maltais: It's I think people want to try out new new data stores. So folks who are wanting to do now analytics or or really look at some stuff at scale, and maybe they're sitting in a Postgres database and they're like, I really want to do ClickHouse.

It's like, great. You can stand up TypeStream. You can just connect to Bezium up right to your Postgres, stand up your ClickHouse, connect, type in the credentials for each and type stream and press play, and it'll just stream it all for you. To be able to do that, by hand, you're deploying There's lots that you're deploying to be able to do that.

And so that those are some of the that's probably the most exciting things. People are like, I wanna explore this interesting data store that I could really leverage and speed things up by 10 x to make that really seamless, really easy to do. It's pretty exciting. Tobias Macey: And as you have been building this product and working on Kafka and getting to know its internals more closely and building a product on top of it, what are some of the most interesting or unexpected challenging lessons that you've learned in the process?

Jevin Maltais: I think the question that keeps coming to mind is why is it so hard to deploy this thing? That's that's just such a It's a shame really. And maybe it's by design so that we all go and buy the Confluence mega price, buy the Confluence package for very expensive. But it is a shame that it's so hard to be able to go and deploy Kafka, the schema registry, all of this with high availability in your cloud provider of choice is the thing that's really the thing that keeps coming back to is like, man, I've Or talking to people and just the I'm trying to be always look at first principles for products and be able to go and figure out the best way to use the right tools you have available.

But when I'm talking to people, just like, Man, that's a lot to be able to go and deploy. I already have Redis. I've already got Postgres. Why don't I just write all my events as Postgres rows?

You're like, yeah, I get it. And I totally do. It's a heavy lift to get to a place, but I think So that's one. And the second is like, yeah, just how underutilized I think Kafka is or all the functionality on top of Kafka that people are just not using, even though they have Kafkas already fully available.

And that's also disappointing because it's like, you have all of this that you can build on top of, but yet we just haven't pushed through to find out what those things are because we're, especially like us who've been doing this for more than fifteen, twenty years is like this whole asynchronous event driven architecture is not something that we're very, you know, that grew up with necessarily, right? We have our data store and if you wanna do some sort of asynchronous events, like, you know, it's still new in the past fifteen years for us.

So these are it's just a different way of thinking, being event driven versus just looking at synchronous table. Tobias Macey: One of the common themes that I've been seeing, particularly with this whole boom of foundation model AI providers is that anytime you're building on top of something that you don't fully control, there is that element of platform risk where if everything changes out from under you, you don't really have a lot of recourse. And so given the fact that we do have so many different API compatible implementations that are at least intended to be drop in replacements for Kafka, I'm curious how you're thinking about that from a sort of operational security standpoint for your own work.

Jevin Maltais: Do you are we talking about, like, the AI the AI world? Tobias Macey: I'm thinking of I'm thinking in terms of what you're building where you're currently deploying on top of Kafka specifically. But there are other alternatives that you could incorporate such as the Pulsar with their Kafka protocol layer or Red Panda that is intended to be a drop in replacement or AutoMQ where Right. You're not vendor locked as it were despite the fact that they're open sourced to Kafka specifically.

Jevin Maltais: Yeah. Yeah. The the major advantage that we have within this realm, I think, is that we, there's so many good like Kafka connectors that we can go and drop backups into S3 of like all, everything that we have, if we really want to, right? Or I want to now move my Kafka, like all my Kafka topics a different distribution of it or into a different kinesis or something.

Well, there's compatibility layers strictly on by connecting topics together where I can now progressively have a new consumer, having whatever it is, like we have Pulsar, I've never tried it, but presumably Pulsar has a way where you can subscribe to Kafka topics or you use a Kafka connector and you can start progressively dropping all of your data, like having it as a second consumer effectively for it, where now you have like the second cluster that's a full replica of the initial one, which I think is really, which is good.

So I think because there's already so many compatibility layers, not just like magic connection of these, like of a replication of a cluster, we can just move the data and have another consumer that will translate it to a different place. Or, you know, you can write a very thin layer that will just, you know, read one message, copy it to your new place that you wanna store it yourself. So I'm not I'm not super concerned about vendor lock in that way because it's so easy for me to get my data and move it around.

Tobias Macey: And so for people who are dealing with these challenges of data replication or data freshness or latencies, what are the cases where you would advocate against using TypeStream and maybe just go whole hog into Kafka directly? Jevin Maltais: Right. We wouldn't solve the problem. So usually, you know, Kafka streams will be one of the fastest ways latency wise because it's going to be Kafka native, it's compiled and it's designed to be able to work in between to do the transformations natively, like in line in the pipeline.

So moving from, we're not losing anything with Kafka because we're just compiling down into that same library. There's not a ton in terms of like latency or performance that you would gain. It's more if you have a very particular type of transformation that I think you want to be able to write and you really need to have kind of that Java accessing attributes and doing some sort of weird tangly thing that maybe we don't support in native TypeStream node that maybe you want to go and write your own Kafka stream process that will go and consume directly off a topic.

Performance wise though, where TypeStream doesn't you'll be fully the same latency you'd get by writing this natively in Java. But you might have to write something in Java if if TypeStream doesn't provide, like, a node or the flexibility that you need in terms of the transformation. Likely, that'd be the reason. Tobias Macey: And as you continue to build and invest in TypeStream and continue to evolve along with Kafka and the broader ecosystem around that?

What are some of the things you have planned for the near to medium term or any particular projects or capabilities you're excited to explore? Jevin Maltais: Yeah. Wanna try it with more with more of these Kafka compatible distributions to see how they perform, I think. Because if one of the advantages, if you have with, I think this thin abstraction that we have on top of it is whether you deploy it with vanilla Kafka or you go deploy with AutoMQ or Memphis or whatever, it should be entirely portable.

And so I want to be able to ensure people that regardless of where you have as your Kafka deployment, that TypeStream should be good to work for you. And the other thing is, yeah, we're looking for more people to try it out. It doesn't have a ton of users right now, but if you think there's a, if someone listening thinks they have a particular use case where they're like, I want to move data between these two data stores, it's a real pain. Or like, I want to be able to get my data and use Kafka as my source of truth.

And I'd love just to have like a REST endpoint to offer to our engineers to be able to query for the latest materialized version. I'd love to talk to them and see if this would be a fit for you. It's it's BSL. So it's like you can go and take a look at the source code yourself.

You can deploy it, you know, for fun, try it out. But we think it's so cool. We just don't want Confluent going, just dragging all that code into their into their deployment, not giving us anything. So Tobias Macey: Are there any other aspects of the work that you're doing on TypeStream specifically or applications for Kafka and the streaming systems, particularly with a product oriented focus that we didn't discuss yet that you'd like to cover before we close out the show?

Jevin Maltais: No. I don't think so. I think those we saw some that's some pretty common yeah. Probably common use cases, maybe some unique ones.

Yeah. No. Not really. I think that's kind of it.

Tobias Macey: For anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the last question, I'd like to get your perspective on what you see as being the biggest gap in the tooling technology or training that's available for data and AI management today. Jevin Maltais: Oh, great. Okay.

Yeah. You can reach me, jevin @typestream.io, or you can check out typestream.io if you wanna learn more.

And the biggest gap, I'm most concerned about kind of data security with the agentic stuff. So I do lots of work with legal tech in my fractional CTO work and passing your data, even though you have like zero data retention to these AI providers is still very concerning for our clients who are working on multimillion or billion dollar deals. And so how we're going to be able to manage that where, you know, just like that whole self hosted idea, we're not going to help self host a model, but like, how do we have that same assurance of like, our data is not going somewhere else, but still running frontier models is like one of the top things that I think we're going to really need to be thinking about.

And then the other thing I think about a lot is what we opened with is how where is going to be the competitive edge for companies with their data in the age of AI. A lot of people are just using wrappers around those frontier models, but I think the real advantage is going to be when we can provide a lot more context and offer up our context of the data that we manage as data and product engineers, to the AI. And I think there's still quite a bit of work on tooling around that and and kind of the security and the assurances of how we're managing that data, for privacy.

That's we still need to work through. Tobias Macey: Alright. Well, thank you very much for taking the time today to join me and share the work that you're doing on TypeStream and just your overall efforts of helping product teams gain more capability out of things like Kafka and these complex data focused tools to solve real world problems. I appreciate all the time and effort that you're putting into that, and I hope you enjoy the rest of your day.

Yeah. Thanks a lot, Tobias. Same to you. Thank you for listening, and don't forget to check out our other shows.

Podcast.net covers the Python language, its community, and the innovative ways it is being used. And the AI Engineering podcast is your guide to the fast moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.

And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com with your story. Just to help other people find the show, please leave a review on Apple podcasts and tell your friends and coworkers.