Managing a Customer Analytics Platform with Airflow at Skimlinks

Julian Larelld: Well, what I'm seeing in the industry is the rise of data contracts. Uh, I think everyone is really, uh, talking about it, but not many, perhaps, are implementing it.

Narrator: You're listening to the Data flowcast, the podcast about Apache Airflow and the world of data and AI around it. Let's get into it.

Kenton Danis: Okay. Hi everyone. Welcome to the dataflow cast. I'm your host, Kenton Danis, and today I'm joined by Julian Larelld, who is director of data engineering at scimlings. Jo. Julian, how are you?

Julian Larelld: Hey Kenton, it's great to meet you and it's a pleasure to be here in the podcast today. So thanks for the invite.

Kenton Danis: Yeah, of course. It's great to have you on. Well, I'm excited to get into it for our listeners. For today's podcast, we're going to talk about how skimlinks uses Airflow to power external analytics for their clients. But before we get into all of the details, I want to start with a little bit of background. So first maybe we'll focus on your company. Can you tell me more about what Skimlinks does?

Julian Larelld: Of course, yes. Well, Skimlinks basically operates in the affiliate marketing industry. What we do is basically we help publishers, websites, uh, content creators, anyone who has an audience to monetize their content. They're writing about basic products in the marketplaces. Uh, so yeah, you probably this week you might have already clicked on a Skimlinks link without even noticing. But yeah, we, we help you set up everything with just like two clicks to basically monetize your content. Why do I say that? Uh, we are a network aggregator. Because publishers, they have their own needs and goals. So there are multiple affiliate networks out there, more than a hundred percent perhaps. But uh, with gamelinks, you take that effort all the way and then with just one contract and then one point portal, you're able to go and get the best rates across the entire industry in a very easy way.

Kenton Danis: Okay, yeah, that makes sense. I think there's so much that goes on behind the hood of ads and marketing online that people just don't even realize because it all works pretty seamlessly. But yeah, so that's very cool that you guys are a big part of that. And then, uh, tell me a little bit about your team and what you're responsible for. Mhm.

Julian Larelld: Well, I'm, uh, a director of the data engineering here at Schemlynx and I manage a team of seven data engineers. We basically own what we call the data platform. And what we are responsible is basically we provide with reporting portal to our clients, external clients which are publishers. And this reporting platform basically is like a BI tool that we have developed. Right. So we give them um, metrics around the performance on um, the content that they're writing about, but also the content they should be writing about to uplift increase that revenue.

Kenton Danis: Okay, interesting. So that sounds like a big part of your business and I understand that a part of your focus is powering the analytics platform that your publisher clients use. Can you tell me a little bit more about how that works?

Julian Larelld: Mhm, sure. So this platform then that we have built serves around 2,000 weekly publisher users that they access it through our skimless hub, what we call it the data platform. Perhaps we can split it into like a few different phases. Uh starts with the ingestion of the data. We have real time sources, you know the performance marketing metrics such as page impressions, clicks and in this case because we operate in the affiliate marketing it will be commissions. So every time someone clicks on a link and end up buying that product, we pass that commission to the publisher after all. So those are basically the metrics that we evaluate and then we made certain calculations. But all that data, uh, we have it in transactional database and then come from different applications and we sync everything to bigquery. Once the data is in bigquery then uh, everything is orchestrated by airflow and this is where it comes. Airflow is our main single tool that we use to orchestrate or our data processes or our data needs. And um, well within BigQuery we have like the classic few layers, raw silver and gold and we generate this report, we have different data marts and then we end up exposing it into a separate database called Apache Truage. This database is the one that the publishers would eventually underneath query in the reporting hub and it gives us the sub second latency uh, that they are after when it comes to uh, a BI tool up to like today's standards. With that because we have so much for our external customers, the reports are mostly the same. So we reutilize most of it to create internal analytics as well. So we count with 100 internal customers from the Department of Marketing, Finance, operations managers, anyone from our internal business that wants to dig into the data. So yeah, we provide them with BigQuery Access or we have Looker as our internal BI tool as well.

Kenton Danis: Okay, yeah, very interesting. So it sounds like airflow is very central to both the product that your company provides and of course for internal analytics as well. I think 100 internal customers sounds very like a lot to me. Well I would love to dig in a little bit deeper into kind of your use of Airflow and how it's helping you power your analytics platform. I imagine that there's a big component of that that's you know, sort of on demand or needs to be updated based on when particular customer data is being, is coming in. Tell me about how you're you know, scheduling dags. Are you using sensors or. Yeah, something like that.

Julian Larelld: As I mentioned, Airflow is the core of our platform. Everything that we orchestrate goes through Airflow and we as uh, data platform provide perhaps like the best practices on how to utilize it and we make sure that it's up and running and task don't get delayed, they don't get scheduled or they don't get ah, like outside the uh, defined SLAs, let's say. But yeah, in a nutshell we have uh, a few different products but we basically use Airflow, uh, as an orchestrator. But as I mentioned all our Data sits in BigQuery and therefore all our business logic sits in SQL. Right. So with Airflow what we mostly use are like bigquery operators to kind of like define that lineage from one task after the other. We do some other complex stuff. Like for example there is uh, a uh, service that we provide to certain publishers, the largest ones actually that we provide them with CSV, Avro parquet, just raw file exports with the raw data in case they need to ingest it to the warehouse. And Airflow plays a critical part in that because that is quite a complex orchestration that we need to do across a few dozens of them.

Kenton Danis: Okay, yeah, interesting. And then so I understand that with that you've also shifted from the sensor based approach to using Airflow data sets which are now airflow assets and airflow 3. What does that look like?

Julian Larelld: Yes. So we've been running Airflow for many many years since Airflow one. Right. So I'm always quite excited when a new version comes out and there's that need for doing the uh, migration and um, um, tackle deprecations. But these new features like dataset or assets, now they're called, has been like uh, an integral part uh over the last few years. We've been historically using sensors as a way of like orchestrating dependencies from different DAGs, particularly task sensors. So making sure that only a uh, particular dag, a particular DAG airflow process would only run when the data is available. Right. So uh, external task sensor would normally be marked as finished and then only then the second DAG will kick in, uh, as long as the schedule, uh, had met already. But we've seen across the industry that within data, uh, things were moving from a cron base from a schedule to more like event driven approach. Right. And we knew that we wanted to be part of that because nobody wants to have like um, a dead time in between tasks or you don't want to constantly be waiting, poking for data to see has it arrived yet, has it arrived yet? Until it finally does. All that created like worker overload from a cost perspective, you'll be utilizing workers for a set amount of time that were not actually doing much labor. But then from another perspective is the amount of uh, tasks that the airflow scheduler will have to organize. Quite often, you know, we have the different types of poke and reschedule to release the worker slot usage, but still it still needs to go to the scheduler. So that migration to the dataset event driven approach was critical to help streamline actually what is uh, a smooth transition from one process to the other as soon as the data becomes available. But also I think the added bonus was this lineage that Airflow provides in the ui. We were able after that migration to be, to see within like a single panel how everything connects together. Right?

Kenton Danis: Yeah, absolutely. That's such a huge benefit of the data sets or assets feature is being able to get that visibility into, yeah, into uh, all of the connections between your dags. So yeah, that's really cool.

Narrator: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for, for data engineering teams. From batch pipelines to training models and wiring up agents, Astro handles the infrastructure upgrades and scaling so you can focus on building. Check it out at astronomer IO.

Kenton Danis: I guess, uh, looking ahead, you have mentioned, you know, on a different topic, plans to kind of open up airflow usage to other engineers across the company so that they can contribute in some way. What is your vision there and kind of how are you going about implementing that?

Julian Larelld: Mhm. That's a project that I'm quite excited about. Yeah, my vision and I think it's starting to become perhaps uh, a bit of a standard across the industry when you have like uh, a central hub of data platform providing services to help other engineers achieve their tasks. The idea is to break down the different silos of other teams across the world running either their own airflow instances or run the jobs or crunch jobs. You don't want to dig in much, but yeah, they're running locally in the laptop so effectively we want to become like the center of excellence. And when it comes to orchestrating something that is reliable, it can run on time and you will have visibility on like alerting notification when something is not quite right by us being the experts in airflow and um, providing the best practice in a safe environment because security is quite important nowadays, uh, with the rise of AI agents and so on. Yeah, I'm hoping to be there within this year and um, provide my fellow engineers the best experience they can possibly get.

Kenton Danis: Okay. Yeah, very cool. This is definitely something that we hear about a lot. And Astronomer has kind of our own take within our platform on how we can open up DAG or pipeline creation to folks who might not know airflow very well or might not know Python. How are you handling that aspect of it? So you want more people to be able to contribute, but as you mentioned, of course you have to provide some guardrails uh, so that best practices are followed. What does that look like in practice for the implementation?

Julian Larelld: Exactly? Well, Astronomer airflow is very tempting, you know, because airflow has been, everyone has been asking for multi m tenancy for years and Astronomer came one day and said look guys, we did it. Here it is, start uh, using it. So it's really tempting and it's something that you don't have to reinvent as well. Right. It's off the shelf available and of course providing these security guard delays. In terms of what does it look like? I guess it's what I was mentioning first, a common standard across like DAG testing, providing SLA or alerts when those are missed, monitoring to make sure that things are running properly. Airflow gives you the possibility to create your own plugins, um, provide a little dashboard if you wanted that is custom to your business. Making sure that each team has a green light when it comes to checking that the data has run, but not only has run, that it has actually the data that they're expecting. Providing baseline for monitoring in terms of standard deviations when it comes to broken or sums over the days or like um, a particular data assets. I think all of that contributes to them being able to focus on what it matters. Right on delivering business value for the projects without worrying that much about the data infrastructure or the cumbersome that airflow may entail.

Kenton Danis: Are there any main challenges that you're anticipating there other than we've kind of talked about the best practices aspect already, but as you look forward to implementing this.

Julian Larelld: Yeah, I think one of the challenges that I've uh, been having over the years is trying to convince the business that we need to upgrade Airflow. It's always such a, not a tedious task, but it requires effort and the business might not always see the benefits of doing so. Of course, like us, uh, engineers, we always get excited about new features, making things in a more elegant way. But for the business, it might be a bit hard to say, yes, let's prioritize doing this and help you achieve what you're after.

Kenton Danis: Yeah, sure, totally. That's always a challenge for data teams. Well, I guess then on a slightly different topic, so you mentioned the DAG factory. I had incorrectly assumed that was part of this platform you were talking about. But is that a different project that you're working on that you could share some details about?

Julian Larelld: Yeah, yeah, of course, yeah. Quite recently, actually, over the last six months, we were able to migrate, uh, quite a longstanding project of us. So we have a data product called the Data Pipe, which, uh, effectively we export raw data to our largest publishers to ingest the data into the warehouse if they need to. Ah, rather than giving them the insights on a reporting layer, we give them the work data so then they can analyze it any way they want. Now, this project coming from Airflow 1, uh, has been long standing and it serves that particular purpose right many, many years ago. But over time, businesses started to rise, requirements changed, and we, moreover, we started to utilize this way of exporting data to, uh, share data within our business as well. So not only it kind of like, it has outgrown itself to like, different needs. So that led us to perhaps not developing it in the most elegant way because it was always in a hurry. We didn't need it by yesterday. And over the years we just kept adding layers that weren't ideal and it wasn't a robust of a system as, um, we would have liked in terms of what we did. So, yeah, but basically it was a single class Python class, doing absolutely everything, generating the dag. If we were to split it by phases, what we did was break it down, break that class down into what we call the single responsibility. Uh, so we create one class for each responsibility, doing first the configuration of it. Who is this intending to be sent to what file format they want? The second one will be actually building the query that they wanted, querying our BigQuery database. The third stage would be actually extracting that data out to the file format that they were after and finally would be sending that data, um, to the cloud provider that they are compatible, they're working with. So we broke that down into multiple classes. That helped us make it more reliable in terms of testing, but also making sure that every section was kind of independent and, um, building more of a robot system on top of this. Because we were used to changing Python files for configuration every time we had to onboard a new client, it was quite risky to do so. So we decided to create our own dagfactory and provided with a YAML format for the configuration. So now when we need to import a new publisher, we basically go to the YAML and, uh, we just add the requirements that we have there, then the name, then the export format and the particular query format that they want. Yeah, it's super easy to integrate. Not only we have the YAML, but on top of that we added a pyrantic layer. So that helps us create something way more stable, because the YAML might be valid, for example, but the key might be mistaken or the value might be incorrect. And you would only notice that on execution time. So one of the main benefits that we see with Pedantic is that the DAG would not be able to render, otherwise it would fail. So, um, I think that was a critical, perhaps improvement that we've done for adding rocks. Nice.

Kenton Danis: Interesting. And so, so the Pedantic layer is kind of like a test that the DAG is going through before you deploy it so that you're catching any failures before it actually runs.

Julian Larelld: Exactly, yes.

Kenton Danis: Interesting. Yeah, that's very cool. I haven't heard of Pedantic being used in that way before. Yeah, that's great. And so, and overall, this project, it sounds like, was meant to kind of help standardize a lot of this repeatable work, as you're, like, onboarding clients into the platform. So I imagine it allows you to. Much faster.

Julian Larelld: Exactly, yes. And, um, it's something that we've been trying to do for years. You know, we had the design in our mind, but we were never, uh, able to prioritize it from a business need. But yeah, over the last six months, you know, the world has changed recently with the AI adoption. So I'm not saying that the AI did it, but it really helped us, guide us through making sure that the decisions that we were taking from a design perspective were sound, but also helping us writing tests, comprehensive tests across each class, and, uh, making sure that the decisions we were taking along the development were correct. Uh, honestly, it's a great help to brainstorm different ways of doing things that you were not thinking before.

Kenton Danis: Yeah, absolutely. That sounds like a great AI success story. And I'm curious, did that work pretty well out of the box for you for an airflow specific context.

Julian Larelld: I guess with AI we are all reinventing ourselves in our roles. There are multiple ways, multiple tools and multiple ways of doing it. You have uh, the ui, you have the client, you have uh, agents running in your own ide. So it was a project that was mostly like uh, a self discovery project as well to learn the ways of AI evaluating different models, uh, starting with definitions on the, on the project rules, adding skills of certain elements. So yeah, I guess it worked well both ways.

Kenton Danis: Okay. Yeah. Awesome. Well that's uh, really cool to hear about and thank you for going through those couple of different use cases. I always love talking to folks where airflow is really powering a huge part of their product. It's so cool to see airflow being used in that way. And yeah, you've got some really creative and interesting projects going on with that. I guess just to close us out, I always like to end with a standard final question per guest and that's what would you most like to see from the Airflow project in the near future?

Julian Larelld: Mhm. Well, I guess we talk about data sets, right? Sensors, data sets, uh, Asset Watchers, I think. Well, Asset Watchers is basically the evolution of data sets. Right. But for external data sources. I think what I'm seeing in the industry is the, the rise of data contracts. I think everyone is really uh, talking about it, but uh, not many perhaps are implementing it. So perhaps helping validate that, uh, Asset Watcher, assigning it with a data contract, making sure that the schema fits what we were expecting, uh, row counts or field types, that kind of thing. If we were able to validate at the time the Asset Watcher is triggered, before we start processing the data, that would be amazing.

Kenton Danis: Yeah, yeah, definitely. I can see where that would be super helpful. And I, I think if I remember correctly, the data contracts was included in the previous airflow survey as one of the features of that folks were requesting. I think it was pretty high up there. So I think, I think you're definitely not alone. As you said, it's def, it's a big focus for the industry right now.

Julian Larelld: So. Absolutely. I mean software engineers, uh, have it, you know, when they do API development, we are engineers as well. Right. In data. So why shouldn't be investing in data contracts?

Kenton Danis: Yeah, absolutely. Totally agree with you there. Well Julian, thank you so much again for coming on. It's been really great speaking with you. Just uh, to close this out. What is the best way for folks to get in touch with you.

Julian Larelld: LinkedIn is the best way. Julian Ralt I'm responsive there, so yeah, reach out. If you just want to talk about airflow, be glad to.

Kenton Danis: Great. Well we will put a link to that in the show notes. Thank you again. As I said, this has been a really great conversation. I have enjoyed speaking with you.

Julian Larelld: It was my pleasure. Thanks Kanten. See you soon.

Narrator: Thank you for listening to the Data flowcast. Check the show notes for links to everything mentioned today. If you're enjoying the podcast, subscribe so you never miss an episode and leave us a five star, uh, review.