Using Airflow for diverse client projects at Accion Labs

Speaker A: You have, uh, a brilliant, uh, LLM, but, uh, it's working with the stale or noisy, uh, data. The output will obviously fall apart. So, um, the POC is really about, uh, treating AI data pipelines with same rigor. We would apply to any production data pipeline.

Speaker B: You're listening to the Data flowcast, the podcast about Apache Airflow and the world of data, uh, and AI around it. We're excited to invite you to Orchestrate Everything, our free online conference on September 16th. Engineering leaders from Lyft, RAMP and Wix will share how they're shipping production AI workflows on Airflow plus a crash course for the new Astronomer Certified AI Orchestration Fundamentals exam. Attendees get a discount code for a free certification, a $150 value. Go to Astronomer Link, Data Flowcast OE link in the show notes. Now let's get into the show.

Speaker C: All right. Hi, everyone. Welcome to the Data Flow Cast. I'm your host, Kenton Danis, and today I'm joined by Chandan Gowda, who is a data engineer at Axion Labs. Chandan, how are you?

Speaker A: Hey, Gitan. So, I'm good and, uh, thanks for having me here. I'm really excited to be here and, uh, I have been a listener of this show quite a while and it feels a bit serious to actually be on the other side of the mic. Thank you for having, um, Me.

Speaker C: Yeah, that's great. Well, I love to hear that and it's so great to have you on. Uh, we'll go ahead and dive into it. So, for our listeners for today's podcast, we're going to talk about how Axiom Labs uses airflow across their client projects, uh, including a financial scoring use case and some early work on AI pipeline orchestration. So both some pretty cool topics. But, uh, before we get into all of that, I want to start with some background. Maybe you can tell me about the company you work for. What does Axion Labs do?

Speaker A: Sure. So Accion Labs is a technology consulting and services firm and we essentially help, uh, businesses to solve the complex IT and data challenges. You can think of us like a partner that comes in, understands the client pain points, and then design and build the solutions around the modern data and AI technologies. So we also work across, uh, industries like bfsi, healthcare, retail. So as we are consulting firm, the problems will vary, so which honestly keep things really interesting day by day. Get a new things to solve.

Speaker C: Yeah, very cool. I used to work in consulting myself, so I relate to the idea of being able to work on different projects and how that keeps Things Interesting. And so tell me a little bit more then about your role and the kinds of projects that you typically work on.

Speaker A: Yeah so uh, earlier like um I started as a data engineer so as you know the technologies will be changing and um, we will be evolving with the market. So currently I'm uh, uh associated with data itself but uh quite more occupied with the gen things. So you can call me as a gen data engineer for now. So my work sits right at uh the intersection of uh, like data engineering and AI where I have to uh build some uh pipelines in one week and uh in another week I will be working on the rack based AI agents like uh developing some internal tools for uh, our internal purposes and like uh, it'll be like uh, all together it's like orchestration plus AI. So I need to make sure that uh data moves reliably and model gets fed with the right inputs and everything runs on schedule without like uh someone babysitting it. So so everything needs to be automated. So everyone are fascinating. And yeah I'm also thinking to automate all the things uh, which is uh absolutely possible with the AI today. So yeah that's pretty much about my role.

Speaker C: Yeah, that's great. That sounds yeah very interesting. And where does airflow typically fit into that? Either with the data engineering piece or maybe the AI engineering piece. I guess we'll get a little bit more into that uh later in the interview. But yeah high level uh, where are you typically using airflow with your client work?

Speaker A: Airflow is um, severe for us in orchestration things uh with the data thing. So um, while um we are doing the data pipelining at the time, airflow will uh help us with orchestrating, bringing the data and we um, can use the triggers to fetch the data, uh do some transformations and put it back. So there uh, we use uh airflow for now. We will talk about this uh, uh when I specifically explain about the projects.

Speaker C: Yeah absolutely. And I'm curious, is airflow something that you are typically recommending or do your clients come in asking for it or both Maybe

Speaker A: I can say like uh it's a mix. So honestly some clients come uh who are already using the airflow and they wanted to scale it up and um, there are others who are like just wanted to evaluate the options and we will assess what is their requirements and we will uh suggest them what fits the best. But in most cases once we lay out the options so airflow will uh win obviously. So especially for the clients who need a fine grained control over their pipeline Logic and they don't want to be locked in into a single cloud provider's ecosystem. So at that time uh, airflow is a savior. So we recommend that uh, like uh, it's like airflow is not a new thing, you know. Right. So it was um, created by the uh, like uh, Maxima Bochiman, uh who is a founder uh like at Airbnb. And it is obviously open source and it has around 30 millions of downloads uh per month. So which is like um, it's easy to convince the clients if they really don't know what is airflow and they uh, wanted to integrate it into their systems. So yeah, um, when we show all of these details and we shows uh, how it solves the pain points, client will be uh, obviously happy with the solution we will be implementing with the airflow.

Speaker C: Yeah, totally makes sense. I think the popularity of the project in some ways speaks for itself. And so you mentioned having you know, the flexibility of not being locked into a certain cloud, uh, as well as you know, preferring having the control and sort of lower overhead compared to maybe a cloud native tool. Tell me a little bit more about that kind of where, where do you think airflow really stands out in that area over other orchestrators?

Speaker A: Like we can say like if we speak about the AWS step functions or um, Azure Data Frame Factory, they are all great only if they are in the cloud. So we cannot um, pull them in locally and we cannot uh, do the orchestration locally. But um, the moment a client says we might move to a different cloud in uh, two years or if we wanted to uh, go for the uh, other technologies or other clouds. So at that time um, uh, we will be in trouble. So shifting the things from uh, let's say from uh, Microsoft to aws, there is a lot of pain points. It requires a huge efforts and um, we have to understand the infrastructure and all. But the magic comes here. So when we use the air for airflow, so it's like infrastructure agnostic so you can containerize it and you can deploy it on Kubernetes or even in the vm, it just works. We just need to contain it and we can deploy it anywhere. So this gives us flexibility to use airflow. Even we are migrating to a different cloud or ecosystems. It'll be easy.

Speaker C: Yeah, totally makes sense and certainly consistent with what we've heard from uh, our own customer base at Astronomer for why people choose Airflow. Uh yeah, very cool. So I would love to then dig into a couple of the specific use cases that you've implemented with your clients uh, that are you know heavily use airflow as the orchestrator. Uh one that I know you've worked on recently is a loan eligibility scoring use case. So maybe you can start by telling me a little bit more about that. What was that project about?

Speaker A: Sure. So this one was uh, one of the most uh concrete uh airflow use case I worked on. So basically the client was a uh financial services company. They needed uh escort loan applicants for eligibility in the real time. So the challenge was that um, the new ah applicant data was arriving continuously. It's like a streaming data so it was coming from uh different multiple sources and um, the scoring models were uh downstream. So we need to manage the trigger. So even if we uh fail to trigger the pipelines within the specified times we would lost the data. So at that time we, we needed to use the airflow. So uh, airflow before airflow we were um, using um manually. We were doing it manually. So we have to set some cron jobs and uh, the. It's like a fragile thing, it can break anytime. So uh, then we uh thought of airflow and we uh got to know about like if we uh use the airflow, how flexible it is and how convenient it can be. So we replace the entire stack with a clean airflow dag. So that handles the ingestion, transformation, model inference and also the result delivery completely from end to end. So everything is managed by the airflow. So by this um, we we reduce the efforts nearly around 25%. So we came from um, 100 to 25% efforts reduction. So this, this was a very uh, what we can say, great change. That uh client was very happy with this and um, from that onwards we don't want to get up in the midnight and check whether the pipeline is uh working or not. It has failed, something has broken. It is like a pain point for even our developers or supporting engineers. So airflow helped us in that point and uh, from that downwards we encountered very minimal um bricks and uh, everything was smooth like a butterfly.

Speaker C: Yeah, that's great. What a great airflow success story. Uh then I'm curious based on the use case how are you scheduling these pipelines? Are you doing more event driven so that when eligibility needs to be determined that the pipeline is actually running or is it scheduled? How does that work?

Speaker A: Great question. Um this is where like uh, I can say airflow really shines. So we use the combination uh approach. The primary trigger was file sensor based. So we have a component like um Airflow's Flow file sensor so or uh, uh we have uh s3 file sensor poles. So any new data arriving at a specific location the moment the um, um batch file lands, the sensor task goes green and um, the rest of the DAC kicks off automatically. So we also had a scheduled fallback also if there is no new data arrived by some time so there must be some problem. So the sensor could not pick it up. So at that time we had to uh, manage the backup also. So we had a scheduled backup which would run for a certain time and um, if there is something in the queue it could pick up. And um. It's like both. It's not like purely event one or it's like uh, not morely like scheduled one also it's like combination uh but uh, I can say the sensor based trigger was handling about 90 um percent of our runs. So it was successful.

Speaker C: Yeah, that's great. And I think testament to the scheduling flexibility of airflow that you're able to manage even the backup solution pretty easily. It sounds like. Yeah that's really cool. Once that DAG or maybe multiple dags kick off, what does the end to end flow look like within airflow? So once that data arrives all the way to you know you've got those eligibility scores landing back in the target system. What's happening in the middle there?

Speaker A: It's like a pretty clean pipeline. Um so once you see it laid out. So step one is ingestion. So airflow pulls the raw applicant records from the source system into our data lake. And um, after pulling that step uh two starts. So step two is uh like a validation and transformations. So we check for the nulls, duplicates and anything that uh requires attention and we will apply our business rules and join in in reference data. Like uh, we will be sourcing some um uh third party data also so like uh credit bureau scores uh for passing on to the model for scorings. So that things on all we were doing the step two and uh step three was uh like model um inference tanks. So this was uh calling our scoring models and which runs in a separate containerized environment and it passes back the eligibility scores and the supporting features which we require for assessing the candidate. So after this has been done the step four step four was like um uh writing back the scores into the client's loan management system. So we have done the scoring, we have done the analysis and we need to report it back to the client. So we have to uh give him in database. So we had a database where uh, we were appending the Results. Um, so throughout all this airflow is uh tracking the task states and it has been retrying the failed task automatically and if it is also sending the alerts if uh again something was breaking. So I can say that airflow sensors are like special uh operators that uh pause the DAG until certain condition is met. So they were like um, mostly used for event driven triggers. And uh, we also had some poke um interval and uh timeout on sensors towards some of the zombie tasks so that um, if a task ah sits in a running state indefinitely it would cause other tasks to be in the queues. So we had set some timeout so that um, if uh something is uh running for infinite time it used to um poke mode and it used to free up that uh worker loads from that checks. So yeah, like this ah our whole flow looks.

Speaker C: Yeah, yeah, that's great. Um, what a cool use case for airflow.

Speaker B: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams. From batch pipelines to training models and wiring up agents. Astro handles the infrastructure upgrades and scaling so you can focus on building. Check it out at Astronomer IO

Speaker C: Uh, okay, well changing topics a little bit to another use case I know you've implemented. I mentioned at the beginning that we would talk about uh using airflow for AI orchestration. Usually very hot topic right now uh, that we're seeing more and more of. And I understand you've been working on a POC around this. Maybe first tell me what is the problem that you're trying to solve here?

Speaker A: Yeah, this is the one I'm most uh excited to talk about right now. So the problem statement is like um, as we are building more on gen and uh rag based application internally and some of them for the clients also. So we realize that the data fading into these models is just as critical as model itself. See uh, model is you can think as a baby so you what you teach the baby is what the baby learns. So if you feed him good data he will uh, speak good and he will be good. If you have teach uh him the bad things, he will uh learn the bad things and they will uh do the bad things. So that we realize while we are doing the POC and you have any like uh, brilliant LLM but it's uh, it's working with the stale or noisy uh data. The out the output will obviously fall apart. So um, the POC is really about uh treating AI data pipelines with same rigor we had applied to any production data pipeline and we used airflow as the orchestrator to uh, whatever the data we are getting, we were not directly feeding it to the model or even we are not training with the noisy data or what we get directly. We used airflow, we did some transformations and then we put that data to the AI model so that it works as expected.

Speaker C: Yeah, that's great. I love that you mentioned uh, treating these pipelines like you would any other pipeline. This is something that my team at Astronomer has been talking about a lot recently. Just the importance of orchestration in general for the same reasons that it's important for ETL pipelines when you're dealing with AI pipelines. Uh, and then so I understand that part of the goal as you mentioned is, is controlling the data that gets fed into the models. Tell me more about what role airflow plays with that specifically.

Speaker A: Okay, you can think of it this way. So in a Rack pipeline you have a knowledge base that needs to stay fresh. So knowledge base in the sense, uh, the data, the data lake, you can um, imagine. So uh, it has to be fresh and the documents get updated. Like um, the new ones will be added or the old ones will get irrelevant. And also there are some cases where we need to update the documents. So whatever we do um, like the data keeps changing and uh, the models should be updated with the updated data. So we are uh, we cannot do this uh, without like uh, what we can say without orchestration tools. We cannot refresh um all the times we cannot refresh the knowledge base. And even the refreshing thing is uh, very expensive. So that too while you are running the models and you are doing a refresh, it costs huge. So with airflow we can build a tag that detects the document changes, rechunks and re embeds only what's changed and it will update the vector store. Instead of uh, doing a whole refresh or whole uh change of the documents or the knowledge base, it only picks up what is changed and it only rechunks it and it only re embeds what is needed. Again we are cutting cost at the vector embedding uh mod and um, like this uh, we will validate the quality and we will have some threshold set and there flow. So if um, the threshold goes below then obviously we are not going to process the data. We will stop that. If it is met then we are going to switch it to the uh, we will be connecting it to the live system. So it's totally controlled, auditable and even you can roll back if something goes wrong. That audit. Right. Is Something you just don't get with ad hoc scripts. That's where again Airflow shines.

Speaker C: Yeah, absolutely. That traceability aspect or the reliability and monitoring is so important. Tell me if this POC goes well. Where do you see this going in the future?

Speaker A: Yeah, the vision is basically like a uh full MLOps orchestration layer for JNA workloads. Right now we are focused uh on the data freshness problem, keeping like embeddings and context uh current. But the next natural step is model uh evaluation pipelines, um triggering automated evaluations whenever the underlying knowledge base changes, flagging regressions before they hit production. And beyond that uh we are looking for a um multi agent orchestration where um Airflow manages the sequencing of uh different AI agents and uh each will be responsible for a specific task in the larger workflow. It's like uh genuinely exciting territory. And um, uh airflow 2 point X introduced that uh task flow API which uh makes it much more uh cleaner to pass data between tasks using xcons. So for AI pipelines this is very powerful and you can pass the embeddings, retrieved context ch or model scores between the DAG tags as like uh Python objects. And additionally the Airflow community has published provider packages for Lang Chain, OpenAI and Hugging faces which making it straightforward to trigger model inference or embedding uh generation directly from the Airflow operators rather than uh going to the third parties.

Speaker C: Yeah, absolutely. There's so much great work going on in the Airflow project uh to make it more usable for AI and agent orchestration. All the providers you mentioned, and now the new common AI provider as well, uh which makes it very generally easy to interact with any model. So that's so cool to hear how you're making use of that already. Uh, I think that's a good segue into my standard final question uh for all of my guests, which is what would you most like to see from the Airflow project in the near future?

Speaker A: Yeah so honestly um, a few things ah come to my mind and number uh one is um, better native support for event driven triggers beyond polling sensors. Something like true pub sub integration at the core, not just via workarounds like um, the new Airflow 3.0-uh data set aware scheduling moves into that direction and I'm really watching that closely and uh hoping for that. And number two is like first um class observability for the AI workloads. Things like uh token usage tracking, model latency histograms and uh embeddings drift detection built into the airflow UI rather than uh, needing to stitch together external monitoring tool. So if we get everything under the one ui so that will be around a great thing. And third one is like I would love to see a better dynamic DAG generation at scale. So the current dynamic task mapping is great but when you have hundreds of dynamically generated tasks the UI will get wild. So it's like more ergonomic grouping and uh filtering at that scale would be like a big quality of life uh improvement. So I'm hoping it uh, when uh Apache Airflow 3.0 which is in active development so it will be bringing more changes and um, a fully decoupled uh DAC processor, improved data set, um driven scheduling and uh, redesigned uh rest APIs. So these will uh, help us in um better uh task execution isolation. So again the data set concept aligns closely with the event driven data pipelines and um, is a key step towards the uh, data aware orchestration. Exactly what modern AI pipelines needed. So we have uh, these all things I think there is no better choice than Airflow.

Speaker C: Yeah, that's great. Those are all good answers and I can see where especially the data sets are now assets. They're called in airflow 3 uh, and the event driven scheduling would be very useful for both of the use cases that you've talked about in this interview. So uh, yeah, definitely lots of stuff to look forward to in the project. Okay, well Chandan has been so great speaking with you. Thank you for sharing uh your experience with Airflow. I always love talking to folks who get to work with lots uh of different projects and implementing Airflow because you get to see so many different implementations and uh, yeah the kind of wide variety of things it's useful for just to close this out. What is the best way for folks to get in touch with you?

Speaker A: So to get in touch with me, LinkedIn is the best place. Just search for Chandan Gowda, data engineer at Action Labs and you will find me. I'm always happy to connect with the folks who are working on the data pipelines in a orchestration challenges. Just feel free to drop a message.

Speaker C: Okay, great. Well we will throw a link to your LinkedIn in the show notes for sure so people can find it. Okay, well thank you again for joining us today. It's been really great speaking with you.

Speaker B: Thank you for listening to the data flowcast. Check the show notes for links to everything mentioned today. If you're enjoying the podcast, subscribe so you never miss an episode and leave us a five star uh review.