Building a custom Tableau provider for Airflow at JLR

Najeeb Sulaiman: It's a git system, so you cannot move your dag across the gate unless the gate is open for you. So it catches. And not just catches, it blocks issues from getting to the production environment.

Narrator: So you're listening to the Data flowcast, the podcast about Apache Airflow and the world of data and AI around it. Let's get into it.

Kenton Danis: Okay. Hi, everyone. Welcome to the Data flowcast. I'm your host, Kenton Danis, and today I'm joined by Najeeb Sulaiman, who is a senior data engineer at jlr. Uh, Najeeb, how are you?

Najeeb Sulaiman: Yeah, I'm fine. And you?

Kenton Danis: I'm, um, good, good. Thanks for joining. Okay. Okay, well, let's go ahead and dive in. So, for today's podcast, we're going to talk about how you use airflow to power your data pipelines at jlr. But, uh, before we get into that, I want to start with some background. Can you tell me, for folks who might not be familiar, what does JLR do as a company?

Najeeb Sulaiman: Yeah, thank you. I mean, first of all, thank you for vitamin. It's a privilege, you know, to be here. So jlr, uh, I want to assume a lot of people would have seen JLR products even if they don't know, you know, they are actually produced by jlr. So JLR is a luxury automotive manufacturer. JLR is the company behind luxury brands, vehicle brands like Range Rover, the vendor, uh, Jaguar and Discovery. So if you've ever seen any Range Rover pass you by on the streets, you know, that's JLR products. And we designed these vehicles across different plants all over the world. And, uh, we also sell these vehicles globally as well. So that's what.

Kenton Danis: Okay, great. Yeah, as you said, I think everybody is familiar with Range Rover, Jaguar, all of those, all of those car brands. So it was certainly, yeah, new information to me about the parent company. But, yeah, that's cool. And I'm interested to hear more, you know, about your team and what you do. So what, what kind of data problems are you solving at jlr?

Najeeb Sulaiman: Yeah, so JLR is a very interesting organization from what I've just said, from data perspective, it's just about the sheer, uh, breadth of what we need to make JLR a data led organization. I mentioned, you know, these luxury brands. You know, there is high expectation, you know, of these luxury brands, different plants all over the world. We saw these vehicles all over the world as well. So you could imagine the am of data across different domain, manufacturing, finance, supply chain, e commerce, et cetera. So to ensure JLR Continues to be a data led organization. I work in a team called Data and Artificial Intelligence Office. We do a lot of things and among what we do is ensuring that JLR continue to be data led. So we provide capabilities for jlr, uh, across these different domain, um, manufacturing supply chain, et cetera, to ensure we harness this data, to ensure that we continue to produce uh, vehicles that delight our customers and also ensure that we continue to improve the roi, reduce cost, improve maintenance, et cetera, to ensure that JLR is data driven. So we do this across um, different domains and also using different technologies and stacks.

Kenton Danis: Yeah. Okay, great. And that makes sense. And where does Airflow fit into that? What sort of orchestration needs do you have within those use cases?

Najeeb Sulaiman: Yeah, so we use Airflow a lot. It's main orchestrator in our data stacks. So I would say it's like the central nervous system of our data stack is the tool that uh, connects everything because we get data across different sources. You know, imagine data coming in milliseconds, you know, from telemetry, data from the manufacturing plant, etc. So within our data stack itself, airflow sits at the center of it. It helps us to connect the dots, third party systems from source, destination and even you know, around it as well. We use it every little. Like I said and I would say it help us to orchestrate things. Just like you already know, FWORK is an orchestrator. So it's scheduling automation. Ensure there is no manual intervention, ensure our systems are reliable, ensure we continue to support, you know, um, the business with data. So it's the integral part the central nervous system does. I'll summarize it for our data stack in JLo.

Kenton Danis: Okay. M. Very cool. I love that description of airflow as the central nervous system. I think that's a really good way of putting it. And then. Yeah, yeah, that's great. And then I'd love to dig into, you know, some of your use cases in a little bit more depth. One of the things that I think you had mentioned to me before was that you, you have a tableau integration. So I would imagine your team is providing some sort of like uh, dashboards for internal users or something like that. Tell me more about that and how airflow fits in there.

Najeeb Sulaiman: Yeah, so that's a very interesting problem we were trying to solve and to give more context because I know a lot of people listen to this and uh, context is very important as well. Tableau is one of our visualization tool. Um, basically when you want to get data into tableau, there are Usually two major modes. We have what we call extract refresh or live connection. So the live connection means every time you run a visualization on Tableau, it goes to your data warehouse and then put the data every time. So if you go to the visualization 10 times, it goes back to the data warehouse to put the data every time you visualize your data. The other, uh, approach is what we call extract refresh. So extract refresh is more like batch processing. So it goes into your data warehouse pulling the data. And if you visualize your data a hundred times, it doesn't go back, you know, it just use the data that's already been extracted and that way you can use your data without going back to your data warehouse. That is trade off, right? But those are available. So now we use extract refresh a lot. So what that means is it's scheduled. For example, you have a dashboard that refreshes 6:00am in the morning. So that means every morning at 6:00am, um, you expect Tableau to go to your data warehouse, put the data that's available there, and then make it available. So when you come into the morning, you already have like data to visualize. Now there's a problem with that. Uh, the problem, which is a major problem, is that it's not data pipeline aware. Because remember you schedule 6am M in the morning, so it is not aware if the data is already refreshing your data warehouse or you still extracting the steal data. You know, maybe something happened and your data pipeline has not refreshed your data warehouse, for example. So that means you could be extracting still data. Ah, because the tableau is not data pipeline area. It just go to the data warehouse. Anyways, the other thing is that if your pipeline refreshes tables earlier than 6am, tableau has to wait till 6am before it goes and pull in the fresh data. So that means it's also not, um, live, uh, we can not need it live. But you cannot get the fresh data as that when it's already available. So that was the problem statement. So, um, what we set out to build is how can we make our dashboard pipeline away. So there is a connection and natural integration between pipeline data. Pipeline refreshes the data warehouse. Tableau is aware, Tableau goes to data warehouse pulling the data. And anytime you are viewing data, you're always seeing the data as that's when the fresh data that's available in the house, basically. So we wanted to solve that problem, you know, and luckily, interestingly, Tableau has a rest API. So what that means is we can programmatically interact with Tableau engineers who love APIs. What we started with is we started with building a Python module that just enter to this API, uh, because there are available endpoints. So there's an endpoint to trigger a refresh using the API, there's an endpoint to do a lot of other things. So we use those endpoints made available by Tableau to trigger these refreshes. So that worked. We're able to just use Python, just write the scripts, we're able to refresh Tableau dashboards when Pipeline finish running. However, that is not scalable because what that means is everybody that needs to do this in every project needs to then copy this code across, uh, those, you know, data pipeline. It's not scalable. It's also not maintainable because when you distribute code like that, it means people will be running on stale version because everybody just duplicate the code. Which is not a very good, uh, practice. But we proved that we can solve the problem basically. So I don't know if you want me to just continue to explain the story or you want me to stop then, you know, on the problem statement.

Kenton Danis: Yeah, I think that's a great overview of, you know, what you're dealing with and kind of why Airflow is involved. I uh, think you may have mentioned that you have a provider that you've built to do this within Airflow. So tell me a little bit more about that because I think the community there is like an open source provider for Tableau, but it sounds like that's not what you're using. So what was the, what was the decision there?

Najeeb Sulaiman: A, uh, very interesting one as well. So like I said, we built something, we evaluated that it worked, but we're like, wait, we use Airflow a lot. I know most of our data pipeline. Why not just use the Airflow Tableau provider to interact with Tableau? Why use our own script? So yeah, we then try to use the Airflow Tableau provider. So naturally, because it's already available, I mean, kudos to the community, very, very strong community that Airflow has. We were able to test this, integrate this, see how it works. However we, it's a roadblock. Not that it won't work. It worked even though was a little bit of, you know, trial. But the roadblock is Tableau Cloud allows different modes of authentication basically. So it has username and password, phc, which is Personal Access token, and also jwt, which is Java Web Token. So it has this mode of authentication to Tableau. However, what happened with The Airflow Community provider, uh, uh, Tableau provider is. It supported the pat, which uh, is personal access token, username and password initially. So that was fine. However, there were a lot of issues with the personal access token because the way Tableau made that available is you cannot have two connection running at the same time with the same pat. So what that means is that you cannot exactly, you cannot do. You cannot run things simultaneously using the pat. But we love parallelism as engineers, so we, it was causing a lot of issues and the maintenance of the provider or airflow generally, they were having to respond to a lot of query from users of this provider. So I read the issues because I actually followed the GitHub issues and I saw the conversation and at some point there has to be like a decision to make to make do. We keep having headaches and people keep coming. There's issue with this and uh, we cannot do it without tabloid getting involved or we just remove that particular feature and support the one that is uh, easier. So yeah, at some point it was deprecated. So PAT was no longer available for authentication. So it was just using your own password. So. And that was the limitation for us, uh, initially M. Because PAT over username and password, you don't, I mean we don't want to use any more password. It's not recommended. So. However, uh, the community provider doesn't allow us support PH any longer. So we are now in the mid crossroad. How do you proceed? So that led us to now going the route. Okay, fine. We'll start from what the community provider already gives and build on top of that to build our own custom provider that then allows us to use our uh, mode of authentication, which is pat. So that was what led us to building our custom provider, uh, because the one available by the community has, you know, duplicated the PAT approach of authenticating to Tableau Cloud.

Kenton Danis: Yeah, okay, very cool. That totally makes sense. Those limitations on authentication can be really tricky to work around, but certainly a benefit of airflow that you can just, just build your own provider and. Sounds like that's working pretty well for you.

Narrator: This episode is brought to you by Astronomer, the team behind astro, a managed airflow platform built for data engineering teams. From batch pipelines to training models and wiring up agents, Astro handles the infrastructure upgrades and scaling so you can focus on building. Check it out at Astronomer IO.

Kenton Danis: Um, yeah. Cool. Well that's, yeah, a really interesting use case. Appreciate you sharing that. One of the other things that I wanted to dig into, a bit of a separate topic was you mentioned using CICD pipelines and how you validate DAGs before they're deployed to production. This is of course a really important topic especially, well, maybe for all companies, but especially for really big ones like a JLR where you've got a lot of airflow usage going on and your production pipelines are really critical. Tell me about what that process looks like and you know, how, how well it works in the day to day in terms of testing your dags.

Najeeb Sulaiman: Yeah, again we, there's also always a problem statement and you already mentioned one of them. One of them is that JLI is a very big organization. So that means, you know, you can imagine the number of engineers, number of teams that runs it, a pipeline. So one of our uh, best practices to use CICD to ensure we follow a very solid pattern across the teams. And the problem statement is this, you can develop airflow locally or in your dev environment, but when you move your dad to production, it's not business as usual. You don't want things to break, you don't want to cause you don't want, you know, your downstream users to be frustrated basically. So the idea is this, how can we ensure that before things get to the production we do our uh, sanity checks to ensure, we check for, you know, errors that could happen so that when someone is trying to push their DAG to production, we already catch common issues that would have affected the production environment. So the way it works is this. We have in our CI CD pipeline we have a couple of checks, in fact we have uh, more called DAG validation, you know, and uh, to talk about the DAG validation, specifically in uh, our CI CD pipeline it mutually checks for things like I'll give you some examples, you know, for example in every, there's what we call top level codes and top level code is more like you can have within your dag, you can have a script or a function that is making external call to a database or to somewhere a third party system. And that is very bad for airflow because Airflow Scheduler is going to pass a DAG every couple of seconds. So what that means is every time airflow is passing the dag, you are making connection to your third party system, external connection. So that's very bad. So that will not essentially break your dag, but it's not efficient way to write a dag. So um, because you know, engineers, we develop and want to push the dag. So we check for things like this. Even though it works, is it working appropriately? Uh, is it following the standard Is it following best practices? So we cache something like that before it gets to the production so it doesn't overwhelm m the resources that is meant to be used by other, you know, dags. Another example is import errors. You know, we want to also check because that's one of the most common failure in DAGs. Import errors. Maybe you are relying on a package on your local environment that is not already available in production. So you want to cache that before it gets deployed so it doesn't cause issues in production. And uh, recently as well, like Airflow 3 compatibility, there are a lot of breaking changes from Flow 2 to FF3. So things like that, you don't want to leave it to you know, guesswork or uh, you know, to vibes. So you want to cache that before it becomes an issue. So we have things like that as well, whereby we check if your Dag is using FO3 do you have import issues? So if you have import issues it lets you know, uh, before it gets to production so you can fix it right there and there. And we have uh, you know, so that's how we kind of just use CI CD pipeline airflow, that validation. A couple of things to ensure we sanity checks everything that is leaving the environment to production environment in airflow.

Kenton Danis: Yeah, totally makes sense. I can see where that would be really important. And do you feel like that's working really well overall? Do you catch most issues before they make it to production?

Najeeb Sulaiman: Yes, yes, it helps a lot. I think one thing that uh. Yeah, I mean it's, it's very useful for iFats because again it's a git system so you cannot move your DAG across the gate unless the gate is open for you. So it catches and not just it blocks issues from getting to the production environment. So yeah, it's a very, very library and um, important part of our orchestration and also like deployment process.

Kenton Danis: Yeah, ah, definitely good to have those guardrails both for just the general errors and the best practices as you mentioned. So um. Yeah, very cool. Well thank you for going through that. But those were both really interesting use cases. Just in the interest of time, I always ask my guests a standard final question. And that is what would you most like to see from the Airflow project in the near future?

Najeeb Sulaiman: Um, Airflow. I'm a very, I'm a fan of Airflow. I. I don't just use airflow, I teach very deep in the flow. So one thing that I would like to see is based on what I've just said as well Regarding testing and validation, there is nothing existing for now in terms of native integration testing for airflow. So why by integration testing is you can do all those, you know, sanity checks, but it would be nice to have like a native integration testing so whereby you can already flag dependency issues. You can flag if your pipeline is going to work well, uh, before you actually deploy. Because sometimes what happened is yes, you can check for import errors, you can check for missing dags, but sometimes it's not just that, uh, it could be dependency issues, you know, and you don't get to cache that until you know it's, it's an environment because you have to run it and to be able to know is this thing working well. So I think it would be very nice to have a native integration testing support in Airflow. So that's one. The other one M on top of my head now is the one I encountered recently as well. So I was building like an observalistic platform to observe airflow infrastructure. So not just the dag, to see how the worker is behaving, the scheduler, the metadatabase, etc. Then I was trying to use hotel, which is opentelemetry, which is more like it's, it's not the next best thing any terms of observability. So that's, that's the future of observability. Airflow has done a good job there, uh, because I found out that there is already like instrumentation for Autel. However it's still very, very lacking behind compared to the native or the defaults or the one, the legacy, let me say legacy Start D. So uh, Start D is legacy and it's already very matured but whatever is still maturing. So I had a couple Roblox I remember to like build my own Grafana dashboard from scratch because nothing existing, nothing's existing already in the community. Even thinking of open sourcing, that I'll think about it. But uh, yeah, it would be very nice to have better observability, especially native hotel observability in Airflow and also Lineage as well. So Airflow can tell you your dags, it can show you your dags, you know, dependencies, but it doesn't give you the lineage, for example dependency and also like what depends on your uh, on your dags. So you can remember, you can imagine the issue of tab dimension. So there's nothing to tell. This particular dashboard is depending on your pipeline. So it's also been nice to have like native lineage and you know into in Airflow. That's very Very good.

Kenton Danis: Yeah, absolutely. Those are all great answers. And you're not the first guest I've had on to talk about hotel support within airflow as well as lineage. So yeah, common issues for a lot of data engineers out there. Well, Najeeb, thank you so much for coming on. It's been really great speaking with you. Just to close us out, what is the best way for folks to get in touch with you?

Najeeb Sulaiman: Yeah, the best is on LinkedIn. Najib Sliman. That's my name. Also on X as well. Najeeb Sliman.

Kenton Danis: Okay, great. We'll throw a link to that stuff in the show notes. Well, thank you again, Najeem. This has been a great conversation.

Najeeb Sulaiman: Thank you so much. Thank you. My pleasure.

Narrator: Thank you for listening to the Data Flowcast. Check the show notes for links to everything mentioned today. If you're enjoying the podcast, subscribe so you never miss an episode and leave us a five star, uh, review.