Running Airflow 3 in a regulated environment at OTPP

Kausi Narayan: Last week was our uh, sunset of hosted uh, deployments and we're fully live on uh, remote deployment, uh, model.

Narrator: You're listening to the Data ah, flowcast, the podcast about Apache Airflow and the world of data and AI around it. Let's get into it.

Kenton Danis: Okay. Hi everyone. Welcome to the Data flowcast. I'm your host, Kenton Danis and today I'm joined by Kausi Narayan, who is a cloud data platform lead of data engineering at Ontario Teachers Pension Plan, which if it's okay, I think I will refer to as OTPP going forward in this interview. Uh, Cassie, it's great to have you on. How are you?

Kausi Narayan: Hey uh, Kenton, thanks for having me. Uh, I'm good, how are you?

Kenton Danis: Yeah, I'm doing really well. I'm excited to have you on. Uh, well let's go ahead and dive into it. So for today's podcast we're going to discuss OTPP's cloud migration journey and your move to airflow 3 and use of remote execution. All very exciting topics. Uh, but before we get into that, maybe let's start with some background. Tell me just at a high level what OTPP does.

Kausi Narayan: Yeah. Ontario Teachers Pension Plan is a global investor, um, and is a uh, fully funded defined benefit pension plan. Uh, we invest in a wide variety of asset classes, uh, primarily to deliver retirement security for Ontario teachers. Uh, we currently have like 346,000 active members in the fund, uh, both working as well as retired teachers in the province of Ontario.

Kenton Danis: Okay, very cool. What do you and your team do?

Kausi Narayan: Uh, my team primarily is um, uh called a data platform team. Uh, we're currently going through uh, um, migration, uh from all of our on prem ecosystem to cloud. So a lot of uh, the data platform, uh, initiatives uh, surrounding this cloud data technology. My team is primarily responsible for uh, enabling standards, patterns, framework, uh, new capabilities within these data cloud platform technologies as well as uh, we also support the business technology teams with uh, their project initiatives, also with onboarding training, um, to help adopt these new technologies and capabilities.

Kenton Danis: Okay, very cool. And you mentioned as part of that that you're currently going through a cloud migration. I'd love to hear a little bit more about that kind of what's driving that change. What is that process looking like?

Kausi Narayan: Yeah, so it's been a multi year journey. Uh, it kickstarted uh, a few years ago and we're still in the midst of that journey, um, primarily to uh, move away from our on Prem data center and uh, move to cloud M architecture. The key capabilities is primarily enable uh, Scalability, resilience of the platform. And also as we all know the business demands are rapidly growing. We wanted to enable new capabilities as well as like you know, meet the growing demands of the business. So those are the key drivers that uh, we wanted to enable cloud uh, uh capabilities.

Kenton Danis: Yeah, that makes sense. And what does the new tech stack look like?

Kausi Narayan: Yeah, from the cloud data platform perspective, uh, we have Snowflake as our uh enterprise data platform and um, we use dbt uh primarily for uh, our data transformation and um, Airflow uh, is sitting in the middle of that ecosystem primarily for data pipelines and orchestration.

Kenton Danis: Okay, great. Uh, so it sounds like airflow is pretty central there. Maybe tell me a little bit about what sort of things you're orchestrating with Airflow.

Kausi Narayan: Yeah, so it's, it's positioned as a data pipeline orchestrator and uh, as an investment uh organization we bring in data from uh, multiple vendor data sources, market data, um, also like our internal enterprise data, data from like you know, order management systems, data from book of record systems. There are so many different data that is needed. Our daily trading and investment decisions. So this data engineering pipeline help bring these data, uh, either being it scheduled or event driven, kind of like use cases.

Kenton Danis: Yeah, absolutely. That totally makes sense. And I understand that OTPP is a customer of Astronomers. Maybe you can tell me a little bit about why you chose to run Airflow on Astro as part of this stack that you've put together.

Kausi Narayan: Yeah, sure. Um, when we started exploring um, Airflow, uh specifically the key features that we were looking for were available. Um, and specifically because of the nature of it being open source community and a very active community and uh, able to release features pretty rapidly uh at the same time ah, we also looked at the infrastructure side of things and managing that infrastructure on our own was an additional overhead and cost associated with it as well. So uh, we started looking into like a managed service providers and um, astronomers stood out uh, in that space. Uh specifically because Astronomer uh, is one of these strong contributor to the airflow open source community. Um, and as and when the airflow releases there is an Astronomer, uh runtime releases right away and um, other capabilities like monitoring uh, observability and rbac. All of that is built natively within Astronomer as a platform. So these are the things that um, we don't want to spend, spend time investing. Rather we wanted to uh, focus on business value and leverage some of these managed service providers.

Kenton Danis: Yeah, absolutely. That's great. I think that's you know our sort of highest level value prop is we can just take care of some of those things so you don't have to. So it's great to hear that that's working out for you. Uh, I also wanted to double click very briefly on. You mentioned uh, that DBT is a big part of your tech stack. And I'm curious how you are running DBT with Airflow and kind of. What, what does that look like?

Kausi Narayan: Yeah, when we were originally experimenting um dbt, we were running a um DBT uh core within Airflow natively using Bash Operator and such. Uh was okay for experimental work. We started with something small models and it was all fine. But then when we started bringing in our real life scenarios and uh, migrating our uh, big uh, uh data transformation projects, uh, we ran into issues of uh, leveraging native airflow for dbt. So that's when we found about DBT Cosmos uh provided by Astronomer, um as well. So we started looking into DBD Cosmos to specifically be able to give that granularity of all of our uh models. It also enabled us to see much better lineage as well as be able to troubleshoot um as well and uh, be precise on where exactly the failure is and rerun from that point onwards rather than running the entire big monolith of data model. So um, that's how I think we started as like you know, plain vanilla. Then slowly we evolved into leveraging Cosmos. Um and um, other thing that we started looking at is also performance boost that we get out of Cosmos now running it in like the new capability of uh, running it in a watch mode, Watcher mode uh also brings in a significant uh performance boost for some of our uh, uh transformation layer. So that's, that's how we started.

Narrator: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams from batch pipelines to training models and wiring up agents. Astro handles the infrastructure upgrades and scaling so you can focus on building. Check it out at Astronomer IO.

Kenton Danis: Yeah absolutely. Watcher Mode was a really exciting uh feature in the release that happened I think a couple weeks ago as we're recording this, maybe a couple months ago by the time this has come out. But yeah, definitely something that was widely requested. I've seen a lot of excitement about it including from my team who was yeah pretty impressed. Yep, yeah, very cool. Uh, and then I also want to touch a little bit on your experience with Airflow 3 because I understand that you've just gone through a migration. You mention um, Astronomer having runtime releases right? With new uh airflow releases which is definitely something we want our customers to be able to um, you know, take advantage of. All of the new things going on in the Airflow project. And Airflow three was a big one. So yeah, tell me about kind of what um, what motivated you to do that upgrade, uh, when you did and kind of what that experience looked like.

Kausi Narayan: Yeah, I think when we started Airflow we weren't way too off given that our journey started recently. We were at airflow, uh version 2.9 and uh, when airflow 3 came about, the main thing that we wanted to learn as part of this, how fast we can upgrade uh, from a version to another version. Also um, we started looking at some of the new capabilities that Airflow 3 offered, which was uh, remote execution capability. Um, so that was something that we actually added from day one. But because we had to, to go live right away with airflow 2.9, it wasn't available. Uh, so that is one of the primary driver as well for us to make a decision to move to airflow 3 and leverage the remote execution capabilities uh, at the same time exercise the upgrade procedures. And it was pretty um, easy for us to upgrade mainly because we were pretty close to the version as well. Uh, we weren't too far off. At the same time we leveraged Astro CLI and some of the linters that can automatically catch some of the issues and uh, any sort of like uh, library that get deprecated and need to be upgraded. All of that were pretty uh, straightforward and uh, very easy for us to manage. So we made a quick call to move to three so that we can enable remote uh, execution capabilities.

Kenton Danis: Yeah, great. Let's talk a little bit more about the remote execution because this is definitely, I think one of the most popular, uh, or I don't know most popular but certainly one of the biggest impact features of airflow 3. Uh, I'm curious what drove uh, your decision to use this feature? Why were you guys looking at it?

Kausi Narayan: Yeah, I think uh, for us being uh, an investment organization, uh, there's a lot of security controls in place, um, and uh, we wanted to make sure that the data doesn't get uh, moved uh, from our ecosystem. There's always uh, infosecurity concerns when the data gets moved in the cloud. Um, so that was one big uh, factor and the other one is the scalability flexibility that we get on our side. Uh, we can up and down the resources according to the different workloads, um, given that the execution plane is going to be running in our kubernetes Cluster. So those are some of the key compelling reasons uh, that uh, we started exploring remote execution. And um, yeah it was pretty successful uh, migration and we uh, did a quick um, pov, uh working closely with um, the product engineers and uh, we learned a lot. Uh, also this capability is new from the product perspective as well. So we provided feedback and uh, we were able to get some updates uh, in the next versions. And uh, it was a very pretty collaborative work that we did uh, with the product engineers and uh, it was a pretty successful migration.

Kenton Danis: Yeah, that's great to hear and I think the compliance piece is really interesting. It's certainly something that we've heard a lot about with financial institutions of various kinds. I think they're one of the top uh, industries I would say, that needs to make use of a feature like remote execution. Before you had this available to you, did you have to do some sort of workaround to manage the data within your security policies?

Kausi Narayan: Um, yeah, we had to come up with. Yeah. So definitely, I think like you know, compliance is always uh, and infosecurity is always at the forefront of our uh, any sort of integration with cloud platform. Uh, so when we had to do uh, this without the remote execution, there is very complex network architecture that we have to come up with. We have to enable like VPN tunnel and we have to handshake through that vpn obviously like you know, there's other network firewall rules and restrictions uh, that we have to enforce which made like the latency a concern as the data flows through. Uh, there is a bit of that latency that we have uh, noticed. And uh. Yeah, so those are the work. I wouldn't call it like workaround, like the complexity in the network architecture. And uh, by moving to remote execution we were able to simply simplify that network architecture as well.

Kenton Danis: Yeah, absolutely. I could see a huge benefit to doing that. Uh, you mentioned, well maybe this seemed easy because it was a simplification, but I know remote uh, execution, setting that up can be not super trivial depending on the environment that you're working in. You mentioned doing a POV before you actually made the move to doing this in production. Kind of. What did that process look like and how was it for you from an implementation standpoint?

Kausi Narayan: Yeah, so I think when we were in the hosted execution we exactly know some of the pain points or concerns specifically as I called out, like from the network complexity, data security and all that stuff. And uh, using those as kind of like our uh, goal, uh, and we came up with some um, key success criteria uh, metrics uh, for us to be able to say hey we're good to go and move forward. So yeah we were able to make those decision pretty fast. And as I mentioned we work with astronomer product engineers. Uh, we had some dedicated resources helping us uh, to uh, set this up on our side and making sure the handshake is happening properly. We were able to spin up a sandbox uh, environment uh for this uh, use case as well to timebox it uh, for I believe we did it in a four to six weeks time box. Uh both from uh, infrastructure setup perspective as well as deploying some of our workloads and validating those through those success criteria to make sure that we're able to meet all of those. And uh, then yeah I think after that it's, it's primarily socializing with our executive team and um, getting the, the sign offs from uh, our leadership to, to move forward.

Kenton Danis: Okay, great. And then so you have now at this point moved beyond that stage and you're using this in production, is that right?

Kausi Narayan: Definitely. Yes. Yes. So we actually uh, last week was our uh, sunset of hosted uh deployments and we're fully uh remote uh deployment model.

Kenton Danis: Okay, cool. That's exciting.

Kausi Narayan: Yes, yes. So we're closely monitoring, we're learning as well. Uh, but so far so good.

Kenton Danis: Yeah, that's great. Love to hear that. Like I said, I think the remote execution feature has been so impactful especially for companies working with sensitive data especially. And so it's really great to hear that success story from your team. Very cool. Uh, well I think that is a good um, tie into my standard final question which I ask of all of my guests. Might be related to airflow 3 and remote execution or it doesn't have to be. Uh, but that is. What would you most like to see from the Airflow project in the near future?

Kausi Narayan: I think like uh, with airflow 3 there are a lot of like you know, new capabilities uh specifically around AI. Everything is like you know going at a rapid pace that we're trying to catch up. Um, I think like the focus for our organization as well is to uh, bring in uh, AI capabilities. A lot of the uh, AI orchestration, um, as well, um, and also uh, like you know stabilizing the remote execution uh platform. So these are the key areas that uh, we would be primarily um, focusing on and closely watching this uh community space as well to see how we can quickly enable these capabilities and evolve the platform.

Kenton Danis: Yeah, absolutely. On the AI front, have you seen or tried the new common AI provider?

Kausi Narayan: Yes, uh, uh, we just recently upgraded to airflow 3.2, so we're in the midst of evaluating some of those. Along with that, uh, AI, there's also other cool features like async operators, um, there's data partition, data assets, um, so a lot of people cool features that primarily fits into our use case that we're looking at as well. So definitely, um, exciting. Um, looking forward to do some POC and uh, roll that out to production.

Kenton Danis: Yeah, that's great. There's so much great stuff going on in the project right now and I know the common AI provider has been generating a lot of excitement. I've just been working on today actually I will be giving some talks at upcoming meetups about the provider and some of its functionality because I think you are not the first person I have heard say. But working with AI from Airflow is a top priority for their team. So I think something definitely I'll be

Kausi Narayan: the first one to sign up for those webinars. I love the webinars, uh, that uh, Airflow community runs. It's amazing, um, to learn all the new capabilities as well as to quickly implement as well with all the sample code repos that you provide. Uh, it's been amazing to use all those, uh, resources.

Kenton Danis: Oh great. I love to hear that. Um, I'll pass along that feedback to my team and other folks at Astronomer, so. Okay, great. Well, thank you so much for going through all of that. So interesting to hear about. Yeah, the journey that your team has been on recently and the success that you've had with Airflow three. Uh, just to close us out, ke, what is the best way for folks to get in touch with you?

Kausi Narayan: I'm um, available in LinkedIn, so feel free to reach out and touch base with me. I'm happy to share our experiences and able to share our uh, a journey as well.

Kenton Danis: Great. Well, we will put a link to that in the show notes. Well, thank you again for coming on. It's been so great chatting with you.

Kausi Narayan: Thank you. Thanks for having us.

Narrator: Kenton. Thank you for listening to the Data flowcast. Check the show notes for links to everything mentioned today. If you're enjoying the podcast, subscribe so you never miss an episode and leave us a five star, uh, review.