The Data Flowcast · 2026-09-17 · 20 min
Key moments - from our scoring
Substance score
61 / 100
Five dimensions, 20 points each
Grupo Vasquez is a major Latin American holding company with over 30 subsidiaries including Ueno Bank (60% market penetration in Paraguay) and operates across financial services, e-commerce, food delivery, and mobility. Lucas Trubbiano leads the data engineering center of excellence, which uses Airflow as a central task orchestrator for scheduling, dependency management, and alerting - deliberately separating orchestration from processing, which is handled by Spark and DBT. The team developed two YAML frameworks to solve a critical bottleneck: code review overhead when non-technical teams attempted to build DAGs. The newer framework uses Jinja templating and custom operators (for tools like DBT, their proprietary data quality tool, and AWS services) to let business users configure only what's customizable while the framework handles testing, validation, and reliability behind the scenes. They've also adopted spec-driven development based on GitHub's spec kit, allowing AI tools to understand available templates and guide users through troubleshooting. This approach has dramatically reduced the data engineering team's support burden while scaling Airflow adoption across the organization.
They built a custom YAML framework where non-technical users configure only what's changeable in pre-built Jinja templates, while the framework handles all validation, testing, and operator selection automatically.
They've built custom operators for open-source tools like DBT, their in-house data quality tool that runs on Spark, and AWS services, triggered directly through the YAML framework.
They developed an in-house data quality tool that runs on Spark and tests data per-partition rather than scanning entire tables, triggered by custom operators that integrate with their YAML framework.
ITTI customized GitHub's spec kit framework to standardize data lifecycle rules and create skills that allow AI tools and users to request DAG creation or debugging against certified templates.
When DAGs fail, Slack alerts include logs and explanations of common errors so users can self-resolve most issues; only complex operator-level problems escalate to the central team.
Our reviewer’s read on each dimension, with quotes from the episode.
The episode contains solid practical insights about framework design (YAML abstraction, Jinja templating, custom operators) and organizational scaling (80% adoption rate, self-service enablement, Slack-driven alerting). However, it lacks depth on implementation specifics - no metrics on performance gains, cost savings, or failure reduction; no discussion of trade-offs or failure cases. The conversation stays at a surface level for much of the runtime without probing into how these solutions actually work mechanically or what problems they created.
we created another framework where we abstract uh more the complexity and only uh, leave the user to configure what we think it's configurable
for example only configure what document by the URL and ah, what sheet they want to convert um into table and the destination. But behind we introduced lot of testing tasks
The YAML abstraction pattern and Jinja templating approach are standard industry practice (the host even mentions DagFactory as a precedent). The spec-driven development idea is borrowed directly from GitHub's Copilot spec-kit framework. While the execution in a 30-company holding is noteworthy, the conceptual frameworks themselves are not novel or contrarian - this is solid engineering practice, not original thinking.
we based our framework on GitHub spec kit already existing framework
the typical DAX factory framework, this project we call Bowie
Lucas Trubbiano is a legitimate data engineering practitioner leading a center of excellence within a large enterprise holding (Grupo Vasquez with 30+ companies), managing 300+ DAGs at scale. He has hands-on responsibility for Airflow infrastructure across multiple business units and has built custom frameworks. However, he's not a globally-recognized figure or founder; he's a strong mid-level operator but not exceptional caliber relative to podcast guest standards for a specialized data engineering show.
I'm leading the data engineering center of excellence
we have almost 300 processes in airflow
The episode includes some concrete numbers (300 DAGs, 80% adoption of new framework, 6,000 survey responses, 60% customer penetration for Ueno bank) and specific tool mentions (Spark, DBT, Google Sheets, AWS). However, most implementation details remain vague: no specific timelines for rollout, no quantified metrics on speed improvements or reliability gains, no named examples of problematic pipelines, and minimal detail on the data quality tool or custom operators beyond broad categorization.
we have almost 300 processes in airflow
more than 80% of our DAX are created with a new framework
The host asks reasonable follow-up questions (e.g., 'how do you manage all of that?', 'tell me more about reliability') but rarely pushes back or challenges claims. There's no probing into failure cases, trade-offs, or why certain architectural choices were made. The conversation feels like a guided walkthrough of the guest's framework rather than investigative dialogue. The host occasionally misses opportunities to dig deeper (e.g., 'Tell me a little bit more about the custom operators' yields only vague categorization, not specifics).
Yeah, that totally makes sense. Makes it a lot easier for you to Institute some guardrails
Yeah, that's great. Sounds like it's been very successful
Computed from the transcript - who did the talking, and the words that came up most.
Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano , Data Engineer at itti , the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python. Key Takeaways: 00:00 Introduction. 01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies. 02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs. 04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML. 05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.
Transcribed and scored by The B2B Podcast Index.
Speaker A: You're listening to the dataflowcast, the podcast about Apache Airflow and the world of data and AI around it. We're excited to announce the launch of the annual Airflow Survey, the largest data engineering survey ever with almost 6,000 responses. This year we are offering survey respondents something special. A free code to take the new Astronomer Certified AI Orchestration Fundamentals Exam $150 value. Just go to Astronomer link AirFlowsurveyMinus2026 or click the link in our show notes. Now let's get into the show.
Speaker B: All right. Hi everyone. Welcome to the dataflow cast. I'm your host, Kenton Danis and today I'm joined by Lucas Trubbiano who is a data engineer at itti, which is the technology company within Grupo Vasquez. Lucas, how are you?
Speaker C: I'm doing great, thank you Kenton. Thanks for the space.
Speaker B: Yeah, of course. It's great to have you on. Well, we'll go ahead and dive in. So for today we're going to talk about how ITTI uses Airflow to orchestrate data across more than 30 companies. And we'll dig into the custom YAML framework that you've built to make Airflow more accessible to non technical teams. Before we get into that, let's start with some background. So tell me about what ITTI and more broadly what group of Esque was does.
Speaker C: Perfect. Well I will start with group of askers that it's the holding of as you said more than 30 companies in Paraguay, uh Latin American country and uh, it's a really big holding but split into very um, specific areas financial and uh digital ecosystem with services through payments, credit and other complementary like uh, E commerce, uh food delivery and mobility also.
Speaker B: Okay, perfect.
Speaker C: Um, two of the main companies of this group are uh, Ueno bank that it's the leading bank in the country with more than 60% of active population. So it's a lot of customers uh for us and uh the other it's the tech consultancy company that brings services to all the other companies. It's uh, a very big ecosystem.
Speaker B: Yeah, that sounds huge. What does your team do within that?
Speaker C: Well right now I'm leading the data engineering center of excellence and our main goal ah is to build a 360 degree view of our customers in order of that we explore some new and challenging business cases like for example streaming analytics or unstructured data processing. But not also we create or explore some business solutions also we standardize and scale then uh, those solutions to other teams across the organization.
Speaker B: Okay, very cool. And where does Airflow fit into all of that?
Speaker C: Well we use uh, airflow mainly as a central task orchestrator. We do not use it for data processing. For this we have SPARK or DBT for example. So we always try and um, our best practice is to use airflow only to define dependencies, scheduling, also monitoring uh, our data pipelines and to handle uh, all types of failures and alerting.
Speaker B: Okay, so very traditional orchestrator kind of offloading the processing paradigm.
Speaker C: Yeah, um, uh right now we have almost uh 300 processes in airflow. So yeah, um, most of them grow in the last year. So it's a high speed ah creating of those processes.
Speaker B: Yeah, that sounds like a lot and I think it leads me into my next question which is you've talked about how broad the business is and how all of these different business units need orchestration. I imagine you have a lot of different people trying to use airflow or requesting that your team do something in airflow for them. How do you manage all of that? How do you ensure that all of the users have the access they need to orchestration using airflow?
Speaker C: Uh, well we can think of it in two ways. Um, first one it's uh, that at uh enterprise scale we prefer to use an airflow as a service. With that users don't worry of manage uh infrastructure and uh, the other thing is to enable self service to our users to our non technical users. We made that by YAML code, uh by custom YAML code. So users only need to focus on what they want to create. And uh, then in CI CD pipeline we handle how to convert that configuration into a fully functional DAG airflow.
Speaker B: Okay, interesting. You mentioned at the top I kind of buried the lead a little bit that one of the ways that you've helped to manage this is developing a custom YAML framework. So tell me a little bit more about that and what problem you were trying to solve.
Speaker C: Perfect. Well we have two YAML frameworks right now implemented. The first one was uh, more customizable based on uh, the typical DAX factory framework, this project we call Bowie. But uh, the real problem was the bottleneck uh in code review of those pipelines because users not all the time use the correct operator or change some parameters uh, in a wrong way. So we decided to create another framework where we abstract uh more the complexity and only uh, leave the user to configure what we think it's configurable and validate all other things um, to be as we want.
Speaker B: Yeah, that totally makes sense. Makes it a lot easier for you to Institute some guardrails and manage everything with lots of different teams using it.
Speaker C: I would like to add um, another interesting insight that Today more than 80% of our DAX are created with a new framework because it's simply uh, faster for the user to create those pipelines. It's a very big uh, percentage of creation with that new framework.
Speaker B: Yeah, that's great. Sounds like it's been very successful and you've had a really wide rollout. Tell me a little bit more about what the framework looks like. I know there are a lot of companies build custom YAML frameworks and then there are of course open source ones like Dagfactory. I'm curious how yours works and what it looks like for somebody from your team to build a pipeline using it.
Speaker C: Uh, well behind the framework we have the typical Python code as a center of excellence. We develop the best DAG to solve some problem and then we put it in a Jinja template. Um, then a user only configure um, what it's customizable in that pipeline through YAML and then we compile these parameters into the Jinja template and compile the Python code. It's very simple, uh, behind but it's also very powerful because uh, behind we wrote Python but ah, user don't know what's behind.
Speaker B: Yeah, so it sounds like very simple for the end user to interact with then.
Speaker C: Yeah. One interesting thing is that um, for example I can give you an example of a uh, user trying to ingest for example a Google sheet file into our lake. They for example only configure what document by the URL and ah, what sheet they want to convert um into table and the destination. But behind we introduced lot of testing tasks and um, user don't know really uh, if there is a pre validation then download and then a processing task for example. Ah, but we introduced that uh, because we think reliability uh, is as much important as the speed. We ensure that in the framework.
Speaker B: Yeah, that makes sense. I'd love to double click a little bit on that reliability piece that you just mentioned since it sounds like that's really important there in addition to the development speed that you're enabling with this framework. So how does that work from a data quality perspective? How is that part of the framework that you've built?
Speaker C: Uh, from the data quality perspective? Uh, it was very challenging for us. We uh, develop our um, self made in house tool because we want something faster that runs over Spark framework and that could test data per partition not to fully scan the tables. So we decided to develop our own tool. And that tool, it's triggered by custom operators we have developed on Airflow.
Speaker B: Okay, interesting. So I'd love to hear a little bit more about the custom operators part of the framework. Can you give me some examples of what that looks like?
Speaker C: Yeah, we have almost three or four kinds of operators, most of us for typical open source tools like DVD for example, um, another for our own data quality tool, and um, also some operators for Amazon Web Services.
Speaker B: Yeah, that sounds like a great way of leveraging Airflow's flexibility.
Speaker C: Yeah, exactly. Um, also it's the flexibility not also to develop those custom operators also to um, explain or document these custom operators maybe in some AI development framework. Um, because not only we, we develop the operators, also our agents know how to use it and it's simpler for the user to explain what they want to create. And our framework, our development framework explain how to create it and uh, how to use it.
Speaker B: Yeah, that's great. Cool. Well, thank you for going through that. This is a really interesting example of a custom framework. Sounds like it's really enabled your team to roll out Airflow more broadly.
Speaker A: This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams. From batch pipelines to training models and wiring up agents. Astro handles the infrastructure, uh, upgrades and scaling so you can focus on building. Check it out at Astronomer IO.
Speaker B: Okay, so I want to change topics a little bit and ask you about AI orchestration since this is a huge topic within the Airflow world right now. I understand from speaking with you before that you are using a spec driven development approach so that product and business users can contribute to templates without needing to know Python. So it sounds like kind of very similar in framework to the YAML framework that you've implemented just to allow more people to interact with your orchestration. But tell me a little bit more about how that works.
Speaker C: Uh, yeah, we, well we created m our own framework but it's not new. We based our framework on GitHub spec kit already existing framework. And uh, as uh, GitHub standardized we customize all the data life cycle rules and create new skills to help users to only explain what they want to create and understand what certified and available templates we have created. And uh, with that in any AI tool they can ask for creating or debugging, maybe some error in a dag and uh, all the tools uh, with that skills will know how to solve it or how to at least guide the user to solve the problem.
Speaker B: Interesting, that makes sense and tell me or I guess give me an example of what that might look like in practice. If a business user has a bug fix or some enhancement they need to implement, what happens then?
Speaker C: Well the full life cycle for us is when um, the process or a dagger fails Dax by the base template have a Slack alerting. So uh, every owner in Slack receives a notification with not only the logs of the error, uh also with uh, some explanation uh if it's a common error. So most of common issues are solved only by the user with the explanation in the Slack thread. But uh, if there is a deeper issue, maybe in some operator, those tickets are scaled automatically to the central sober team.
Speaker B: Yeah, that's really cool. I love hearing how you've just made Airflow and all of your orchestration so much more accessible to all of these other teams through these frameworks. That's great.
Speaker C: Yeah. Um, with that we have a lot of teams using the tool so for us that users can solve the bugs by themselves. It's very important because in the other hand we will uh, be a uh, bottleneck in the organization. So we don't want that to happen.
Speaker B: Yeah, I would imagine your team would spend a lot of time just fixing other people's bugs.
Speaker C: Yeah, exactly. Yeah. Yeah.
Speaker B: That's great that you've been able to do that and kind of free up your team for more uh yeah, just more new work uh, while also supporting so many business units. That's really cool.
Speaker C: Yeah, we think a lot of how we can continue um, improving that topic. Um, Airflow offers a lot of options. A uh few days ago we were thinking about how to uh, maybe trigger an agent when we have um, some alert or issue also from the Airflow instance, maybe to create a code um, to create a pull request with a bug fix. Uh, so Airflow have, have lots of features very interesting to explore and to implement.
Speaker B: Yeah, absolutely. I think that kind of self healing pipelines as you're describing is a very popular topic right now. It's certainly one that Astronomer is interested in as well I think speaking of kind of forward looking Airflow projects and features, that's a good lead in to my standard final question for all My guess which is what would you most like to see from the Airflow project in the near future?
Speaker C: Well we are very anxious about moving to the new Airflow 3 version because in this new version uh, you have solved a lot of uh, things we need to uh, maybe create some legacy code. Right now we are maintaining uh some of these features are for example the backfilling features, uh, for people that need to reprocess, uh, every. Any table. And another topic that it's growing a lot in our organization at least. Uh, it's event driven orchestration because we have a lot of, um, streaming pipelines right now and I think, uh, this is a very interesting topic. And another cool feature that we are expecting, uh, it's the Human in the loop for launching agents in the. In the same DAX processes. It's very useful for orchestrating, uh, generative AI projects.
Speaker B: Yeah, absolutely. Human in the Loop is a really cool feature. There's a lot of AI enablement or AI features to enable AI orchestration and the airflow kind of 3.1 to 3.3 where we're currently at. So yeah, great to hear that you'll take advantage of some of those.
Speaker C: Yeah. And um, not also to leave AI work alone, also to introduce a human review in that loop or in that processes. Yeah, it's a very cool feature for us.
Speaker B: Yeah, absolutely. Cool. Well, Lucas, thank you so much for going through all of that. It's been really interesting to hear about how you're supporting so many different business units with airflow. Uh, just to close us out, what is the best way for folks to get in touch with you?
Speaker C: Well, by LinkedIn, you can search for the company or our personal LinkedIns are open. It was a pleasure not only to talk, but also to share how we solved some problems. Uh, we have. Thank you for the space.
Speaker B: Yeah, thank you again for joining.
Speaker A: Thank you for listening to the data flowcast. Check the show notes for links to everything mentioned today. If you're enjoying the podcast, subscribe so you never miss an episode and leave us a five star, uh, review.
Other episodes covering the same guests and topics, from across The B2B Podcast Index.
Orion at Gravity: Trustworthy AI Analysts for the EnterpriseData Engineering Podcast · on dbt92 / 100
Is Your AI Actually Worth What You're Spending? with Parker ConradStrictlyVC Download · on dbt86 / 100
Code Review Is a Taste Problem | David Poll @GitHubHangar DX Podcast · on Spec-driven development81 / 100
323 - David Yanacek on 20 Years of Innovation at AWSCode with Jason · on Spec-driven development80 / 100
044 - Synthesia: Data Director - Why Data Teams Should Stop Trying to Be in Every RoomThe Stacked Data Podcast · on dbt79 / 100
From Billable Hours to Business Outcomes: How AI Is Rewriting the Rules of IT ConsultingEvolving the Enterprise · on Spec-driven development78 / 100
Using Airflow for diverse client projects at Accion Labs79 / 100
What's New in Apache Airflow® 3.377 / 100
Running Airflow 3 in a regulated environment at OTPP57 / 100
Managing a Customer Analytics Platform with Airflow at Skimlinks66 / 100
Building a custom Tableau provider for Airflow at JLR69 / 100