37 - Building AI Responsibly While the Rules Are Still Being Written - with Rishu Gandhi, Senior Data Engineer (Fortune 500 Bank)

Honestly, the conversation themselves gives me optimism, like the fact that we're having this conversation. And nowadays I've been seeing responsible AI showing up in job descriptions, engineering blogs, and broad level conversations. And that is the real cultural moment and not just the noise. You are listening to the Product Builders Podcast.

Each week on the show, we bring you conversations with experts and innovators building digital products. Our conversations help you gain behind the scenes insights into building some of today's most innovative companies. Subscribe and be sure to check out our website for more at productbuilderspodcast.com.

is Rishu Gandhi, a senior data engineer in machine learning and AI cybersecurity at a Fortune 50 bank. Rishu, welcome to the show. Thank you so much, Mark, for inviting and I'm very excited to talk about Responsible AI. Likewise.

And before we dig into that topic, I do want to start at the top and give our listeners more context about your background. So when your friends or family ask what you do, how do you explain it in a way that clicks and that they can understand? Yeah, that's a very good question. So I usually tell them that I build the bridge between the data that has to travel from point A to point B or to an AI model.

So think of data as something that needs to, let's say, travel from point A to an AI model. So I, as a data engineer, I build the pipes, the filtration logic, and all the routing. So by the time it arrives, it is clean, structured, and actually useful. Because whatever data that we feed in to our AI model, we want to make sure that it doesn't have any historical biases, it doesn't have any null values, it doesn't have any duplications.

So the AI learns smartly. I would say I'm the kind of that glue that kind of brings both the sites together. So it's a part of architecture, it's a part of like plumbing, and part of quality control. Great.

I love that analogy. That definitely helps understand what you do in an easy way. And when we use the term responsible AI, I think that might feel abstract sometimes. Can you define what that means in practice?

What does it actually mean when we say responsible AI? Sure. So people hear responsible AI and think it's some kind of buzzword. But in practice, it really comes down to three big questions.

So can you explain what your AI model did and why? Another question is, can you catch it when it's wrong before the damage is done? And can you stop it before it causes any real harm? So if you cannot explain a decision, then you cannot defend it.

And if you cannot defend it, then you probably should not be deploying it at scale. So responsibility is not something you kind of figure it out, you know, at the end. It is, it shouldn't be a part of your design when you build an AI solution. So that's how I would define it.

That's great. And I'm sure we'll come back to those questions later in this conversation. But when did this work become something that you were really interested in, you know, making sure that responsibility in building these systems was integrated into your flow? Yeah, honestly, I would say it was not a one specific moment at work or anything.

It was more watching the broader trajectory where the AI is being applied and how fast it's evolving. You know, now we have started applying AIs into domains like healthcare, hiring, criminal justice and financial systems. And let's be honest, these are not just fancy technical experiments, right? These are systems making real decisions about real people.

If somebody doesn't get a job because an AI model chooses not to filter his or her resume, a patient gets a wrong recommendation because an AI made the wrong call and the person affected has no idea that AI was even involved, let alone question it. So it's that accumulation of risk across so many domains is what made this feel very urgent. And I realized that people building these systems, people like me are often the last line of defense before something gets deployed. And that should not be.

So there is a weight worth taking this seriously. If not now, then I don't know when. So that's why I believe it's really important. Awesome.

So let's zoom out a bit and set the stage. I think one of the things I mentioned at the start of this conversation was how fast technology is moving. You don't have to work in tech to see how quickly things are evolving on the AI front. And that's critical to our conversation today.

And I think the interesting dynamic that exists is that AI is moving so fast, but the rules, the guardrails, the frameworks that you're talking about are not. So that creates a lot of gray area for people actually building products. Can you give us a sense of where regulations stand right now in relation to keeping pace with the technology? So yeah, the honest answer is regulation is real.

It is growing, but I believe it's still playing catch up. And the landscape is genuinely very fragmented right now. The EU is the furthest along. So the EU AI Act really classifies systems by risk levels.

So there's unacceptable, high, limited, and minimal risk. And it imposes real obligations around transparency and human oversight, with full enforcement starting, I believe, this year in August. In the U.S.

, we are still playing catch up. The previous administration tried to impose some regulations, but then it was reversed because they wanted to focus more on the innovation and not more on the regulations. But I think we are kind of paying the price in a way. So I would say it's still catching up.

But on the standard side, I know that NIST released a cybersecurity AI framework profile in December 2025. So the industry is trying to self where the binding law does not exist yet So we do have some regulations and I think some are catching up So let hope that we come up with better regulations Thank you And when there aren clear rules yet or kind of abiding organizations who becomes responsible for defining what good AI looks like? Is it at the state level or is it truly down to the product building level and the organizations investing in tech?

Yeah, you know, I would say it's not just one department's job. Every decision is, I would say, responsible action. So I would say it falls on the builders, whether we want that responsibility or not. So because when there's no rulebook, right, the product teams, engineers, data scientists are making ethical judgment calls every single day.

And here's what I always push back on. It is not one department problem. The product manager setting the success metric, right? the designer shaping how output gets communicated, the engineer deciding what goes into the training data.

So every one of us is equally responsible, whether it's labeled that way or not. So I believe when you don't have a rulebook, your design decisions become the rulebook. I love that thought and that insight that it's not one department's problem. I think it becomes very easy for people to pass the buck and say, you know, it's the responsibility of the UX team or the data team or this team.

And when that happens, things fall through the cracks. Responsibility gets thrown out the door. And so it's very easy to say, you know, we tried our best. But in actuality, all you did was just pass responsibility to other people and nothing happened, right?

Exactly. So from your perspective, what are some of the biggest risks if responsible AI isn't truly taken seriously? Yeah. So I would say to demonstrate this, right?

Let's say I'll give you three concrete domain examples where the stakes are real. So let's talk about social media. So let's say when an AI recommender is built purely for engagement, optimizing for clicks and time on screen, we may get a lot of misleading contact, which ends up creating echo chambers and social unrest. So the model kind of hit its target perfectly, but the target was wrong.

So that is an irresponsible design by choice, right? It was not a technical failure, but we're still not being responsible to what we're showing to our users. Just because it drives engagement doesn't mean it's right. Another domain like healthcare, right?

Let's say there's a medicine advisor AI app without high accuracy or in low biases that can recommend the wrong treatment or reflect historical bias based on the training data. So it may suggest one medicine to a patient that it suggested another patient, but every patient is different. So as a doctor, they cannot just trust the AI app. They have to be able to explain it, why it reached that decision.

So in this case, the risk is we're dealing with human lives. Similar goes into like financial and recommendation system, right? Let's say you have your AI financial advisor, which optimizes for maximum profit over quality. So you end up getting biased recommendations and a harmful experience.

So in all these domains, the stakes are really high. That's great. And one of my questions that I was going to ask is, are there specific verticals where risks feel even more critical, right? So I guess healthcare, finance, when you're dealing with things like that, that truly affect the individual, that's probably where you might see it.

But on a similar line, do you think the public perception of AI is going to also shape how companies approach responsibility? Do you think public is even aware that it's something that they should be considering and concerned about right now? Absolutely. And the fact that we're having this conversation because we know that it is highly urgent and it is something that every company kind of needs to look into their own models that they're building and ensuring what they're building is actually right.

So if you like, you know, in terms of, I would say in terms of Publix unrest, it's very much real now because you see there's new models that come out every single day. There's always something new that's evolving in the AI and it's becoming practically impossible to really play the catch up at this point. But, you know, it's more scarier that we don't have the right guardrails. So I believe people are increasingly becoming aware of the situation and we're all hoping that we enforce the proper guardrails.

And then you mentioned this earlier in the US, right? We have prioritized innovation over safety, if you will, right now. Do you think companies fully understand the responsibility they're taking on when they start considering and introducing AI into their products? Is that something you think they're aware of or are they also still playing catch up?

I believe now they're definitely becoming more aware because now I've seen that within the companies, there are designated AI teams, more like a risk teams, which have incorporated all the regulations regarding AI and they want to make sure that all the solutions, they kind of go through that, right? And there's another, I guess, business aspect of it too. So let's say if a company is building an AI solution and obviously they intend to make their solutions globally, right, for targeted to global audiences.

So if they want to comply in the EU, they're going to have to comply with the EU's AI Act. In order to do that, they're already kind of putting their regulations and even if their user is sitting in Asia or in United States, right, because they don't want to build two separate AI solutions for two regions. So in a way, I think it's improving for sure. Great.

Right. And so we've already spoken about this. The responsibility right now, at least, is heavily leaning on the actual product builders, the people making decisions about how these systems behave before any formal rules are in place. So if that responsibility falls on the builders, the product teams, the engineers, the designers, the next question is what that looks like in practice, right?

Because I think, I hope most teams aren't trying to build something irresponsible. They just trying to move fast and ship So I like to get into how people should be thinking about this as they build So that said when teams are building AI driven products are there principles that they can keep in mind to start laying the foundation for a responsible AI in their product? Yeah, absolutely. So I think, you know, a few things, right?

So a few things that actually work in practice. So let's start with input validation, which is basically you control what goes into the model, which is again, the data. So if a system was designed for a specific type of data, then let's make sure and enforce that. So we do not let it improvise on the inputs it was never built for.

So this is actually falling behind the idea of whole RAG solution, right? Where we provide a specific domain knowledge so that the model doesn't hallucinate a lot. So this is where the input validation and all those things come into handy. Another is confidence thresholds with human in the loop process.

So high confidence cases get automated, it's okay. We have the AI so we can automate decisions that we know we're highly confident about, but the ambiguous ones, we can have them escalated to a human. And this is, I would say, especially critical in agentic AI systems, where knowing when to hand back control is one of the most important design decisions you can make, right? So it's not automation replacing humans.

It is automation working with humans where it matters. Another one I would say in practice is production monitoring. So this one is underrated. We sometimes think that, okay, I've tested my AI solutions a lot and I know it's doing what it's supposed to do, so I'm just going to deploy it.

Sure, but the model's drift. The role changes. A model performing, well, six months ago might be quietly degrading today, so you need to watch how it behaves after it is live and not just before. And another is, I would say, clear ownership.

So someone must be accountable if something was wrong, not just who wrote the code, but who owns the outcome. So that accountability gap is where most failures slip through quietly. So I think all of these are some of the things that we can really put in practice. Amazing.

And I know when it comes to AI, another big key is transparency, right? About how data is being used, et cetera. Can you speak about why building transparency and making that a key principle as well is so important for building trust when we talk about responsible AI? Yeah, absolutely.

So let's say, you know, you go to a doctor and doctor said, hey, trust me and take this pill with no explanation. You would be very uncomfortable, right? Like even if the pill works. So AI is the same in that sense.

Let's say if a system makes a decision that affects someone, a hiring filter, a content recommendation, like I mentioned in the earlier example. So let's say it does that, but nobody can explain why. The people impacted have no way to question it, appeal it, even understand what happened. So that is not just a UX problem.

It is a fairness problem. So it is transparency is everything because it protects the builders too. Because when you can explain the system, you can defend it. You can also audit it and you can improve it.

And when you cannot, you're just kind of flying blind, hoping nothing breaks in a way that matters. So transparency is not just about earning user trust. It is building something you yourself can stand behind. The first thing that came to mind when you talked about the analogy, I got a vision of like the matrix when you've got like the red pill and the blue pill.

Yes. And like Neo just had to take one and trust it. So I think very relevant analogy. Thanks for sharing that.

And this is similar to my previous question about principles, but when we talk about guardrails, are there specific guardrails that teams can introduce when developing AI products as well? Yes, I think similar to those practical tips that I mentioned in the earlier question, similar ways. You need to make sure the input validation is real. You really need to have the control over what goes into the model.

Confidence thresholds with human in the loop. We need to make sure that for those edge cases, we keep the human in the loop. The idea is not to replace humans. The idea is to let AI take decisions on those tasks, which are very simple.

and we know we're confident, like we can confidently trust the AI, but the edge cases, we should keep human in the loop. I don't think we're still there yet to have fully AI automated and we should not be for multiple reasons. So that is definitely another guardrail. And like I mentioned, production monitoring, once you deploy your AI solution, but the monitoring doesn't stop there.

You need to be able to monitor your model, how it behaves. You may need to fine tune it. So that is equally important. And having clear ownership, someone must be accountable if something goes wrong.

So there has to be guardrails, I would say, in the entire SDLC or MDLC lifecycle process. Yeah. And you said something earlier that really stood out to me. It was that we need to know when to hand back control to the human.

And it is not about AI replacing us. it's about being smart about its use of AI within our workflows. And I think that really shifts the mindset from, you know, moving fast at all costs to moving fast, but with intention. Because I think those early decisions really shape everything that follows.

So one of the things that keeps coming up in this conversation, right, we've been talking about trust a lot, because you can build something incredibly powerful, but if people don't trust it, they're not going to use it. So I do want to spend a few minutes talking about how this evolves from here. What actually creates trust in AI systems from an end user perspective? Yeah, so I think in terms of the trust, like I mentioned, your systems need to be explainable.

So you need to know why it took a certain decision based on what were the metrics that your AI model kind of used to build a certain decision. and you need to be able to trace it back to, this is the decision that it made based on these features. What data did it use? So you need to be able to trace it back to the data it comes from because let's say your data can be biased, right?

Your data itself can be clustered around one specific domain. So when I say biased let say you are building some kind of AI solution to filter out the people resume right But let say if the data that is used to train that model if all of them are coming from a very prestigious school right So it's going to take that as a reference and think that, hey, anyone that does not filter these very prestigious qualities, I'm just going to remove it. But what if it could be true for one type of job description, but what about the other one?

So we need to be able to really tell why it made a certain decision. And that's how it works in humans, right? Let's say when you make a decision, I come and ask you, okay, Mark, like, why did you choose that? What was your thought process?

That helps me understand what are the different dots that you were able to connect. So I want to be able to do the same thing with the AI. So there are different ways to say the same thing, but it really comes down to this core logic. And you mentioned something that I want to dig into, but how do you communicate AI decisions to users without overwhelming them?

Is it built into the flow itself? How do we educate people into how our information, how AI is being used, all of the things surrounding this topic? Yeah. So again, this is where the guardrails and the regulations also come into handy.

As per some regulations you are required to say, or maybe it could be in different forms. One could be that you're citing your sources. So I know I'm, I don't want to name them, but like some of the tools, like, you know, when you're actually like, some people are using it for research and those things, you can go and let's say you asked something in a chatbot, right? It gives you where it got the information from, like it gives you citations of the sources.

So again, that is one way of letting the users know is that, hey, this is where I'm getting the information from. So all those things are becoming increasingly important. And it is part of kind of having your users trust your solutions more. Because if you want users to adopt your AI solutions, then they have to trust your tools to begin with.

So it is important for larger adoption as well. Awesome. And I know we've talked about the industry catching up to things, but looking ahead, what gives you optimism about the future of AI? Yeah, you know, honestly, the conversation themselves gives me optimism, like the fact that we're having this conversation.

And nowadays I've been seeing responsible AI showing up in job descriptions, engineering blogs, and broad level conversations. And that is the real cultural moment and not just a noise. I would say maybe like a few years ago when the first chat GPT was released, I think at the time we were all like, oh, there's this really cool thing. Like, I'm gonna go try it out.

I'm gonna, I can do this. I can do that. In terms of imagination, like we thought like we could do so many different things. but then now we're kind of taking a pause we're still doing amazing things but it's like okay but is it safe so i think that i do see that cultural shift which does give me optimism that i know ai is here but it is something that we can utilize for advantage and we can use it responsibly so the technology is i think also maturing in ways that make responsibility more you know more achievable like rag is now a production grade system and gives you much more control over what the model knows in sight.

So I think that all put together, that does give me a lot of optimism. Yeah, I love that. I love that sentiment. I also see it myself in that teams are thinking about things a lot earlier on in the process and not just layering it on top of something at the end, which is great to see.

Obviously, we have a long way to go in terms of building true responsibility and standards across the industry, but it's nice to see that people are considering it from the front. They're having the conversations like us. And so I know we've covered a lot here today. Before we close out, I do like to do a little segment where I throw a few statements your way.

And I just want your quick take, no overthinking, just kind of gut reaction and maybe, you know, 30 seconds or less. Yes. So let me see here. What is something about AI that excites you that people aren't necessarily talking about right now?

What really excites me is that there are certain things that I, certain tasks that I keep doing daily, what if AI can take over those tasks for me and then I can focus my valuable time into, you know, doing something that I truly enjoy. So it's just a thought. I mean, but I think that's something that I think about all the time. Awesome.

Is there one word you would remove from the AI conversation right now? Wow, that's a really good question. I think one word is, you know, it's funny, but I would probably removed the word AI itself because everyone's like, oh, you know, that is AI. Something happens, right?

It's like, oh, it's AI. Not everything is an AI. Sometimes it's a simple automation. So I think people are really kind of, people assume that it's just, it's AI, but sometimes it is just optimizations.

It's automation. And those things have been here long before AI came in. So that's my take on it. I love that.

Let me do one more here. Another one word question here, but what's one word that defines the future of AI for you? I think regulation for sure. Regulation will define the future of AI.

Awesome. I love this segment. I know I kind of threw some things your way, but loved your take on the questions I posed. Risha, this was such a great conversation, especially right now when so many companies are moving quickly with AI, but not always taking the time to consider the responsibility that comes with it.

We'll link to our work and anything you'd recommend in the show notes. And for those of you tuning in, thanks as always for listening. If you found this episode helpful, please give it a share. This is the Product Builders Podcast, and we'll see you next time.

Thank you for listening, and we hope you found this episode insightful. If you enjoyed the show, please leave a five-star review. You can find more information and links to all the resources mentioned in today's episode at productbuilderspodcast.com.

This episode is brought to by Majestic Apps. We imagine, design, and build digital products. With clients like Chevrolet, AudioMac, IBM, Barefoot, and more, you can be sure you're in good hands. Reach out to us at MajesticApps.

com.