Building Trust with AI Agents - Henri Terho

Welcome to Software Testing Unleashed, the podcast for testers developers and all software people who want to create best quality software. My name is Ritchie. I'm a software-quality coach author & keynote speaker And happy to bring you an episode live from The Hustaf Conference in Budapest. Gustav is one of the greatest conferences I've ever been to.

Top quality talks, amazing people and that unique mix of friendly and intense atmosphere you only find at truly great events! If you have every get a chance go there seriously You will love it. My guest today is Henry Tero. He's senior consultant with broad experience in regulated industries And also he very passionate about AI & Quality Assurance.

And that's what we talked about. How can we build trust into the AI world? how to deal with all the AI agents and how Can We Deal With Testing & Quality in The AI Era, and now enjoy the episode! Hello Henry nice To Have You On The Show Here.

Hey, thank you for inviting me. It's really good to be here and all the cool props on everything that you have it's great. Yeah Thank You very much. And Zillac said To Me you Have to Invite Henry.

he was in Episode one. I think of podcast Okay? Now when He Says i have to invite somewhere. sure i will follow That.

i would like to thank Zilla But also your talk. so i know that its worth that you are Here. Very fun that me and Zilla did the talk together, because we kind of had this like a old test thing. Maybe if young AI is I'm not that young anymore but still it's they'll kind of like.

We have nice report going on And you're gonna do this again A couple more times so It's gonna get even better. So i really enjoyed. Yeah as well was great topic Because its'so important now. You talked about AI engines Especially About The Trust Thing.

Let's go into it. What is the problem? There are a lot of problems, but I think that the main ones have kind like this same problem we had with humans and everything else on there. But now because its an automated system We really need to solve these problems Because when you do it faster You multiply your mistakes And you multiply your successes.

So building systems where we can actually trust in AI In the way i can trust my colleagues and everybody else Is very important. Now with the current AI systems, we want to talk about systems and not just models. That you want a system around an AI model. be that source of trust?

You have a process, checks and balances, guardrails... And there's lots of stuff built around it! You go back to basics like the Oracle problem or communication problems. how do you communicate what I want?

The same problem with humans as well. Now you have to do the AI and also their Oracle. side of the problem is that how Do I make sure? That this stuff that comes out it's actually true or he's actually what i want?

Yeah, so those two things How would they build that? if you can answer those then you have the trust that we're looking for. That's a problem with humans too, as you mentioned. so yeah but you say...

we have to deal with it. what do I have? Do you have concrete example of this problem how its shows up in reality? Yeah if think about like you ask AI agent take a generic like chat gpt or something, you say that hey I'm small business owner.

i am doing this in software can make me a business strategy and of course from there on get the very generic business strategy. what do you expect to be? You have all points in it nicely done but then offcourse thats what they model does. its only predicts if doesn't think it doesn't analyse but works magically in that way.

Of course we got generate one. it doesn't have probably what you exactly have. It doesn't know what you want or, might be totally wrong. You are in a different country but didn't realize to say the hate.

actually I'm working Germany so take Germany into account But i am working Finland and selling this other countries. So take that into account Because typically when people interact with AI They kind of used to fix my code now, and then they are angry that why didn't this work? But you get the same experience with developers or anybody else. The production is broken, fix it!

Okay, it would be nice to know what's broken. Can you get me the logs? can help we figure that out? where do I find out The actual complaint?

What was broken? You know how to write a proper bug report in a concert. So this is the same communication problem now with AI than all bug reports and all of That. How do you convey actually what Do you want?

yeah because because as we must We've filled A lot Of those gaps by our experience pretty much. but then off course the systems don't have Experience And your not working With kind of pros But an AI model to computer. So you have to figure out that your prompt is good, your requirements are good and all of this that you're giving the system is actually good and specific enough so that the system can work. And I think it's a communication problem we have with humans.

We encounter daily in our job The other side... Well when talking about Oracle problems how do they know if what we did was right? and the AI model actually outputs something. And this might get really philosophical if you really dig into what's right in a company context, is it just that?

hey my boss asked for this? That may be totally valid reason to do it yourself. of course I'll do because my boss asks me maybe not the best idea but hey i was asked to do then how im measured so thats where i got pay from Or then the company will make money if this feature works or something like that. So you see these problems in everywhere.

and how do you build, or codify... these best practices now into your AI system? is what we have to do.. to build that trust of those systems as well.

And Is it possible? Can can be done with the AI systems from nowadays? I think so. The technology's there.

I can give the AI system my credit card, access to accounts and chat. All personal chats no problem but somehow i have trust that this system does good things with it. How do you do? Q&A testing validation all typical tools we already had.

Those are pretty much the same problem. You subcontract a piece of software from somewhere, how do you know that they've delivered what you want? Yeah! Do end-to-end testing or do acceptance testing...

or some kind validation? where did we get what wanted? and then it might fail because.. you actually didn't deliver proper specs.

so again communicating wrong things Or when in fact they didn't delivery Where can prove this isn't exactly what we wanted? So the same stuff happens. But in reality, I think it's hard for people to do their Q&A stuff with AI because they're such a fluffy unknown... Yeah.

And that is also part of the problem seen now with AI. That might be talking really technical about AI and how the matrices go How probabilities calculate. Somebody just started asking if there was an AI recipe for cake. When you talk about AI currently really hard to know at which level are we talking about and how does it impact me or what's going to happen.

And I think that, We also talked a little bit about the whats holding us back. is there lot of fears on. why do i fear making mistakes? Do I fear for my career that AI will take my testing career out from me?

those fears now can dominate. I cannot speak. Dominance mode kind of like now. that first, because it's a big change.

It is huge and thats why i think its interesting. but Its totally natural to be afraid at the level of change Of course. Yeah yeah Thats true Because there are huge impact in all over world Especially testing QA. But i think it will raise up qa To become much more important than just programming.

What do we have to do? Additionally, or special with AI what we are not doing with our software nowadays. Yeah good question I think it comes down a lot to repetition and doing like. when you think about software testing is typically very What's the word?

Like You input something and you always get the same answer. So what's the world for that? It's deterministic. yeah so testing is easy because well Software should be deterministic and if it isn't then you know, You have a problem.

but with AI models of course being non-deterministic. You Have to kind Of like shift your thinking in testing also not running Like. Is this green? No!

You're looking at over time...you are Looking At trends..you Are looking at like mean Time between failures. It's the same thing that machine shops airplane like these guys do That they're Not looking just one point test in a sense, but they are looking at trends.

So I think that's also going to change a lot on how we look at testing that you just cannot run a single test say it is okay and then go for because there will be Black Swan events. type of stuff happens all these models Because You Cannot Predict It. And Also Of Course The One Thing That We Really Have To Think About Is Not Just about the Testing and Validation But Building These Systems is that the models change underneath because companies update it. And how do you manage a system where components constantly keep changing, so to figure out some kind of stability and something too?

actually deliver something. I think that's a real change also in how we see software testing because typically, besides all the safety stuff... We have this case where you just see our expected result at screen okay? And it doesn't work because there is another quality.

now Yeah, I think that's maybe even like the separation between validation and testing kind of what's going on. And i think terminology will evolve as well. What are we talking about to facilitate it? We can more clearly indicate whether you're even testing or whatever You've been doing here.

That's gonna be also a big an interesting change in their field at all yeah Or Like As he said It's Gonna Be. I Think it's gonna Be much More About BDD is more up Much More About TDD stuff because that's one. How do I write down what i actually want in a proper way, make that into some kind of acceptance criteria. And then basically because code is going to become more and more black box Because you just let the AI generate an application for you.

Yeah yeah. Then their only source of truth Is your test and spec. If it passes this thing specified It's fine. The software think its fine without talking into the code.

Yeah, I like that idea. you mentioned BDD and TDD because often it's just used as tools to get something done with testing or test automation but in reality this is something we have got a same understanding of. what they are doing there As your mention. We need to do it with AI too So we need more information that the eyes know what it should do.

And I think one of the big things also is going to change, maybe too much spec in the beginning because everybody's gonna overreact and write huge specs then end up in very... In the same problems with code. now have millions of lines of spec which should probably be conflicting and maybe the systems, that's why behave in strange ways. And you're going to start de-packing I think your spec more than your code actually at this point.

so That is also gonna be an interesting change. The abstraction level rises quite much. So can i even... Because how do we write a spec for let us say salesforce?

It´s huge system. Please write me open. what does it do? I have no idea how to do that.

Yeah, yeah! That's true. but i think people get annoyed about it because if we want with AI... We wanted all very fast and quick.

just give me. And now i have put a lot of information had write down on the stuff and put into AI ...I think the people don't like it. But people don't like work.

Of course, if I can just say to my development team fix the problem and then go get coffee of course that's better than hey... I figured it out. It is this function in here. i found out This actually a problem.

Can you fix this one? And maybe change it to be this way. Yeah.. This more worked on first option.

Of course I want the first one! That was good shift in the view. How can we get that into our teams, this idea of doing that? That's a really good question.

But I think it also somehow naturally happened because in my career doing programming and doing stuff at the university are now testing on quality. I haven't experienced things like people from for example marketing come to me saying you're the guy with AI who could help us with these problems. Yeah, I like that hasn't happened with software. Hey you are the software architect.

could You help us solve this problem? yeah That hasn't happen. but now a day and you get a lot of these like inbound. hey This is how really cool.

we want to be part of it. And can you help us with this? yeah, and this way i also think that much Of the change is gonna come not from they're Like IT. It's coming up on the outside.

where there kind of like value is because who don't have this History of writing code or being deeply knowledge about my personal identity Being i'm sql expert which sadly I think is now going to be a less and less relevant skill set. But like, hey! My marketing...I need this tool actually track our clients and figure out what's actually going on of it.

just much closer the value with dangerous word but i think we can use it here. that close to the value creation at point up. yeah business thing they're doing in the softwares gonna enabling that is a much more, let's say approachable enabler because you don't have to know magic runes of JavaScript or Python. You can just prompt it and it democratizes the computers.

I really like that. Yeah, I think this is one big point that now everyone can build software more or less. But quality question. We don't know it and we as testers...

One of our main skills why are testing? Is when you're building trust into software what do you think? how does changes in the AI world get its most important for us to be the same? What's your opinion on that?

It also shows a lot of the biases that we have. I'm actually doing a little bit research on my PEG topic about how we've done major business decisions just by doing software, because if you didn't realize some coder has made major business decision like locking us to Azure or one day. Nobody even thought back in today. and hey great!

We got application done but it affects everything. so big thing is going happen now. We have to understand a little bit of the business side as well and figure that out. And how do you make sure in many steps, we are building right things?

Again it comes down to making sure there is some kind of platform for doing this so they can have a single source of truth about what your company does. That's also going be interesting. You all have to start writing down. what do you think we actually are going to do?

Because AI is gonna be the number one interface. So that's an interesting change, figuring out how to communicate it with everybody and everybody like hey! We've got this project AI which knows about all of our documents on projects. Ask him...

ask the AI for questions when he was onboarding For example. And then in other side as well. How can I make sure they're right? We need to build systems again on a platform or some kind of system to check like multiple AIs.

discussing this, that is actually a good thing. Some of these problems are kind-of. you hear them and think they're easy. but for example You might have SharePoint where we had twenty different versions of the document which was right one.

It's super difficult problem when thinking about it? When did lot context? Did we send this to client? Which ones did we sent Have improved afterwards?

And how do we figure out? It's now surfacing a lot of these problems that humans have solved. The problem we had already and didn't solve in the last game! We can solve them with AI, but they're bubbling up to the surface saying actually you need something about this.

so I think it is also quite cool. What do you think? As testers as quality people what are our skills today? how should we evolve that work.

I think we are in a very low position, but i think the testing skill set and mindset is in a good position for AI because as said it's much more about hey what are criteria they're actually doing? how can you write this into spec? How do we write these things to test case ? How could be verify what were doing so that baseline already really good compared many other places in IT Because thats basically main interface to interface with these systems.

and how do you design them? Yeah. And on top of that, kind of like start figuring out How the build this kind of Statistical testing system such as it's not your know make test go green type Of a thing they're very care about one single test. but These trends and how they go and how Do You Not just work As a tester in Your own small box But Like Expand your skillset in many other places In The whole Thing?

because I think Testing and validation is gonna eat parts of DevOps is going to eat part of programming as well. So expanding in those directions, I think it's gonna be something that's very valuable for you as well. and what we've heard talks here from Robb on many others talking about teach people multiple roles so they'll make a better tester because then have more context. yeah the same thing with AI if there are more contexts.

but Well, of course I'm biased because i'm a tester as well. But the skill set is already very suitable for the new world and that's why I wouldn't be scared. The number one fear that I've tried to allay today too many people That AI isn't going take your job. They're not gonna automate you.

It's going to augment you and the way that you work. And there is a lot of work done around this area, so... I think it would be nice if we have also this mission or this vision for our community! Henry, thank you very much that you were here and gave us your insights from the talk.

It was really great! I appreciate it. And yeah i wish a good rest of the conference. It's now sliding in the finish here...

and hope we see as soon at another conference or maybe next year? four weeks, we're going to be an automation start doing the same thing again. so version two point zero. So it means that I'm gonna be there and continuing the journey of telling this too as many people as you can!

Great to hear us yeah thank you very much. have a good time.