Boilerplate in Seconds: AI Handles Setup, Engineers Handle Logic - Klaudia Dussa Zieger

We are at the position that we need to decide how much liberty, how much autonomy do we want to give these systems? So it's like Going to the left, going to the right, going to the middle. What do we want? An autonomous agent that is able to book Appointments for me that is able to do some money transfer stuff for me, and I have to give him access to all the systems From my experience in projects, usability is the first characteristic that is dropped if time is really an issue.

I think with the advent of more autonomous systems I think the the gap between using AI and testing AI is closing again because the more autonomy you give to an AI agent, the better it has to be. Welcome to Software Testing Unleashed. I'm Richie and I'm happy to share another episode from the Testcon 5 conference from Mauritius. My guest today was Klaudia Dussa Zieger.

We talked about how to use language models in Testcon. Testing in a very very practical way. Why small functions like writing keyword documentation work so well, why reviews only got good with reasoning models, and where AI is still No help at all. That all you hear in this episode, and now enjoy Hello Klaudia, great to have you here in the show.

Hi Richie, my pleasure, my pleasure. It's the first time you are in my English podcast. That's true. Two times in the German one.

Mm-hmm. And now the first time. And we're we're both from Germany. I'm from Austria, but it's still counts.

Yeah, it still counts. And uh but we are here on Mauritius, uh the test conference. I I I like the the spirit here. It's a very active community in software quality, software testing here on Horizon's Yeah, I think you can feel that it is a a rising community.

It hasn't started all that long ago. I think the board now is six or seven years old, something around these lines. It's the th their fifth conference, so it's still relatively long. And you can see it the I think also the average age if I compare it to conferences in Germany is way younger.

Way younger, perhaps not as young as the conference in Kenya, because they were also very young. But yeah, it's a very young, lively, energetic community for sure. And you gave uh a speech yesterday, uh uh one of the first slots here, uh, about LLM in testing Yes. And uh that's I think a good point to start in in to look a little deeper because since we have LLMs now on the plate since a few years and Often some speeches is are only about yes we have to use it, but they are not very specific and not very concrete And uh now we can maybe go a little bit deeper.

How can we really use this stuff in testing? Yeah, okay. So we have been looking Let me perhaps start a little further or a little going a little bit further back. We started out actually trying out different AIs already in 2019.

But that was a visual classification algorithm in order to recognize moving objects on the GUI. No LLMs, no generative AI, no neural networks. But for the last three Three and a half years we are looking into applications of language models, how you could actually employ them in a smart way in software testing, and given the fact that we are working with a lot of text with a lot of language, specifications, test scripts, reports. Yeah?

We thought it might be really worth trying. And then we tried to do little pieces. So our intention was not to come up with a fine-tuned general testing LLM but really have s small functionalities that would really benefit the user. And we started with the test specification.

Actually that was a a larger prototype where we used an approach where we fed in additional project specific information. So it was a RAG approach, retrieved augmented retrieval augmented generation. Where you use with a standard generic model and we started out with Lama 3-1 And then we move to clothes three five and just by the numbers you can see it's already a couple a couple of years ago. It's like legacies of you know, one of the a side note, one of the issues or or I think the challenges I personally have with with the language models is they're evolving so fast.

The frequency is going down. At the beginning it was about half a year, and now I think every couple of weeks you have a new model, which is really tough. Tough Yeah. Yeah, but coming back to the language models, yes, we used it to look at requirements and derive test specifications out of it.

And that after several iterations, because at the beginning we had a complete cycle without any human intervention that worked but not too well. I mean the intention was to actually create something so that you wouldn't start on a white piece of paper, but you would start with a draft version. So in order to get around the the the writing block, yeah, so that you could have something you could work with. And then we presented that to the testers in that company and they felt a little bit left out so we thought two birds with one stone.

improve the quality of the final result and get them into the boat so that they can see that they have a say to it, that they can control it at least a little bit. And they had a very strong opinion. that they wanted to share their experience. So for them it was important that they had one spot where they could feed in additional information.

And that's what we actually did. Which can be very valuable too. Yes, yes So that worked well and that was the starting point. And then we decided, um, and now I'm speaking not as the ISDQB president, but as um a consultant coming from IMUS and we do have a test management tool.

And then we decided to add little AI functionality in the test management tool. In our tool you can um Do test specification using keywords, not only for test automation, but also for manual testing. And given the fact that the keywords are basically code, right? It's a script, it's an automated script They are doing programming and they don't like to document like most of the programmers.

So the first tiny little bit that we actually wrote within the tool was a functionality where you press a button It reads through the implementation of the keyword and it comes up with a header. So you get a short header, you get the description of the parameters, you get a l longer description what the keyword is doing. They just have to press a button Read through it and then they're done. And they were quite, you know, very easy, very small test, but they were quite happy with it.

And the amazing or the the the result now is that the descriptions are more unique. Not unique, uniform. You know, everybody now uses it. It's not doesn't very much so that would work really well.

Then the next thing was given the fact that we have a distributed development environment or distributed team, the quality of the implementations of the keywords was different. And we said ah why not use an LLM for reviews? Given the fact that we have Rules how keywords need to be implemented. We have a glossary which terms you're allowed to use and stuff like that.

So we played around and try to feed in as additional information our rules, our glossary and see if the LLM could do a good review. And there it really turned out that you needed to be very specific about which LLM you're using. So the Normal Chat GPT versions didn't work, you had to use one of their reasoning models. So we ended up with O1 and did it there.

And it worked actually quite well. So it catches most of the more obvious stuff but you get it very fast you get it like this right within a second or two you do typically two or three cycles and then formally the keyword looks pretty good And then the impl uh the the test automation guys, the engineers came and they said, then we can look at the logic of the keyword if everything is fine. So they also appreciated that. So that that were like the first two little implementations we had in the in our tool in the test bench.

And then the next one was actually coming up. when we decided um that we needed a translation functionality and that kind of sounds a little funny but it works really well with language models A translation functionality from a technical defect description to a description which a domain user could understand. I mean if you anticipate some erroneous behavior you typically have a very nice error message but if you really run into a technical problem where you ha get a defect message which is thrown by the system it's typically very technical And since we are working with a lot of domain testers, they don't understand what is in there.

And we figured, well, if we feed the defect message plus the test case that led to the defect into an LLM it could do the translation language right it's language again and that also worked very nicely. That sounds great. I I like their approach to to to to look at different spots in there And when you said about the review, uh I one question because uh when I look at development teams uh who are now using uh AI for reviewing, they are very happy. uh to have a neutral instance uh more or less and objective not the senior dev who is blaming them or uh another who is uh do you have any feedback from the From the people who get whose code get reviewed uh by by your AI?

Um no, not really. We got feedback that they not with respect to this issue, so that You have somebody who's blaming somebody else. So that was not the issue. But they were very happy that they got rid of the tedious reviewing task where you can just come up with the plain simple stuff so they could get rid of all the boring stuff and really focus on the interesting issues.

So that was their feedback. The other thing, perhaps because the the team is The team knows each other despite the fact that they're in two different locations. They know each other for more than ten years. So they're very well connected.

And I don't think that that was really the issue. At least they didn't tell me, let's put it that way. But they were happy to get the boring stuff out of the way in a very fast manner and and that worked really well. So that was actually the starting point.

And then when we talked or when I talked to a couple of our colleagues, of my colleagues, um, they were using Copilot for coding without even talking about it, right And then interesting enough, when I talked to one of the guys just floor below me, he said he's not only using it for coding, he's also using it to develop scripts for Vagrant so that it can set up the environment automatically and I thought that was actually quite interesting because that is the DevOps approach, right?

You build up the test environment automatically And he said, yeah, it's kind of a boilerplate thing. He gets 95%. The last five percent he has to do by himself. And he knows what the system is supposed to look or that the script is supposed to look like so he can easily check it And for him it's just also a question of efficiency.

He just puts it in, gets it within seconds. It would take him perhaps half an hour, perhaps. Perhaps 15 minutes, perhaps a little longer, but it's faster. He just has to review it and then he's fine.

So that also works. And did you uh have their um more iterations to get it run uh with review or is it just a one shot where the review thing yeah no the review thing took six took several iterations because we started out I think with chat GPT40 And it didn't work too well there. And then, you know, i if you look at the language models, you have to really be careful. about which year you're talking about because the reasoning models haven't been there from the beginning.

So we started out with a normal chat model. And then we we saw coming looking at the results that they were not good enough and then we figured perhaps we need to move to the newer reasoning models. and really try out a reasoning model and that did the trick. So for the documentation of the keywords we use the standard model.

For the review we use the reasoning model. So you need to There isn't I think I think there won't be the one generic solution that covers everything. I think you need to be more specific what you really want to do and you need different skills for that. I think what you say is so important that uh we have to be clear what is the problem we want to solve and and you specified very very clear problems you w or or things you want to to resolve by by AI.

Actually you know for us it was an experiment. Is it gonna work? Can we make it work or not? And the the same approach we also took as an experiment or perhaps even more as a challenge when we tried to look at the usage of large language models for the digital accessibility tests.

Because we have a group that just deals with digital accessibility testing and we always had a discussion how much can you automate? And they were all 40%, but we can't do more because if we look at the text, we need to evaluate it and say it's okay or if it's not okay, so there's always a human person needed. And we said Let's just try it. It's like challenge, accept it, let's just try it.

And we'd also just tried it out again as an experiment with this new agent technology, the setup where you use NCP and stuff like that. Um and we just used it for the search of the alternative or or for the generation of the alternative text for pictures, where you need to interpret the picture. That for me was actually the I thought that would be the challenge. And and it worked.

It worked. Again, it doesn't work perfectly, so sometimes the system is proposing something where where I would say it's okay, but where the specialist for it uh digital accessibility said it's a duplication, it doesn't make sense but it still gives them a gain in efficiency. So it's just doing it faster. It just goes through all the picture like uh on the fly.

And then you get the result. Again, you get the results back and you as the human in the loop have to look at it and say, it's okay, it's not okay. To understand that we don't have don't need the hundred percent solution out of the so we can review it when we have the the people there who who see see what the outcome and then can reflect on that. Yeah.

Yeah. And I think this is also Right now for me the the the the interesting point because I think right now given that MCP was developed and MCP the the model context protocol where you connect different uh functionality for AI systems for large language ones that was just developed and released in November 2024. So a little bit over a year, not quite one and a half So everything is very still very new and we don't have years and years of experience. And with this technology, the number of AI agents that run autonomously is increasing.

Yeah, now our first version, there was nothing autonomous. We told it every step what it's supposed to do. But now it is increasing, and I think right now we are at the position that we need to decide how much liberty, how much autonomy do we want to give these systems? So it's like Going to the left, going to the right, going to the middle.

What do we want? And then and and again we're back to some of the basics of testing. It's a risk-based approach again Do I want to accept that if I have to give this the autonomous agent my banking information, my health information and stuff like that? And you know, even if I look at Austria, you coming from Austria, open claw from the Austrian developer.

Scary scary. Scary. An autonomous agent that is able to book appointments for me that is able to do some money transfer stuff for me and I have to give him access to all the systems. Well interesting if you want to put it nicely.

I think a little scary, but Correct, correct. And and I read in an interview that he has the vision that we will have personalized AI agents And they should be as so easy that his mother can work with them. So that is a completely, for me, a complete new level, right? Because I'm pretty sure that his mother will not be able to judge whether the thing is working correctly or not.

But the level of of automation, the level of access rights you have to give OpenClaw that is Woo. Yeah, yeah. You see well come back to t to testing. Did you uh did you have some experiments where you then said okay it doesn't work?

It it is not a way we can go there with an I AI. Yes, we uh well experiments perhaps is a little too strong a word. We have um two projects that are actually doing a little bit of reverse engineering and for older existing embedded code doing unit tests because they need to do it due to regulatory reasons. And that is very cumbersome.

It's like a little bit like arcology, right? You need to look into it, you need to look into the details, you need to figure out in C code. And I was asking and actually one of our X AI experts is working in these projects and I was asking him Do you think an AI could help your language model, could help your kneeset? No.

No. Because you need to look into the existing code so it's not about building code. It's doing the unit test for these codes. And they don't want to just cover it on the line coverage basis.

They really want to write test cases that make sense and he says that level of expertise is not there. He doesn't see it. It's not like writing a header for a keyword. It's not like reviewing against rules.

You really have to Concentrate, dig into ex into the existing code, which of course is not well documented, right? So I don't want to say how good or bad the quality of the code is, but definitely it's not documented There are no requirements, so you have to do everything just by looking at the code and from his point of view for that kind of for that particular problem he says no. No I think you would need to invest a lot of time to train the models or somehow to get the models up to the level that they could understand it.

And by that time perhaps That's his viewpoint. He's doing it by himself. Or actually it's a team of four, but they're doing it by themselves. So he said, and I was just asking that question last week because I was very curious.

He said no. In that particular instance doesn't see an opportunity. They're not good enough yet. uh the higher level of test automation.

Where do you there see possibilities to use LLMs or is it not really Well I think for for test automation for the coding itself you can use Copilot and I think that would work as well as for normal programming. I mean automation is nothing else than programming. So yeah, I think that would work. Um if we are looking at the test automation on the system level, if you're going through the graphical user interface, I think you will have a number of AI applications that do the recognition of the objects that if they move or if they're slightly named differently or stuff like that.

I think that would also work. Um there have been applications that you use the word embeddings from LLMs in order to find Different path identifiers if your first one breaks. So that goes into the direction of self-healing test cases. There was a presentation uh by Manika Shihari at the RoboCon 2025.

And I was just asking this year, the Robocon is always in February, so that was two weeks two months ago And that I they're not pursuing that idea anymore. So um I need to dig a little deeper why they're not doing it. When it came up last year we also had a discussion with our specialist and they said If they use the embeddings in order to do self-healings, of course they are covering up real mistakes. Yeah, and it's hard to say Whether it was accidental or whether there's a real mistake where they really should look at.

So there are some I mean definitely for the normal coding Um and for the more it let me call it more advanced stuff, um we'll need to see. Yeah, okay. When uh we can we cannot really predict the future. uh up to now with AI it's it's even not awesome.

It's crazy, honestly it's crazy. But do you have any any wish uh for the AI To support us in the testing process where we have now a leg or a gap now. What do you can you imagine where the AI will help us in the future as a tester? Where we have a gap or something you wanna get rid of in the testing process.

I think if the AI would really give us the the speed, the efficiency, that we could get the the typical stuff like all the functional tests done so that we can concentrate more on security issues, on performance issues, on usability. We were just discussing it yesterday at the conference. From my experience in projects, usability is the first characteristic that is dropped if time is really an issue, right? because functionality has to be there and then performance and then security.

So perhaps to speed up the process that we can take care of these characteristics, which typically are neglected if you don't have to do that. Personally, I would love to see more applications for doing the test reporting since writing the information, summarizing the information is something that is I personally I don't love to write Yeah. If I have to write something, it it's for me it's effort. So if I get support there, getting stuff together, writing, and that is a good application for a language model since it's a piece of text.

I think that would be nice. More support there. Yeah, let me think. What else?

Yeah, that's for I would say that for right now that that would be it That would be and an another thing, and perhaps let me just state that, I think with the advent of more autonomous systems I think the the gap between using AI and testing AI is closing again because the more autonomy you give to an AI agent, the better it has to be. So far We were talking about two different sides of the same metal, but I have the feeling now the two different paths converge again because people are now really looking at the quality of the output of the LLMs because it has to be good enough that you can automatically go into next processing step or at least allow the agent to continue doing it by himself.

And that was not necessary before because you had always the human in between. So all these agents when we're talking about agent farms or orchestrated teams, I think the quality of the agents themselves become more and more important. And from just using Gen AI to support testing, now we're moving a little bit back to testing Gen AI. Which given the hype that we have around Jennifer wasn't it that much in focus before.

But I think we're getting there now. I have the impression uh where we're closing that gap. Yeah we will see. So hop hopefully uh you're the Imbus um test bench developer here this episode and now making the module for test reporting more AI driven for you Claudia, thank you very much that you were here in the show uh that you gave us in some insights on how to use AI throughout the testing process in a concrete and specific way.

Yeah my pleasure, my pleasure. And yeah, I wish you a great conference. Next week is ICTQB meeting here in uh Mauritius. Two, a lot of work, a lot of work groups.

We will release the new AI testing syllabus. So yeah. A lot of AI related stuff too. Yeah, it's it's important.

Thank you very much. Thank you, Regime.