Why Traditional Testing Fails for AI Systems - Dušanka Lečić

Bugs are I I would say like invisible. Why invisible? Because they are not in the code In traditional testing everything is like pass and fail. Input output.

But uh here is uh uh very different. Like we have uh for the same input we have a lot of different outputs, some of them can be similar, but uh yeah still non-determinism is completely there. Uh first uh time that uh I uh saw the term stress testing In terms of uh testing Chad Bolt, I was like performance testing? No.

But no, of course, stress testing for Chad Bolt testing is uh uh like uh checking misspelling some biggest terms, some typos , some bad wording that usually user can use during some frustration moments. Yeah, some magic tool. I didn't find it actually that is doing everything instead of me. Yeah, I I'm joking a bit, but uh yes, I I think uh we are missing tools still.

Good tools that are doing the stuff. Welcome to Software Testing Unleashed, the podcast for testers, developers and test automation engineers and all the people in the software development process who want to create. Great amazing quality software. My today's episode was recorded at TestWarez, a great testing conference in Poland.

The days were full packed with great quality content, and I had a great time discussing and chatting with the Poland. And the international software test community. If you haven't been here, you should definitely go to one of the next Test Warriors conferences. You'll find a link to the conference in the show notes.

My guest today is Dusanka Lecic. She is a test lead in Department manager specialized in performance testing, agile methodologies, and engineering excellence. We talk today about how to test a chatbot. Dusanka shares with us her experiences on testing a chatbot, how to set Set up such a test, how to deal with prompting and so on.

And now enjoy the episode Hi Dusanka, great to have you here on the show. Thank you. Thank you very much. I'm happy to be here and to share some knowledge.

uh and uh my expertise. Yeah that's great. We had uh we I had the opportunity to have lunch with you today. Yes we had very nice time.

Yeah it was was great to to talk and chatting around And we are here at the Testvores uh comp com conference. It's uh a great nice conference. I like it very much here. I'm very happy to be here.

My first time. Yes. Uh I'm uh very pleased that I'm also for the first time here and uh People are really great, uh crowded. Sometimes, like on conferences, I really like crowd and it's amazing, different people different uh knowledges, uh different perspectives about everything.

I really like it here and uh this is not just My first conference in terms of representing my company Levi 9. I'm So and so on on the conferences and uh try to uh learn something from different guys that are here and also to share Yeah, yeah, that that's always a good good reason to to be on such a conference here. Yeah. I read in your abstract in the program here uh you were dealing with testing of chatbots And I said, ooh, we have to talk about this in the in the podcast too because uh it's it's it's the I think the primary reason uh uh when we see a chatbot and we think about our klassical traditional test cases Put something in, get something out that doesn't work.

Exactly. Uh for me it was very interesting, very challenging because uh I think majority of the people, especially in IT, uh thinking about the chat boats like Yeah, I put some question, I got some answer. I don't care about it, but we are not uh uh deep diving in it What if answer is not correct? And uh on some questions we are not familiar with the answers and Do not know is it correct or no.

And uh yes is uh very uh different uh from traditional testing. In traditional testing everything is like pass and fail. Input output. But here is uh uh very different.

Like we have uh for the same input we have a lot of different outputs. Some of them can be similar. But uh yeah, still non-determinism is completely there. So we uh have in traditional testing one valid case, others are like negative.

test cases but uh in uh this uh testing chatbot uh we have many valid test cases. And we need to be careful about it actually. So let let's uh structure that a little bit and and look into it. to to get good tests there and the good quality information out of that that is a very interesting question now for me because uh when I started to to test jackboat I was like like majority of the people, uh but then uh I figure out that uh I need to deep dive what is under that query under that question that I'm putting in the chatbot.

To deep dive in chunks, to deep dive uh in queries, in the in prompts. in chunking strategies, in uh everything that is uh completely, I think, uh still unfamiliar to majority And and how how did you do that? How was your approach to this? First approach was completely manually because I'm not familiar with it.

So I started from a small parts. Okay? What I want from this chat boat. I want to have some accuracy, correctness, some relevance.

I want to see that he's answering me correctly. I thought about uh first uh like some human needs, what I really want. I I want uh to not repeat some things uh I want uh to not be frustrated. That is I think the most important Because I I think uh everyone felt it in one moment with uh with chatbots.

But then of course after that uh there is a lot of technical things that uh we need to cover in the sense of uh testing chem checkboat sorry chatboats. So first of all uh I started to think about What is chunking? How I get chunks? Do I need to do it manually, automatically?

How to do it? Is it good enough or no? uh what are the semantic boundaries that we are putting uh like to uh improve our chunking strategies uh and so on so what are Uh actually uh when I first time uh uh saw a term rag system, I was like What is that and do I need it? Yes, of course.

Most of the chat boats uh is uh built uh on base uh of rag system and we need to uh have So retrieval logic, we need to test it. So not just response. Response is important, but also what we are retrieving. What is our prompt?

what we need to fix in one moment, uh how to trace, uh, uh how to retest. So a lot of things are uh Like were first unfamiliar to me and I started to discovering uh and it was uh starting to be Very interesting, like uh I uh found uh some uh good uh things that uh like uh were a challenge for me, uh like vector embeddings, how to test them. What I expect uh uh is that good enough or no? Uh how to write some uh test cases, how to produce negative test scenario Uh first uh time that uh I uh saw the term stress testing in terms of uh testing chat boat I was like performance testing No.

But no, of course, stress testing for chatbot testing is uh uh like uh checking misspellings, ambiguous terms, uh Some typos, some bad wording that usually user can use during some frustration moments. So you you you looked deep into into the the chatbot and and and clustered it in different areas. What you look so who When I what I understand is that you build your own test strategy by by researching what is inside this chatbot and how can you cluster that. Uh yeah I tried to divide it, uh not just like my own strategy, but uh I had some small like framework more than uh more like uh guidance uh for my testing what is really important for me like uh to preserve the context So if I have multi-turn interactions with chatboat, I want to remember what I said before.

I don't want to repeat Myself and I want to preserve the context. Context is very important. Not just content. We are uh we need to mix those things.

Uh also that is like uh uh that uh especially framework is a chat framework but yeah with dots like ch eight Like first of all, C is context retention that is really important age for hallucination control. So we really don't want to have some made up facts, some uh misleading uh answers. A for accuracy and relevance, so we need correctness, we need some relevant answers. I don't want uh any fact.

I don't want any answer. I want answer on my question and that is also really important. And T is uh for testing workflow. What is testing workflow here So to trace everything to fix what is wrong and of course to retest.

We need to retest things with some similar or same queries that we tested first time before fixing. And and uh w when I think I I didn't uh uh test the chatbot up to now, so but I think it's it's uh it could be sometimes not easy to say if an if the the result I get, not especially from Jetpot, but also insight. Uh if it's correct in the main of in the thing of our quality thinking. So uh maybe I I I I would suggest there are more answers I get out there than than I I get uh really uh uh knowledge out of that.

Yes you are really uh correct with with this uh thinking uh it is very hard to say yeah this is tested and this is okay and for me uh I think it's best uh to to use uh uh like a combined approach, uh hybrid approach, manual testing, so we need really to to know what is going on under the questions, under the queries, under the responses and retrievals so to understand everything uh to check everything it's very consumption like time consuming and uh of course uh we also have uh that uh automated part that is really important that makes our lives much easier.

And in terms also chatbot testing. Yeah yeah And how do you deal with documentation in this stuff to to make it uh repeatable and and to to know what you have done? in uh in your testing workflows and so it's also it's a bit harder than we usually uh think because now we have uh more uh valid uh test uh scenarios, uh more positive test cases, and uh yeah usually some uh tools that we can use uh in terms of uh creating test scenarios are useful. Uh do we automate it Yes, we always can automate also that part and uh make our life a bit easier in terms of creating creating documentation.

Because I think also our colleague here, one of the speakers, Mitko, mentioned uh and uh he had a very interesting story uh about AI also. Um so everything that You can use help of AI, use it. It really saves time. In terms of test plans, but It's not so different from traditional testing.

Majority, I think, like differences are actually in those valid test cases that we need. to have more than uh in traditional where is just one valid test case. Did you use any Literature research paper or something like that to get into the how to test a uh a chatpot Uh actually I read uh a lot of things. Uh I cannot even remember everything and uh Because I was not uh the only girl uh in in the team.

Uh we have uh more team members and we searched a lot of things like uh news some articles uh researcher researching uh papers uh some um slits, some uh konference speech, uh everything was in the game uh to get more and more understanding and information and trying to implement everything about it like about overlapping, about topkey scores that we have for different chunks, about those unknown things. that uh were unfamiliar to us. But a lot of things uh you need to be familiar first, to read it, to research it, to uh search of uh what kind of database we need, why vector database uh and what is so specific and so on And when you did your older test stuff, what were were the typical uh uh errors and failures, bugs you find in in in the chatbot?

Not at all, actually. You know, in traditional testing we are always finding errors in the code. Then we are doing our debugging uh and yeah of course opening the bug and uh putting some screenshots and uh describing everything that we can Uh but from the perspective of testing, Chadboat is uh completely different because uh bugs are I I would say like invisible. Why invisible?

Because they are not in the code. It can be also the code, of course But uh sometimes uh bugs in uh like in retrieving the information from the database uh bugs sometimes can be in uh prompts itself. because uh sometimes we have uh some bad structure of prompt but also sometimes we have uh bug in in generation of response. So it's uh very different uh from traditional testing.

And very very different bug types too, you you find okay. Yeah. And uh but uh w was it all easy when you when you find something maybe date from the data uh stuff or for the training data behind that uh what which was which was easy to fix uh or Something uh is easy to fix, but again something is not so easy to fix. So we need to really be able to log everything that we can Uh so uh traceability is very important here to log uh every information, not just query, not just response.

so to log also chunks to know why we retrieved uh some chunks, uh what was retrieved uh on which way uh so everything that we can that can help us in one moment To fix the bug. Sometimes we need to retrain our model, to update it, to start to figure out about some strategies to handle all those situations. uh but also yeah error can happen uh and uh we need also to have that in mind that uh if we doing everything to prevent it and it is happening that we can accept it uh and to uh have some lessons from there uh to maybe implement some uh fallback logic uh um for our chat boat.

Also maybe to implement some additional logic for questioning like uh to chat boat be able to uh ask a user Are you thinking about this or that? Uh to ask for uh more explanation what we are asking about. Also yeah to try to use all feedback that we have from user uh all errors to retrain our model to make it better. But yeah it's it's normal to have some errors in the end.

Yeah sure. That's that's that's I think that's a part why we test so uh Uh when we come to test documentation, did you use the same tools for uh as you use for uh writing your test cases and stuff uh like a Jira or something or an Excel spreadsheet or another test management tool? Or do you did you use different tools for Managing your test cases and um so most of the tools that we have now, test management tools, has some uh uh built AI like some add-in or uh in itself uh so basically we can use it.

But also we tried to automate though that process, a particular process, so to make it easier. uh to to type everything because it's very hard and uh time consuming. Yeah, yeah, that's true. Uh and do you have um in the process where or where in your now with with uh a a lot of experience about the chatbot testing Is there a tool that you missed all the time where you said that would be great if there would be a tool that helps me with this problem?

Yeah, some magic tool. I didn't find it actually. That is doing everything instead of me. Yeah, I'm joking a bit, but uh yes, I I think uh we are missing tools still.

uh good tools that are doing the stuff. Like here we used a lot of different tools to try it and one of the most successful tools was uh ragas uh for testing rags systems uh but uh I think uh that we uh cannot test everything like with it. just one part then a lot of manual testing again and uh I think it's not just in in the chatbot testing uh In majority things we can use AI in some parts, but in some where is hard infrastructure like subsystems or something like that. Maybe it will change during the next period, but for now we we miss some tools.

And may maybe it's it's because it's uh I think a very very new topic to test this stuff. They are just individual customized solutions and then they will sometime get uh become a product uh and and a suite for testing maybe a chatboard or something like that to combine these tools, but that's m maybe the the future. Yeah Yeah Dusanka thank you very much for this insights. Uh uh I think just Testing is a thing I I want to try it out someday.

Yes, I encourage you to try it. Just to play around with it and to see what is happening, to uh try to to learn Uh from it but also uh to try to train it a bit uh with questions and everything, I think It's very useful and interesting. Yeah. So thank you on this invitation also.

Yeah, thank you for your time and enjoy the rest of the conference and then uh Safe travel home. And to the next conference, I know you're going to a Yes. Next conference is expected to me And uh yeah, thank you on this invitation and uh also enjoy in the rest of the conference. Thank you very much.